BuildSeniorData / MLAuto-graded

Score a model's answers into a pass rate

Fix an eval harness so its pass rate is right, an empty dataset doesn't crash it, and one model error doesn't stop the run.run_eval(dataset, model, scorer) in eval_harness.py runs a model over a labelled dataset, scores each output and reports a pass rate. Right now the pass rate and counts don't agree with what the scorer says, and a model that raises on one case takes down the whole run. Make the report match the scorer case by case. When the model raises, record that case as failed and carry on.

35 min

python

Start lab

You're done when

A case whose model call raises doesn't stop the run.
That case shows up in the report as failed. It isn't skipped or counted as a pass.
The reported pass rate equals passed divided by total.
The passed and total counts match the per-case results.
Each case's pass or fail flag matches what the scorer said for it.
An empty dataset reports a pass rate of 0.0 with zero passed and zero total, without an error.

What it measures

AI evaluation and QATurning model outputs into a pass rate you can trust.

AI system designHow a model error is counted, and what that does to the pass rate.

Consistent countsWhether the totals and the rate agree with the per-case results.

Your environment

Terminal + editorA full shell in the browser

Claude CodePre-authenticated, no key setup

Isolated microVMYours alone, wiped afterwards

Start this lab

35 min

Start labYou'll sign in to Anthropos to start. The lab opens in a private cloud workspace with Claude Code ready to use.

AI Labs is part of Anthropos Workforce. Not on your plan yet? Book a demo