BuildSeniorData / MLAuto-graded

Score a model's answers into a pass rate

Measuring AI output quality honestly is the skill here: scoring arithmetic that still reports truthfully when the model fails.Evals are how you actually know an AI system works — before you ship a prompt, a retrieval pipeline or an agent, you need a loop that runs the model over a labelled dataset, scores each output and reports a pass rate. That harness is the most reused tool in AI engineering, and the one in eval_harness.py is broken. Make run_eval correct and resilient, including when a case raises rather than returning an answer.

35 min

python

Start lab

You're done when

A case whose model call raises does not bring down the whole run.
That raising case is recorded as failed rather than skipped or counted as a pass.
The reported pass rate is arithmetically correct.
The passed and total counts agree with the per-case outcomes.
Each case carries an accurate individual pass or fail flag.

What it measures

AI evaluation and QABuilding the measurement loop that makes an AI system's quality a number rather than an impression.

AI system designDeciding what an error should count as — a question that changes what the pass rate means.

Robustness in aggregationKeeping counts and rates consistent with per-case results, including on the empty and all-failing edges.

Your environment

Terminal + editorA full shell in the browser

Claude CodePre-authenticated, no key setup

Isolated microVMYours alone, wiped afterwards

Start this lab

35 min

Start labYou'll sign in to Anthropos to start. The lab opens in a private cloud workspace with Claude Code ready to use.

AI Labs is part of Anthropos Workforce. Not on your plan yet? Book a demo