Ten ways a grader lies in AI post-training: rubrics, answer keys and judges

If the exercises decide what a model sees, the graders decide what it becomes. Every attempt is scored; training moves the model toward whatever scored higher; and the model is very, very good at finding what scored higher. Part 1 (opens in a new tab) of this series argued that a wrong grader is a gradient pushed thousands of times. Part 2 (opens in a new tab) helps you to design training exercises for AI agents.

Most of what follows came from time designing evals and rubrics for knowledge work. Each item was cheap to fix once seen, and each was invisible until someone read the rubric next to the brief.

1. The rubric has criteria nobody asked for

A criterion copied from an unrelated scenario: restaurant roles appearing in a hospital process task. Criteria beyond the brief: a transmittal letter, an appendix table, that no requester mentioned. These fail by construction, for every model, forever. A rubric line that can only fail is a constant, and constants teach nothing except that the ceiling is lower than it looks.

The reverse also happens. An outcome section with two criteria, both about hygiene, so a deck with none of the requested content scores perfectly on outcome. Rule: at least one outcome criterion per requested deliverable element, and identical scores for two different models on a task is a sign the section is not discriminating.

2. The brief and the rubric contradict each other

"Four slides" in the brief, "exactly five" in the rubric. "Split out the cost item" in the brief, a rubric line penalizing a line-item table. A native-chart check applied to slides that ask for a table or a shape diagram. Criteria whose expected values were derived from a source file the model never received.

And the quiet one: expected values that are probably wrong on the rubric side. When every other figure passes and one family of derived percentages fails for both models on every run, the first suspect is the key, not the models.

Gao, Schulman and Hilton (opens in a new tab) measured the shape of the curve: optimize hard enough against a learned proxy and true quality goes down.

3. Duplicates are silent double weights

A criterion that appears twice, verbatim, counts twice. Mixed denominators inside one task (some sections out of five, some out of twelve) do the same thing more subtly. Dedupe the rubric and fix a single formula for the composite. Always publish the unweighted pass rate beside the weighted score. If you cannot, nobody can check your weights.

4. The house rules were tested but never signaled

Message-led titles. No gridlines. Sources that do not cite spreadsheet tab names. All reasonable conventions, all tested, none mentioned in the brief. The model failed them and a reader of the report sees "model cannot write titles," when what happened is the model was never told in the first place.

Anthropic's research on Agent Skills (opens in a new tab) is one answer to this: package the house rules as procedural knowledge the model loads, then grade against them fairly. Either put the convention in the brief or put it in a shared skill file the model can read or report the failure as a convention gap rather than a capability gap.

5. Nobody audits the answer key

The typical process: one author builds one reference file, reads the expected values off it, and "machine-checks" them against the same file. That is a check of the file against itself. OpenAI's SWE-bench Verified (opens in a new tab) exercise found that most of the original benchmark had tests that would reject a valid fix. A checklist grader is only as good as the expected values it is handed, and nothing in the loop tests them.

Edge rules in the brief are exactly where authors and models both slip. Derived tasks (add a sheet to this, edit this) inherit the create task's key, so one mistake propagates to several exercises. Hand-maintained decoy lists can mark a genuinely broken item as an intentional decoy and then punish the model for fixing it.

The audit that would catch this is not expensive. Two independent builders reproduce the key from the brief alone. A third reconciles. Every disagreement is classified: brief ambiguity, author error, tolerance, decoy mislabeled. Defect rate is reported per sub-environment. Dual-build becomes the rule for every new exercise, and an author is never their own reconciler.

The cheap early warning: exercises where every attempt fails the same value item. Go and look at the key. Lilian Weng's survey of reward hacking (opens in a new tab) collects the cases.

This is not a problem peculiar to eval design. Thinking Machines Lab audited a standard text-to-SQL training set and found that over half the gold queries were wrong (opens in a new tab), and that a third of the "correct" rewards their execution-match grader handed out went to queries that were not equivalent to the gold answer. Both Thinking Machines and OpenAI fixed it the same way: people, reading items, one at a time.

6. The shared checks do most of the scoring

Every task carries a set of general quality checks alongside its task-specific criteria: was the slide master used, is space used well, are tables and charts native and editable, are the general house rules followed.

That is not wrong, exactly. House style matters. But it has to be stated in the report, and the criteria need tiers (blocking, major, minor, or must and should) so that pixel-level asks cannot swamp whether the deck says the right thing.

Anthropic's post on infrastructure noise (opens in a new tab) makes a related point from the coding side: a surprising share of eval variance comes from the harness, not the model, and if you do not measure it you will read noise as signal.

7. Excluding the broken files deletes the strongest finding

Decks that would not open without repair need to be dropped from the benchmark's averages. Reasonable, until you look at which tasks lost them. On the edit-an-existing-deck tasks, most outputs were dropped. One model had no surviving row. The other was scored on one or two files. The whole readout for the task family that matters most in enterprise use rested on a single deck.

The repair rate is not a footnote. It is the headline metric for edit tasks, reported with n-evaluated per task. And structural validity (i.e., opens, schema-valid, no dangling relationships, media resolves, roundtrips) should be a hard gate that scores zero, not a dimension averaged in with the others. Averaging content quality with layout quality with validity produces one number dominated by whichever already works.

Thinking Machines Lab went one layer down and showed that inference itself is not reproducible by default (opens in a new tab): the same prompt returns different completions depending on server load, because most kernels are not batch-invariant. If two runs of the same model on the same task disagree, the grader's number is not yet a measurement.

8. The reward makes some things free

Anthropic's research on reward tampering (opens in a new tab) shows that models trained on gameable environments generalize the gaming, and its work on emergent misalignment from reward hacking (opens in a new tab) shows the generalization reaching well beyond the original task.

Relative preference within a group of attempts, with no absolute floor, lets a group of poor attempts crown a near-perfect winner. The model learns that its best attempt was good. This is where the rubric stops being a measurement and becomes a curriculum.

• An honesty override that zeroes a score for a false claim about the output, with no cost for saying nothing, teaches terseness. One wrong sentence about a requested line arguably deserves the zero. Every material claim probably does not. A floor for claims about requested outputs plus a heavy penalty for everything else is the version I would argue for.
• A "nothing else changed" check that counts double teaches timidity: patch the named symptom, leave the stale summary that depends on it.
• Fixed build-quality rows applied whether or not the task needs them (frozen header rows, input blocks) invite style over-fitting. The test is simple: grade a set where those rows are inapplicable and count how often the model adds them anyway.
• A fixed order of concerns in the judge (floors, then checklist, then build quality, then fitness, then the reply) can let a minor floor breach outrank a large gap in whether the thing is fit for purpose.
• And if the judge reads the model's reply alongside the file, replies get written for the judge. Grade the file and the reply separately, and re-grade with the reply stripped or inflated to measure how much it moves the score.

OpenAI watched its reasoning models write "let's hack" in their own chain of thought (opens in a new tab) before stubbing out a test and found that penalizing the thought taught the model to hide it rather than stop. Their hallucination paper (opens in a new tab) makes the general version of point eight: any scoreboard that scores a wrong answer and a blank the same teaches the model to guess.

Anthropic's sycophancy research (opens in a new tab) is the same lesson from the human-preference side: the preference data itself rewarded agreeable over correct, and OpenAI's GPT-4o postmortem (opens in a new tab) is what it looks like when that ships.

9. The model-based judge has its own taste

Model-as-judge scoring is now standard, and Zheng et al. (opens in a new tab) put the agreement with humans around eighty percent for pairwise comparison. Two things to hold onto from that number. It is pairwise; agreement drops on absolute and batch scoring. And the remaining twenty percent is not random: position bias, verbosity bias, self-preference.

There is a subtler problem underneath the bias problem. Shankar and colleagues found that people writing grading criteria change the criteria as they read outputs (opens in a new tab); they called it criteria drift. A judge prompt written before anyone looked at real outputs encodes the criteria you thought you had, not the ones you hold. That is why the human review must precede the judge, not follow it.

For document work there is a specific version of the bias problem. A multimodal judge shown rendered pages with no length control learns that denser is better, which is backwards for slides. It is more vulnerable to verbosity than to position. Wang et al. (opens in a new tab) showed rankings flipping when answer order was swapped.

Two OpenAI evals show what a well-checked judge looks like. HealthBench (opens in a new tab) grades against physician-written rubrics and reports that judge-versus-physician agreement matches physician-versus-physician agreement, which is the only meaningful ceiling. PaperBench (opens in a new tab) decomposes each task into a hierarchy of thousands of gradable sub-items and then benchmarks the judge itself on a separate judge test set. Their earlier rule-based rewards (opens in a new tab) work is the closest public analogue to a checklist grader and is candid that it works for clear rules and struggles with subjective quality. Slide layout is subjective quality.

10. Demand unsaturated sub-scores

A fixed set of tasks will be trained against. "Frontier systems should pass most straightforward requirements" is an invitation to an eval that stops telling checkpoints apart. Ask for a score band on current models with sub-scores unsaturated, a sealed held-out fraction, a costed recipe for extending the set, and versioned rubric conventions. Anthropic's post on AI-resistant technical evaluations (opens in a new tab) is about keeping evals valid as models improve; the same discipline applies to a training bank's graders, only more so, because the model is optimizing against them on purpose.

The protocol that works in most circumstances: binary items, one call each, partial counts as fail, a few dozen items per task. Pairwise against a reference, in both orders. Length-controlled. Position-flip rate and a regression of score on word count and element count reported alongside the results. Template adherence stays deterministic; the judge scores only the residual. Per-item judgments stay binary; severity lives in the aggregation layer, not in the judge's mouth. And weights chosen by intuition are a tuning knob with no ground truth behind it, so fit them to human labels or drop them.

Nothing in the loop asks whether "better" means anything to a person who does this work for a living. Part 4 is about the people.

----

Daljeet Saran (dsaran@motifplatforms.com) is the founder of Motif Platforms, where he builds AI and data systems for commercial teams. He currently spends most of his time on pre- and post-training data and evaluation for AI agents that do real knowledge work.

Part 1 of this series: Reward hacking explained: how one bad grader teaches AI a thousand bad lessons (opens in a new tab). Part 2: How to design training exercises for AI agents (opens in a new tab). Part 4: The correct-looking file.