The Saturated Leaderboard

By the end of this session you will be able to:

• Put a confidence interval on any reported benchmark score and say whether a five-point gap is signal or noise
• Work out a benchmark's effective ceiling from its label error rate and its human baseline, and say how much headroom is left
• State exactly what a public benchmark result licenses you to claim about your own task

One point on GPQA Diamond is two questions

GPQA Diamond has 198 questions. That is the whole set. When a model card reports 93.0 percent, it means the model got 184 of them right, and 92.4 percent would mean 183. A single question moves the headline number by half a point.

The numbers you are comparing are estimates from a sample, and every estimate has a spread. The standard error of a proportion is sqrt(p * (1 - p) / n). At p = 0.90 and n = 198 that is 2.1 points, so the 95 percent interval around a reported 90.0 runs from 85.8 to 94.2. Two models reported at 88 and 93 have intervals that overlap across most of their range.

MMLU is bigger, about 14,000 questions, so its sampling noise is under half a point. Its problem is different, and worse.

The ceiling is not one hundred

Are We Done with MMLU? re-annotated the benchmark and estimated that 6.49 percent of MMLU questions contain errors: wrong gold answers, unanswerable questions, multiple correct options. In the Virology subset, 57 percent of the analysed questions had errors.

If roughly one question in fifteen has a broken label, the highest score a perfectly correct model can get is around 93.5 percent, and every point above that is the model agreeing with a mistake. Frontier models report MMLU in the high eighties to low nineties. They are already inside the band where the benchmark cannot tell a right answer from a wrong key. A model scoring 91 instead of 88 may simply have learned the errors better.

GPQA has the opposite shape and the same conclusion. Domain experts who hold or are pursuing PhDs in the relevant field reach 65 percent on it, 74 percent discounting mistakes they identified in retrospect. Skilled non-experts with over thirty minutes and unrestricted web access reach 34 percent. So a model at 90 percent is not "smarter than a PhD" in any sense you can ship. It is answering four-option questions in a format the experts were not optimised for, on a set small enough that the top of the range is a handful of items wide.

Before you compare two scores, ask what the ceiling of that benchmark actually is. Above it, higher is not better, it is just differently wrong.

Do the arithmetic before you believe the gap

Here is the whole test for a reported five-point spread on GPQA Diamond.

import math
 
n = 198                      # GPQA Diamond has 198 questions
score_a, score_b = 0.880, 0.930
 
def stderr(p, n):
    return math.sqrt(p * (1 - p) / n)
 
for name, p in (("model A", score_a), ("model B", score_b)):
    s = stderr(p, n)
    print(f"{name}: {p*100:.1f}%  95% CI [{(p-1.96*s)*100:.1f}, {(p+1.96*s)*100:.1f}]")
 
se_gap = math.sqrt(stderr(score_a, n) ** 2 + stderr(score_b, n) ** 2)
gap = (score_b - score_a) * 100
half = 1.96 * se_gap * 100
print(f"gap: {gap:.1f} points, 95% CI [{gap-half:.1f}, {gap+half:.1f}]")

That prints a gap of 5.0 points with a 95 percent interval of -0.8 to 10.8. The interval contains zero, which means the data are consistent with model A being the better one.

You can object that this test is too conservative, and you are right. Both models answered the same 198 questions, so the correct test is paired: McNemar's test on the items where the two models disagree. Suppose model B gets 16 items that A misses and A gets 6 that B misses.

import math
 
b, c = 16, 6                 # B correct/A wrong, A correct/B wrong
chi2 = (abs(b - c) - 1) ** 2 / (b + c)
p = math.erfc(math.sqrt(chi2 / 2))
print(f"discordant items: {b + c}, net gap: {b - c} items")
print(f"McNemar chi2 = {chi2:.2f}, p = {p:.3f}")

That gives p = 0.055. The stronger test still does not clear the bar, and here is the part that should decide the argument for you: you cannot run it anyway. McNemar needs per-item results for both models. Leaderboards and model cards publish one number per model. The paired test is unavailable to you, so the conservative unpaired bound above is the best you have, and a five-point spread does not survive it.

The benchmark is not your task

Suppose the gap were real. GPQA Diamond measures the ability to pick one of four lettered options on graduate biology, physics and chemistry questions, scored by matching a letter. If your product reads a customer thread, decides whether a refund applies, calls a refund tool with the right arguments, and declines when policy says no, then not one of those 198 questions exercises argument construction, tool selection, or refusal calibration. A benchmark result is evidence about the benchmark. Transfer to your task is an assumption, and it is the assumption that keeps failing.

Score also depends on the harness. Prompt format, whether chain of thought is allowed, how the answer is extracted from free text, and how many times each item is run all move the number by several points. Two labs both reporting "GPQA Diamond" are frequently not reporting the same measurement.

If you want a number you can reason about, produce it yourself with a reported standard error. Inspect AI 0.3 does this by default:

pip install inspect-ai inspect-evals
export ANTHROPIC_API_KEY=sk-ant-your-key-here
inspect eval inspect_evals/gpqa_diamond --model anthropic/claude-opus-5 --epochs 3

Inspect prints accuracy with its standard error next to it, and --epochs 3 runs each item three times so you see run-to-run variance rather than a single lucky pass. GPQA is gated on Hugging Face, so accept the terms and set HF_TOKEN first. You still end up with a number about GPQA, not about your product, but at least it carries an error bar.

Try it

Open the model card for the model you use today, and for the model you are considering switching to. Copy their GPQA Diamond scores into score_a and score_b in the first script and run it.

Then find the threshold: hold score_a fixed and raise score_b until the printed interval on the gap stops containing zero.

Success condition: you can state, in one sentence, the smallest GPQA Diamond gap that clears 95 percent confidence at n = 198 near the top of the range (it lands around 6 points), and whether the gap you were about to act on clears it. If it does not, you have just avoided a migration you had no evidence for.

Common mistakes

• Reading a leaderboard delta as a product decision. It is an estimate on a small sample, from a task that is not yours, scored by a harness you did not control. Treat it as a weak prior on general capability and nothing more.
• Assuming the ceiling is 100 percent. With 6.49 percent of MMLU labels broken, the ceiling is about 93.5, and scores near it are measuring agreement with mistakes. Always look up the benchmark's label audit and its human baseline before reading a gap near the top.
• Comparing numbers from two different reports. Different prompt templates, answer extraction, and epoch counts move a score by more than the gap you are trying to detect. Compare only numbers you produced under one harness.
• Running the eval once and calling it a measurement. A single pass gives you no variance estimate. Use --epochs and report the spread, or you will chase differences that vanish on the next run.
• Concluding that benchmarks are useless. They work as a floor check: a model that fails GPQA badly will not do your hard task either. They fail as a ranking between two models already near the ceiling.

Where this goes next

You now know that a saturated score cannot rank models for your task. The next session, Contamination, Measured, asks a harder question about the same scores: whether the model has seen the test set already. You will run canary strings, n-gram overlap and verbatim recall probes against the benchmark you were about to trust.