Benchmarks

A benchmark is a fixed set of tasks with an automatic grader. The score is the share of tasks the model gets right. Run it and watch: the same model can land on different scores.

In plain wordsLike a driving test with 20 manoeuvres: a good driver can still fail one on a bad day. More manoeuvres give a fairer picture.

score = \frac{tasks passed}{tasks in the set}

One run is one sample. Small sets wobble more.

True skill70%

This run—

True skill70%

Last runs

pass@1 and pass@k

Coding benchmarks often let a model try several times. pass@k is the chance that at least one of k tries is correct. Labs estimate it from n samples, c of which were right.

In plain wordsLike letting a student retake a quiz five times and keeping the best try. Useful, but not the same as getting it right first time.

pass@ k = 1 - \frac{( k n - c )}{( k n )}

pass@5 always looks better than pass@1. Only compare like with like.

Samples n20Correct c6Tries k5

pass@130.0%

pass@587.1%

How sure is a score?

With N questions and a score p, the standard error tells you how much the score would move if you drew a fresh set of questions. The 95% interval is about two standard errors either side.

In plain wordsLike polling 200 people before an election: the result is close to the truth, but a small lead may be luck.

S E = \frac{p ( 1 - p )}{N} 95% interval \approx p \pm 1.96 \cdot S E

On a 200-question test, a two-point lead is often inside the noise.

Model A78%Model B80%Questions N200

72.3% to 83.7%

74.5% to 85.5%

A 2-point gap on 200 questions could be noise.

Same name, different test

Two numbers with the same benchmark name can come from different setups. Before comparing, check these five things.

In plain wordsLike comparing two runners when one wore spikes and the other ran uphill. Same race name, different conditions.

Compare within one column and one setup, never across.

0 of 5 checked. Tap each one you have confirmed.

Saturation and contamination

When top models all score near 100%, a benchmark stops telling them apart (saturation). If test questions leak into training data, scores rise without real skill (contamination). Newer, harder or private sets are the answer.

In plain wordsLike a school exam everyone has already seen the answers to: high marks stop meaning much.

A high score on an old public test means less than it used to.

Illustrative curve, not real data

Points per dollar

The best model for you is rarely the top scorer. Divide the score by what one request costs to see value, then check that the cheaper model is good enough for your task.

In plain wordsLike choosing a car: the fastest one is not the best buy if a cheaper one gets you there just as well.

value = \frac{score}{cost per request}

Pick the cheapest model that clears your bar, not the highest bar.

Model AScore82%Cost per request$0.60

Model BScore74%Cost per request$0.12

Model A

137 points per $

Model B

617 points per $

Model B gives 4.5× more score per dollar.

Ready to compare real models?

Open Compare