DictionaryEvaluation
Humanity's Last Exam
A very hard exam of 2,500 expert questions across many subjects, built because AIs were acing older tests.
Definition
A benchmark of 2,500 expert-written questions across dozens of subjects, including mathematics, the humanities, and the natural sciences, in multiple-choice and short-answer formats with unambiguous, automatically gradable answers. Written by more than 1,150 contributing experts in response to the saturation of benchmarks like MMLU, it is usually reported without tools, and frontier models show low accuracy and poor calibration on it.
Example
A question might ask for a precise result in advanced number theory with one exact answer that is checked automatically, with no partial credit.