DictionaryEvaluation
GPQA
Very hard science questions that even smart people with Google struggle to answer, used to test expert-level AI knowledge.
Definition
A Graduate-Level Google-Proof Q&A benchmark (Rein et al., 2023): 448 multiple-choice questions in biology, physics, and chemistry written by domain experts. Experts with or pursuing PhDs reached 65% (74% excluding mistakes they later identified), while skilled non-experts with unrestricted web access reached 34% despite spending over 30 minutes per question. GPQA Diamond, the hardest high-quality subset, is the version usually reported for new models.
Example
A chemistry question about a reaction mechanism has four options; non-experts with web search get only about a third of such questions right.