All benchmarksImages and diagrams In plain words Example taskSource
MMMU
College exam questions that require reading images.
- Tasks
- 11,500
- Released
- Nov 2023
- Made by
- IN.AI Research, University of Waterloo, The Ohio State University and others
- Score
- Percent answered correctly
Like a university exam where every question comes with a chart, diagram, scan or sheet music you must understand to answer.
Read a Gallup chart of wellbeing scores and pick the probability that a score falls in a given range.
A question from college exams, quizzes or textbooks with one or more images. Mostly multiple choice, some short answers.How it is marked
The answer is compared with the correct one.
Humans: College senior students scored 76% to 89% on the validation set (worst to best) source
Why it matters
Real work mixes words with visuals. This checks whether AI can reason over both together.
Good to know
- Covers 30 subjects across art, business, science, medicine, humanities and engineering.
- The human experts were college seniors who could consult textbooks.
- A harder follow-up, MMMU-Pro, also exists; check which one a score refers to.