Which AI models can you trust with your board papers?

What it measures

A general knowledge exam of roughly sixteen thousand multiple choice questions across fifty seven school and university subjects, from elementary mathematics to professional law. It was the standard headline number in vendor announcements for years. It tests what a model absorbed during training, with no documents supplied.

Why it does not answer the board question

It asks about the world, never about your documents. A model can score at the top of MMLU and still invent a figure in your quarterly pack, because the exam never gives it a document to be faithful to. It is also multiple choice, so a model that guesses well looks identical to a model that knows.

What it measures

The harder rebuild of MMLU. It raises the number of answer options per question from four to ten and filters out the easy items, so guessing is worth less and step by step reasoning is worth more. It is the knowledge benchmark most vendors now quote instead of MMLU.

Why it does not answer the board question

It is still an exam about general knowledge, still multiple choice, and still document free. It tells you whether a model is well read. It says nothing about whether it will cite the right page of your own management report, or admit that a quarter is missing from the pack.

What it measures

Graduate level science questions in biology, physics and chemistry, written by PhD holders and deliberately made hard to look up. The Diamond subset holds the 198 questions that two domain experts answered correctly and that most skilled non experts got wrong even with the internet open. It is the standard proof that a model can reason at expert level in science.

Why it does not answer the board question

Board work is not graduate chemistry. Nothing in GPQA asks a model to stay inside a supplied document set, to cite a source, or to say that it does not know. A model that reasons brilliantly about reaction mechanisms can still be the model that confidently misreads your covenant schedule.

What it measures

Two and a half thousand questions at the outer edge of expert knowledge, crowdsourced from academics across more than a hundred subjects and filtered so that models of the day could not answer them. It was built as a deliberately unsaturated replacement for MMLU and GPQA. The work was published in Nature in January 2026.

Why it does not answer the board question

It measures the ceiling of specialist knowledge, which is the opposite end of the scale from board work. Directors do not need a model that knows the frontier of topology. They need a model that reads forty ordinary documents without embellishing them, and Humanity's Last Exam never puts a document in front of the model.

What it measures

Small coloured grid puzzles where the model must infer the rule from two or three examples and apply it to a new grid. It was designed in 2019 to measure fluid intelligence, which is the ability to solve a problem you have never seen, rather than recall. It resisted scale for five years, which is exactly why it became famous.

Why it does not answer the board question

It is a puzzle about visual rules with no text, no documents and no sources. It is a useful signal about raw novel problem solving, and it is silent on every property a board cares about: faithfulness to your papers, traceability, and willingness to say nothing when the answer is not there.

What it measures

The 2025 rebuild of the grid puzzle test, with a thousand training tasks and three hundred and sixty evaluation tasks split across a public, a semi private and a private set. Every task was solved by at least two people in under two attempts, so a human baseline exists. Keeping part of the set private is a deliberate defence against training on the test.

Why it does not answer the board question

Still no documents, still no citations, still no way to score honesty. ARC-AGI-2 answers whether a system can adapt to something genuinely new. A director's question is whether a system can be boringly reliable on something entirely familiar.

What it measures

Launched on 25 March 2026, this is the first interactive test in the series. Instead of static puzzles the model is dropped into hundreds of hand built turn based environments with no instructions, no rules and no stated goal, and has to work out by experiment what winning looks like. It measures learning from experience rather than answering from knowledge.

Why it does not answer the board question

It is the most honest measure yet of how far frontier systems are from human style learning, and it is entirely orthogonal to document work. It tells a board how much headroom the technology still has. It does not tell a board which model to point at the next board pack.

What it measures

Five hundred real bug reports from open source software projects, hand checked so that each one is solvable and each test is fair. The model gets the codebase and the issue, and has to produce a patch that makes the project's own tests pass. It became the reference number for agentic coding ability.

Why it does not answer the board question

It is coding only. Board papers are not code, and the pass or fail criterion is a unit test, which is exactly the thing board work does not have. A model that is excellent at patching Python has no demonstrated ability to keep to a house style, cite a page number, or refuse to answer.

What it measures

Whether an AI agent can drive a real command line to finish real technical work end to end: compile a project, configure a system, train a small model, recover a broken environment. Unlike SWE-bench it scores whole workflows rather than a single patch. Version 4.0, released in 2026, holds sixty six tasks after retiring the ones frontier agents had already mastered.

Why it does not answer the board question

This is a benchmark for the CTO's engineering platform, not for the boardroom. It measures whether an agent can operate infrastructure. It measures nothing about reading a management report faithfully, and its verifiers are scripts, which board documents do not come with.

What it measures

Whether an agent can handle a real customer conversation while obeying a written policy and using tools correctly. A simulated customer talks to the agent across several turns in retail, airline or telecom scenarios, and the agent must change the right records without breaking the rules it was given. The second version adds cases where the customer also acts, not just the agent.

Why it does not answer the board question

It is the closest public benchmark to policy discipline, which is one of the five things a board cares about, but the policy is a short service script and the world is a small simulated database. A board's house rules are longer, vaguer and applied to prose, and nothing here scores whether an answer is traceable to a document.

What it measures

Whether an agent can persist on the open web until it finds a fact that is genuinely hard to locate. It holds 1,266 questions built by working backwards from a verified fact, so the answer is difficult to find and easy to check once found. It measures stamina and search creativity rather than knowledge.

Why it does not answer the board question

The whole point is the open internet, and the whole point of a board assistant is the closed document set. A model that is brilliant at finding a stranger's paper online is not thereby a model that stays inside your own quarterly pack and refuses to fill gaps from the web.

What it measures

Whether models can produce the actual deliverables of paid professional work. It covers 1,320 tasks drawn from forty four occupations across the nine sectors that contribute most to United States GDP, written by practitioners with an average of fourteen years of experience. Outputs are real work products: documents, slide decks, spreadsheets and diagrams, graded by expert reviewers against the professional's own version.

Why it does not answer the board question

This is the closest any large public benchmark gets to office work, and it is still not the board question. It grades whether the deliverable is good, not whether every claim in it is traceable to a supplied document. A convincing memo with one invented figure can score well on quality and still be the thing that costs a CFO an audit finding.

What it measures

Whether an AI agent can do multi week knowledge work inside a messy pile of company files. Four projects covering data science, product management, banking and strategy contain ninety one linked tasks and thousands of input files: Slack exports, emails, meeting transcripts, spreadsheets, PDFs and board materials. Each task is scored three ways: a pass or fail rubric for correctness and evidence use, and two head to head ratings for analytical quality and presentation.

Why it does not answer the board question

Of everything on this page, this is the nearest neighbour: real files, real deliverables, an explicit check on evidence use. It still does not isolate the five properties a board must sign off on. There is no separate score for how often the model invents, no separate score for whether it says it does not know, and no score for holding house rules over a long conversation. It also runs in English on generic corporate material, not on Dutch board papers.

What it measures

Whether a model can answer the question a professional would actually type while working inside a specific long PDF. It holds one hundred tasks across ten professional domains, grounded in 4,592 source pages and graded against 1,275 separate criteria. The headline score is the share of attempts where every single criterion passes, which is a deliberately strict bar.

Why it does not answer the board question

It is the right shape and the wrong subject. The documents are professional PDFs, not a board's own mixed pack of minutes, spreadsheets, slides and contradictory drafts, and each task is one question about one document rather than a conversation that spans forty of them. It also does not score whether the model admits that something is absent.

What it measures

The real usable length of a model's context window, as opposed to the number printed on the box. It generates synthetic tasks at controlled lengths in four kinds: retrieval, tracing a fact through several hops, aggregating many items, and long form question answering. The output is the length at which a model stops performing, which is what a director should be told.

Why it does not answer the board question

The haystack is synthetic filler, not your minutes and your annual accounts. Real board documents share vocabulary, repeat each other and contradict each other, which is a harder and different problem than finding planted variables in noise. RULER gives you a ceiling, not a behaviour.

What it measures

Long context ability when the answer cannot be found by matching words. The classic needle in a haystack test lets a model spot the sentence that shares vocabulary with the question. NoLiMa removes that shortcut by writing needles that share almost no words with the question, so the model has to make the connection by meaning.

Why it does not answer the board question

It is the sharpest available warning that advertised context windows overstate real capability, and it is still a retrieval puzzle in fiction. It does not test whether a model attributes what it found to the right document, nor whether it stops when the link is not there.

What it measures

Deep understanding and reasoning over realistic long documents, rather than retrieval. It holds 503 hard multiple choice questions over contexts from eight thousand to two million words, in six categories including multi document question answering, long dialogue history and long structured data. Every item has annotated evidence so a grader can check where the answer came from.

Why it does not answer the board question

It is multiple choice. A board assistant writes prose, and the entire risk lies in what it writes when nobody offers it four options. Picking the right option among four says nothing about whether the model would have invented a fifth.

What it measures

How well a model still understands a story as the story gets longer. Thirty stories are cut into versions of increasing length that preserve the same key details, and the same thirty six questions are asked at every length. Answering requires tracking who knew what when, which is comprehension rather than search.

Why it does not answer the board question

The material is fiction from a creative writing community, and its shape is a narrative with a timeline. Board material is a heterogeneous pack of minutes, tables, policies and slides, where the hard part is not following a plot but resolving two documents that disagree.

What it measures

Not a leaderboard but a controlled study, and one of the most useful pieces of evidence a board can be handed. Chroma tested eighteen models across a range of input lengths and showed that performance does not hold steady and then fall off a cliff at the limit: it degrades gradually and unevenly from far below the advertised window. The term context rot comes from this work.

Why it does not answer the board question

It is a study of one failure mode, not a ranking you can buy from. It tells you that a large context window is not a promise, which is the single most important correction to vendor marketing a board can absorb. It does not tell you which model to choose, and it says nothing about citation or honesty.

What it measures

Reasoning across several real long documents at once, deliberately built to look like knowledge work rather than a synthetic puzzle. One hundred hard questions each come with around a hundred thousand tokens of input drawn from company reports, industry reports, government consultations, academic papers, legal texts, marketing material and survey reports. Answers cannot be looked up in one place; they have to be assembled.

Why it does not answer the board question

It measures whether the model can reach the right answer across documents. It does not measure whether the model shows you which page it came from, and it does not include questions whose answer is deliberately absent, which is the test that separates a careful assistant from a confident one.

What it measures

Whether a model's long form answer stays inside the document it was given. It holds 1,719 examples, half public and half held back, each pairing a document of up to about thirty two thousand tokens with a request such as summarise, rewrite or answer questions. Answers are judged on two things at once: did it follow the request, and is every statement supported by the document.

Why it does not answer the board question

This is the closest public benchmark to the factuality metric a board needs, and it stops one step short. It checks that statements are supported by the document, but it does not require the model to point at where, so an answer can pass without being traceable. It also works with one document at a time, while a board pack is a set of documents that disagree with each other.

What it measures

How often a model adds something to a summary that the source document does not say. Each model summarises the same set of documents and a separate evaluation model, HHEM, checks each summary against its source. The published number is the share of summaries judged unfaithful, so lower is better, and it is one of the few public numbers that reads like a risk figure rather than a grade.

Why it does not answer the board question

It measures one narrow behaviour, summarisation of a single short to medium document, and Vectara says so explicitly. A board assistant answers questions across dozens of documents, which is a different and harder failure surface. A low hallucination rate here is necessary and nowhere near sufficient.

What it measures

Whether a model knows the facts it claims to know, and whether it will admit when it does not. It holds 4,326 short questions with one indisputable answer each, collected adversarially so that the models of the day got them wrong. Crucially it scores three outcomes rather than two: correct, incorrect, and not attempted.

Why it does not answer the board question

The not attempted category makes SimpleQA the nearest public relative of the honesty metric a board needs, and it is about world knowledge, not about your documents. Being willing to say I do not know about a historical date is not the same as being willing to say a figure is not in the pack when a director is waiting for an answer.

What it measures

A cleaned rebuild of SimpleQA: a thousand prompts filtered for duplicate questions, topic imbalance and wrong answer keys, with an improved grading prompt. It exists because the original set had enough label noise to distort comparisons between close models. It measures the same thing, more reliably.

Why it does not answer the board question

It is a better ruler for the same wrong wall. Parametric knowledge, meaning what the model remembers from training, is exactly the thing a board assistant should not be relying on when it answers a question about the company's own numbers.

What it measures

Whether a model does exactly what it was told, using instructions a computer can check without judgement. Around five hundred prompts carry twenty five types of verifiable instruction such as write more than four hundred words, answer in JSON, or begin every paragraph with a question. Scoring is mechanical, so there is no judge and no room for taste.

Why it does not answer the board question

It is a single instruction, checked once, in one message. Board discipline is a set of house rules that has to survive twenty follow up questions and a user who pushes back. IFEval never tests whether the rule is still being followed at question fifteen.

What it measures

Whether instruction following generalises to constraints the model has never been trained on. It introduces fifty eight new and deliberately unfamiliar verifiable constraints, precisely because models had overfitted to the small set used by IFEval. It was accepted at NeurIPS 2025 in the datasets and benchmarks track.

Why it does not answer the board question

It is the sharpest available evidence that a high instruction following score can be an artefact of training rather than a capability, which is directly relevant to a board that plans to give an assistant house rules. It still tests single turn compliance with mechanical constraints, not sustained adherence to a policy written in prose.

What it measures

Whether a model holds itself together over a long conversation. It scores four things that break in practice: keeping an instruction given in the first message alive to the end, remembering details the user mentioned in passing, editing a document reliably across several rounds of revision, and staying coherent instead of agreeing with whatever the user last said.

Why it does not answer the board question

This is the closest public work to the discipline metric a board needs, and it is measured on general conversation, not on a document set with house rules. Nothing in it checks whether the model's claims trace back to a source, and its self coherence category is about not flattering the user rather than about not inventing a number.

What it measures

Which answer people prefer. Visitors type a prompt, see two anonymous answers side by side, and pick the better one; the votes become an Elo style rating. It is the most cited public ranking in the industry and it measures taste at scale, not correctness.

Why it does not answer the board question

A vote records which answer looked better to an anonymous person who has no way to verify it. That rewards confident, well formatted, comprehensive prose, which is precisely the presentation style a fabricated figure arrives in. Nothing in the mechanism can distinguish a correct answer from a persuasive one, and no voter has your documents.

What it measures

One number that combines several benchmarks, so buyers have a single ranking to point at. Version 4.3 blends ten evaluations into four weighted categories: agents at thirty percent, general capability at thirty percent, coding at twenty percent and scientific reasoning at twenty percent. The component list is published, which makes it one of the more transparent composites available.

Why it does not answer the board question

A composite inherits the blind spots of every benchmark inside it, and this one contains no measure of citation and no measure of whether a model admits it does not know. Averaging also hides exactly the thing a board needs to see: a model can be first overall and last on the one property that decides whether you can forward its output to the audit committee.

What it measures

A family of more than twenty leaderboards run on evaluation sets that are deliberately kept private, so models cannot be trained on them. Topics range from agentic and software engineering ability to frontier reasoning and safety behaviour, with prompts written and graded by verified domain experts. The private set is the point: it is a direct answer to contamination.

Why it does not answer the board question

Privacy solves contamination and creates a different problem: you have to take the result on trust, because you cannot inspect the questions or check whether they resemble your work. The domains are also research and engineering domains, not a board pack in Dutch, and there is no separate citation or honesty score.

What it measures

A general capability benchmark designed so the questions cannot have been in the training data. Roughly one sixth of the questions is replaced every month, drawing on recently published papers, news and datasets, so the whole set turns over twice a year. Everything is graded against an objective answer key, with no model acting as judge.

Why it does not answer the board question

It is the best available defence against contamination among the general capability benchmarks, and its categories are still reasoning, coding, mathematics, data analysis, language and instruction following. None of those is faithfulness to a document set, and none of them scores what happens when the answer simply is not there.