A gold standard for automated policy coding. Three domains, three units of analysis, four independent coders on every text — and machines scored only against the items where all four agreed.
The Easy tier: the 3,677 held-out items on which all four human coders independently agreed. Every entrant is scored on the same items under the protocol below. Accuracy is agreement with the four-coder label; κ is Cohen's kappa against it. Runtime is wall-clock for the full set, which is a property of the deployment as much as the model — read it next to where it ran, not on its own.
Read the numbers against the right baseline. These items are the subset on which four trained humans agreed unanimously — the unanimous-agreement subset, the easiest items by construction. A κ here is therefore not comparable with the human inter-coder κ, which is measured across the whole corpus including the contested items. The honest reading of 75% accuracy is: on text where four coders saw the same thing, the model still disagrees with all four once in four.
A second tier, harder by construction: the 2,245 items where exactly three of the four coders agreed and one dissented. The label is the majority; the dissent is withheld with it. Easy asks whether a system can match a unanimous human verdict. Intermediate asks whether it lands on the majority side of a split one — which is the case that actually decides a coding project.
Same protocol, same prompt, same withheld-label rule — only the item set differs, so the two tiers are directly comparable. Twenty-three runs are scored here; nothing is still queued. The tier sizes shown for Intermediate (2,245) and Difficult (954) predate the final exclusion rules; the manuscript uses 2,232 and 942, and these two tiers will be re-scored on that basis. Two things stand out. Every model drops sharply — the leader loses 16.1 points moving from unanimous items to split ones, a fair estimate of how much of an Easy score depends on the question being easy. And the order changes: Qwen3.8-27B rises from 6th on Easy to 2nd here, while the two fine-tuned classifiers spread apart. An Easy rank does not predict an Intermediate rank, which is the case for running both. 365 of the majority labels are “no policy content.”
The 954 items where two coders agreed and the other two each chose something different. Every item carries exactly three distinct codes from four trained humans. There is no consensus here to match, so the reference label is a plurality of two in four — half the panel said something else. Read the accuracy figure accordingly: it measures whether a system lands on the largest bloc, not whether it is right.
Because a majority does not exist, each row also reports two figures the other tiers do not need: how often the model matched one of the two dissenting coders, and how often it answered outside all four human codes. That second number is the one worth watching — it separates a model picking a defensible human reading from one that is simply wrong. Twenty-three runs are scored; nothing is still queued. The tier sizes shown for Intermediate (2,245) and Difficult (954) predate the final exclusion rules; the manuscript uses 2,232 and 942, and these two tiers will be re-scored on that basis. Two results carry beyond this page. First, the fine-tuned classifiers do not hold up: poltextlab-english-cap-v5 wins Easy and Intermediate and places 6th here, and ParlaCAP falls from 10th to 14th, while general models climb — Qwen3.8-27B goes 6th, 2nd, 1st across the three tiers. Task-specific fine-tuning buys agreement with consensus, not with contested judgement. Second, most of what looks like failure is not: for the leading models, roughly five in six answers are a code some human coder chose. The exceptions are the bottom of the board, where GLM-4.7-Flash answers outside the human panel on 30.3% and DeepSeek-V4-Flash, run with its reasoning switched off, on 33.0% — twice the leaders — which is the honest way to tell error apart from disagreement. The split concentrates at the no-policy-content boundary: it is the plurality label on 200 items, but on 372 it is a dissenter who called an item no-policy.
Texts are released; the gold labels are withheld and held by the benchmark operator. There is no training split. Teams bring models and workflows that are already trained, and are scored only by submission against the held-out labels — the design Kaggle competitions and the GLUE and SuperGLUE suites use to keep a test set from leaking into the models it measures.
A closed — or private-label, or held-out — benchmark releases the material to be coded and keeps the correct answers. There is no training split to fit against and no public key to check yourself against; the only way to learn your score is to submit and be told. Kaggle competitions work this way, and so do the GLUE and SuperGLUE suites that set the convention in language evaluation.
Once labels are public they end up in training data — scraped, redistributed, or pasted into a prompt. A model trained afterwards may have seen the answers, and the benchmark keeps producing numbers while quietly measuring memory instead of skill. Nothing in the score tells you which one you are looking at.
On one of our own classifiers, an older deployed model scored macro F1 0.894 on a held-out split against a newly trained model’s 0.828 — an apparently decisive win. Every one of those test rows turned out to sit in the older model’s own training files. Its published card, scored on a clean set, reports 0.57. The flattering number was contamination, and only checking the overlap revealed it.
With a private key there is no gradient to climb by resubmission. A system is scored on text it has never been fitted to, which is the only condition under which a leaderboard position says anything about the next corpus.
Every entrant gets the same items, the same prompt, the same truncation and the same decoding settings, published in advance. Two scores differ because the systems differ, not because one had a better scaffold.
Every item was coded independently by four trained humans, and only the units where all four agreed enter the benchmark. The label a model is scored against is therefore not one annotator’s judgement but four converging — which is what makes a disagreement with it meaningful.
Every item’s text, in full
The domain and unit of analysis
Source metadata — congress, chamber, speaker, outlet, section, date, URL
The exact prompt, truncation limit and decoding settings
The codebook: 21 CAP major topics plus a no-policy-content code
Human reliability for every domain and unit is reported in the accompanying manuscript, which is under review
The gold labels — held only by the benchmark operator
Any training split: there is none; entrants bring already-trained systems
Per-item results, so no entrant can reconstruct the key from its own score
The bills summary field, which paraphrases a bill’s topic
Existing machine predictions carried in the source sheets
Closed does not mean secretive: everything needed to reproduce the conditions is public, only the answers are not.
Without per-item feedback you learn your score, not where you went wrong. We return aggregate breakdowns by domain, unit and topic — enough to diagnose, not enough to reconstruct the key.
Someone has to run every submission. That is the price of the key staying private, and it caps how fast the leaderboard can move.
A closed test protects the key from leaking; it does not make the key correct. That is why human reliability is reported in the accompanying manuscript (under review) rather than left out — a benchmark whose own labels disagree cannot certify a model past that point.
Fixed and published in advance, so that a difference between two entrants is a difference between the systems and not between their harnesses. This is the prompt every entrant on the leaderboard below was scored under, verbatim.
| Items | 3,677 — every unit on which all four human coders independently agreed |
| Task | one CAP major topic code per item |
| Prompt | prompted models: the text below, verbatim, one item per call (file prompt_v1.txt, SHA-256 8181c223…f197d9). Fine-tuned classifiers: no prompt — supply a one-to-one mapping from your labels to the CAP codes |
| Truncation | 6,000 characters (affects 204 of the 3,677 items: long media bodies, whole speeches and bill documents) |
| Temperature | 0 |
| Max tokens | 400 — high enough that a reasoning model can finish; a short answer simply uses fewer |
| Metadata | supplied in the input file, deliberately kept out of the prompt — entrants are scored on the text |
| Unparseable answer | counts as wrong, never as missing |
You are coding a text to the Comparative Agendas Project (CAP) major topic scheme.
Read the text and assign exactly ONE code from this list:
1 Macroeconomics
2 Civil Rights, Minority Issues, and Civil Liberties
3 Health
4 Agriculture
5 Labor and Employment
6 Education
7 Environment
8 Energy
9 Immigration
10 Transportation
12 Law, Crime, and Family Issues
13 Social Welfare
14 Housing and Community Development
15 Banking, Finance, and Domestic Commerce
16 Defense
17 Space, Science, Technology and Communications
18 Foreign Trade
19 International Affairs and Foreign Aid
20 Government Operations
21 Public Lands and Water Management
23 Culture and Arts
999 No policy content
Rules:
- Code the policy topic the text is ABOUT, not the institution that produced it.
- 20 Government Operations is a substantive policy topic (the machinery of
government, appropriations process, agency structure, elections administration).
It is not a residual category for text you cannot place.
- Use 999 only when the text carries no policy content at all.
- Answer with the code number and nothing else. No explanation, no punctuation.
TEXT:
{TEXT}Bring a trained system. There is no training split to fit against, and no submission is scored twice under different conditions.
| 1 · Request the items | Email poltextlab@poltextlab.com with the subject CAPBENCH-English submission. You get the 3,677 items as JSONL — item_id, text, and source metadata. No labels. |
| 2 · Code them | One CAP major topic per item, under the protocol above. Prompted models use the published prompt verbatim; a fine-tuned classifier submits its own label mapping instead. |
| 3 · Return | JSONL of {item_id, code}, one line per item, plus a short description of the system, its version, and the hardware and wall-clock — those go on the board next to the score. |
| 4 · Scoring | Done by the benchmark operator against the withheld labels. You receive accuracy, κ, and breakdowns by domain, unit and topic — never per-item results, which would let the key be reconstructed. |
One entry per system version. An unparseable answer counts as wrong, not as missing, so return a code for every item.
The benchmark is open for submissions across all three domains. Entrants are scored off-box against the withheld labels and posted to the scoreboard above; the round is continuous, and new entrants are added as they are run.
Scores are provisional while the round is open, and an entrant may be re-run if the protocol changes — in which case every entrant is re-run, never one alone.