Benchmark explainer
Crowdsourced LLM evaluation via blind pairwise human preference votes on real user conversations (Chatbot Arena).
LMCA measures LLM quality by crowdsourced blind pairwise human preference votes on real user conversations, the Chatbot Arena setup: users submit a prompt, receive two anonymous model responses side by side, and vote for the better one, with Elo rankings aggregating the votes. Developed by LMSYS at UC Berkeley (Chiang et al.).
Discrimination
76.8/100Does it still separate models?
Saturation headroom
99.3/100How far from ceiling / clustered at the top?
Contamination resistance
83.3/100Public vs held-out; training-leak risk.
Harness comparability
100/100Is it apples-to-apples, or mixed / vendor-optimized?
Freshness
100/100How old is the benchmark?
Contamination history: User prompts and votes publicly released (lmsys/chatbot_arena_conversations) — prompts may end up in training data, inflating scores.
Limitations: Vote distribution skews to categories users prefer; new models lack sufficient vote depth (cold-start); user base demographics bias prompts; static benchmarks can saturate but Arena ranking continually shifts.
| Model | Score | Evidence |
|---|---|---|
| Claude Fable 5.2 | 83.07 | unverified |
| Claude Sonnet 5.5 | 80.31 | unverified |
| Claude Opus 5.5 | 80.27 | unverified |
| Claude Fable 5.1 | 77.01 | unverified |
| Claude Opus 5 | 75.82 | unverified |
LMCA: LMCA measures LLM quality by crowdsourced blind pairwise human preference votes on real user conversations, the Chatbot Arena setup: users submit a prompt, receive two anonymous model responses side by side, and vote for the better one, with Elo rankings aggregating the votes. Developed by LMSYS at UC Berkeley (Chiang et al.).
Claude Fable 5.2 leads LMCA at 83.07. The full leaderboard above lists every recorded measurement, not just the headline number.
126 models have recorded scores on LMCA, spanning a score spread of 76.75.
tensor.news grades LMCA A for integrity (score 90.3/100), ranking #10 of 62 benchmarks we assess across discrimination, saturation, contamination resistance, harness comparability, and freshness.
No — LMCA still has headroom and continues to discriminate between models rather than bunching them at the ceiling.
Its test set is unknown, so contamination risk is on the table: a high score may partly reflect training-data overlap rather than capability.
Scores are reported under a consistent harness, so comparisons on LMCA are reasonably apples-to-apples.
Every LMCA measurement is source-backed and tagged reproduced, self-reported, or unverified; the underlying record cites arxiv.org/abs/2403.04132.