- Task type
- unknown
- Metric
- normalized performance in Epoch CSV, rendered as accuracy (%)
- Score ceiling
- 100
- Human baseline
- Human preference vote IS the baseline (the metric being aggregated).
- Test-set privacy
- unknown
- Construction
- Open conversational platform at lmarena.ai; users submit prompts, get two anonymous model responses side-by-side, vote for the better one; Elo ranks aggregate the votes.
- Key assumptions
- Real-user prompts and blind pairwise preference is the most production-relevant quality signal; Elo captures incremental capability differences cleanly.
- Limitations
- Vote distribution skews to categories users prefer; new models lack sufficient vote depth (cold-start); user base demographics bias prompts; static benchmarks can saturate but Arena ranking continually shifts.
- Contamination history
- User prompts and votes publicly released (lmsys/chatbot_arena_conversations) — prompts may end up in training data, inflating scores.
- Harness notes
- Epoch CSV reports model, model version, performance, score source, benchmark release date, and model date; optimized-scaffold flags are carried forward from prior Epoch exports (column removed upstream ~2026-09).
- Authors
- LMSYS @ UC Berkeley (Chiang et al.)
- Released
- unknown
- License
- CC BY 4.0