Hugging Face · coding
April 16, 2024 · Read the original
Researchers from UC Berkeley, MIT, and Cornell developed LiveCodeBench, a benchmark for measuring LLMs' code generation capabilities, and released a leaderboard based on it. LiveCodeBench collects coding problems over time from LeetCode, AtCoder, and CodeForces and annotates them with release dates, so models can be evaluated on problem sets from different time windows to help detect and prevent contamination. It covers four scenarios: code generation, self-repair, code execution, and test output prediction, each evaluated with the Pass@1 metric.
Written from the article itself at capture time and checked against it before publication: every number and every name in it comes from the source. It is our summary, not the lab’s wording — the original is linked above.
The announcement says what the lab claims. The record says what was measured, by whom, and whether anyone reproduced it. Read both before believing either.