benchmark

MLE-bench

ML competition engineering under bounded compute.

AI-R&D capability

What this measure tells us

Measures practical training and data-science work.

Revised 2026 tasks and percentile scoring are incompatible with 2024 medal rates.
Public competition history creates contamination risk.

Benchmark version

Metric

Results to compare

Protocol

2026-09-03 · 72 tasks

Replaces tasks and scores percentile rank against a generated reference distribution; up to three leaderboard submissions.

Published results · Revised, Astra card snapshot
Key	System / organization	Metric	Reported result	Date	Protocol	Verification	Evidence
—	GPT-6 Astra	Reference-distribution percentile	not reported	2026-09-03	mle-revised-p	lab reported	GPT-6 Astra System Card ↗

2024-10-09 · 75 tasks

Mean any-medal fraction per attempt; not pass@k.

direct comparison

Different seed counts affect precision; no significance inference from rounded means.

Published results · Original paper v1 (2024)
Key	System / organization	Metric	Reported result	Date	Protocol	Verification	Evidence
4	o1-preview / AIDE	Any medal rate	16.9 ± 1.1%	2024-10-09	mle-o1-p	benchmark author reported	MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering (v1)↗
1	GPT-4o / AIDE	Any medal rate	8.7 ± 0.5%	2024-10-09	mle-4o-p	benchmark author reported	MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering (v1)↗
3	Llama 3.1 405B / AIDE	Any medal rate	3.0 ± 1.0%	2024-10-09	mle-llama-p	benchmark author reported	MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering (v1)↗
2	Claude 3.5 Sonnet / AIDE	Any medal rate	7.6 ± 1.8%	2024-10-09	mle-claude-p	benchmark author reported	MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering (v1)↗

No applicable reference points are published for this measure.