benchmark

RE-Bench

Research engineering environments with expert baselines.

AI-R&D capability

What this measure tells us

Compares agent and expert work under explicit resource budgets.

Seven selected environments do not cover the full research process.
Best-of-k allocations and subsets must be kept distinct.

Results by version

Benchmark version

Metric

Results to compare

Protocol

Google five-task subset, November 2025

2025-11 · 5 tasks

Two internet-requiring tasks omitted; 16 attempts × 2 hours; 24 runs used in bootstrap.

Published results · Google five-task subset, November 2025
Key	System / organization	Metric	Reported result	Date	Protocol	Verification	Evidence
—	Gemini 3 Pro / METR Modular adaptation	Reported assessment	Google reports Gemini 3 Pro remains below its ML R&D alert threshold.	2025-11	rebench-gemini-p	lab reported	Gemini 3 Pro Frontier Safety Framework Report, v2 ↗

Original November 2024 report

2024-11-22 · 7 tasks

No chart heights digitized; retain source qualitative comparison.

No published results for this version.

Version lineage

Google five-task subset, November 2025Original November 2024 report

Reference points

No applicable reference points are published for this measure.

Availability

What is publicly available
Resource	Status
public description	yes
public results	yes
public tasks	yes
public code	yes
public evaluation service	unknown

Official sources

Evaluating frontier AI R&D capabilities of language model agents against human expertsEnvironment descriptions; Results; What do we mean by time budget?Original source ↗