Tracker

benchmark

CoBench

Diagnose historical internal R&D failures from infrastructure snapshots.

AI-R&D capability

What this measure tells us

Measures a component of research engineering.

  • Environment changed between versions; scores are not comparable across versions.
  • Private historical infrastructure and model-graded root-cause rubrics limit external replication.

Results by version

2.1

2026-09-22 · 500 tasks

One attempt per problem; environment drift within the reported comparison.

CoBench · Reported score

limited comparison

Source reports no statistically distinguishable difference (paired p≈0.2). Environment changes prevent attributing the point difference solely to the model.

CoBench: Reported scoreCategorical point plot with no connecting line. Source explicitly reports these systems under the same evaluation design.507090percentEstimated requirement · 85%2026-09-22: 53.4 percent · Claude Mythos 5.1153.42026-09-22: 53.2 percent · Claude Opus 5253.22026-09-22: 55.8 percent · Claude Opus 5.5355.8
Published results · 2.1
KeySystem / organizationMetricReported resultDateProtocolVerificationEvidence
2Claude Opus 5Reported score53.2%2026-09-22co21-earlierlab reported
1Claude Mythos 5.1Reported score53.4%2026-09-22co21-earlierlab reported
3Claude Opus 5.5Reported score55.8%2026-09-22co21-laterlab reported

Prior version (label unspecified)

Release date not reported

Historical values restated in the Opus 5.5 report; original version equivalence is not established.

Published results · Prior version (label unspecified)
KeySystem / organizationMetricReported resultDateProtocolVerificationEvidence
—Claude Opus 5Reported score59.6%2026-09-22co-priorlab reported
—Claude Mythos 5.1Reported score57.6%2026-09-22co-priorlab reported

Version lineage

Prior version (label unspecified) ← 2.1Prior version (label unspecified) → 2.1

Reference points

estimated necessary capability

85 · percent

Anthropic expects a research-staff substitute to score at least 85% on the prior version and estimates carryover to 2.1.

Logical role: estimated necessary condition · applicability: hypothesized

Estimated carryover only. Reaching 85% would not establish researcher substitution.

Availability

What is publicly available
ResourceStatus
public descriptionyes
public resultsyes
public tasksno
public codeunknown
public evaluation serviceunknown

Official sources