Evidence record
GRB internal research engineering
August 2026 report snapshot · Gemini 3.1 Pro / preliminary GRB runs · 2026-08
Reported result · numeric
around 16%
Average pass@1 · percent
lab reported · extraction review: agent checked
Source and extraction
Published 2026-08
- Gemini 3.7 Flash Frontier Safety Framework ReportPrinted page 37; PDF index 36; Results, footnote 10Original source ↗
Evaluation setup
| comparability caveats | No detailed transcript analysis of older runs. |
|---|
Not reported: split, task count, task snapshot, attempts per task, run count, selection rule, aggregate method, token budget, wall clock budget, hardware, training budget, inference budget, monetary cost, tool access, internet access, filtering, scaffold, evaluator version, human intervention, task exclusions, contamination concerns.
Comparability
limited comparison
- Rough older estimate; no automatic delta.
Limitations
- Approximate preliminary estimate, not a confident baseline.
- Approximately 20% of tasks had bugs that could cause false negatives.
- Older result is a preliminary estimate.