Evidence record
GRB internal research engineering
August 2026 report snapshot · Gemini 3.7 Flash / GRB scaffold · 2026-08
Reported result · numeric
27%
Average pass@1 · percent
lab reported · extraction review: agent checked
Source and extraction
Published 2026-08
- Gemini 3.7 Flash Frontier Safety Framework ReportPrinted page 37; PDF index 36; ResultsOriginal source ↗
Evaluation setup
| task count | 74 |
|---|---|
| aggregate method | Average pass@1 |
| wall clock budget | 2 hours per task |
| scaffold | Scoped internal agent scaffold |
| comparability caveats | Bugged tasks retained. |
Not reported: split, task snapshot, attempts per task, run count, selection rule, token budget, hardware, training budget, inference budget, monetary cost, tool access, internet access, filtering, evaluator version, human intervention, task exclusions, contamination concerns.
Comparability
limited comparison
- Rough older estimate; no automatic delta.
Limitations
- Bugged tasks retained; adjusted hypothetical 47% is not an observed score.
- Approximately 20% of tasks had bugs that could cause false negatives.
- Older result is a preliminary estimate.