PR #196 adds the complete 56-task evaluation of Codex + GPT-6 Astra xhigh with skill, reporting cumulative reward 43.02/56, mean 78.2182/100 over 55 validly scored tasks with 32 perfect scores, and matched comparisons where Astra's average exceeds Luna, Terra, and Flash. The report explicitly states this cannot be claimed as controlled full-56 SOTA and Astra's 56-task mean should not be directly compared to smaller subsets of other models.
Automatically summarized from source discussions. Sources and the full record are available below.
Discussion points
The PR records execution details including host sleep, memory wait, H16 Harbor metadata double-counted token, H8 output symlink failure causing grader abort, and states there was no selective rerun by score, no no-skill control, and no variance data from repeated runs.Source 1 ↗Source 2 ↗Source 3 ↗Source 4 ↗
Plugins and versions
The PR affects the benchmark index, adding a full 56-task report, per-task CSV, source and summary JSON, while keeping the original 18-task report unchanged.Source ↗
Review record and full evidence
Evidence & timeline
Verification
Not yet verified
Maintainer response
Untriaged
Author response
No response recorded
Earliest source found so far
Unknown
Sep 9, 2026, 01:09 AM
Research & review record
Lifecycle
Emerging
Source scope
5 automatically linked original sources; source relationships and independence are not human reviewed
Attribution confidence · 0%
First found by this site · Sep 9, 2026, 01:58 AM
Evidence, test conditions and updates
Research & review record · Reviewed propagation graph
06
Reviewed propagation graph
00
No source-to-source propagation relation recorded.
The lead source, “benchmark: step5-preview S1-S24 双臂运行归档 + 双判分结果,” says: “StepFun Step 5 Preview（OpenCode Zen 供应，公开发布名；预览端点背后精确 checkpoint 未经独立验证；opencode 编排）在 S 系列基准上的完整 24 题双臂运行、双判分与重跑 follow-up 归档。.” This is currently the only captured source.
The lead source, “feat(benchmark): S18 terminal sprite render trap (half-block SGR leak, trim ghosting, timer pinning),” says: “Summary Adds benchmark task S18-terminal-sprite-render-trap (static, read-only report): a terminal half-block sprite renderer whose three independent rendering bugs (SGR background.” 36 related sources are linked for comparison to determine whether they describe the same event.
The lead source, “feat(benchmark): H22 not-listed trap — the package that installs but never registers,” says: “Distilled from a real top-tier failure: anywhere-labs/dsh-desktop (22k★) failed our 50-plugin live verification on 09-01 with exactly this shape — dsh plugin add succeeds, the entr.” 31 related sources are linked for comparison to determine whether they describe the same event.
