Repository navigation
Record the first eval review, and say when to run the next - #133
Merged
Merged
Conversation
…t claim Adds .plans/scenario-difficulty-research.md: how our scenarios and metric compare with other vendor and research benchmarks, every scenario classified by what makes it hard, and the decisions taken from it. AGENTS.md points to it from Plans. AGENTS.md said Vercel and Convex write prompts as requirement lists. Convex does; Vercel's current Next.js eval prompts are short task statements. Co-Authored-By: Claude Opus 5.5 <[email protected]>
The first commit kept the conclusions and dropped the evidence: the per- benchmark survey detail and the scripts that computed other suites' saturation lived only in temporary files. Adds .plans/eval-reviews/: the vendor and research surveys from November, scripts that recompute our saturation and four other suites' from live published data, and a README saying when to review (a scripted check before every release; a full review when half the suite is passed by every arm on the first attempt, the model set changes, or six months pass), what a review does, a prompt to start one, and an index. Co-Authored-By: Claude Opus 5.5 <[email protected]>
leggetter
changed the title
- ours.py now prints the pooled first-attempt rates and the per-snapshot count of scenarios with a failing arm, which the findings quoted but no script reproduced; external.py prints supabase's per-run figure - Convex read at 92 and 73 everywhere, dated, since one run per model drifts - "most" frontier passes becomes "many"; "six" builds becomes seven; Outpost docs reads are scoped to v0.5.0 and called fetches; seeds are counted in requests; Terminal-Bench trials marked inferred - The research survey's 35-65% bottom line names the benchmarks it is true of, since its own table has two that score higher - AGENTS.md no longer gives Convex a reason for its requirement lists that the survey does not support Co-Authored-By: Claude Opus 5.5 <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The first eval review, kept so the next one starts from its evidence. It asks whether our scenarios still separate agents, and whether a pass reflects good docs and skills or an easy scenario.
.plans/scenario-difficulty-research.md. Mostly easy scenarios, and best of three hides what failure is left. It compares our metric and scenarios with other vendor and research benchmarks, classifies all nineteen scenarios by what makes them hard, and records the decisions..plans/eval-reviews/2026-11-vendor-survey.md(supabase, Next.js, Convex, Clerk, Stripe, Auth0, Upstash and others) and.plans/eval-reviews/2026-11-research-survey.md(Terminal-Bench, τ-bench, MCPMark, MCP-Atlas, ITBench and others). Each claim carries its source link, and anything not reproduced from data is marked unverified.scripts/ours.pyrecomputes our saturation fromresults/.scripts/external.pyrecomputes supabase, Next.js, Convex and Stripe saturation from their live published data..plans/eval-reviews/README.md.Follow-up
The decisions are #134 to #142, with a graduation rule proposed on #47.
🤖 Generated with Claude Code