[leggetter]

What this is for

#131: every defect v0.5.0 found could have been caught before the run that found it. That list is the Codex PATH miss (#114), a skills pin behind main (#26), a Hookdeck CLI pin behind (#125), leftovers in the Outpost project (#111) and overlapping dispatches (#127). A matrix costs about $100 and six hours. This is a script that takes minutes, run before the dispatch.

What changed

• pnpm preflight: apps/framework/scripts/preflight.ts. It is wired in the framework package.json like triage/report-results, plus a root alias. It prints a report, takes --json (use pnpm -s preflight --json for clean JSON), and exits 1 on a failed assertion (2 on bad arguments).
• Decisions (1–5, plus 16):
  • Hookdeck CLI pin and skills CLI pin (both build args of the sandbox image) against npm latest.
  • Agent CLI pins against npm latest. The pins are read from the runner objects through the new registeredAgentPins().
  • Skills gitlink against hookdeck/agent-skills main, listing which skills changed. It notes the compare API's caps and a diverged pin.
  • Model IDs against each provider's /v1/models.
  • Base prompt against the last non-draft release.
  • Each one prints the current value, the latest and the value at the last release, then decision required: record in the release notes. None of them fails the script.
• Assertions (6–14):
  • 6, provider keys: one real completion per provider. It sends the exact request the workflow's "Check provider keys are live" step sends, parsed out of the YAML rather than copied.
  • 7, spend: an estimate from the AGENTS.md cost table, using fail rates derived from results/latest.json with k bracketed at 1–3 (Review and merge or drop the Codex cost-reporting branch #6). It never claims to read a balance, and needs --confirm-balance (or y on a TTY).
  • 8, secrets: gh secret list against every secrets.* the workflow reads. The key of an agent no experiment uses is reported but does not fail.
  • 9, main is green: if HEAD is a clean checkout of main, it runs the four commands locally. Otherwise it reads the Checks run of the pull request that produced main, and fails if main's first parent was committed after that run started.
  • 10, nothing running: no unfinished eval-refresh run.
  • 11, repository variables: EVALS_PUBLISH and EVALS_MAX_MISSING_FRACTION must match --hold-publish and --max-missing-fraction.
  • 12, projects are pristine:
    • evals-ci holds exactly DEFAULT_ISSUE_TRIGGERS, using the reset's own matcher, and no other resources.
    • The Outpost project's gateway side matches OUTPOST_GATEWAY_PRISTINE, compared by id.
    • There are no Outpost tenants.
    • There is at least one enabled operator event destination. Its target is checked without sending it anything, and its URL is printed with the path masked.
  • 13, skill links: every absolute or root-relative hookdeck.com/docs URL in the pinned skills returns 200. A redirect warns with the final URL, and a 429 backs off once before failing.
  • 14, harness issues: every open harness issue must be named in --accept-harness.
• Canary (15): for each agent, it compares the image definition hash, the Hookdeck and skills CLI pins and that agent's CLI pin against the commit of the newest published run that has cells for that agent. It also compares each experiment's model against its newest row. If anything moved, it fails and prints the two-cell gh workflow run command with EVALS_PUBLISH=false. --canary-run <id> then verifies the finished run:
  • the run concluded success
  • there is one Claude and one Codex cell
  • each cell's "Run evals" and "Check artifacts for credentials" steps succeeded
  • the publish job was skipped
  • the log has no sandbox self-check failed
  • no downloaded raw result shows PINNED CLI NOT ON PATH
• --skip <id> prints a banner and the skipped ids in the result line.
• Redaction: every result string goes through redactArtifactDeep before the report is laid out.
• AGENTS.md: a "Before dispatching a matrix" subsection under Runs.
• Small exports, all read-only:
  • basePromptFor moves to apps/framework/lib/base-prompt.ts. The string is unchanged, so no harness fingerprint moves.
  • @hookdeck-evals/core exports registeredAgentPins().
  • @hookdeck-evals/hookdeck exports issueTriggerMatchesDeclared, OUTPOST_GATEWAY_PRISTINE, the OperatorEventDestination type, redactArtifactDeep and redactSecretsForArtifact.

Verified

• pnpm install: ok

• pnpm format:check: exit 0

• pnpm typecheck: exit 0

• pnpm -r build: exit 0

• pnpm check: exit 0, framework 215 tests

• pnpm -r test: exit 0. core 115, sandbox 49, hookdeck 70, web 71, framework 215.

• test/preflight.test.ts covers the pure logic. test/preflight-checks.test.ts runs the real checks against fake Deps: canary run selection after a Claude-only publish, the main-green fallback, the secrets filter and the projects-pristine evaluation.

• I did not run actionlint, because no workflow changed.

• A real run against a full benchmark matrix exits 1 on three assertions:

  • skill links: 7 dead links, exactly the set in The event-gateway skill links agents to API reference pages that return 404 #104, plus 3 redirects
  • open harness issues: 18 not accepted
  • spend: not confirmed

  It found five decisions: the Hookdeck CLI pin is one major behind, the skills CLI is two minors behind, the agent CLIs are behind, the skills pin is 4 commits behind main, and claude-sonnet-5-5 exists.

Not done, and why

• The provider-key preflight only runs in CI, not for a local run #42 stays open. The workflow keeps its own curl preflight. This script reads that step rather than replacing it, and moving the check into the run path is The provider-key preflight only runs in CI, not for a local run #42's own change.
• Not exercised for real:
  • --canary-run: no canary run exists yet.
  • The interactive y/N prompt.
  • The local branch of check 9, which only runs on a clean checkout of main.
• Check 12 can only warn on the operator event destination. It is a Hookdeck source URL in a project neither key can see. Sending it a request to prove it exists would be a write, so the check stops at a warning.
• Known limits of the canary:
  • It is per agent, as Nothing checks the CLI pins, skills pin, project state or credit before a matrix is dispatched #131 specifies. One Codex cell on a new image clears it for both Codex experiments.
  • It compares each runner's default CLI version, so an experiment that overrides cliVersion would not be seen. No experiment does today.

Observations, not in this diff

• AI_GATEWAY_API_KEY: the workflow reads it, but it is not a repository secret. Only opencode uses it, and no experiment uses opencode.
• gpt-5.6: it is not in OpenAI's /v1/models, which lists only gpt-5.6-sol, -luna and -terra. That matches AGENTS.md's "resolves to gpt-5.6-sol".
• The Checks workflow runs on pull_request only, so main commits carry no check runs. That is why check 9 reads the PR's run.
• Three skill links redirect to moved pages: /docs/outpost, /docs/outpost/destinations and /docs/projects.

Closes #131

🤖 Generated with Claude Code