alerting-001 passes an alert that can never fire #128

benchmark-alerting-001-delivery-alerts checks each delivery trigger on its own, for whether it is enabled, covers the fulfilment connection and has a channel. Hookdeck binds each issue to one trigger, and a connection-specific final_attempt trigger loses to the default * / first_attempt trigger (#). Since #116 and #118 the scenario starts from that default with channels: {}, so an agent that adds a scoped final_attempt trigger beside it has built an alert that binds to a trigger that tells nobody. The scorer passes it.

Evidence

In run 42250228870 on 6 November, five of six arms did exactly that and all six passed. Only codex-gpt-5.4-mini-no-skills gave the default a channel, which is the only one of the six alerts that would reach anyone.

Those six rows are not in the v0.5.0 snapshot. The run checked out a commit from before the Codex re-run had published, and its export merged against that older results file, so results/latest.json kept the alerting-001 rows measured before #116 and #118 (#). Those earlier rows started from a project with no default delivery trigger, so an agent's scoped trigger had nothing to lose to, and this defect does not apply to them.

Second gap

channels.email.member_emails: [] is valid on the live API and notifies nobody. The scorer counts any non-empty channels object as a channel, so it would pass that too.

Fix

Score the outcome, not the configuration. After the agent finishes, cause one failing delivery on the fulfilment connection, read the issue_trigger_id of the issue that opens, and require that trigger to have a channel that reaches someone. On 6 November a first-attempt issue opened within a two-minute poll. This does not copy Hookdeck's precedence rules into a public scorer, so it cannot drift if they change. Then re-run all six experiments at three attempts; results either side are not comparable. Planned for v0.6.0.