Protocol
Test an intervention across a replay cohort
One rescued trace is a story. A cohort begins to look like evidence.
When it fits
- One replay looks better after a model, prompt, tool or routing change and you need to know whether the improvement is stable enough to matter.
When to avoid it
- A cohort is not the future. Rare events and distribution shifts may be absent, so production monitoring still matters.
Why it matters
Collect a bounded set of representative checkpoints from real runs. Apply the same intervention to each, then compare task outcomes, decisions, cost and failure types. This reveals whether a local fix generalizes across the cases that motivated it.
Steps
- Define the production slice the cohort should represent.
- Replay each checkpoint with the same intervention.
- Measure outcomes and important failure categories, not only trace similarity.
- Review cases that improve, regress or remain unchanged before release.
An example
Before moving support triage to a cheaper model, replay a sample of real triage checkpoints and compare correct routing, escalation and cost across the cohort.
Check your result
The release decision cites cohort-level outcomes and notable regressions rather than one favorable replay.
Keep this limit in mind
- A cohort is not the future. Rare events and distribution shifts may be absent, so production monitoring still matters.
Connected ideas
Useful withEvaluate the same agent task across multiple trials
Evidence and sources
Comparing an intervention across multiple checkpointed runs can reveal outcome changes that a single replay would not represent reliably.
A replay cohort only represents the production slice it was sampled from; rare cases and future distribution shifts can remain uncovered.
Your Agents Need a Save Button · 10:54-15:10, apply the intervention across a cohort and use the cohort to change the release decision