Protocol
Re-benchmark AI when the model or task changes
An AI capability map expires faster than most process documentation.
Before you use it
- Use when
- A workflow has an old judgment about AI capability but the model, tools, prompt, data or task shape has materially changed.
Keep this limit in mind
- A small benchmark can miss rare failures; keep production monitoring and consequential review even after a strong re-test.
An idea to explore.
Try this idea.
Inspect the example, try the idea in your situation, then check your next move.
Why it matters
Keep a small representative benchmark for important AI-assisted tasks and rerun it after material model or workflow changes. Compare quality, failure modes and review burden with the previous version. Promote or reduce AI responsibility from observed results rather than release notes or reputation.
Steps
- Representative task cases are saved with pass criteria.
- The model/tool/configuration version is recorded.
- Material workflow changes trigger re-evaluation.
- New failure modes are compared with old ones.
- Responsibility changes only after the new evidence is reviewed.
An example
After moving from one coding agent to another, rerun repository-specific tasks instead of assuming a benchmark leaderboard predicts your workflow.
Check your result
A material AI-system change produces an updated local capability judgment backed by comparable cases.
Save and practise this idea
My learning
Read. Recall. Repeat.
Save this idea and return to it on any device.
Keep exploring
Connected ideas
You are readingRe-benchmark AI when the model or task changes
Ideas that link here
These ideas link to this one. Check their conditions and limits before use.
Useful with this ideaRe-test old harness rules when the model changesUseful with this idea
Own the task eval before shopping for a better model
Evidence and sources
The jagged-frontier study notes that AI capabilities and their useful task boundary change rapidly, so knowledge workers cannot rely on a fixed map of what AI can do well.
The study itself does not measure every subsequent model generation; reassessment is a practical implication, not a measured effect size.
Navigating the Jagged Technological Frontier: Field Experimental Evidence of the Effects of Artificial Intelligence on Knowledge Worker Productivity and Quality · Discussion of expanding and changing frontier
Edition and review
Version 0.1.0
No review details are supplied for this edition.