Protocol
Re-test old harness rules when the model changes
Yesterday's workaround can become tomorrow's tax.
Before you use it
- Use when
- A workflow has accumulated scaffolding, retries or context-reset tricks around an older model.
Keep this limit in mind
- Change one component at a time where possible; removing several together can hide interactions and make regression diagnosis ambiguous.
An idea to explore.
Try this idea.
Inspect the example, try the idea in your situation, then check your next move.
Why it matters
After a material model or harness upgrade, rerun representative evals while removing or simplifying one workaround at a time. Keep a component only if it still improves quality, reliability, cost or recovery. Do not preserve elaborate context resets, prompt rituals or reviewer loops merely because an earlier model needed them.
Steps
- Choose a representative baseline suite.
- Remove or simplify one harness component.
- Run repeated comparable trials.
- Inspect quality, cost and failure mode changes.
- Keep, revise or retire the component.
An example
A forced context reset that prevented drift on an older coding model may be unnecessary after a model upgrade with stronger long-context behavior.
Check your result
Every expensive harness component has current evidence of value on the model you actually run.
Save and practise this idea
My learning
Read. Recall. Repeat.
Save this idea and return to it on any device.
Keep exploring
Connected ideas
You are readingRe-test old harness rules when the model changes
Evidence and sources
Anthropic reports that context-reset scaffolding that helped one model could be removed for a stronger later model, illustrating that harness components should be re-tested as models change.
Do not infer that resets or compaction are broadly obsolete; test on the actual model and task.
Harness design for long-running application development · Context resets and later simplification
All sources (1)
Edition and review
Version 0.1.0
No review details are supplied for this edition.