Protocol

Re-test old harness rules when the model changes

Yesterday's workaround can become tomorrow's tax.

When it fits

A workflow has accumulated scaffolding, retries or context-reset tricks around an older model.

When to avoid it

Change one component at a time where possible; removing several together can hide interactions and make regression diagnosis ambiguous.

Why it matters

After a material model or harness upgrade, rerun representative evals while removing or simplifying one workaround at a time. Keep a component only if it still improves quality, reliability, cost or recovery. Do not preserve elaborate context resets, prompt rituals or reviewer loops merely because an earlier model needed them.

Steps

Choose a representative baseline suite.
Remove or simplify one harness component.
Run repeated comparable trials.
Inspect quality, cost and failure mode changes.
Keep, revise or retire the component.

An example

A forced context reset that prevented drift on an older coding model may be unnecessary after a model upgrade with stronger long-context behavior.

Check your result

Every expensive harness component has current evidence of value on the model you actually run.

Keep this limit in mind

Change one component at a time where possible; removing several together can hide interactions and make regression diagnosis ambiguous.

Connected ideas

Evidence and sources

Supports

Anthropic reports that context-reset scaffolding that helped one model could be removed for a stronger later model, illustrating that harness components should be re-tested as models change.

Do not infer that resets or compaction are broadly obsolete; test on the actual model and task.

Harness design for long-running application development · Context resets and later simplification

All sources (1)

Harness design for long-running application development