Protocol

Re-test old harness rules when the model changes

Yesterday's workaround can become tomorrow's tax.

About 1 min read

Before you use it

Use when
  • A workflow has accumulated scaffolding, retries or context-reset tricks around an older model.

Keep this limit in mind

  • Change one component at a time where possible; removing several together can hide interactions and make regression diagnosis ambiguous.

An idea to explore.

Try this idea.

Inspect the example, try the idea in your situation, then check your next move.

Keep reading

Why it matters

After a material model or harness upgrade, rerun representative evals while removing or simplifying one workaround at a time. Keep a component only if it still improves quality, reliability, cost or recovery. Do not preserve elaborate context resets, prompt rituals or reviewer loops merely because an earlier model needed them.

Steps

  1. Choose a representative baseline suite.
  2. Remove or simplify one harness component.
  3. Run repeated comparable trials.
  4. Inspect quality, cost and failure mode changes.
  5. Keep, revise or retire the component.

An example

A forced context reset that prevented drift on an older coding model may be unnecessary after a model upgrade with stronger long-context behavior.

Check your result

Every expensive harness component has current evidence of value on the model you actually run.

Save and practise this idea

My learning

Read. Recall. Repeat.

Save this idea and return to it on any device.

Use email
Keep exploring

Connected ideas

You are readingRe-test old harness rules when the model changes

Evidence and sources
Supports

Anthropic reports that context-reset scaffolding that helped one model could be removed for a stronger later model, illustrating that harness components should be re-tested as models change.

Do not infer that resets or compaction are broadly obsolete; test on the actual model and task.

Harness design for long-running application development · Context resets and later simplification

All sources (1)
Edition and review

Version 0.1.0

No review details are supplied for this edition.

Read this idea as JSON · Release details