Protocol

Compare prompt versions on saved test cases

A better answer on one example can hide a new mistake elsewhere.

About 2 min read

Before you use it

Use when
  • You reuse one prompt each week and assume a fluent answer means success.

Keep this limit in mind

  • Three cases are an initial smoke test, not proof of general reliability or safety. Provider changes or hidden settings can limit reproducibility.

An idea to explore.

Try this idea.

Inspect the example, try the idea in your situation, then check your next move.

Keep reading

Why it matters

Save an ordinary case, an ambiguous case and a known failure, with the result each should produce. Compare the old and new prompts using the same inputs and checks. Start each attempt without leftover conversation or test state. Keep the model and tools unchanged where possible; record any change so you do not attribute it to the prompt.

Steps

  1. Save both prompts, the visible model label, relevant settings and tool access, plus each output and its pass or fail result. Mark unavailable details as unknown.
  2. Inspect failures against the agreed checks. If a decision depends on consistency, repeat cases under the recorded conditions and report failures as well as passes.

An example

An invented inbox test expects a meeting request to be labelled scheduling, a request mixing scheduling and billing to be flagged for review, and an automatic reply to be ignored. A rewrite fixes the mixed request but labels the automatic reply scheduling: keep both results.

Check your result

The comparison shows which saved cases passed or failed under which conditions. One passing attempt does not establish consistent performance.

Save and practise this idea

My learning

Read. Recall. Repeat.

Save this idea and return to it on any device.

Use email
Keep exploring

Connected ideas

You are readingCompare prompt versions on saved test cases

Evidence and sources
Supports

Anthropic describes evaluation tasks with defined inputs and success criteria, records of attempts, repeated trials for variable model outputs, and clean, stable test environments.

Claim limit: Provider engineering guidance, not experimental proof that a three-case personal prompt check establishes reliability. Recording visible conditions is an editorial adaptation; hidden provider settings may prevent exact reproduction. Source reading history: selected_passages; 2026-10-10; Evaluation structure: task, trial, transcript and harness; Step 4: stable environment. The cited official page was retrieved on 2026-10-10 and the named passages were inspected. Card-specific adaptations are independent editorial reasoning, not validated improvements. Selected definitions and Step 4 reread on 2026-10-12; this does not constitute full scientific review. Agent release review: Agent checked the exact card fields and its claim, source, locator, example, check and stated boundary. This bounded projection preserves the source pack’s screened evidence scope and does not claim independent scientific or human editorial review.

Demystifying evals for AI agents · Evaluation structure: task, trial, transcript and harness; Step 4: stable environment

All sources (1)
Edition and review

Version 0.1.0 · Published

Edition checked · 2026-11-13

Read this idea as JSON · Release details