Protocol

Keep acceptance examples out of the prompt workshop

The demonstration has had a lot of rehearsal. The next real case has not.

About 1 min read

Before you use it

Use when
  • A prompt improves on familiar examples and you need to know whether it improved beyond them.

Keep this limit in mind

  • A tiny or unrepresentative holdout remains weak evidence. Repeatedly selecting changes from the same holdout can contaminate it too.

An idea to explore.

Try this idea.

Inspect the example, try the idea in your situation, then check your next move.

Keep reading

Why it matters

Separate development examples from a held-back acceptance set. Use the first group to revise prompts and rules; use the second to evaluate the frozen candidate. Once acceptance failures guide another revision, those cases are no longer untouched evidence of generalization.

Steps

  1. Write expected outcomes for representative, permission-safe cases before testing.
  2. Version the prompt, model configuration and evaluation set together.
  3. Report held-back performance separately from results on examples used during development.

An example

A document extractor is tuned on familiar layouts, then tested on previously held-back layouts with known expected fields.

Check your result

The report identifies which cases influenced development and which first appeared at evaluation.

Save and practise this idea

My learning

Read. Recall. Repeat.

Save this idea and return to it on any device.

Use email
Keep exploring

Connected ideas

You are readingKeep acceptance examples out of the prompt workshop

Evidence and sources
Supports

Using evaluation data to make development choices can create optimistic performance estimates; the scikit-learn guidance keeps test data out of model choices.

A separate acceptance set for prompts is an application of this principle; representative sampling and repeated-use contamination still matter.

Common pitfalls and recommended practices · Data leakage

All sources (1)
Edition and review

Version 0.1.0

No review details are supplied for this edition.

Read this idea as JSON · Release details