Protocol
Keep acceptance examples out of the prompt workshop
The demonstration has had a lot of rehearsal. The next real case has not.
When it fits
- A prompt improves on familiar examples and you need to know whether it improved beyond them.
When to avoid it
- A tiny or unrepresentative holdout remains weak evidence. Repeatedly selecting changes from the same holdout can contaminate it too.
Why it matters
Separate development examples from a held-back acceptance set. Use the first group to revise prompts and rules; use the second to evaluate the frozen candidate. Once acceptance failures guide another revision, those cases are no longer untouched evidence of generalization.
Steps
- Write expected outcomes for representative, permission-safe cases before testing.
- Version the prompt, model configuration and evaluation set together.
- Report held-back performance separately from results on examples used during development.
An example
A document extractor is tuned on familiar layouts, then tested on previously held-back layouts with known expected fields.
Check your result
The report identifies which cases influenced development and which first appeared at evaluation.
Keep this limit in mind
- A tiny or unrepresentative holdout remains weak evidence. Repeatedly selecting changes from the same holdout can contaminate it too.
Connected ideas
Useful withTest what should stay unchanged when the input changes
Evidence and sources
Using evaluation data to make development choices can create optimistic performance estimates; the scikit-learn guidance keeps test data out of model choices.
A separate acceptance set for prompts is an application of this principle; representative sampling and repeated-use contamination still matter.
Common pitfalls and recommended practices · Data leakage