Protocol
Keep acceptance examples out of the prompt workshop
The demonstration has had a lot of rehearsal. The next real case has not.
Before you use it
- Use when
- A prompt improves on familiar examples and you need to know whether it improved beyond them.
Keep this limit in mind
- A tiny or unrepresentative holdout remains weak evidence. Repeatedly selecting changes from the same holdout can contaminate it too.
An idea to explore.
Try this idea.
Inspect the example, try the idea in your situation, then check your next move.
Why it matters
Separate development examples from a held-back acceptance set. Use the first group to revise prompts and rules; use the second to evaluate the frozen candidate. Once acceptance failures guide another revision, those cases are no longer untouched evidence of generalization.
Steps
- Write expected outcomes for representative, permission-safe cases before testing.
- Version the prompt, model configuration and evaluation set together.
- Report held-back performance separately from results on examples used during development.
An example
A document extractor is tuned on familiar layouts, then tested on previously held-back layouts with known expected fields.
Check your result
The report identifies which cases influenced development and which first appeared at evaluation.
Save and practise this idea
My learning
Read. Recall. Repeat.
Save this idea and return to it on any device.
Keep exploring
Connected ideas
You are readingKeep acceptance examples out of the prompt workshop
Ideas that link here
These ideas link to this one. Check their conditions and limits before use.
Useful with this ideaJudge the expensive error separately from the common oneUseful with this idea
Swap answer order before trusting an AI judge's winner
Evidence and sources
Using evaluation data to make development choices can create optimistic performance estimates; the scikit-learn guidance keeps test data out of model choices.
A separate acceptance set for prompts is an application of this principle; representative sampling and repeated-use contamination still matter.
Common pitfalls and recommended practices · Data leakage
All sources (1)
Edition and review
Version 0.1.0
No review details are supplied for this edition.