Protocol
Test what should stay unchanged when the input changes
Sometimes you cannot name the whole answer, but you can name what must not change.
Before you use it
- Use when
- You lack a complete answer key but know a relationship that correct outputs must obey.
Keep this limit in mind
- Do not assume that paraphrases preserve meaning or that order is irrelevant in every task. Incorrect invariants create incorrect tests.
An idea to explore.
Try this idea.
Inspect the example, try the idea in your situation, then check your next move.
Why it matters
Define a justified input transformation and the expected relationship between outputs. Reordering independent records should not change their extracted values; changing a relevant negation should affect the meaning. These behavioral tests expose inconsistencies that a single example can miss.
Steps
- State the task-specific invariant or expected direction before running the test.
- Create a controlled pair that changes only the relevant feature.
- Compare outputs and inspect failures before expanding the test family.
An example
An extractor receives the same independent records in a different order. The per-record results should remain the same even if output order changes.
Check your result
A failure can be explained against a written relationship, not merely a feeling that the answers differ.
Save and practise this idea
My learning
Read. Recall. Repeat.
Save this idea and return to it on any device.
Keep exploring
Connected ideas
You are readingTest what should stay unchanged when the input changes
Ideas that link here
These ideas link to this one. Check their conditions and limits before use.
Useful with this ideaKeep acceptance examples out of the prompt workshopUseful with this idea
Give the second check evidence the first answer did not create
Evidence and sources
CheckList provides perturbation-based invariance and directional tests for expected model behavior under controlled input changes.
A change is only an invariant when the task's meaning and expected answer really should remain unchanged.
CheckList: behavioral testing of NLP models · Perturbing data for INVs and DIRs
All sources (1)
Edition and review
Version 0.1.0
No review details are supplied for this edition.