Protocol
Test what should stay unchanged when the input changes
Sometimes you cannot name the whole answer, but you can name what must not change.
When it fits
- You lack a complete answer key but know a relationship that correct outputs must obey.
When to avoid it
- Do not assume that paraphrases preserve meaning or that order is irrelevant in every task. Incorrect invariants create incorrect tests.
Why it matters
Define a justified input transformation and the expected relationship between outputs. Reordering independent records should not change their extracted values; changing a relevant negation should affect the meaning. These behavioral tests expose inconsistencies that a single example can miss.
Steps
- State the task-specific invariant or expected direction before running the test.
- Create a controlled pair that changes only the relevant feature.
- Compare outputs and inspect failures before expanding the test family.
An example
An extractor receives the same independent records in a different order. The per-record results should remain the same even if output order changes.
Check your result
A failure can be explained against a written relationship, not merely a feeling that the answers differ.
Keep this limit in mind
- Do not assume that paraphrases preserve meaning or that order is irrelevant in every task. Incorrect invariants create incorrect tests.
Evidence and sources
CheckList provides perturbation-based invariance and directional tests for expected model behavior under controlled input changes.
A change is only an invariant when the task's meaning and expected answer really should remain unchanged.
CheckList: behavioral testing of NLP models · Perturbing data for INVs and DIRs