Protocol

Test what should stay unchanged when the input changes

Sometimes you cannot name the whole answer, but you can name what must not change.

About 1 min read

Before you use it

Use when
  • You lack a complete answer key but know a relationship that correct outputs must obey.

Keep this limit in mind

  • Do not assume that paraphrases preserve meaning or that order is irrelevant in every task. Incorrect invariants create incorrect tests.

An idea to explore.

Try this idea.

Inspect the example, try the idea in your situation, then check your next move.

Keep reading

Why it matters

Define a justified input transformation and the expected relationship between outputs. Reordering independent records should not change their extracted values; changing a relevant negation should affect the meaning. These behavioral tests expose inconsistencies that a single example can miss.

Steps

  1. State the task-specific invariant or expected direction before running the test.
  2. Create a controlled pair that changes only the relevant feature.
  3. Compare outputs and inspect failures before expanding the test family.

An example

An extractor receives the same independent records in a different order. The per-record results should remain the same even if output order changes.

Check your result

A failure can be explained against a written relationship, not merely a feeling that the answers differ.

Save and practise this idea

My learning

Read. Recall. Repeat.

Save this idea and return to it on any device.

Use email
Keep exploring

Connected ideas

You are readingTest what should stay unchanged when the input changes

Evidence and sources
Supports

CheckList provides perturbation-based invariance and directional tests for expected model behavior under controlled input changes.

A change is only an invariant when the task's meaning and expected answer really should remain unchanged.

CheckList: behavioral testing of NLP models · Perturbing data for INVs and DIRs

All sources (1)
Edition and review

Version 0.1.0

No review details are supplied for this edition.

Read this idea as JSON · Release details