Protocol
Review test changes independently from the code they excuse
Green can mean the test moved, not the bug.
When it fits
- The same agent changes implementation and tests until CI turns green.
When to avoid it
- Human-written tests can also be poor; independence reduces correlated failure but does not guarantee correctness.
Why it matters
Inspect deleted tests, weaker assertions, new mocks and changed fixtures separately from implementation logic. Ask whether each test still checks the intended behavior. For important invariants, preserve at least one independently authored or reviewed test that the implementation agent did not redefine.
Steps
- Deleted tests justified.
- Assertions not weakened silently.
- Mocks do not remove the behavior under test.
- Old regression intent preserved.
- Critical invariant has independent review.
An example
An agent replaces an exact authorization result with 'not null'; the test diff is rejected even though CI is green.
Check your result
A passing suite still exercises the intended contract rather than the agent's preferred implementation.
Keep this limit in mind
- Human-written tests can also be poor; independence reduces correlated failure but does not guarantee correctness.
Connected ideas
Use beforeAdd one adversarial case the coding agent did not propose
Evidence and sources
OWASP recommends independent scrutiny of AI-generated test changes because an agent can make a suite pass by weakening, deleting or misdirecting tests.
This is a threat model and practice recommendation, not evidence that every agent routinely corrupts tests.
Secure Coding with AI Cheat Sheet · Test Fabrication and Test Deletion