Protocol
Evaluate what memory retrieves, not just what it stores
A perfect archive can still produce a bad working memory.
When it fits
- A memory-enabled agent still misses old findings or fills context with irrelevant notes.
When to avoid it
- Recall targets can overfit known questions; periodically add new tasks and audit whether the retrieval policy generalizes.
Why it matters
Create recall tests from real tasks: which stored items should appear, which should stay out, and how high the important item should rank. Measure misses and distracting over-recall separately. When a task fails, inspect whether the needed fact was absent from storage, retrieved poorly or ignored after retrieval; those are different failures.
Steps
- Memory quality has a task-level recall and noise test, not only a count of saved items.
An example
For a recurring customer migration, test whether the agent retrieves the current mapping exception without also loading months of unrelated status notes.
Check your result
Memory quality has a task-level recall and noise test, not only a count of saved items.
Keep this limit in mind
- Recall targets can overfit known questions; periodically add new tasks and audit whether the retrieval policy generalizes.
Evidence and sources
The same memory-harness work argues that retrieval ranking and recall policy can determine whether stored memory helps or wastes context.
Results come from a particular research-agent experiment and should be re-tested locally.
Memory Harnesses for Long-Running Research Agents · Bad memory spends tokens in the wrong direction; recall policy as evaluation target