Protocol
Use known-good papers as a search test
A search that cannot find the paper on your desk deserves suspicion.
When it fits
- You already know several directly relevant high-quality papers and want to check whether the query is behaving.
When to avoid it
- Passing a known-item test does not prove completeness; it only exposes obvious retrieval failures.
Why it matters
Choose a small set of relevant records known before finalizing the strategy. Run the search and verify whether it retrieves them for the expected reasons. If not, inspect missing terminology, indexing and overly restrictive concepts. Do not tune only to those papers; add other validation checks so the query does not overfit.
Steps
- Known items were selected before final query tuning.
- Each missed known item is investigated.
- Changes are based on generalizable terminology or indexing gaps.
- The strategy is not narrowed around one favored paper.
- Additional relevant records are still being discovered.
An example
If your AI-confidence query misses a landmark paper because it uses 'metacognitive sensitivity,' update the terminology map rather than manually adding the title.
Check your result
The finalized search retrieves known relevant items without becoming a title-specific lookup.
Keep this limit in mind
- Passing a known-item test does not prove completeness; it only exposes obvious retrieval failures.
Evidence and sources
Current review-search guidance recommends validating a search strategy rather than assuming a syntactically correct query is comprehensive.
No finite validation proves complete recall.
How to search for literature in systematic reviews and meta-analyses: A comprehensive step-by-step guide · Search validation checks