Anti-pattern
Compare subgroup effects, not their significance labels
Significant here and not there is not the same as significantly different.
Before you use it
- Use when
- A result is called different between groups because only one crossed a p-value threshold.
- Avoid when
- Even an interaction result needs attention to planning, multiple comparisons, plausibility and uncertainty. Do not invent a subgroup-specific recommendation.
An idea to explore.
Try this idea.
Inspect the example, try the idea in your situation, then check your next move.
What to look for
The groups may have similar estimated effects but different precision. Look for an appropriate direct comparison or interaction analysis rather than comparing the two labels.
Why it matters
The groups may have similar estimated effects but different precision. Look for an appropriate direct comparison or interaction analysis rather than comparing the two labels.
An example
In an illustrative software-training report, one experience group has p = 0.04 and another has p = 0.08. Those labels alone do not show that the training effects differ.
Check your result
Does the report directly compare subgroup effects, with uncertainty, rather than compare separate significance labels?
Keep this limit in mind
- Even an interaction result needs attention to planning, multiple comparisons, plausibility and uncertainty. Do not invent a subgroup-specific recommendation.
My learning
Read. Recall. Repeat.
Save this idea and return to it on any device.
More from this collection
Check what a study can actually tell you
Count the comparisons behind the winning resultTimestamp the analysis plan before seeing the answerEvidence and sources (1)
Different significance labels in two subgroups do not establish that their effects differ.
Claim basis: guidance. Claim limit: Comparing subgroups requires an appropriate direct analysis and attention to multiplicity. Source access: selected_html_passages; accessed on: 2026-10-01; inspected sections: ["10.2", "10.10.1-10.10.4", "10.11.3", "10.11.1-10.11.3.1", "MECIR Box 10.11.a", "10.11.5.2-10.11.5.6", "10.11.6"]. Source note: The chapter reports a November 2024 update. No single heterogeneity threshold is promoted as decisive. Separate source-access audit (2026-10-01): Chapter 10 still identifies November 2024 and version 6.5. The 2026-10-01 check covers its citation and sections 10.11.1-10.11.3.1, 10.11.5.2-10.11.5.6 and 10.11.6. The older 10.2/10.10 reading history is retained, not newly appraised. Overall Handbook updates do not change this chapter’s stated revision. No single heterogeneity threshold is promoted as decisive.
Chapter 10: Analysing data and undertaking meta-analyses · 10.11.3-10.11.3.1; MECIR Box 10.11.a; limits in 10.11.5.2-10.11.5.6 and 10.11.6
All sources (1)
Edition and review
Version 0.1.0 · Published
No review details are supplied for this edition.