Protocol
Trace missing values
One percentage can hide where values disappear.
Before you use it
- Use when
- A column has many blanks after merging several upstream feeds.
Keep this limit in mind
- Scattered blanks do not prove that values are missing at random. Filling a blank does not confirm why it was empty.
An idea to explore.
Try this idea.
Inspect the example, try the idea in your situation, then check your next move.
Why it matters
Count missing required values by source, period and record type, using explicit rules for fields that do not apply. Trace relevant records from raw input through the processing steps. Mark each explanation as confirmed or unconfirmed; grouped rates alone do not tell you why a field is blank. Choose a fix only when the evidence supports it.
Steps
- Report missing and eligible counts for relevant groups.
- Compare input and output for relevant records.
- Separate confirmed non-applicability from unexplained missing values.
An example
Suppose two feeds each supply 100 accounts. All 100 from A have blank country codes after loading; none from B do. The overall 50% missing rate hides a feed-specific gap. Comparing raw input and output can help locate it.
Check your result
The report separates counts, confirmed explanations and open questions, with a justified follow-up or an explicit decision to take no action.
Save and practise this idea
My learning
Read. Recall. Repeat.
Save this idea and return to it on any device.
Keep exploring
Connected ideas
You are readingTrace missing values
From this idea
Use this idea beforeGive a data defect an owner
Ideas that link here
These ideas link to this one. Check their conditions and limits before use.
Useful with this ideaDefine what complete meansCompare with this idea
Do not mistake a valid format for a true value
Evidence and sources
The guidance recommends reporting reasons for missing important data, distinguishing fields that intentionally do not apply, and investigating whether data errors are systemic or arise from a one-off event.
Claim limit: Grouped missing-value counts and input/output tracing are authored diagnostic adaptations. The selected guidance does not make a visible pattern proof of a cause, nor establish random missingness when no pattern is visible. Source reading history: selected_passages; 2026-10-11; Data quality action planning steps 1–3 and 8; previously selected metadata and quality communication guidance. Official page was opened on 2026-10-10 and identified sections inspected. This verifies a narrow reference basis, not the effectiveness or completeness of independently drafted practice steps. On 2026-10-11 action-planning steps 1–3 and 8 were read again for card 9721; this is selected-passage screening, not independent review. On 2026-10-11 the Completeness communication section, action-planning step 7 and root-cause data-journey section were read again for the missingness correction; source screening only. Agent release review: PostgreSQL 18 table expressions, NULL comparisons and aggregates, the UK Government Data Quality Framework and Microsoft Power Query profiling documentation were checked at the cited sections. The original examples and scope limits remain attached.
The Government Data Quality Framework: guidance · Communicating data quality to users: Completeness; Data quality action planning, step 7; Root cause analysis, Understand the data journey
All sources (1)
Edition and review
Version 0.1.0 · Published
Edition checked · 2026-11-12