Protocol

Trace missing values

One percentage can hide where values disappear.

About 1 min read

Before you use it

Use when
  • A column has many blanks after merging several upstream feeds.

Keep this limit in mind

  • Scattered blanks do not prove that values are missing at random. Filling a blank does not confirm why it was empty.

An idea to explore.

Try this idea.

Inspect the example, try the idea in your situation, then check your next move.

Keep reading

Why it matters

Count missing required values by source, period and record type, using explicit rules for fields that do not apply. Trace relevant records from raw input through the processing steps. Mark each explanation as confirmed or unconfirmed; grouped rates alone do not tell you why a field is blank. Choose a fix only when the evidence supports it.

Steps

  1. Report missing and eligible counts for relevant groups.
  2. Compare input and output for relevant records.
  3. Separate confirmed non-applicability from unexplained missing values.

An example

Suppose two feeds each supply 100 accounts. All 100 from A have blank country codes after loading; none from B do. The overall 50% missing rate hides a feed-specific gap. Comparing raw input and output can help locate it.

Check your result

The report separates counts, confirmed explanations and open questions, with a justified follow-up or an explicit decision to take no action.

Save and practise this idea

My learning

Read. Recall. Repeat.

Save this idea and return to it on any device.

Use email
Keep exploring

Connected ideas

You are readingTrace missing values

Evidence and sources
Supports

The guidance recommends reporting reasons for missing important data, distinguishing fields that intentionally do not apply, and investigating whether data errors are systemic or arise from a one-off event.

Claim limit: Grouped missing-value counts and input/output tracing are authored diagnostic adaptations. The selected guidance does not make a visible pattern proof of a cause, nor establish random missingness when no pattern is visible. Source reading history: selected_passages; 2026-10-11; Data quality action planning steps 1–3 and 8; previously selected metadata and quality communication guidance. Official page was opened on 2026-10-10 and identified sections inspected. This verifies a narrow reference basis, not the effectiveness or completeness of independently drafted practice steps. On 2026-10-11 action-planning steps 1–3 and 8 were read again for card 9721; this is selected-passage screening, not independent review. On 2026-10-11 the Completeness communication section, action-planning step 7 and root-cause data-journey section were read again for the missingness correction; source screening only. Agent release review: PostgreSQL 18 table expressions, NULL comparisons and aggregates, the UK Government Data Quality Framework and Microsoft Power Query profiling documentation were checked at the cited sections. The original examples and scope limits remain attached.

The Government Data Quality Framework: guidance · Communicating data quality to users: Completeness; Data quality action planning, step 7; Root cause analysis, Understand the data journey

All sources (1)
Edition and review

Version 0.1.0 · Published

Edition checked · 2026-11-12

Read this idea as JSON · Release details