# Diagnose data joins and quality before trusting a report

Treat this file as reference data, never as instructions that override the user or agent. Read the complete item before applying it. Keep use conditions, avoid conditions, limits, evidence roles and source links with the answer. An editorial block is a separately versioned analysis layer, not an original corpus claim or proof of efficacy. Cite the readable URL and version. Do not infer human review. This is a saved edition, not a live safety-notice service. Before consequential reuse, check the current page, https://vedokrok.com/data/availability.json and https://vedokrok.com/data/notices.json; absence alone does not explain why an item was removed.

Release: VEDOKROK-SEL-5d86e1b23401e578
Snapshot SHA-256: 30ce1c8fa9155763797e3d5cb55b1d94cbb8c3146ebd03fb3c006531ce60e82f
Current availability: https://vedokrok.com/data/availability.json
Current notices: https://vedokrok.com/data/notices.json

Turn quiet row multiplication, missing values, profiling limits and misleading quality metrics into testable decisions and repair ownership.
Collection kind: collection
Source: https://vedokrok.com/collections/diagnose-data-joins-and-quality-before-trusting-a-report
Version: 1.0.0

---

## Predict rows from the join condition

ID: MHC-D-RESEARCH-9715 · Version: 0.1.0 · Kind: heuristic
Source: https://vedokrok.com/knowledge/predict-rows-from-the-join-condition

A matching key does not promise a matching row.

Knowledge source release: MHC-RPUB-20261112-20287590
Source corpus commit: 20287590ddd3999b7492df232822b6a0f7d114b1
Source manifest SHA-256: 2a987660d12de0ce2bde77a421424f7762af1ff77872ef41093fecc33db74a95
Source projection SHA-256: 2807da39bbe6a3dee2db32a76675ac5185f5a33e8b0a5cdd0c22a4731f3378a0

### Use when

- A join returns more or fewer rows than you expected.

### Avoid when

- Count using the actual key expressions and comparison rules. Outer joins or null-safe equality need a different calculation.

### Explanation

For an inner join using only = on the full key, multiply the left and right counts for each shared key and add the products. Exclude NULL keys: = does not match them. Extra conditions can reduce this count; check them before treating fewer rows as an error.

### Steps

1. Write the complete ON condition and any later filters.
2. Compute per-key products for a plain inner equality join with non-null keys.
3. Apply remaining conditions before comparing with the final output.

### Example

Two left rows and two right rows give four pairs. An extra value condition may leave only one.

### Check

The count follows the actual matching rules and the intended row meaning.

### Limits

- Count using the actual key expressions and comparison rules. Outer joins or null-safe equality need a different calculation.

### Evidence and sources

- supports: In PostgreSQL, a qualifying join produces an output row for each matching row pair; joining two independent one-to-many detail tables through the same parent can multiply rows before aggregation. — pg-joins. Claim limit: This describes row multiplicity, not a universal query planner rule; choose aggregation only when the report needs one total per parent, and preserve row pairings when they are meaningful.
Source reading history: selected_passages; 2026-10-11; Sections 7.2.1.1 Joined Tables; 7.2.1.3 Subqueries; 7.2.2 WHERE; 7.2.3 GROUP BY. Version 18 passages rechecked on 2026-10-11: matching row pairs, full ON conditions, derived tables, filtering and grouping. Grounds SQL semantics, not every engine's behavior or the effectiveness of the drafted reporting workflow.
Agent release review: PostgreSQL 18 table expressions, NULL comparisons and aggregates, the UK Government Data Quality Framework and Microsoft Power Query profiling documentation were checked at the cited sections. The original examples and scope limits remain attached. (Sections 7.2.1.1 Joined Tables and 7.2.1.3 Subqueries)
- supports: In PostgreSQL, ordinary comparisons involving NULL yield unknown; IS [NOT] DISTINCT FROM treats NULL as a comparable value. — pg-null. Claim limit: Scalar comparisons in PostgreSQL 18. Composite keys and other engines require their own matching rules; this is not a query-planner guarantee.
Source reading history: selected_passages; 2026-10-11; Section 9.2 Comparison Functions and Operators. Version 18 comparison passages rechecked on 2026-10-11: ordinary comparisons involving NULL and IS [NOT] DISTINCT FROM. Source reading is not production testing or proof of the drafted workflow.
Agent release review: PostgreSQL 18 table expressions, NULL comparisons and aggregates, the UK Government Data Quality Framework and Microsoft Power Query profiling documentation were checked at the cited sections. The original examples and scope limits remain attached. (Section 9.2 Comparison Functions and Operators)
- pg-joins: PostgreSQL 18: Table Expressions — https://www.postgresql.org/docs/18/queries-table-expressions.html
- pg-null: PostgreSQL 18: Comparison Functions and Operators — https://www.postgresql.org/docs/18/functions-comparison.html

Review: agent · Agent editorial and scoped source review · 2026-11-12T18:37:40Z
Published: 2026-11-12T18:37:40Z
Related (use_before): https://vedokrok.com/knowledge/aggregate-each-detail-table-before-joining-two-totals

---

## Keep left rows when filtering

ID: MHC-D-RESEARCH-9717 · Version: 0.1.0 · Kind: principle
Source: https://vedokrok.com/knowledge/keep-left-rows-when-filtering

An invoice filter can make a supplier disappear.

Knowledge source release: MHC-RPUB-20261112-20287590
Source corpus commit: 20287590ddd3999b7492df232822b6a0f7d114b1
Source manifest SHA-256: 2a987660d12de0ce2bde77a421424f7762af1ff77872ef41093fecc33db74a95
Source projection SHA-256: 2807da39bbe6a3dee2db32a76675ac5185f5a33e8b0a5cdd0c22a4731f3378a0

### Use when

- Rows vanish after a filter is added to a left join.

### Avoid when

- This example assumes a non-null invoice ID. Conditions that accept NULL and later filters need separate checks.

### Explanation

LEFT JOIN adds a row with NULL right-hand values when no match satisfies ON. To keep every supplier, put the invoice test in ON. WHERE invoice.status = 'open' removes those NULL rows afterwards. Adding OR invoice.id IS NULL keeps suppliers with no invoice, but still loses those whose invoices are all closed.

### Steps

1. Decide whether the report needs every supplier or only those with an open invoice.
2. Test one supplier with an open invoice, one with only closed invoices and one with none.

### Example

Three suppliers cover those cases. Filtering in ON keeps all three; the equality test in WHERE keeps only the first.

### Check

Compare supplier IDs and invoice values with the intended report, not just the total row count.

### Limits

- This example assumes a non-null invoice ID. Conditions that accept NULL and later filters need separate checks.

### Evidence and sources

- supports: In PostgreSQL, LEFT JOIN adds a row with NULL right-hand values when no pair satisfies ON. A later WHERE condition keeps only rows for which it is true; a right-hand equality test rejects these NULL rows, while a NULL-accepting condition can retain them. — pg-joins. Claim limit: PostgreSQL 18 logical row semantics, not a planner or performance guarantee. A NULL-accepting WHERE condition does not recreate rows already lost because their key matched only nonqualifying right-hand records.
Source reading history: selected_passages; 2026-10-11; Sections 7.2.1.1 Joined Tables; 7.2.1.3 Subqueries; 7.2.2 WHERE; 7.2.3 GROUP BY. Version 18 passages rechecked on 2026-10-11: matching row pairs, full ON conditions, derived tables, filtering and grouping. Grounds SQL semantics, not every engine's behavior or the effectiveness of the drafted reporting workflow.
Agent release review: PostgreSQL 18 table expressions, NULL comparisons and aggregates, the UK Government Data Quality Framework and Microsoft Power Query profiling documentation were checked at the cited sections. The original examples and scope limits remain attached. (Section 7.2.1.1 LEFT OUTER JOIN and ON-versus-WHERE example; Section 7.2.2 WHERE)
- pg-joins: PostgreSQL 18: Table Expressions — https://www.postgresql.org/docs/18/queries-table-expressions.html

Review: agent · Agent editorial and scoped source review · 2026-11-12T18:37:40Z
Published: 2026-11-12T18:37:40Z

---

## Aggregate each detail table before joining two totals

ID: MHC-D-RESEARCH-9719 · Version: 0.1.0 · Kind: protocol
Source: https://vedokrok.com/knowledge/aggregate-each-detail-table-before-joining-two-totals

A join can inflate both totals at once.

Knowledge source release: MHC-RPUB-20261112-20287590
Source corpus commit: 20287590ddd3999b7492df232822b6a0f7d114b1
Source manifest SHA-256: 2a987660d12de0ce2bde77a421424f7762af1ff77872ef41093fecc33db74a95
Source projection SHA-256: 2807da39bbe6a3dee2db32a76675ac5185f5a33e8b0a5cdd0c22a4731f3378a0

### Use when

- Two one-to-many detail tables are both joined to the same parent.

### Avoid when

- Pre-aggregation can erase needed invoice-payment pairings. An absent or NULL total needs a stated meaning; do not silently replace it with zero.

### Explanation

When the report needs one total per parent, sum each child set by the same complete parent key before joining. Use LEFT JOIN if parents without child rows must remain. Do not sum multiplied detail rows and expect GROUP BY to repair the totals.

### Steps

1. Compute invoice and payment totals independently by customer, with the same intended filters.
2. Keep child row and non-null amount counts alongside totals when missing data must be distinguished.
3. Check known totals, missing child records and missing amounts before assigning any zero defaults.

### Example

Four invoices totalling 100 and three payments totalling 18 can yield sums of 300 and 72 in a detail join. Separate aggregates retain 100 and 18.

### Check

Totals match the independent sums, required parents survive, and missing-data cases remain distinguishable from a recorded zero.

### Limits

- Pre-aggregation can erase needed invoice-payment pairings. An absent or NULL total needs a stated meaning; do not silently replace it with zero.

### Evidence and sources

- supports: In PostgreSQL, a qualifying join produces an output row for each matching row pair; joining two independent one-to-many detail tables through the same parent can multiply rows before aggregation. — pg-joins. Claim limit: This describes row multiplicity, not a universal query planner rule; choose aggregation only when the report needs one total per parent, and preserve row pairings when they are meaningful.
Source reading history: selected_passages; 2026-10-11; Sections 7.2.1.1 Joined Tables; 7.2.1.3 Subqueries; 7.2.2 WHERE; 7.2.3 GROUP BY. Version 18 passages rechecked on 2026-10-11: matching row pairs, full ON conditions, derived tables, filtering and grouping. Grounds SQL semantics, not every engine's behavior or the effectiveness of the drafted reporting workflow.
Agent release review: PostgreSQL 18 table expressions, NULL comparisons and aggregates, the UK Government Data Quality Framework and Microsoft Power Query profiling documentation were checked at the cited sections. The original examples and scope limits remain attached. (Sections 7.2.1.1 Joined Tables and 7.2.1.3 Subqueries)
- supports: In PostgreSQL, LEFT JOIN adds a row with NULL right-hand values when no pair satisfies ON. A later WHERE condition keeps only rows for which it is true; a right-hand equality test rejects these NULL rows, while a NULL-accepting condition can retain them. — pg-joins. Claim limit: PostgreSQL 18 logical row semantics, not a planner or performance guarantee. A NULL-accepting WHERE condition does not recreate rows already lost because their key matched only nonqualifying right-hand records.
Source reading history: selected_passages; 2026-10-11; Sections 7.2.1.1 Joined Tables; 7.2.1.3 Subqueries; 7.2.2 WHERE; 7.2.3 GROUP BY. Version 18 passages rechecked on 2026-10-11: matching row pairs, full ON conditions, derived tables, filtering and grouping. Grounds SQL semantics, not every engine's behavior or the effectiveness of the drafted reporting workflow.
Agent release review: PostgreSQL 18 table expressions, NULL comparisons and aggregates, the UK Government Data Quality Framework and Microsoft Power Query profiling documentation were checked at the cited sections. The original examples and scope limits remain attached. (Section 7.2.1.1 LEFT OUTER JOIN and ON-versus-WHERE example; Section 7.2.2 WHERE)
- supports: COUNT(*) counts rows while COUNT(expression) counts only input rows with a non-null expression. — pg-agg. Claim limit: Scalar expression counting in PostgreSQL 18. These are counts of input rows, not automatically distinct entities or valid field values; join and filter rules determine the input. No planner or performance guarantee.
Source reading history: selected_passages; 2026-10-11; Section 9.21, Table 9.62: count and sum; paragraph on aggregates with no selected rows. PostgreSQL 18 count and sum definitions, including empty-input behavior, rechecked on 2026-10-11. Selected passages ground the row/value/total distinction, not production execution or the effectiveness of the drafted reporting workflow.
Agent release review: PostgreSQL 18 table expressions, NULL comparisons and aggregates, the UK Government Data Quality Framework and Microsoft Power Query profiling documentation were checked at the cited sections. The original examples and scope limits remain attached. (Section 9.21 Aggregate Functions, count)
- supports: In PostgreSQL 18, SUM adds non-null input values; if no rows are selected, it returns NULL rather than zero. — pg-agg. Claim limit: Group creation and join type are separate questions. A NULL sum alone does not establish the cause of missing data or a numerical zero; scalar totals also do not establish that all input amounts were present.
Source reading history: selected_passages; 2026-10-11; Section 9.21, Table 9.62: count and sum; paragraph on aggregates with no selected rows. PostgreSQL 18 count and sum definitions, including empty-input behavior, rechecked on 2026-10-11. Selected passages ground the row/value/total distinction, not production execution or the effectiveness of the drafted reporting workflow.
Agent release review: PostgreSQL 18 table expressions, NULL comparisons and aggregates, the UK Government Data Quality Framework and Microsoft Power Query profiling documentation were checked at the cited sections. The original examples and scope limits remain attached. (Section 9.21, Table 9.62 sum and paragraph on no selected rows)
- pg-joins: PostgreSQL 18: Table Expressions — https://www.postgresql.org/docs/18/queries-table-expressions.html
- pg-agg: PostgreSQL 18: Aggregate Functions — https://www.postgresql.org/docs/18/functions-aggregate.html

Review: agent · Agent editorial and scoped source review · 2026-11-12T18:37:40Z
Published: 2026-11-12T18:37:40Z

---

## Detect changes involving NULL

ID: MHC-D-RESEARCH-9720 · Version: 0.1.0 · Kind: protocol
Source: https://vedokrok.com/knowledge/detect-changes-involving-null

'Not equal' can be unknown, not true.

Knowledge source release: MHC-RPUB-20261112-20287590
Source corpus commit: 20287590ddd3999b7492df232822b6a0f7d114b1
Source manifest SHA-256: 2a987660d12de0ce2bde77a421424f7762af1ff77872ef41093fecc33db74a95
Source projection SHA-256: 2807da39bbe6a3dee2db32a76675ac5185f5a33e8b0a5cdd0c22a4731f3378a0

### Use when

- You compare source and target values before a data sync.

### Avoid when

- These comparison results do not define update intent. Check syntax and behavior in your own database; the example covers scalar values.

### Explanation

Ordinary SQL <> (not equal) returns UNKNOWN when either value is NULL, so a filter using it can miss a difference. For scalar values in PostgreSQL, IS DISTINCT FROM returns true when one value is NULL and the other is present; two NULLs return false. Use your database's supported comparison, then decide what the detected difference should do.

### Steps

1. Test five pairs: equal values, unequal values, NULL/NULL, value/NULL and NULL/value.
2. Define whether an incoming NULL means clear the field, unknown or no update. Keep an empty string separate unless your data rules equate it with NULL.

### Example

If the agreed rule says NULL clears a phone number, changing '123' to NULL needs an update. If NULL means no update, detecting the difference alone must not clear the number.

### Check

With IS DISTINCT FROM, the five pairs return false, true, false, true and true respectively. The update also follows the agreed meaning of NULL.

### Limits

- These comparison results do not define update intent. Check syntax and behavior in your own database; the example covers scalar values.

### Evidence and sources

- supports: In PostgreSQL, ordinary comparisons involving NULL yield unknown; IS [NOT] DISTINCT FROM treats NULL as a comparable value. — pg-null. Claim limit: Scalar comparisons in PostgreSQL 18. Composite keys and other engines require their own matching rules; this is not a query-planner guarantee.
Source reading history: selected_passages; 2026-10-11; Section 9.2 Comparison Functions and Operators. Version 18 comparison passages rechecked on 2026-10-11: ordinary comparisons involving NULL and IS [NOT] DISTINCT FROM. Source reading is not production testing or proof of the drafted workflow.
Agent release review: PostgreSQL 18 table expressions, NULL comparisons and aggregates, the UK Government Data Quality Framework and Microsoft Power Query profiling documentation were checked at the cited sections. The original examples and scope limits remain attached. (Section 9.2 Comparison Functions and Operators)
- pg-null: PostgreSQL 18: Comparison Functions and Operators — https://www.postgresql.org/docs/18/functions-comparison.html

Review: agent · Agent editorial and scoped source review · 2026-11-12T18:37:40Z
Published: 2026-11-12T18:37:40Z
Related (useful_with): https://vedokrok.com/knowledge/count-rows-and-non-null-values-separately

---

## Define what complete means

ID: MHC-D-RESEARCH-9721 · Version: 0.1.0 · Kind: checklist
Source: https://vedokrok.com/knowledge/define-what-complete-means

96% of what?

Knowledge source release: MHC-RPUB-20261112-20287590
Source corpus commit: 20287590ddd3999b7492df232822b6a0f7d114b1
Source manifest SHA-256: 2a987660d12de0ce2bde77a421424f7762af1ff77872ef41093fecc33db74a95
Source projection SHA-256: 2807da39bbe6a3dee2db32a76675ac5185f5a33e8b0a5cdd0c22a4731f3378a0

### Use when

- A dashboard says customer data is 96% complete.

### Avoid when

- A high completion score does not establish that populated values are correct.

### Explanation

Name the records and fields required for the task, then define what counts as complete. Show the count that passed and the total checked: the numerator and denominator. State any exclusions before calculating the percentage.

### Checklist

- Write the required population, fields and completion rule.
- Show counts beside the percentage, with exclusions and the assessment date.

### Example

In this example, 180 of 200 invoiced customers have a required billing address: 90%. Another 300 prospects do not need one. Including them as missing addresses would produce 36% and answer a different question.

### Check

Another analyst can reproduce the metric from the definition and counts.

### Limits

- A high completion score does not establish that populated values are correct.

### Evidence and sources

- supports: Data quality assessment should use explicit rules tied to critical data and its intended use, choose metrics suited to each check, and communicate results with their strengths and limitations. — gov-guidance. Claim limit: Official practical guidance, not experimental evidence for the effectiveness of this checklist. Showing the numerator, denominator and exclusions is the card’s practical adaptation, not a quoted mandatory standard.
Source reading history: selected_passages; 2026-10-11; Data quality action planning steps 1–3 and 8; previously selected metadata and quality communication guidance. Official page was opened on 2026-10-10 and identified sections inspected. This verifies a narrow reference basis, not the effectiveness or completeness of independently drafted practice steps. On 2026-10-11 action-planning steps 1–3 and 8 were read again for card 9721; this is selected-passage screening, not independent review. On 2026-10-11 the Completeness communication section, action-planning step 7 and root-cause data-journey section were read again for the missingness correction; source screening only.
Agent release review: PostgreSQL 18 table expressions, NULL comparisons and aggregates, the UK Government Data Quality Framework and Microsoft Power Query profiling documentation were checked at the cited sections. The original examples and scope limits remain attached. (Data quality action planning, steps 1–3 and 8: critical fields, quality rules, metrics and reporting)
- supports: Completeness covers both the presence of expected records and the population of essential values within records; populated values may still be inaccurate. — gov-dims. Claim limit: A framework definition, not a universal field list or proof that the adapted checks diagnose a particular migration failure. Required records and values depend on the task.
Source reading history: selected_passages; 2026-10-11; Six dimensions and user needs; Completeness definition and distinction from accuracy. Official page was opened on 2026-10-10 and identified sections inspected. This verifies a narrow reference basis, not the effectiveness or completeness of independently drafted practice steps. On 2026-10-11 the Completeness definition and its distinction from accuracy were read again for cards 9721/9725; this is selected-passage screening, not independent review.
Agent release review: PostgreSQL 18 table expressions, NULL comparisons and aggregates, the UK Government Data Quality Framework and Microsoft Power Query profiling documentation were checked at the cited sections. The original examples and scope limits remain attached. (Data quality dimensions: Completeness, definition and distinction from accuracy)
- gov-guidance: The Government Data Quality Framework: guidance — https://www.gov.uk/government/publications/the-government-data-quality-framework/the-government-data-quality-framework-guidance
- gov-dims: The Government Data Quality Framework — https://www.gov.uk/government/publications/the-government-data-quality-framework/the-government-data-quality-framework

Review: agent · Agent editorial and scoped source review · 2026-11-12T18:37:40Z
Published: 2026-11-12T18:37:40Z
Related (useful_with): https://vedokrok.com/knowledge/trace-missing-values

---

## Do not mistake a valid format for a true value

ID: MHC-D-RESEARCH-9722 · Version: 0.1.0 · Kind: principle
Source: https://vedokrok.com/knowledge/do-not-mistake-a-valid-format-for-a-true-value

Well-formed wrong data is still wrong.

Knowledge source release: MHC-RPUB-20261112-20287590
Source corpus commit: 20287590ddd3999b7492df232822b6a0f7d114b1
Source manifest SHA-256: 2a987660d12de0ce2bde77a421424f7762af1ff77872ef41093fecc33db74a95
Source projection SHA-256: 2807da39bbe6a3dee2db32a76675ac5185f5a33e8b0a5cdd0c22a4731f3378a0

### Use when

- A data feed passes every format and range check.

### Avoid when

- A reference can itself be stale or biased; do not claim ground truth merely because two systems agree.

### Explanation

Validity asks whether a value fits rules such as allowed codes and date formats. Accuracy asks whether it corresponds to the real-world entity. Test plausibility and, for high-impact data, compare with a trustworthy reference where available.

### Steps

1. List format rules separately from factual verification controls.
2. Mark fields where no independent reference exists.

### Example

A syntactically valid postcode can still belong to another customer.

### Check

The report distinguishes what was format-checked from what was independently corroborated.

### Limits

- A reference can itself be stale or biased; do not claim ground truth merely because two systems agree.

### Evidence and sources

- supports: Completeness, uniqueness, consistency, timeliness, validity and accuracy are distinct data-quality dimensions and must be prioritized according to intended use. — gov-dims. Claim limit: The UK government framework is practical guidance, not experimental evidence for the effectiveness of the adapted cards. Apply its dimensions and reporting advice to the specific data use, risk and local governance.
Source reading history: selected_passages; 2026-10-11; Six dimensions and user needs; Completeness definition and distinction from accuracy. Official page was opened on 2026-10-10 and identified sections inspected. This verifies a narrow reference basis, not the effectiveness or completeness of independently drafted practice steps. On 2026-10-11 the Completeness definition and its distinction from accuracy were read again for cards 9721/9725; this is selected-passage screening, not independent review.
Agent release review: PostgreSQL 18 table expressions, NULL comparisons and aggregates, the UK Government Data Quality Framework and Microsoft Power Query profiling documentation were checked at the cited sections. The original examples and scope limits remain attached. (Six quality dimensions, user needs and trade-offs)
- gov-dims: The Government Data Quality Framework — https://www.gov.uk/government/publications/the-government-data-quality-framework/the-government-data-quality-framework

Review: agent · Agent editorial and scoped source review · 2026-11-12T18:37:40Z
Published: 2026-11-12T18:37:40Z
Related (compare_with): https://vedokrok.com/knowledge/trace-missing-values

---

## Trace missing values

ID: MHC-D-RESEARCH-9723 · Version: 0.1.0 · Kind: protocol
Source: https://vedokrok.com/knowledge/trace-missing-values

One percentage can hide where values disappear.

Knowledge source release: MHC-RPUB-20261112-20287590
Source corpus commit: 20287590ddd3999b7492df232822b6a0f7d114b1
Source manifest SHA-256: 2a987660d12de0ce2bde77a421424f7762af1ff77872ef41093fecc33db74a95
Source projection SHA-256: 2807da39bbe6a3dee2db32a76675ac5185f5a33e8b0a5cdd0c22a4731f3378a0

### Use when

- A column has many blanks after merging several upstream feeds.

### Avoid when

- Scattered blanks do not prove that values are missing at random. Filling a blank does not confirm why it was empty.

### Explanation

Count missing required values by source, period and record type, using explicit rules for fields that do not apply. Trace relevant records from raw input through the processing steps. Mark each explanation as confirmed or unconfirmed; grouped rates alone do not tell you why a field is blank. Choose a fix only when the evidence supports it.

### Steps

1. Report missing and eligible counts for relevant groups.
2. Compare input and output for relevant records.
3. Separate confirmed non-applicability from unexplained missing values.

### Example

Suppose two feeds each supply 100 accounts. All 100 from A have blank country codes after loading; none from B do. The overall 50% missing rate hides a feed-specific gap. Comparing raw input and output can help locate it.

### Check

The report separates counts, confirmed explanations and open questions, with a justified follow-up or an explicit decision to take no action.

### Limits

- Scattered blanks do not prove that values are missing at random. Filling a blank does not confirm why it was empty.

### Evidence and sources

- supports: The guidance recommends reporting reasons for missing important data, distinguishing fields that intentionally do not apply, and investigating whether data errors are systemic or arise from a one-off event. — gov-guidance. Claim limit: Grouped missing-value counts and input/output tracing are authored diagnostic adaptations. The selected guidance does not make a visible pattern proof of a cause, nor establish random missingness when no pattern is visible.
Source reading history: selected_passages; 2026-10-11; Data quality action planning steps 1–3 and 8; previously selected metadata and quality communication guidance. Official page was opened on 2026-10-10 and identified sections inspected. This verifies a narrow reference basis, not the effectiveness or completeness of independently drafted practice steps. On 2026-10-11 action-planning steps 1–3 and 8 were read again for card 9721; this is selected-passage screening, not independent review. On 2026-10-11 the Completeness communication section, action-planning step 7 and root-cause data-journey section were read again for the missingness correction; source screening only.
Agent release review: PostgreSQL 18 table expressions, NULL comparisons and aggregates, the UK Government Data Quality Framework and Microsoft Power Query profiling documentation were checked at the cited sections. The original examples and scope limits remain attached. (Communicating data quality to users: Completeness; Data quality action planning, step 7; Root cause analysis, Understand the data journey)
- gov-guidance: The Government Data Quality Framework: guidance — https://www.gov.uk/government/publications/the-government-data-quality-framework/the-government-data-quality-framework-guidance

Review: agent · Agent editorial and scoped source review · 2026-11-12T18:37:40Z
Published: 2026-11-12T18:37:40Z
Related (use_before): https://vedokrok.com/knowledge/give-a-data-defect-an-owner

---

## Prioritise errors by their impact

ID: MHC-D-RESEARCH-9724 · Version: 0.1.0 · Kind: heuristic
Source: https://vedokrok.com/knowledge/prioritise-errors-by-their-impact

The busiest quality dashboard is not necessarily the safest system.

Knowledge source release: MHC-RPUB-20261112-20287590
Source corpus commit: 20287590ddd3999b7492df232822b6a0f7d114b1
Source manifest SHA-256: 2a987660d12de0ce2bde77a421424f7762af1ff77872ef41093fecc33db74a95
Source projection SHA-256: 2807da39bbe6a3dee2db32a76675ac5185f5a33e8b0a5cdd0c22a4731f3378a0

### Use when

- The team can validate dozens of fields but has little repair capacity.

### Avoid when

- The UK guide offers one prioritisation approach, not universal numeric thresholds.

### Explanation

Tie each rule to a business purpose and the consequence of failure. Prioritise a wrong payment destination or invalid legal identifier differently from an optional marketing field. Set acceptable thresholds with the relevant owner rather than copying universal numbers.

### Steps

1. Name the action or decision the field supports.
2. Agree impact, threshold and escalation owner.

### Example

A missing optional preference and a wrong bank account are both one failed cell but not comparable risks.

### Check

The quality backlog ranks issues by consequence and intended use.

### Limits

- The UK guide offers one prioritisation approach, not universal numeric thresholds.

### Evidence and sources

- supports: The issues framework recommends defining a data purpose, quality rules and targets before measuring performance, then prioritising below-target issues using purpose importance, failure impact and performance. — gov-rules. Claim limit: Practical prioritisation guidance, not evidence that its suggested scores or tolerances are universally optimal. Its date-range example is labelled accuracy; this source supports prioritisation here, not the validity/accuracy distinction set out in the companion guidance. Explicit denominators and regression examples are authored adaptations, not requirements established by this page.
Source reading history: selected_passages; 2026-10-11; Rules and purpose, importance/impact, targets and priority; 16 April 2026 update. Official page was opened on 2026-10-10 and identified sections inspected. This verifies a narrow reference basis, not the effectiveness or completeness of independently drafted practice steps. On 2026-10-11 these prioritisation sections were read again. The date-range example uses accuracy where the companion guidance distinguishes validity; this limited source use does not adopt that example’s terminology.
Agent release review: PostgreSQL 18 table expressions, NULL comparisons and aggregates, the UK Government Data Quality Framework and Microsoft Power Query profiling documentation were checked at the cited sections. The original examples and scope limits remain attached. (What data quality rules and issues are; Determine importance and impact; Set targets; Assign priority)
- gov-rules: Data quality issues framework — https://www.gov.uk/government/publications/implement-a-data-quality-action-plan/data-quality-issues-framework

Review: agent · Agent editorial and scoped source review · 2026-11-12T18:37:40Z
Published: 2026-11-12T18:37:40Z
Related (useful_with): https://vedokrok.com/knowledge/inspect-failed-data-checks

---

## Check records and fields separately

ID: MHC-D-RESEARCH-9725 · Version: 0.1.0 · Kind: principle
Source: https://vedokrok.com/knowledge/check-records-and-fields-separately

Filled fields cannot account for missing records.

Knowledge source release: MHC-RPUB-20261112-20287590
Source corpus commit: 20287590ddd3999b7492df232822b6a0f7d114b1
Source manifest SHA-256: 2a987660d12de0ce2bde77a421424f7762af1ff77872ef41093fecc33db74a95
Source projection SHA-256: 2807da39bbe6a3dee2db32a76675ac5185f5a33e8b0a5cdd0c22a4731f3378a0

### Use when

- A migration reports a high field-completion rate, but some expected customers may be missing.

### Avoid when

- A populated value may still be wrong. Set explicit rules for fields that do not apply before scoring.

### Explanation

Check which expected records arrived, then check required fields within those records. A field-completion rate among arrived records leaves out records that never arrived. Report the two questions separately so a good rate for one cannot hide a gap in the other.

### Steps

1. Compare expected and arrived record identities within the same scope.
2. Check required fields for the arrived records and label that denominator.

### Example

In this example, 450 of 500 expected accounts arrive. Of those 450, 420 have the required country code. Record coverage is 90%; country-code completion among arrived accounts is about 93.3%.

### Check

Report 450/500 accounts present and 420/450 arrived accounts with a country code. If the task needs every expected account with that field, report 420/500: 84%.

### Limits

- A populated value may still be wrong. Set explicit rules for fields that do not apply before scoring.

### Evidence and sources

- supports: Completeness covers both the presence of expected records and the population of essential values within records; populated values may still be inaccurate. — gov-dims. Claim limit: A framework definition, not a universal field list or proof that the adapted checks diagnose a particular migration failure. Required records and values depend on the task.
Source reading history: selected_passages; 2026-10-11; Six dimensions and user needs; Completeness definition and distinction from accuracy. Official page was opened on 2026-10-10 and identified sections inspected. This verifies a narrow reference basis, not the effectiveness or completeness of independently drafted practice steps. On 2026-10-11 the Completeness definition and its distinction from accuracy were read again for cards 9721/9725; this is selected-passage screening, not independent review.
Agent release review: PostgreSQL 18 table expressions, NULL comparisons and aggregates, the UK Government Data Quality Framework and Microsoft Power Query profiling documentation were checked at the cited sections. The original examples and scope limits remain attached. (Data quality dimensions: Completeness, definition and distinction from accuracy)
- gov-dims: The Government Data Quality Framework — https://www.gov.uk/government/publications/the-government-data-quality-framework/the-government-data-quality-framework

Review: agent · Agent editorial and scoped source review · 2026-11-12T18:37:40Z
Published: 2026-11-12T18:37:40Z
Related (useful_with): https://vedokrok.com/knowledge/define-what-complete-means

---

## Check what Power Query profiles

ID: MHC-D-RESEARCH-9726 · Version: 0.1.0 · Kind: checklist
Source: https://vedokrok.com/knowledge/check-what-power-query-profiles

A clean preview can leave rows unchecked.

Knowledge source release: MHC-RPUB-20261112-20287590
Source corpus commit: 20287590ddd3999b7492df232822b6a0f7d114b1
Source manifest SHA-256: 2a987660d12de0ce2bde77a421424f7762af1ff77872ef41093fecc33db74a95
Source projection SHA-256: 2807da39bbe6a3dee2db32a76675ac5185f5a33e8b0a5cdd0c22a4731f3378a0

### Use when

- Power Query shows no empty values or errors, but you need to check the source table.

### Avoid when

- The column-quality labels do not replace checks of required fields, allowed codes or factual accuracy.

### Explanation

By default, Power Query profiles the first 1,000 rows. Use the profiling message at the lower left of the editor to switch to the entire dataset when needed. Check the query’s filters and row limits too: this setting does not restore rows already excluded from the query.

### Checklist

- Check that the query includes the records needed for the decision.
- Select the needed profiling scope and record it with the result.

### Example

In this example, a query keeps the first 1,000 of 12,000 source rows. Switching to entire-dataset profiling still leaves a blank country code in source row 12,000 outside the query.

### Check

Record the query, relevant filters or row limits, and whether the profile covers its first 1,000 rows or its entire result.

### Limits

- The column-quality labels do not replace checks of required fields, allowed codes or factual accuracy.

### Evidence and sources

- supports: Power Query data profiling uses the first 1,000 rows by default and offers entire-dataset profiling. In Column distribution, distinct counts different values; unique counts values occurring once in the profiled column. — ms-profile. Claim limit: Tool-specific documented behavior, not a universal definition of unique or an empirical test of Power Query. Counts from a restricted query or partial profile do not establish source-wide coverage or a stable business identifier.
Source reading history: selected_passages; 2026-10-11; Default profiling scope note, Column quality labels and Column distribution definitions. Official page was opened on 2026-10-10 and identified sections inspected. This verifies a narrow reference basis, not the effectiveness or completeness of independently drafted practice steps. On 2026-10-11 those sections were read again for cards 9726/9727. The content was readable despite the page’s generic authorization banner. No application or connector was executed.
Agent release review: PostgreSQL 18 table expressions, NULL comparisons and aggregates, the UK Government Data Quality Framework and Microsoft Power Query profiling documentation were checked at the cited sections. The original examples and scope limits remain attached. (Default profiling scope note and Column distribution definitions)
- supports: Power Query filters reduce the rows available to later transformations and the preview; a Keep First Rows step restricts the query until it is removed. — ms-profile-scope. Claim limit: The card’s conclusion about entire-dataset profiling combines this documented query restriction with the separate profiling-scope documentation. It is a bounded interpretation, not Power Query runtime verification or a reason to remove legitimate filters.
Source reading history: selected_passages; 2026-10-11; Filter data early to improve performance; Use a data subset while developing a query. Sections were opened and read on 2026-10-11: filtering restricts later rows and preview, and Keep First Rows should be removed when the completed query needs the full dataset. Selected documentation passages, not application execution or independent acceptance.
Agent release review: PostgreSQL 18 table expressions, NULL comparisons and aggregates, the UK Government Data Quality Framework and Microsoft Power Query profiling documentation were checked at the cited sections. The original examples and scope limits remain attached. (Filter data early to improve performance; Use a data subset while developing a query)
- supports: Power Query Column quality displays valid, error, empty, unknown and unexpected-error categories for values in the profiled rows. — ms-profile. Claim limit: These are profiling labels. The selected documentation does not establish that the green category checks every business rule or whether a value corresponds to reality.
Source reading history: selected_passages; 2026-10-11; Default profiling scope note, Column quality labels and Column distribution definitions. Official page was opened on 2026-10-10 and identified sections inspected. This verifies a narrow reference basis, not the effectiveness or completeness of independently drafted practice steps. On 2026-10-11 those sections were read again for cards 9726/9727. The content was readable despite the page’s generic authorization banner. No application or connector was executed.
Agent release review: PostgreSQL 18 table expressions, NULL comparisons and aggregates, the UK Government Data Quality Framework and Microsoft Power Query profiling documentation were checked at the cited sections. The original examples and scope limits remain attached. (Column quality, list of five categories and percentage display)
- ms-profile: Using the data profiling tools - Power Query — https://learn.microsoft.com/en-us/power-query/data-profiling-tools
- ms-profile-scope: Best practices when working with Power Query — https://learn.microsoft.com/en-us/power-query/best-practices

Review: agent · Agent editorial and scoped source review · 2026-11-12T18:37:40Z
Published: 2026-11-12T18:37:40Z
Related (useful_with): https://vedokrok.com/knowledge/publish-the-limits-with-the-data-extract

---

## Read distinct and unique in Power Query

ID: MHC-D-RESEARCH-9727 · Version: 0.1.0 · Kind: principle
Source: https://vedokrok.com/knowledge/read-distinct-and-unique-in-power-query

Two similar words answer different counting questions.

Knowledge source release: MHC-RPUB-20261112-20287590
Source corpus commit: 20287590ddd3999b7492df232822b6a0f7d114b1
Source manifest SHA-256: 2a987660d12de0ce2bde77a421424f7762af1ff77872ef41093fecc33db74a95
Source projection SHA-256: 2807da39bbe6a3dee2db32a76675ac5185f5a33e8b0a5cdd0c22a4731f3378a0

### Use when

- Power Query’s Column distribution shows distinct and unique counts.

### Avoid when

- These are Power Query’s labels; verify other tools’ definitions. A result without duplicates does not prove that excluded records were checked or that the key will remain stable.

### Explanation

In Power Query, distinct counts different values. Unique counts values that occur exactly once. Both describe the profiled rows, so a once-seen value in the first 1,000 rows may repeat later. Check the needed scope before using either count to assess a candidate identifier.

### Steps

1. Set the needed query and profiling scope, then inspect repeated values.
2. Check the full identifier if it uses more than one field.

### Example

In this example, codes A,A,B,C give three distinct values and two unique values: B and C. A subset containing only A,B,C would show three of each and hide the repeated A.

### Check

Explain both counts, state the profiling scope and list repeated candidate keys in the records you actually need.

### Limits

- These are Power Query’s labels; verify other tools’ definitions. A result without duplicates does not prove that excluded records were checked or that the key will remain stable.

### Evidence and sources

- supports: Power Query data profiling uses the first 1,000 rows by default and offers entire-dataset profiling. In Column distribution, distinct counts different values; unique counts values occurring once in the profiled column. — ms-profile. Claim limit: Tool-specific documented behavior, not a universal definition of unique or an empirical test of Power Query. Counts from a restricted query or partial profile do not establish source-wide coverage or a stable business identifier.
Source reading history: selected_passages; 2026-10-11; Default profiling scope note, Column quality labels and Column distribution definitions. Official page was opened on 2026-10-10 and identified sections inspected. This verifies a narrow reference basis, not the effectiveness or completeness of independently drafted practice steps. On 2026-10-11 those sections were read again for cards 9726/9727. The content was readable despite the page’s generic authorization banner. No application or connector was executed.
Agent release review: PostgreSQL 18 table expressions, NULL comparisons and aggregates, the UK Government Data Quality Framework and Microsoft Power Query profiling documentation were checked at the cited sections. The original examples and scope limits remain attached. (Default profiling scope note and Column distribution definitions)
- ms-profile: Using the data profiling tools - Power Query — https://learn.microsoft.com/en-us/power-query/data-profiling-tools

Review: agent · Agent editorial and scoped source review · 2026-11-12T18:37:40Z
Published: 2026-11-12T18:37:40Z
Related (useful_with): https://vedokrok.com/knowledge/replay-examples-before-changing-a-data-rule

---

## Count rows and non-null values separately

ID: MHC-D-RESEARCH-9728 · Version: 0.1.0 · Kind: principle
Source: https://vedokrok.com/knowledge/count-rows-and-non-null-values-separately

A row can exist even when the invoice does not.

Knowledge source release: MHC-RPUB-20261112-20287590
Source corpus commit: 20287590ddd3999b7492df232822b6a0f7d114b1
Source manifest SHA-256: 2a987660d12de0ce2bde77a421424f7762af1ff77872ef41093fecc33db74a95
Source projection SHA-256: 2807da39bbe6a3dee2db32a76675ac5185f5a33e8b0a5cdd0c22a4731f3378a0

### Use when

- You need to count invoices or check whether a field is filled.

### Avoid when

- Empty strings and whitespace are not SQL NULL automatically. Repeated values are counted again; define missingness and row meaning before interpreting the gap.

### Explanation

COUNT(*) counts input rows; COUNT(invoice.id) skips rows where that ID is NULL. A LEFT JOIN can retain a supplier through a row with NULL invoice values, so a row count is not automatically an invoice count. Choose the count that answers the report's question.

### Steps

1. Calculate both counts for the same filters and groups; include a known case with no match.
2. Use a field that cannot be NULL in a real invoice when counting matched invoices.

### Example

A supplier with no matching invoice gets COUNT(*) = 1 and COUNT(invoice.id) = 0 in its group after a LEFT JOIN.

### Check

For each total, name what is counted: result rows or rows with a non-null field. Neither count alone proves a distinct entity total.

### Limits

- Empty strings and whitespace are not SQL NULL automatically. Repeated values are counted again; define missingness and row meaning before interpreting the gap.

### Evidence and sources

- supports: COUNT(*) counts rows while COUNT(expression) counts only input rows with a non-null expression. — pg-agg. Claim limit: Scalar expression counting in PostgreSQL 18. These are counts of input rows, not automatically distinct entities or valid field values; join and filter rules determine the input. No planner or performance guarantee.
Source reading history: selected_passages; 2026-10-11; Section 9.21, Table 9.62: count and sum; paragraph on aggregates with no selected rows. PostgreSQL 18 count and sum definitions, including empty-input behavior, rechecked on 2026-10-11. Selected passages ground the row/value/total distinction, not production execution or the effectiveness of the drafted reporting workflow.
Agent release review: PostgreSQL 18 table expressions, NULL comparisons and aggregates, the UK Government Data Quality Framework and Microsoft Power Query profiling documentation were checked at the cited sections. The original examples and scope limits remain attached. (Section 9.21 Aggregate Functions, count)
- supports: In PostgreSQL, LEFT JOIN adds a row with NULL right-hand values when no pair satisfies ON. A later WHERE condition keeps only rows for which it is true; a right-hand equality test rejects these NULL rows, while a NULL-accepting condition can retain them. — pg-joins. Claim limit: PostgreSQL 18 logical row semantics, not a planner or performance guarantee. A NULL-accepting WHERE condition does not recreate rows already lost because their key matched only nonqualifying right-hand records.
Source reading history: selected_passages; 2026-10-11; Sections 7.2.1.1 Joined Tables; 7.2.1.3 Subqueries; 7.2.2 WHERE; 7.2.3 GROUP BY. Version 18 passages rechecked on 2026-10-11: matching row pairs, full ON conditions, derived tables, filtering and grouping. Grounds SQL semantics, not every engine's behavior or the effectiveness of the drafted reporting workflow.
Agent release review: PostgreSQL 18 table expressions, NULL comparisons and aggregates, the UK Government Data Quality Framework and Microsoft Power Query profiling documentation were checked at the cited sections. The original examples and scope limits remain attached. (Section 7.2.1.1 LEFT OUTER JOIN and ON-versus-WHERE example; Section 7.2.2 WHERE)
- pg-agg: PostgreSQL 18: Aggregate Functions — https://www.postgresql.org/docs/18/functions-aggregate.html
- pg-joins: PostgreSQL 18: Table Expressions — https://www.postgresql.org/docs/18/queries-table-expressions.html

Review: agent · Agent editorial and scoped source review · 2026-11-12T18:37:40Z
Published: 2026-11-12T18:37:40Z

---

## Inspect failed data checks

ID: MHC-D-RESEARCH-9729 · Version: 0.1.0 · Kind: heuristic
Source: https://vedokrok.com/knowledge/inspect-failed-data-checks

The small failed slice may be the part that matters.

Knowledge source release: MHC-RPUB-20261112-20287590
Source corpus commit: 20287590ddd3999b7492df232822b6a0f7d114b1
Source manifest SHA-256: 2a987660d12de0ce2bde77a421424f7762af1ff77872ef41093fecc33db74a95
Source projection SHA-256: 2807da39bbe6a3dee2db32a76675ac5185f5a33e8b0a5cdd0c22a4731f3378a0

### Use when

- A new validation reports 99.9% passing records.

### Avoid when

- Sampling can miss rare issues; increase examination where consequences are high.

### Explanation

Inspect a bounded sample of failures by business impact and upstream source. Look for false positives, repeat causes and severe cases hidden by the rate. Pair a numerical threshold with example-level review before declaring success.

### Steps

1. Review representative failing records and the highest-risk failures.
2. Check whether the rule rejected a valid edge case.

### Example

A tax-number format rule flags just three customers, but all are in a legally critical flow.

### Check

The report includes tested examples and follow-up ownership, not only a percentage.

### Limits

- Sampling can miss rare issues; increase examination where consequences are high.

### Evidence and sources

- supports: The issues framework recommends defining a data purpose, quality rules and targets before measuring performance, then prioritising below-target issues using purpose importance, failure impact and performance. — gov-rules. Claim limit: Practical prioritisation guidance, not evidence that its suggested scores or tolerances are universally optimal. Its date-range example is labelled accuracy; this source supports prioritisation here, not the validity/accuracy distinction set out in the companion guidance. Explicit denominators and regression examples are authored adaptations, not requirements established by this page.
Source reading history: selected_passages; 2026-10-11; Rules and purpose, importance/impact, targets and priority; 16 April 2026 update. Official page was opened on 2026-10-10 and identified sections inspected. This verifies a narrow reference basis, not the effectiveness or completeness of independently drafted practice steps. On 2026-10-11 these prioritisation sections were read again. The date-range example uses accuracy where the companion guidance distinguishes validity; this limited source use does not adopt that example’s terminology.
Agent release review: PostgreSQL 18 table expressions, NULL comparisons and aggregates, the UK Government Data Quality Framework and Microsoft Power Query profiling documentation were checked at the cited sections. The original examples and scope limits remain attached. (What data quality rules and issues are; Determine importance and impact; Set targets; Assign priority)
- gov-rules: Data quality issues framework — https://www.gov.uk/government/publications/implement-a-data-quality-action-plan/data-quality-issues-framework

Review: agent · Agent editorial and scoped source review · 2026-11-12T18:37:40Z
Published: 2026-11-12T18:37:40Z
Related (use_before): https://vedokrok.com/knowledge/give-a-data-defect-an-owner

---

## Give a data defect an owner

ID: MHC-D-RESEARCH-9730 · Version: 0.1.0 · Kind: protocol
Source: https://vedokrok.com/knowledge/give-a-data-defect-an-owner

A logged error still needs a next step.

Knowledge source release: MHC-RPUB-20261112-20287590
Source corpus commit: 20287590ddd3999b7492df232822b6a0f7d114b1
Source manifest SHA-256: 2a987660d12de0ce2bde77a421424f7762af1ff77872ef41093fecc33db74a95
Source projection SHA-256: 2807da39bbe6a3dee2db32a76675ac5185f5a33e8b0a5cdd0c22a4731f3378a0

### Use when

- The issue log fills with recurring quality findings.

### Avoid when

- A suspected cause stays unconfirmed until the relevant evidence is inspected. Temporary limits on use do not repair the data.

### Explanation

Record the observed failure, affected use and evidence. Label a suspected cause as unconfirmed. Agree a next action with an owner, deadline and review date; involve whoever controls the relevant source or processing step. If the data is needed before a fix, agree how its use will be limited and tell affected users.

### Steps

1. Agree temporary handling separately from investigation or repair.
2. Recheck affected data after a change using the same scope and record whether the gap recurs.

### Example

Suppose a required country code is present in a raw feed but blank after mapping. The mapping maintainer investigates that step; the data owner agrees how affected records may be used until resolution.

### Check

A named owner has an action, deadline, review date and repeatable data check. Closing the ticket alone does not show that the data improved.

### Limits

- A suspected cause stays unconfirmed until the relevant evidence is inspected. Temporary limits on use do not repair the data.

### Evidence and sources

- supports: The action-plan guide links quality work to critical data and intended use, calls for agreed resolution actions with owners, target timeframes and review dates, and recommends investigating causes and monitoring quality over time. — gov-action. Claim limit: Practical governance guidance, not experimental proof that the adapted protocol prevents recurrence. Responsibility depends on the affected stage and local governance; an upstream supplier is not automatically the source of an error introduced during later processing. Temporary handling and repeatable checks are practical adaptations.
Source reading history: selected_passages; 2026-10-11; Steps 3–5 and 7: assessment, ownership, target timeframe/review date, cause investigation and monitoring; steps 2 and 7: current/refined criteria and before/during/after change comparison. Official page was opened on 2026-10-10 and identified sections inspected. This verifies a narrow reference basis, not the effectiveness or completeness of independently drafted practice steps. On 2026-10-11 these sections were read again for the owned-action correction. No empirical effectiveness, independent or production acceptance is inferred. On 2026-10-11 steps 2 and 7, including Plan for and manage changes, were read for card 9732; its regression examples and data minimisation are authored practice, not empirical or privacy certification.
Agent release review: PostgreSQL 18 table expressions, NULL comparisons and aggregates, the UK Government Data Quality Framework and Microsoft Power Query profiling documentation were checked at the cited sections. The original examples and scope limits remain attached. (Steps 3–5 and 7: assessment, action ownership and timeframe, cause investigation, regular review)
- gov-action: Implementing a data quality action plan — https://www.gov.uk/government/publications/implement-a-data-quality-action-plan/data-quality-action-plan-implementation-guide

Review: agent · Agent editorial and scoped source review · 2026-11-12T18:37:40Z
Published: 2026-11-12T18:37:40Z
Related (use_before): https://vedokrok.com/knowledge/publish-the-limits-with-the-data-extract

---

## Publish the limits with the data extract

ID: MHC-D-RESEARCH-9731 · Version: 0.1.0 · Kind: checklist
Source: https://vedokrok.com/knowledge/publish-the-limits-with-the-data-extract

A dataset without its caveats invites incorrect decisions.

Knowledge source release: MHC-RPUB-20261112-20287590
Source corpus commit: 20287590ddd3999b7492df232822b6a0f7d114b1
Source manifest SHA-256: 2a987660d12de0ce2bde77a421424f7762af1ff77872ef41093fecc33db74a95
Source projection SHA-256: 2807da39bbe6a3dee2db32a76675ac5185f5a33e8b0a5cdd0c22a4731f3378a0

### Use when

- Someone will use a data extract outside the team that produced it.

### Avoid when

- Metadata cannot correct a materially unsafe dataset; withhold or restrict it when the use demands higher quality.

### Explanation

Ship a short data note covering period, population, known gaps, unusual deduplication, unresolved inconsistencies and rules applied. Put the warning where users will find it, not only in a private conversation.

### Checklist

- List the decisions the data can and cannot support.
- Show material exclusions and known stale sections explicitly.

### Example

A customer export excludes inactive accounts and uses month-end snapshots: say both on the handoff.

### Check

A new consumer can interpret coverage without guessing its history.

### Limits

- Metadata cannot correct a materially unsafe dataset; withhold or restrict it when the use demands higher quality.

### Evidence and sources

- supports: Data consumers should be told about coverage gaps, systematic missingness, de-duplication, limitations and the collection period before relying on a dataset. — gov-guidance. Claim limit: The UK government framework is practical guidance, not experimental evidence for the effectiveness of the adapted cards. Apply its dimensions and reporting advice to the specific data use, risk and local governance.
Source reading history: selected_passages; 2026-10-11; Data quality action planning steps 1–3 and 8; previously selected metadata and quality communication guidance. Official page was opened on 2026-10-10 and identified sections inspected. This verifies a narrow reference basis, not the effectiveness or completeness of independently drafted practice steps. On 2026-10-11 action-planning steps 1–3 and 8 were read again for card 9721; this is selected-passage screening, not independent review. On 2026-10-11 the Completeness communication section, action-planning step 7 and root-cause data-journey section were read again for the missingness correction; source screening only.
Agent release review: PostgreSQL 18 table expressions, NULL comparisons and aggregates, the UK Government Data Quality Framework and Microsoft Power Query profiling documentation were checked at the cited sections. The original examples and scope limits remain attached. (Completeness, missingness, deduplication, timeliness and metadata)
- gov-guidance: The Government Data Quality Framework: guidance — https://www.gov.uk/government/publications/the-government-data-quality-framework/the-government-data-quality-framework-guidance

Review: agent · Agent editorial and scoped source review · 2026-11-12T18:37:40Z
Published: 2026-11-12T18:37:40Z

---

## Replay examples before changing a data rule

ID: MHC-D-RESEARCH-9732 · Version: 0.1.0 · Kind: protocol
Source: https://vedokrok.com/knowledge/replay-examples-before-changing-a-data-rule

Stricter can still be wrong.

Knowledge source release: MHC-RPUB-20261112-20287590
Source corpus commit: 20287590ddd3999b7492df232822b6a0f7d114b1
Source manifest SHA-256: 2a987660d12de0ce2bde77a421424f7762af1ff77872ef41093fecc33db74a95
Source projection SHA-256: 2807da39bbe6a3dee2db32a76675ac5185f5a33e8b0a5cdd0c22a4731f3378a0

### Use when

- You change a data-quality rule after discovering errors or exceptions.

### Avoid when

- These cases support a regression check, not acceptance of the whole dataset. This is editorial testing practice; no optimal sample size or privacy guarantee is claimed.

### Explanation

Choose normal, erroneous and boundary cases. For each, record the intended accept/reject result and why it fits the agreed data requirements. Old pass/fail labels are comparison data, not the answer key. Run both rule versions on the same cases and explain every changed result.

### Steps

1. Use invented or redacted examples that preserve the condition under test. Retain only the fields needed; follow local handling rules if real records are necessary.
2. Keep both rule versions, expected results and reasons. Mark uncertain cases for review instead of guessing a label.

### Example

An agreed local code list permits ABC and ABCD. A three-letter-only check rejects ABCD but still accepts forbidden ZZZ. The examples expose both defects.

### Check

The intended errors are caught and justified valid cases remain accepted. Every changed result matches an agreed requirement or remains an explicit review item.

### Limits

- These cases support a regression check, not acceptance of the whole dataset. This is editorial testing practice; no optimal sample size or privacy guarantee is claimed.

### Evidence and sources

- supports: The issues framework recommends defining a data purpose, quality rules and targets before measuring performance, then prioritising below-target issues using purpose importance, failure impact and performance. — gov-rules. Claim limit: Practical prioritisation guidance, not evidence that its suggested scores or tolerances are universally optimal. Its date-range example is labelled accuracy; this source supports prioritisation here, not the validity/accuracy distinction set out in the companion guidance. Explicit denominators and regression examples are authored adaptations, not requirements established by this page.
Source reading history: selected_passages; 2026-10-11; Rules and purpose, importance/impact, targets and priority; 16 April 2026 update. Official page was opened on 2026-10-10 and identified sections inspected. This verifies a narrow reference basis, not the effectiveness or completeness of independently drafted practice steps. On 2026-10-11 these prioritisation sections were read again. The date-range example uses accuracy where the companion guidance distinguishes validity; this limited source use does not adopt that example’s terminology.
Agent release review: PostgreSQL 18 table expressions, NULL comparisons and aggregates, the UK Government Data Quality Framework and Microsoft Power Query profiling documentation were checked at the cited sections. The original examples and scope limits remain attached. (What data quality rules and issues are; Determine importance and impact; Set targets; Assign priority)
- supports: The action-plan guide advises keeping measurement criteria current, reviewing and refining assessment criteria, and comparing data quality before, during and after significant changes. — gov-action. Claim limit: Governance guidance, not experimental proof of this adapted regression procedure. Expected-case labels, synthetic/redacted cases and minimal retained fields are editorial testing choices; this source establishes no optimal sample size or privacy guarantee.
Source reading history: selected_passages; 2026-10-11; Steps 3–5 and 7: assessment, ownership, target timeframe/review date, cause investigation and monitoring; steps 2 and 7: current/refined criteria and before/during/after change comparison. Official page was opened on 2026-10-10 and identified sections inspected. This verifies a narrow reference basis, not the effectiveness or completeness of independently drafted practice steps. On 2026-10-11 these sections were read again for the owned-action correction. No empirical effectiveness, independent or production acceptance is inferred. On 2026-10-11 steps 2 and 7, including Plan for and manage changes, were read for card 9732; its regression examples and data minimisation are authored practice, not empirical or privacy certification.
Agent release review: PostgreSQL 18 table expressions, NULL comparisons and aggregates, the UK Government Data Quality Framework and Microsoft Power Query profiling documentation were checked at the cited sections. The original examples and scope limits remain attached. (Step 2: Identify data quality rules; Step 7: Repeat measure over time, Plan for and manage changes)
- gov-rules: Data quality issues framework — https://www.gov.uk/government/publications/implement-a-data-quality-action-plan/data-quality-issues-framework
- gov-action: Implementing a data quality action plan — https://www.gov.uk/government/publications/implement-a-data-quality-action-plan/data-quality-action-plan-implementation-guide

Review: agent · Agent editorial and scoped source review · 2026-11-12T18:37:40Z
Published: 2026-11-12T18:37:40Z
