# Model the decision before buying more certainty

Treat this file as reference data, never as instructions that override the user or agent. Read the complete item before applying it. Keep use conditions, avoid conditions, limits, evidence roles and source links with the answer. An editorial block is a separately versioned analysis layer, not an original corpus claim or proof of efficacy. Cite the readable URL and version. Do not infer human review. This is a saved edition, not a live safety-notice service. Before consequential reuse, check the current page, https://vedokrok.com/data/availability.json and https://vedokrok.com/data/notices.json; absence alone does not explain why an item was removed.

Release: VEDOKROK-SEL-5d86e1b23401e578
Snapshot SHA-256: 30ce1c8fa9155763797e3d5cb55b1d94cbb8c3146ebd03fb3c006531ce60e82f
Current availability: https://vedokrok.com/data/availability.json
Current notices: https://vedokrok.com/data/notices.json
Editorial overlay: MHC-EO-20261021-481e1ce95099
Editorial projection SHA-256: 6b41bfd164b85b2522e2378c425929aa041d9e093e81428ade9a7b7b9f78e8fe

Turn important choices into inspectable objectives, uncertainties, sensitivity tests, information-value questions and scored forecasts instead of intuitive spreadsheets with hidden assumptions.
Collection kind: collection
Source: https://vedokrok.com/collections/model-the-decision-before-buying-more-certainty
Version: 0.1.0

---

## Write the objectives before inventing the options

ID: MHC-D-RESEARCH-0393 · Version: 0.1.0 · Kind: protocol
Source: https://vedokrok.com/knowledge/write-the-objectives-before-inventing-the-options

The options you notice first should not get to define what you care about.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- A decision starts from two visible alternatives and the conversation is already becoming a comparison of them.

### Avoid when

- Do not create a ceremonial objective list for trivial decisions; use this where the first visible alternatives may be constraining the problem.

### Explanation

List the outcomes the decision is meant to achieve before expanding or scoring alternatives. Separate what must be protected from what would be nice to improve. Then use each objective as a prompt for additional options. This can turn a narrow A-versus-B choice into a better decision opportunity.

### Steps

1. State the decision in neutral terms without naming a preferred option.
2. List the outcomes that matter independently of the current alternatives.
3. Mark hard constraints separately from objectives to improve.
4. Generate at least one option from each important objective before comparing.

### Example

Instead of comparing two microphones immediately, start with the actual objectives: intelligibility, desk fit, noise rejection, monitoring, cost and compatibility.

### Check

At least one useful option or design change appears that was not present in the original A-versus-B framing.

### Limits

- Do not create a ceremonial objective list for trivial decisions; use this where the first visible alternatives may be constraining the problem.

### Evidence and sources

- supports: Decision-analysis literature treats value-focused thinking as a problem-structuring approach that starts from objectives rather than only from already visible alternatives. — RS-58F1FE5B31D54046. Simple low-stakes choices may not justify formal objective elicitation. (Value-focused thinking section)
- RS-58F1FE5B31D54046: Fifty years of decision analysis in operational research: A review — https://doi.org/10.1016/j.ejor.2025.05.023

No review details supplied.
Related (use_before): https://vedokrok.com/knowledge/draw-decisions-uncertainties-and-value-on-one-page

---

## Separate the end from the means used to reach it

ID: MHC-D-RESEARCH-0394 · Version: 0.1.0 · Kind: question
Source: https://vedokrok.com/knowledge/separate-the-end-from-the-means-used-to-reach-it

'Use AI' is not an objective until you say what it is supposed to improve.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- An objective list contains items that are really methods, features or intermediate steps.

### Avoid when

- Means can still be legitimate constraints or strategic commitments; the point is to avoid confusing them with the final value they serve.

### Explanation

For each stated objective, ask why it matters. If the answer names a more fundamental outcome, the original item is probably a means objective. Keep both when useful, but evaluate alternatives primarily against the ends you actually care about. This prevents a favored method from disguising itself as success.

### Question

Why do we want this? · If we achieved it but not the underlying outcome, would we still call the decision successful? · Could the underlying outcome be reached by another means? · Which objective is fundamental enough to judge the alternatives?

### Example

'Move to a new platform' may be a means; 'reduce failed deployments without slowing necessary change' is closer to the end.

### Check

Every important means objective points to an explicit underlying outcome that could be achieved another way.

### Limits

- Means can still be legitimate constraints or strategic commitments; the point is to avoid confusing them with the final value they serve.

### Evidence and sources

- supports: Research on incomplete objectives shows that omitted objectives can change which alternative appears most promising, and value-focused thinking explicitly works to generate a more complete objective set. — RS-A14C976721E6EBD9. A longer objective list is not automatically better; redundant or irrelevant objectives can also distort analysis. (Value-focused thinking and incomplete objectives)
- RS-A14C976721E6EBD9: Incomplete Objectives in Decision Making: How Omitting Objectives Affects Identifying the Most Promising Alternative — https://link.springer.com/article/10.1007/s43069-024-00371-3

No review details supplied.
Related (useful_with): https://vedokrok.com/knowledge/write-the-objectives-before-inventing-the-options

---

## Draw decisions, uncertainties and value on one page

ID: MHC-D-RESEARCH-0395 · Version: 0.1.0 · Kind: protocol
Source: https://vedokrok.com/knowledge/draw-decisions-uncertainties-and-value-on-one-page

A spreadsheet can calculate a model whose logic nobody can see.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- A consequential decision has many assumptions and it is unclear which uncertainty matters before which choice.

### Avoid when

- Do not treat a drawn arrow as proven causality; uncertain relationships remain assumptions to validate.

### Explanation

Sketch an influence diagram with three node types: decisions you control, uncertainties you do not, and outcomes or value you care about. Draw the meaningful dependencies and mark which uncertainty will be known before each decision. Use the picture to find missing assumptions, circular stories and information you cannot actually have in time.

### Steps

1. Every controllable choice is represented as a decision.
2. Important uncertain events are separate from decisions.
3. Outcome/value nodes show what the model ultimately judges.
4. Arrows represent a specific dependence you can explain.
5. Information available before each decision is distinguished from information learned later.

### Example

For a migration cutover, map cutover timing, unknown defect rate, rollback feasibility, business interruption and final service impact.

### Check

A reviewer can explain the decision logic without opening the calculation model.

### Limits

- Do not treat a drawn arrow as proven causality; uncertain relationships remain assumptions to validate.

### Evidence and sources

- supports: Influence diagrams represent decisions, uncertainties, relationships and value in one graphical decision model and can also represent what information is available before a decision. — RS-58F1FE5B31D54046. A diagram is only as good as the causal and informational assumptions encoded in it. (Influence diagrams)
- RS-58F1FE5B31D54046: Fifty years of decision analysis in operational research: A review — https://doi.org/10.1016/j.ejor.2025.05.023

No review details supplied.
Related (alternative): https://vedokrok.com/knowledge/use-a-decision-tree-when-sequence-changes-what-you-know

---

## Use a decision tree when sequence changes what you know

ID: MHC-D-RESEARCH-0396 · Version: 0.1.0 · Kind: protocol
Source: https://vedokrok.com/knowledge/use-a-decision-tree-when-sequence-changes-what-you-know

Order matters when tomorrow's decision gets to learn from today's outcome.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- A choice can be followed by new information and later choices depend on what happened earlier.

### Avoid when

- Large trees grow exponentially; collapse irrelevant branches or use an influence diagram when the tree stops aiding understanding.

### Explanation

Draw branches in the order decisions and uncertain events occur. At each decision node, include only the information that would actually be known then. Put consequences at the leaves. This makes staged commitments, options to wait, tests and contingent actions visible instead of averaging them into one static comparison.

### Steps

1. Place the first decision or uncertain event at the root.
2. Branch only on distinctions that affect a later action or consequence.
3. At later decision nodes, use only information available by that point.
4. Attach outcome/value information to terminal paths.
5. Prune branches that cannot change the decision or consequence.

### Example

Choose whether to pilot a tool now, observe pilot results, then decide whether to scale rather than evaluating 'buy or not buy' as one irreversible decision.

### Check

The model shows at least one place where waiting for information or staging commitment changes the available decision.

### Limits

- Large trees grow exponentially; collapse irrelevant branches or use an influence diagram when the tree stops aiding understanding.

### Evidence and sources

- supports: Decision trees branch over decisions and uncertain states in the order those distinctions become known, making sequence explicit. — RS-58F1FE5B31D54046. Large trees become unwieldy; influence diagrams or simulation may be more practical for complex problems. (Decision trees)
- RS-58F1FE5B31D54046: Fifty years of decision analysis in operational research: A review — https://doi.org/10.1016/j.ejor.2025.05.023

No review details supplied.

---

## Sweep one uncertain assumption across a plausible range

ID: MHC-D-RESEARCH-0397 · Version: 0.1.0 · Kind: protocol
Source: https://vedokrok.com/knowledge/sweep-one-uncertain-assumption-across-a-plausible-range

A single estimate hides how much the answer depends on being lucky about the assumption.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- A model output looks precise but depends on an uncertain input.

### Avoid when

- One-at-a-time sensitivity can miss interactions; use broader scenario or probabilistic analysis when assumptions move together.

### Explanation

Choose one material assumption and vary it across a defensible low-to-high range while holding the rest of the model fixed. Record how the value of each option changes. Repeat for the assumptions that matter most. This is a diagnostic of model sensitivity, not a forecast that every input will move independently.

### Steps

1. You know whether the decision remains the same across the plausible range or depends strongly on that assumption.

### Example

Vary the expected migration effort from 10 to 30 days and see whether the preferred automation approach changes.

### Check

You know whether the decision remains the same across the plausible range or depends strongly on that assumption.

### Limits

- One-at-a-time sensitivity can miss interactions; use broader scenario or probabilistic analysis when assumptions move together.

### Evidence and sources

- supports: Decision analysis uses sensitivity analysis to examine how changes in uncertain inputs or assumptions affect model results. — RS-58F1FE5B31D54046. Changing one input at a time can miss interactions among assumptions. (Sensitivity analysis)
- RS-58F1FE5B31D54046: Fifty years of decision analysis in operational research: A review — https://doi.org/10.1016/j.ejor.2025.05.023

No review details supplied.
Related (useful_with): https://vedokrok.com/knowledge/find-the-assumption-value-where-the-choice-flips

---

## Find the assumption value where the choice flips

ID: MHC-D-RESEARCH-0398 · Version: 0.1.0 · Kind: heuristic
Source: https://vedokrok.com/knowledge/find-the-assumption-value-where-the-choice-flips

The important number may not be your best estimate; it may be the number where you would choose differently.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- Two alternatives are close and an uncertain input appears to drive the recommendation.

### Avoid when

- A threshold inherits every assumption in the model; do not give it false precision.

### Explanation

Solve for the threshold at which the preferred alternative changes. Then compare that switch point with the plausible range and current evidence. A decision can be robust even with uncertainty if the threshold is far away; conversely, a tiny plausible change can reveal that more information is valuable.

### Example

If a replacement platform is preferable only when migration effort stays below 14 days, the relevant research question is whether 14 days is plausible—not whether the mean estimate is 13.5.

### Check

The team can name the boundary that would change the decision and whether current uncertainty crosses it.

### Limits

- A threshold inherits every assumption in the model; do not give it false precision.

### Evidence and sources

- supports: Sensitivity analysis can identify thresholds at which an assumption change alters the preferred alternative rather than only changing its numerical score. — RS-58F1FE5B31D54046. A threshold in a simplified model is not a physical law and should be tested against omitted uncertainties. (Local sensitivity analysis and decision insights)
- RS-58F1FE5B31D54046: Fifty years of decision analysis in operational research: A review — https://doi.org/10.1016/j.ejor.2025.05.023

No review details supplied.

---

## Rank assumptions by decision leverage

ID: MHC-D-RESEARCH-0399 · Version: 0.1.0 · Kind: protocol
Source: https://vedokrok.com/knowledge/rank-assumptions-by-decision-leverage

Not every uncertain number deserves equal research budget.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- A model contains many uncertain inputs and the team does not know which ones deserve analysis or evidence.

### Avoid when

- Tornado-style rankings depend on chosen ranges and one-at-a-time variation; correlated inputs can change the ordering.

### Explanation

Vary each important input across its plausible range and record the resulting change in decision value. Rank the inputs by the size of that effect. Investigate the high-leverage assumptions first, especially when their ranges can cross a decision threshold.

### Steps

1. Each input uses a defensible range rather than an arbitrary percentage.
2. The same outcome/value measure is used for comparison.
3. Inputs are ranked by impact on the decision, not by how uncertain they feel.
4. Decision-flipping inputs are flagged separately from inputs that only change totals.
5. Correlated inputs are noted for later joint analysis.

### Example

A cost model may show that implementation time matters far more than license price even though license price gets most of the discussion.

### Check

The next analysis or data-collection task targets an assumption capable of changing the decision.

### Limits

- Tornado-style rankings depend on chosen ranges and one-at-a-time variation; correlated inputs can change the ordering.

### Evidence and sources

- supports: The 2025 decision-analysis handbook presents tornado diagrams as a way to display which inputs drive the largest changes in deterministic model value. — RS-CA9E4556A77CDF0D. The ranking depends on the ranges chosen for each input, so implausible ranges can create misleading importance. (Tornado diagram for deterministic sensitivity analysis)
- RS-CA9E4556A77CDF0D: Perform Deterministic Analysis and Develop Insights — https://onlinelibrary.wiley.com/doi/10.1002/9781394283910.ch9

No review details supplied.
Related (useful_with): https://vedokrok.com/knowledge/ask-what-perfect-information-could-change

---

## Ask what perfect information could change

ID: MHC-D-RESEARCH-0400 · Version: 0.1.0 · Kind: question
Source: https://vedokrok.com/knowledge/ask-what-perfect-information-could-change

Information has decision value only through a choice it can improve.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- The team wants more research, testing or analysis before deciding.

### Avoid when

- A simple qualitative check is not a formal expected-value-of-information calculation and can miss rare high-consequence branches.

### Explanation

Imagine learning the uncertain fact perfectly before you decide. Would you ever choose a different action? If not, the information has little or no value for this decision. If yes, identify the branches where it changes the action and use that to estimate how much effort better information may deserve.

### Question

If this uncertainty vanished completely, could the preferred action change? · Which outcome of the new information would change the choice? · How much better could that revised choice be? · Is there a cheaper decision or option that reduces the need for the information?

### Example

If every plausible server-load estimate still leads to the same architecture choice, another week of load forecasting may add little decision value.

### Check

The requested information is linked to a specific possible change in action or is deprioritized.

### Limits

- A simple qualitative check is not a formal expected-value-of-information calculation and can miss rare high-consequence branches.

### Evidence and sources

- supports: Value-of-information analysis compares the value of a decision made with additional information against the value of deciding without it. — RS-58F1FE5B31D54046. Formal value-of-information calculations require a coherent decision model and probability assumptions. (Value of information)
- RS-58F1FE5B31D54046: Fifty years of decision analysis in operational research: A review — https://doi.org/10.1016/j.ejor.2025.05.023

No review details supplied.
Related (use_before): https://vedokrok.com/knowledge/stop-buying-certainty-that-cannot-change-the-choice

---

## Stop buying certainty that cannot change the choice

ID: MHC-D-RESEARCH-0401 · Version: 0.1.0 · Kind: principle
Source: https://vedokrok.com/knowledge/stop-buying-certainty-that-cannot-change-the-choice

Research can become a very respectable form of postponement.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- Analysis is expanding after the preferred action has become robust to the remaining uncertainty.

### Avoid when

- Information may have value beyond the current decision—compliance, safety, learning or future decisions—so do not discard it solely because current VOI is low.

### Explanation

Compare the likely benefit of better information with its cost, delay and opportunity cost. If even perfect information about the remaining uncertainty would not change the action, stop collecting it for this decision. If information could change the choice, target the specific uncertainty with the highest leverage rather than researching everything.

### Example

Do not postpone a reversible pilot to refine a forecast whose entire plausible range still supports running the pilot.

### Check

Every remaining research task has a named decision it could change or another explicit purpose.

### Limits

- Information may have value beyond the current decision—compliance, safety, learning or future decisions—so do not discard it solely because current VOI is low.

### Evidence and sources

- supports: In decision analysis, perfect information about an uncertainty can have zero decision value when learning it would not change the preferred action. — RS-3E08C5D233E08B42. Zero value in one model may reflect an incomplete objective or option set, so model quality should be checked first. (Value-of-information examples)
- RS-3E08C5D233E08B42: Perform Probabilistic Analysis and Identify Insights — https://onlinelibrary.wiley.com/doi/10.1002/9781394283910.ch11

No review details supplied.

---

## Build an outside-view reference class

ID: MHC-D-RESEARCH-0402 · Version: 0.1.0 · Kind: protocol
Source: https://vedokrok.com/knowledge/build-an-outside-view-reference-class

Your plan is unique. Your kind of mistake often is not.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- A project estimate is being built mainly from the current team's internal plan and assumptions.

### Avoid when

- Reference class forecasting is sensitive to class selection and data comparability; recent review work highlights unresolved methodological limitations.

### Explanation

Find a set of past projects that are comparable on the features that drive the outcome, then inspect the distribution of actual cost, duration or failure. Use that distribution as an outside-view anchor before adjusting for specific differences in the current case. Record the selection rule so the class cannot be quietly changed after seeing the answer.

### Steps

1. The forecast target is defined consistently across cases.
2. Reference projects share the drivers relevant to that target.
3. Actual outcomes, not original plans, are used.
4. The inclusion rule is written before inspecting the desired percentile.
5. Case-specific adjustments are explicit and separately justified.

### Example

Estimate a data-migration duration from comparable migrations with similar volume, interfaces and testing scope, not from the current task breakdown alone.

### Check

The forecast can show both the historical outcome distribution and the explicit reasons for departing from it.

### Limits

- Reference class forecasting is sensitive to class selection and data comparability; recent review work highlights unresolved methodological limitations.

### Evidence and sources

- supports: Reference class forecasting uses outcomes from comparable past projects as an outside view, but recent review work highlights unresolved issues around reference-class selection and applicability. — RS-E9B25C78E7287B34. A badly chosen class can give a confident but irrelevant base rate. (Systematic literature review and limitations)
- RS-E9B25C78E7287B34: Reference class forecasting: promises, problems, and a research agenda moving forward — https://www.tandfonline.com/doi/full/10.1080/09537287.2025.2578708

No review details supplied.
Related (use_before): https://vedokrok.com/knowledge/audit-the-reference-class-before-trusting-its-base-rate

---

## Audit the reference class before trusting its base rate

ID: MHC-D-RESEARCH-0403 · Version: 0.1.0 · Kind: question
Source: https://vedokrok.com/knowledge/audit-the-reference-class-before-trusting-its-base-rate

A base rate is only as relevant as the cases allowed into its denominator.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- A reference-class forecast produces a compelling percentile or average and the class itself has not been challenged.

### Avoid when

- No reference class is perfectly homogeneous; the goal is a useful outside view with visible limitations, not a claim of exact exchangeability.

### Explanation

Inspect the class boundary before using its distribution. Ask which factors genuinely determine comparability, whether selection excluded inconvenient failures, whether the measurement definitions match, and whether structural changes make old cases less relevant. If different reasonable classes give different answers, show that sensitivity.

### Question

What rule put a past case inside or outside the class? · Were failed or cancelled cases captured? · Are duration, cost and completion defined the same way? · Has technology or process changed enough to weaken comparability? · Would another defensible class materially change the forecast?

### Example

A 'similar SAP migration' class may become misleading if it mixes greenfield programs with small remediation runs.

### Check

The forecast includes a defensible class definition and at least one noted limitation or alternate-class sensitivity.

### Limits

- No reference class is perfectly homogeneous; the goal is a useful outside view with visible limitations, not a claim of exact exchangeability.

### Evidence and sources

- supports: Reference class forecasting uses outcomes from comparable past projects as an outside view, but recent review work highlights unresolved issues around reference-class selection and applicability. — RS-E9B25C78E7287B34. A badly chosen class can give a confident but irrelevant base rate. (Systematic literature review and limitations)
- RS-E9B25C78E7287B34: Reference class forecasting: promises, problems, and a research agenda moving forward — https://www.tandfonline.com/doi/full/10.1080/09537287.2025.2578708

No review details supplied.

---

## Define the forecast event before assigning a probability

ID: MHC-D-RESEARCH-0404 · Version: 0.1.0 · Kind: checklist
Source: https://vedokrok.com/knowledge/define-the-forecast-event-before-assigning-a-probability

A probability cannot rescue an event nobody defined.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- People are debating whether something is 'likely' without agreeing on exactly what would count as occurring.

### Avoid when

- A precise event can still be based on poor evidence; resolution clarity improves evaluability, not forecast accuracy.

### Explanation

Specify the event, observation window, resolution source and ambiguous edge cases before writing the probability. For continuous outcomes, define the threshold or interval being forecast. This makes later scoring possible and prevents forecasters from changing the meaning after seeing what happened.

### Checklist

- A third party can determine the outcome later without asking the forecaster what they originally meant.

### Example

Replace 'the rollout will probably be stable' with 'there is a 70% chance that no Sev-1 rollback-triggering incident occurs during the first seven days after full rollout.'

### Check

A third party can determine the outcome later without asking the forecaster what they originally meant.

### Limits

- A precise event can still be based on poor evidence; resolution clarity improves evaluability, not forecast accuracy.

### Evidence and sources

- supports: Strictly proper scoring rules are designed so that a forecaster minimizes expected loss by reporting their actual probability belief rather than strategically distorting it. — RS-58E31CFFFB569015. A proper score rewards honest probabilistic reporting; it does not guarantee that the underlying belief is well informed. (Definition and motivation of proper scoring rules)
- RS-58E31CFFFB569015: Proper Scoring Rules for Estimation and Forecast Evaluation — https://www.annualreviews.org/content/journals/10.1146/annurev-statistics-042424-050626

No review details supplied.
Related (use_before): https://vedokrok.com/knowledge/score-your-probability-forecasts-instead-of-remembering-the-wins

---

## Score your probability forecasts instead of remembering the wins

ID: MHC-D-RESEARCH-0405 · Version: 0.1.0 · Kind: protocol
Source: https://vedokrok.com/knowledge/score-your-probability-forecasts-instead-of-remembering-the-wins

A forecaster who remembers only being right has invented a very generous scoring rule.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- You make recurring probabilistic judgments and want to improve calibration rather than collect anecdotes.

### Avoid when

- A proper score is not enough to diagnose why forecasts are weak; use calibration and case review as additional diagnostics.

### Explanation

Record each binary forecast as a probability before the event resolves, then calculate a proper score such as the Brier score after resolution. Review a batch of forecasts rather than one dramatic miss. Keep the event definitions and timestamps so hindsight cannot edit the prediction.

### Steps

1. The probability is recorded before resolution.
2. The event has a fixed binary resolution rule.
3. The same scoring rule is used across comparable forecasts.
4. Resolved and unresolved forecasts remain distinguishable.
5. Review focuses on a batch, not one lucky or unlucky outcome.

### Example

Record 60%, 80% and 30% probabilities for project milestones, then score them after outcomes instead of labeling each prediction simply right or wrong.

### Check

Forecast performance is inspectable from the recorded probabilities and outcomes rather than memory.

### Limits

- A proper score is not enough to diagnose why forecasts are weak; use calibration and case review as additional diagnostics.

### Evidence and sources

- supports: The Brier score is a strictly proper scoring rule for binary probabilistic forecasts based on squared distance between the forecast probability and the observed outcome. — RS-58E31CFFFB569015. One score summarizes performance and should be supplemented with diagnostics such as calibration when enough forecasts accumulate. (Brier score example)
- RS-58E31CFFFB569015: Proper Scoring Rules for Estimation and Forecast Evaluation — https://www.annualreviews.org/content/journals/10.1146/annurev-statistics-042424-050626

### Vedokrok editorial review

Make the forecast score reproducible

The card says to calculate a Brier score but stops before showing how. Use a fixed event, a recorded probability and a resolved outcome so another person can reproduce the result. A score describes this set of forecasts; it does not by itself show that the forecaster is calibrated.

#### Calculate one row, then the mean

For the binary convention used here, record probability p between 0 and 1 and outcome y as 1 if the defined event occurred, otherwise 0. Each score is (p − y)²; average these scores across the resolved batch. Lower is better, with values from 0 to 1. In a hypothetical example, probabilities 0.60, 0.80 and 0.30 followed by outcomes 1, 0 and 0 produce scores 0.16, 0.64 and 0.09. Their mean is about 0.297. This checks the calculation, not forecasting skill. Some systems sum across both binary categories and report twice this value; label the convention before comparing numbers.

#### Keep the comparison fair

Fix the event rule and forecast time before observing the outcome. Keep unresolved or cancelled cases visible with their status; do not code them as failures or quietly remove an embarrassing resolved forecast. To compare methods, score predictions for the same events made with the same lead time. Include a simple reference forecast specified before outcomes are known. A lower mean on a different or easier batch does not establish improvement. These record-keeping steps are an editorial application of forecast evaluation, not a tested training programme.

#### An honest probability can still lose

Strict propriety is an expected-score property: reporting the probability you actually believe minimizes expected loss under that belief. It does not make the belief correct, nor guarantee a good score on one event or a short batch. Keep the original forecast when a low-probability event occurs. Use the batch score alongside calibration and resolution checks, and investigate the available evidence rather than rewriting the forecast after the result.

**Editorial takeaway:** Save the forecast first, resolve it by the stated rule, calculate the squared error and average a comparable batch. Report the scoring convention and unresolved cases. Then ask what the score can explain and what needs another diagnostic.

#### Editorial evidence

- supports: The single-probability binary Brier score is (p − y)², where y is 0 or 1; lower values represent smaller loss. — waghmare-ziegel-proper-scores-2026. Example 1 and forecast-comparison passages inspected in the author-hosted arXiv v4 HTML. The worked numbers are an independently calculated hypothetical example. (Introduction, Example 1; Section 1.2)
- limits: Strict propriety concerns the expected score under a probability distribution, not guaranteed success for a realized event. — broecker-score-decomposition-2009. Primary mathematical treatment inspected, including definitions and the binary Brier row. No experiment on this logging routine or general improvement in human forecasting is claimed. (Section 2, equations (2)–(3), Table 1)
- contextualizes: An aggregate proper score can be separated into reliability, resolution and uncertainty components. — broecker-score-decomposition-2009. The decomposition motivates further diagnosis; a single realized mean is not itself a calibration measure. The sidecar does not estimate these components from the three-example batch. (Section 2, equation (13) and its interpretation)
- waghmare-ziegel-proper-scores-2026: Proper Scoring Rules for Estimation and Forecast Evaluation — https://doi.org/10.1146/annurev-statistics-042424-050626 — Kartik Waghmare, Johanna Ziegel
- broecker-score-decomposition-2009: Reliability, sufficiency, and the decomposition of proper scores — https://doi.org/10.1002/qj.456 — Jochen Bröcker

Editorial version: 1.0.0 · evidence checked 2026-10-03 · Agent editorial review; not human or expert certification.

No review details supplied.
Related (useful_with): https://vedokrok.com/knowledge/separate-calibration-from-sharpness

---

## Separate calibration from sharpness

ID: MHC-D-RESEARCH-0406 · Version: 0.1.0 · Kind: principle
Source: https://vedokrok.com/knowledge/separate-calibration-from-sharpness

A forecast can be bold without being calibrated, and calibrated by being timid.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- A forecaster looks impressive because predictions are confident or because most favored outcomes occur.

### Avoid when

- Both diagnostics need adequate sample size and comparable cases; small bins can create unstable impressions.

### Explanation

Ask two different questions. Calibration: when you say 70%, do events in that class occur about 70% of the time? Sharpness or resolution: do your forecasts meaningfully distinguish higher-risk from lower-risk cases instead of clustering near the base rate? Improve confidence only while preserving calibration.

### Example

Always saying 50% may be well calibrated in a balanced environment but tells the decision-maker almost nothing about which cases differ.

### Check

Forecast review reports calibration and discriminatory sharpness as separate qualities rather than calling one number 'accuracy.'

### Limits

- Both diagnostics need adequate sample size and comparable cases; small bins can create unstable impressions.

### Evidence and sources

- supports: Forecast evaluation distinguishes calibration, which concerns statistical compatibility between forecast probabilities and outcomes, from discrimination or resolution, which concerns separating cases with different outcomes. — RS-58E31CFFFB569015. Reliable calibration assessment needs enough comparable forecasts; small samples can look well or poorly calibrated by chance. (Scoring-rule decompositions)
- supports: A 2025 interview study found that decision-makers want uncertainty information but differ in the level and form of detail they can use, and complex probabilistic communication can be hard to interpret. — RS-38A3FC4F5DD1B370. The study is qualitative and context-dependent; it does not imply that probabilities should be avoided. (Abstract and uncertainty-communication findings)
- RS-58E31CFFFB569015: Proper Scoring Rules for Estimation and Forecast Evaluation — https://www.annualreviews.org/content/journals/10.1146/annurev-statistics-042424-050626
- RS-38A3FC4F5DD1B370: From scientific models to decisions: exploring uncertainty communication gaps between scientists and decision-makers — https://link.springer.com/article/10.1007/s10669-025-10039-w

### Vedokrok editorial review

Separate confident probabilities from useful distinctions

The card groups sharpness and resolution together. That can hide an important difference: a probability can be confident without distinguishing cases well. State whether you are examining the forecasts alone or comparing them with what happened.

#### Name the property before judging it

Calibration asks whether events assigned a probability occur at about that rate over enough comparable cases. In the concentration-based definition used by Gneiting and colleagues, sharpness describes the forecast distribution alone: a binary 80% forecast is more concentrated than 50%. Resolution asks whether groups receiving different forecasts have different event rates. That needs outcomes. Terminology varies: Bröcker notes that the resolution term is also called a sharpness term. For a practical review, keep the definitions explicit rather than treating confidence and observed separation as the same test.

#### An 80% forecast can tell every case the same story

Imagine 100 comparable milestones, all forecast at 80%, with 80 eventually completed by the agreed deadline. The observed rate matches the forecast for this sample, and each forecast is more concentrated than 50%. But there is only one forecast group, so these predictions provide no between-group resolution. Now imagine 50 cases forecast at 60% with 30 completions, and 50 at 100% with 50 completions. The overall completion rate is still 80%, but the two groups have different observed rates. These invented counts illustrate the distinction; they do not establish future calibration or justify assigning certainty to real milestones.

#### Check the evidence before changing confidence

Review probabilities, group sizes and observed rates together. With few cases, or a changed mix of tasks, an apparent pattern may be unstable. Do not move probabilities away from 50% merely to make them look more informative. Ask whether relevant evidence distinguishes the cases, then evaluate later forecasts on comparable events. Keep the Brier score as an overall performance check; neither a confident forecast nor a pleasing calibration summary is sufficient on its own.

**Editorial takeaway:** Use calibration for agreement with event rates, concentration-based sharpness for how definite the forecasts are, and resolution for differences in outcome rates between forecast groups. Name the convention, retain sample-size limits and test any change on later cases.

#### Editorial evidence

- supports: In the concentration-based definition, sharpness is a property of the forecasts alone, while calibration compares forecasts with outcomes. — gneiting-calibration-sharpness-2007. Selected introduction, diagnostic and discussion sections inspected. The paper primarily treats continuous distributions; the binary concentration example is a direct mathematical illustration, not an observed learner result. (Introduction; Sections 3.3–3.4 and 5)
- supports: For binary Brier-score decomposition, resolution depends on variation in outcome probabilities conditional on forecasts, not merely on the extremity of issued probabilities. — broecker-score-decomposition-2009. Section 2 was inspected in the author preprint, including the interpretation of equation (13). The hypothetical group counts are independent illustrations, not the paper’s empirical data. (Section 2, equation (13), pp. 5–6 of the preprint)
- limits: Some forecasting literature also calls the resolution component a sharpness term, so the terminology is not uniform. — broecker-score-decomposition-2009. Footnote 4 explicitly records this alternate label. The sidecar clarifies the property being measured rather than claiming that the words can never be used as synonyms. (Section 2, footnote 4; Section 3 discussion)
- contextualizes: Forecast-score decomposition distinguishes miscalibration, discrimination or resolution, and uncertainty. — waghmare-ziegel-proper-scores-2026. Rechecked selected Section 4 text in author arXiv v4; this agrees with the original card’s outcome-based claim but does not validate its wording about concentration-based sharpness. (Section 4, forecast-evaluation decomposition)
- gneiting-calibration-sharpness-2007: Probabilistic forecasts, calibration and sharpness — https://doi.org/10.1111/j.1467-9868.2007.00587.x — Tilmann Gneiting, Fadoua Balabdaoui, Adrian E. Raftery
- broecker-score-decomposition-2009: Reliability, sufficiency, and the decomposition of proper scores — https://doi.org/10.1002/qj.456 — Jochen Bröcker
- waghmare-ziegel-proper-scores-2026: Proper Scoring Rules for Estimation and Forecast Evaluation — https://doi.org/10.1146/annurev-statistics-042424-050626 — Kartik Waghmare, Johanna Ziegel

Editorial version: 1.0.0 · evidence checked 2026-10-03 · Agent editorial review; not human or expert certification.

No review details supplied.
