# Find the user problem before building more machinery

Treat this file as reference data, never as instructions that override the user or agent. Read the complete item before applying it. Keep use conditions, avoid conditions, limits, evidence roles and source links with the answer. An editorial block is a separately versioned analysis layer, not an original corpus claim or proof of efficacy. Cite the readable URL and version. Do not infer human review. This is a saved edition, not a live safety-notice service. Before consequential reuse, check the current page, https://vedokrok.com/data/availability.json and https://vedokrok.com/data/notices.json; absence alone does not explain why an item was removed.

Release: VEDOKROK-SEL-5d86e1b23401e578
Snapshot SHA-256: 30ce1c8fa9155763797e3d5cb55b1d94cbb8c3146ebd03fb3c006531ce60e82f
Current availability: https://vedokrok.com/data/availability.json
Current notices: https://vedokrok.com/data/notices.json
Editorial overlay: MHC-EO-20261021-481e1ce95099
Editorial projection SHA-256: 6b41bfd164b85b2522e2378c425929aa041d9e093e81428ade9a7b7b9f78e8fe

Ten distinct research and pilot methods for observing tasks, testing structure and delivering an initial real benefit.
Collection kind: collection
Source: https://vedokrok.com/collections/find-the-user-problem-before-building-more-machinery
Version: 0.1.0

---

## Walk through the task as a first-time user

ID: MHC-D-RESEARCH-1090 · Version: 0.1.0 · Kind: protocol
Source: https://vedokrok.com/knowledge/walk-through-the-task-as-a-first-time-user

Knowing where the button is can hide how hard it is to find.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- You need to inspect a new flow before arranging user sessions.

### Avoid when

- An expert imagining a newcomer is still an expert. Test important assumptions with representative users.

### Explanation

Write the task and the actions required. At each action, inspect what a newcomer would need to know and what the interface actually reveals. Record possible failures as hypotheses, not as user research results.

### Steps

1. Ask whether the user would form this immediate goal and notice the available action.
2. Ask whether the label connects the action to the goal and the result makes progress clear.
3. Record the missing cue and a change or user test that could examine it.

### Example

A training app offers Start baseline, but nothing explains that this creates a recording. The walkthrough flags the unexplained term.

### Check

Can each predicted difficulty be traced to a particular step and missing piece of information?

### Limits

- An expert imagining a newcomer is still an expert. Test important assumptions with representative users.

### Evidence and sources

- supports: A cognitive walkthrough inspects whether a new user would form the relevant goal, notice an action, connect it to the goal and recognise progress. — RS-9494823B724BD182. Inspectors predict difficulties; a walkthrough does not observe users succeeding. (Four questions for each action.)
- RS-9494823B724BD182: Evaluate Interface Learnability with Cognitive Walkthroughs — https://www.nngroup.com/articles/cognitive-walkthroughs/

### Vedokrok editorial review

Can a new user see the next move?

A cognitive walkthrough can expose where a first-time user loses the connection between a goal, the right action and visible progress. It is much weaker as a substitute for observing real users.

#### Turn a vague review into a task-level inspection

Teams know what their own interface means. A new user does not. A cognitive walkthrough makes that gap inspectable by following one concrete task and asking what a newcomer would need at each step: would they form the immediate goal, notice the available action, connect the label to the goal and recognise progress after acting? This structure can turn “the navigation is confusing” into a reviewable hypothesis. For example, a first-time user who wants a CSV report may see only a control called Share and receive no clear export confirmation. The finding now points to two specific risks: the action may not be associated with the goal, and the result may not confirm progress.

#### Use the method early, but name the instrument

The original method was designed around learnability in walk-up-and-use interfaces. Modern practice applies the same task-based logic to prototypes and product flows, where it can reveal missing cues before a broader study is practical. The reviewer remains part of the measuring instrument. Reviewers choose the task, assume what the user already knows and judge what that user is likely to notice or understand. Those judgments are useful design hypotheses, not observations of representative users. Record the assumed user knowledge, the exact step and the missing cue so a later change or user session can test the prediction.

#### Do not turn one method comparison into a universal ranking

A real-world comparison of heuristic evaluation, guidelines, cognitive walkthroughs and usability testing found different strengths and costs across the four approaches. In that study, the walkthrough helped define user goals and assumptions but was tedious and missed some general or recurring problems. A later methodological review warned that design choices in influential method-comparison experiments weakened simple claims about which usability-evaluation method is generally superior. The practical conclusion is not that walkthroughs fail. It is that the method answers a narrower question about first-use learnability, while other methods may be better for observed behaviour, broad interface coverage or recurring problems.

#### Start with the primary route, then expand by consequence

The main successful route is an efficient first pass: can a plausible newcomer move from the starting state to the intended result without insider knowledge? If that route already breaks, the team has a concrete place to improve. A clean happy path says little about error recovery, alternate routes, accessibility needs, interruptions or behaviour under real conditions. Add those cases when they are common or consequential. Sensitive, safety-critical or hard-to-reverse flows need stronger usability or professional review; confidence from an internal walkthrough is not enough.

#### Keep walkthrough findings in their proper evidence class

A cognitive walkthrough is an inspection, not a user test. It differs from heuristic evaluation because it follows a specific task step by step and concentrates on first-use learnability, while heuristic evaluation checks a broader interface against general usability principles. The methods can complement each other, but neither makes the other redundant. Treat every walkthrough result as a reviewer-generated hypothesis. When a decision depends on what representative users actually notice, misunderstand, avoid or recover from, direct observation provides the stronger evidence.

**Editorial takeaway:** Use a cognitive walkthrough to locate a first-time user's action-and-feedback breaks before broad testing. Tie every finding to one task step, an assumed piece of user knowledge and a missing cue. Then treat the result as a hypothesis about learnability, not as observed user evidence or proof that the whole interface works.

#### Editorial evidence

- supports: The original cognitive-walkthrough paper presents a theoretically structured, question-based evaluation of individual interface actions for walk-up-and-use systems. — UI-PAPER-1990-COGNITIVE-WALKTHROUGH. This supports a task-level inspection of goals, actions and visible progress. It does not turn reviewer predictions into observed user behaviour. (Method description and walkthrough form)
- contextualizes: A real-world study compared heuristic evaluation, software guidelines, cognitive walkthroughs and usability testing on one pre-release interface and reported different advantages and disadvantages for the methods. — UI-PAPER-1991-REAL-WORLD-COMPARISON. The walkthrough helped define user goals and assumptions in this study, but it was also described as tedious and missed some general and recurring problems. One product evaluation does not establish a universal method ranking. (Abstract; Table 6 and Discussion)
- limits: Work applying cognitive walkthroughs to more complex interfaces reported practical issues and recommendations for using the method beyond simple walk-up-and-use systems. — UI-PAPER-1992-COMPLEX-WALKTHROUGHS. This supports preserving task selection, evaluator assumptions and interpretation as part of the method rather than treating a walkthrough as an automatic usability score. (Case study and recommendations)
- challenges: A methodological review of experiments comparing usability-evaluation methods argued that experimental-design choices can substantially weaken the conclusions drawn from those comparisons. — UI-REVIEW-1998-DAMAGED-MERCHANDISE. This challenges broad superiority claims across usability methods. It does not show that cognitive walkthroughs have no value for their narrower learnability question. (Abstract and review conclusions)
- contextualizes: Current practitioner guidance defines a cognitive walkthrough as a task-based inspection of learnability performed by reviewers rather than users and recommends combining it with other evaluation methods. — UI-GUIDE-2022-NNG-WALKTHROUGH. This is practical method guidance, not comparative efficacy evidence. Its distinction between reviewer inspection and user testing supports the editorial boundary used here. (Defining Cognitive Walkthroughs; When Should a Cognitive Walkthrough Be Conducted?)
- UI-PAPER-1990-COGNITIVE-WALKTHROUGH: Testing a walkthrough methodology for theory-based design of walk-up-and-use interfaces — https://doi.org/10.1145/97243.97279 — Clayton Lewis, Peter G. Polson, Cathleen Wharton, John Rieman
- UI-PAPER-1991-REAL-WORLD-COMPARISON: User interface evaluation in the real world: a comparison of four techniques — https://doi.org/10.1145/108844.108862 — Robin Jeffries, James R. Miller, Cathleen Wharton, Kathy Uyeda
- UI-PAPER-1992-COMPLEX-WALKTHROUGHS: Applying cognitive walkthroughs to more complex user interfaces — https://doi.org/10.1145/142750.142864 — Cathleen Wharton, John Rieman, Clayton Lewis, Peter Polson
- UI-REVIEW-1998-DAMAGED-MERCHANDISE: Damaged Merchandise? A Review of Experiments That Compare Usability Evaluation Methods — https://doi.org/10.1207/s15327051hci1303_2 — Wayne D. Gray, Marilyn C. Salzman
- UI-GUIDE-2022-NNG-WALKTHROUGH: Evaluate Interface Learnability with Cognitive Walkthroughs — https://www.nngroup.com/articles/cognitive-walkthroughs/ — Kim Flaherty

Editorial version: 2.0.0 · evidence checked 2026-09-27 · Agent editorial review; not human or expert certification.

No review details supplied.
Related (useful_with): https://vedokrok.com/knowledge/give-the-user-a-goal-and-keep-your-instructions-out-of-the-result

---

## Test where people start, not where you tell them to click

ID: MHC-D-RESEARCH-1091 · Version: 0.1.0 · Kind: protocol
Source: https://vedokrok.com/knowledge/test-where-people-start-not-where-you-tell-them-to-click

The first wrong turn may happen before any form appears.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- People cannot find how to begin an important task.

### Avoid when

- A correct first click does not show that the task can be completed. A static screenshot cannot reveal later interaction failures.

### Explanation

Show the interface with a realistic goal, but no navigation hints. Ask the participant to choose where they would begin. Preserve the actual first choice before asking for an explanation; a reason supplied afterwards is not the click itself.

### Steps

1. Write the need in the participant’s language without repeating the target label.
2. Record the first selected area and any uncertainty.
3. Compare choices with the intended start, then test the rest of the journey separately.

### Example

Ask where someone would go to practise a difficult conversation, rather than asking them to click Practice.

### Check

Does the proposed label or placement change help a fresh participant choose a workable starting point?

### Limits

- A correct first click does not show that the task can be completed. A static screenshot cannot reveal later interaction failures.

### Evidence and sources

- supports: A first-click test records where a participant begins a stated task on an interface or image. — RS-3940FF26CDAFFDE4. It measures the starting choice, not completion or overall usability. (Definition and limitations.)
- RS-3940FF26CDAFFDE4: First click testing guide — https://www.lyssna.com/guides/first-click-testing-guide/

### Vedokrok editorial review

A first click diagnoses the route, not the whole task

A first-click test is useful because it isolates the opening route choice before recovery, search, or moderator help changes the path. That makes it a sharp diagnostic for labels and information scent. It does not show that the rest of the task works, and the result can move when the task wording, available routes, or interface state changes.

#### The first choice is a routing signal

People tend to choose links and regions that appear semantically close to their goal. Blackmon's web-navigation studies found much higher success when one information patch clearly matched the user's goal, while competing patches reduced success when only one led to the answer. This supports a narrow use of first-click testing: it can reveal whether the opening choice has strong information scent or whether several routes look equally plausible. The paper also warns against generalising the model too broadly, so a first click should be read as evidence about this task and this interface, not as a universal law of navigation.

#### The headline effect depends on how the test is built

A famous first-click claim is that choosing the right first path makes eventual success dramatically more likely. A later MeasuringU analysis found a much smaller gap on five live websites: average completion was 78% after an optimal first choice and 70% after a non-optimal one. In separate tree tests, where search and other alternative routes were removed, the gap was far larger: 80% versus 14%. The useful lesson is not one fixed multiplier. The strength of the first-choice signal changes with the task, the available recovery routes, and how 'optimal' is defined.

#### A screenshot removes some of the interface

Static click tests can still be informative, but they are not a perfect stand-in for live use. In a 2023 randomized study of 130 participants across five websites, first-click patterns on screenshots and live sites were broadly similar on average. Yet dynamic functionality produced important local differences, with the largest first-click gap reaching 40% under the stricter coding scheme. Use a static image when the research question is mainly 'where would you start?'. If hover states, search, expanding menus, loading, or later decisions matter, follow with an interactive test.

#### Task wording is part of the measuring instrument

A route can look successful simply because the task text supplied the answer. Nielsen Norman Group's guidance recommends realistic, actionable scenarios that do not repeat the exact interface wording when that wording would give away the path. The opposite mistake is also possible: replacing a normal term with an unnatural synonym can make the task harder for the wrong reason. Write the goal in language a real participant could plausibly use, then check whether the scenario itself points toward one label before treating the click distribution as a design finding.

#### Record the click before you ask for the story

The observed first choice and the participant's later explanation answer different questions. Preserve the click, hesitation, or change of direction first; then ask what the participant expected to find. Post-task explanation is not automatically unreliable. Guan and colleagues compared stimulated retrospective reports with eye movements and found that the reports could give a valid account of attended information and useful detail about strategies, with a low risk of fabrication in their protocol. Treat the explanation as interpretation around the behavior, not as a replacement for the behavior itself.

#### Three questions before you trust the heatmap

Q: Is every non-target first click an error? A: No. Search, in-page links, shortcuts, and alternate navigation can all be valid routes. Define success around a workable path, not only the route the design team expected.

Q: Is a strong first-click score enough? A: No. It says little about later comprehension, form completion, recovery, or dynamic interaction. Test the rest of the journey separately when those outcomes matter.

Q: Should task wording avoid every word that appears in the interface? A: No. Avoid wording that gives away the route, but keep normal domain language when paraphrasing it would make the scenario artificial or confusing.

**Editorial takeaway:** Use first-click testing to diagnose the opening route, not to certify the whole journey. Give a realistic goal without handing over the answer, preserve the first choice before discussion, and treat plausible alternative routes as data rather than automatic errors. When downstream completion or dynamic behavior matters, follow the first-click result with a live interaction test.

#### Editorial evidence

- supports: Initial route choice is strongly influenced by the semantic similarity between the user's goal and available information patches. — blackmon-2011. The studies support information scent as a useful mechanism for interpreting opening navigation choices. (Abstract; Study 1 and Study 2 summary.)
- limits: The information-scent findings should not be treated as a universal model of navigation across all interfaces and tasks. — blackmon-2011. The author explicitly cautions against generalising too broadly from the reported studies. (Abstract conclusion.)
- challenges: The relationship between an optimal first choice and eventual success varies substantially with the testing method and available routes. — sauro-2013. Live-site studies showed a much smaller completion gap than tree tests, challenging any single universal first-click multiplier. (Table 1; First Choices in Navigation Only; conclusions.)
- limits: Static click tests can approximate live-site first-click patterns, but dynamic functionality can produce meaningful local differences. — sauro-et-al-2023. The randomized comparison found broadly similar distributions on average while still observing substantial differences for some interactive regions. (Study Overview; Summary and Discussion.)
- contextualizes: Task scenarios should be realistic and actionable without giving away the interface path through the wording itself. — mccloskey-2014. Task wording is therefore part of the measurement setup rather than a neutral wrapper around the interface. (Summary and task-scenario guidelines.)
- contextualizes: Stimulated retrospective reports can provide useful information about what users attended to and how they reasoned after a task. — guan-et-al-2006. This supports using post-task explanation as a second source of evidence while keeping it distinct from the observed first click. (Abstract.)
- blackmon-2011: Information scent determines attention allocation and link selection among multiple information patches on a webpage — https://doi.org/10.1080/0144929X.2011.599041 — Marilyn Hughes Blackmon
- sauro-2013: The Importance of the First Choice in Website Navigation — https://measuringu.com/first-choice/ — Jeff Sauro
- sauro-et-al-2023: Do Click Tests Predict Live Site Clicks? — https://measuringu.com/do-click-tests-predict-live-site-clicks/ — Jeff Sauro, Will Schiavone, David Du, Jim Lewis
- mccloskey-2014: Turn User Goals into Task Scenarios for Usability Testing — https://www.nngroup.com/articles/task-scenarios-usability-testing/ — Marieke McCloskey
- guan-et-al-2006: The validity of the stimulated retrospective think-aloud method as measured by eye tracking — https://doi.org/10.1145/1124772.1124961 — Zhiwei Guan, Shirley Lee, Elisabeth Cuddihy, Judith Ramey

Editorial version: 1.0.0 · evidence checked 2026-09-26 · Agent editorial review; not human or expert certification.

No review details supplied.
Related (useful_with): https://vedokrok.com/knowledge/give-the-user-a-goal-and-keep-your-instructions-out-of-the-result

---

## Test the navigation before polishing the page

ID: MHC-D-RESEARCH-1092 · Version: 0.1.0 · Kind: protocol
Source: https://vedokrok.com/knowledge/test-the-navigation-before-polishing-the-page

A beautiful menu can still send people to the wrong shelf.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- A growing content library has plausible categories but uncertain findability.

### Avoid when

- Tree testing deliberately removes visual context. A good result still needs validation in the built interface.

### Explanation

Represent the navigation as a simple text hierarchy. Give participants realistic information needs and let them choose a destination. Record routes, backtracking and final choices before changing the labels or structure.

### Steps

1. Choose important retrieval tasks, including items that could plausibly belong in two places.
2. Keep task wording independent of the category labels.
3. Review mistaken routes, revise one suspected grouping or label, and test fresh tasks or participants.

### Example

For a corpus library, ask where to find help checking an AI-generated answer, not where to find AI assurance.

### Check

Can people reach a useful destination without relying on images or moderator hints?

### Limits

- Tree testing deliberately removes visual context. A good result still needs validation in the built interface.

### Evidence and sources

- supports: Tree testing evaluates whether people can locate destinations in a hierarchy without the visual interface. — RS-DEBD82EF96CFBE20. Visual cues and interaction behavior remain outside this test. (Text-only tree and task procedure.)
- RS-DEBD82EF96CFBE20: Tree Testing: Fast, Iterative Evaluation of Menu Labels and Categories — https://www.nngroup.com/articles/tree-testing/

No review details supplied.
Related (compare_with): https://vedokrok.com/knowledge/let-users-group-the-content-before-naming-the-categories
Related (useful_with): https://vedokrok.com/knowledge/give-the-user-a-goal-and-keep-your-instructions-out-of-the-result

---

## Let users group the content before naming the categories

ID: MHC-D-RESEARCH-1093 · Version: 0.1.0 · Kind: protocol
Source: https://vedokrok.com/knowledge/let-users-group-the-content-before-naming-the-categories

The organisation chart is not the reader’s map.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- Your category names reflect the project’s history more than readers’ needs.

### Avoid when

- Similarity judgements are not navigation success. Mixed audiences may need more than one route to the same content.

### Explanation

In an open card sort, participants group representative content and name the groups. Ask what belongs together and why. Use recurring patterns and disagreements to propose structures; do not turn one person’s arrangement into universal taxonomy.

### Steps

1. Prepare understandable content labels without embedding your preferred categories.
2. Let participants group and name them, including uncertain or overlapping items.
3. Compare the rationales, draft a hierarchy and test whether people can find real items in it.

### Example

Readers place an exercise about asking for clarification beside communication skills rather than grammar theory. That is a categorisation hypothesis worth testing.

### Check

Can you explain which user grouping evidence led to each proposed category?

### Limits

- Similarity judgements are not navigation success. Mixed audiences may need more than one route to the same content.

### Evidence and sources

- supports: Open card sorting asks participants to group content and name their groups. — RS-CA8F830846B90092. Preferred groupings do not prove that users can find an item in a finished menu. (Open card sorting definition.)
- RS-CA8F830846B90092: Card Sorting: Uncover Users’ Mental Models — https://www.nngroup.com/articles/card-sorting-definition/

No review details supplied.

---

## Simulate the interaction without pretending the system exists

ID: MHC-D-RESEARCH-1094 · Version: 0.1.0 · Kind: protocol
Source: https://vedokrok.com/knowledge/simulate-the-interaction-without-pretending-the-system-exists

A human behind the prototype is not a production benchmark.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- You need to test a proposed interaction before building its automation.

### Avoid when

- Disclosure may change behavior; concealment introduces ethical issues. Follow an approved research protocol, and never use the prototype for consequential live decisions.

### Explanation

Use a human operator to supply responses while a participant tries a bounded prototype. Explain that parts are simulated and agree what will be recorded. This transparent variant can expose interaction needs without presenting human judgement as an implemented model.

### Steps

1. Specify the task, allowed responses and operator assistance.
2. Run the interaction and log what the operator supplied or repaired.
3. Separate interface findings from capability, reliability and cost questions still untested.

### Example

A speech-practice prototype receives manually prepared feedback. The team records which feedback users understand, not an invented AI accuracy score.

### Check

Can every apparent system capability be traced to implemented behavior or declared human assistance?

### Limits

- Disclosure may change behavior; concealment introduces ethical issues. Follow an approved research protocol, and never use the prototype for consequential live decisions.

### Evidence and sources

- supports: Wizard of Oz prototyping can use a human operator to simulate responses of a proposed system. — RS-DCAC328873CBBBF2. Simulated capability does not establish production performance, capacity or cost. (Method definition and operator role.)
- supports: The government research guidance calls for participants to understand the activity and how their data will be used before agreeing voluntarily. — RS-B809229D425E94B5. This scoped guidance does not settle every legal basis or organisational approval requirement. (Informed consent and explaining the research.)
- RS-DCAC328873CBBBF2: The Wizard of Oz Method in UX — https://www.nngroup.com/articles/wizard-of-oz/
- RS-B809229D425E94B5: Getting informed consent for user research — https://www.gov.uk/service-manual/user-research/getting-users-consent-for-research

### Vedokrok editorial review

Prototype the conversation, not the capability

A human-operated prototype can answer whether an interaction is useful before the automation exists. The trap is treating smooth human performance as evidence that the eventual system will be equally fast, reliable or cheap.

#### Borrow human capability to test the part that is still uncertain

Wizard-of-Oz prototyping is useful when the interaction question arrives before the engineering is ready. A 2026 systematic review of 194 SIGCHI papers found the method used across many domains for reasons including technical limitations, adaptive flexibility, experimental control and cost. That supports a narrow claim: a person can stand in for missing system behaviour so a team can learn about the interaction. It does not show that the missing automation is feasible. A good session therefore starts by naming the question the prototype is meant to answer — for example, whether users understand a feedback format — and leaves accuracy, latency, scale and operating cost for later tests.

#### Transparency changes the research question

Classic Wizard-of-Oz studies usually hide the operator so participants experience the system as autonomous. The current card deliberately uses a transparent variant. That can be the safer and more honest choice for ordinary product discovery, but it changes what the result means. Once people know a human is involved, the session cannot tell you how they would react to apparent machine autonomy in the same way. It can still reveal whether the workflow makes sense, whether a response is useful, what users ask next and where the interaction breaks. In other words, disclosure does not make the prototype useless; it moves the claim from “this is how users respond to an autonomous system” toward “this is how they respond to this interaction with declared human assistance.”

#### The wizard is part of the measuring instrument

Human flexibility is the advantage and the main source of noise. Riek’s systematic review of 54 human–robot interaction experiments found that most studies constrained what the wizard produced, but very few reported measuring wizard error or pre-experiment wizard training. The broader 2026 review again identified wizard variability and bias, timing and performance constraints, ecological validity, deception and ethics, and transparency and reproducibility as recurring concerns. For practical prototyping, constrain everything that is not under study: define allowed responses, record what the operator supplied or repaired, log delays and deviations, and make operator freedom explicit. If the wizard quietly rescues every awkward case, the prototype can become a test of the wizard’s skill rather than the design.

#### Does telling participants about the operator ruin the test?

No, but it rules out some conclusions. A disclosed operator is compatible with testing flow, comprehension, desired response shape and whether an interaction is worth pursuing. It is weaker evidence for questions that depend on believing the system is autonomous, such as perceived agency or some forms of trust. If that distinction matters, write it into the study question before the session rather than explaining it away afterwards.

#### How tightly should the wizard be scripted?

Tightly enough that irrelevant operator differences do not become the result, but not so tightly that you remove the behaviour you are trying to explore. If wording is the variable, standardize timing and available actions. If you are discovering what users may ask, allow bounded flexibility but log every novel response. There is no universal script length; the useful rule is to control what should stay constant and expose what is intentionally allowed to vary.

#### When should the wizard leave the loop?

When the main uncertainty moves from interaction value to implementation performance. Once you need to know real recognition accuracy, model failure modes, latency, throughput, unit cost or behaviour at scale, human simulation becomes the wrong instrument. Houde and Hill’s prototyping framework is useful here: a prototype should be judged by the design question it was built to answer, not by how complete it looks. A convincing Wizard-of-Oz session can justify building the next technical experiment; it cannot stand in for that experiment.

**Editorial takeaway:** Use human simulation to learn cheaply about an interaction before automation exists, and label the boundary precisely. A transparent wizard can test usefulness and flow, but smooth human performance is not evidence of production capability. Script and log the operator, then switch to real implementation tests when feasibility, reliability, latency, cost or scale become the question.

#### Editorial evidence

- supports: A 2026 systematic review coded 194 Wizard-of-Oz papers from SIGCHI venues and found the method used for multiple motivations, including overcoming technical limitations, adaptive flexibility, experimental control and cost-efficiency. — yang-du-du-huang-2026-woz-review. This supports using Wizard-of-Oz as a prototyping and research method. It does not establish that a simulated capability is technically feasible or that one Wizard-of-Oz setup generalizes to every product context. (Abstract; Sections 4.3 and 5.2)
- limits: The 2026 review identifies recurring methodological concerns including temporal and performance constraints, wizard variability and bias, ecological validity, deception and ethics, and transparency and reproducibility. — yang-du-du-huang-2026-woz-review. These concerns are directly relevant to interpreting a human-operated prototype as evidence. The review synthesizes reported concerns across studies rather than estimating one pooled effect size. (Section 4.4)
- challenges: A systematic review of 54 HRI Wizard-of-Oz experiments found that 90.7% constrained wizard production, while 3.7% reported measuring wizard error and 5.4% reported pre-experiment wizard training. — riek-2012-woz-hri-review. The sample is limited to HRI publications from 2001–2011, so the exact percentages should not be generalized to all modern HCI work. They illustrate how underreported operator control can weaken reproducibility. (Abstract)
- contextualizes: Early Wizard-of-Oz methodology work treated human simulation as a way to study proposed interactive behaviour before the final system existed, while emphasizing practical control of the simulated interaction. — dahlback-jonsson-ahrenberg-1993. This is foundational method work focused on natural-language interfaces, not evidence that every transparent human-operated prototype produces valid behavioural estimates. (Article abstract and methodological discussion)
- contextualizes: Houde and Hill argue that prototypes should be designed and interpreted around the specific design question they represent, distinguishing role, look-and-feel and implementation concerns. — houde-hill-1997-prototypes. This prototyping framework supports the editorial distinction between testing interaction value and testing implementation capability; it is not a Wizard-of-Oz efficacy study. (Publisher summary and prototype model)
- yang-du-du-huang-2026-woz-review: Mapping the Wizards' Path: A Systematic Review of Wizard-of-Oz in HCI — https://doi.org/10.1145/3772318.3791174 — Ruoxuan Yang, Yuwei Du, Hongyang Du, Kaibin Huang
- riek-2012-woz-hri-review: Wizard of Oz studies in HRI: a systematic review and new reporting guidelines — https://doi.org/10.5898/JHRI.1.1.Riek — Laurel D. Riek
- dahlback-jonsson-ahrenberg-1993: Wizard of Oz studies — why and how — https://doi.org/10.1016/0950-7051(93)90017-N — Nils Dahlbäck, Arne Jönsson, Lars Ahrenberg
- houde-hill-1997-prototypes: What do Prototypes Prototype? — https://doi.org/10.1016/B978-044481862-1/50082-0 — Stephanie Houde, Charles Hill

Editorial version: 1.0.0 · evidence checked 2026-09-24 · Agent editorial review; not human or expert certification.

No review details supplied.
Related (useful_with): https://vedokrok.com/knowledge/give-the-user-a-goal-and-keep-your-instructions-out-of-the-result
Related (compare_with): https://vedokrok.com/knowledge/deliver-the-small-service-before-automating-its-machinery

---

## Record the episode while its context is still available

ID: MHC-D-RESEARCH-1095 · Version: 0.1.0 · Kind: protocol
Source: https://vedokrok.com/knowledge/record-the-episode-while-its-context-is-still-available

Last Tuesday’s frustration is more useful than usually annoying.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- A problem happens occasionally and interviews produce vague averages.

### Avoid when

- Recording can change behavior. Missing entries are not proof that the event never happened; protect personal details and allow withdrawal.

### Explanation

Ask participants to make a short entry after a relevant event over an agreed period. Capture the situation, attempted action and result. Keep the burden small enough to be realistic, then discuss selected entries rather than treating the diary as complete telemetry.

### Steps

1. Define the event worth recording and provide one simple example.
2. Ask for context and outcome, not a long daily essay.
3. Track missed or delayed entries and explore contrasting episodes in follow-up.

### Example

Someone records when they abandon a practice session: the available time, chosen exercise and point of interruption.

### Check

Do the entries reveal concrete situations that could change the design or the next research question?

### Limits

- Recording can change behavior. Missing entries are not proof that the event never happened; protect personal details and allow withdrawal.

### Evidence and sources

- supports: Diary studies collect participant accounts of activities and experiences over a period of time. — RS-D169CDD346BD1C48. Accounts can be selective, delayed or missing; the proposed small diary is not a validated universal format. (Definition, logging instructions and follow-up.)
- RS-D169CDD346BD1C48: Diary Studies: Understanding Long-Term User Behavior and Experiences — https://www.nngroup.com/articles/diary-studies/

### Vedokrok editorial review

Capture the context, then account for what went missing

A diary moves the report closer to the event, which can preserve context that a later interview turns into a vague average. But it creates a different measurement problem: people can forget, skip, simplify, or change what they do because logging interrupts the task. Treat the diary as sampled self-report, not hidden telemetry.

#### Closer to the event changes the evidence you can collect

Diary methods are useful when the episode is intermittent and the surrounding situation matters. Bolger, Davis and Rafaeli describe diaries as frequent reports of events and experiences in daily life, allowing researchers to capture particulars that traditional retrospective designs often miss. A later review of ecological momentary assessment makes the same advantage more explicit: reporting near the moment can reduce some retrospective bias and preserve variation across contexts and time. This supports the card's core move — record the situation, attempted action and result while they are still recoverable — but not the stronger claim that a near-event report is a complete or objective record.

#### Silence in the diary is ambiguous

No entry can mean that the event did not happen. It can also mean the participant forgot, the prompt arrived at a bad time, the task was too demanding, or the participant gradually stopped engaging. Arndt and Rose show this clearly in an event-contingent diary study: daily reminders increased participation, yet 27% of learners still submitted no diary entries, and participation differed across learner groups. That makes missingness part of the method, not empty space to ignore. Record opportunities, delays and missed entries where practical, then avoid turning 'not logged' into 'did not occur'.

#### The act of recording can disturb the episode

Carter and Mankoff describe a basic trade-off in HCI diary studies. Asking participants to answer questions as events happen can preserve context, but the questions can distract from the main task; without an objective observer, the researcher also cannot fully verify how the log matches the event. Their work on media capture suggests one practical option: make the immediate capture lightweight, then use the captured cue in a later discussion. That reduces interruption but increases the distance from the event. There is no free format: choose how much to capture now according to what would be lost if you wait.

#### Keep the entry small, then do the interpretation later

For ordinary product discovery, a compact event record is usually more useful than a daily essay. Define the event in concrete terms, ask for the immediate context, the attempted action and the result, and keep a timestamp or delay marker when the tool allows it. If a screenshot, short note or other cue can be collected without creating privacy or workload problems, it can support the follow-up. After the diary period, compare contrasting episodes and ask about gaps instead of averaging everything into 'usually'. When system logs or other non-sensitive traces exist, use them to check timing or opportunity; do not treat either source as automatically superior.

#### Questions worth asking before you trust the pattern

Does no entry mean no event? No — first check whether the person had a realistic chance and reason to log it. Should every entry be immediate? Not necessarily; immediate capture can interrupt the task, while delayed capture loses detail, so keep the first capture as small as the research question allows. Can analytics replace the diary? Usually not when the missing information is intention, context or an off-system workaround; analytics can show recorded behaviour, while a diary can add the participant's account of why and what happened around it. Can reminders solve missingness? They may improve participation, but they do not remove self-selection or guarantee that every relevant episode was captured.

**Editorial takeaway:** Use a diary when rare or distributed episodes matter more than a retrospective average. Capture only the context needed to reconstruct the episode, track delay and missingness where possible, and use follow-up or other traces to test the story. A diary is valuable precisely because it is close to real life; it is not a complete log of real life.

#### Editorial evidence

- supports: Frequent diary reports can capture particulars of daily events and experiences that are difficult to recover with more traditional retrospective designs. — bolger-davis-rafaeli-2003-diary-methods. This supports collecting repeated event-close reports. It does not make those reports complete, unbiased or objectively verified. (Abstract)
- contextualizes: Momentary self-report is used partly to reduce retrospective bias, but technology and protocol design create competing concerns around engagement, reporting burden, data validity and honest disclosure. — doherty-balaskas-doherty-2020-ema-design. This supports the core trade-off in the editorial analysis: moving measurement closer to the event can improve context while creating participation and measurement costs. (Abstract; Research Highlights; section 1.1)
- challenges: Participant-captured diary data can interrupt the activity being studied, and researchers may be unable to verify how closely a participant log matches the underlying event without another observation source. — carter-mankoff-2005-diary-media. This challenges any interpretation of a diary as passive telemetry and supports keeping immediate capture lightweight. (Introduction and diary-method discussion)
- limits: In an event-contingent educational diary, reminders improved participation but substantial non-participation remained, and engagement differed across participant groups, creating a risk of self-selection bias. — arndt-rose-2022-diary-data. This limits conclusions from missing entries and from patterns based only on the participants who kept logging. (Abstract; exemplar study two; reflections on participation)
- supports: Event-contingent reports completed soon after an event can preserve contextual details that are harder to obtain from later retrospective questionnaires. — arndt-rose-2022-diary-data. This supports asking for a short context-rich entry near the event while preserving the paper's cautions about burden and selective participation. (Exemplar study two; discussion of event-contingent diaries)
- bolger-davis-rafaeli-2003-diary-methods: Diary Methods: Capturing Life as It Is Lived — https://doi.org/10.1146/annurev.psych.54.101601.145030 — Niall Bolger, Angelina Davis, Eshkol Rafaeli
- carter-mankoff-2005-diary-media: When Participants Do the Capturing: The Role of Media in Diary Studies — https://doi.org/10.1145/1054972.1055098 — Scott Carter, Jennifer Mankoff
- doherty-balaskas-doherty-2020-ema-design: The Design of Ecological Momentary Assessment Technologies — https://doi.org/10.1093/iwcomp/iwaa019 — Kevin Doherty, Andreas Balaskas, Gavin Doherty
- arndt-rose-2022-diary-data: Capturing life as it is truly lived? Improving diary data in educational research — https://doi.org/10.1080/1743727X.2022.2094360 — Henriette L. Arndt, Heath Rose

Editorial version: 1.0.0 · evidence checked 2026-09-26 · Agent editorial review; not human or expert certification.

No review details supplied.
Related (useful_with): https://vedokrok.com/knowledge/observe-the-workaround-where-the-work-happens

---

## Observe the workaround where the work happens

ID: MHC-D-RESEARCH-1096 · Version: 0.1.0 · Kind: protocol
Source: https://vedokrok.com/knowledge/observe-the-workaround-where-the-work-happens

The missing requirement may be on the sticky note beside the screen.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- A workflow sounds simple in interviews but remains difficult in practice.

### Avoid when

- Research presence can change work. Do not observe sensitive records, bystanders or hazardous tasks without appropriate safeguards.

### Explanation

With permission, observe a real task in its normal setting. Notice tools, interruptions, handoffs and unofficial workarounds. Separate what you see from your explanation, and ask about unclear decisions when interruption is safe.

### Steps

1. Agree the task, recording limits and information that must stay out of view.
2. Record the actual sequence and surrounding constraints without coaching it into your preferred flow.
3. Ask what the workaround accomplishes before proposing its removal.

### Example

A user copies an identifier into a private checklist because the application gives no reliable completion signal.

### Check

Can the design problem be explained through an observed constraint rather than an imagined user preference?

### Limits

- Research presence can change work. Do not observe sensitive records, bystanders or hazardous tasks without appropriate safeguards.

### Evidence and sources

- supports: Contextual research observes activities in the environment where people normally perform them. — RS-7BB2C88BA31E91CF. Observation does not prove causes or eliminate the effect of being observed. (Method purpose and observation procedure.)
- RS-7BB2C88BA31E91CF: Contextual research and observation — https://www.gov.uk/service-manual/user-research/contextual-research-and-observation

No review details supplied.
Related (useful_with): https://vedokrok.com/knowledge/give-the-user-a-goal-and-keep-your-instructions-out-of-the-result

---

## Include the people whose access needs your tests miss

ID: MHC-D-RESEARCH-1097 · Version: 0.1.0 · Kind: protocol
Source: https://vedokrok.com/knowledge/include-the-people-whose-access-needs-your-tests-miss

An accessibility score is not a person completing the task.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- An interface passes automated checks but some people still cannot use it.

### Avoid when

- Do not ask people to simulate a disability as a replacement for involving users. Participation must be accessible and voluntary.

### Explanation

Include participants with relevant disabilities and assistive technologies in realistic task evaluation. Ask about their usual setup and access needs before the session. Combine their findings with standards-based checks instead of making either stand for the other.

### Steps

1. Recruit across the relevant needs rather than treating one participant as representative of everyone.
2. Support the participant’s familiar tools and agreed accommodations.
3. Record barriers, successful workarounds and untested combinations explicitly.

### Example

A keyboard check passes, but a screen-reader user cannot tell whether a recording has finished processing.

### Check

Can the team connect each finding to a task, setup and observed barrier, while stating what was not tested?

### Limits

- Do not ask people to simulate a disability as a replacement for involving users. Participation must be accessible and voluntary.

### Evidence and sources

- supports: W3C recommends involving users with disabilities alongside standards-based accessibility evaluation. — RS-789B0E22EE7CFA3B. Neither a small user sample nor an automated scan certifies complete accessibility. (Combining user involvement with conformance evaluation.)
- RS-789B0E22EE7CFA3B: Involving Users in Evaluating Web Accessibility — https://www.w3.org/WAI/test-evaluate/involving-users/

No review details supplied.
Related (useful_with): https://vedokrok.com/knowledge/give-the-user-a-goal-and-keep-your-instructions-out-of-the-result
Related (useful_with): https://vedokrok.com/knowledge/announce-the-result-without-stealing-the-user-s-place

---

## Give the user a goal and keep your instructions out of the result

ID: MHC-D-RESEARCH-1098 · Version: 0.1.0 · Kind: protocol
Source: https://vedokrok.com/knowledge/give-the-user-a-goal-and-keep-your-instructions-out-of-the-result

A tour proves that you can explain the interface.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- You need to know whether people can use a service without coaching.

### Avoid when

- Small studies identify problems in chosen situations. They do not establish population-wide success rates or every accessibility requirement.

### Explanation

Set a realistic task and observe the participant’s attempt. Use neutral prompts and record where assistance occurs. A completed task after a rescue is useful evidence of a problem, not an unassisted success.

### Steps

1. Describe the desired outcome without naming the controls to use.
2. Observe actions, errors and recovery before asking for opinions.
3. Separate independent completion, assisted completion and non-completion in the findings.

### Example

Ask a participant to resume an unfinished lesson. Do not tell them that it is hidden under History.

### Check

Can someone reviewing the notes tell what the interface supported and what the moderator supplied?

### Limits

- Small studies identify problems in chosen situations. They do not establish population-wide success rates or every accessibility requirement.

### Evidence and sources

- supports: Moderated usability testing observes participants attempting relevant tasks with a service or prototype. — RS-ACE67B532B933567. Facilitator assistance changes what an observed completion establishes. (Preparing tasks and conducting the session.)
- RS-ACE67B532B933567: Using moderated usability testing — https://www.gov.uk/service-manual/user-research/using-moderated-usability-testing

No review details supplied.

---

## Deliver the small service before automating its machinery

ID: MHC-D-RESEARCH-1099 · Version: 0.1.0 · Kind: protocol
Source: https://vedokrok.com/knowledge/deliver-the-small-service-before-automating-its-machinery

An automation plan can hide an untested service.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- You are building infrastructure for a benefit nobody has received yet.

### Avoid when

- Manual success does not prove automation feasibility, scalable economics or broad demand. Do not promise support capacity you do not have.

### Explanation

Offer a clearly bounded manual pilot to a suitable user. Agree the output, limits and human involvement, then deliver the useful result. Record the effort and repeat demand before deciding which part deserves software.

### Steps

1. Choose a small outcome you can actually deliver without pretending the service is automated.
2. Agree scope, access, confidentiality and any payment terms.
3. Record delivery effort, revisions and whether the user would seek the service again.

### Example

Before building a full corpus subscription system, a pilot delivers a small, source-checked selection for one recurring work problem.

### Check

Did the user receive a useful outcome, and can you explain the cost of repeating the service?

### Limits

- Manual success does not prove automation feasibility, scalable economics or broad demand. Do not promise support capacity you do not have.

### Evidence and sources

- supports: Graham describes manually serving early users as a way to learn before scaling a service. — RS-39A2667CF2BEBFB8. Practitioner advice, not causal evidence that a concierge pilot predicts commercial success. (Manual work discussion.)
- RS-39A2667CF2BEBFB8: Do Things that Don’t Scale — https://paulgraham.com/ds.html

No review details supplied.
Related (useful_with): https://vedokrok.com/knowledge/record-the-episode-while-its-context-is-still-available
