Protocol

Prototype the conversation, not the capability

A human-operated prototype can answer whether an interaction is useful before the automation exists. The trap is treating smooth human performance as evidence that the eventual system will be equally fast, reliable or cheap.

About 5 min read

Before you use it

Use when
  • You need to test a proposed interaction before building its automation.

Keep this limit in mind

  • Disclosure may change behavior; concealment introduces ethical issues. Follow an approved research protocol, and never use the prototype for consequential live decisions.

Current editorial reading

Borrow human capability to test the part that is still uncertain

Wizard-of-Oz prototyping is useful when the interaction question arrives before the engineering is ready. A 2026 systematic review of 194 SIGCHI papers found the method used across many domains for reasons including technical limitations, adaptive flexibility, experimental control and cost. That supports a narrow claim: a person can stand in for missing system behaviour so a team can learn about the interaction. It does not show that the missing automation is feasible. A good session therefore starts by naming the question the prototype is meant to answer — for example, whether users understand a feedback format — and leaves accuracy, latency, scale and operating cost for later tests.

Transparency changes the research question

Classic Wizard-of-Oz studies usually hide the operator so participants experience the system as autonomous. The current card deliberately uses a transparent variant. That can be the safer and more honest choice for ordinary product discovery, but it changes what the result means. Once people know a human is involved, the session cannot tell you how they would react to apparent machine autonomy in the same way. It can still reveal whether the workflow makes sense, whether a response is useful, what users ask next and where the interaction breaks. In other words, disclosure does not make the prototype useless; it moves the claim from “this is how users respond to an autonomous system” toward “this is how they respond to this interaction with declared human assistance.”

The wizard is part of the measuring instrument

Human flexibility is the advantage and the main source of noise. Riek’s systematic review of 54 human–robot interaction experiments found that most studies constrained what the wizard produced, but very few reported measuring wizard error or pre-experiment wizard training. The broader 2026 review again identified wizard variability and bias, timing and performance constraints, ecological validity, deception and ethics, and transparency and reproducibility as recurring concerns. For practical prototyping, constrain everything that is not under study: define allowed responses, record what the operator supplied or repaired, log delays and deviations, and make operator freedom explicit. If the wizard quietly rescues every awkward case, the prototype can become a test of the wizard’s skill rather than the design.

Does telling participants about the operator ruin the test?

No, but it rules out some conclusions. A disclosed operator is compatible with testing flow, comprehension, desired response shape and whether an interaction is worth pursuing. It is weaker evidence for questions that depend on believing the system is autonomous, such as perceived agency or some forms of trust. If that distinction matters, write it into the study question before the session rather than explaining it away afterwards.

How tightly should the wizard be scripted?

Tightly enough that irrelevant operator differences do not become the result, but not so tightly that you remove the behaviour you are trying to explore. If wording is the variable, standardize timing and available actions. If you are discovering what users may ask, allow bounded flexibility but log every novel response. There is no universal script length; the useful rule is to control what should stay constant and expose what is intentionally allowed to vary.

When should the wizard leave the loop?

When the main uncertainty moves from interaction value to implementation performance. Once you need to know real recognition accuracy, model failure modes, latency, throughput, unit cost or behaviour at scale, human simulation becomes the wrong instrument. Houde and Hill’s prototyping framework is useful here: a prototype should be judged by the design question it was built to answer, not by how complete it looks. A convincing Wizard-of-Oz session can justify building the next technical experiment; it cannot stand in for that experiment.

Take it with you

Use human simulation to learn cheaply about an interaction before automation exists, and label the boundary precisely. A transparent wizard can test usefulness and flow, but smooth human performance is not evidence of production capability. Script and log the operator, then switch to real implementation tests when feasibility, reliability, latency, cost or scale become the question.

Sources and evidence (4)

What we checked

Supports

A 2026 systematic review coded 194 Wizard-of-Oz papers from SIGCHI venues and found the method used for multiple motivations, including overcoming technical limitations, adaptive flexibility, experimental control and cost-efficiency.

This supports using Wizard-of-Oz as a prototyping and research method. It does not establish that a simulated capability is technically feasible or that one Wizard-of-Oz setup generalizes to every product context.

Mapping the Wizards' Path: A Systematic Review of Wizard-of-Oz in HCI · Abstract; Sections 4.3 and 5.2

Limits

The 2026 review identifies recurring methodological concerns including temporal and performance constraints, wizard variability and bias, ecological validity, deception and ethics, and transparency and reproducibility.

These concerns are directly relevant to interpreting a human-operated prototype as evidence. The review synthesizes reported concerns across studies rather than estimating one pooled effect size.

Mapping the Wizards' Path: A Systematic Review of Wizard-of-Oz in HCI · Section 4.4

Challenges

A systematic review of 54 HRI Wizard-of-Oz experiments found that 90.7% constrained wizard production, while 3.7% reported measuring wizard error and 5.4% reported pre-experiment wizard training.

The sample is limited to HRI publications from 2001–2011, so the exact percentages should not be generalized to all modern HCI work. They illustrate how underreported operator control can weaken reproducibility.

Wizard of Oz studies in HRI: a systematic review and new reporting guidelines · Abstract

Provides context

Early Wizard-of-Oz methodology work treated human simulation as a way to study proposed interactive behaviour before the final system existed, while emphasizing practical control of the simulated interaction.

This is foundational method work focused on natural-language interfaces, not evidence that every transparent human-operated prototype produces valid behavioural estimates.

Wizard of Oz studies — why and how · Article abstract and methodological discussion

Provides context

Houde and Hill argue that prototypes should be designed and interpreted around the specific design question they represent, distinguishing role, look-and-feel and implementation concerns.

This prototyping framework supports the editorial distinction between testing interaction value and testing implementation capability; it is not a Wizard-of-Oz efficacy study.

What do Prototypes Prototype? · Publisher summary and prototype model

Editorial sources (4)

Editorial version 1.0.0 · Evidence checked 2026-09-24

Save and practise this idea

My learning

Read. Recall. Repeat.

Save this idea and return to it on any device.

Use email
Keep exploring

Connected ideas

You are readingPrototype the conversation, not the capability

Original edition evidence and sources

These records belong to the original released edition. The current editorial reading has its own evidence above; neither set silently replaces the other.

Supports

Wizard of Oz prototyping can use a human operator to simulate responses of a proposed system.

Simulated capability does not establish production performance, capacity or cost.

The Wizard of Oz Method in UX · Method definition and operator role.

Supports

The government research guidance calls for participants to understand the activity and how their data will be used before agreeing voluntarily.

This scoped guidance does not settle every legal basis or organisational approval requirement.

Getting informed consent for user research · Informed consent and explaining the research.

All sources (2)
Original edition, downloads and provenance

The reading above is editorial version 1.0.0, bound to this exact source edition. The original data and saved downloads retain that edition and the separate editorial layer. They may contain earlier wording; use the current reading above when interpreting it.

Version 0.1.0

No review details are supplied for this edition.

Read the original edition and overlay as JSON · Release details