Checklist
Segment AI gains by experience before standardizing the workflow
An average gain can hide opposite effects across the experience curve.
Before you use it
- Use when
- A team reports one average AI productivity gain and is considering the same workflow for novices and experts.
- Avoid when
- Do not label people permanently as low- or high-skill from one metric. The cited pattern comes from one customer-support setting and may reverse for other tasks, tools or forms of expertise.
An idea to explore.
Try this idea.
Inspect the example, try the idea in your situation, then check your next move.
Checklist
- Define experience or baseline-performance groups before reading the result.
- Measure speed and quality separately for each group.
- Inspect whether review burden also changes by group.
- Look for a quality loss among strong performers even when the overall average improves.
- Allow different AI defaults, permissions or review rules when the evidence supports them.
Why it matters
In a 5,172-agent workplace study, AI assistance produced substantial gains for newer and lower-skill workers but little productivity benefit for the most experienced workers, with a small quality decline among the highest-skill group. When roles differ in experience or baseline performance, report AI effects by meaningful skill band. A workflow that is excellent scaffolding for a novice may be friction or standardization pressure for an expert.
An example
A support copilot may help a new consultant draft a technically complete first response while making a senior consultant slower if they must remove generic wording and restore context-specific judgment.
Check your result
The rollout decision can show who improved, who did not, and whether any group traded quality for speed.
Keep this limit in mind
- Do not label people permanently as low- or high-skill from one metric. The cited pattern comes from one customer-support setting and may reverse for other tasks, tools or forms of expertise.
My learning
Read. Recall. Repeat.
Save this idea and return to it on any device.
Connected ideas
Useful withBenchmark human plus AI against the stronger solo mode
Evidence and sources (2)
In a staggered deployment to 5,172 customer-support agents, AI assistance increased issues resolved per hour by 15% on average, with much larger gains among less experienced and lower-skill workers.
The study concerns one firm, occupation and AI assistant; the magnitude and distribution of gains may differ in other work.
Generative AI at Work · Abstract and Introduction
The customer-support study reported that less experienced and lower-skill workers improved across productivity measures, while the most experienced and highest-skill workers saw little productivity gain and a small decline in conversation quality.
Worker skill was measured within the studied support environment; the pattern should not be generalized to every expert or every AI system.
Generative AI at Work · Introduction, heterogeneity results and conclusion
All sources (1)
Edition and review
Version 0.2.0 · Published
No review details are supplied for this edition.