Collection
Operate AI agents as systems you can replay, verify and constrain
Add practical 2026 agent-engineering techniques for replay, evaluation, tool routing, computer use, scoped authorization and deterministic execution without turning conference claims into universal guarantees.
20 elements · Explore in any order.
1 checklist · 1 heuristic · 6 principles · 11 protocols · 1 template
20 elements
Replay from the failure checkpoint, not from memory
A reproducible failure is more useful than a fresh attempt that happens to succeed.
Read the idea ChecklistCapture semantic boundaries, not only raw logs
A log can be complete and still omit the boundary where meaning changed.
Read the idea ProtocolTest an intervention across a replay cohort
One rescued trace is a story. A cohort begins to look like evidence.
Read the idea TemplateTreat done as an evidence object
Done is not a green check mark. It is a set of claims someone must be able to inspect.
Read the idea PrincipleGive verification an independent evidence path
A second opinion is weak when it only rereads the first opinion.
Read the idea ProtocolTry to break the evaluator before trusting it
If the agent can game the judge, a higher score can mean a worse system.
Read the idea PrincipleSimulate users who behave like users
A simulator that helps the agent too much is grading a different job.
Read the idea PrinciplePut uncertainty around consequential eval differences
A two-point score gap is not a verdict until you know how noisy the estimate is.
Read the idea ProtocolMine traces for failure categories before changing prompts
A low score tells you that something is wrong. A failure taxonomy tells you where to work.
Read the idea ProtocolRoute to the cheapest model that still passes your task eval
Do not pay for intelligence the task does not use—but do not guess where the boundary is.
Read the idea ProtocolRetrieve tool schemas just in time
A tool can exist without occupying attention on every turn.
Read the idea PrincipleMeasure the active tool set, not the catalog size
The catalog is inventory. The working set is cognitive load.
Read the idea ProtocolRetrieve code before sending code
The cheapest token is the irrelevant file you never send.
Read the idea HeuristicTurn repeated browser work into persistent programs
Use the browser to learn the path; use code to stop relearning it every time.
Read the idea ProtocolPin the browser environment when comparing agent behavior
If the browser moves under your feet, you do not know what the agent change caused.
Read the idea ProtocolExchange broad credentials for task-scoped access
Do not hand an agent the master key when it needs one door for one job.
Read the idea PrincipleSeparate user-visible UI state from model-visible context
What the user can see and what the model must receive are different design decisions.
Read the idea ProtocolGive computer-use tasks a start state and success check
A screenshot is not a test case until you know where the task starts and what state counts as success.
Read the idea PrincipleLet the model propose; let deterministic code commit
Use the model for judgment; use the control plane for authority.
Read the idea ProtocolTurn a production finding into a reusable eval artifact
A failure is expensive twice if the system can forget it.
Read the idea