Checklist
Capture semantic boundaries, not only raw logs
A log can be complete and still omit the boundary where meaning changed.
When it fits
- A production agent failure disappears when developers try to reproduce it locally.
When to avoid it
- Do not turn observability into indiscriminate data retention. Minimize personal data, secrets and irrelevant payloads.
Checklist
- Capture the model decision that selected the action.
- Capture the exact parsed tool arguments and returned result.
- Record policy or validation decisions around the call.
- Keep enough state to replay the boundary without storing unrelated sensitive data.
Why it matters
Record the inputs and outputs at the model and tool boundaries that matter to the task: model request and response identifiers, parsed arguments, tool results, policy decisions and the state carried into the next step. This creates an execution envelope that can replay the bad decision against repaired code.
An example
A trading assistant interpreted dollars as shares. The useful trace preserves the model's action, parsed amount, tool contract and validation result at the order boundary.
Check your result
A developer can replay the consequential boundary without inventing missing inputs or re-prompting the model for a similar decision.
Keep this limit in mind
- Do not turn observability into indiscriminate data retention. Minimize personal data, secrets and irrelevant payloads.
Connected ideas
Useful withTrack tool errors, tool calls, runtime and tokens as agent diagnostics
Evidence and sources
Recording semantic model and tool boundaries can preserve the decision context needed to replay a production failure against repaired enforcement code.
Capturing more data is not automatically useful or safe; retain only the boundaries needed for diagnosis while respecting privacy and secret-handling requirements.
Your Agent Failed in Prod. Good Luck Reproducing It. · 4:51-13:14, recover the run, record semantic boundaries, replay the bad decision and retain the execution envelope