Checklist
Track tool errors, tool calls, runtime and tokens as agent diagnostics
The path to the answer reveals inefficiency before users notice the bill or latency.
When it fits
- Agent evaluation records only final pass/fail.
When to avoid it
- Lower cost or fewer calls are not automatically better if they reduce robustness or coverage.
Checklist
- Count tool calls by tool.
- Count validation/tool errors.
- Track total runtime and major tool latency.
- Track token/context consumption.
- Flag repeated or redundant calls on successful tasks too.
Why it matters
Alongside task success, record number of tool calls, tool errors, runtime, token use and repeated calls. These metrics help detect confused tool selection, bloated responses, loops and regressions that final accuracy can hide.
An example
Two agents both pass, but one needs 7 tool calls and the other 48 with five schema errors; that difference deserves investigation.
Check your result
Efficiency and failure-path regressions are visible even when final success stays constant.
Keep this limit in mind
- Lower cost or fewer calls are not automatically better if they reduce robustness or coverage.
Evidence and sources
Supports
Anthropic recommends tracking runtime, tool-call count, token consumption and tool errors in addition to top-level accuracy.
Lower cost or fewer calls are not automatically better if they reduce robustness or coverage.
Writing effective tools for AI agents—using AI agents · See source record