Protocol
Grade the final state and the agent trajectory separately
The destination can be correct while the route is unacceptable.
When it fits
- An agent can reach the right answer through risky, wasteful or policy-breaking intermediate actions.
When to avoid it
- Overly prescriptive trajectory graders can punish creative valid solutions; inspect the path only where the path itself matters.
Why it matters
Use outcome graders for the resulting files, data or decision, and separate trajectory checks for properties that matter during execution: unauthorized tool calls, destructive retries, excessive loops, fabricated sources or other process constraints. Do not grade every stylistic path when only the end state matters; trajectory rules should protect real risks or costs.
Steps
- A run can pass the result while still failing a meaningful process constraint, and the reports distinguish those failures.
An example
A deployment agent must produce the correct configuration and must not bypass the approval gate on the way there.
Check your result
A run can pass the result while still failing a meaningful process constraint, and the reports distinguish those failures.
Keep this limit in mind
- Overly prescriptive trajectory graders can punish creative valid solutions; inspect the path only where the path itself matters.
Connected ideas
Useful withMonitor the whole agent trajectory, not only individual allowed actions
Evidence and sources
Agent evals may grade both end-state outcomes and transcript or trajectory properties because a correct-looking final artifact can hide undesirable process behavior.
Do not overconstrain legitimate alternative paths when only the end state matters.
Demystifying evals for AI agents · Graders can evaluate outcome or transcript