Protocol

Grade the final state and the agent trajectory separately

The destination can be correct while the route is unacceptable.

About 1 min read

Before you use it

Use when
  • An agent can reach the right answer through risky, wasteful or policy-breaking intermediate actions.

Keep this limit in mind

  • Overly prescriptive trajectory graders can punish creative valid solutions; inspect the path only where the path itself matters.

An idea to explore.

Try this idea.

Inspect the example, try the idea in your situation, then check your next move.

Keep reading

Why it matters

Use outcome graders for the resulting files, data or decision, and separate trajectory checks for properties that matter during execution: unauthorized tool calls, destructive retries, excessive loops, fabricated sources or other process constraints. Do not grade every stylistic path when only the end state matters; trajectory rules should protect real risks or costs.

Steps

  1. A run can pass the result while still failing a meaningful process constraint, and the reports distinguish those failures.

An example

A deployment agent must produce the correct configuration and must not bypass the approval gate on the way there.

Check your result

A run can pass the result while still failing a meaningful process constraint, and the reports distinguish those failures.

Save and practise this idea

My learning

Read. Recall. Repeat.

Save this idea and return to it on any device.

Use email
Keep exploring

Connected ideas

You are readingGrade the final state and the agent trajectory separately

Evidence and sources
Supports

Agent evals may grade both end-state outcomes and transcript or trajectory properties because a correct-looking final artifact can hide undesirable process behavior.

Do not overconstrain legitimate alternative paths when only the end state matters.

Demystifying evals for AI agents · Graders can evaluate outcome or transcript

All sources (1)
Edition and review

Version 0.1.0

No review details are supplied for this edition.

Read this idea as JSON · Release details