Monitor the agent after deployment

When it fits

• An AI workflow is useful enough to run repeatedly in real conditions.

When to avoid it

• NIST notes that AI monitoring practice is still developing; a metric set should not be presented as a complete safety methodology.

Why it matters

Define a small set of post-deployment signals tied to real failure modes: invalid actions, corrections, refusals, escalation, unexpected tool use, latency or other task-specific outcomes. Review incidents and field behavior for conditions not represented in predeployment tests. Feed verified failures back into controls and evaluation rather than treating monitoring as a dashboard decoration.

Steps

1. Choose signals linked to concrete failure or harm modes.
2. Capture enough context to investigate without collecting unnecessary sensitive data.
3. Review unexpected behavior and user corrections on a defined cadence or trigger.
4. Turn confirmed new failure modes into a control, test case or product decision.

Check your result

A real-world failure has a route from detection to investigation and, when warranted, to a changed control or evaluation.

Keep this limit in mind

• NIST notes that AI monitoring practice is still developing; a metric set should not be presented as a complete safety methodology.

Connected ideas