Protocol
Validate model output for the system that will consume it
Readable text becomes a different risk when the next component treats it as code.
When it fits
- LLM output will be executed, rendered or interpreted by another component.
When to avoid it
- Validation rules must match the actual sink and can themselves contain vulnerabilities.
Why it matters
Treat model output as untrusted input at the downstream boundary. Validate allowed fields and ranges, use parameterized interfaces where relevant, encode for the destination context, and reject output that requests effects outside the contract. A generic 'valid JSON' check is not enough when the payload later becomes SQL, HTML, a path or a shell argument.
Steps
- Output becomes SQL or another query language.: Use parameterized operations and allowlisted semantics rather than string execution.
- Output is rendered in a browser or message.: Apply the required context-aware encoding and content controls.
- Output controls a tool call.: Validate the function, arguments, resource scope and authorization before execution.
An example
Parse an agent's file-edit request into an allowlisted path and operation instead of interpolating its text into a shell command.
Check your result
Malformed or out-of-contract model output is rejected before the downstream component acts on it.
Keep this limit in mind
- Validation rules must match the actual sink and can themselves contain vulnerabilities.
Connected ideas
Use beforeLog what the agent did, not only what it said
Evidence and sources
OWASP recommends treating model output as untrusted input to downstream components and validating or encoding it for the destination context.
Validation must match the sink; a JSON shape check does not sanitize a shell command, SQL fragment or HTML payload.
LLM05:2025 Improper Output Handling · Prevention and Mitigation Strategies