Insecure Output Handling

Description

LLM outputs are passed to downstream systems or rendered without validation. This creates injection vulnerabilities when outputs contain malicious code or commands.

Risk

Treating LLM outputs as trusted data enables injection attacks in downstream systems. Outputs may contain SQL injection, XSS, command injection, or other malicious payloads that execute when processed by backend systems or rendered in user interfaces. This can lead to data breaches, system compromise, and privilege escalation.

Attack Scenarios

An LLM-powered search feature generates SQL queries from natural language. Attacker crafts input causing the LLM to output malicious SQL that drops tables when executed.

A content generation system renders LLM outputs directly in HTML without sanitization, allowing the model to inject JavaScript that steals session tokens from other users.

Mitigations

Output Validation

Validate and sanitize all LLM outputs before use. Treat outputs as untrusted user input and apply the same security controls.

Parameterized Interfaces

Use parameterized queries, prepared statements, and safe APIs when passing LLM outputs to backend systems. Never construct commands or queries through string concatenation.

Content Security Policy

Apply CSP headers and output encoding when rendering LLM content in web applications. Use context-aware escaping based on output destination.

Least Privilege Execution

Run downstream systems with minimal required permissions. Limit the impact of successful injection attacks through defense in depth.

Code Examples

# Bad: Direct execution of LLM output
query = llm.complete(f"Generate SQL for: {user_request}")
db.execute(query)  # SQL injection risk

# Good: Validate and use parameterized queries
import re
from typing import Optional

def validate_sql_output(llm_output: str) -> Optional[dict]:
    # Parse structured output, reject freeform SQL
    if not llm_output.startswith('{'):
        return None
    parsed = json.loads(llm_output)
    allowed_ops = ['SELECT', 'INSERT', 'UPDATE']
    if parsed.get('operation') not in allowed_ops:
        return None
    return parsed

llm_output = llm.complete(f"Generate query plan: {user_request}")
query_plan = validate_sql_output(llm_output)
if query_plan:
    # Use safe parameterized execution
    db.execute(
        "SELECT * FROM products WHERE category = ?",
        (query_plan['category'],)
    )

Evidence Requirements

Code reviews showing output validation before downstream processing
Use of parameterized queries and prepared statements
Output encoding and CSP implementation for web rendering
Security testing reports validating injection resistance
Documentation of output handling security requirements