Agent Goal Hijack

Description

Attackers alter agent objectives through indirect prompt injection embedded in retrieved content, causing agents to pursue adversarial goals instead of their intended mission.

Mitigations

Instruction-Data Separation

Architect a clear boundary between system instructions (trusted) and retrieved content (untrusted). Never concatenate raw retrieved content directly into the system prompt. Use structured message formats where retrieved data is placed in explicitly labeled user-content blocks that the model is instructed to treat as data only.

Content Sandboxing

Process all retrieved content through a sanitization layer before it reaches the agent's context window:

• Strip invisible Unicode characters (zero-width spaces, direction overrides, homoglyphs)
• Remove HTML/CSS that could hide text (display:none, font-size:0, color matching background)
• Extract and discard metadata fields that could carry injection payloads
• Normalize whitespace and encoding to surface hidden content

Goal Validation Checkpoints

Implement periodic goal validation throughout the agent's execution:

• Before each tool call, verify the proposed action aligns with the original task objective
• Maintain an immutable copy of the original goal that cannot be overwritten by retrieved content
• Use a separate, isolated model call to evaluate whether the agent's current trajectory matches its mandate
• Log all goal-state transitions for post-hoc audit

Output Filtering

Apply output-side controls as a defense-in-depth measure:

• Block outbound requests to domains not on the approved allowlist
• Flag and quarantine responses that contain data patterns matching sensitive information (PII, credentials, financial data)
• Rate-limit tool invocations to prevent rapid exfiltration

Behavioral Canaries

Inject known-benign canary instructions into test documents to verify that agents ignore embedded directives. Monitor for canary activation as an early warning of goal hijack susceptibility.

Code Examples

Goal Integrity Checker

import hashlib
from dataclasses import dataclass
from typing import Optional


@dataclass
class AgentGoal:
    objective: str
    constraints: list[str]
    allowed_tools: list[str]
    allowed_domains: list[str]
    fingerprint: str


class GoalIntegrityChecker:
    """Validates that agent behavior remains aligned with original objectives."""

    def __init__(self, goal: AgentGoal):
        self._original_goal = goal
        self._fingerprint = self._compute_fingerprint(goal)
        self._action_log: list[dict] = []

    def _compute_fingerprint(self, goal: AgentGoal) -> str:
        payload = f"{goal.objective}|{'|'.join(goal.constraints)}"
        return hashlib.sha256(payload.encode()).hexdigest()

    def validate_action(self, proposed_action: dict) -> dict:
        """Check proposed action against original goal before execution."""
        result = {"allowed": True, "violations": []}

        # Verify goal integrity hasn't been tampered with
        if self._compute_fingerprint(self._original_goal) != self._fingerprint:
            result["allowed"] = False
            result["violations"].append("CRITICAL: Goal fingerprint mismatch — possible hijack")
            return result

        # Check tool is in allowlist
        tool = proposed_action.get("tool", "")
        if tool not in self._original_goal.allowed_tools:
            result["allowed"] = False
            result["violations"].append(f"Tool '{tool}' not in allowed set")

        # Check target domain is permitted
        target = proposed_action.get("target_domain", "")
        if target and target not in self._original_goal.allowed_domains:
            result["allowed"] = False
            result["violations"].append(f"Domain '{target}' not in allowed domains")

        # Log every action for audit trail
        self._action_log.append({
            "action": proposed_action,
            "result": result,
        })

        return result


def sanitize_retrieved_content(raw_content: str) -> str:
    """Strip content that could carry hidden injection payloads."""
    import re
    # Remove zero-width characters
    cleaned = re.sub(r'[\u200b\u200c\u200d\u2060\ufeff]', '', raw_content)
    # Remove HTML tags that hide content
    cleaned = re.sub(r'<[^>]*style=[^>]*display\s*:\s*none[^>]*>.*?</[^>]*>', '', cleaned, flags=re.DOTALL)
    cleaned = re.sub(r'<[^>]*style=[^>]*font-size\s*:\s*0[^>]*>.*?</[^>]*>', '', cleaned, flags=re.DOTALL)
    # Collapse excessive whitespace
    cleaned = re.sub(r'\s{10,}', ' ', cleaned)
    return cleaned.strip()

Evidence Requirements

• Goal fingerprint audit log showing all action validation decisions
• Content sanitization reports documenting stripped injection attempts
• Canary activation monitoring dashboard with alerting thresholds
• Incident response records for any detected goal deviation events
• Periodic red team assessment reports targeting agent goal integrity