Description
Continuous monitoring of deployed AI systems for performance degradation, drift, security threats, and unexpected behaviors.
Implementation Guidance
Monitoring Dimensions
Monitor deployed AI systems across: model performance (accuracy, precision, recall over time), data drift (input distribution changes from training baseline), concept drift (relationship changes between features and outcomes), system health (latency, throughput, error rates), and security (anomalous queries, potential adversarial inputs).
Alerting and Thresholds
Define quantitative alerting thresholds for each monitoring dimension. Configure automated alerts routed to appropriate teams: model drift alerts to data science, security anomalies to security operations, performance degradation to system owners. Include escalation for sustained threshold breaches.
Monitoring Infrastructure
Deploy observability infrastructure: prediction logging with input/output capture, statistical distribution monitoring, fairness metric dashboards refreshed on production data, and integration with organizational SIEM for security-relevant events.
Evidence Requirements
- Monitoring configuration documentation per AI system
- Alerting threshold definitions and justification
- Monitoring dashboard screenshots or access records
- Alert history and response records