UBC — AI Solutions

Audit — Dynamic Documentation Platform

Living documentation that stays in sync with engineering, cloud, AI, and compliance activity. Documents update themselves as the organization changes.

Back to library
SOP
Fresh

SOP-AIDLC-08 — Monitoring, Drift & Feedback Loop

Monitors AI systems for quality, drift, cost, and abuse; feeds signals back into evaluation.

Owner: MLOps LeadApprover: AI Platform DirectorVersion: 1.8Updated: 12/18/1969

1. Purpose

Detect drift and degradation early; keep evaluations aligned to reality.

2. Scope

All production AI systems.

3. Definitions

  • Drift — A statistically significant change in data or model behaviour vs. baseline.
  • Feedback loop — The pipeline that turns production signals into new evaluation cases.

4. Roles & Responsibilities (RACI)

ActivityRACICadence
Instrument monitorsMLOpsMLOps LeadAI EngineerTeamPer system
Investigate driftAI EngineerMLOps LeadData EngineerCouncilAs triggered
Update golden setsAI Evaluation TeamAI Evaluation LeadClinical SMETeamMonthly

5. Procedure

  1. Emit per-request telemetry (latency, cost, guardrail triggers, refusals, confidence).
  2. Compute drift metrics against a rolling baseline; alert on breach.
  3. Investigate alerts within SLA and open a CAPA if a systematic issue is found.
  4. Turn interesting production cases into new golden-set entries.

6. Inputs & Outputs

Inputs

  • Telemetry
  • User feedback
  • Incident tickets

Outputs

  • Drift dashboards
  • Updated golden sets
  • CAPA records

7. Controls & Metrics

MetricTarget
Drift alerts investigated within SLA≥ 95%
Feedback cases added to eval / month≥ 20

8. Exceptions & Escalation

  • Sudden severe drift (> 5σ) auto-pauses the affected system and pages on-call.

9. Records & Retention

RecordRetention
Drift reports5 years
Feedback cases5 years

10. Related Documents

  • SOP-AIDLC-05
  • SOP-AIDLC-09

11. References

  • NIST AI RMF Manage

12. Revision History

See the Versions tab for the full change history maintained by the Auto-Doc Engine.

Document code: SOP-AIDLC-08 · Aligned to NIST AI RMF Measure/Manage. Controlled document — reproduction outside the UBC QMS requires the Quality Manager's approval.

Suggested by AI
No suggestions right now.
Frameworks
AI Governance
SOC 2
ISO 9001