UBC — AI Solutions

Audit — Dynamic Documentation Platform

Living documentation that stays in sync with engineering, cloud, AI, and compliance activity. Documents update themselves as the organization changes.

Back to library
SOP
Fresh

SOP-AIDLC-05 — Evaluation (Accuracy, Hallucination, Safety, Red-Team)

Runs a standard evaluation battery on every AI release, including red-team attacks and safety probes.

Owner: AI Evaluation LeadApprover: AI Platform DirectorVersion: 2.5Updated: 12/20/1969

1. Purpose

Provide objective evidence that AI systems meet accuracy, safety, and robustness targets.

2. Scope

Every AI release, including prompt-only changes.

3. Definitions

  • Hallucination rate — Rate at which the model produces fabricated content.
  • Red-team — Structured adversarial testing to find failure modes.

4. Roles & Responsibilities (RACI)

ActivityRACICadence
Define eval batteryAI Evaluation LeadAI Platform DirectorClinical SMETeamPer model
Execute evaluationsAI Evaluation TeamAI Evaluation LeadEngineersCouncilEvery release
Red-teamRed-Team LeadAI Platform DirectorSecurityCouncilPer major release

5. Procedure

  1. Run the golden-set accuracy evaluation on every release.
  2. Measure hallucination rate using a curated adversarial set.
  3. Run safety probes (self-harm, jailbreak, PHI leakage, off-topic).
  4. For Tier 1/2 systems, conduct red-team exercises before major releases.
  5. Publish an Evaluation Report with pass/fail against thresholds.

6. Inputs & Outputs

Inputs

  • Golden sets
  • Adversarial sets
  • Prior evaluation reports

Outputs

  • Evaluation Report
  • Red-team Report

7. Controls & Metrics

MetricTarget
Releases with signed Evaluation Reports100%
Tier 1/2 releases with red-team100%

8. Exceptions & Escalation

  • Prompt-only hotfixes may skip red-team but require abbreviated eval and post-hoc review.

9. Records & Retention

RecordRetention
Evaluation Reports10 years
Red-team Reports10 years

10. Related Documents

  • SOP-AIDLC-04
  • SOP-AIDLC-06

11. References

  • NIST AI RMF Measure
  • OWASP LLM Top 10

12. Revision History

See the Versions tab for the full change history maintained by the Auto-Doc Engine.

Document code: SOP-AIDLC-05 · Aligned to NIST AI RMF Measure / ISO/IEC 42001 §9. Controlled document — reproduction outside the UBC QMS requires the Quality Manager's approval.

Suggested by AI
No suggestions right now.
Frameworks
AI Governance
ISO 9001
CMMI-DEV