UBC — AI Solutions

Audit — Dynamic Documentation Platform

Living documentation that stays in sync with engineering, cloud, AI, and compliance activity. Documents update themselves as the organization changes.

Back to library
SOP
Fresh

SOP-AIDLC-03 — Dataset Curation, Labeling & Bias Assessment

Curates and labels datasets with quality controls and evaluates for bias.

Owner: ML Engineering LeadApprover: AI Platform DirectorVersion: 1.9Updated: 11/22/1969

1. Purpose

Deliver high-quality, representative datasets with documented bias analysis.

2. Scope

Training, fine-tuning, and evaluation datasets.

3. Definitions

  • Golden set — A curated evaluation dataset with expert-verified labels.
  • Inter-annotator agreement — A measure of labeling consistency across annotators.

4. Roles & Responsibilities (RACI)

ActivityRACICadence
Define labeling schemaML EngineerML LeadClinical SMETeamPer dataset
Label & QCAnnotatorAnnotation LeadML EngineerTeamPer batch
Assess biasML EngineerML LeadGRCCouncilPer dataset

5. Procedure

  1. Design a labeling schema with definitions and examples.
  2. Train annotators; require ≥ 0.8 inter-annotator agreement on a calibration set.
  3. Label in batches with 10% double-labeled for QC.
  4. Analyse the dataset for demographic and clinical representativeness.
  5. Publish a Datasheet-for-Datasets record.

6. Inputs & Outputs

Inputs

  • Approved data source
  • Labeling schema

Outputs

  • Labeled dataset
  • Datasheet
  • Bias analysis report

7. Controls & Metrics

MetricTarget
Datasets with datasheets100%
Inter-annotator agreement≥ 0.8

8. Exceptions & Escalation

  • Emergency labeling passes require post-hoc QC within 5 business days.

9. Records & Retention

RecordRetention
DatasheetsLife of dataset + 5 years
Bias analysesLife of dataset + 5 years

10. Related Documents

  • SOP-AIDLC-02
  • SOP-AIDLC-05

11. References

  • Datasheets for Datasets (Gebru et al.)
  • NIST AI RMF Measure

12. Revision History

See the Versions tab for the full change history maintained by the Auto-Doc Engine.

Document code: SOP-AIDLC-03 · Aligned to NIST AI RMF Measure / ISO/IEC 42001 §8.4. Controlled document — reproduction outside the UBC QMS requires the Quality Manager's approval.

Suggested by AI
No suggestions right now.
Frameworks
AI Governance
ISO 9001