VideoDetect/STORY-08.md

STORY-08: Active Learning Pipeline

Epic

E3: Active Learning — As an ML engineer, I can fine-tune the classifier head with reviewed data.

Related Requirements

ID	Requirement
FR-06	Active learning pipeline (label ingestion → fine-tuning → deployment): Batch retraining only. No online learning. Versioned model swaps.
NFR-04	Determinism & Reproducibility: Config-seeded randomness, versioned models
NFR-05	Fault Tolerance: Auto-retry on transient failures; skip & log on fatal errors
TC-01	Hardware: 2× Tesla P40 24GB (compute capability 5.2, PCIe 3.0, no Tensor Cores)
TC-02	CUDA/Torch Compatibility: CUDA ≤ 11.8, PyTorch ≤ 2.1.0, FP32 inference only

Description

Implement the active learning pipeline that ingests annotated review data, fine-tunes the classifier head, validates the candidate model, and deploys it if it meets quality gates. This enables incremental model improvement using human-labeled data from the review queue.

Scope

In Scope

• Label ingestion from review queue (CSV/JSON export or direct DB query)
• Dataset versioning and preparation for training
• Head-only fine-tuning on face crops (freeze backbone, train classification head)
• Training with AdamW, configurable learning rate, early stopping
• Candidate model validation on held-out validation set
• F1 score and ECE (Expected Calibration Error) calculation
• Quality gate validation (ΔF1 > 0.02, ECE < 0.08)
• Model registry updates (ACTIVE, CANDIDATE, ARCHIVED states)
• Hot reload of new TensorRT engine
• Rollback on regression

Out of Scope

• Frame sampling (covered in STORY-03)
• Face detection (covered in STORY-04)
• Classification inference (covered in STORY-05)
• Review UI (covered in STORY-07)
• Monitoring dashboards (covered in STORY-09)

Deliverables

8.1 Label Ingestion

File: src/active_learning/label_ingestor.py

Features:

• Data Source: Query review_queue table for annotated labels

  SELECT video_id, ground_truth, contributing_frames
  FROM review_queue
  WHERE annotated = true AND ground_truth IS NOT NULL
  

• Crop Extraction: Extract face crops from stored paths or re-extract from video
• Dataset Versioning: Create versioned dataset directory structure

  /data/training/v2.0.0/
      crops/
          class_0/  (negative samples)
          class_1/  (positive samples)
      labels.csv
      metadata.json
  

• Train/Val Split: 80/20 split (stratified by class)
• Augmentation: Apply standard augmentations at training time (not pre-computed)
  • Random horizontal flip
  • Random color jitter
  • Random affine transform (±10 degrees rotation, ±10% scale)

8.2 Training Pipeline

File: src/active_learning/trainer.py

Features:

• Model Loading: Load pre-trained MobileNetV3 backbone (frozen)

  model = load_pretrained_mobilenetv3()
  for param in model.features.parameters():
      param.requires_grad = False
  # Replace classification head
  model.classifier = nn.Sequential(
      nn.Linear(1280, 256),
      nn.ReLU(),
      nn.Dropout(0.3),
      nn.Linear(256, 2)  # binary classification
  )
  

• Head-Only Fine-Tuning: Only train the classification head
  • Backbone weights are frozen (no gradient updates)
  • Head weights are trainable
  • This is faster and requires less data than full fine-tuning
• Optimizer: AdamW with configurable parameters
  • Learning rate: 1e-3 (default), configurable
  • Weight decay: 1e-2
  • Betas: (0.9, 0.999)
• Loss Function: Binary Cross-Entropy with class weights (if imbalanced)
• Training Configuration:
  • Epochs: 10-30 (configurable)
  • Batch size: 32
  • Early stopping: patience=5 epochs (no validation improvement)
  • Learning rate scheduler: ReduceLROnPlateau (factor=0.5, patience=3)
• Checkpointing: Save checkpoint to /models/candidate/ after each epoch

  /models/candidate/
      v2.0.0_epoch_01.pt
      v2.0.0_epoch_02.pt
      ...
      v2.0.0_best.pt  (best validation F1)
  

8.3 Validation Pipeline

File: src/active_learning/validator.py

Features:

• Validation Set: Held-out 20% of annotated data (never seen during training)
• Metrics Calculation:
  • F1 Score: Macro F1 on validation set
  • ECE (Expected Calibration Error):

    def compute_ece(predictions, labels, n_bins=15):
        bin_boundaries = np.linspace(0, 1, n_bins + 1)
        ece = 0.0
        for i in range(n_bins):
            mask = (predictions >= bin_boundaries[i]) & (predictions < bin_boundaries[i+1])
            if mask.sum() > 0:
                bin_confidence = predictions[mask].mean()
                bin_accuracy = labels[mask].mean()
                ece += (mask.sum() / len(predictions)) * abs(bin_confidence - bin_accuracy)
        return ece
    

  • Accuracy, Precision, Recall: Standard classification metrics
• Quality Gates:
  • ΔF1 > 0.02 (improvement over current model)
  • ECE < 0.08 (calibration acceptable)
  • Both gates must pass for deployment

8.4 Model Registry & Deployment

File: src/active_learning/registry.py

Features:

• Model Registry: Update DB models table

  -- Promote candidate to active
  UPDATE models SET status = 'ACTIVE' WHERE version = 'v2.0.0';
  UPDATE models SET status = 'ARCHIVED' WHERE status = 'ACTIVE' AND version != 'v2.0.0';
  

• TensorRT Engine Build: Convert candidate model to TensorRT engine

  trtexec --onnx=models/candidate/v2.0.0.onnx \
          --saveEngine=models/candidate/v2.0.0.trt \
          --fp32 --maxBatch=32 --workspace=2048
  

• Hot Reload: Signal worker process to reload new engine
  • Option A: Restart worker container (docker-compose restart worker)
  • Option B: In-process reload (graceful, no downtime)
  • Default: In-process reload via signal handling
• Rollback: If new model causes issues, rollback to archived model

  UPDATE models SET status = 'ACTIVE' WHERE version = 'v1.2.0';
  UPDATE models SET status = 'ARCHIVED' WHERE version = 'v2.0.0';
  

8.5 Training Configuration

File: config.yaml (updates)

New fields:

active_learning:
  enabled: true
  min_annotated_samples: 100  # minimum labeled data to trigger training
  training:
    epochs: 20
    batch_size: 32
    learning_rate: 1e-3
    weight_decay: 1e-2
    early_stopping_patience: 5
    lr_scheduler: ReduceLROnPlateau
    lr_factor: 0.5
    lr_patience: 3
  validation:
    val_split: 0.2
    min_f1_improvement: 0.02
    max_ece: 0.08
  deployment:
    auto_deploy: true  # deploy if quality gates pass
    hot_reload: true
    rollback_enabled: true
  augmentation:
    horizontal_flip: true
    color_jitter: true
    affine: true
    affine_degrees: 10
    affine_scale: 0.1

Acceptance Criteria

Functional

• Annotated labels are ingested from review_queue table correctly
• Dataset is versioned with proper directory structure
• Train/val split is stratified by class (80/20)
• Data augmentation is applied at training time (not pre-computed)
• Backbone weights are frozen during fine-tuning (verified by checking requires_grad)
• Classification head is trainable and receives gradient updates
• AdamW optimizer is used with correct parameters
• Early stopping works (training stops if no improvement for patience epochs)
• Candidate model is saved to /models/candidate/ with correct versioning
• F1 score is calculated correctly on validation set
• ECE is calculated correctly on validation set
• Quality gates are enforced (ΔF1 > 0.02 AND ECE < 0.08)
• Model registry is updated (ACTIVE, CANDIDATE, ARCHIVED states)
• TensorRT engine is built from candidate model
• Hot reload deploys new model without downtime
• Rollback restores previous model if deployment fails

Non-Functional

• Training completes in < 4 hours on Tesla P40 (typical dataset: 1000 samples)
• Validation completes in < 30 minutes
• TensorRT engine build completes in < 10 minutes
• Hot reload completes in < 30 seconds
• Training uses < 18GB GPU memory
• Training is deterministic (same data + same config → same results)

Technical Constraints

• CUDA 11.8 compatible (verified via torch.version.cuda)
• PyTorch ≤ 2.1.0 (verified via torch.version)
• FP32 training only (no mixed precision)
• No Tensor Cores used (CC 5.2 constraint)
• Model versioning follows semantic versioning (MAJOR.MINOR.PATCH)
• All training hyperparameters are configurable via config.yaml

Dependencies

• Prerequisites: STORY-01 (Foundation), STORY-05 (Classification — provides base model), STORY-07 (Review UI — provides labeled data)
• Depends on: None (runs independently, triggered by labeled data threshold)
• Enables: STORY-05 (provides new model for inference)

Risks & Mitigations

Risk	Mitigation
Insufficient labeled data for meaningful fine-tuning	Set min_annotated_samples threshold (e.g., 100); wait until reached
Head-only fine-tuning may not be enough for domain shift	Offer full fine-tuning as option; document limitations
Model regression in production	Strict quality gates; keep previous model in ARCHIVED state for quick rollback
Long training times on Tesla P40	Head-only training is faster; batch size tuning; early stopping
ECE calculation sensitivity	Use proper binning; report ECI with confidence intervals

Estimated Effort

• Sprint: 7-8
• Story Points: 34
• Dependencies: STORY-01, STORY-05, STORY-07