After story 1

2026-08-03 11:30:49 -04:00
commit 82590c392f
32 changed files with 4375 additions and 0 deletions
@@ -0,0 +1,245 @@
+# STORY-08: Active Learning Pipeline
+
+## Epic
+**E3: Active Learning** — As an ML engineer, I can fine-tune the classifier head with reviewed data.
+
+## Related Requirements
+| ID | Requirement |
+|----|------|
+| FR-06 | Active learning pipeline (label ingestion → fine-tuning → deployment): Batch retraining only. No online learning. Versioned model swaps. |
+| NFR-04 | Determinism & Reproducibility: Config-seeded randomness, versioned models |
+| NFR-05 | Fault Tolerance: Auto-retry on transient failures; skip & log on fatal errors |
+| TC-01 | Hardware: 2× Tesla P40 24GB (compute capability 5.2, PCIe 3.0, no Tensor Cores) |
+| TC-02 | CUDA/Torch Compatibility: CUDA ≤ 11.8, PyTorch ≤ 2.1.0, FP32 inference only |
+
+## Description
+Implement the active learning pipeline that ingests annotated review data, fine-tunes the classifier head, validates the candidate model, and deploys it if it meets quality gates. This enables incremental model improvement using human-labeled data from the review queue.
+
+## Scope
+
+### In Scope
+- Label ingestion from review queue (CSV/JSON export or direct DB query)
+- Dataset versioning and preparation for training
+- Head-only fine-tuning on face crops (freeze backbone, train classification head)
+- Training with AdamW, configurable learning rate, early stopping
+- Candidate model validation on held-out validation set
+- F1 score and ECE (Expected Calibration Error) calculation
+- Quality gate validation (ΔF1 > 0.02, ECE < 0.08)
+- Model registry updates (ACTIVE, CANDIDATE, ARCHIVED states)
+- Hot reload of new TensorRT engine
+- Rollback on regression
+
+### Out of Scope
+- Frame sampling (covered in STORY-03)
+- Face detection (covered in STORY-04)
+- Classification inference (covered in STORY-05)
+- Review UI (covered in STORY-07)
+- Monitoring dashboards (covered in STORY-09)
+
+## Deliverables
+
+### 8.1 Label Ingestion
+**File:** `src/active_learning/label_ingestor.py`
+
+Features:
+- **Data Source:** Query `review_queue` table for annotated labels
+  ```sql
+  SELECT video_id, ground_truth, contributing_frames
+  FROM review_queue
+  WHERE annotated = true AND ground_truth IS NOT NULL
+  ```
+- **Crop Extraction:** Extract face crops from stored paths or re-extract from video
+- **Dataset Versioning:** Create versioned dataset directory structure
+  ```
+  /data/training/v2.0.0/
+      crops/
+          class_0/  (negative samples)
+          class_1/  (positive samples)
+      labels.csv
+      metadata.json
+  ```
+- **Train/Val Split:** 80/20 split (stratified by class)
+- **Augmentation:** Apply standard augmentations at training time (not pre-computed)
+  - Random horizontal flip
+  - Random color jitter
+  - Random affine transform (±10 degrees rotation, ±10% scale)
+
+### 8.2 Training Pipeline
+**File:** `src/active_learning/trainer.py`
+
+Features:
+- **Model Loading:** Load pre-trained MobileNetV3 backbone (frozen)
+  ```python
+  model = load_pretrained_mobilenetv3()
+  for param in model.features.parameters():
+      param.requires_grad = False
+  # Replace classification head
+  model.classifier = nn.Sequential(
+      nn.Linear(1280, 256),
+      nn.ReLU(),
+      nn.Dropout(0.3),
+      nn.Linear(256, 2)  # binary classification
+  )
+  ```
+- **Head-Only Fine-Tuning:** Only train the classification head
+  - Backbone weights are frozen (no gradient updates)
+  - Head weights are trainable
+  - This is faster and requires less data than full fine-tuning
+- **Optimizer:** AdamW with configurable parameters
+  - Learning rate: 1e-3 (default), configurable
+  - Weight decay: 1e-2
+  - Betas: (0.9, 0.999)
+- **Loss Function:** Binary Cross-Entropy with class weights (if imbalanced)
+- **Training Configuration:**
+  - Epochs: 10-30 (configurable)
+  - Batch size: 32
+  - Early stopping: patience=5 epochs (no validation improvement)
+  - Learning rate scheduler: ReduceLROnPlateau (factor=0.5, patience=3)
+- **Checkpointing:** Save checkpoint to `/models/candidate/` after each epoch
+  ```
+  /models/candidate/
+      v2.0.0_epoch_01.pt
+      v2.0.0_epoch_02.pt
+      ...
+      v2.0.0_best.pt  (best validation F1)
+  ```
+
+### 8.3 Validation Pipeline
+**File:** `src/active_learning/validator.py`
+
+Features:
+- **Validation Set:** Held-out 20% of annotated data (never seen during training)
+- **Metrics Calculation:**
+  - **F1 Score:** Macro F1 on validation set
+  - **ECE (Expected Calibration Error):**
+    ```python
+    def compute_ece(predictions, labels, n_bins=15):
+        bin_boundaries = np.linspace(0, 1, n_bins + 1)
+        ece = 0.0
+        for i in range(n_bins):
+            mask = (predictions >= bin_boundaries[i]) & (predictions < bin_boundaries[i+1])
+            if mask.sum() > 0:
+                bin_confidence = predictions[mask].mean()
+                bin_accuracy = labels[mask].mean()
+                ece += (mask.sum() / len(predictions)) * abs(bin_confidence - bin_accuracy)
+        return ece
+    ```
+  - **Accuracy, Precision, Recall:** Standard classification metrics
+- **Quality Gates:**
+  - ΔF1 > 0.02 (improvement over current model)
+  - ECE < 0.08 (calibration acceptable)
+  - Both gates must pass for deployment
+
+### 8.4 Model Registry & Deployment
+**File:** `src/active_learning/registry.py`
+
+Features:
+- **Model Registry:** Update DB `models` table
+  ```sql
+  -- Promote candidate to active
+  UPDATE models SET status = 'ACTIVE' WHERE version = 'v2.0.0';
+  UPDATE models SET status = 'ARCHIVED' WHERE status = 'ACTIVE' AND version != 'v2.0.0';
+  ```
+- **TensorRT Engine Build:** Convert candidate model to TensorRT engine
+  ```bash
+  trtexec --onnx=models/candidate/v2.0.0.onnx \
+          --saveEngine=models/candidate/v2.0.0.trt \
+          --fp32 --maxBatch=32 --workspace=2048
+  ```
+- **Hot Reload:** Signal worker process to reload new engine
+  - Option A: Restart worker container (`docker-compose restart worker`)
+  - Option B: In-process reload (graceful, no downtime)
+  - Default: In-process reload via signal handling
+- **Rollback:** If new model causes issues, rollback to archived model
+  ```sql
+  UPDATE models SET status = 'ACTIVE' WHERE version = 'v1.2.0';
+  UPDATE models SET status = 'ARCHIVED' WHERE version = 'v2.0.0';
+  ```
+
+### 8.5 Training Configuration
+**File:** `config.yaml` (updates)
+
+New fields:
+```yaml
+active_learning:
+  enabled: true
+  min_annotated_samples: 100  # minimum labeled data to trigger training
+  training:
+    epochs: 20
+    batch_size: 32
+    learning_rate: 1e-3
+    weight_decay: 1e-2
+    early_stopping_patience: 5
+    lr_scheduler: ReduceLROnPlateau
+    lr_factor: 0.5
+    lr_patience: 3
+  validation:
+    val_split: 0.2
+    min_f1_improvement: 0.02
+    max_ece: 0.08
+  deployment:
+    auto_deploy: true  # deploy if quality gates pass
+    hot_reload: true
+    rollback_enabled: true
+  augmentation:
+    horizontal_flip: true
+    color_jitter: true
+    affine: true
+    affine_degrees: 10
+    affine_scale: 0.1
+```
+
+## Acceptance Criteria
+
+### Functional
+- [ ] Annotated labels are ingested from review_queue table correctly
+- [ ] Dataset is versioned with proper directory structure
+- [ ] Train/val split is stratified by class (80/20)
+- [ ] Data augmentation is applied at training time (not pre-computed)
+- [ ] Backbone weights are frozen during fine-tuning (verified by checking requires_grad)
+- [ ] Classification head is trainable and receives gradient updates
+- [ ] AdamW optimizer is used with correct parameters
+- [ ] Early stopping works (training stops if no improvement for patience epochs)
+- [ ] Candidate model is saved to /models/candidate/ with correct versioning
+- [ ] F1 score is calculated correctly on validation set
+- [ ] ECE is calculated correctly on validation set
+- [ ] Quality gates are enforced (ΔF1 > 0.02 AND ECE < 0.08)
+- [ ] Model registry is updated (ACTIVE, CANDIDATE, ARCHIVED states)
+- [ ] TensorRT engine is built from candidate model
+- [ ] Hot reload deploys new model without downtime
+- [ ] Rollback restores previous model if deployment fails
+
+### Non-Functional
+- [ ] Training completes in < 4 hours on Tesla P40 (typical dataset: 1000 samples)
+- [ ] Validation completes in < 30 minutes
+- [ ] TensorRT engine build completes in < 10 minutes
+- [ ] Hot reload completes in < 30 seconds
+- [ ] Training uses < 18GB GPU memory
+- [ ] Training is deterministic (same data + same config → same results)
+
+### Technical Constraints
+- [ ] CUDA 11.8 compatible (verified via torch.version.cuda)
+- [ ] PyTorch ≤ 2.1.0 (verified via torch.__version__)
+- [ ] FP32 training only (no mixed precision)
+- [ ] No Tensor Cores used (CC 5.2 constraint)
+- [ ] Model versioning follows semantic versioning (MAJOR.MINOR.PATCH)
+- [ ] All training hyperparameters are configurable via config.yaml
+
+## Dependencies
+- **Prerequisites:** STORY-01 (Foundation), STORY-05 (Classification — provides base model), STORY-07 (Review UI — provides labeled data)
+- **Depends on:** None (runs independently, triggered by labeled data threshold)
+- **Enables:** STORY-05 (provides new model for inference)
+
+## Risks & Mitigations
+| Risk | Mitigation |
+|------|--|
+| Insufficient labeled data for meaningful fine-tuning | Set min_annotated_samples threshold (e.g., 100); wait until reached |
+| Head-only fine-tuning may not be enough for domain shift | Offer full fine-tuning as option; document limitations |
+| Model regression in production | Strict quality gates; keep previous model in ARCHIVED state for quick rollback |
+| Long training times on Tesla P40 | Head-only training is faster; batch size tuning; early stopping |
+| ECE calculation sensitivity | Use proper binning; report ECI with confidence intervals |
+
+## Estimated Effort
+- **Sprint:** 7-8
+- **Story Points:** 34
+- **Dependencies:** STORY-01, STORY-05, STORY-07