# STORY-08: Active Learning Pipeline

## Epic
**E3: Active Learning** — As an ML engineer, I can fine-tune the classifier head with reviewed data.

## Related Requirements
| ID | Requirement |
|----|------|
| FR-06 | Active learning pipeline (label ingestion → fine-tuning → deployment): Batch retraining only. No online learning. Versioned model swaps. |
| NFR-04 | Determinism & Reproducibility: Config-seeded randomness, versioned models |
| NFR-05 | Fault Tolerance: Auto-retry on transient failures; skip & log on fatal errors |
| TC-01 | Hardware: 2× Tesla P40 24GB (compute capability 5.2, PCIe 3.0, no Tensor Cores) |
| TC-02 | CUDA/Torch Compatibility: CUDA ≤ 11.8, PyTorch ≤ 2.1.0, FP32 inference only |

## Description
Implement the active learning pipeline that ingests annotated review data, fine-tunes the classifier head, validates the candidate model, and deploys it if it meets quality gates. This enables incremental model improvement using human-labeled data from the review queue.

## Scope

### In Scope
- Label ingestion from review queue (CSV/JSON export or direct DB query)
- Dataset versioning and preparation for training
- Head-only fine-tuning on face crops (freeze backbone, train classification head)
- Training with AdamW, configurable learning rate, early stopping
- Candidate model validation on held-out validation set
- F1 score and ECE (Expected Calibration Error) calculation
- Quality gate validation (ΔF1 > 0.02, ECE < 0.08)
- Model registry updates (ACTIVE, CANDIDATE, ARCHIVED states)
- Hot reload of new TensorRT engine
- Rollback on regression

### Out of Scope
- Frame sampling (covered in STORY-03)
- Face detection (covered in STORY-04)
- Classification inference (covered in STORY-05)
- Review UI (covered in STORY-07)
- Monitoring dashboards (covered in STORY-09)

## Deliverables

### 8.1 Label Ingestion
**File:** `src/active_learning/label_ingestor.py`

Features:
- **Data Source:** Query `review_queue` table for annotated labels
  ```sql
  SELECT video_id, ground_truth, contributing_frames
  FROM review_queue
  WHERE annotated = true AND ground_truth IS NOT NULL
  ```
- **Crop Extraction:** Extract face crops from stored paths or re-extract from video
- **Dataset Versioning:** Create versioned dataset directory structure
  ```
  /data/training/v2.0.0/
      crops/
          class_0/  (negative samples)
          class_1/  (positive samples)
      labels.csv
      metadata.json
  ```
- **Train/Val Split:** 80/20 split (stratified by class)
- **Augmentation:** Apply standard augmentations at training time (not pre-computed)
  - Random horizontal flip
  - Random color jitter
  - Random affine transform (±10 degrees rotation, ±10% scale)

### 8.2 Training Pipeline
**File:** `src/active_learning/trainer.py`

Features:
- **Model Loading:** Load pre-trained MobileNetV3 backbone (frozen)
  ```python
  model = load_pretrained_mobilenetv3()
  for param in model.features.parameters():
      param.requires_grad = False
  # Replace classification head
  model.classifier = nn.Sequential(
      nn.Linear(1280, 256),
      nn.ReLU(),
      nn.Dropout(0.3),
      nn.Linear(256, 2)  # binary classification
  )
  ```
- **Head-Only Fine-Tuning:** Only train the classification head
  - Backbone weights are frozen (no gradient updates)
  - Head weights are trainable
  - This is faster and requires less data than full fine-tuning
- **Optimizer:** AdamW with configurable parameters
  - Learning rate: 1e-3 (default), configurable
  - Weight decay: 1e-2
  - Betas: (0.9, 0.999)
- **Loss Function:** Binary Cross-Entropy with class weights (if imbalanced)
- **Training Configuration:**
  - Epochs: 10-30 (configurable)
  - Batch size: 32
  - Early stopping: patience=5 epochs (no validation improvement)
  - Learning rate scheduler: ReduceLROnPlateau (factor=0.5, patience=3)
- **Checkpointing:** Save checkpoint to `/models/candidate/` after each epoch
  ```
  /models/candidate/
      v2.0.0_epoch_01.pt
      v2.0.0_epoch_02.pt
      ...
      v2.0.0_best.pt  (best validation F1)
  ```

### 8.3 Validation Pipeline
**File:** `src/active_learning/validator.py`

Features:
- **Validation Set:** Held-out 20% of annotated data (never seen during training)
- **Metrics Calculation:**
  - **F1 Score:** Macro F1 on validation set
  - **ECE (Expected Calibration Error):**
    ```python
    def compute_ece(predictions, labels, n_bins=15):
        bin_boundaries = np.linspace(0, 1, n_bins + 1)
        ece = 0.0
        for i in range(n_bins):
            mask = (predictions >= bin_boundaries[i]) & (predictions < bin_boundaries[i+1])
            if mask.sum() > 0:
                bin_confidence = predictions[mask].mean()
                bin_accuracy = labels[mask].mean()
                ece += (mask.sum() / len(predictions)) * abs(bin_confidence - bin_accuracy)
        return ece
    ```
  - **Accuracy, Precision, Recall:** Standard classification metrics
- **Quality Gates:**
  - ΔF1 > 0.02 (improvement over current model)
  - ECE < 0.08 (calibration acceptable)
  - Both gates must pass for deployment

### 8.4 Model Registry & Deployment
**File:** `src/active_learning/registry.py`

Features:
- **Model Registry:** Update DB `models` table
  ```sql
  -- Promote candidate to active
  UPDATE models SET status = 'ACTIVE' WHERE version = 'v2.0.0';
  UPDATE models SET status = 'ARCHIVED' WHERE status = 'ACTIVE' AND version != 'v2.0.0';
  ```
- **TensorRT Engine Build:** Convert candidate model to TensorRT engine
  ```bash
  trtexec --onnx=models/candidate/v2.0.0.onnx \
          --saveEngine=models/candidate/v2.0.0.trt \
          --fp32 --maxBatch=32 --workspace=2048
  ```
- **Hot Reload:** Signal worker process to reload new engine
  - Option A: Restart worker container (`docker-compose restart worker`)
  - Option B: In-process reload (graceful, no downtime)
  - Default: In-process reload via signal handling
- **Rollback:** If new model causes issues, rollback to archived model
  ```sql
  UPDATE models SET status = 'ACTIVE' WHERE version = 'v1.2.0';
  UPDATE models SET status = 'ARCHIVED' WHERE version = 'v2.0.0';
  ```

### 8.5 Training Configuration
**File:** `config.yaml` (updates)

New fields:
```yaml
active_learning:
  enabled: true
  min_annotated_samples: 100  # minimum labeled data to trigger training
  training:
    epochs: 20
    batch_size: 32
    learning_rate: 1e-3
    weight_decay: 1e-2
    early_stopping_patience: 5
    lr_scheduler: ReduceLROnPlateau
    lr_factor: 0.5
    lr_patience: 3
  validation:
    val_split: 0.2
    min_f1_improvement: 0.02
    max_ece: 0.08
  deployment:
    auto_deploy: true  # deploy if quality gates pass
    hot_reload: true
    rollback_enabled: true
  augmentation:
    horizontal_flip: true
    color_jitter: true
    affine: true
    affine_degrees: 10
    affine_scale: 0.1
```

## Acceptance Criteria

### Functional
- [ ] Annotated labels are ingested from review_queue table correctly
- [ ] Dataset is versioned with proper directory structure
- [ ] Train/val split is stratified by class (80/20)
- [ ] Data augmentation is applied at training time (not pre-computed)
- [ ] Backbone weights are frozen during fine-tuning (verified by checking requires_grad)
- [ ] Classification head is trainable and receives gradient updates
- [ ] AdamW optimizer is used with correct parameters
- [ ] Early stopping works (training stops if no improvement for patience epochs)
- [ ] Candidate model is saved to /models/candidate/ with correct versioning
- [ ] F1 score is calculated correctly on validation set
- [ ] ECE is calculated correctly on validation set
- [ ] Quality gates are enforced (ΔF1 > 0.02 AND ECE < 0.08)
- [ ] Model registry is updated (ACTIVE, CANDIDATE, ARCHIVED states)
- [ ] TensorRT engine is built from candidate model
- [ ] Hot reload deploys new model without downtime
- [ ] Rollback restores previous model if deployment fails

### Non-Functional
- [ ] Training completes in < 4 hours on Tesla P40 (typical dataset: 1000 samples)
- [ ] Validation completes in < 30 minutes
- [ ] TensorRT engine build completes in < 10 minutes
- [ ] Hot reload completes in < 30 seconds
- [ ] Training uses < 18GB GPU memory
- [ ] Training is deterministic (same data + same config → same results)

### Technical Constraints
- [ ] CUDA 11.8 compatible (verified via torch.version.cuda)
- [ ] PyTorch ≤ 2.1.0 (verified via torch.__version__)
- [ ] FP32 training only (no mixed precision)
- [ ] No Tensor Cores used (CC 5.2 constraint)
- [ ] Model versioning follows semantic versioning (MAJOR.MINOR.PATCH)
- [ ] All training hyperparameters are configurable via config.yaml

## Dependencies
- **Prerequisites:** STORY-01 (Foundation), STORY-05 (Classification — provides base model), STORY-07 (Review UI — provides labeled data)
- **Depends on:** None (runs independently, triggered by labeled data threshold)
- **Enables:** STORY-05 (provides new model for inference)

## Risks & Mitigations
| Risk | Mitigation |
|------|--|
| Insufficient labeled data for meaningful fine-tuning | Set min_annotated_samples threshold (e.g., 100); wait until reached |
| Head-only fine-tuning may not be enough for domain shift | Offer full fine-tuning as option; document limitations |
| Model regression in production | Strict quality gates; keep previous model in ARCHIVED state for quick rollback |
| Long training times on Tesla P40 | Head-only training is faster; batch size tuning; early stopping |
| ECE calculation sensitivity | Use proper binning; report ECI with confidence intervals |

## Estimated Effort
- **Sprint:** 7-8
- **Story Points:** 34
- **Dependencies:** STORY-01, STORY-05, STORY-07