246 lines
9.4 KiB
Markdown
246 lines
9.4 KiB
Markdown
# STORY-08: Active Learning Pipeline
|
||
|
||
## Epic
|
||
**E3: Active Learning** — As an ML engineer, I can fine-tune the classifier head with reviewed data.
|
||
|
||
## Related Requirements
|
||
| ID | Requirement |
|
||
|----|------|
|
||
| FR-06 | Active learning pipeline (label ingestion → fine-tuning → deployment): Batch retraining only. No online learning. Versioned model swaps. |
|
||
| NFR-04 | Determinism & Reproducibility: Config-seeded randomness, versioned models |
|
||
| NFR-05 | Fault Tolerance: Auto-retry on transient failures; skip & log on fatal errors |
|
||
| TC-01 | Hardware: 2× Tesla P40 24GB (compute capability 5.2, PCIe 3.0, no Tensor Cores) |
|
||
| TC-02 | CUDA/Torch Compatibility: CUDA ≤ 11.8, PyTorch ≤ 2.1.0, FP32 inference only |
|
||
|
||
## Description
|
||
Implement the active learning pipeline that ingests annotated review data, fine-tunes the classifier head, validates the candidate model, and deploys it if it meets quality gates. This enables incremental model improvement using human-labeled data from the review queue.
|
||
|
||
## Scope
|
||
|
||
### In Scope
|
||
- Label ingestion from review queue (CSV/JSON export or direct DB query)
|
||
- Dataset versioning and preparation for training
|
||
- Head-only fine-tuning on face crops (freeze backbone, train classification head)
|
||
- Training with AdamW, configurable learning rate, early stopping
|
||
- Candidate model validation on held-out validation set
|
||
- F1 score and ECE (Expected Calibration Error) calculation
|
||
- Quality gate validation (ΔF1 > 0.02, ECE < 0.08)
|
||
- Model registry updates (ACTIVE, CANDIDATE, ARCHIVED states)
|
||
- Hot reload of new TensorRT engine
|
||
- Rollback on regression
|
||
|
||
### Out of Scope
|
||
- Frame sampling (covered in STORY-03)
|
||
- Face detection (covered in STORY-04)
|
||
- Classification inference (covered in STORY-05)
|
||
- Review UI (covered in STORY-07)
|
||
- Monitoring dashboards (covered in STORY-09)
|
||
|
||
## Deliverables
|
||
|
||
### 8.1 Label Ingestion
|
||
**File:** `src/active_learning/label_ingestor.py`
|
||
|
||
Features:
|
||
- **Data Source:** Query `review_queue` table for annotated labels
|
||
```sql
|
||
SELECT video_id, ground_truth, contributing_frames
|
||
FROM review_queue
|
||
WHERE annotated = true AND ground_truth IS NOT NULL
|
||
```
|
||
- **Crop Extraction:** Extract face crops from stored paths or re-extract from video
|
||
- **Dataset Versioning:** Create versioned dataset directory structure
|
||
```
|
||
/data/training/v2.0.0/
|
||
crops/
|
||
class_0/ (negative samples)
|
||
class_1/ (positive samples)
|
||
labels.csv
|
||
metadata.json
|
||
```
|
||
- **Train/Val Split:** 80/20 split (stratified by class)
|
||
- **Augmentation:** Apply standard augmentations at training time (not pre-computed)
|
||
- Random horizontal flip
|
||
- Random color jitter
|
||
- Random affine transform (±10 degrees rotation, ±10% scale)
|
||
|
||
### 8.2 Training Pipeline
|
||
**File:** `src/active_learning/trainer.py`
|
||
|
||
Features:
|
||
- **Model Loading:** Load pre-trained MobileNetV3 backbone (frozen)
|
||
```python
|
||
model = load_pretrained_mobilenetv3()
|
||
for param in model.features.parameters():
|
||
param.requires_grad = False
|
||
# Replace classification head
|
||
model.classifier = nn.Sequential(
|
||
nn.Linear(1280, 256),
|
||
nn.ReLU(),
|
||
nn.Dropout(0.3),
|
||
nn.Linear(256, 2) # binary classification
|
||
)
|
||
```
|
||
- **Head-Only Fine-Tuning:** Only train the classification head
|
||
- Backbone weights are frozen (no gradient updates)
|
||
- Head weights are trainable
|
||
- This is faster and requires less data than full fine-tuning
|
||
- **Optimizer:** AdamW with configurable parameters
|
||
- Learning rate: 1e-3 (default), configurable
|
||
- Weight decay: 1e-2
|
||
- Betas: (0.9, 0.999)
|
||
- **Loss Function:** Binary Cross-Entropy with class weights (if imbalanced)
|
||
- **Training Configuration:**
|
||
- Epochs: 10-30 (configurable)
|
||
- Batch size: 32
|
||
- Early stopping: patience=5 epochs (no validation improvement)
|
||
- Learning rate scheduler: ReduceLROnPlateau (factor=0.5, patience=3)
|
||
- **Checkpointing:** Save checkpoint to `/models/candidate/` after each epoch
|
||
```
|
||
/models/candidate/
|
||
v2.0.0_epoch_01.pt
|
||
v2.0.0_epoch_02.pt
|
||
...
|
||
v2.0.0_best.pt (best validation F1)
|
||
```
|
||
|
||
### 8.3 Validation Pipeline
|
||
**File:** `src/active_learning/validator.py`
|
||
|
||
Features:
|
||
- **Validation Set:** Held-out 20% of annotated data (never seen during training)
|
||
- **Metrics Calculation:**
|
||
- **F1 Score:** Macro F1 on validation set
|
||
- **ECE (Expected Calibration Error):**
|
||
```python
|
||
def compute_ece(predictions, labels, n_bins=15):
|
||
bin_boundaries = np.linspace(0, 1, n_bins + 1)
|
||
ece = 0.0
|
||
for i in range(n_bins):
|
||
mask = (predictions >= bin_boundaries[i]) & (predictions < bin_boundaries[i+1])
|
||
if mask.sum() > 0:
|
||
bin_confidence = predictions[mask].mean()
|
||
bin_accuracy = labels[mask].mean()
|
||
ece += (mask.sum() / len(predictions)) * abs(bin_confidence - bin_accuracy)
|
||
return ece
|
||
```
|
||
- **Accuracy, Precision, Recall:** Standard classification metrics
|
||
- **Quality Gates:**
|
||
- ΔF1 > 0.02 (improvement over current model)
|
||
- ECE < 0.08 (calibration acceptable)
|
||
- Both gates must pass for deployment
|
||
|
||
### 8.4 Model Registry & Deployment
|
||
**File:** `src/active_learning/registry.py`
|
||
|
||
Features:
|
||
- **Model Registry:** Update DB `models` table
|
||
```sql
|
||
-- Promote candidate to active
|
||
UPDATE models SET status = 'ACTIVE' WHERE version = 'v2.0.0';
|
||
UPDATE models SET status = 'ARCHIVED' WHERE status = 'ACTIVE' AND version != 'v2.0.0';
|
||
```
|
||
- **TensorRT Engine Build:** Convert candidate model to TensorRT engine
|
||
```bash
|
||
trtexec --onnx=models/candidate/v2.0.0.onnx \
|
||
--saveEngine=models/candidate/v2.0.0.trt \
|
||
--fp32 --maxBatch=32 --workspace=2048
|
||
```
|
||
- **Hot Reload:** Signal worker process to reload new engine
|
||
- Option A: Restart worker container (`docker-compose restart worker`)
|
||
- Option B: In-process reload (graceful, no downtime)
|
||
- Default: In-process reload via signal handling
|
||
- **Rollback:** If new model causes issues, rollback to archived model
|
||
```sql
|
||
UPDATE models SET status = 'ACTIVE' WHERE version = 'v1.2.0';
|
||
UPDATE models SET status = 'ARCHIVED' WHERE version = 'v2.0.0';
|
||
```
|
||
|
||
### 8.5 Training Configuration
|
||
**File:** `config.yaml` (updates)
|
||
|
||
New fields:
|
||
```yaml
|
||
active_learning:
|
||
enabled: true
|
||
min_annotated_samples: 100 # minimum labeled data to trigger training
|
||
training:
|
||
epochs: 20
|
||
batch_size: 32
|
||
learning_rate: 1e-3
|
||
weight_decay: 1e-2
|
||
early_stopping_patience: 5
|
||
lr_scheduler: ReduceLROnPlateau
|
||
lr_factor: 0.5
|
||
lr_patience: 3
|
||
validation:
|
||
val_split: 0.2
|
||
min_f1_improvement: 0.02
|
||
max_ece: 0.08
|
||
deployment:
|
||
auto_deploy: true # deploy if quality gates pass
|
||
hot_reload: true
|
||
rollback_enabled: true
|
||
augmentation:
|
||
horizontal_flip: true
|
||
color_jitter: true
|
||
affine: true
|
||
affine_degrees: 10
|
||
affine_scale: 0.1
|
||
```
|
||
|
||
## Acceptance Criteria
|
||
|
||
### Functional
|
||
- [ ] Annotated labels are ingested from review_queue table correctly
|
||
- [ ] Dataset is versioned with proper directory structure
|
||
- [ ] Train/val split is stratified by class (80/20)
|
||
- [ ] Data augmentation is applied at training time (not pre-computed)
|
||
- [ ] Backbone weights are frozen during fine-tuning (verified by checking requires_grad)
|
||
- [ ] Classification head is trainable and receives gradient updates
|
||
- [ ] AdamW optimizer is used with correct parameters
|
||
- [ ] Early stopping works (training stops if no improvement for patience epochs)
|
||
- [ ] Candidate model is saved to /models/candidate/ with correct versioning
|
||
- [ ] F1 score is calculated correctly on validation set
|
||
- [ ] ECE is calculated correctly on validation set
|
||
- [ ] Quality gates are enforced (ΔF1 > 0.02 AND ECE < 0.08)
|
||
- [ ] Model registry is updated (ACTIVE, CANDIDATE, ARCHIVED states)
|
||
- [ ] TensorRT engine is built from candidate model
|
||
- [ ] Hot reload deploys new model without downtime
|
||
- [ ] Rollback restores previous model if deployment fails
|
||
|
||
### Non-Functional
|
||
- [ ] Training completes in < 4 hours on Tesla P40 (typical dataset: 1000 samples)
|
||
- [ ] Validation completes in < 30 minutes
|
||
- [ ] TensorRT engine build completes in < 10 minutes
|
||
- [ ] Hot reload completes in < 30 seconds
|
||
- [ ] Training uses < 18GB GPU memory
|
||
- [ ] Training is deterministic (same data + same config → same results)
|
||
|
||
### Technical Constraints
|
||
- [ ] CUDA 11.8 compatible (verified via torch.version.cuda)
|
||
- [ ] PyTorch ≤ 2.1.0 (verified via torch.__version__)
|
||
- [ ] FP32 training only (no mixed precision)
|
||
- [ ] No Tensor Cores used (CC 5.2 constraint)
|
||
- [ ] Model versioning follows semantic versioning (MAJOR.MINOR.PATCH)
|
||
- [ ] All training hyperparameters are configurable via config.yaml
|
||
|
||
## Dependencies
|
||
- **Prerequisites:** STORY-01 (Foundation), STORY-05 (Classification — provides base model), STORY-07 (Review UI — provides labeled data)
|
||
- **Depends on:** None (runs independently, triggered by labeled data threshold)
|
||
- **Enables:** STORY-05 (provides new model for inference)
|
||
|
||
## Risks & Mitigations
|
||
| Risk | Mitigation |
|
||
|------|--|
|
||
| Insufficient labeled data for meaningful fine-tuning | Set min_annotated_samples threshold (e.g., 100); wait until reached |
|
||
| Head-only fine-tuning may not be enough for domain shift | Offer full fine-tuning as option; document limitations |
|
||
| Model regression in production | Strict quality gates; keep previous model in ARCHIVED state for quick rollback |
|
||
| Long training times on Tesla P40 | Head-only training is faster; batch size tuning; early stopping |
|
||
| ECE calculation sensitivity | Use proper binning; report ECI with confidence intervals |
|
||
|
||
## Estimated Effort
|
||
- **Sprint:** 7-8
|
||
- **Story Points:** 34
|
||
- **Dependencies:** STORY-01, STORY-05, STORY-07
|