After story 1
This commit is contained in:
+187
@@ -0,0 +1,187 @@
|
||||
# STORY-05: Classification & Confidence Aggregation
|
||||
|
||||
## Epic
|
||||
**E1: Core Pipeline** — I can classify face crops and aggregate video confidence.
|
||||
|
||||
## Related Requirements
|
||||
| ID | Requirement |
|
||||
|----|------|
|
||||
| FR-03 | Binary demographic classification of detected face crops: Outputs probability p ∈ [0,1] for target class |
|
||||
| FR-04 | Video-level confidence aggregation & threshold routing: Aggregates frame-level scores → video confidence C; Routes to MATCH, REVIEW, or SKIP |
|
||||
| NFR-01 | Throughput: ≥ 30 videos/hour/GPU |
|
||||
| NFR-03 | GPU Memory Safety: ≤ 18GB per GPU sustained |
|
||||
| NFR-04 | Determinism & Reproducibility: Config-seeded randomness, versioned models |
|
||||
| TC-01 | Hardware: 2× Tesla P40 24GB (compute capability 5.2, PCIe 3.0, no Tensor Cores) |
|
||||
| TC-02 | CUDA/Torch Compatibility: CUDA ≤ 11.8, PyTorch ≤ 2.1.0, FP32 inference only |
|
||||
|
||||
## Description
|
||||
Implement demographic classification of detected face crops using a lightweight classifier (MobileNetV3 or equivalent). Apply temperature scaling for calibrated confidence scores. Aggregate frame-level confidence to video-level confidence using configurable aggregation logic. Route videos to MATCH, REVIEW, or SKIP based on threshold comparison.
|
||||
|
||||
## Scope
|
||||
|
||||
### In Scope
|
||||
- MobileNetV3 (or equivalent) classification model via TensorRT FP32
|
||||
- Temperature scaling for confidence calibration
|
||||
- Frame-level confidence computation (softmax with temperature)
|
||||
- Video-level confidence aggregation (max or weighted mean, configurable)
|
||||
- Threshold-based routing (T_high, T_low)
|
||||
- Routing decision assignment (MATCH, REVIEW, SKIP)
|
||||
- Confidence score persistence to DB and output files
|
||||
|
||||
### Out of Scope
|
||||
- Frame sampling (covered in STORY-03)
|
||||
- Face detection (covered in STORY-04)
|
||||
- Results export format (covered in STORY-06)
|
||||
- Review UI (covered in STORY-07)
|
||||
- Active learning / model retraining (covered in STORY-08)
|
||||
|
||||
## Deliverables
|
||||
|
||||
### 5.1 Classification Model
|
||||
**File:** `models/classifier/`
|
||||
|
||||
Components:
|
||||
- **Source Model:** MobileNetV3-small (or equivalent lightweight classifier)
|
||||
- Trained on demographic dataset (fairness considerations documented)
|
||||
- Binary classification: Black male subject (class 1) vs. not (class 0)
|
||||
- **ONNX Export:** `classifier.onnx`
|
||||
- Input: `[1, 3, 224, 224]` (RGB, resized face crop)
|
||||
- Output: `[1, 2]` (logits for class 0 and class 1)
|
||||
- **TensorRT Engine:** `classifier.trt` (FP32)
|
||||
- Builder config: max_batch_size=32, max_workspace_size=2GB
|
||||
- **Calibration Data:** Temperature parameter T stored with model
|
||||
- **Model Metadata:** `model.json` with architecture, training dataset, validation metrics
|
||||
|
||||
### 5.2 Classification Runner
|
||||
**File:** `src/classifier.py`
|
||||
|
||||
Core components:
|
||||
- **Model Loader:** Load TensorRT engine at startup
|
||||
```python
|
||||
class FaceClassifier:
|
||||
def __init__(self, engine_path, temperature=1.0, device='cuda'):
|
||||
self.engine = load_trt_engine(engine_path)
|
||||
self.context = self.engine.create_execution_context()
|
||||
self.temperature = temperature
|
||||
self.input_shape = (1, 3, 224, 224)
|
||||
```
|
||||
- **Preprocessing:** Convert face crop to model input
|
||||
- Resize to 224×224
|
||||
- Normalize (ImageNet statistics)
|
||||
- Convert to tensor (NCHW format)
|
||||
- **Inference:** Run classification on face crops
|
||||
- Dynamic batching (shared with face detection batcher)
|
||||
- Execute TensorRT engine
|
||||
- Get raw logits output
|
||||
- **Temperature Scaling:** Apply temperature to logits before softmax
|
||||
```python
|
||||
def calibrated_softmax(logits, temperature):
|
||||
scaled_logits = logits / temperature
|
||||
return softmax(scaled_logits, axis=-1)
|
||||
```
|
||||
- **Frame-level Confidence:** Extract probability for target class (class 1)
|
||||
```python
|
||||
p_i = calibrated_probs[:, 1] # probability of target class
|
||||
```
|
||||
|
||||
### 5.3 Confidence Aggregation
|
||||
**File:** `src/aggregator.py`
|
||||
|
||||
Features:
|
||||
- **Aggregation Strategies (configurable):**
|
||||
1. **Max:** `C = max(p_i)` — use highest confidence frame
|
||||
2. **Weighted Mean:** `C = softmax(α·mean(p_i) + β·var(p_i))`
|
||||
3. **Top-K Mean:** `C = mean(top_k(p_i))` — average of top K confidences
|
||||
- **Default Strategy:** Max (most conservative, aligns with FR-04)
|
||||
- **Parameter Configuration:** α, β, K configurable in config.yaml
|
||||
- **Variance Calculation:** Compute variance of frame-level confidences (for uncertainty estimation)
|
||||
|
||||
### 5.4 Threshold Routing
|
||||
**File:** `src/router.py`
|
||||
|
||||
Features:
|
||||
- **Threshold Comparison:**
|
||||
```python
|
||||
if C >= T_high: # default 0.75
|
||||
routing = 'MATCH'
|
||||
elif C >= T_low: # default 0.45
|
||||
routing = 'REVIEW'
|
||||
else:
|
||||
routing = 'SKIP'
|
||||
```
|
||||
- **Configurable Thresholds:** T_high and T_low in config.yaml
|
||||
- **Routing Decision Logging:** Log routing decision with confidence score
|
||||
- **Edge Case Handling:**
|
||||
- No faces detected → routing = SKIP (with confidence = 0.0)
|
||||
- All frames have same confidence → routing based on threshold comparison
|
||||
- Confidence exactly at threshold → use >= comparison (inclusive)
|
||||
|
||||
### 5.5 Configuration Updates
|
||||
**File:** `config.yaml` (updates)
|
||||
|
||||
New fields:
|
||||
```yaml
|
||||
classifier:
|
||||
model: mobilenetv3-small
|
||||
model_path: /models/classifier/classifier.trt
|
||||
input_size: 224
|
||||
temperature: 1.0 # calibration temperature
|
||||
default_strategy: max # max, weighted_mean, top_k_mean
|
||||
|
||||
aggregation:
|
||||
strategy: max
|
||||
alpha: 1.0 # for weighted_mean
|
||||
beta: 0.1 # for weighted_mean
|
||||
top_k: 3 # for top_k_mean
|
||||
|
||||
routing:
|
||||
T_high: 0.75
|
||||
T_low: 0.45
|
||||
no_faces_decision: SKIP
|
||||
```
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
### Functional
|
||||
- [ ] MobileNetV3 model loads and runs inference via TensorRT FP32
|
||||
- [ ] Classification outputs calibrated probability p ∈ [0,1] for target class
|
||||
- [ ] Temperature scaling is applied correctly (verified on test set)
|
||||
- [ ] Frame-level confidence scores are deterministic (same input → same output)
|
||||
- [ ] Max aggregation produces C = max(p_i) correctly
|
||||
- [ ] Weighted mean aggregation produces correct result with configurable α, β
|
||||
- [ ] Threshold routing assigns exactly one of: MATCH, REVIEW, SKIP
|
||||
- [ ] Videos with no detected faces are routed to SKIP with confidence 0.0
|
||||
- [ ] Confidence exactly at threshold uses >= comparison (inclusive)
|
||||
|
||||
### Non-Functional
|
||||
- [ ] Classification for 16 face crops (batched) completes in < 1 second on Tesla P40
|
||||
- [ ] GPU memory usage stays ≤ 18GB per GPU during classification
|
||||
- [ ] Aggregation computation is negligible (< 10ms per video)
|
||||
- [ ] Temperature parameter is stored with model checkpoint
|
||||
- [ ] All routing decisions are logged with confidence score and threshold values
|
||||
|
||||
### Technical Constraints
|
||||
- [ ] TensorRT engine is FP32 only
|
||||
- [ ] No Tensor Cores used (CC 5.2 constraint)
|
||||
- [ ] CUDA 11.8 compatible
|
||||
- [ ] ONNX model exports without errors
|
||||
- [ ] Confidence scores are reproducible (deterministic inference)
|
||||
- [ ] Aggregation strategy is configurable without code changes
|
||||
|
||||
## Dependencies
|
||||
- **Prerequisites:** STORY-01 (Foundation), STORY-03 (Frame Sampling), STORY-04 (Face Detection — provides crops)
|
||||
- **Depends on:** None (runs after face detection)
|
||||
- **Enables:** STORY-06 (Results Persistence), STORY-07 (Review UI — provides routing decisions)
|
||||
|
||||
## Risks & Mitigations
|
||||
| Risk | Mitigation |
|
||||
|------|--|
|
||||
| Temperature scaling parameters need tuning | Evaluate on held-out calibration set during training cycle |
|
||||
| Max aggregation may be overly conservative | Offer weighted mean as alternative; allow per-job strategy selection |
|
||||
| Model bias concerns with demographic classification | Document training dataset; audit fairness metrics; legal review required |
|
||||
| TensorRT engine build time | Pre-build and cache engines; version control model files |
|
||||
|
||||
## Estimated Effort
|
||||
- **Sprint:** 5-6 (second half)
|
||||
- **Story Points:** 21
|
||||
- **Dependencies:** STORY-01, STORY-03, STORY-04
|
||||
Reference in New Issue
Block a user