# Implementation Plan: Video Classification System

## Overview
This document outlines the phased implementation plan for the Video Classification System. The plan breaks down the requirements into logical work packages (Sprints/Phases) that can be implemented, tested, and integrated sequentially.

**Key Dependencies:**
1.  **Infrastructure** must be available before any processing logic can run.
2.  **Database Schema** must be finalized early to support metadata logging.
3.  **Model Training Data** is a bottleneck; the initial model must be ready or a placeholder used for pipeline testing.

---

## Phase 1: Foundation & Infrastructure (Sprint 1-2)
**Goal:** Establish the Docker environment, Database schema, and basic connectivity. No video processing yet, just the "skeleton."

### 1.1 Environment Setup
- [ ] **Docker Compose Structure:** Create `docker-compose.yml` defining services for:
    -   `mariadb`: Persistent volume for metadata.
    -   `worker`: Build context for PyTorch/TensorRT environment. Include `nvidia-container-toolkit` config for GPU passthrough.
    -   `ui`: Placeholder container (e.g., nginx serving a static "Under Construction" page) to verify network connectivity.
- [ ] **Base Images:**
    -   Create a custom Dockerfile for the worker based on `nvidia/cuda:11.8.0-runtime-ubuntu20.04`.
    -   Install PyTorch (1.13+ with CUDA 11.8), OpenCV, FFmpeg-python, and TensorRT prerequisites.
- [ ] **Volume Mounts:**
    -   Map local NVMe path to `/scratch` in worker container (tmpfs preferred for speed).
    -   Map NAS/SMB path to `/data/input` and `/data/output` in worker container.
    -   Map `/models` and `/data/training` for persistence.

### 1.2 Database Schema Design
- [ ] **MariaDB Schema:** Define tables in SQL migration script:
    -   `videos`: `id`, `file_path`, `file_hash`, `resolution_w`, `resolution_h`, `codec`, `status` (NEW, PENDING, PROCESSING, COMPLETED, UNSCANNABLE, ERROR), `last_scan_time`, `last_processed_time`.
    -   `processing_logs`: `id`, `video_id`, `model_version`, `frame_count`, `confidence_score`, `routing_decision`, `processed_at`.
    -   `models`: `version`, `status` (ACTIVE, CANDIDATE, ARCHIVED), `path`, `calibration_temp`.
- [ ] **Connection Layer:** Implement a Python module (`db_connector.py`) with connection pooling to MariaDB.

### 1.3 Configuration Management
- [ ] **Config File:** Create `config.yaml` for thresholds (`T_high`, `T_low`), sampling intervals, and GPU settings.
- [ ] **Logging Setup:** Configure structured logging (JSON format) to stdout for Docker capture.

**Exit Criteria:**
- `docker-compose up` starts DB and Worker containers successfully.
- Worker can connect to DB and create tables.
- GPU is visible inside the Worker container (`nvidia-smi`).

---

## Phase 2: Core Ingestion & Codec Handling (Sprint 3-4)
**Goal:** Implement the directory scanner, video probing, and robust error handling for codecs/resolutions.

### 2.1 Directory Scanner
- [ ] **Scanner Service:** Implement a background thread or separate script that walks `/data/input`.
- [ ] **File Detection:** Identify new files (not in DB) and update status to `PENDING`.
- [ ] **Deduplication:** Use `file_hash` (SHA-256 of first 1MB or full file if small) to prevent re-processing identical files.

### 2.2 Video Probing & Metadata Extraction
- [ ] **FFmpeg Probe:** For each `PENDING` file, run `ffprobe` to extract:
    -   Codec name.
    -   Resolution (width/height).
    -   Duration.
- [ ] **Codec Validation:** Maintain a whitelist of supported codecs (H.264, H.265, VP8, VP9, etc.).
    -   If codec is unsupported or `ffprobe` fails: Set status to `UNSCANNABLE`.
    -   Log the specific error code/reason.
- [ ] **Resolution Handling:** Store resolution in DB. Implement logic to downscale high-res frames (4K) if necessary to fit VRAM constraints during inference.

### 2.3 Batch Orchestration Skeleton
- [ ] **Job Queue:** Implement a simple priority queue or DB-based locking mechanism to assign files to Worker processes.
- [ ] **State Management:** Ensure atomic transitions from `PENDING` → `PROCESSING` to avoid duplicate processing.

**Exit Criteria:**
- New files in `/data/input` appear in DB with correct metadata within 60 seconds.
- Unsupported codec files are marked `UNSCANNABLE` without crashing the scanner.
- DB accurately reflects the current state of the 30TB corpus.

---

## Phase 3: Inference Pipeline & Model Integration (Sprint 5-6)
**Goal:** Implement frame sampling, face detection, classification, and confidence aggregation.

### 3.1 Frame Sampling
- [ ] **FFmpeg Extraction:** Implement logic to extract 1 frame per 30 seconds uniformly.
    -   Handle variable FPS gracefully.
    -   Save frames to `/scratch/tmp/` with naming convention `{video_id}_{timestamp}.jpg`.
- [ ] **Memory Management:** Ensure frames are deleted from scratch space immediately after processing the video to prevent filling NVMe.

### 3.2 Face Detection
- [ ] **Model Integration:** Load YOLOv8n (or chosen lightweight detector) via ONNX Runtime or TensorRT.
- [ ] **Batching:** Implement dynamic batching for face detection to maximize GPU utilization.
- [ ] **Crop Generation:** Extract face crops, resize to inference input size (e.g., 224x224).

### 3.3 Classification & Aggregation
- [ ] **Model Integration:** Load MobileNetV3 (or chosen classifier) via TensorRT FP32.
- [ ] **Inference:** Run classification on face crops.
- [ ] **Aggregation Logic:**
    -   Calculate frame-level confidence `p_i`.
    -   Apply video-level aggregation (e.g., `max(p_i)` or weighted mean).
    -   Apply temperature scaling for calibration.
- [ ] **Routing Decision:** Compare final confidence `C` against `T_high` and `T_low`. Assign `MATCH`, `REVIEW`, or `SKIP`.

### 3.4 Result Persistence
- [ ] **DB Update:** Update `videos` table with `processed_at`, `model_version`, and status `COMPLETED`.
- [ ] **Log Entry:** Insert record into `processing_logs`.
- [ ] **Export:** Write summary JSON/Parquet file to `/data/output` for analytics.

**Exit Criteria:**
- A sample video is processed end-to-end: Frames extracted → Faces detected → Classified → Confidence calculated → DB updated.
- GPU memory usage stays under 18GB per GPU.
- Throughput meets baseline (>30 videos/hour/GPU on test set).

---

## Phase 4: Active Learning & Review UI (Sprint 7-8)
**Goal:** Enable human-in-the-loop correction and model retraining.

### 4.1 Review UI
- [ ] **UI Setup:** Deploy Label Studio or custom Flask UI.
- [ ] **Data Feed:** Query DB for videos with routing_decision = `REVIEW`.
- [ ] **Frontend:** Display video player + top-k contributing frames. Allow annotator to toggle label (True/False) or correct classification.
- [ ] **Export:** Generate CSV/JSON export of annotated data with ground truth.

### 4.2 Fine-Tuning Pipeline
- [ ] **Data Loader:** Script to ingest exported annotations and prepare dataset for training.
- [ ] **Training Job:**
    -   Head-only fine-tuning on face crops.
    -   Use AdamW, LR=1e-3, Early Stopping.
    -   Save checkpoint to `/models/candidate/`.
- [ ] **Validation:**
    -   Run candidate model on held-out validation set.
    -   Calculate F1 score and ECE (Expected Calibration Error).
    -   If ΔF1 > 0.02 and ECE < 0.08, promote model to `ACTIVE`.

### 4.3 Model Swapping
- [ ] **Registry Update:** Update DB `models` table to mark new version as `ACTIVE` and old as `ARCHIVED`.
- [ ] **Hot Reload:** Restart Worker containers or signal process to reload new TensorRT engine.

**Exit Criteria:**
- Annotator can label a review video.
- Labeled data triggers a fine-tuning job.
- New model is validated and deployed automatically.

---

## Phase 5: Observability, Monitoring & Hardening (Sprint 9+)
**Goal:** Ensure system reliability, debuggability, and long-term stability.

### 5.1 Metrics & Logging
- [ ] **Prometheus Integration:** Expose metrics:
    -   `videos_processed_total` (by routing decision).
    -   `gpu_utilization`, `gpu_memory_used`.
    -   `queue_depth` (PENDING/PROCESSING counts).
    -   `confidence_distribution` histogram.
- [ ] **Grafana Dashboard:** Create dashboards for:
    -   Throughput (videos/hour).
    -   Error rates (UNSCANNABLE/ERROR counts).
    -   Confidence drift alerts.

### 5.2 Fault Tolerance & Resumption
- [ ] **Crash Recovery:** Implement checkpointing. If Worker dies, mark `PROCESSING` jobs as `PENDING` again.
- [ ] **Idempotency:** Ensure re-processing a file does not duplicate DB entries or outputs.
- [ ] **Retry Logic:** Auto-retry transient errors (e.g., network blip during NAS access) up to 3 times.

### 5.3 Drift Detection
- [ ] **Weekly Job:** Compare current confidence distribution to baseline.
- [ ] **Alerting:** Send alert if `p(C > 0.5)` shifts >10% or Review Queue grows unbounded.

**Exit Criteria:**
- Dashboard shows real-time health of the system.
- System recovers gracefully from forced Worker termination.
- Alerts trigger correctly on simulated drift/errors.

---

## Risk Mitigation

| Risk | Mitigation Strategy |
|------|---------------------|
| **GPU OOM** | Implement strict batch size caps; downscale 4K frames to 1080p/720p before inference. |
| **Slow NAS I/O** | Cache frames in local NVMe `tmpfs`; minimize disk writes until final export. |
| **Codec Variance** | Extensive unit testing of `ffprobe` logic against known "bad" files. Graceful degradation to `UNSCANNABLE`. |
| **Model Regression** | Strict validation gate before model swap. Keep previous model in `ARCHIVED` state for quick rollback. |
| **30TB Scan Time** | Incremental scanning using `file_hash` and `last_modified` timestamps. Avoid full rescan. |

---

## Deliverables Checklist
- [ ] `docker-compose.yml`
- [ ] Worker Dockerfile & Requirements.txt
- [ ] MariaDB SQL Schema
- [ ] Python Source Code (`src/`)
    -   `scanner.py`
    -   `processor.py`
    -   `model_runner.py`
    -   `db_handler.py`
    -   `train.py`
- [ ] Config Files (`config.yaml`)
- [ ] Grafana Dashboard JSON
- [ ] Documentation (`README.md`, `API_DOCS.md`)