VideoDetect/Plan.md

Implementation Plan: Video Classification System

Overview

This document outlines the phased implementation plan for the Video Classification System. The plan breaks down the requirements into logical work packages (Sprints/Phases) that can be implemented, tested, and integrated sequentially.

Key Dependencies:

1. Infrastructure must be available before any processing logic can run.
2. Database Schema must be finalized early to support metadata logging.
3. Model Training Data is a bottleneck; the initial model must be ready or a placeholder used for pipeline testing.

---

Phase 1: Foundation & Infrastructure (Sprint 1-2)

Goal: Establish the Docker environment, Database schema, and basic connectivity. No video processing yet, just the "skeleton."

1.1 Environment Setup

• Docker Compose Structure: Create docker-compose.yml defining services for:
  • mariadb: Persistent volume for metadata.
  • worker: Build context for PyTorch/TensorRT environment. Include nvidia-container-toolkit config for GPU passthrough.
  • ui: Placeholder container (e.g., nginx serving a static "Under Construction" page) to verify network connectivity.
• Base Images:
  • Create a custom Dockerfile for the worker based on nvidia/cuda:11.8.0-runtime-ubuntu20.04.
  • Install PyTorch (1.13+ with CUDA 11.8), OpenCV, FFmpeg-python, and TensorRT prerequisites.
• Volume Mounts:
  • Map local NVMe path to /scratch in worker container (tmpfs preferred for speed).
  • Map NAS/SMB path to /data/input and /data/output in worker container.
  • Map /models and /data/training for persistence.

1.2 Database Schema Design

• MariaDB Schema: Define tables in SQL migration script:
  • videos: id, file_path, file_hash, resolution_w, resolution_h, codec, status (NEW, PENDING, PROCESSING, COMPLETED, UNSCANNABLE, ERROR), last_scan_time, last_processed_time.
  • processing_logs: id, video_id, model_version, frame_count, confidence_score, routing_decision, processed_at.
  • models: version, status (ACTIVE, CANDIDATE, ARCHIVED), path, calibration_temp.
• Connection Layer: Implement a Python module (db_connector.py) with connection pooling to MariaDB.

1.3 Configuration Management

• Config File: Create config.yaml for thresholds (T_high, T_low), sampling intervals, and GPU settings.
• Logging Setup: Configure structured logging (JSON format) to stdout for Docker capture.

Exit Criteria:

• docker-compose up starts DB and Worker containers successfully.
• Worker can connect to DB and create tables.
• GPU is visible inside the Worker container (nvidia-smi).

---

Phase 2: Core Ingestion & Codec Handling (Sprint 3-4)

Goal: Implement the directory scanner, video probing, and robust error handling for codecs/resolutions.

2.1 Directory Scanner

• Scanner Service: Implement a background thread or separate script that walks /data/input.
• File Detection: Identify new files (not in DB) and update status to PENDING.
• Deduplication: Use file_hash (SHA-256 of first 1MB or full file if small) to prevent re-processing identical files.

2.2 Video Probing & Metadata Extraction

• FFmpeg Probe: For each PENDING file, run ffprobe to extract:
  • Codec name.
  • Resolution (width/height).
  • Duration.
• Codec Validation: Maintain a whitelist of supported codecs (H.264, H.265, VP8, VP9, etc.).
  • If codec is unsupported or ffprobe fails: Set status to UNSCANNABLE.
  • Log the specific error code/reason.
• Resolution Handling: Store resolution in DB. Implement logic to downscale high-res frames (4K) if necessary to fit VRAM constraints during inference.

2.3 Batch Orchestration Skeleton

• Job Queue: Implement a simple priority queue or DB-based locking mechanism to assign files to Worker processes.
• State Management: Ensure atomic transitions from PENDING → PROCESSING to avoid duplicate processing.

Exit Criteria:

• New files in /data/input appear in DB with correct metadata within 60 seconds.
• Unsupported codec files are marked UNSCANNABLE without crashing the scanner.
• DB accurately reflects the current state of the 30TB corpus.

---

Phase 3: Inference Pipeline & Model Integration (Sprint 5-6)

Goal: Implement frame sampling, face detection, classification, and confidence aggregation.

3.1 Frame Sampling

• FFmpeg Extraction: Implement logic to extract 1 frame per 30 seconds uniformly.
  • Handle variable FPS gracefully.
  • Save frames to /scratch/tmp/ with naming convention {video_id}_{timestamp}.jpg.
• Memory Management: Ensure frames are deleted from scratch space immediately after processing the video to prevent filling NVMe.

3.2 Face Detection

• Model Integration: Load YOLOv8n (or chosen lightweight detector) via ONNX Runtime or TensorRT.
• Batching: Implement dynamic batching for face detection to maximize GPU utilization.
• Crop Generation: Extract face crops, resize to inference input size (e.g., 224x224).

3.3 Classification & Aggregation

• Model Integration: Load MobileNetV3 (or chosen classifier) via TensorRT FP32.
• Inference: Run classification on face crops.
• Aggregation Logic:
  • Calculate frame-level confidence p_i.
  • Apply video-level aggregation (e.g., max(p_i) or weighted mean).
  • Apply temperature scaling for calibration.
• Routing Decision: Compare final confidence C against T_high and T_low. Assign MATCH, REVIEW, or SKIP.

3.4 Result Persistence

• DB Update: Update videos table with processed_at, model_version, and status COMPLETED.
• Log Entry: Insert record into processing_logs.
• Export: Write summary JSON/Parquet file to /data/output for analytics.

Exit Criteria:

• A sample video is processed end-to-end: Frames extracted → Faces detected → Classified → Confidence calculated → DB updated.
• GPU memory usage stays under 18GB per GPU.
• Throughput meets baseline (>30 videos/hour/GPU on test set).

---

Phase 4: Active Learning & Review UI (Sprint 7-8)

Goal: Enable human-in-the-loop correction and model retraining.

4.1 Review UI

• UI Setup: Deploy Label Studio or custom Flask UI.
• Data Feed: Query DB for videos with routing_decision = REVIEW.
• Frontend: Display video player + top-k contributing frames. Allow annotator to toggle label (True/False) or correct classification.
• Export: Generate CSV/JSON export of annotated data with ground truth.

4.2 Fine-Tuning Pipeline

• Data Loader: Script to ingest exported annotations and prepare dataset for training.
• Training Job:
  • Head-only fine-tuning on face crops.
  • Use AdamW, LR=1e-3, Early Stopping.
  • Save checkpoint to /models/candidate/.
• Validation:
  • Run candidate model on held-out validation set.
  • Calculate F1 score and ECE (Expected Calibration Error).
  • If ΔF1 > 0.02 and ECE < 0.08, promote model to ACTIVE.

4.3 Model Swapping

• Registry Update: Update DB models table to mark new version as ACTIVE and old as ARCHIVED.
• Hot Reload: Restart Worker containers or signal process to reload new TensorRT engine.

Exit Criteria:

• Annotator can label a review video.
• Labeled data triggers a fine-tuning job.
• New model is validated and deployed automatically.

---

Phase 5: Observability, Monitoring & Hardening (Sprint 9+)

Goal: Ensure system reliability, debuggability, and long-term stability.

5.1 Metrics & Logging

• Prometheus Integration: Expose metrics:
  • videos_processed_total (by routing decision).
  • gpu_utilization, gpu_memory_used.
  • queue_depth (PENDING/PROCESSING counts).
  • confidence_distribution histogram.
• Grafana Dashboard: Create dashboards for:
  • Throughput (videos/hour).
  • Error rates (UNSCANNABLE/ERROR counts).
  • Confidence drift alerts.

5.2 Fault Tolerance & Resumption

• Crash Recovery: Implement checkpointing. If Worker dies, mark PROCESSING jobs as PENDING again.
• Idempotency: Ensure re-processing a file does not duplicate DB entries or outputs.
• Retry Logic: Auto-retry transient errors (e.g., network blip during NAS access) up to 3 times.

5.3 Drift Detection

• Weekly Job: Compare current confidence distribution to baseline.
• Alerting: Send alert if p(C > 0.5) shifts >10% or Review Queue grows unbounded.

Exit Criteria:

• Dashboard shows real-time health of the system.
• System recovers gracefully from forced Worker termination.
• Alerts trigger correctly on simulated drift/errors.

---

Risk Mitigation

Risk	Mitigation Strategy
GPU OOM	Implement strict batch size caps; downscale 4K frames to 1080p/720p before inference.
Slow NAS I/O	Cache frames in local NVMe tmpfs; minimize disk writes until final export.
Codec Variance	Extensive unit testing of ffprobe logic against known "bad" files. Graceful degradation to UNSCANNABLE.
Model Regression	Strict validation gate before model swap. Keep previous model in ARCHIVED state for quick rollback.
30TB Scan Time	Incremental scanning using file_hash and last_modified timestamps. Avoid full rescan.

---

Deliverables Checklist

• docker-compose.yml
• Worker Dockerfile & Requirements.txt
• MariaDB SQL Schema
• Python Source Code (src/)
  • scanner.py
  • processor.py
  • model_runner.py
  • db_handler.py
  • train.py
• Config Files (config.yaml)
• Grafana Dashboard JSON
• Documentation (README.md, API_DOCS.md)