# VideoDetect - Video Classification System

A production-grade video classification system for processing large-scale video corpora (~30TB) to classify videos based on demographic presence using deep learning models.

## Architecture

```
┌─────────────┐     ┌──────────────┐     ┌─────────────┐
│  Scanner    │────▶│  Processor   │────▶│  Results    │
│  (STORY-02) │     │  (STORY-03/04)│     │  (STORY-05) │
└─────────────┘     └──────────────┘     └─────────────┘
       │                    │                    │
       ▼                    ▼                    ▼
┌─────────────┐     ┌──────────────┐     ┌─────────────┐
│  MariaDB    │◀───▶│  GPU (P40)   │◀───▶│  Export     │
│  (Metadata) │     │  (Inference) │     │  (Parquet)  │
└─────────────┘     └──────────────┘     └─────────────┘
       │
       ▼
┌─────────────┐     ┌──────────────┐
│  Review UI  │◀───▶│ Active Learn │
│  (STORY-06) │     │  (STORY-07)  │
└─────────────┘     └──────────────┘
       │
       ▼
┌─────────────┐
│ Monitoring  │
│ (STORY-08)  │
└─────────────┘
```

## Quick Start

### Prerequisites
- Docker & Docker Compose v2+
- NVIDIA Container Toolkit
- 2× Tesla P40 GPUs (or compatible)
- NAS/SMB mount at `/mnt/nas` (configurable via env vars)

### Environment Variables

```bash
# Database
export DB_ROOT_PASSWORD=your_root_password
export DB_PASSWORD=your_db_password

# Paths (optional, defaults shown)
export NAS_INPUT_PATH=/mnt/nas/input
export NAS_OUTPUT_PATH=/mnt/nas/output
export MODELS_PATH=/mnt/nas/models
export TRAINING_PATH=/mnt/nas/training

# Grafana admin password
export GRAFANA_PASSWORD=your_grafana_password
```

### Start the Stack

```bash
docker-compose up -d
```

### Verify Services

```bash
# Check all containers are running
docker-compose ps

# Check GPU visibility in worker
docker exec videodetect-worker nvidia-smi

# Check database connectivity
docker exec videodetect-mariadb mysql -u videodetect -p videodetect -e "SHOW TABLES;"

# Access UI
open http://localhost:5000

# Access Grafana
open http://localhost:3000  (admin / your_grafana_password)

# Access Prometheus
open http://localhost:9090
```

## Project Structure

```
VideoDetect/
├── docker-compose.yml          # Multi-service orchestration
├── config.yaml                 # All configuration
├── db/
│   └── schema.sql              # MariaDB schema
├── worker/
│   ├── Dockerfile              # Worker container image
│   └── requirements.txt        # Python dependencies
├── ui/
│   ├── Dockerfile              # UI container image
│   └── review/                 # Review UI templates
├── src/
│   ├── main.py                 # Worker entry point
│   ├── db_connector.py         # Database connection layer
│   ├── config_loader.py        # Configuration management
│   ├── logging_config.py       # Logging setup
│   ├── scanner.py              # Directory scanner (STORY-02)
│   ├── prober.py               # Video probing (STORY-02)
│   ├── frame_sampler.py        # Frame extraction (STORY-03)
│   ├── face_detector.py        # Face detection (STORY-04)
│   ├── classifier.py           # Classification (STORY-05)
│   ├── aggregator.py           # Confidence aggregation (STORY-05)
│   ├── router.py               # Threshold routing (STORY-05)
│   ├── result_updater.py       # Result persistence (STORY-06)
│   ├── data_export.py          # Parquet/JSONL export (STORY-06)
│   ├── review_api.py           # Review API (STORY-07)
│   ├── active_learning/        # Active learning pipeline (STORY-08)
│   │   ├── label_ingestor.py
│   │   ├── trainer.py
│   │   ├── validator.py
│   │   └── registry.py
│   ├── metrics.py              # Prometheus metrics (STORY-09)
│   ├── crash_recovery.py       # Crash recovery (STORY-09)
│   └── drift_detector.py       # Drift detection (STORY-09)
├── monitoring/
│   ├── prometheus/
│   │   └── prometheus.yml      # Prometheus config
│   └── grafana/
│       ├── dashboards/         # Grafana dashboard JSONs
│       └── provisioning/       # Grafana data source config
├── STORY-01.md through STORY-09.md  # Implementation stories
├── Plan.md                     # Implementation plan
└── Requirements.md             # Requirements document
```

## Implementation Stories

| Story | Description | Sprint | Points |
|-------|-------------|--------|--------|
| STORY-01 | Foundation & Infrastructure | 1-2 | 13 |
| STORY-02 | Core Ingestion & Codec Handling | 3-4 | 21 |
| STORY-03 | Frame Sampling | 5 | 13 |
| STORY-04 | Face Detection | 5-6 | 21 |
| STORY-05 | Classification & Confidence | 5-6 | 21 |
| STORY-06 | Results Persistence & Export | 5-6 | 13 |
| STORY-07 | Review Interface | 7 | 21 |
| STORY-08 | Active Learning Pipeline | 7-8 | 34 |
| STORY-09 | Observability & Hardening | 9+ | 34 |

## Hardware Requirements

- **GPUs:** 2× Tesla P40 24GB (compute capability 5.2)
- **CUDA:** ≤ 11.8
- **PyTorch:** ≤ 2.1.0
- **Inference:** FP32 only (no Tensor Cores)
- **Storage:** NVMe/SSD for scratch, NAS/SMB for input/output
- **RAM:** ≥ 32GB system RAM

## Key Design Decisions

1. **TensorRT FP32 only** — Tesla P40 has no Tensor Cores; FP16 would not provide benefit
2. **YOLOv8n + MobileNetV3** — Lightweight models optimized for throughput over accuracy
3. **Max aggregation** — Most conservative confidence strategy; use highest frame confidence
4. **Temperature scaling** — Calibrates confidence scores without retraining
5. **Head-only fine-tuning** — Faster retraining with frozen backbone
6. **Parquet export** — Efficient columnar format for analytics
7. **No auth/SSL** — Per TC-06, internal LAN only

## License

Proprietary — VideoDetect Project