After story 1

2026-08-03 11:30:49 -04:00
commit 82590c392f
32 changed files with 4375 additions and 0 deletions
@@ -0,0 +1,168 @@
+# VideoDetect - Video Classification System
+
+A production-grade video classification system for processing large-scale video corpora (~30TB) to classify videos based on demographic presence using deep learning models.
+
+## Architecture
+
+```
+┌─────────────┐     ┌──────────────┐     ┌─────────────┐
+│  Scanner    │────▶│  Processor   │────▶│  Results    │
+│  (STORY-02) │     │  (STORY-03/04)│     │  (STORY-05) │
+└─────────────┘     └──────────────┘     └─────────────┘
+       │                    │                    │
+       ▼                    ▼                    ▼
+┌─────────────┐     ┌──────────────┐     ┌─────────────┐
+│  MariaDB    │◀───▶│  GPU (P40)   │◀───▶│  Export     │
+│  (Metadata) │     │  (Inference) │     │  (Parquet)  │
+└─────────────┘     └──────────────┘     └─────────────┘
+       │
+       ▼
+┌─────────────┐     ┌──────────────┐
+│  Review UI  │◀───▶│ Active Learn │
+│  (STORY-06) │     │  (STORY-07)  │
+└─────────────┘     └──────────────┘
+       │
+       ▼
+┌─────────────┐
+│ Monitoring  │
+│ (STORY-08)  │
+└─────────────┘
+```
+
+## Quick Start
+
+### Prerequisites
+- Docker & Docker Compose v2+
+- NVIDIA Container Toolkit
+- 2× Tesla P40 GPUs (or compatible)
+- NAS/SMB mount at `/mnt/nas` (configurable via env vars)
+
+### Environment Variables
+
+```bash
+# Database
+export DB_ROOT_PASSWORD=your_root_password
+export DB_PASSWORD=your_db_password
+
+# Paths (optional, defaults shown)
+export NAS_INPUT_PATH=/mnt/nas/input
+export NAS_OUTPUT_PATH=/mnt/nas/output
+export MODELS_PATH=/mnt/nas/models
+export TRAINING_PATH=/mnt/nas/training
+
+# Grafana admin password
+export GRAFANA_PASSWORD=your_grafana_password
+```
+
+### Start the Stack
+
+```bash
+docker-compose up -d
+```
+
+### Verify Services
+
+```bash
+# Check all containers are running
+docker-compose ps
+
+# Check GPU visibility in worker
+docker exec videodetect-worker nvidia-smi
+
+# Check database connectivity
+docker exec videodetect-mariadb mysql -u videodetect -p videodetect -e "SHOW TABLES;"
+
+# Access UI
+open http://localhost:5000
+
+# Access Grafana
+open http://localhost:3000  (admin / your_grafana_password)
+
+# Access Prometheus
+open http://localhost:9090
+```
+
+## Project Structure
+
+```
+VideoDetect/
+├── docker-compose.yml          # Multi-service orchestration
+├── config.yaml                 # All configuration
+├── db/
+│   └── schema.sql              # MariaDB schema
+├── worker/
+│   ├── Dockerfile              # Worker container image
+│   └── requirements.txt        # Python dependencies
+├── ui/
+│   ├── Dockerfile              # UI container image
+│   └── review/                 # Review UI templates
+├── src/
+│   ├── main.py                 # Worker entry point
+│   ├── db_connector.py         # Database connection layer
+│   ├── config_loader.py        # Configuration management
+│   ├── logging_config.py       # Logging setup
+│   ├── scanner.py              # Directory scanner (STORY-02)
+│   ├── prober.py               # Video probing (STORY-02)
+│   ├── frame_sampler.py        # Frame extraction (STORY-03)
+│   ├── face_detector.py        # Face detection (STORY-04)
+│   ├── classifier.py           # Classification (STORY-05)
+│   ├── aggregator.py           # Confidence aggregation (STORY-05)
+│   ├── router.py               # Threshold routing (STORY-05)
+│   ├── result_updater.py       # Result persistence (STORY-06)
+│   ├── data_export.py          # Parquet/JSONL export (STORY-06)
+│   ├── review_api.py           # Review API (STORY-07)
+│   ├── active_learning/        # Active learning pipeline (STORY-08)
+│   │   ├── label_ingestor.py
+│   │   ├── trainer.py
+│   │   ├── validator.py
+│   │   └── registry.py
+│   ├── metrics.py              # Prometheus metrics (STORY-09)
+│   ├── crash_recovery.py       # Crash recovery (STORY-09)
+│   └── drift_detector.py       # Drift detection (STORY-09)
+├── monitoring/
+│   ├── prometheus/
+│   │   └── prometheus.yml      # Prometheus config
+│   └── grafana/
+│       ├── dashboards/         # Grafana dashboard JSONs
+│       └── provisioning/       # Grafana data source config
+├── STORY-01.md through STORY-09.md  # Implementation stories
+├── Plan.md                     # Implementation plan
+└── Requirements.md             # Requirements document
+```
+
+## Implementation Stories
+
+| Story | Description | Sprint | Points |
+|-------|-------------|--------|--------|
+| STORY-01 | Foundation & Infrastructure | 1-2 | 13 |
+| STORY-02 | Core Ingestion & Codec Handling | 3-4 | 21 |
+| STORY-03 | Frame Sampling | 5 | 13 |
+| STORY-04 | Face Detection | 5-6 | 21 |
+| STORY-05 | Classification & Confidence | 5-6 | 21 |
+| STORY-06 | Results Persistence & Export | 5-6 | 13 |
+| STORY-07 | Review Interface | 7 | 21 |
+| STORY-08 | Active Learning Pipeline | 7-8 | 34 |
+| STORY-09 | Observability & Hardening | 9+ | 34 |
+
+## Hardware Requirements
+
+- **GPUs:** 2× Tesla P40 24GB (compute capability 5.2)
+- **CUDA:** ≤ 11.8
+- **PyTorch:** ≤ 2.1.0
+- **Inference:** FP32 only (no Tensor Cores)
+- **Storage:** NVMe/SSD for scratch, NAS/SMB for input/output
+- **RAM:** ≥ 32GB system RAM
+
+## Key Design Decisions
+
+1. **TensorRT FP32 only** — Tesla P40 has no Tensor Cores; FP16 would not provide benefit
+2. **YOLOv8n + MobileNetV3** — Lightweight models optimized for throughput over accuracy
+3. **Max aggregation** — Most conservative confidence strategy; use highest frame confidence
+4. **Temperature scaling** — Calibrates confidence scores without retraining
+5. **Head-only fine-tuning** — Faster retraining with frozen backbone
+6. **Parquet export** — Efficient columnar format for analytics
+7. **No auth/SSL** — Per TC-06, internal LAN only
+
+## License
+
+Proprietary — VideoDetect Project