After story 1

STORY-01.md

@@ -0,0 +1,170 @@
+# STORY-01: Foundation & Infrastructure
+
+## Epic
+**E6: Infrastructure** — As a DevOps engineer, I can deploy the entire stack via Docker Compose.
+
+## Related Requirements
+| ID | Requirement |
+|----|-------------|
+| TC-01 | Hardware: 2× Tesla P40 24GB (compute capability 5.2, PCIe 3.0, no Tensor Cores) |
+| TC-02 | CUDA/Torch Compatibility: CUDA ≤ 11.8, PyTorch ≤ 2.1.0, FP32 inference only |
+| TC-03 | Storage I/O: Fast local NVMe/SSD for temp frame cache; shared NAS/SMB for video input/output |
+| TC-04 | Framework Stack: PyTorch → ONNX → TensorRT FP32; FFmpeg/OpenCV; MariaDB |
+| TC-05 | Deployment Model: Docker Compose orchestrates all services; GPUs passed via nvidia-container-toolkit |
+| TC-06 | Network Security: Internal LAN only; no reverse proxy, SSL, or auth |
+| NFR-04 | Determinism & Reproducibility: Config-seeded randomness, versioned models |
+
+## Description
+Establish the Docker environment, database schema, and basic connectivity. No video processing logic yet — just the operational skeleton that all subsequent stories depend on.
+
+## Scope
+
+### In Scope
+- Docker Compose multi-service orchestration
+- Custom Worker Dockerfile with CUDA 11.8 + PyTorch 2.1.0 + TensorRT FP32
+- MariaDB schema design and migration scripts
+- Volume mounts for NVMe scratch, NAS input/output, and model persistence
+- Configuration management (config.yaml + environment variables)
+- Structured logging setup (JSON format to stdout)
+- Database connection layer with connection pooling
+- Network connectivity verification between all services
+
+### Out of Scope
+- Video processing logic (covered in STORY-02 through STORY-05)
+- Model loading or inference (covered in STORY-04)
+- Review UI functionality (covered in STORY-06)
+- Monitoring dashboards (covered in STORY-08)
+- Active learning pipeline (covered in STORY-07)
+
+## Deliverables
+
+### 1.1 Docker Compose Structure
+**File:** `docker-compose.yml`
+
+Services defined:
+- `mariadb`: MariaDB 10.11+ with persistent volume
+- `worker`: PyTorch/TensorRT inference worker with GPU passthrough
+- `ui`: Placeholder service (nginx serving static page) for network verification
+
+Key configurations:
+- `nvidia-container-toolkit` runtime configuration for GPU passthrough
+- Volume mounts:
+  - NVMe → `/scratch` (tmpfs for speed)
+  - NAS/SMB → `/data/input` and `/data/output`
+  - Persistent → `/models` and `/data/training`
+- Network bridge for inter-service communication
+- Resource limits (GPU memory caps per NFR-03)
+
+### 1.2 Worker Dockerfile
+**File:** `worker/Dockerfile`
+
+Base image: `nvidia/cuda:11.8.0-runtime-ubuntu22.04`
+
+Installed packages:
+- PyTorch 2.1.0 (CUDA 11.8, FP32 only)
+- TensorRT 8.6+ (FP32)
+- OpenCV 4.8+
+- FFmpeg 5.x + ffprobe
+- ONNX Runtime
+- Python 3.10+
+- Required system libraries (libcudnn8, libglib2.0, etc.)
+
+### 1.3 Database Schema
+**File:** `db/schema.sql`
+
+Tables:
+- `videos`: `id` (BIGINT PK), `file_path` (VARCHAR), `file_hash` (CHAR(64)), `resolution_w` (INT), `resolution_h` (INT), `codec` (VARCHAR), `duration` (FLOAT), `status` (ENUM: NEW, PENDING, PROCESSING, COMPLETED, UNSCANNABLE, ERROR), `last_scan_time` (DATETIME), `last_processed_time` (DATETIME), `created_at` (DATETIME), `updated_at` (DATETIME)
+- `processing_logs`: `id` (BIGINT PK), `video_id` (BIGINT FK), `model_version` (VARCHAR), `frame_count` (INT), `confidence_score` (FLOAT), `routing_decision` (ENUM: MATCH, REVIEW, SKIP), `processed_at` (DATETIME), `error_message` (TEXT)
+- `models`: `version` (VARCHAR PK), `status` (ENUM: ACTIVE, CANDIDATE, ARCHIVED), `path` (VARCHAR), `calibration_temp` (FLOAT), `f1_score` (FLOAT), `ece_score` (FLOAT), `deployed_at` (DATETIME), `created_at` (DATETIME)
+- `review_queue`: `id` (BIGINT PK), `video_id` (BIGINT FK), `confidence_score` (FLOAT), `routing_decision` (ENUM: REVIEW), `annotated` (BOOLEAN), `ground_truth` (BOOLEAN), `annotated_at` (DATETIME), `created_at` (DATETIME)
+
+Indexes:
+- `idx_videos_status` on `videos(status)`
+- `idx_videos_file_hash` on `videos(file_hash)` (UNIQUE)
+- `idx_videos_last_scan` on `videos(last_scan_time)`
+- `idx_processing_logs_video` on `processing_logs(video_id)`
+- `idx_models_status` on `models(status)`
+- `idx_review_queue_annotated` on `review_queue(annotated)`
+
+### 1.4 Database Connection Layer
+**File:** `src/db_connector.py`
+
+Features:
+- Connection pooling (DBUtils + PyMySQL)
+- Configurable pool size (min=5, max=20)
+- Automatic reconnection on disconnect
+- Context manager support
+- Prepared statements for all queries
+- Transaction support for atomic state transitions
+
+### 1.5 Configuration Management
+**File:** `config.yaml`
+
+Contents:
+- `sampling`: `interval_seconds: 30`, `override_per_job: true`
+- `thresholds`: `T_high: 0.75`, `T_low: 0.45`
+- `gpu`: `max_memory_gb: 18`, `batch_size: auto`, `device: cuda`
+- `storage`: `scratch_path: /scratch`, `input_path: /data/input`, `output_path: /data/output`
+- `database`: `host: mariadb`, `port: 3306`, `pool_size: 20`
+- `logging`: `format: json`, `level: INFO`
+- `model`: `face_detector: yolo8n`, `classifier: mobilenetv3`, `input_size: 224`
+
+**File:** `src/config_loader.py`
+
+Features:
+- Load config.yaml with environment variable overrides
+- Validate all required fields
+- Provide typed accessors (e.g., `config.thresholds.T_high`)
+- Hot-reload support for config changes
+
+### 1.6 Logging Setup
+**File:** `src/logging_config.py`
+
+- JSON structured logging via `python-json-logger`
+- Log levels: DEBUG, INFO, WARNING, ERROR, CRITICAL
+- Fields: timestamp, level, service, video_id, message, metadata (key-value pairs)
+- Log rotation: 100MB per file, 10 files max
+- All logs to stdout for Docker capture
+
+## Acceptance Criteria
+
+### Functional
+- [ ] `docker-compose up` starts MariaDB, Worker, and UI containers successfully
+- [ ] Worker container can connect to MariaDB and execute schema migrations
+- [ ] GPU is visible inside Worker container (`nvidia-smi` shows Tesla P40)
+- [ ] Volume mounts are accessible and writable in all containers
+- [ ] Config.yaml loads correctly with all required fields validated
+- [ ] Structured logging produces valid JSON output in all services
+- [ ] Database connection pool handles concurrent connections (test with 20 simultaneous)
+
+### Non-Functional
+- [ ] Worker container starts within 30 seconds
+- [ ] MariaDB starts within 15 seconds
+- [ ] GPU memory usage in Worker is < 2GB at idle (before model loading)
+- [ ] All services communicate over Docker internal network (no host network exposure except UI port)
+- [ ] Schema migration is idempotent (running twice produces same result)
+
+### Technical Constraints
+- [ ] CUDA version in Worker is 11.8 (verified via `torch.version.cuda`)
+- [ ] PyTorch version ≤ 2.1.0 (verified via `torch.__version__`)
+- [ ] TensorRT runs in FP32 mode only
+- [ ] No Tensor Cores used (compute capability 5.2 constraint respected)
+- [ ] No SSL, auth, or reverse proxy configured (TC-06)
+
+## Dependencies
+- **Prerequisites:** NVIDIA Container Toolkit installed on host, Docker Compose v2+, NAS/SMB mounts configured
+- **Depends on:** None (this is the foundational story)
+- **Enables:** STORY-02 (Ingestion), STORY-03 (Orchestration), STORY-04 (Inference), STORY-05 (Results), STORY-06 (Review UI), STORY-07 (Active Learning), STORY-08 (Monitoring)
+
+## Risks & Mitigations
+| Risk | Mitigation |
+|------|-----------|
+| Tesla P40 (CC 5.2) incompatible with newer TensorRT | Use TensorRT 8.6 which supports CC 5.x; test early |
+| CUDA 11.8 + PyTorch 2.1.0 dependency conflicts | Pin all versions in Dockerfile; use nvidia base image |
+| NAS/SMB mount latency affects processing | Use local tmpfs for scratch; only read from NAS |
+| MariaDB connection pool exhaustion | Monitor pool metrics; tune pool_size based on worker count |
+
+## Estimated Effort
+- **Sprint:** 1-2
+- **Story Points:** 13
+- **Dependencies:** None