After story 1
This commit is contained in:
+70
@@ -0,0 +1,70 @@
|
|||||||
|
# VideoDetect .gitignore
|
||||||
|
|
||||||
|
# Python
|
||||||
|
__pycache__/
|
||||||
|
*.py[cod]
|
||||||
|
*$py.class
|
||||||
|
*.so
|
||||||
|
*.egg
|
||||||
|
*.egg-info/
|
||||||
|
dist/
|
||||||
|
build/
|
||||||
|
.eggs/
|
||||||
|
*.whl
|
||||||
|
|
||||||
|
# Virtual environments
|
||||||
|
venv/
|
||||||
|
env/
|
||||||
|
.venv/
|
||||||
|
|
||||||
|
# IDE
|
||||||
|
.vscode/
|
||||||
|
.idea/
|
||||||
|
*.swp
|
||||||
|
*.swo
|
||||||
|
*~
|
||||||
|
|
||||||
|
# Docker
|
||||||
|
.docker/
|
||||||
|
|
||||||
|
# Environment
|
||||||
|
.env
|
||||||
|
.env.*
|
||||||
|
|
||||||
|
# OS
|
||||||
|
.DS_Store
|
||||||
|
Thumbs.db
|
||||||
|
|
||||||
|
# Logs
|
||||||
|
*.log
|
||||||
|
logs/
|
||||||
|
|
||||||
|
# Data (large files)
|
||||||
|
data/
|
||||||
|
*.mp4
|
||||||
|
*.mov
|
||||||
|
*.avi
|
||||||
|
*.mkv
|
||||||
|
|
||||||
|
# Models (large files)
|
||||||
|
models/*.pt
|
||||||
|
models/*.onnx
|
||||||
|
!models/.gitkeep
|
||||||
|
|
||||||
|
# Scratch
|
||||||
|
scratch/
|
||||||
|
*.tmp
|
||||||
|
*.temp
|
||||||
|
|
||||||
|
# Testing
|
||||||
|
.pytest_cache/
|
||||||
|
.coverage
|
||||||
|
htmlcov/
|
||||||
|
|
||||||
|
# Monitoring
|
||||||
|
prometheus/data/
|
||||||
|
grafana/data/
|
||||||
|
|
||||||
|
# OS
|
||||||
|
.DS_Store
|
||||||
|
Thumbs.db
|
||||||
@@ -0,0 +1,196 @@
|
|||||||
|
# Implementation Plan: Video Classification System
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
This document outlines the phased implementation plan for the Video Classification System. The plan breaks down the requirements into logical work packages (Sprints/Phases) that can be implemented, tested, and integrated sequentially.
|
||||||
|
|
||||||
|
**Key Dependencies:**
|
||||||
|
1. **Infrastructure** must be available before any processing logic can run.
|
||||||
|
2. **Database Schema** must be finalized early to support metadata logging.
|
||||||
|
3. **Model Training Data** is a bottleneck; the initial model must be ready or a placeholder used for pipeline testing.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Phase 1: Foundation & Infrastructure (Sprint 1-2)
|
||||||
|
**Goal:** Establish the Docker environment, Database schema, and basic connectivity. No video processing yet, just the "skeleton."
|
||||||
|
|
||||||
|
### 1.1 Environment Setup
|
||||||
|
- [ ] **Docker Compose Structure:** Create `docker-compose.yml` defining services for:
|
||||||
|
- `mariadb`: Persistent volume for metadata.
|
||||||
|
- `worker`: Build context for PyTorch/TensorRT environment. Include `nvidia-container-toolkit` config for GPU passthrough.
|
||||||
|
- `ui`: Placeholder container (e.g., nginx serving a static "Under Construction" page) to verify network connectivity.
|
||||||
|
- [ ] **Base Images:**
|
||||||
|
- Create a custom Dockerfile for the worker based on `nvidia/cuda:11.8.0-runtime-ubuntu20.04`.
|
||||||
|
- Install PyTorch (1.13+ with CUDA 11.8), OpenCV, FFmpeg-python, and TensorRT prerequisites.
|
||||||
|
- [ ] **Volume Mounts:**
|
||||||
|
- Map local NVMe path to `/scratch` in worker container (tmpfs preferred for speed).
|
||||||
|
- Map NAS/SMB path to `/data/input` and `/data/output` in worker container.
|
||||||
|
- Map `/models` and `/data/training` for persistence.
|
||||||
|
|
||||||
|
### 1.2 Database Schema Design
|
||||||
|
- [ ] **MariaDB Schema:** Define tables in SQL migration script:
|
||||||
|
- `videos`: `id`, `file_path`, `file_hash`, `resolution_w`, `resolution_h`, `codec`, `status` (NEW, PENDING, PROCESSING, COMPLETED, UNSCANNABLE, ERROR), `last_scan_time`, `last_processed_time`.
|
||||||
|
- `processing_logs`: `id`, `video_id`, `model_version`, `frame_count`, `confidence_score`, `routing_decision`, `processed_at`.
|
||||||
|
- `models`: `version`, `status` (ACTIVE, CANDIDATE, ARCHIVED), `path`, `calibration_temp`.
|
||||||
|
- [ ] **Connection Layer:** Implement a Python module (`db_connector.py`) with connection pooling to MariaDB.
|
||||||
|
|
||||||
|
### 1.3 Configuration Management
|
||||||
|
- [ ] **Config File:** Create `config.yaml` for thresholds (`T_high`, `T_low`), sampling intervals, and GPU settings.
|
||||||
|
- [ ] **Logging Setup:** Configure structured logging (JSON format) to stdout for Docker capture.
|
||||||
|
|
||||||
|
**Exit Criteria:**
|
||||||
|
- `docker-compose up` starts DB and Worker containers successfully.
|
||||||
|
- Worker can connect to DB and create tables.
|
||||||
|
- GPU is visible inside the Worker container (`nvidia-smi`).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Phase 2: Core Ingestion & Codec Handling (Sprint 3-4)
|
||||||
|
**Goal:** Implement the directory scanner, video probing, and robust error handling for codecs/resolutions.
|
||||||
|
|
||||||
|
### 2.1 Directory Scanner
|
||||||
|
- [ ] **Scanner Service:** Implement a background thread or separate script that walks `/data/input`.
|
||||||
|
- [ ] **File Detection:** Identify new files (not in DB) and update status to `PENDING`.
|
||||||
|
- [ ] **Deduplication:** Use `file_hash` (SHA-256 of first 1MB or full file if small) to prevent re-processing identical files.
|
||||||
|
|
||||||
|
### 2.2 Video Probing & Metadata Extraction
|
||||||
|
- [ ] **FFmpeg Probe:** For each `PENDING` file, run `ffprobe` to extract:
|
||||||
|
- Codec name.
|
||||||
|
- Resolution (width/height).
|
||||||
|
- Duration.
|
||||||
|
- [ ] **Codec Validation:** Maintain a whitelist of supported codecs (H.264, H.265, VP8, VP9, etc.).
|
||||||
|
- If codec is unsupported or `ffprobe` fails: Set status to `UNSCANNABLE`.
|
||||||
|
- Log the specific error code/reason.
|
||||||
|
- [ ] **Resolution Handling:** Store resolution in DB. Implement logic to downscale high-res frames (4K) if necessary to fit VRAM constraints during inference.
|
||||||
|
|
||||||
|
### 2.3 Batch Orchestration Skeleton
|
||||||
|
- [ ] **Job Queue:** Implement a simple priority queue or DB-based locking mechanism to assign files to Worker processes.
|
||||||
|
- [ ] **State Management:** Ensure atomic transitions from `PENDING` → `PROCESSING` to avoid duplicate processing.
|
||||||
|
|
||||||
|
**Exit Criteria:**
|
||||||
|
- New files in `/data/input` appear in DB with correct metadata within 60 seconds.
|
||||||
|
- Unsupported codec files are marked `UNSCANNABLE` without crashing the scanner.
|
||||||
|
- DB accurately reflects the current state of the 30TB corpus.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Phase 3: Inference Pipeline & Model Integration (Sprint 5-6)
|
||||||
|
**Goal:** Implement frame sampling, face detection, classification, and confidence aggregation.
|
||||||
|
|
||||||
|
### 3.1 Frame Sampling
|
||||||
|
- [ ] **FFmpeg Extraction:** Implement logic to extract 1 frame per 30 seconds uniformly.
|
||||||
|
- Handle variable FPS gracefully.
|
||||||
|
- Save frames to `/scratch/tmp/` with naming convention `{video_id}_{timestamp}.jpg`.
|
||||||
|
- [ ] **Memory Management:** Ensure frames are deleted from scratch space immediately after processing the video to prevent filling NVMe.
|
||||||
|
|
||||||
|
### 3.2 Face Detection
|
||||||
|
- [ ] **Model Integration:** Load YOLOv8n (or chosen lightweight detector) via ONNX Runtime or TensorRT.
|
||||||
|
- [ ] **Batching:** Implement dynamic batching for face detection to maximize GPU utilization.
|
||||||
|
- [ ] **Crop Generation:** Extract face crops, resize to inference input size (e.g., 224x224).
|
||||||
|
|
||||||
|
### 3.3 Classification & Aggregation
|
||||||
|
- [ ] **Model Integration:** Load MobileNetV3 (or chosen classifier) via TensorRT FP32.
|
||||||
|
- [ ] **Inference:** Run classification on face crops.
|
||||||
|
- [ ] **Aggregation Logic:**
|
||||||
|
- Calculate frame-level confidence `p_i`.
|
||||||
|
- Apply video-level aggregation (e.g., `max(p_i)` or weighted mean).
|
||||||
|
- Apply temperature scaling for calibration.
|
||||||
|
- [ ] **Routing Decision:** Compare final confidence `C` against `T_high` and `T_low`. Assign `MATCH`, `REVIEW`, or `SKIP`.
|
||||||
|
|
||||||
|
### 3.4 Result Persistence
|
||||||
|
- [ ] **DB Update:** Update `videos` table with `processed_at`, `model_version`, and status `COMPLETED`.
|
||||||
|
- [ ] **Log Entry:** Insert record into `processing_logs`.
|
||||||
|
- [ ] **Export:** Write summary JSON/Parquet file to `/data/output` for analytics.
|
||||||
|
|
||||||
|
**Exit Criteria:**
|
||||||
|
- A sample video is processed end-to-end: Frames extracted → Faces detected → Classified → Confidence calculated → DB updated.
|
||||||
|
- GPU memory usage stays under 18GB per GPU.
|
||||||
|
- Throughput meets baseline (>30 videos/hour/GPU on test set).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Phase 4: Active Learning & Review UI (Sprint 7-8)
|
||||||
|
**Goal:** Enable human-in-the-loop correction and model retraining.
|
||||||
|
|
||||||
|
### 4.1 Review UI
|
||||||
|
- [ ] **UI Setup:** Deploy Label Studio or custom Flask UI.
|
||||||
|
- [ ] **Data Feed:** Query DB for videos with routing_decision = `REVIEW`.
|
||||||
|
- [ ] **Frontend:** Display video player + top-k contributing frames. Allow annotator to toggle label (True/False) or correct classification.
|
||||||
|
- [ ] **Export:** Generate CSV/JSON export of annotated data with ground truth.
|
||||||
|
|
||||||
|
### 4.2 Fine-Tuning Pipeline
|
||||||
|
- [ ] **Data Loader:** Script to ingest exported annotations and prepare dataset for training.
|
||||||
|
- [ ] **Training Job:**
|
||||||
|
- Head-only fine-tuning on face crops.
|
||||||
|
- Use AdamW, LR=1e-3, Early Stopping.
|
||||||
|
- Save checkpoint to `/models/candidate/`.
|
||||||
|
- [ ] **Validation:**
|
||||||
|
- Run candidate model on held-out validation set.
|
||||||
|
- Calculate F1 score and ECE (Expected Calibration Error).
|
||||||
|
- If ΔF1 > 0.02 and ECE < 0.08, promote model to `ACTIVE`.
|
||||||
|
|
||||||
|
### 4.3 Model Swapping
|
||||||
|
- [ ] **Registry Update:** Update DB `models` table to mark new version as `ACTIVE` and old as `ARCHIVED`.
|
||||||
|
- [ ] **Hot Reload:** Restart Worker containers or signal process to reload new TensorRT engine.
|
||||||
|
|
||||||
|
**Exit Criteria:**
|
||||||
|
- Annotator can label a review video.
|
||||||
|
- Labeled data triggers a fine-tuning job.
|
||||||
|
- New model is validated and deployed automatically.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Phase 5: Observability, Monitoring & Hardening (Sprint 9+)
|
||||||
|
**Goal:** Ensure system reliability, debuggability, and long-term stability.
|
||||||
|
|
||||||
|
### 5.1 Metrics & Logging
|
||||||
|
- [ ] **Prometheus Integration:** Expose metrics:
|
||||||
|
- `videos_processed_total` (by routing decision).
|
||||||
|
- `gpu_utilization`, `gpu_memory_used`.
|
||||||
|
- `queue_depth` (PENDING/PROCESSING counts).
|
||||||
|
- `confidence_distribution` histogram.
|
||||||
|
- [ ] **Grafana Dashboard:** Create dashboards for:
|
||||||
|
- Throughput (videos/hour).
|
||||||
|
- Error rates (UNSCANNABLE/ERROR counts).
|
||||||
|
- Confidence drift alerts.
|
||||||
|
|
||||||
|
### 5.2 Fault Tolerance & Resumption
|
||||||
|
- [ ] **Crash Recovery:** Implement checkpointing. If Worker dies, mark `PROCESSING` jobs as `PENDING` again.
|
||||||
|
- [ ] **Idempotency:** Ensure re-processing a file does not duplicate DB entries or outputs.
|
||||||
|
- [ ] **Retry Logic:** Auto-retry transient errors (e.g., network blip during NAS access) up to 3 times.
|
||||||
|
|
||||||
|
### 5.3 Drift Detection
|
||||||
|
- [ ] **Weekly Job:** Compare current confidence distribution to baseline.
|
||||||
|
- [ ] **Alerting:** Send alert if `p(C > 0.5)` shifts >10% or Review Queue grows unbounded.
|
||||||
|
|
||||||
|
**Exit Criteria:**
|
||||||
|
- Dashboard shows real-time health of the system.
|
||||||
|
- System recovers gracefully from forced Worker termination.
|
||||||
|
- Alerts trigger correctly on simulated drift/errors.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Risk Mitigation
|
||||||
|
|
||||||
|
| Risk | Mitigation Strategy |
|
||||||
|
|------|---------------------|
|
||||||
|
| **GPU OOM** | Implement strict batch size caps; downscale 4K frames to 1080p/720p before inference. |
|
||||||
|
| **Slow NAS I/O** | Cache frames in local NVMe `tmpfs`; minimize disk writes until final export. |
|
||||||
|
| **Codec Variance** | Extensive unit testing of `ffprobe` logic against known "bad" files. Graceful degradation to `UNSCANNABLE`. |
|
||||||
|
| **Model Regression** | Strict validation gate before model swap. Keep previous model in `ARCHIVED` state for quick rollback. |
|
||||||
|
| **30TB Scan Time** | Incremental scanning using `file_hash` and `last_modified` timestamps. Avoid full rescan. |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Deliverables Checklist
|
||||||
|
- [ ] `docker-compose.yml`
|
||||||
|
- [ ] Worker Dockerfile & Requirements.txt
|
||||||
|
- [ ] MariaDB SQL Schema
|
||||||
|
- [ ] Python Source Code (`src/`)
|
||||||
|
- `scanner.py`
|
||||||
|
- `processor.py`
|
||||||
|
- `model_runner.py`
|
||||||
|
- `db_handler.py`
|
||||||
|
- `train.py`
|
||||||
|
- [ ] Config Files (`config.yaml`)
|
||||||
|
- [ ] Grafana Dashboard JSON
|
||||||
|
- [ ] Documentation (`README.md`, `API_DOCS.md`)
|
||||||
@@ -0,0 +1,168 @@
|
|||||||
|
# VideoDetect - Video Classification System
|
||||||
|
|
||||||
|
A production-grade video classification system for processing large-scale video corpora (~30TB) to classify videos based on demographic presence using deep learning models.
|
||||||
|
|
||||||
|
## Architecture
|
||||||
|
|
||||||
|
```
|
||||||
|
┌─────────────┐ ┌──────────────┐ ┌─────────────┐
|
||||||
|
│ Scanner │────▶│ Processor │────▶│ Results │
|
||||||
|
│ (STORY-02) │ │ (STORY-03/04)│ │ (STORY-05) │
|
||||||
|
└─────────────┘ └──────────────┘ └─────────────┘
|
||||||
|
│ │ │
|
||||||
|
▼ ▼ ▼
|
||||||
|
┌─────────────┐ ┌──────────────┐ ┌─────────────┐
|
||||||
|
│ MariaDB │◀───▶│ GPU (P40) │◀───▶│ Export │
|
||||||
|
│ (Metadata) │ │ (Inference) │ │ (Parquet) │
|
||||||
|
└─────────────┘ └──────────────┘ └─────────────┘
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
┌─────────────┐ ┌──────────────┐
|
||||||
|
│ Review UI │◀───▶│ Active Learn │
|
||||||
|
│ (STORY-06) │ │ (STORY-07) │
|
||||||
|
└─────────────┘ └──────────────┘
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
┌─────────────┐
|
||||||
|
│ Monitoring │
|
||||||
|
│ (STORY-08) │
|
||||||
|
└─────────────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
## Quick Start
|
||||||
|
|
||||||
|
### Prerequisites
|
||||||
|
- Docker & Docker Compose v2+
|
||||||
|
- NVIDIA Container Toolkit
|
||||||
|
- 2× Tesla P40 GPUs (or compatible)
|
||||||
|
- NAS/SMB mount at `/mnt/nas` (configurable via env vars)
|
||||||
|
|
||||||
|
### Environment Variables
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Database
|
||||||
|
export DB_ROOT_PASSWORD=your_root_password
|
||||||
|
export DB_PASSWORD=your_db_password
|
||||||
|
|
||||||
|
# Paths (optional, defaults shown)
|
||||||
|
export NAS_INPUT_PATH=/mnt/nas/input
|
||||||
|
export NAS_OUTPUT_PATH=/mnt/nas/output
|
||||||
|
export MODELS_PATH=/mnt/nas/models
|
||||||
|
export TRAINING_PATH=/mnt/nas/training
|
||||||
|
|
||||||
|
# Grafana admin password
|
||||||
|
export GRAFANA_PASSWORD=your_grafana_password
|
||||||
|
```
|
||||||
|
|
||||||
|
### Start the Stack
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docker-compose up -d
|
||||||
|
```
|
||||||
|
|
||||||
|
### Verify Services
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Check all containers are running
|
||||||
|
docker-compose ps
|
||||||
|
|
||||||
|
# Check GPU visibility in worker
|
||||||
|
docker exec videodetect-worker nvidia-smi
|
||||||
|
|
||||||
|
# Check database connectivity
|
||||||
|
docker exec videodetect-mariadb mysql -u videodetect -p videodetect -e "SHOW TABLES;"
|
||||||
|
|
||||||
|
# Access UI
|
||||||
|
open http://localhost:5000
|
||||||
|
|
||||||
|
# Access Grafana
|
||||||
|
open http://localhost:3000 (admin / your_grafana_password)
|
||||||
|
|
||||||
|
# Access Prometheus
|
||||||
|
open http://localhost:9090
|
||||||
|
```
|
||||||
|
|
||||||
|
## Project Structure
|
||||||
|
|
||||||
|
```
|
||||||
|
VideoDetect/
|
||||||
|
├── docker-compose.yml # Multi-service orchestration
|
||||||
|
├── config.yaml # All configuration
|
||||||
|
├── db/
|
||||||
|
│ └── schema.sql # MariaDB schema
|
||||||
|
├── worker/
|
||||||
|
│ ├── Dockerfile # Worker container image
|
||||||
|
│ └── requirements.txt # Python dependencies
|
||||||
|
├── ui/
|
||||||
|
│ ├── Dockerfile # UI container image
|
||||||
|
│ └── review/ # Review UI templates
|
||||||
|
├── src/
|
||||||
|
│ ├── main.py # Worker entry point
|
||||||
|
│ ├── db_connector.py # Database connection layer
|
||||||
|
│ ├── config_loader.py # Configuration management
|
||||||
|
│ ├── logging_config.py # Logging setup
|
||||||
|
│ ├── scanner.py # Directory scanner (STORY-02)
|
||||||
|
│ ├── prober.py # Video probing (STORY-02)
|
||||||
|
│ ├── frame_sampler.py # Frame extraction (STORY-03)
|
||||||
|
│ ├── face_detector.py # Face detection (STORY-04)
|
||||||
|
│ ├── classifier.py # Classification (STORY-05)
|
||||||
|
│ ├── aggregator.py # Confidence aggregation (STORY-05)
|
||||||
|
│ ├── router.py # Threshold routing (STORY-05)
|
||||||
|
│ ├── result_updater.py # Result persistence (STORY-06)
|
||||||
|
│ ├── data_export.py # Parquet/JSONL export (STORY-06)
|
||||||
|
│ ├── review_api.py # Review API (STORY-07)
|
||||||
|
│ ├── active_learning/ # Active learning pipeline (STORY-08)
|
||||||
|
│ │ ├── label_ingestor.py
|
||||||
|
│ │ ├── trainer.py
|
||||||
|
│ │ ├── validator.py
|
||||||
|
│ │ └── registry.py
|
||||||
|
│ ├── metrics.py # Prometheus metrics (STORY-09)
|
||||||
|
│ ├── crash_recovery.py # Crash recovery (STORY-09)
|
||||||
|
│ └── drift_detector.py # Drift detection (STORY-09)
|
||||||
|
├── monitoring/
|
||||||
|
│ ├── prometheus/
|
||||||
|
│ │ └── prometheus.yml # Prometheus config
|
||||||
|
│ └── grafana/
|
||||||
|
│ ├── dashboards/ # Grafana dashboard JSONs
|
||||||
|
│ └── provisioning/ # Grafana data source config
|
||||||
|
├── STORY-01.md through STORY-09.md # Implementation stories
|
||||||
|
├── Plan.md # Implementation plan
|
||||||
|
└── Requirements.md # Requirements document
|
||||||
|
```
|
||||||
|
|
||||||
|
## Implementation Stories
|
||||||
|
|
||||||
|
| Story | Description | Sprint | Points |
|
||||||
|
|-------|-------------|--------|--------|
|
||||||
|
| STORY-01 | Foundation & Infrastructure | 1-2 | 13 |
|
||||||
|
| STORY-02 | Core Ingestion & Codec Handling | 3-4 | 21 |
|
||||||
|
| STORY-03 | Frame Sampling | 5 | 13 |
|
||||||
|
| STORY-04 | Face Detection | 5-6 | 21 |
|
||||||
|
| STORY-05 | Classification & Confidence | 5-6 | 21 |
|
||||||
|
| STORY-06 | Results Persistence & Export | 5-6 | 13 |
|
||||||
|
| STORY-07 | Review Interface | 7 | 21 |
|
||||||
|
| STORY-08 | Active Learning Pipeline | 7-8 | 34 |
|
||||||
|
| STORY-09 | Observability & Hardening | 9+ | 34 |
|
||||||
|
|
||||||
|
## Hardware Requirements
|
||||||
|
|
||||||
|
- **GPUs:** 2× Tesla P40 24GB (compute capability 5.2)
|
||||||
|
- **CUDA:** ≤ 11.8
|
||||||
|
- **PyTorch:** ≤ 2.1.0
|
||||||
|
- **Inference:** FP32 only (no Tensor Cores)
|
||||||
|
- **Storage:** NVMe/SSD for scratch, NAS/SMB for input/output
|
||||||
|
- **RAM:** ≥ 32GB system RAM
|
||||||
|
|
||||||
|
## Key Design Decisions
|
||||||
|
|
||||||
|
1. **TensorRT FP32 only** — Tesla P40 has no Tensor Cores; FP16 would not provide benefit
|
||||||
|
2. **YOLOv8n + MobileNetV3** — Lightweight models optimized for throughput over accuracy
|
||||||
|
3. **Max aggregation** — Most conservative confidence strategy; use highest frame confidence
|
||||||
|
4. **Temperature scaling** — Calibrates confidence scores without retraining
|
||||||
|
5. **Head-only fine-tuning** — Faster retraining with frozen backbone
|
||||||
|
6. **Parquet export** — Efficient columnar format for analytics
|
||||||
|
7. **No auth/SSL** — Per TC-06, internal LAN only
|
||||||
|
|
||||||
|
## License
|
||||||
|
|
||||||
|
Proprietary — VideoDetect Project
|
||||||
+182
@@ -0,0 +1,182 @@
|
|||||||
|
# Video Classification System: Requirements Document
|
||||||
|
**Version:** 1.2
|
||||||
|
**Author:** AI Architecture Consultant
|
||||||
|
**Target Audience:** Engineering Leads, ML Engineers, DevOps, Product/Project Managers
|
||||||
|
**Purpose:** Define functional, technical, and operational requirements to drive implementation story creation and
|
||||||
|
sprint planning.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. System Overview & Objectives
|
||||||
|
The system processes a large corpus of video files (~30TB) as a local background batch job to classify each video into two
|
||||||
|
categories:
|
||||||
|
- `MATCH`: Video contains at least one Black male subject
|
||||||
|
- `NO_MATCH`: Video does not contain the target demographic
|
||||||
|
|
||||||
|
The system prioritizes **throughput and speed** over maximum accuracy, outputs **calibrated confidence scores**,
|
||||||
|
routes low-confidence results to a **manual review queue**, and supports an **active learning loop** to
|
||||||
|
incrementally improve model performance using reviewed samples.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. Functional Requirements
|
||||||
|
| ID | Requirement | Priority | Notes |
|
||||||
|
|----|-------------|----------|-------|
|
||||||
|
| FR-01 | Configurable frame sampling interval (default: 1 frame per 30 seconds) | P0 | Must support override per
|
||||||
|
job/batch. Uniform temporal sampling preferred. |
|
||||||
|
| FR-02 | Face detection on all sampled frames | P0 | Lightweight detector only; no full-body or scene analysis. |
|
||||||
|
| FR-03 | Binary demographic classification of detected face crops | P0 | Outputs probability `p ∈ [0,1]` for
|
||||||
|
target class. |
|
||||||
|
| FR-04 | Video-level confidence aggregation & threshold routing | P0 | Aggregates frame-level scores → video
|
||||||
|
confidence `C`. Routes to `MATCH`, `REVIEW`, or `SKIP`. |
|
||||||
|
| FR-05 | Manual review interface for low-confidence videos | P1 | Displays video + contributing frames/crops +
|
||||||
|
model confidence. Supports binary labeling. |
|
||||||
|
| FR-06 | Active learning pipeline (label ingestion → fine-tuning → deployment) | P1 | Batch retraining only. No
|
||||||
|
online learning. Versioned model swaps. |
|
||||||
|
| FR-07 | Metadata logging & audit trail | P1 | Stores video ID, timestamps, frame counts, confidence scores,
|
||||||
|
routing decision, model version. |
|
||||||
|
| FR-08 | Batch job orchestration & crash recovery | P0 | Supports resume, parallel GPU scheduling, and
|
||||||
|
deterministic IDempotent processing. |
|
||||||
|
| FR-09 | Directory Scanning & Sync | P0 | Process to scan input directories, detect new/removed files, and sync state to MariaDB. |
|
||||||
|
| FR-10 | Codec & Resolution Detection & Handling | P0 | Detect video properties; handle unsupported codecs by flagging files as `UNSCANNABLE`. |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3. Non-Functional Requirements
|
||||||
|
| ID | Requirement | Target | Notes |
|
||||||
|
|----|-------------|--------|-------|
|
||||||
|
| NFR-01 | Throughput | ≥ 30 videos/hour/GPU (≈ 60 videos/hour total) | Baseline; tunable via sampling interval &
|
||||||
|
batch size. |
|
||||||
|
| NFR-02 | Latency per video | ≤ 45 seconds end-to-end (15-min avg video) | Excludes I/O bottlenecks; measured at
|
||||||
|
compute stage. |
|
||||||
|
| NFR-03 | GPU Memory Safety | ≤ 18GB per GPU sustained | Leaves headroom for OS, queues, and peak allocation. |
|
||||||
|
| NFR-04 | Determinism & Reproducibility | Config-seeded randomness, versioned models | Enables auditability and
|
||||||
|
rollback. |
|
||||||
|
| NFR-05 | Fault Tolerance | Auto-retry on transient failures; skip & log on fatal errors | Prevents batch
|
||||||
|
poisoning. |
|
||||||
|
| NFR-06 | Observability | Prometheus/Grafana metrics + structured logging | Tracks FPS, queue depth, confidence
|
||||||
|
distribution, drift alerts. |
|
||||||
|
| NFR-07 | Data Volume Handling | Efficient indexing for ~30TB dataset | Metadata stored in MariaDB; file existence verified via hashing/checksums if needed. |
|
||||||
|
| NFR-08 | Codec Agnosticism | Handle H.264, H.265, VP8, VP9, AV1, MJPEG, etc. | Unsupported codecs flagged gracefully. |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4. Technical & Environmental Constraints
|
||||||
|
| ID | Constraint | Details |
|
||||||
|
|----|------------|---------|
|
||||||
|
| TC-01 | Hardware | 2× Tesla P40 24GB (compute capability 5.2, PCIe 3.0, **no Tensor Cores**) |
|
||||||
|
| TC-02 | CUDA/Torch Compatibility | CUDA ≤ 11.8, PyTorch ≤ 2.1.0, FP32 inference only |
|
||||||
|
| TC-03 | Storage I/O | Fast local NVMe/SSD for temp frame cache; shared NAS/SMB for video input/output. 30TB Capacity. |
|
||||||
|
| TC-04 | Framework Stack | PyTorch → ONNX → TensorRT FP32; FFmpeg/OpenCV for sampling; **MariaDB** for metadata/state |
|
||||||
|
| TC-05 | Deployment Model | **Docker Compose** orchestrates all services (Workers, DB, UI). GPUs passed via `nvidia-container-toolkit`. |
|
||||||
|
| TC-06 | Network Security | Internal LAN only. No reverse proxy, SSL, or auth required for UI. |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5. Data & Storage Architecture
|
||||||
|
| Layer | Specification |
|
||||||
|
|-------|---------------|
|
||||||
|
| **Input** | Raw video files (MP4, MOV, AVI, MKV) totaling ~30TB. Growth is slow (archival nature). Resolutions: 480p–4K. Codecs: Mixed. |
|
||||||
|
| **Scratch** | `tmpfs` or fast local SSD for extracted frames & face crops. Auto-cleaned post-job. |
|
||||||
|
| **Metadata Store** | **MariaDB** database storing: `file_path`, `file_hash`, `resolution`, `codec`, `last_scan_time`, `last_processed_time`, `model_version_used`, `confidence_score`, `routing_decision`, `status` (OK, UNSCANNABLE, ERROR). |
|
||||||
|
| **Results** | Parquet/JSON lines exported for analytics: `{video_id, model_version, sample_count, confidence_scores[], video_confidence, routing, processed_at}` |
|
||||||
|
| **Training Data** | Versioned directory structure: `/data/v1/crops/`, `/data/v1/labels/`. Augmentation pipeline
|
||||||
|
applied at training time. |
|
||||||
|
| **Model Registry** | `/models/` with semantic versioning. Active, candidate, and archived states tracked. |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 6. Confidence Scoring & Routing Logic
|
||||||
|
| Stage | Specification |
|
||||||
|
|-------|---------------|
|
||||||
|
| **Frame-Level** | Raw logits → temperature-scaled softmax → calibrated `p_i ∈ [0,1]` |
|
||||||
|
| **Video-Level** | `C = max(p_i)` OR `C = softmax(α·mean(p_i) + β·var(p_i))` (configurable) |
|
||||||
|
| **Routing Thresholds** | `C ≥ T_high` → `MATCH` <br> `T_low ≤ C < T_high` → `REVIEW` <br> `C < T_low` → `SKIP`
|
||||||
|
<br> *(Default: T_high=0.75, T_low=0.45)* |
|
||||||
|
| **Calibration** | Temperature scaling evaluated on held-out set every training cycle. Stores `T` with model
|
||||||
|
checkpoint. |
|
||||||
|
| **Error Handling** | If FFmpeg/OpenCV fails to decode frame or detect codec: Set status to `UNSCANNABLE`. Log error. Do not retry indefinitely. |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 7. Active Learning Pipeline
|
||||||
|
| Component | Specification |
|
||||||
|
|-----------|---------------|
|
||||||
|
| **Review Queue** | Lightweight web UI (Label Studio or custom Flask/FastAPI). Shows video player, top-k
|
||||||
|
contributing frames, model confidence, label toggle. Accessible via internal IP:Port. |
|
||||||
|
| **Label Export** | CSV/JSON export with `{video_id, frame_timestamps, crops_paths, ground_truth}` |
|
||||||
|
| **Fine-Tuning** | Head-only fine-tuning on face crops. 10–30 epochs, AdamW, LR=1e-3, early stopping. Runs on
|
||||||
|
idle GPU slot or off-peak schedule. |
|
||||||
|
| **Deployment** | Candidate model validated against held-out set → auto-swap if `ΔF1 > 0.02` & confidence
|
||||||
|
calibration passes. Rollback on regression. |
|
||||||
|
| **Drift Monitoring** | Weekly confidence histogram comparison. Alerts if `p(C > 0.5)` shifts >10% or review
|
||||||
|
queue grows unbounded. |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 8. Acceptance Criteria
|
||||||
|
| Area | Criteria |
|
||||||
|
|------|----------|
|
||||||
|
| **Sampling** | Configurable interval honored ±1 frame; handles variable FPS & codec edge cases. |
|
||||||
|
| **Inference** | Sustained ≥ 30 videos/hour/GPU at default sampling; GPU RAM ≤ 18GB. |
|
||||||
|
| **Confidence** | Calibration error (ECE) ≤ 0.08 on validation set; thresholds configurable via YAML/ENV. |
|
||||||
|
| **Routing** | 100% of videos assigned to exactly one bucket; metadata persisted before cleanup. |
|
||||||
|
| **Active Learning** | Labeled reviews → fine-tune → model swap → next batch uses new weights. Full loop < 48h. |
|
||||||
|
| **Reliability** | Batch resumes after crash without reprocessing; idempotent file handling. |
|
||||||
|
| **Syncing** | Scanning process accurately identifies new, modified, and deleted files in the 30TB corpus relative to DB state. |
|
||||||
|
| **Codec Handling** | Files with unsupported codecs are marked `UNSCANNABLE` in DB without crashing the batch. |
|
||||||
|
| **Deployment** | `docker-compose.yml` launches DB, Worker(s), and UI. UI accessible on LAN via standard port. |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 9. Implementation Story Mapping
|
||||||
|
| Epic | User Story | Acceptance Criteria | Priority |
|
||||||
|
|------|------------|---------------------|----------|
|
||||||
|
| **E1: Core Pipeline** | As an engineer, I can configure frame sampling interval and extract frames uniformly. |
|
||||||
|
FFmpeg/OpenCV extracts frames at interval; handles variable duration; logs frame count. | P0 |
|
||||||
|
| **E1** | As an engineer, I can run face detection on sampled frames in batch. | YOLOv8n/RetinaFace runs at
|
||||||
|
320²/416²; batch size auto-tuned to VRAM; outputs crop tensors. | P0 |
|
||||||
|
| **E1** | I can classify face crops and aggregate video confidence. | MobileNetV3 head outputs
|
||||||
|
calibrated `p`; aggregation logic configurable; results saved to Parquet. | P0 |
|
||||||
|
| **E1** | As a system, I can detect video resolution and codec. | FFmpeg probe extracts width, height, codec name. Stored in DB. | P0 |
|
||||||
|
| **E1** | As a system, I can handle unsupported codecs. | If decoding fails, file marked `UNSCANNABLE`. Error logged. | P0 |
|
||||||
|
| **E2: Routing & Review** | As an operator, I can route videos based on confidence thresholds. | High→MATCH,
|
||||||
|
Mid→REVIEW, Low→SKIP; thresholds in config; routing logged. | P0 |
|
||||||
|
| **E2** | As an annotator, I can view low-confidence videos and label them. | UI shows video + frames; label
|
||||||
|
saved; export triggers training pipeline. | P1 |
|
||||||
|
| **E3: Active Learning** | As an ML engineer, I can fine-tune the classifier head with reviewed data. | Dataset
|
||||||
|
versioned; head-only training; early stopping; ECE validated. | P1 |
|
||||||
|
| **E3** | As a system, I can swap models safely and roll back on regression. | Model registry with
|
||||||
|
candidate/active states; auto-validate; rollback script. | P1 |
|
||||||
|
| **E4: Operations** | As a DevOps, I can schedule, monitor, and resume batch jobs. | Multi-GPU process split;
|
||||||
|
crash recovery; Prometheus metrics; structured logs. | P0 |
|
||||||
|
| **E4** | As a lead, I can track confidence drift and review queue health. | Dashboard shows distribution, queue
|
||||||
|
size, throughput; alerts on thresholds. | P1 |
|
||||||
|
| **E5: Data Management** | As a system, I can scan directories and sync file state to MariaDB. | Detects new/removed files; updates `last_scan_time`; handles 30TB path space efficiently. | P0 |
|
||||||
|
| **E6: Infrastructure** | As a DevOps, I can deploy the entire stack via Docker Compose. | `docker-compose.yml` includes DB, Worker, UI. GPU passthrough configured. No SSL/Auth. | P0 |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 10. Assumptions & Open Questions
|
||||||
|
| Item | Status |
|
||||||
|
|------|--------|
|
||||||
|
| Dataset size & growth rate | ✅ ~30TB, slow growth (archival). |
|
||||||
|
| Video codec/resolution distribution | ✅ Varies 480p–4K. Virtually all codecs present. Unsupported ones flagged. |
|
||||||
|
| Review annotation bandwidth | ❓ Determines acceptable queue backlog & SLA |
|
||||||
|
| Retraining frequency preference | ❓ Nightly, weekly, or queue-depth triggered? |
|
||||||
|
| UI deployment environment | ✅ Internal LAN only. No security/SSL needed. Docker Compose managed. |
|
||||||
|
| Legal/ethics sign-off | ✅ Assumed handled per prompt; document retention policy still needed |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Next Steps
|
||||||
|
1. **Stakeholder Review**: Validate thresholds, throughput targets, and review workflow.
|
||||||
|
2. **Sprint 0 Setup**: Provision Docker env, install CUDA 11.8 + PyTorch 2.0.1, set up MariaDB schema & temp storage. Configure GPU passthrough in `docker-compose`.
|
||||||
|
3. **MVP Build**: Implement E1 stories → baseline throughput & confidence pipeline.
|
||||||
|
4. **Calibration & Routing**: Add ECE scaling, threshold routing, metadata persistence.
|
||||||
|
5. **Review & Active Learning**: Deploy UI, wiring label export → fine-tune → model swap.
|
||||||
|
6. **Data Syncing**: Implement directory scanner (E5) to maintain MariaDB state against 30TB filesystem.
|
||||||
|
7. **Hardening**: Multi-GPU scheduling, crash recovery, monitoring, drift alerts.
|
||||||
|
|
||||||
|
Let me know if you want this exported as a Confluence/Markdown template, or if you'd like detailed technical specs
|
||||||
|
for any specific story (e.g., FFmpeg sampling logic, TensorRT export pipeline, or Label Studio integration schema).
|
||||||
+170
@@ -0,0 +1,170 @@
|
|||||||
|
# STORY-01: Foundation & Infrastructure
|
||||||
|
|
||||||
|
## Epic
|
||||||
|
**E6: Infrastructure** — As a DevOps engineer, I can deploy the entire stack via Docker Compose.
|
||||||
|
|
||||||
|
## Related Requirements
|
||||||
|
| ID | Requirement |
|
||||||
|
|----|-------------|
|
||||||
|
| TC-01 | Hardware: 2× Tesla P40 24GB (compute capability 5.2, PCIe 3.0, no Tensor Cores) |
|
||||||
|
| TC-02 | CUDA/Torch Compatibility: CUDA ≤ 11.8, PyTorch ≤ 2.1.0, FP32 inference only |
|
||||||
|
| TC-03 | Storage I/O: Fast local NVMe/SSD for temp frame cache; shared NAS/SMB for video input/output |
|
||||||
|
| TC-04 | Framework Stack: PyTorch → ONNX → TensorRT FP32; FFmpeg/OpenCV; MariaDB |
|
||||||
|
| TC-05 | Deployment Model: Docker Compose orchestrates all services; GPUs passed via nvidia-container-toolkit |
|
||||||
|
| TC-06 | Network Security: Internal LAN only; no reverse proxy, SSL, or auth |
|
||||||
|
| NFR-04 | Determinism & Reproducibility: Config-seeded randomness, versioned models |
|
||||||
|
|
||||||
|
## Description
|
||||||
|
Establish the Docker environment, database schema, and basic connectivity. No video processing logic yet — just the operational skeleton that all subsequent stories depend on.
|
||||||
|
|
||||||
|
## Scope
|
||||||
|
|
||||||
|
### In Scope
|
||||||
|
- Docker Compose multi-service orchestration
|
||||||
|
- Custom Worker Dockerfile with CUDA 11.8 + PyTorch 2.1.0 + TensorRT FP32
|
||||||
|
- MariaDB schema design and migration scripts
|
||||||
|
- Volume mounts for NVMe scratch, NAS input/output, and model persistence
|
||||||
|
- Configuration management (config.yaml + environment variables)
|
||||||
|
- Structured logging setup (JSON format to stdout)
|
||||||
|
- Database connection layer with connection pooling
|
||||||
|
- Network connectivity verification between all services
|
||||||
|
|
||||||
|
### Out of Scope
|
||||||
|
- Video processing logic (covered in STORY-02 through STORY-05)
|
||||||
|
- Model loading or inference (covered in STORY-04)
|
||||||
|
- Review UI functionality (covered in STORY-06)
|
||||||
|
- Monitoring dashboards (covered in STORY-08)
|
||||||
|
- Active learning pipeline (covered in STORY-07)
|
||||||
|
|
||||||
|
## Deliverables
|
||||||
|
|
||||||
|
### 1.1 Docker Compose Structure
|
||||||
|
**File:** `docker-compose.yml`
|
||||||
|
|
||||||
|
Services defined:
|
||||||
|
- `mariadb`: MariaDB 10.11+ with persistent volume
|
||||||
|
- `worker`: PyTorch/TensorRT inference worker with GPU passthrough
|
||||||
|
- `ui`: Placeholder service (nginx serving static page) for network verification
|
||||||
|
|
||||||
|
Key configurations:
|
||||||
|
- `nvidia-container-toolkit` runtime configuration for GPU passthrough
|
||||||
|
- Volume mounts:
|
||||||
|
- NVMe → `/scratch` (tmpfs for speed)
|
||||||
|
- NAS/SMB → `/data/input` and `/data/output`
|
||||||
|
- Persistent → `/models` and `/data/training`
|
||||||
|
- Network bridge for inter-service communication
|
||||||
|
- Resource limits (GPU memory caps per NFR-03)
|
||||||
|
|
||||||
|
### 1.2 Worker Dockerfile
|
||||||
|
**File:** `worker/Dockerfile`
|
||||||
|
|
||||||
|
Base image: `nvidia/cuda:11.8.0-runtime-ubuntu22.04`
|
||||||
|
|
||||||
|
Installed packages:
|
||||||
|
- PyTorch 2.1.0 (CUDA 11.8, FP32 only)
|
||||||
|
- TensorRT 8.6+ (FP32)
|
||||||
|
- OpenCV 4.8+
|
||||||
|
- FFmpeg 5.x + ffprobe
|
||||||
|
- ONNX Runtime
|
||||||
|
- Python 3.10+
|
||||||
|
- Required system libraries (libcudnn8, libglib2.0, etc.)
|
||||||
|
|
||||||
|
### 1.3 Database Schema
|
||||||
|
**File:** `db/schema.sql`
|
||||||
|
|
||||||
|
Tables:
|
||||||
|
- `videos`: `id` (BIGINT PK), `file_path` (VARCHAR), `file_hash` (CHAR(64)), `resolution_w` (INT), `resolution_h` (INT), `codec` (VARCHAR), `duration` (FLOAT), `status` (ENUM: NEW, PENDING, PROCESSING, COMPLETED, UNSCANNABLE, ERROR), `last_scan_time` (DATETIME), `last_processed_time` (DATETIME), `created_at` (DATETIME), `updated_at` (DATETIME)
|
||||||
|
- `processing_logs`: `id` (BIGINT PK), `video_id` (BIGINT FK), `model_version` (VARCHAR), `frame_count` (INT), `confidence_score` (FLOAT), `routing_decision` (ENUM: MATCH, REVIEW, SKIP), `processed_at` (DATETIME), `error_message` (TEXT)
|
||||||
|
- `models`: `version` (VARCHAR PK), `status` (ENUM: ACTIVE, CANDIDATE, ARCHIVED), `path` (VARCHAR), `calibration_temp` (FLOAT), `f1_score` (FLOAT), `ece_score` (FLOAT), `deployed_at` (DATETIME), `created_at` (DATETIME)
|
||||||
|
- `review_queue`: `id` (BIGINT PK), `video_id` (BIGINT FK), `confidence_score` (FLOAT), `routing_decision` (ENUM: REVIEW), `annotated` (BOOLEAN), `ground_truth` (BOOLEAN), `annotated_at` (DATETIME), `created_at` (DATETIME)
|
||||||
|
|
||||||
|
Indexes:
|
||||||
|
- `idx_videos_status` on `videos(status)`
|
||||||
|
- `idx_videos_file_hash` on `videos(file_hash)` (UNIQUE)
|
||||||
|
- `idx_videos_last_scan` on `videos(last_scan_time)`
|
||||||
|
- `idx_processing_logs_video` on `processing_logs(video_id)`
|
||||||
|
- `idx_models_status` on `models(status)`
|
||||||
|
- `idx_review_queue_annotated` on `review_queue(annotated)`
|
||||||
|
|
||||||
|
### 1.4 Database Connection Layer
|
||||||
|
**File:** `src/db_connector.py`
|
||||||
|
|
||||||
|
Features:
|
||||||
|
- Connection pooling (DBUtils + PyMySQL)
|
||||||
|
- Configurable pool size (min=5, max=20)
|
||||||
|
- Automatic reconnection on disconnect
|
||||||
|
- Context manager support
|
||||||
|
- Prepared statements for all queries
|
||||||
|
- Transaction support for atomic state transitions
|
||||||
|
|
||||||
|
### 1.5 Configuration Management
|
||||||
|
**File:** `config.yaml`
|
||||||
|
|
||||||
|
Contents:
|
||||||
|
- `sampling`: `interval_seconds: 30`, `override_per_job: true`
|
||||||
|
- `thresholds`: `T_high: 0.75`, `T_low: 0.45`
|
||||||
|
- `gpu`: `max_memory_gb: 18`, `batch_size: auto`, `device: cuda`
|
||||||
|
- `storage`: `scratch_path: /scratch`, `input_path: /data/input`, `output_path: /data/output`
|
||||||
|
- `database`: `host: mariadb`, `port: 3306`, `pool_size: 20`
|
||||||
|
- `logging`: `format: json`, `level: INFO`
|
||||||
|
- `model`: `face_detector: yolo8n`, `classifier: mobilenetv3`, `input_size: 224`
|
||||||
|
|
||||||
|
**File:** `src/config_loader.py`
|
||||||
|
|
||||||
|
Features:
|
||||||
|
- Load config.yaml with environment variable overrides
|
||||||
|
- Validate all required fields
|
||||||
|
- Provide typed accessors (e.g., `config.thresholds.T_high`)
|
||||||
|
- Hot-reload support for config changes
|
||||||
|
|
||||||
|
### 1.6 Logging Setup
|
||||||
|
**File:** `src/logging_config.py`
|
||||||
|
|
||||||
|
- JSON structured logging via `python-json-logger`
|
||||||
|
- Log levels: DEBUG, INFO, WARNING, ERROR, CRITICAL
|
||||||
|
- Fields: timestamp, level, service, video_id, message, metadata (key-value pairs)
|
||||||
|
- Log rotation: 100MB per file, 10 files max
|
||||||
|
- All logs to stdout for Docker capture
|
||||||
|
|
||||||
|
## Acceptance Criteria
|
||||||
|
|
||||||
|
### Functional
|
||||||
|
- [ ] `docker-compose up` starts MariaDB, Worker, and UI containers successfully
|
||||||
|
- [ ] Worker container can connect to MariaDB and execute schema migrations
|
||||||
|
- [ ] GPU is visible inside Worker container (`nvidia-smi` shows Tesla P40)
|
||||||
|
- [ ] Volume mounts are accessible and writable in all containers
|
||||||
|
- [ ] Config.yaml loads correctly with all required fields validated
|
||||||
|
- [ ] Structured logging produces valid JSON output in all services
|
||||||
|
- [ ] Database connection pool handles concurrent connections (test with 20 simultaneous)
|
||||||
|
|
||||||
|
### Non-Functional
|
||||||
|
- [ ] Worker container starts within 30 seconds
|
||||||
|
- [ ] MariaDB starts within 15 seconds
|
||||||
|
- [ ] GPU memory usage in Worker is < 2GB at idle (before model loading)
|
||||||
|
- [ ] All services communicate over Docker internal network (no host network exposure except UI port)
|
||||||
|
- [ ] Schema migration is idempotent (running twice produces same result)
|
||||||
|
|
||||||
|
### Technical Constraints
|
||||||
|
- [ ] CUDA version in Worker is 11.8 (verified via `torch.version.cuda`)
|
||||||
|
- [ ] PyTorch version ≤ 2.1.0 (verified via `torch.__version__`)
|
||||||
|
- [ ] TensorRT runs in FP32 mode only
|
||||||
|
- [ ] No Tensor Cores used (compute capability 5.2 constraint respected)
|
||||||
|
- [ ] No SSL, auth, or reverse proxy configured (TC-06)
|
||||||
|
|
||||||
|
## Dependencies
|
||||||
|
- **Prerequisites:** NVIDIA Container Toolkit installed on host, Docker Compose v2+, NAS/SMB mounts configured
|
||||||
|
- **Depends on:** None (this is the foundational story)
|
||||||
|
- **Enables:** STORY-02 (Ingestion), STORY-03 (Orchestration), STORY-04 (Inference), STORY-05 (Results), STORY-06 (Review UI), STORY-07 (Active Learning), STORY-08 (Monitoring)
|
||||||
|
|
||||||
|
## Risks & Mitigations
|
||||||
|
| Risk | Mitigation |
|
||||||
|
|------|-----------|
|
||||||
|
| Tesla P40 (CC 5.2) incompatible with newer TensorRT | Use TensorRT 8.6 which supports CC 5.x; test early |
|
||||||
|
| CUDA 11.8 + PyTorch 2.1.0 dependency conflicts | Pin all versions in Dockerfile; use nvidia base image |
|
||||||
|
| NAS/SMB mount latency affects processing | Use local tmpfs for scratch; only read from NAS |
|
||||||
|
| MariaDB connection pool exhaustion | Monitor pool metrics; tune pool_size based on worker count |
|
||||||
|
|
||||||
|
## Estimated Effort
|
||||||
|
- **Sprint:** 1-2
|
||||||
|
- **Story Points:** 13
|
||||||
|
- **Dependencies:** None
|
||||||
+182
@@ -0,0 +1,182 @@
|
|||||||
|
# STORY-02: Core Ingestion & Codec Handling
|
||||||
|
|
||||||
|
## Epic
|
||||||
|
**E5: Data Management** — As a system, I can scan directories and sync file state to MariaDB.
|
||||||
|
**E1: Core Pipeline** — As a system, I can detect video resolution and codec.
|
||||||
|
|
||||||
|
## Related Requirements
|
||||||
|
| ID | Requirement |
|
||||||
|
|----|---------|
|
||||||
|
| FR-09 | Directory Scanning & Sync: Process to scan input directories, detect new/removed files, and sync state to MariaDB |
|
||||||
|
| FR-10 | Codec & Resolution Detection & Handling: Detect video properties; handle unsupported codecs by flagging files as UNSCANNABLE |
|
||||||
|
| NFR-07 | Data Volume Handling: Efficient indexing for ~30TB dataset |
|
||||||
|
| NFR-08 | Codec Agnosticism: Handle H.264, H.265, VP8, VP9, AV1, MJPEG, etc. |
|
||||||
|
| NFR-05 | Fault Tolerance: Auto-retry on transient failures; skip & log on fatal errors |
|
||||||
|
|
||||||
|
## Description
|
||||||
|
Implement the directory scanner, video probing, and robust error handling for codecs and resolutions. This story enables the system to discover new files in the 30TB corpus, extract their metadata, validate codec support, and maintain accurate database state.
|
||||||
|
|
||||||
|
## Scope
|
||||||
|
|
||||||
|
### In Scope
|
||||||
|
- Background directory scanner service that walks `/data/input`
|
||||||
|
- File detection and deduplication via SHA-256 hashing
|
||||||
|
- Video probing via ffprobe (codec, resolution, duration)
|
||||||
|
- Codec whitelist/blacklist validation
|
||||||
|
- UNSCANNABLE status for unsupported or corrupt files
|
||||||
|
- Priority queue / DB-based locking for job assignment
|
||||||
|
- Atomic state transitions (PENDING → PROCESSING)
|
||||||
|
- Efficient handling of 30TB directory structure (incremental scanning)
|
||||||
|
|
||||||
|
### Out of Scope
|
||||||
|
- Frame extraction (covered in STORY-04)
|
||||||
|
- Face detection and classification (covered in STORY-04)
|
||||||
|
- Confidence scoring and routing (covered in STORY-05)
|
||||||
|
- Review UI (covered in STORY-06)
|
||||||
|
- Active learning pipeline (covered in STORY-07)
|
||||||
|
- Monitoring dashboards (covered in STORY-08)
|
||||||
|
|
||||||
|
## Deliverables
|
||||||
|
|
||||||
|
### 2.1 Directory Scanner Service
|
||||||
|
**File:** `src/scanner.py`
|
||||||
|
|
||||||
|
Core components:
|
||||||
|
- **Walker:** Recursive directory walker with configurable depth and path filters
|
||||||
|
- **Incremental Sync:** Compare current filesystem state against DB `last_scan_time` — only process new/modified files
|
||||||
|
- **File Detection:** Identify files not in DB or with status NEW/PENDING
|
||||||
|
- **Hash Computation:** SHA-256 of first 1MB (or full file if < 1MB) for deduplication
|
||||||
|
- **Scan Scheduler:** Configurable interval (default: every 60 seconds) via cron-like scheduler
|
||||||
|
- **Concurrency:** Multi-threaded walker with configurable worker count (default: 8 threads)
|
||||||
|
- **Error Handling:** Per-file error isolation — scanner continues on individual file failures
|
||||||
|
|
||||||
|
Key behaviors:
|
||||||
|
- On scan start: query DB for files with `last_scan_time < now()` or `status IN ('NEW', 'PENDING')`
|
||||||
|
- For each file: compute hash → check DB for duplicate → if new, insert with status PENDING
|
||||||
|
- For removed files: mark as REMOVED in DB (optional, configurable)
|
||||||
|
- Update `last_scan_time` on `videos` table after successful scan
|
||||||
|
|
||||||
|
### 2.2 Video Probing Module
|
||||||
|
**File:** `src/prober.py`
|
||||||
|
|
||||||
|
Features:
|
||||||
|
- **ffprobe wrapper:** Execute ffprobe with optimized arguments for speed
|
||||||
|
```bash
|
||||||
|
ffprobe -v error -select_streams v:0 -show_entries stream=codec_name,width,height,duration,r_frame_rate -show_entries format=duration -of json <file>
|
||||||
|
```
|
||||||
|
- **Metadata extraction:** Parse JSON output for:
|
||||||
|
- `codec_name`: Video codec identifier
|
||||||
|
- `width`, `height`: Resolution
|
||||||
|
- `duration`: Video duration in seconds
|
||||||
|
- `r_frame_rate`: Frame rate (for sampling calculations)
|
||||||
|
- **Error handling:** Catch ffprobe failures (corrupt files, unsupported formats)
|
||||||
|
- **Timeout:** ffprobe execution limited to 10 seconds per file
|
||||||
|
|
||||||
|
### 2.3 Codec Validation
|
||||||
|
**File:** `src/codec_validator.py`
|
||||||
|
|
||||||
|
Codec whitelist (supported):
|
||||||
|
- H.264 (avc1)
|
||||||
|
- H.265 (hevc)
|
||||||
|
- VP8 (vp8)
|
||||||
|
- VP9 (vp9)
|
||||||
|
- AV1 (av01)
|
||||||
|
- MJPEG (mjpeg)
|
||||||
|
- MPEG-4 (mp4v)
|
||||||
|
|
||||||
|
Codec blacklist (unsupported):
|
||||||
|
- Theora
|
||||||
|
- DivX/Xvid (legacy)
|
||||||
|
- ProRes (requires special handling)
|
||||||
|
- Any codec not in whitelist
|
||||||
|
|
||||||
|
Behavior:
|
||||||
|
- If codec not in whitelist → set status UNSCANNABLE, log reason
|
||||||
|
- If ffprobe fails → set status UNSCANNABLE, log error code
|
||||||
|
- If file is not a valid video → set status UNSCANNABLE, log reason
|
||||||
|
- UNSCANNABLE files are excluded from processing pipeline
|
||||||
|
|
||||||
|
### 2.4 Batch Orchestration Skeleton
|
||||||
|
**File:** `src/orchestrator.py`
|
||||||
|
|
||||||
|
Components:
|
||||||
|
- **Job Queue:** Priority queue based on file modification time (newest first)
|
||||||
|
- **DB-based Locking:** Atomic state transition using UPDATE ... WHERE status = 'PENDING'
|
||||||
|
- **Worker Pool:** Configurable number of worker processes (default: 2, one per GPU)
|
||||||
|
- **State Machine:**
|
||||||
|
```
|
||||||
|
NEW → PENDING → PROCESSING → COMPLETED
|
||||||
|
→ UNSCANNABLE
|
||||||
|
→ ERROR
|
||||||
|
```
|
||||||
|
- **Concurrency Control:** Max concurrent processing per GPU (default: 1 video at a time per worker)
|
||||||
|
- **Idempotency:** Re-processing a file does not duplicate DB entries or outputs
|
||||||
|
|
||||||
|
### 2.5 Configuration Updates
|
||||||
|
**File:** `config.yaml` (updates to STORY-01)
|
||||||
|
|
||||||
|
New fields:
|
||||||
|
```yaml
|
||||||
|
scanner:
|
||||||
|
scan_interval_seconds: 60
|
||||||
|
walker_threads: 8
|
||||||
|
ffprobe_timeout_seconds: 10
|
||||||
|
hash_algorithm: sha256
|
||||||
|
hash_chunk_size_mb: 1
|
||||||
|
|
||||||
|
codec:
|
||||||
|
whitelist: [avc1, hevc, vp8, vp9, av01, mjpeg, mp4v]
|
||||||
|
default_status_on_error: UNSCANNABLE
|
||||||
|
|
||||||
|
queue:
|
||||||
|
priority: modification_time # newest first
|
||||||
|
max_concurrent_per_gpu: 1
|
||||||
|
lock_timeout_seconds: 300
|
||||||
|
```
|
||||||
|
|
||||||
|
## Acceptance Criteria
|
||||||
|
|
||||||
|
### Functional
|
||||||
|
- [ ] New files in `/data/input` appear in DB with correct metadata within 60 seconds of placement
|
||||||
|
- [ ] File hash deduplication prevents re-processing identical files
|
||||||
|
- [ ] ffprobe correctly extracts codec, resolution, and duration for all supported codecs
|
||||||
|
- [ ] Unsupported codec files are marked UNSCANNABLE without crashing the scanner
|
||||||
|
- [ ] Corrupt/unreadable files are marked UNSCANNABLE with appropriate error logged
|
||||||
|
- [ ] Atomic state transitions: PENDING → PROCESSING succeeds only once (no duplicate processing)
|
||||||
|
- [ ] Removed files are detected and marked REMOVED (if configured)
|
||||||
|
- [ ] Scanner handles 30TB directory structure without OOM (memory < 500MB during scan)
|
||||||
|
- [ ] Re-running scanner is idempotent — no duplicate entries or states
|
||||||
|
|
||||||
|
### Non-Functional
|
||||||
|
- [ ] Scanner completes full 30TB directory walk in < 4 hours (incremental: < 10 minutes for typical day)
|
||||||
|
- [ ] ffprobe timeout (10s) is enforced — does not hang on corrupt files
|
||||||
|
- [ ] Scanner memory usage stays < 500MB regardless of directory depth
|
||||||
|
- [ ] Hash computation for 100MB file completes in < 5 seconds on NVMe
|
||||||
|
- [ ] Worker pool respects GPU count (2 workers for 2 GPUs)
|
||||||
|
|
||||||
|
### Technical Constraints
|
||||||
|
- [ ] SHA-256 hash is deterministic and reproducible
|
||||||
|
- [ ] Codec detection matches ffprobe output exactly
|
||||||
|
- [ ] Resolution stored as integers (width, height)
|
||||||
|
- [ ] Duration stored as float (seconds)
|
||||||
|
- [ ] All scanner errors are logged with video_id, error_code, and file_path
|
||||||
|
- [ ] Scanner does not modify video files (read-only operation)
|
||||||
|
|
||||||
|
## Dependencies
|
||||||
|
- **Prerequisites:** STORY-01 (Foundation & Infrastructure) — DB schema, connection layer, config
|
||||||
|
- **Depends on:** None (runs in parallel with STORY-03 if needed)
|
||||||
|
- **Enables:** STORY-04 (Inference Pipeline), STORY-05 (Results & Export)
|
||||||
|
|
||||||
|
## Risks & Mitigations
|
||||||
|
| Risk | Mitigation |
|
||||||
|
|------|--|
|
||||||
|
| 30TB directory walk is extremely slow | Incremental scanning using last_scan_time; only walk new directories |
|
||||||
|
| ffprobe hangs on corrupt files | Enforce 10-second timeout via subprocess timeout |
|
||||||
|
| Hash computation on slow NAS is bottleneck | Hash only first 1MB; full hash optional for verification |
|
||||||
|
| Concurrent scanner + worker conflicts | DB-based locking; scanner only writes PENDING, workers read PENDING |
|
||||||
|
| Network mount latency | Cache directory listings; batch DB operations |
|
||||||
|
|
||||||
|
## Estimated Effort
|
||||||
|
- **Sprint:** 3-4
|
||||||
|
- **Story Points:** 21
|
||||||
|
- **Dependencies:** STORY-01
|
||||||
+152
@@ -0,0 +1,152 @@
|
|||||||
|
# STORY-03: Frame Sampling
|
||||||
|
|
||||||
|
## Epic
|
||||||
|
**E1: Core Pipeline** — As an engineer, I can configure frame sampling interval and extract frames uniformly.
|
||||||
|
|
||||||
|
## Related Requirements
|
||||||
|
| ID | Requirement |
|
||||||
|
|----|---------|
|
||||||
|
| FR-01 | Configurable frame sampling interval (default: 1 frame per 30 seconds); Must support override per job/batch; Uniform temporal sampling preferred |
|
||||||
|
| NFR-02 | Latency per video: ≤ 45 seconds end-to-end (15-min avg video) |
|
||||||
|
| NFR-03 | GPU Memory Safety: ≤ 18GB per GPU sustained |
|
||||||
|
| NFR-05 | Fault Tolerance: Auto-retry on transient failures; skip & log on fatal errors |
|
||||||
|
| TC-03 | Storage I/O: Fast local NVMe/SSD for temp frame cache |
|
||||||
|
|
||||||
|
## Description
|
||||||
|
Implement frame extraction from processed videos using FFmpeg/OpenCV. Extract one frame per configurable interval (default: 30 seconds) uniformly across the video duration. Handle variable FPS, high-resolution frames, and manage scratch space efficiently.
|
||||||
|
|
||||||
|
## Scope
|
||||||
|
|
||||||
|
### In Scope
|
||||||
|
- Uniform temporal frame sampling via FFmpeg `-ss` timestamp extraction
|
||||||
|
- Configurable sampling interval (default: 30 seconds, override per job)
|
||||||
|
- Variable FPS handling (calculate correct timestamps)
|
||||||
|
- Frame extraction to JPEG format for storage efficiency
|
||||||
|
- Scratch space management (NVMe tmpfs with auto-cleanup)
|
||||||
|
- High-resolution frame handling (downscale 4K to fit VRAM constraints)
|
||||||
|
- Frame naming convention: `{video_id}_{timestamp}.jpg`
|
||||||
|
- Memory management (delete frames after processing)
|
||||||
|
|
||||||
|
### Out of Scope
|
||||||
|
- Face detection (covered in STORY-04)
|
||||||
|
- Classification (covered in STORY-05)
|
||||||
|
- Confidence scoring (covered in STORY-05)
|
||||||
|
- Results export (covered in STORY-06)
|
||||||
|
- Review UI (covered in STORY-07)
|
||||||
|
|
||||||
|
## Deliverables
|
||||||
|
|
||||||
|
### 3.1 Frame Sampler Module
|
||||||
|
**File:** `src/frame_sampler.py`
|
||||||
|
|
||||||
|
Core functionality:
|
||||||
|
- **Timestamp Calculator:** Compute uniform sampling timestamps
|
||||||
|
```python
|
||||||
|
def calculate_timestamps(duration, interval):
|
||||||
|
"""Return list of timestamps for frame extraction."""
|
||||||
|
count = max(1, int(duration / interval))
|
||||||
|
step = duration / count
|
||||||
|
return [i * step for i in range(count)]
|
||||||
|
```
|
||||||
|
- **FFmpeg Extraction:** Extract frames at computed timestamps
|
||||||
|
```bash
|
||||||
|
ffmpeg -ss {timestamp} -i {video_path} -vframes 1 -q:v 2 -f jpeg {output_path}
|
||||||
|
```
|
||||||
|
- **Variable FPS Handling:** Adjust timestamps for videos with variable frame rates
|
||||||
|
- **Quality Control:** JPEG quality factor 2 (high quality, reasonable size)
|
||||||
|
- **Error Handling:** Per-frame error isolation — skip failed frames, log errors, continue
|
||||||
|
|
||||||
|
### 3.2 Resolution Handling
|
||||||
|
**File:** `src/frame_sampler.py` (resolution handling section)
|
||||||
|
|
||||||
|
Features:
|
||||||
|
- **Resolution Detection:** Use metadata from prober (STORY-02)
|
||||||
|
- **Downscale Logic:** Auto-downscale frames > 1080p to fit VRAM
|
||||||
|
- 4K (3840x2160) → 1080p (1920x1080)
|
||||||
|
- 2K (2560x1440) → 720p (1280x720)
|
||||||
|
- ≤ 1080p → no change
|
||||||
|
- **FFmpeg Scale Filter:** Apply during extraction
|
||||||
|
```bash
|
||||||
|
ffmpeg -ss {ts} -i {video} -vf "scale=1920:1080:force_original_aspect_ratio=decrease,pad=1920:1080:(ow-iw)/2:(oh-ih)/2" -vframes 1 -q:v 2 -f jpeg {output}
|
||||||
|
```
|
||||||
|
- **Aspect Ratio Preservation:** Letterbox/pillarbox to maintain original aspect ratio
|
||||||
|
|
||||||
|
### 3.3 Scratch Space Management
|
||||||
|
**File:** `src/scratch_manager.py`
|
||||||
|
|
||||||
|
Features:
|
||||||
|
- **Directory Structure:** `/scratch/{video_id}/frames/` for each video
|
||||||
|
- **Naming Convention:** `{video_id}_{timestamp_ms}.jpg`
|
||||||
|
- **Auto-Cleanup:** Delete all frames after video processing completes
|
||||||
|
- **Space Monitoring:** Alert if scratch usage exceeds 80% of tmpfs capacity
|
||||||
|
- **Cleanup on Error:** Ensure frames are cleaned up even if processing fails
|
||||||
|
- **Concurrent Access:** Per-video directory isolation (no conflicts between workers)
|
||||||
|
|
||||||
|
### 3.4 Configuration Updates
|
||||||
|
**File:** `config.yaml` (updates to STORY-01/02)
|
||||||
|
|
||||||
|
New fields:
|
||||||
|
```yaml
|
||||||
|
sampling:
|
||||||
|
interval_seconds: 30
|
||||||
|
override_per_job: true
|
||||||
|
quality: 2 # JPEG quality (1-31, lower=better)
|
||||||
|
format: jpeg
|
||||||
|
|
||||||
|
resolution:
|
||||||
|
max_height: 1080
|
||||||
|
preserve_aspect_ratio: true
|
||||||
|
pad_to_square: false # Will be done in face detection if needed
|
||||||
|
|
||||||
|
scratch:
|
||||||
|
path: /scratch
|
||||||
|
max_usage_percent: 80
|
||||||
|
auto_cleanup: true
|
||||||
|
cleanup_on_error: true
|
||||||
|
```
|
||||||
|
|
||||||
|
## Acceptance Criteria
|
||||||
|
|
||||||
|
### Functional
|
||||||
|
- [ ] Frames are extracted at uniform intervals (±1 frame tolerance) across video duration
|
||||||
|
- [ ] Default interval of 30 seconds produces correct number of frames for test videos
|
||||||
|
- [ ] Per-job interval override works (e.g., 10-second interval for specific videos)
|
||||||
|
- [ ] Variable FPS videos produce evenly spaced timestamps (not evenly spaced frames)
|
||||||
|
- [ ] 4K frames are downsampled to 1080p without crashing
|
||||||
|
- [ ] Aspect ratio is preserved during downscaling (no distortion)
|
||||||
|
- [ ] Frames are saved as high-quality JPEGs (quality factor 2)
|
||||||
|
- [ ] Frame naming follows convention: `{video_id}_{timestamp_ms}.jpg`
|
||||||
|
- [ ] Scratch space is cleaned up after video processing (no leftover frames)
|
||||||
|
- [ ] Failed frame extraction does not stop processing of other frames
|
||||||
|
|
||||||
|
### Non-Functional
|
||||||
|
- [ ] Frame extraction for a 15-minute video completes in < 10 seconds on NVMe
|
||||||
|
- [ ] Scratch space usage per video < 50MB (typical case)
|
||||||
|
- [ ] Memory usage during extraction < 500MB per worker
|
||||||
|
- [ ] FFmpeg process timeout enforced (30 seconds per frame)
|
||||||
|
- [ ] No frames left in scratch after processing completes (verified by directory check)
|
||||||
|
|
||||||
|
### Technical Constraints
|
||||||
|
- [ ] FFmpeg uses `-ss` for accurate timestamp seeking (not frame-by-frame)
|
||||||
|
- [ ] Downscaling uses FFmpeg scale filter (not post-processing resize)
|
||||||
|
- [ ] JPEG quality is consistent across all extracted frames
|
||||||
|
- [ ] Timestamps are in seconds with millisecond precision
|
||||||
|
- [ ] Scratch directory is created per-video and isolated from other workers
|
||||||
|
|
||||||
|
## Dependencies
|
||||||
|
- **Prerequisites:** STORY-01 (Foundation), STORY-02 (Ingestion — provides metadata)
|
||||||
|
- **Depends on:** None (runs after ingestion, before STORY-04)
|
||||||
|
- **Enables:** STORY-04 (Face Detection — provides frames), STORY-05 (Classification)
|
||||||
|
|
||||||
|
## Risks & Mitigations
|
||||||
|
| Risk | Mitigation |
|
||||||
|
|------|--|
|
||||||
|
| FFmpeg seeking is inaccurate for some codecs | Use `-accurate_seek` flag; fall back to frame-by-frame if needed |
|
||||||
|
| 4K downscaling increases processing time | Use FFmpeg scale filter (GPU-accelerated if available) |
|
||||||
|
| Scratch space fills up with many concurrent videos | Monitor usage; implement cleanup queue; alert at 80% |
|
||||||
|
| Variable FPS causes uneven frame distribution | Calculate timestamps based on duration, not frame count |
|
||||||
|
|
||||||
|
## Estimated Effort
|
||||||
|
- **Sprint:** 5 (first half)
|
||||||
|
- **Story Points:** 13
|
||||||
|
- **Dependencies:** STORY-01, STORY-02
|
||||||
+168
@@ -0,0 +1,168 @@
|
|||||||
|
# STORY-04: Face Detection
|
||||||
|
|
||||||
|
## Epic
|
||||||
|
**E1: Core Pipeline** — As an engineer, I can run face detection on sampled frames in batch.
|
||||||
|
|
||||||
|
## Related Requirements
|
||||||
|
| ID | Requirement |
|
||||||
|
|----|-------|
|
||||||
|
| FR-02 | Face detection on all sampled frames: Lightweight detector only; no full-body or scene analysis |
|
||||||
|
| NFR-03 | GPU Memory Safety: ≤ 18GB per GPU sustained |
|
||||||
|
| NFR-01 | Throughput: ≥ 30 videos/hour/GPU |
|
||||||
|
| TC-01 | Hardware: 2× Tesla P40 24GB (compute capability 5.2, PCIe 3.0, no Tensor Cores) |
|
||||||
|
| TC-02 | CUDA/Torch Compatibility: CUDA ≤ 11.8, PyTorch ≤ 2.1.0, FP32 inference only |
|
||||||
|
| TC-04 | Framework Stack: PyTorch → ONNX → TensorRT FP32 |
|
||||||
|
|
||||||
|
## Description
|
||||||
|
Implement face detection on all sampled frames using a lightweight detector (YOLOv8n or equivalent). Convert the model to ONNX and TensorRT FP32 for inference. Implement dynamic batching to maximize GPU utilization while staying within VRAM constraints. Generate face crops for classification.
|
||||||
|
|
||||||
|
## Scope
|
||||||
|
|
||||||
|
### In Scope
|
||||||
|
- YOLOv8n face detection model (or equivalent lightweight detector)
|
||||||
|
- ONNX export and TensorRT FP32 engine conversion
|
||||||
|
- Dynamic batching for face detection inference
|
||||||
|
- Face crop extraction and resizing to inference input size (224×224)
|
||||||
|
- GPU memory management (stay within 18GB per GPU)
|
||||||
|
- Batch size auto-tuning based on available VRAM
|
||||||
|
- Face bounding box output with confidence scores
|
||||||
|
- Handling frames with no detected faces
|
||||||
|
|
||||||
|
### Out of Scope
|
||||||
|
- Frame sampling (covered in STORY-03)
|
||||||
|
- Demographic classification (covered in STORY-05)
|
||||||
|
- Confidence aggregation (covered in STORY-05)
|
||||||
|
- Results export (covered in STORY-06)
|
||||||
|
- Review UI (covered in STORY-07)
|
||||||
|
|
||||||
|
## Deliverables
|
||||||
|
|
||||||
|
### 4.1 Face Detection Model
|
||||||
|
**File:** `models/face_detector/`
|
||||||
|
|
||||||
|
Components:
|
||||||
|
- **Source Model:** YOLOv8n (Ultralytics) trained on face detection dataset (WIDER Face or equivalent)
|
||||||
|
- **ONNX Export:** `face_detector.onnx` with proper input/output shapes
|
||||||
|
- Input: `[1, 3, 640, 640]` (RGB, BGR depending on model)
|
||||||
|
- Output: `[1, num_anchors, 4+1]` (bbox coordinates + confidence)
|
||||||
|
- **TensorRT Engine:** `face_detector.trt` (FP32, built for Tesla P40 CC 5.2)
|
||||||
|
- Builder config: max_batch_size=32, max_workspace_size=4GB
|
||||||
|
- Serialization for runtime loading
|
||||||
|
- **Model Metadata:** `model.json` with input shape, normalization params, calibration data
|
||||||
|
|
||||||
|
### 4.2 Face Detection Runner
|
||||||
|
**File:** `src/face_detector.py`
|
||||||
|
|
||||||
|
Core components:
|
||||||
|
- **Model Loader:** Load TensorRT engine at startup
|
||||||
|
```python
|
||||||
|
class FaceDetector:
|
||||||
|
def __init__(self, engine_path, device='cuda'):
|
||||||
|
self.engine = load_trt_engine(engine_path)
|
||||||
|
self.context = self.engine.create_execution_context()
|
||||||
|
self.input_shape = (1, 3, 640, 640)
|
||||||
|
self.device = device
|
||||||
|
```
|
||||||
|
- **Preprocessing:** Convert frame to model input format
|
||||||
|
- Resize to 640×640
|
||||||
|
- Normalize (mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])
|
||||||
|
- Convert to NHWC/NCHW as required by TensorRT
|
||||||
|
- **Inference:** Run batched detection on multiple frames
|
||||||
|
- Dynamic batching: accumulate frames until batch_size reached or timeout
|
||||||
|
- Execute TensorRT engine
|
||||||
|
- Parse outputs (bounding boxes + confidence scores)
|
||||||
|
- **Post-processing:** NMS (Non-Maximum Suppression) to remove duplicate detections
|
||||||
|
- IoU threshold: 0.45
|
||||||
|
- Confidence threshold: 0.25 (low to avoid missing faces)
|
||||||
|
- **Crop Generation:** Extract face crops from original frames
|
||||||
|
- Resize crops to 224×224 for classification input
|
||||||
|
- Save crops to `/scratch/{video_id}/crops/`
|
||||||
|
|
||||||
|
### 4.3 Dynamic Batching
|
||||||
|
**File:** `src/batcher.py`
|
||||||
|
|
||||||
|
Features:
|
||||||
|
- **Batch Accumulator:** Collect face detections across frames
|
||||||
|
- Max batch size: auto-tuned based on available VRAM (start at 16, adjust dynamically)
|
||||||
|
- Timeout: 100ms (process batch even if not full)
|
||||||
|
- **VRAM Monitoring:** Track GPU memory usage and adjust batch size
|
||||||
|
- If VRAM > 16GB: reduce batch size by 25%
|
||||||
|
- If VRAM < 10GB: increase batch size by 25%
|
||||||
|
- **Frame Ordering:** Maintain frame-to-batch mapping for correct crop assignment
|
||||||
|
|
||||||
|
### 4.4 GPU Memory Management
|
||||||
|
**File:** `src/gpu_manager.py`
|
||||||
|
|
||||||
|
Features:
|
||||||
|
- **Memory Tracking:** Monitor GPU memory usage continuously
|
||||||
|
- **Peak Allocation:** Ensure total GPU memory < 18GB (leaving 6GB headroom)
|
||||||
|
- **Cleanup:** Explicitly free TensorRT buffers after each batch
|
||||||
|
- **Per-Video Limits:** Max faces per video (e.g., 100) to prevent memory issues
|
||||||
|
- **Fallback:** If VRAM is critically low, reduce batch size and retry
|
||||||
|
|
||||||
|
### 4.5 Configuration Updates
|
||||||
|
**File:** `config.yaml` (updates)
|
||||||
|
|
||||||
|
New fields:
|
||||||
|
```yaml
|
||||||
|
face_detection:
|
||||||
|
model: yolo8n
|
||||||
|
model_path: /models/face_detector/face_detector.trt
|
||||||
|
input_size: 640
|
||||||
|
confidence_threshold: 0.25
|
||||||
|
iou_threshold: 0.45
|
||||||
|
max_faces_per_frame: 10
|
||||||
|
max_faces_per_video: 100
|
||||||
|
|
||||||
|
batching:
|
||||||
|
max_batch_size: 16
|
||||||
|
batch_timeout_ms: 100
|
||||||
|
vram_target_gb: 16
|
||||||
|
vram_reduce_threshold_gb: 16
|
||||||
|
vram_increase_threshold_gb: 10
|
||||||
|
```
|
||||||
|
|
||||||
|
## Acceptance Criteria
|
||||||
|
|
||||||
|
### Functional
|
||||||
|
- [ ] YOLOv8n model loads and runs inference via TensorRT FP32
|
||||||
|
- [ ] Face detection correctly identifies faces in test frames (mAP > 0.50 on WIDER Face val)
|
||||||
|
- [ ] No faces detected in frames without faces (false positive rate < 5%)
|
||||||
|
- [ ] Face crops are correctly extracted and resized to 224×224
|
||||||
|
- [ ] Dynamic batching works correctly (multiple frames processed in single batch)
|
||||||
|
- [ ] Batch size adjusts based on available VRAM
|
||||||
|
- [ ] Frames with no detected faces are handled gracefully (no crashes, no crops generated)
|
||||||
|
- [ ] NMS correctly removes duplicate detections (IoU > 0.45)
|
||||||
|
- [ ] Face bounding boxes are accurate (center coordinates within 10 pixels of ground truth)
|
||||||
|
|
||||||
|
### Non-Functional
|
||||||
|
- [ ] GPU memory usage stays ≤ 18GB per GPU during face detection
|
||||||
|
- [ ] Face detection for 10 frames (batched) completes in < 2 seconds on Tesla P40
|
||||||
|
- [ ] Batch processing throughput: ≥ 50 faces/second/GPU
|
||||||
|
- [ ] Model loading time < 5 seconds at startup
|
||||||
|
- [ ] No GPU OOM errors during extended processing
|
||||||
|
|
||||||
|
### Technical Constraints
|
||||||
|
- [ ] TensorRT engine is FP32 only (no FP16, no INT8)
|
||||||
|
- [ ] No Tensor Cores used (CC 5.2 constraint)
|
||||||
|
- [ ] CUDA 11.8 compatible (verified via torch.version.cuda)
|
||||||
|
- [ ] ONNX model exports without errors (verified via onnx.checker)
|
||||||
|
- [ ] Face crops are saved in JPEG format at high quality
|
||||||
|
|
||||||
|
## Dependencies
|
||||||
|
- **Prerequisites:** STORY-01 (Foundation), STORY-03 (Frame Sampling — provides frames)
|
||||||
|
- **Depends on:** None (runs after frame sampling)
|
||||||
|
- **Enables:** STORY-05 (Classification — provides face crops)
|
||||||
|
|
||||||
|
## Risks & Mitigations
|
||||||
|
| Risk | Mitigation |
|
||||||
|
|------|--|
|
||||||
|
| Tesla P40 (CC 5.2) has no Tensor Cores — slower inference | Use TensorRT engine caching; maximize batch size to compensate |
|
||||||
|
| YOLOv8n may be too large for VRAM with classification | Use YOLOv8n (smallest variant); monitor VRAM closely |
|
||||||
|
| Many faces per frame could cause VRAM spike | Limit max faces per frame; reduce batch size if needed |
|
||||||
|
| TensorRT engine build time is long | Pre-build engines; cache on disk; skip rebuild if model unchanged |
|
||||||
|
|
||||||
|
## Estimated Effort
|
||||||
|
- **Sprint:** 5-6 (second half)
|
||||||
|
- **Story Points:** 21
|
||||||
|
- **Dependencies:** STORY-01, STORY-03
|
||||||
+187
@@ -0,0 +1,187 @@
|
|||||||
|
# STORY-05: Classification & Confidence Aggregation
|
||||||
|
|
||||||
|
## Epic
|
||||||
|
**E1: Core Pipeline** — I can classify face crops and aggregate video confidence.
|
||||||
|
|
||||||
|
## Related Requirements
|
||||||
|
| ID | Requirement |
|
||||||
|
|----|------|
|
||||||
|
| FR-03 | Binary demographic classification of detected face crops: Outputs probability p ∈ [0,1] for target class |
|
||||||
|
| FR-04 | Video-level confidence aggregation & threshold routing: Aggregates frame-level scores → video confidence C; Routes to MATCH, REVIEW, or SKIP |
|
||||||
|
| NFR-01 | Throughput: ≥ 30 videos/hour/GPU |
|
||||||
|
| NFR-03 | GPU Memory Safety: ≤ 18GB per GPU sustained |
|
||||||
|
| NFR-04 | Determinism & Reproducibility: Config-seeded randomness, versioned models |
|
||||||
|
| TC-01 | Hardware: 2× Tesla P40 24GB (compute capability 5.2, PCIe 3.0, no Tensor Cores) |
|
||||||
|
| TC-02 | CUDA/Torch Compatibility: CUDA ≤ 11.8, PyTorch ≤ 2.1.0, FP32 inference only |
|
||||||
|
|
||||||
|
## Description
|
||||||
|
Implement demographic classification of detected face crops using a lightweight classifier (MobileNetV3 or equivalent). Apply temperature scaling for calibrated confidence scores. Aggregate frame-level confidence to video-level confidence using configurable aggregation logic. Route videos to MATCH, REVIEW, or SKIP based on threshold comparison.
|
||||||
|
|
||||||
|
## Scope
|
||||||
|
|
||||||
|
### In Scope
|
||||||
|
- MobileNetV3 (or equivalent) classification model via TensorRT FP32
|
||||||
|
- Temperature scaling for confidence calibration
|
||||||
|
- Frame-level confidence computation (softmax with temperature)
|
||||||
|
- Video-level confidence aggregation (max or weighted mean, configurable)
|
||||||
|
- Threshold-based routing (T_high, T_low)
|
||||||
|
- Routing decision assignment (MATCH, REVIEW, SKIP)
|
||||||
|
- Confidence score persistence to DB and output files
|
||||||
|
|
||||||
|
### Out of Scope
|
||||||
|
- Frame sampling (covered in STORY-03)
|
||||||
|
- Face detection (covered in STORY-04)
|
||||||
|
- Results export format (covered in STORY-06)
|
||||||
|
- Review UI (covered in STORY-07)
|
||||||
|
- Active learning / model retraining (covered in STORY-08)
|
||||||
|
|
||||||
|
## Deliverables
|
||||||
|
|
||||||
|
### 5.1 Classification Model
|
||||||
|
**File:** `models/classifier/`
|
||||||
|
|
||||||
|
Components:
|
||||||
|
- **Source Model:** MobileNetV3-small (or equivalent lightweight classifier)
|
||||||
|
- Trained on demographic dataset (fairness considerations documented)
|
||||||
|
- Binary classification: Black male subject (class 1) vs. not (class 0)
|
||||||
|
- **ONNX Export:** `classifier.onnx`
|
||||||
|
- Input: `[1, 3, 224, 224]` (RGB, resized face crop)
|
||||||
|
- Output: `[1, 2]` (logits for class 0 and class 1)
|
||||||
|
- **TensorRT Engine:** `classifier.trt` (FP32)
|
||||||
|
- Builder config: max_batch_size=32, max_workspace_size=2GB
|
||||||
|
- **Calibration Data:** Temperature parameter T stored with model
|
||||||
|
- **Model Metadata:** `model.json` with architecture, training dataset, validation metrics
|
||||||
|
|
||||||
|
### 5.2 Classification Runner
|
||||||
|
**File:** `src/classifier.py`
|
||||||
|
|
||||||
|
Core components:
|
||||||
|
- **Model Loader:** Load TensorRT engine at startup
|
||||||
|
```python
|
||||||
|
class FaceClassifier:
|
||||||
|
def __init__(self, engine_path, temperature=1.0, device='cuda'):
|
||||||
|
self.engine = load_trt_engine(engine_path)
|
||||||
|
self.context = self.engine.create_execution_context()
|
||||||
|
self.temperature = temperature
|
||||||
|
self.input_shape = (1, 3, 224, 224)
|
||||||
|
```
|
||||||
|
- **Preprocessing:** Convert face crop to model input
|
||||||
|
- Resize to 224×224
|
||||||
|
- Normalize (ImageNet statistics)
|
||||||
|
- Convert to tensor (NCHW format)
|
||||||
|
- **Inference:** Run classification on face crops
|
||||||
|
- Dynamic batching (shared with face detection batcher)
|
||||||
|
- Execute TensorRT engine
|
||||||
|
- Get raw logits output
|
||||||
|
- **Temperature Scaling:** Apply temperature to logits before softmax
|
||||||
|
```python
|
||||||
|
def calibrated_softmax(logits, temperature):
|
||||||
|
scaled_logits = logits / temperature
|
||||||
|
return softmax(scaled_logits, axis=-1)
|
||||||
|
```
|
||||||
|
- **Frame-level Confidence:** Extract probability for target class (class 1)
|
||||||
|
```python
|
||||||
|
p_i = calibrated_probs[:, 1] # probability of target class
|
||||||
|
```
|
||||||
|
|
||||||
|
### 5.3 Confidence Aggregation
|
||||||
|
**File:** `src/aggregator.py`
|
||||||
|
|
||||||
|
Features:
|
||||||
|
- **Aggregation Strategies (configurable):**
|
||||||
|
1. **Max:** `C = max(p_i)` — use highest confidence frame
|
||||||
|
2. **Weighted Mean:** `C = softmax(α·mean(p_i) + β·var(p_i))`
|
||||||
|
3. **Top-K Mean:** `C = mean(top_k(p_i))` — average of top K confidences
|
||||||
|
- **Default Strategy:** Max (most conservative, aligns with FR-04)
|
||||||
|
- **Parameter Configuration:** α, β, K configurable in config.yaml
|
||||||
|
- **Variance Calculation:** Compute variance of frame-level confidences (for uncertainty estimation)
|
||||||
|
|
||||||
|
### 5.4 Threshold Routing
|
||||||
|
**File:** `src/router.py`
|
||||||
|
|
||||||
|
Features:
|
||||||
|
- **Threshold Comparison:**
|
||||||
|
```python
|
||||||
|
if C >= T_high: # default 0.75
|
||||||
|
routing = 'MATCH'
|
||||||
|
elif C >= T_low: # default 0.45
|
||||||
|
routing = 'REVIEW'
|
||||||
|
else:
|
||||||
|
routing = 'SKIP'
|
||||||
|
```
|
||||||
|
- **Configurable Thresholds:** T_high and T_low in config.yaml
|
||||||
|
- **Routing Decision Logging:** Log routing decision with confidence score
|
||||||
|
- **Edge Case Handling:**
|
||||||
|
- No faces detected → routing = SKIP (with confidence = 0.0)
|
||||||
|
- All frames have same confidence → routing based on threshold comparison
|
||||||
|
- Confidence exactly at threshold → use >= comparison (inclusive)
|
||||||
|
|
||||||
|
### 5.5 Configuration Updates
|
||||||
|
**File:** `config.yaml` (updates)
|
||||||
|
|
||||||
|
New fields:
|
||||||
|
```yaml
|
||||||
|
classifier:
|
||||||
|
model: mobilenetv3-small
|
||||||
|
model_path: /models/classifier/classifier.trt
|
||||||
|
input_size: 224
|
||||||
|
temperature: 1.0 # calibration temperature
|
||||||
|
default_strategy: max # max, weighted_mean, top_k_mean
|
||||||
|
|
||||||
|
aggregation:
|
||||||
|
strategy: max
|
||||||
|
alpha: 1.0 # for weighted_mean
|
||||||
|
beta: 0.1 # for weighted_mean
|
||||||
|
top_k: 3 # for top_k_mean
|
||||||
|
|
||||||
|
routing:
|
||||||
|
T_high: 0.75
|
||||||
|
T_low: 0.45
|
||||||
|
no_faces_decision: SKIP
|
||||||
|
```
|
||||||
|
|
||||||
|
## Acceptance Criteria
|
||||||
|
|
||||||
|
### Functional
|
||||||
|
- [ ] MobileNetV3 model loads and runs inference via TensorRT FP32
|
||||||
|
- [ ] Classification outputs calibrated probability p ∈ [0,1] for target class
|
||||||
|
- [ ] Temperature scaling is applied correctly (verified on test set)
|
||||||
|
- [ ] Frame-level confidence scores are deterministic (same input → same output)
|
||||||
|
- [ ] Max aggregation produces C = max(p_i) correctly
|
||||||
|
- [ ] Weighted mean aggregation produces correct result with configurable α, β
|
||||||
|
- [ ] Threshold routing assigns exactly one of: MATCH, REVIEW, SKIP
|
||||||
|
- [ ] Videos with no detected faces are routed to SKIP with confidence 0.0
|
||||||
|
- [ ] Confidence exactly at threshold uses >= comparison (inclusive)
|
||||||
|
|
||||||
|
### Non-Functional
|
||||||
|
- [ ] Classification for 16 face crops (batched) completes in < 1 second on Tesla P40
|
||||||
|
- [ ] GPU memory usage stays ≤ 18GB per GPU during classification
|
||||||
|
- [ ] Aggregation computation is negligible (< 10ms per video)
|
||||||
|
- [ ] Temperature parameter is stored with model checkpoint
|
||||||
|
- [ ] All routing decisions are logged with confidence score and threshold values
|
||||||
|
|
||||||
|
### Technical Constraints
|
||||||
|
- [ ] TensorRT engine is FP32 only
|
||||||
|
- [ ] No Tensor Cores used (CC 5.2 constraint)
|
||||||
|
- [ ] CUDA 11.8 compatible
|
||||||
|
- [ ] ONNX model exports without errors
|
||||||
|
- [ ] Confidence scores are reproducible (deterministic inference)
|
||||||
|
- [ ] Aggregation strategy is configurable without code changes
|
||||||
|
|
||||||
|
## Dependencies
|
||||||
|
- **Prerequisites:** STORY-01 (Foundation), STORY-03 (Frame Sampling), STORY-04 (Face Detection — provides crops)
|
||||||
|
- **Depends on:** None (runs after face detection)
|
||||||
|
- **Enables:** STORY-06 (Results Persistence), STORY-07 (Review UI — provides routing decisions)
|
||||||
|
|
||||||
|
## Risks & Mitigations
|
||||||
|
| Risk | Mitigation |
|
||||||
|
|------|--|
|
||||||
|
| Temperature scaling parameters need tuning | Evaluate on held-out calibration set during training cycle |
|
||||||
|
| Max aggregation may be overly conservative | Offer weighted mean as alternative; allow per-job strategy selection |
|
||||||
|
| Model bias concerns with demographic classification | Document training dataset; audit fairness metrics; legal review required |
|
||||||
|
| TensorRT engine build time | Pre-build and cache engines; version control model files |
|
||||||
|
|
||||||
|
## Estimated Effort
|
||||||
|
- **Sprint:** 5-6 (second half)
|
||||||
|
- **Story Points:** 21
|
||||||
|
- **Dependencies:** STORY-01, STORY-03, STORY-04
|
||||||
+181
@@ -0,0 +1,181 @@
|
|||||||
|
# STORY-06: Results Persistence & Export
|
||||||
|
|
||||||
|
## Epic
|
||||||
|
**E1: Core Pipeline** — As a system, I can persist processing results and export data for analytics.
|
||||||
|
|
||||||
|
## Related Requirements
|
||||||
|
| ID | Requirement |
|
||||||
|
|----|------|
|
||||||
|
| FR-07 | Metadata logging & audit trail: Stores video ID, timestamps, frame counts, confidence scores, routing decision, model version |
|
||||||
|
| FR-04 | Video-level confidence aggregation & threshold routing: Results saved to Parquet |
|
||||||
|
| NFR-05 | Fault Tolerance: Metadata persisted before cleanup |
|
||||||
|
| NFR-07 | Data Volume Handling: Efficient indexing for ~30TB dataset |
|
||||||
|
| TC-03 | Storage I/O: Shared NAS/SMB for video input/output |
|
||||||
|
|
||||||
|
## Description
|
||||||
|
Implement result persistence, metadata logging, and data export. Update the database with processing results, insert processing log entries, and export summary data in Parquet/JSONL format for analytics. Ensure all persistence happens before scratch space cleanup to prevent data loss.
|
||||||
|
|
||||||
|
## Scope
|
||||||
|
|
||||||
|
### In Scope
|
||||||
|
- Database update for processed videos (status, confidence, routing, timestamps)
|
||||||
|
- Processing log entry insertion (atomic transaction)
|
||||||
|
- Parquet/JSONL export to `/data/output`
|
||||||
|
- Confidence score persistence (frame-level and video-level)
|
||||||
|
- Model version tracking in DB and export
|
||||||
|
- Idempotent result persistence (re-processing doesn't duplicate entries)
|
||||||
|
- Transaction-safe updates (all-or-nothing)
|
||||||
|
|
||||||
|
### Out of Scope
|
||||||
|
- Frame sampling (covered in STORY-03)
|
||||||
|
- Face detection (covered in STORY-04)
|
||||||
|
- Classification (covered in STORY-05)
|
||||||
|
- Review UI (covered in STORY-07)
|
||||||
|
- Active learning pipeline (covered in STORY-08)
|
||||||
|
|
||||||
|
## Deliverables
|
||||||
|
|
||||||
|
### 6.1 Result Updater
|
||||||
|
**File:** `src/result_updater.py`
|
||||||
|
|
||||||
|
Features:
|
||||||
|
- **Video Record Update:** Update `videos` table with processing results
|
||||||
|
```sql
|
||||||
|
UPDATE videos SET
|
||||||
|
status = 'COMPLETED',
|
||||||
|
last_processed_time = NOW(),
|
||||||
|
confidence_score = %s,
|
||||||
|
routing_decision = %s,
|
||||||
|
model_version = %s,
|
||||||
|
frame_count = %s,
|
||||||
|
updated_at = NOW()
|
||||||
|
WHERE id = %s AND status = 'PROCESSING'
|
||||||
|
```
|
||||||
|
- **Atomic Transaction:** Wrap video update + log insert in single transaction
|
||||||
|
- **State Guard:** Only update if status is PROCESSING (prevents duplicate processing)
|
||||||
|
- **Error Handling:** Rollback transaction on any error; re-queue video as PENDING
|
||||||
|
|
||||||
|
### 6.2 Processing Logger
|
||||||
|
**File:** `src/processing_logger.py`
|
||||||
|
|
||||||
|
Features:
|
||||||
|
- **Log Entry Insertion:** Insert record into `processing_logs` table
|
||||||
|
```python
|
||||||
|
def insert_processing_log(video_id, model_version, frame_count,
|
||||||
|
confidence_score, routing_decision):
|
||||||
|
cursor.execute("""
|
||||||
|
INSERT INTO processing_logs
|
||||||
|
(video_id, model_version, frame_count, confidence_score,
|
||||||
|
routing_decision, processed_at)
|
||||||
|
VALUES (%s, %s, %s, %s, %s, NOW())
|
||||||
|
""", (video_id, model_version, frame_count,
|
||||||
|
confidence_score, routing_decision))
|
||||||
|
```
|
||||||
|
- **Frame-Level Confidence Storage:** Store all frame-level confidence scores
|
||||||
|
- Option A: JSON array in processing_logs.confidence_scores (TEXT column)
|
||||||
|
- Option B: Separate `frame_confidences` table (for large datasets)
|
||||||
|
- Default: JSON array in processing_logs (simpler, sufficient for typical frame counts)
|
||||||
|
- **Audit Trail:** All log entries include timestamp, model version, and routing decision
|
||||||
|
|
||||||
|
### 6.3 Data Export
|
||||||
|
**File:** `src/data_export.py`
|
||||||
|
|
||||||
|
Features:
|
||||||
|
- **Parquet Export:** Write summary data to Parquet format
|
||||||
|
```python
|
||||||
|
# Schema:
|
||||||
|
# video_id (int64), file_path (string), file_hash (string),
|
||||||
|
# model_version (string), sample_count (int32),
|
||||||
|
# confidence_scores (list<float>), video_confidence (float),
|
||||||
|
# routing (string), processed_at (timestamp)
|
||||||
|
```
|
||||||
|
- **JSONL Export:** Alternative line-delimited JSON format for streaming analytics
|
||||||
|
```json
|
||||||
|
{"video_id": 12345, "file_path": "/data/input/video.mp4",
|
||||||
|
"model_version": "v1.2.0", "sample_count": 8,
|
||||||
|
"confidence_scores": [0.82, 0.79, 0.85, 0.76, 0.81, 0.83, 0.78, 0.80],
|
||||||
|
"video_confidence": 0.85, "routing": "MATCH", "processed_at": "2026-08-03T10:30:00Z"}
|
||||||
|
```
|
||||||
|
- **Output Directory:** `/data/output/{model_version}/` organized by model version
|
||||||
|
- **File Naming:** `{model_version}_{batch_id}.parquet` and `{model_version}_{batch_id}.jsonl`
|
||||||
|
- **Batch ID:** Sequential counter or timestamp-based (e.g., `v1.2.0_20260803_103000`)
|
||||||
|
- **Compression:** Parquet with Snappy compression
|
||||||
|
- **Export Trigger:** Export after all videos in batch are processed (or periodically)
|
||||||
|
|
||||||
|
### 6.4 Scratch Cleanup
|
||||||
|
**File:** `src/scratch_manager.py` (cleanup section)
|
||||||
|
|
||||||
|
Features:
|
||||||
|
- **Post-Processing Cleanup:** Delete all frames and crops after results are persisted
|
||||||
|
- **Cleanup Order:** Results → Logs → Export → Cleanup (critical ordering)
|
||||||
|
- **Cleanup Verification:** Verify DB update before deleting scratch files
|
||||||
|
- **Cleanup on Error:** If persistence fails, do NOT delete scratch (allow retry)
|
||||||
|
- **Cleanup Logging:** Log cleanup completion with video_id and files removed
|
||||||
|
|
||||||
|
### 6.5 Configuration Updates
|
||||||
|
**File:** `config.yaml` (updates)
|
||||||
|
|
||||||
|
New fields:
|
||||||
|
```yaml
|
||||||
|
results:
|
||||||
|
persist_before_cleanup: true
|
||||||
|
transaction_safe: true
|
||||||
|
state_guard: true # only update PROCESSING → COMPLETED
|
||||||
|
|
||||||
|
export:
|
||||||
|
format: parquet # parquet, jsonl, or both
|
||||||
|
output_path: /data/output
|
||||||
|
compression: snappy
|
||||||
|
batch_size: 100 # export after N videos
|
||||||
|
include_frame_confidences: true
|
||||||
|
```
|
||||||
|
|
||||||
|
## Acceptance Criteria
|
||||||
|
|
||||||
|
### Functional
|
||||||
|
- [ ] Video status transitions from PROCESSING to COMPLETED after successful processing
|
||||||
|
- [ ] Processing log entry is inserted for every processed video
|
||||||
|
- [ ] Frame-level confidence scores are stored (all p_i values)
|
||||||
|
- [ ] Video-level confidence score is stored correctly
|
||||||
|
- [ ] Routing decision is persisted accurately
|
||||||
|
- [ ] Model version is recorded in both videos and processing_logs tables
|
||||||
|
- [ ] Parquet export produces valid, readable files with correct schema
|
||||||
|
- [ ] JSONL export produces valid JSON lines with correct fields
|
||||||
|
- [ ] Export files are organized by model version in /data/output/
|
||||||
|
- [ ] Scratch space is cleaned up after persistence (verified by directory check)
|
||||||
|
- [ ] Re-processing a video updates existing records (no duplicates)
|
||||||
|
- [ ] Transaction rollback on error prevents partial state updates
|
||||||
|
|
||||||
|
### Non-Functional
|
||||||
|
- [ ] Result persistence completes in < 5 seconds per video
|
||||||
|
- [ ] Parquet export for 100 videos completes in < 10 seconds
|
||||||
|
- [ ] JSONL export for 100 videos completes in < 10 seconds
|
||||||
|
- [ ] Export files are valid (verified by parquet.read_table and jsonl parsing)
|
||||||
|
- [ ] Scratch cleanup frees > 95% of allocated space per video
|
||||||
|
- [ ] No data loss if worker crashes between processing and persistence (PROCESSING state preserved)
|
||||||
|
|
||||||
|
### Technical Constraints
|
||||||
|
- [ ] All DB updates use parameterized queries (no SQL injection)
|
||||||
|
- [ ] Transaction isolation level is READ COMMITTED or higher
|
||||||
|
- [ ] Parquet files use Snappy compression (not gzip, for speed)
|
||||||
|
- [ ] JSONL files are UTF-8 encoded
|
||||||
|
- [ ] Confidence scores are stored with float64 precision
|
||||||
|
- [ ] Timestamps are in UTC (ISO 8601 format)
|
||||||
|
|
||||||
|
## Dependencies
|
||||||
|
- **Prerequisites:** STORY-01 (Foundation — DB schema), STORY-05 (Classification — provides results)
|
||||||
|
- **Depends on:** None (runs after classification)
|
||||||
|
- **Enables:** STORY-07 (Review UI — provides routing decisions), STORY-08 (Active Learning — provides training data)
|
||||||
|
|
||||||
|
## Risks & Mitigations
|
||||||
|
| Risk | Mitigation |
|
||||||
|
|------|--|
|
||||||
|
| NAS write latency slows export | Batch writes; use local tmpfs for export staging |
|
||||||
|
| Parquet library compatibility with CUDA container | Test pyarrow in Worker container early; pin version |
|
||||||
|
| Transaction rollback leaves video in PROCESSING state | DB lock timeout (5 min) prevents permanent lock |
|
||||||
|
| Scratch cleanup before persistence causes data loss | Enforce ordering: persist → verify → cleanup |
|
||||||
|
|
||||||
|
## Estimated Effort
|
||||||
|
- **Sprint:** 5-6 (second half)
|
||||||
|
- **Story Points:** 13
|
||||||
|
- **Dependencies:** STORY-01, STORY-05
|
||||||
+216
@@ -0,0 +1,216 @@
|
|||||||
|
# STORY-07: Review Interface
|
||||||
|
|
||||||
|
## Epic
|
||||||
|
**E2: Routing & Review** — As an annotator, I can view low-confidence videos and label them.
|
||||||
|
|
||||||
|
## Related Requirements
|
||||||
|
| ID | Requirement |
|
||||||
|
|----|------|
|
||||||
|
| FR-05 | Manual review interface for low-confidence videos: Displays video + contributing frames/crops + model confidence; Supports binary labeling |
|
||||||
|
| FR-07 | Metadata logging & audit trail: Stores annotated labels for active learning |
|
||||||
|
| NFR-06 | Observability: Tracks review queue depth |
|
||||||
|
| TC-06 | Network Security: Internal LAN only; no auth required |
|
||||||
|
|
||||||
|
## Description
|
||||||
|
Implement a lightweight web-based review interface for annotating low-confidence videos. Display the video player, top-k contributing frames, model confidence scores, and allow annotators to toggle the ground truth label. Support CSV/JSON export of annotated data for active learning.
|
||||||
|
|
||||||
|
## Scope
|
||||||
|
|
||||||
|
### In Scope
|
||||||
|
- Lightweight web UI (Flask/FastAPI) serving on internal LAN
|
||||||
|
- Query DB for videos with routing_decision = REVIEW
|
||||||
|
- Video player with playback controls
|
||||||
|
- Display top-k contributing frames (highest confidence frames)
|
||||||
|
- Display model confidence scores per frame
|
||||||
|
- Binary label toggle (True/False — target class present or not)
|
||||||
|
- Label persistence to DB (review_queue table)
|
||||||
|
- CSV/JSON export of annotated data with ground truth
|
||||||
|
- Accessible via internal IP:Port (no auth, no SSL)
|
||||||
|
|
||||||
|
### Out of Scope
|
||||||
|
- Frame sampling (covered in STORY-03)
|
||||||
|
- Face detection (covered in STORY-04)
|
||||||
|
- Classification (covered in STORY-05)
|
||||||
|
- Confidence aggregation (covered in STORY-05)
|
||||||
|
- Active learning pipeline / model retraining (covered in STORY-08)
|
||||||
|
- Monitoring dashboards (covered in STORY-09)
|
||||||
|
|
||||||
|
## Deliverables
|
||||||
|
|
||||||
|
### 7.1 Review Backend
|
||||||
|
**File:** `src/review_api.py`
|
||||||
|
|
||||||
|
API Endpoints:
|
||||||
|
- `GET /api/review/queue` — List videos in review queue
|
||||||
|
- Query: `SELECT * FROM review_queue WHERE annotated = false ORDER BY created_at DESC`
|
||||||
|
- Pagination: 20 items per page
|
||||||
|
- Response: `{videos: [...], total: N, page: P, per_page: 20}`
|
||||||
|
- `GET /api/review/{video_id}` — Get video details for annotation
|
||||||
|
- Response: `{video_id, file_path, confidence_score, routing_decision, model_version, frame_count, contributing_frames: [{timestamp, crop_path, confidence}], video_duration}`
|
||||||
|
- `POST /api/review/{video_id}/label` — Submit annotation
|
||||||
|
- Body: `{ground_truth: true/false, notes: string (optional)}`
|
||||||
|
- Updates: `review_queue.annotated = true`, `review_queue.ground_truth = value`, `review_queue.annotated_at = NOW()`
|
||||||
|
- Response: `{status: 'annotated', video_id, ground_truth}`
|
||||||
|
- `GET /api/review/export` — Export annotated data
|
||||||
|
- Query params: `format=csv|json`, `annotated=true/false`, `date_from`, `date_to`
|
||||||
|
- Response: File download with annotated data
|
||||||
|
- `GET /api/review/stats` — Review queue statistics
|
||||||
|
- Response: `{total_in_queue: N, annotated_today: N, avg_confidence: F, confidence_distribution: {...}}`
|
||||||
|
|
||||||
|
### 7.2 Review Frontend
|
||||||
|
**File:** `ui/review/`
|
||||||
|
|
||||||
|
Pages:
|
||||||
|
- **Queue Page (`/`):** List of videos awaiting review
|
||||||
|
- Table columns: Video ID, File Path, Confidence Score, Model Version, Date Added, Actions (View)
|
||||||
|
- Sortable by confidence, date, file path
|
||||||
|
- Filter by confidence range, model version
|
||||||
|
- Pagination (20 items per page)
|
||||||
|
- **Annotation Page (`/review/{video_id}`):** Video annotation interface
|
||||||
|
- Video player with playback controls (HTML5 `<video>` element)
|
||||||
|
- Top-k contributing frames displayed as thumbnails (k=5 default)
|
||||||
|
- Confidence scores displayed per frame
|
||||||
|
- Label toggle button (True/False) with confirmation
|
||||||
|
- Optional notes field
|
||||||
|
- Submit button (saves to DB via API)
|
||||||
|
- Navigation: Previous/Next video in queue
|
||||||
|
|
||||||
|
### 7.3 Data Export
|
||||||
|
**File:** `src/review_export.py`
|
||||||
|
|
||||||
|
Features:
|
||||||
|
- **CSV Export:**
|
||||||
|
```csv
|
||||||
|
video_id,file_path,confidence_score,routing_decision,model_version,ground_truth,annotated_at,contributing_frames
|
||||||
|
12345,/data/input/video.mp4,0.62,REVIEW,v1.2.0,true,2026-08-03T10:30:00Z,"[{'timestamp': 30.0, 'crop_path': '/scratch/12345/crops/30000.jpg', 'confidence': 0.82}, ...]"
|
||||||
|
```
|
||||||
|
- **JSON Export:**
|
||||||
|
```json
|
||||||
|
[
|
||||||
|
{
|
||||||
|
"video_id": 12345,
|
||||||
|
"file_path": "/data/input/video.mp4",
|
||||||
|
"confidence_score": 0.62,
|
||||||
|
"routing_decision": "REVIEW",
|
||||||
|
"model_version": "v1.2.0",
|
||||||
|
"ground_truth": true,
|
||||||
|
"annotated_at": "2026-08-03T10:30:00Z",
|
||||||
|
"contributing_frames": [
|
||||||
|
{"timestamp": 30.0, "crop_path": "/scratch/12345/crops/30000.jpg", "confidence": 0.82}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
]
|
||||||
|
```
|
||||||
|
- **Export Options:**
|
||||||
|
- Filter by annotation status (annotated/unannotated)
|
||||||
|
- Filter by date range
|
||||||
|
- Filter by model version
|
||||||
|
- Filter by ground truth label
|
||||||
|
- **Output Location:** `/data/output/reviews/`
|
||||||
|
|
||||||
|
### 7.4 UI Container
|
||||||
|
**File:** `ui/Dockerfile`
|
||||||
|
|
||||||
|
Base image: `python:3.10-slim`
|
||||||
|
|
||||||
|
Installed packages:
|
||||||
|
- Flask 3.0+ or FastAPI 0.100+ (lightweight web framework)
|
||||||
|
- Jinja2 3.1+ (template engine)
|
||||||
|
- PyMySQL (database connection for API)
|
||||||
|
- gunicorn (WSGI server)
|
||||||
|
|
||||||
|
### 7.5 Docker Compose Update
|
||||||
|
**File:** `docker-compose.yml` (update)
|
||||||
|
|
||||||
|
Add UI service:
|
||||||
|
```yaml
|
||||||
|
ui:
|
||||||
|
build:
|
||||||
|
context: ./ui
|
||||||
|
dockerfile: Dockerfile
|
||||||
|
ports:
|
||||||
|
- "5000:5000"
|
||||||
|
volumes:
|
||||||
|
- ./ui:/app/ui
|
||||||
|
environment:
|
||||||
|
- DB_HOST=mariadb
|
||||||
|
- DB_PORT=3306
|
||||||
|
- DB_NAME=videodetect
|
||||||
|
- DB_USER=videodetect
|
||||||
|
- DB_PASSWORD=${DB_PASSWORD}
|
||||||
|
depends_on:
|
||||||
|
- mariadb
|
||||||
|
networks:
|
||||||
|
- videodetect-network
|
||||||
|
```
|
||||||
|
|
||||||
|
### 7.6 Database Update
|
||||||
|
**File:** `db/schema.sql` (review_queue table — from STORY-01)
|
||||||
|
|
||||||
|
The `review_queue` table was defined in STORY-01. This story populates and queries it.
|
||||||
|
|
||||||
|
### 7.7 Configuration Updates
|
||||||
|
**File:** `config.yaml` (updates)
|
||||||
|
|
||||||
|
New fields:
|
||||||
|
```yaml
|
||||||
|
review_ui:
|
||||||
|
host: "0.0.0.0"
|
||||||
|
port: 5000
|
||||||
|
per_page: 20
|
||||||
|
top_k_frames: 5
|
||||||
|
export_path: /data/output/reviews
|
||||||
|
auth_enabled: false # per TC-06
|
||||||
|
ssl_enabled: false # per TC-06
|
||||||
|
```
|
||||||
|
|
||||||
|
## Acceptance Criteria
|
||||||
|
|
||||||
|
### Functional
|
||||||
|
- [ ] Review queue displays all videos with routing_decision = REVIEW and annotated = false
|
||||||
|
- [ ] Video player loads and plays the video correctly
|
||||||
|
- [ ] Top-k contributing frames are displayed as thumbnails with confidence scores
|
||||||
|
- [ ] Annotator can toggle label (True/False) and submit
|
||||||
|
- [ ] Submitted label is persisted to DB (review_queue table)
|
||||||
|
- [ ] Annotated videos are removed from the default queue view
|
||||||
|
- [ ] CSV export produces valid CSV with all required fields
|
||||||
|
- [ ] JSON export produces valid JSON with all required fields
|
||||||
|
- [ ] Export includes ground truth labels and contributing frame data
|
||||||
|
- [ ] UI is accessible via http://<server-ip>:5000 (no auth, no SSL)
|
||||||
|
- [ ] Pagination works correctly (20 items per page)
|
||||||
|
- [ ] Sort and filter operations work on the queue page
|
||||||
|
|
||||||
|
### Non-Functional
|
||||||
|
- [ ] Queue page loads in < 2 seconds (with 1000+ videos in queue)
|
||||||
|
- [ ] Video player loads in < 3 seconds
|
||||||
|
- [ ] Label submission completes in < 1 second
|
||||||
|
- [ ] Export of 1000 annotated videos completes in < 10 seconds
|
||||||
|
- [ ] UI uses < 100MB RAM at idle
|
||||||
|
- [ ] No authentication or SSL configured (per TC-06)
|
||||||
|
|
||||||
|
### Technical Constraints
|
||||||
|
- [ ] UI runs in Docker container (per TC-05)
|
||||||
|
- [ ] No reverse proxy configured
|
||||||
|
- [ ] No SSL certificates configured
|
||||||
|
- [ ] No authentication mechanism configured
|
||||||
|
- [ ] All API responses are JSON
|
||||||
|
- [ ] Database queries use parameterized statements
|
||||||
|
- [ ] Export files are UTF-8 encoded
|
||||||
|
|
||||||
|
## Dependencies
|
||||||
|
- **Prerequisites:** STORY-01 (Foundation — DB schema), STORY-05 (Classification — provides routing decisions)
|
||||||
|
- **Depends on:** None (can be built in parallel with STORY-04, STORY-05)
|
||||||
|
- **Enables:** STORY-08 (Active Learning — provides labeled training data)
|
||||||
|
|
||||||
|
## Risks & Mitigations
|
||||||
|
| Risk | Mitigation |
|
||||||
|
|------|--|
|
||||||
|
| Video playback in browser requires compatible format | Serve videos in web-compatible format (H.264 MP4); transcode if needed |
|
||||||
|
| Crop paths may not be accessible from UI container | Store crop paths in DB; serve via API endpoint |
|
||||||
|
| No auth means anyone on LAN can access | Acceptable per TC-06; document in security notes |
|
||||||
|
| Large review queue slows page loads | Implement server-side pagination; lazy load thumbnails |
|
||||||
|
|
||||||
|
## Estimated Effort
|
||||||
|
- **Sprint:** 7
|
||||||
|
- **Story Points:** 21
|
||||||
|
- **Dependencies:** STORY-01, STORY-05
|
||||||
+245
@@ -0,0 +1,245 @@
|
|||||||
|
# STORY-08: Active Learning Pipeline
|
||||||
|
|
||||||
|
## Epic
|
||||||
|
**E3: Active Learning** — As an ML engineer, I can fine-tune the classifier head with reviewed data.
|
||||||
|
|
||||||
|
## Related Requirements
|
||||||
|
| ID | Requirement |
|
||||||
|
|----|------|
|
||||||
|
| FR-06 | Active learning pipeline (label ingestion → fine-tuning → deployment): Batch retraining only. No online learning. Versioned model swaps. |
|
||||||
|
| NFR-04 | Determinism & Reproducibility: Config-seeded randomness, versioned models |
|
||||||
|
| NFR-05 | Fault Tolerance: Auto-retry on transient failures; skip & log on fatal errors |
|
||||||
|
| TC-01 | Hardware: 2× Tesla P40 24GB (compute capability 5.2, PCIe 3.0, no Tensor Cores) |
|
||||||
|
| TC-02 | CUDA/Torch Compatibility: CUDA ≤ 11.8, PyTorch ≤ 2.1.0, FP32 inference only |
|
||||||
|
|
||||||
|
## Description
|
||||||
|
Implement the active learning pipeline that ingests annotated review data, fine-tunes the classifier head, validates the candidate model, and deploys it if it meets quality gates. This enables incremental model improvement using human-labeled data from the review queue.
|
||||||
|
|
||||||
|
## Scope
|
||||||
|
|
||||||
|
### In Scope
|
||||||
|
- Label ingestion from review queue (CSV/JSON export or direct DB query)
|
||||||
|
- Dataset versioning and preparation for training
|
||||||
|
- Head-only fine-tuning on face crops (freeze backbone, train classification head)
|
||||||
|
- Training with AdamW, configurable learning rate, early stopping
|
||||||
|
- Candidate model validation on held-out validation set
|
||||||
|
- F1 score and ECE (Expected Calibration Error) calculation
|
||||||
|
- Quality gate validation (ΔF1 > 0.02, ECE < 0.08)
|
||||||
|
- Model registry updates (ACTIVE, CANDIDATE, ARCHIVED states)
|
||||||
|
- Hot reload of new TensorRT engine
|
||||||
|
- Rollback on regression
|
||||||
|
|
||||||
|
### Out of Scope
|
||||||
|
- Frame sampling (covered in STORY-03)
|
||||||
|
- Face detection (covered in STORY-04)
|
||||||
|
- Classification inference (covered in STORY-05)
|
||||||
|
- Review UI (covered in STORY-07)
|
||||||
|
- Monitoring dashboards (covered in STORY-09)
|
||||||
|
|
||||||
|
## Deliverables
|
||||||
|
|
||||||
|
### 8.1 Label Ingestion
|
||||||
|
**File:** `src/active_learning/label_ingestor.py`
|
||||||
|
|
||||||
|
Features:
|
||||||
|
- **Data Source:** Query `review_queue` table for annotated labels
|
||||||
|
```sql
|
||||||
|
SELECT video_id, ground_truth, contributing_frames
|
||||||
|
FROM review_queue
|
||||||
|
WHERE annotated = true AND ground_truth IS NOT NULL
|
||||||
|
```
|
||||||
|
- **Crop Extraction:** Extract face crops from stored paths or re-extract from video
|
||||||
|
- **Dataset Versioning:** Create versioned dataset directory structure
|
||||||
|
```
|
||||||
|
/data/training/v2.0.0/
|
||||||
|
crops/
|
||||||
|
class_0/ (negative samples)
|
||||||
|
class_1/ (positive samples)
|
||||||
|
labels.csv
|
||||||
|
metadata.json
|
||||||
|
```
|
||||||
|
- **Train/Val Split:** 80/20 split (stratified by class)
|
||||||
|
- **Augmentation:** Apply standard augmentations at training time (not pre-computed)
|
||||||
|
- Random horizontal flip
|
||||||
|
- Random color jitter
|
||||||
|
- Random affine transform (±10 degrees rotation, ±10% scale)
|
||||||
|
|
||||||
|
### 8.2 Training Pipeline
|
||||||
|
**File:** `src/active_learning/trainer.py`
|
||||||
|
|
||||||
|
Features:
|
||||||
|
- **Model Loading:** Load pre-trained MobileNetV3 backbone (frozen)
|
||||||
|
```python
|
||||||
|
model = load_pretrained_mobilenetv3()
|
||||||
|
for param in model.features.parameters():
|
||||||
|
param.requires_grad = False
|
||||||
|
# Replace classification head
|
||||||
|
model.classifier = nn.Sequential(
|
||||||
|
nn.Linear(1280, 256),
|
||||||
|
nn.ReLU(),
|
||||||
|
nn.Dropout(0.3),
|
||||||
|
nn.Linear(256, 2) # binary classification
|
||||||
|
)
|
||||||
|
```
|
||||||
|
- **Head-Only Fine-Tuning:** Only train the classification head
|
||||||
|
- Backbone weights are frozen (no gradient updates)
|
||||||
|
- Head weights are trainable
|
||||||
|
- This is faster and requires less data than full fine-tuning
|
||||||
|
- **Optimizer:** AdamW with configurable parameters
|
||||||
|
- Learning rate: 1e-3 (default), configurable
|
||||||
|
- Weight decay: 1e-2
|
||||||
|
- Betas: (0.9, 0.999)
|
||||||
|
- **Loss Function:** Binary Cross-Entropy with class weights (if imbalanced)
|
||||||
|
- **Training Configuration:**
|
||||||
|
- Epochs: 10-30 (configurable)
|
||||||
|
- Batch size: 32
|
||||||
|
- Early stopping: patience=5 epochs (no validation improvement)
|
||||||
|
- Learning rate scheduler: ReduceLROnPlateau (factor=0.5, patience=3)
|
||||||
|
- **Checkpointing:** Save checkpoint to `/models/candidate/` after each epoch
|
||||||
|
```
|
||||||
|
/models/candidate/
|
||||||
|
v2.0.0_epoch_01.pt
|
||||||
|
v2.0.0_epoch_02.pt
|
||||||
|
...
|
||||||
|
v2.0.0_best.pt (best validation F1)
|
||||||
|
```
|
||||||
|
|
||||||
|
### 8.3 Validation Pipeline
|
||||||
|
**File:** `src/active_learning/validator.py`
|
||||||
|
|
||||||
|
Features:
|
||||||
|
- **Validation Set:** Held-out 20% of annotated data (never seen during training)
|
||||||
|
- **Metrics Calculation:**
|
||||||
|
- **F1 Score:** Macro F1 on validation set
|
||||||
|
- **ECE (Expected Calibration Error):**
|
||||||
|
```python
|
||||||
|
def compute_ece(predictions, labels, n_bins=15):
|
||||||
|
bin_boundaries = np.linspace(0, 1, n_bins + 1)
|
||||||
|
ece = 0.0
|
||||||
|
for i in range(n_bins):
|
||||||
|
mask = (predictions >= bin_boundaries[i]) & (predictions < bin_boundaries[i+1])
|
||||||
|
if mask.sum() > 0:
|
||||||
|
bin_confidence = predictions[mask].mean()
|
||||||
|
bin_accuracy = labels[mask].mean()
|
||||||
|
ece += (mask.sum() / len(predictions)) * abs(bin_confidence - bin_accuracy)
|
||||||
|
return ece
|
||||||
|
```
|
||||||
|
- **Accuracy, Precision, Recall:** Standard classification metrics
|
||||||
|
- **Quality Gates:**
|
||||||
|
- ΔF1 > 0.02 (improvement over current model)
|
||||||
|
- ECE < 0.08 (calibration acceptable)
|
||||||
|
- Both gates must pass for deployment
|
||||||
|
|
||||||
|
### 8.4 Model Registry & Deployment
|
||||||
|
**File:** `src/active_learning/registry.py`
|
||||||
|
|
||||||
|
Features:
|
||||||
|
- **Model Registry:** Update DB `models` table
|
||||||
|
```sql
|
||||||
|
-- Promote candidate to active
|
||||||
|
UPDATE models SET status = 'ACTIVE' WHERE version = 'v2.0.0';
|
||||||
|
UPDATE models SET status = 'ARCHIVED' WHERE status = 'ACTIVE' AND version != 'v2.0.0';
|
||||||
|
```
|
||||||
|
- **TensorRT Engine Build:** Convert candidate model to TensorRT engine
|
||||||
|
```bash
|
||||||
|
trtexec --onnx=models/candidate/v2.0.0.onnx \
|
||||||
|
--saveEngine=models/candidate/v2.0.0.trt \
|
||||||
|
--fp32 --maxBatch=32 --workspace=2048
|
||||||
|
```
|
||||||
|
- **Hot Reload:** Signal worker process to reload new engine
|
||||||
|
- Option A: Restart worker container (`docker-compose restart worker`)
|
||||||
|
- Option B: In-process reload (graceful, no downtime)
|
||||||
|
- Default: In-process reload via signal handling
|
||||||
|
- **Rollback:** If new model causes issues, rollback to archived model
|
||||||
|
```sql
|
||||||
|
UPDATE models SET status = 'ACTIVE' WHERE version = 'v1.2.0';
|
||||||
|
UPDATE models SET status = 'ARCHIVED' WHERE version = 'v2.0.0';
|
||||||
|
```
|
||||||
|
|
||||||
|
### 8.5 Training Configuration
|
||||||
|
**File:** `config.yaml` (updates)
|
||||||
|
|
||||||
|
New fields:
|
||||||
|
```yaml
|
||||||
|
active_learning:
|
||||||
|
enabled: true
|
||||||
|
min_annotated_samples: 100 # minimum labeled data to trigger training
|
||||||
|
training:
|
||||||
|
epochs: 20
|
||||||
|
batch_size: 32
|
||||||
|
learning_rate: 1e-3
|
||||||
|
weight_decay: 1e-2
|
||||||
|
early_stopping_patience: 5
|
||||||
|
lr_scheduler: ReduceLROnPlateau
|
||||||
|
lr_factor: 0.5
|
||||||
|
lr_patience: 3
|
||||||
|
validation:
|
||||||
|
val_split: 0.2
|
||||||
|
min_f1_improvement: 0.02
|
||||||
|
max_ece: 0.08
|
||||||
|
deployment:
|
||||||
|
auto_deploy: true # deploy if quality gates pass
|
||||||
|
hot_reload: true
|
||||||
|
rollback_enabled: true
|
||||||
|
augmentation:
|
||||||
|
horizontal_flip: true
|
||||||
|
color_jitter: true
|
||||||
|
affine: true
|
||||||
|
affine_degrees: 10
|
||||||
|
affine_scale: 0.1
|
||||||
|
```
|
||||||
|
|
||||||
|
## Acceptance Criteria
|
||||||
|
|
||||||
|
### Functional
|
||||||
|
- [ ] Annotated labels are ingested from review_queue table correctly
|
||||||
|
- [ ] Dataset is versioned with proper directory structure
|
||||||
|
- [ ] Train/val split is stratified by class (80/20)
|
||||||
|
- [ ] Data augmentation is applied at training time (not pre-computed)
|
||||||
|
- [ ] Backbone weights are frozen during fine-tuning (verified by checking requires_grad)
|
||||||
|
- [ ] Classification head is trainable and receives gradient updates
|
||||||
|
- [ ] AdamW optimizer is used with correct parameters
|
||||||
|
- [ ] Early stopping works (training stops if no improvement for patience epochs)
|
||||||
|
- [ ] Candidate model is saved to /models/candidate/ with correct versioning
|
||||||
|
- [ ] F1 score is calculated correctly on validation set
|
||||||
|
- [ ] ECE is calculated correctly on validation set
|
||||||
|
- [ ] Quality gates are enforced (ΔF1 > 0.02 AND ECE < 0.08)
|
||||||
|
- [ ] Model registry is updated (ACTIVE, CANDIDATE, ARCHIVED states)
|
||||||
|
- [ ] TensorRT engine is built from candidate model
|
||||||
|
- [ ] Hot reload deploys new model without downtime
|
||||||
|
- [ ] Rollback restores previous model if deployment fails
|
||||||
|
|
||||||
|
### Non-Functional
|
||||||
|
- [ ] Training completes in < 4 hours on Tesla P40 (typical dataset: 1000 samples)
|
||||||
|
- [ ] Validation completes in < 30 minutes
|
||||||
|
- [ ] TensorRT engine build completes in < 10 minutes
|
||||||
|
- [ ] Hot reload completes in < 30 seconds
|
||||||
|
- [ ] Training uses < 18GB GPU memory
|
||||||
|
- [ ] Training is deterministic (same data + same config → same results)
|
||||||
|
|
||||||
|
### Technical Constraints
|
||||||
|
- [ ] CUDA 11.8 compatible (verified via torch.version.cuda)
|
||||||
|
- [ ] PyTorch ≤ 2.1.0 (verified via torch.__version__)
|
||||||
|
- [ ] FP32 training only (no mixed precision)
|
||||||
|
- [ ] No Tensor Cores used (CC 5.2 constraint)
|
||||||
|
- [ ] Model versioning follows semantic versioning (MAJOR.MINOR.PATCH)
|
||||||
|
- [ ] All training hyperparameters are configurable via config.yaml
|
||||||
|
|
||||||
|
## Dependencies
|
||||||
|
- **Prerequisites:** STORY-01 (Foundation), STORY-05 (Classification — provides base model), STORY-07 (Review UI — provides labeled data)
|
||||||
|
- **Depends on:** None (runs independently, triggered by labeled data threshold)
|
||||||
|
- **Enables:** STORY-05 (provides new model for inference)
|
||||||
|
|
||||||
|
## Risks & Mitigations
|
||||||
|
| Risk | Mitigation |
|
||||||
|
|------|--|
|
||||||
|
| Insufficient labeled data for meaningful fine-tuning | Set min_annotated_samples threshold (e.g., 100); wait until reached |
|
||||||
|
| Head-only fine-tuning may not be enough for domain shift | Offer full fine-tuning as option; document limitations |
|
||||||
|
| Model regression in production | Strict quality gates; keep previous model in ARCHIVED state for quick rollback |
|
||||||
|
| Long training times on Tesla P40 | Head-only training is faster; batch size tuning; early stopping |
|
||||||
|
| ECE calculation sensitivity | Use proper binning; report ECI with confidence intervals |
|
||||||
|
|
||||||
|
## Estimated Effort
|
||||||
|
- **Sprint:** 7-8
|
||||||
|
- **Story Points:** 34
|
||||||
|
- **Dependencies:** STORY-01, STORY-05, STORY-07
|
||||||
+360
@@ -0,0 +1,360 @@
|
|||||||
|
# STORY-09: Observability, Monitoring & Hardening
|
||||||
|
|
||||||
|
## Epic
|
||||||
|
**E4: Operations** — As a DevOps engineer, I can schedule, monitor, and resume batch jobs.
|
||||||
|
|
||||||
|
## Related Requirements
|
||||||
|
| ID | Requirement |
|
||||||
|
|----|------|
|
||||||
|
| NFR-05 | Fault Tolerance: Auto-retry on transient failures; skip & log on fatal errors |
|
||||||
|
| NFR-06 | Observability: Prometheus/Grafana metrics + structured logging; Tracks FPS, queue depth, confidence distribution, drift alerts |
|
||||||
|
| NFR-01 | Throughput: ≥ 30 videos/hour/GPU |
|
||||||
|
| NFR-03 | GPU Memory Safety: ≤ 18GB per GPU sustained |
|
||||||
|
| NFR-04 | Determinism & Reproducibility: Config-seeded randomness, versioned models |
|
||||||
|
| NFR-07 | Data Volume Handling: Efficient indexing for ~30TB dataset |
|
||||||
|
|
||||||
|
## Description
|
||||||
|
Implement comprehensive monitoring, metrics collection, fault tolerance, and system hardening. Expose Prometheus metrics for throughput, GPU utilization, queue depth, and confidence distribution. Build Grafana dashboards for real-time system health. Implement crash recovery, idempotency, retry logic, and drift detection.
|
||||||
|
|
||||||
|
## Scope
|
||||||
|
|
||||||
|
### In Scope
|
||||||
|
- Prometheus metrics exposure (videos processed, GPU utilization, queue depth, confidence distribution)
|
||||||
|
- Grafana dashboard configuration (throughput, error rates, confidence drift)
|
||||||
|
- Crash recovery with checkpointing
|
||||||
|
- Idempotent processing guarantees
|
||||||
|
- Retry logic for transient errors (up to 3 attempts)
|
||||||
|
- Weekly drift detection job
|
||||||
|
- Alerting on confidence distribution shifts and review queue growth
|
||||||
|
- Worker health checks and auto-restart
|
||||||
|
|
||||||
|
### Out of Scope
|
||||||
|
- Frame sampling (covered in STORY-04)
|
||||||
|
- Face detection (covered in STORY-05)
|
||||||
|
- Classification (covered in STORY-06)
|
||||||
|
- Review UI (covered in STORY-08)
|
||||||
|
- Active learning pipeline (covered in STORY-09)
|
||||||
|
|
||||||
|
## Deliverables
|
||||||
|
|
||||||
|
### 9.1 Prometheus Metrics
|
||||||
|
**File:** `src/metrics.py`
|
||||||
|
|
||||||
|
Metrics to expose:
|
||||||
|
- **Counter Metrics:**
|
||||||
|
- `videos_processed_total` (label: routing_decision=MATCH|REVIEW|SKIP|UNSCANNABLE|ERROR)
|
||||||
|
- `videos_processed_by_model_total` (label: model_version)
|
||||||
|
- `frames_extracted_total`
|
||||||
|
- `faces_detected_total`
|
||||||
|
- `inference_errors_total` (label: error_type)
|
||||||
|
- `retry_attempts_total` (label: step=probe|extract|detect|classify)
|
||||||
|
- **Gauge Metrics:**
|
||||||
|
- `gpu_utilization_percent` (label: gpu_id)
|
||||||
|
- `gpu_memory_used_bytes` (label: gpu_id)
|
||||||
|
- `gpu_memory_free_bytes` (label: gpu_id)
|
||||||
|
- `queue_depth_pending`
|
||||||
|
- `queue_depth_processing`
|
||||||
|
- `queue_depth_review`
|
||||||
|
- `scratch_usage_bytes`
|
||||||
|
- `scratch_usage_percent`
|
||||||
|
- **Histogram Metrics:**
|
||||||
|
- `video_processing_duration_seconds` (label: routing_decision)
|
||||||
|
- `confidence_score_distribution` (buckets: 0.0, 0.1, 0.2, ..., 0.9, 1.0)
|
||||||
|
- `frame_count_per_video`
|
||||||
|
- `face_count_per_video`
|
||||||
|
- **Summary Metrics:**
|
||||||
|
- `throughput_videos_per_hour` (calculated from counter)
|
||||||
|
- `average_confidence` (calculated from histogram)
|
||||||
|
|
||||||
|
Implementation:
|
||||||
|
```python
|
||||||
|
from prometheus_client import start_http_server, Counter, Gauge, Histogram
|
||||||
|
|
||||||
|
# Start metrics HTTP server
|
||||||
|
start_http_server(9090) # /metrics endpoint
|
||||||
|
|
||||||
|
# Define metrics
|
||||||
|
videos_processed = Counter(
|
||||||
|
'videos_processed_total',
|
||||||
|
'Total videos processed',
|
||||||
|
['routing_decision']
|
||||||
|
)
|
||||||
|
|
||||||
|
gpu_memory = Gauge(
|
||||||
|
'gpu_memory_used_bytes',
|
||||||
|
'GPU memory usage',
|
||||||
|
['gpu_id']
|
||||||
|
)
|
||||||
|
|
||||||
|
confidence_hist = Histogram(
|
||||||
|
'confidence_score_distribution',
|
||||||
|
'Video confidence scores',
|
||||||
|
buckets=[0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0]
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
### 9.2 Grafana Dashboards
|
||||||
|
**File:** `monitoring/grafana/dashboards/`
|
||||||
|
|
||||||
|
Dashboard 1: **System Overview**
|
||||||
|
- Panels:
|
||||||
|
- Throughput (videos/hour) — line chart, 1h window
|
||||||
|
- Queue depth (pending/processing/review) — stacked bar
|
||||||
|
- GPU utilization (both GPUs) — line chart
|
||||||
|
- GPU memory usage (both GPUs) — line chart
|
||||||
|
- Error rate (errors/100 videos) — bar chart
|
||||||
|
- Confidence distribution — histogram
|
||||||
|
|
||||||
|
Dashboard 2: **Processing Details**
|
||||||
|
- Panels:
|
||||||
|
- Processing duration per video — scatter plot
|
||||||
|
- Frame count per video — histogram
|
||||||
|
- Face count per video — histogram
|
||||||
|
- Confidence score by model version — box plot
|
||||||
|
- Routing decision distribution — pie chart
|
||||||
|
|
||||||
|
Dashboard 3: **Active Learning**
|
||||||
|
- Panels:
|
||||||
|
- Review queue size over time — line chart
|
||||||
|
- Annotation rate (labels/day) — bar chart
|
||||||
|
- Model version timeline — timeline panel
|
||||||
|
- F1 score by model version — line chart
|
||||||
|
- ECE by model version — line chart
|
||||||
|
|
||||||
|
**File:** `monitoring/grafana/dashboards/system_overview.json`
|
||||||
|
**File:** `monitoring/grafana/dashboards/processing_details.json`
|
||||||
|
**File:** `monitoring/grafana/dashboards/active_learning.json`
|
||||||
|
|
||||||
|
### 9.3 Crash Recovery
|
||||||
|
**File:** `src/crash_recovery.py`
|
||||||
|
|
||||||
|
Features:
|
||||||
|
- **Checkpointing:** Periodically save processing state
|
||||||
|
```python
|
||||||
|
def save_checkpoint(video_id, state, progress):
|
||||||
|
checkpoint = {
|
||||||
|
'video_id': video_id,
|
||||||
|
'state': state, # 'processing', 'extracting', 'detecting', 'classifying'
|
||||||
|
'progress': progress, # dict of step -> completed
|
||||||
|
'timestamp': datetime.utcnow().isoformat()
|
||||||
|
}
|
||||||
|
with open(f'/scratch/checkpoints/{video_id}.json', 'w') as f:
|
||||||
|
json.dump(checkpoint, f)
|
||||||
|
```
|
||||||
|
- **Recovery on Startup:** Scan for PROCESSING videos and re-queue them
|
||||||
|
```sql
|
||||||
|
UPDATE videos SET status = 'PENDING', updated_at = NOW()
|
||||||
|
WHERE status = 'PROCESSING' AND updated_at < NOW() - INTERVAL 5 MINUTE
|
||||||
|
```
|
||||||
|
- **Lock Timeout:** 5 minutes (videos stuck in PROCESSING beyond this are re-queued)
|
||||||
|
- **Re-queue Logic:** Only re-queue if worker is down (detected via health check)
|
||||||
|
- **Idempotency:** Re-processing a video produces same results (no duplicates)
|
||||||
|
|
||||||
|
### 9.4 Retry Logic
|
||||||
|
**File:** `src/retry.py`
|
||||||
|
|
||||||
|
Features:
|
||||||
|
- **Retry Decorator:**
|
||||||
|
```python
|
||||||
|
def retry(max_attempts=3, delay=1.0, backoff=2.0, exceptions=(Exception,)):
|
||||||
|
def decorator(func):
|
||||||
|
def wrapper(*args, **kwargs):
|
||||||
|
last_exception = None
|
||||||
|
for attempt in range(max_attempts):
|
||||||
|
try:
|
||||||
|
return func(*args, **kwargs)
|
||||||
|
except exceptions as e:
|
||||||
|
last_exception = e
|
||||||
|
if attempt < max_attempts - 1:
|
||||||
|
wait = delay * (backoff ** attempt)
|
||||||
|
time.sleep(wait)
|
||||||
|
raise last_exception
|
||||||
|
return wrapper
|
||||||
|
return decorator
|
||||||
|
```
|
||||||
|
- **Applicable Steps:** ffprobe, frame extraction, face detection, classification, DB writes
|
||||||
|
- **Transient Errors:** Network timeout, GPU OOM, file lock contention
|
||||||
|
- **Non-Retryable Errors:** Codec unsupported, corrupt file, invalid path → skip and log
|
||||||
|
- **Retry Logging:** Log each retry attempt with error type and delay
|
||||||
|
|
||||||
|
### 9.5 Drift Detection
|
||||||
|
**File:** `src/drift_detector.py`
|
||||||
|
|
||||||
|
Features:
|
||||||
|
- **Weekly Job:** Compare current confidence distribution to baseline
|
||||||
|
```python
|
||||||
|
def detect_drift(current_confidences, baseline_confidences, threshold=0.10):
|
||||||
|
# Compare p(C > 0.5) shift
|
||||||
|
current_high = sum(1 for c in current_confidences if c > 0.5) / len(current_confidences)
|
||||||
|
baseline_high = sum(1 for c in baseline_confidences if c > 0.5) / len(baseline_confidences)
|
||||||
|
shift = abs(current_high - baseline_high)
|
||||||
|
|
||||||
|
if shift > threshold:
|
||||||
|
alert(f"Confidence drift detected: {shift:.2%} shift in p(C > 0.5)")
|
||||||
|
return True
|
||||||
|
return False
|
||||||
|
```
|
||||||
|
- **Baseline:** Stored in DB or config (computed from last training cycle)
|
||||||
|
- **Alert Conditions:**
|
||||||
|
- `p(C > 0.5)` shifts > 10% from baseline
|
||||||
|
- Review queue grows unbounded (> 1000 items for > 24 hours)
|
||||||
|
- Throughput drops below 20 videos/hour/GPU for > 1 hour
|
||||||
|
- Error rate exceeds 5% for any 1-hour window
|
||||||
|
- **Alert Channels:** Email, Slack webhook, or log entry (configurable)
|
||||||
|
|
||||||
|
### 9.6 Worker Health Checks
|
||||||
|
**File:** `src/health_check.py`
|
||||||
|
|
||||||
|
Features:
|
||||||
|
- **Health Endpoint:** `/health` returns worker status
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"status": "healthy",
|
||||||
|
"gpu_available": true,
|
||||||
|
"gpu_memory_used_gb": 12.5,
|
||||||
|
"queue_depth": 42,
|
||||||
|
"uptime_seconds": 86400,
|
||||||
|
"videos_processed_today": 156,
|
||||||
|
"last_error": null
|
||||||
|
}
|
||||||
|
```
|
||||||
|
- **Auto-Restart:** Docker restart policy for worker container
|
||||||
|
```yaml
|
||||||
|
restart: unless-stopped
|
||||||
|
healthcheck:
|
||||||
|
test: ["CMD", "curl", "-f", "http://localhost:9090/health"]
|
||||||
|
interval: 30s
|
||||||
|
timeout: 10s
|
||||||
|
retries: 3
|
||||||
|
```
|
||||||
|
- **GPU Health:** Monitor GPU temperature and error counts
|
||||||
|
- **Disk Health:** Monitor scratch space usage and NAS connectivity
|
||||||
|
|
||||||
|
### 9.7 Docker Compose Update
|
||||||
|
**File:** `docker-compose.yml` (update)
|
||||||
|
|
||||||
|
Add monitoring services:
|
||||||
|
```yaml
|
||||||
|
prometheus:
|
||||||
|
image: prom/prometheus:v2.48.0
|
||||||
|
volumes:
|
||||||
|
- ./monitoring/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
|
||||||
|
- prometheus_data:/prometheus
|
||||||
|
ports:
|
||||||
|
- "9090:9090"
|
||||||
|
networks:
|
||||||
|
- videodetect-network
|
||||||
|
|
||||||
|
grafana:
|
||||||
|
image: grafana/grafana:10.2.0
|
||||||
|
volumes:
|
||||||
|
- grafana_data:/var/lib/grafana
|
||||||
|
- ./monitoring/grafana/dashboards:/etc/grafana/provisioning/dashboards
|
||||||
|
environment:
|
||||||
|
- GF_SECURITY_ADMIN_PASSWORD=${GRAFANA_PASSWORD}
|
||||||
|
ports:
|
||||||
|
- "3000:3000"
|
||||||
|
networks:
|
||||||
|
- videodetect-network
|
||||||
|
|
||||||
|
volumes:
|
||||||
|
prometheus_data:
|
||||||
|
grafana_data:
|
||||||
|
```
|
||||||
|
|
||||||
|
### 9.8 Configuration Updates
|
||||||
|
**File:** `config.yaml` (updates)
|
||||||
|
|
||||||
|
New fields:
|
||||||
|
```yaml
|
||||||
|
monitoring:
|
||||||
|
prometheus:
|
||||||
|
enabled: true
|
||||||
|
port: 9090
|
||||||
|
metrics_path: /metrics
|
||||||
|
grafana:
|
||||||
|
enabled: true
|
||||||
|
port: 3000
|
||||||
|
alerts:
|
||||||
|
confidence_drift_threshold: 0.10
|
||||||
|
review_queue_max_size: 1000
|
||||||
|
review_queue_max_age_hours: 24
|
||||||
|
throughput_min_videos_per_hour: 20
|
||||||
|
throughput_min_duration_hours: 1
|
||||||
|
error_rate_threshold: 0.05
|
||||||
|
error_rate_window_hours: 1
|
||||||
|
drift_detection:
|
||||||
|
enabled: true
|
||||||
|
schedule: weekly # cron: 0 2 * * 0 (Sundays at 2 AM)
|
||||||
|
baseline_source: db # db or config
|
||||||
|
crash_recovery:
|
||||||
|
lock_timeout_minutes: 5
|
||||||
|
auto_requeue: true
|
||||||
|
retry:
|
||||||
|
max_attempts: 3
|
||||||
|
initial_delay: 1.0
|
||||||
|
backoff_factor: 2.0
|
||||||
|
retryable_errors:
|
||||||
|
- timeout
|
||||||
|
- gpu_oom
|
||||||
|
- file_lock
|
||||||
|
non_retryable_errors:
|
||||||
|
- codec_unsupported
|
||||||
|
- file_corrupt
|
||||||
|
- invalid_path
|
||||||
|
```
|
||||||
|
|
||||||
|
## Acceptance Criteria
|
||||||
|
|
||||||
|
### Functional
|
||||||
|
- [ ] Prometheus metrics are exposed at `/metrics` endpoint and queryable
|
||||||
|
- [ ] All required metrics are present (counters, gauges, histograms, summaries)
|
||||||
|
- [ ] Grafana dashboards load and display correct data
|
||||||
|
- [ ] System Overview dashboard shows throughput, queue depth, GPU metrics
|
||||||
|
- [ ] Processing Details dashboard shows duration, frame count, confidence distribution
|
||||||
|
- [ ] Active Learning dashboard shows review queue, annotation rate, model versions
|
||||||
|
- [ ] Crash recovery re-queues stuck PROCESSING videos on worker restart
|
||||||
|
- [ ] Idempotent processing: re-processing produces same results (no duplicates)
|
||||||
|
- [ ] Retry logic retries transient errors up to 3 times with exponential backoff
|
||||||
|
- [ ] Non-retryable errors are skipped and logged (no infinite retry)
|
||||||
|
- [ ] Drift detection runs weekly and alerts on > 10% confidence shift
|
||||||
|
- [ ] Review queue growth alert triggers when queue > 1000 for > 24 hours
|
||||||
|
- [ ] Worker health check returns correct status
|
||||||
|
- [ ] Auto-restart triggers on health check failure
|
||||||
|
|
||||||
|
### Non-Functional
|
||||||
|
- [ ] Metrics collection overhead < 2% of CPU
|
||||||
|
- [ ] Grafana dashboard loads in < 3 seconds
|
||||||
|
- [ ] Crash recovery completes in < 30 seconds
|
||||||
|
- [ ] Drift detection completes in < 5 minutes
|
||||||
|
- [ ] Alert delivery completes in < 60 seconds
|
||||||
|
- [ ] Prometheus data retention: 30 days (configurable)
|
||||||
|
- [ ] Grafana data source refresh: 30 seconds
|
||||||
|
|
||||||
|
### Technical Constraints
|
||||||
|
- [ ] Prometheus metrics follow naming conventions (unit suffixes, proper labels)
|
||||||
|
- [ ] Grafana dashboards are JSON-exportable and version-controlled
|
||||||
|
- [ ] Crash recovery is idempotent (running twice produces same result)
|
||||||
|
- [ ] Retry logic does not retry non-transient errors
|
||||||
|
- [ ] Drift detection baseline is stored and versioned
|
||||||
|
- [ ] Health check endpoint responds in < 1 second
|
||||||
|
- [ ] All alerts are logged with timestamp and context
|
||||||
|
|
||||||
|
## Dependencies
|
||||||
|
- **Prerequisites:** STORY-01 (Foundation), STORY-05 (Classification — provides metrics data), STORY-08 (Active Learning — provides model metrics)
|
||||||
|
- **Depends on:** None (can be built in parallel with other stories)
|
||||||
|
- **Enables:** Production deployment and long-term operation
|
||||||
|
|
||||||
|
## Risks & Mitigations
|
||||||
|
| Risk | Mitigation |
|
||||||
|
|------|--|
|
||||||
|
| Prometheus metrics cardinality explosion | Limit label cardinality; use histograms instead of individual values |
|
||||||
|
| Grafana dashboard load time with large datasets | Use Prometheus aggregations; pre-compute panels |
|
||||||
|
| Crash recovery misses in-flight writes | Use DB transactions; lock timeout prevents permanent locks |
|
||||||
|
| Drift detection baseline becomes stale | Update baseline with each training cycle |
|
||||||
|
| Alert fatigue from too many alerts | Tune thresholds; implement alert grouping |
|
||||||
|
|
||||||
|
## Estimated Effort
|
||||||
|
- **Sprint:** 9+
|
||||||
|
- **Story Points:** 34
|
||||||
|
- **Dependencies:** STORY-01, STORY-05, STORY-08
|
||||||
+248
@@ -0,0 +1,248 @@
|
|||||||
|
# VideoDetect Configuration
|
||||||
|
# All values can be overridden by environment variables (e.g., VD_SAMPLING_INTERVAL_SECONDS)
|
||||||
|
|
||||||
|
# -----------------------------------------------------
|
||||||
|
# Frame Sampling
|
||||||
|
# -----------------------------------------------------
|
||||||
|
sampling:
|
||||||
|
interval_seconds: 30
|
||||||
|
override_per_job: true
|
||||||
|
quality: 2 # JPEG quality (1-31, lower=better quality)
|
||||||
|
format: jpeg
|
||||||
|
|
||||||
|
# -----------------------------------------------------
|
||||||
|
# Confidence Thresholds
|
||||||
|
# -----------------------------------------------------
|
||||||
|
thresholds:
|
||||||
|
T_high: 0.75 # C >= T_high → MATCH
|
||||||
|
T_low: 0.45 # T_low <= C < T_high → REVIEW; C < T_low → SKIP
|
||||||
|
|
||||||
|
# -----------------------------------------------------
|
||||||
|
# GPU Configuration
|
||||||
|
# -----------------------------------------------------
|
||||||
|
gpu:
|
||||||
|
max_memory_gb: 18
|
||||||
|
batch_size: auto # auto-tune based on available VRAM
|
||||||
|
device: cuda
|
||||||
|
|
||||||
|
# -----------------------------------------------------
|
||||||
|
# Storage Paths
|
||||||
|
# -----------------------------------------------------
|
||||||
|
storage:
|
||||||
|
scratch_path: /scratch
|
||||||
|
input_path: /data/input
|
||||||
|
output_path: /data/output
|
||||||
|
models_path: /models
|
||||||
|
training_path: /data/training
|
||||||
|
|
||||||
|
# -----------------------------------------------------
|
||||||
|
# Database
|
||||||
|
# -----------------------------------------------------
|
||||||
|
database:
|
||||||
|
host: mariadb
|
||||||
|
port: 3306
|
||||||
|
name: videodetect
|
||||||
|
user: videodetect
|
||||||
|
password: videodetect123
|
||||||
|
pool_size: 20
|
||||||
|
pool_min: 5
|
||||||
|
pool_recycle: 3600
|
||||||
|
|
||||||
|
# -----------------------------------------------------
|
||||||
|
# Logging
|
||||||
|
# -----------------------------------------------------
|
||||||
|
logging:
|
||||||
|
format: json
|
||||||
|
level: INFO
|
||||||
|
rotation_max_bytes: 104857600 # 100MB
|
||||||
|
rotation_backup_count: 10
|
||||||
|
|
||||||
|
# -----------------------------------------------------
|
||||||
|
# Model Configuration
|
||||||
|
# -----------------------------------------------------
|
||||||
|
model:
|
||||||
|
face_detector: yolo8n
|
||||||
|
face_detector_path: /models/face_detector/face_detector.trt
|
||||||
|
classifier: mobilenetv3-small
|
||||||
|
classifier_path: /models/classifier/classifier.trt
|
||||||
|
input_size: 224
|
||||||
|
|
||||||
|
# -----------------------------------------------------
|
||||||
|
# Directory Scanner
|
||||||
|
# -----------------------------------------------------
|
||||||
|
scanner:
|
||||||
|
scan_interval_seconds: 60
|
||||||
|
walker_threads: 8
|
||||||
|
ffprobe_timeout_seconds: 10
|
||||||
|
hash_algorithm: sha256
|
||||||
|
hash_chunk_size_mb: 1
|
||||||
|
|
||||||
|
# -----------------------------------------------------
|
||||||
|
# Codec Validation
|
||||||
|
# -----------------------------------------------------
|
||||||
|
codec:
|
||||||
|
whitelist:
|
||||||
|
- avc1 # H.264
|
||||||
|
- hevc # H.265
|
||||||
|
- vp8
|
||||||
|
- vp9
|
||||||
|
- av01 # AV1
|
||||||
|
- mjpeg
|
||||||
|
- mp4v
|
||||||
|
default_status_on_error: UNSCANNABLE
|
||||||
|
|
||||||
|
# -----------------------------------------------------
|
||||||
|
# Processing Queue
|
||||||
|
# -----------------------------------------------------
|
||||||
|
queue:
|
||||||
|
priority: modification_time # newest first
|
||||||
|
max_concurrent_per_gpu: 1
|
||||||
|
lock_timeout_seconds: 300
|
||||||
|
|
||||||
|
# -----------------------------------------------------
|
||||||
|
# Face Detection
|
||||||
|
# -----------------------------------------------------
|
||||||
|
face_detection:
|
||||||
|
model: yolo8n
|
||||||
|
model_path: /models/face_detector/face_detector.trt
|
||||||
|
input_size: 640
|
||||||
|
confidence_threshold: 0.25
|
||||||
|
iou_threshold: 0.45
|
||||||
|
max_faces_per_frame: 10
|
||||||
|
max_faces_per_video: 100
|
||||||
|
|
||||||
|
# -----------------------------------------------------
|
||||||
|
# Batching
|
||||||
|
# -----------------------------------------------------
|
||||||
|
batching:
|
||||||
|
max_batch_size: 16
|
||||||
|
batch_timeout_ms: 100
|
||||||
|
vram_target_gb: 16
|
||||||
|
vram_reduce_threshold_gb: 16
|
||||||
|
vram_increase_threshold_gb: 10
|
||||||
|
|
||||||
|
# -----------------------------------------------------
|
||||||
|
# Classification
|
||||||
|
# -----------------------------------------------------
|
||||||
|
classifier:
|
||||||
|
model: mobilenetv3-small
|
||||||
|
model_path: /models/classifier/classifier.trt
|
||||||
|
input_size: 224
|
||||||
|
temperature: 1.0 # calibration temperature
|
||||||
|
default_strategy: max # max, weighted_mean, top_k_mean
|
||||||
|
|
||||||
|
# -----------------------------------------------------
|
||||||
|
# Confidence Aggregation
|
||||||
|
# -----------------------------------------------------
|
||||||
|
aggregation:
|
||||||
|
strategy: max
|
||||||
|
alpha: 1.0 # for weighted_mean
|
||||||
|
beta: 0.1 # for weighted_mean
|
||||||
|
top_k: 3 # for top_k_mean
|
||||||
|
|
||||||
|
# -----------------------------------------------------
|
||||||
|
# Routing
|
||||||
|
# -----------------------------------------------------
|
||||||
|
routing:
|
||||||
|
T_high: 0.75
|
||||||
|
T_low: 0.45
|
||||||
|
no_faces_decision: SKIP
|
||||||
|
|
||||||
|
# -----------------------------------------------------
|
||||||
|
# Results Persistence
|
||||||
|
# -----------------------------------------------------
|
||||||
|
results:
|
||||||
|
persist_before_cleanup: true
|
||||||
|
transaction_safe: true
|
||||||
|
state_guard: true # only update PROCESSING → COMPLETED
|
||||||
|
|
||||||
|
# -----------------------------------------------------
|
||||||
|
# Data Export
|
||||||
|
# -----------------------------------------------------
|
||||||
|
export:
|
||||||
|
format: parquet # parquet, jsonl, or both
|
||||||
|
output_path: /data/output
|
||||||
|
compression: snappy
|
||||||
|
batch_size: 100 # export after N videos
|
||||||
|
include_frame_confidences: true
|
||||||
|
|
||||||
|
# -----------------------------------------------------
|
||||||
|
# Review UI
|
||||||
|
# -----------------------------------------------------
|
||||||
|
review_ui:
|
||||||
|
host: "0.0.0.0"
|
||||||
|
port: 5000
|
||||||
|
per_page: 20
|
||||||
|
top_k_frames: 5
|
||||||
|
export_path: /data/output/reviews
|
||||||
|
auth_enabled: false
|
||||||
|
ssl_enabled: false
|
||||||
|
|
||||||
|
# -----------------------------------------------------
|
||||||
|
# Active Learning
|
||||||
|
# -----------------------------------------------------
|
||||||
|
active_learning:
|
||||||
|
enabled: true
|
||||||
|
min_annotated_samples: 100
|
||||||
|
training:
|
||||||
|
epochs: 20
|
||||||
|
batch_size: 32
|
||||||
|
learning_rate: 0.001
|
||||||
|
weight_decay: 0.01
|
||||||
|
early_stopping_patience: 5
|
||||||
|
lr_scheduler: ReduceLROnPlateau
|
||||||
|
lr_factor: 0.5
|
||||||
|
lr_patience: 3
|
||||||
|
validation:
|
||||||
|
val_split: 0.2
|
||||||
|
min_f1_improvement: 0.02
|
||||||
|
max_ece: 0.08
|
||||||
|
deployment:
|
||||||
|
auto_deploy: true
|
||||||
|
hot_reload: true
|
||||||
|
rollback_enabled: true
|
||||||
|
augmentation:
|
||||||
|
horizontal_flip: true
|
||||||
|
color_jitter: true
|
||||||
|
affine: true
|
||||||
|
affine_degrees: 10
|
||||||
|
affine_scale: 0.1
|
||||||
|
|
||||||
|
# -----------------------------------------------------
|
||||||
|
# Monitoring
|
||||||
|
# -----------------------------------------------------
|
||||||
|
monitoring:
|
||||||
|
prometheus:
|
||||||
|
enabled: true
|
||||||
|
port: 9090
|
||||||
|
metrics_path: /metrics
|
||||||
|
grafana:
|
||||||
|
enabled: true
|
||||||
|
port: 3000
|
||||||
|
alerts:
|
||||||
|
confidence_drift_threshold: 0.10
|
||||||
|
review_queue_max_size: 1000
|
||||||
|
review_queue_max_age_hours: 24
|
||||||
|
throughput_min_videos_per_hour: 20
|
||||||
|
throughput_min_duration_hours: 1
|
||||||
|
error_rate_threshold: 0.05
|
||||||
|
error_rate_window_hours: 1
|
||||||
|
drift_detection:
|
||||||
|
enabled: true
|
||||||
|
schedule: "0 2 * * 0" # cron: Sundays at 2 AM
|
||||||
|
baseline_source: db
|
||||||
|
crash_recovery:
|
||||||
|
lock_timeout_minutes: 5
|
||||||
|
auto_requeue: true
|
||||||
|
retry:
|
||||||
|
max_attempts: 3
|
||||||
|
initial_delay: 1.0
|
||||||
|
backoff_factor: 2.0
|
||||||
|
retryable_errors:
|
||||||
|
- timeout
|
||||||
|
- gpu_oom
|
||||||
|
- file_lock
|
||||||
|
non_retryable_errors:
|
||||||
|
- codec_unsupported
|
||||||
|
- file_corrupt
|
||||||
|
- invalid_path
|
||||||
+122
@@ -0,0 +1,122 @@
|
|||||||
|
--
|
||||||
|
-- VideoDetect Database Schema
|
||||||
|
-- Version: 1.0.0
|
||||||
|
-- Created: 2026-08-03
|
||||||
|
--
|
||||||
|
|
||||||
|
-- Create database (if not exists)
|
||||||
|
CREATE DATABASE IF NOT EXISTS videodetect
|
||||||
|
CHARACTER SET utf8mb4
|
||||||
|
COLLATE utf8mb4_unicode_ci;
|
||||||
|
|
||||||
|
USE videodetect;
|
||||||
|
|
||||||
|
-- -----------------------------------------------------
|
||||||
|
-- Table: videos
|
||||||
|
-- Stores metadata for each video file in the corpus
|
||||||
|
-- -----------------------------------------------------
|
||||||
|
CREATE TABLE IF NOT EXISTS videos (
|
||||||
|
id BIGINT AUTO_INCREMENT PRIMARY KEY,
|
||||||
|
file_path VARCHAR(2048) NOT NULL,
|
||||||
|
file_hash CHAR(64) NOT NULL COMMENT 'SHA-256 hash of file',
|
||||||
|
resolution_w INT DEFAULT NULL,
|
||||||
|
resolution_h INT DEFAULT NULL,
|
||||||
|
codec VARCHAR(50) DEFAULT NULL,
|
||||||
|
duration FLOAT DEFAULT NULL COMMENT 'Duration in seconds',
|
||||||
|
status ENUM('NEW', 'PENDING', 'PROCESSING', 'COMPLETED', 'UNSCANNABLE', 'ERROR')
|
||||||
|
NOT NULL DEFAULT 'NEW' COMMENT 'Processing status',
|
||||||
|
last_scan_time DATETIME DEFAULT NULL,
|
||||||
|
last_processed_time DATETIME DEFAULT NULL,
|
||||||
|
confidence_score FLOAT DEFAULT NULL COMMENT 'Video-level confidence score',
|
||||||
|
routing_decision ENUM('MATCH', 'REVIEW', 'SKIP') DEFAULT NULL COMMENT 'Routing decision',
|
||||||
|
model_version VARCHAR(50) DEFAULT NULL COMMENT 'Model version used for processing',
|
||||||
|
frame_count INT DEFAULT NULL COMMENT 'Number of frames sampled',
|
||||||
|
error_message TEXT DEFAULT NULL,
|
||||||
|
created_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||||
|
updated_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
|
||||||
|
|
||||||
|
UNIQUE KEY uk_file_hash (file_hash),
|
||||||
|
INDEX idx_videos_status (status),
|
||||||
|
INDEX idx_videos_last_scan (last_scan_time),
|
||||||
|
INDEX idx_videos_status_last_scan (status, last_scan_time),
|
||||||
|
INDEX idx_videos_file_path (file_path(255))
|
||||||
|
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
|
||||||
|
|
||||||
|
-- -----------------------------------------------------
|
||||||
|
-- Table: processing_logs
|
||||||
|
-- Audit trail for each video processing job
|
||||||
|
-- -----------------------------------------------------
|
||||||
|
CREATE TABLE IF NOT EXISTS processing_logs (
|
||||||
|
id BIGINT AUTO_INCREMENT PRIMARY KEY,
|
||||||
|
video_id BIGINT NOT NULL,
|
||||||
|
model_version VARCHAR(50) NOT NULL,
|
||||||
|
frame_count INT NOT NULL DEFAULT 0,
|
||||||
|
confidence_score FLOAT DEFAULT NULL,
|
||||||
|
confidence_scores JSON DEFAULT NULL COMMENT 'Frame-level confidence scores',
|
||||||
|
routing_decision ENUM('MATCH', 'REVIEW', 'SKIP') NOT NULL,
|
||||||
|
processed_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||||
|
error_message TEXT DEFAULT NULL,
|
||||||
|
FOREIGN KEY (video_id) REFERENCES videos(id) ON DELETE CASCADE,
|
||||||
|
INDEX idx_processing_logs_video (video_id),
|
||||||
|
INDEX idx_processing_logs_processed_at (processed_at),
|
||||||
|
INDEX idx_processing_logs_routing (routing_decision)
|
||||||
|
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
|
||||||
|
|
||||||
|
-- -----------------------------------------------------
|
||||||
|
-- Table: models
|
||||||
|
-- Model registry tracking all deployed and archived models
|
||||||
|
-- -----------------------------------------------------
|
||||||
|
CREATE TABLE IF NOT EXISTS models (
|
||||||
|
version VARCHAR(50) PRIMARY KEY,
|
||||||
|
status ENUM('ACTIVE', 'CANDIDATE', 'ARCHIVED') NOT NULL DEFAULT 'CANDIDATE',
|
||||||
|
path VARCHAR(2048) NOT NULL,
|
||||||
|
calibration_temp FLOAT DEFAULT NULL COMMENT 'Temperature scaling parameter',
|
||||||
|
f1_score FLOAT DEFAULT NULL COMMENT 'F1 score on validation set',
|
||||||
|
ece_score FLOAT DEFAULT NULL COMMENT 'Expected Calibration Error',
|
||||||
|
deployed_at DATETIME DEFAULT NULL,
|
||||||
|
created_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||||
|
INDEX idx_models_status (status)
|
||||||
|
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
|
||||||
|
|
||||||
|
-- -----------------------------------------------------
|
||||||
|
-- Table: review_queue
|
||||||
|
-- Videos awaiting manual annotation
|
||||||
|
-- -----------------------------------------------------
|
||||||
|
CREATE TABLE IF NOT EXISTS review_queue (
|
||||||
|
id BIGINT AUTO_INCREMENT PRIMARY KEY,
|
||||||
|
video_id BIGINT NOT NULL,
|
||||||
|
confidence_score FLOAT NOT NULL,
|
||||||
|
routing_decision ENUM('REVIEW') NOT NULL DEFAULT 'REVIEW',
|
||||||
|
annotated BOOLEAN NOT NULL DEFAULT FALSE,
|
||||||
|
ground_truth BOOLEAN DEFAULT NULL COMMENT 'True label from annotator',
|
||||||
|
annotated_at DATETIME DEFAULT NULL,
|
||||||
|
notes TEXT DEFAULT NULL,
|
||||||
|
created_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||||
|
FOREIGN KEY (video_id) REFERENCES videos(id) ON DELETE CASCADE,
|
||||||
|
INDEX idx_review_queue_annotated (annotated),
|
||||||
|
INDEX idx_review_queue_created (created_at)
|
||||||
|
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
|
||||||
|
|
||||||
|
-- -----------------------------------------------------
|
||||||
|
-- Table: scan_history
|
||||||
|
-- Tracks directory scan operations for incremental sync
|
||||||
|
-- -----------------------------------------------------
|
||||||
|
CREATE TABLE IF NOT EXISTS scan_history (
|
||||||
|
id BIGINT AUTO_INCREMENT PRIMARY KEY,
|
||||||
|
scan_start DATETIME NOT NULL,
|
||||||
|
scan_end DATETIME DEFAULT NULL,
|
||||||
|
files_discovered INT DEFAULT 0,
|
||||||
|
files_new INT DEFAULT 0,
|
||||||
|
files_modified INT DEFAULT 0,
|
||||||
|
files_removed INT DEFAULT 0,
|
||||||
|
files_unscannable INT DEFAULT 0,
|
||||||
|
duration_seconds FLOAT DEFAULT NULL,
|
||||||
|
status ENUM('RUNNING', 'COMPLETED', 'FAILED') NOT NULL DEFAULT 'RUNNING',
|
||||||
|
error_message TEXT DEFAULT NULL
|
||||||
|
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
|
||||||
|
|
||||||
|
-- -----------------------------------------------------
|
||||||
|
-- Insert default model entry
|
||||||
|
-- -----------------------------------------------------
|
||||||
|
INSERT IGNORE INTO models (version, status, path, calibration_temp, created_at)
|
||||||
|
VALUES ('v0.0.0-placeholder', 'ACTIVE', '/models/placeholder', 1.0, NOW());
|
||||||
@@ -0,0 +1,146 @@
|
|||||||
|
version: '3.8'
|
||||||
|
|
||||||
|
services:
|
||||||
|
mariadb:
|
||||||
|
image: mariadb:10.11
|
||||||
|
container_name: videodetect-mariadb
|
||||||
|
restart: unless-stopped
|
||||||
|
environment:
|
||||||
|
MYSQL_ROOT_PASSWORD: ${DB_ROOT_PASSWORD:-rootpass}
|
||||||
|
MYSQL_DATABASE: videodetect
|
||||||
|
MYSQL_USER: videodetect
|
||||||
|
MYSQL_PASSWORD: ${DB_PASSWORD:-videodetect123}
|
||||||
|
ports:
|
||||||
|
- "3306:3306"
|
||||||
|
volumes:
|
||||||
|
- mariadb_data:/var/lib/mysql
|
||||||
|
- ./db/schema.sql:/docker-entrypoint-initdb.d/01-schema.sql:ro
|
||||||
|
networks:
|
||||||
|
- videodetect-network
|
||||||
|
healthcheck:
|
||||||
|
test: ["CMD", "healthcheck.sh", "--connect", "--innodb_initialized"]
|
||||||
|
interval: 10s
|
||||||
|
timeout: 5s
|
||||||
|
retries: 5
|
||||||
|
deploy:
|
||||||
|
resources:
|
||||||
|
limits:
|
||||||
|
memory: 2G
|
||||||
|
|
||||||
|
worker:
|
||||||
|
build:
|
||||||
|
context: ./worker
|
||||||
|
dockerfile: Dockerfile
|
||||||
|
container_name: videodetect-worker
|
||||||
|
restart: unless-stopped
|
||||||
|
runtime: nvidia
|
||||||
|
environment:
|
||||||
|
- NVIDIA_VISIBLE_DEVICES=all
|
||||||
|
- NVIDIA_DRIVER_CAPABILITIES=compute,utility
|
||||||
|
- DB_HOST=mariadb
|
||||||
|
- DB_PORT=3306
|
||||||
|
- DB_NAME=videodetect
|
||||||
|
- DB_USER=videodetect
|
||||||
|
- DB_PASSWORD=${DB_PASSWORD:-videodetect123}
|
||||||
|
- CONFIG_PATH=/app/config.yaml
|
||||||
|
volumes:
|
||||||
|
- /dev/null:/dev/null # tmpfs mounted at /scratch in container
|
||||||
|
- ${NAS_INPUT_PATH:-/mnt/nas/input}:/data/input:ro
|
||||||
|
- ${NAS_OUTPUT_PATH:-/mnt/nas/output}:/data/output
|
||||||
|
- ${MODELS_PATH:-/mnt/nas/models}:/models
|
||||||
|
- ${TRAINING_PATH:-/mnt/nas/training}:/data/training
|
||||||
|
tmpfs:
|
||||||
|
- /scratch:noexec,nosuid,size=100G
|
||||||
|
deploy:
|
||||||
|
resources:
|
||||||
|
reservations:
|
||||||
|
devices:
|
||||||
|
- driver: nvidia
|
||||||
|
count: all
|
||||||
|
capabilities: [gpu]
|
||||||
|
limits:
|
||||||
|
memory: 24G
|
||||||
|
networks:
|
||||||
|
- videodetect-network
|
||||||
|
healthcheck:
|
||||||
|
test: ["CMD", "python3", "-c", "import torch; print(torch.cuda.is_available())"]
|
||||||
|
interval: 30s
|
||||||
|
timeout: 10s
|
||||||
|
retries: 3
|
||||||
|
start_period: 60s
|
||||||
|
|
||||||
|
ui:
|
||||||
|
build:
|
||||||
|
context: ./ui
|
||||||
|
dockerfile: Dockerfile
|
||||||
|
container_name: videodetect-ui
|
||||||
|
restart: unless-stopped
|
||||||
|
ports:
|
||||||
|
- "5000:5000"
|
||||||
|
environment:
|
||||||
|
- DB_HOST=mariadb
|
||||||
|
- DB_PORT=3306
|
||||||
|
- DB_NAME=videodetect
|
||||||
|
- DB_USER=videodetect
|
||||||
|
- DB_PASSWORD=${DB_PASSWORD:-videodetect123}
|
||||||
|
- FLASK_ENV=production
|
||||||
|
volumes:
|
||||||
|
- ./ui:/app/ui
|
||||||
|
networks:
|
||||||
|
- videodetect-network
|
||||||
|
depends_on:
|
||||||
|
mariadb:
|
||||||
|
condition: service_healthy
|
||||||
|
deploy:
|
||||||
|
resources:
|
||||||
|
limits:
|
||||||
|
memory: 512M
|
||||||
|
|
||||||
|
prometheus:
|
||||||
|
image: prom/prometheus:v2.48.0
|
||||||
|
container_name: videodetect-prometheus
|
||||||
|
restart: unless-stopped
|
||||||
|
ports:
|
||||||
|
- "9090:9090"
|
||||||
|
volumes:
|
||||||
|
- ./monitoring/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro
|
||||||
|
- prometheus_data:/prometheus
|
||||||
|
networks:
|
||||||
|
- videodetect-network
|
||||||
|
deploy:
|
||||||
|
resources:
|
||||||
|
limits:
|
||||||
|
memory: 1G
|
||||||
|
|
||||||
|
grafana:
|
||||||
|
image: grafana/grafana:10.2.0
|
||||||
|
container_name: videodetect-grafana
|
||||||
|
restart: unless-stopped
|
||||||
|
ports:
|
||||||
|
- "3000:3000"
|
||||||
|
environment:
|
||||||
|
- GF_SECURITY_ADMIN_PASSWORD=${GRAFANA_PASSWORD:-admin}
|
||||||
|
- GF_USERS_ALLOW_SIGN_UP=false
|
||||||
|
volumes:
|
||||||
|
- grafana_data:/var/lib/grafana
|
||||||
|
- ./monitoring/grafana/provisioning:/etc/grafana/provisioning:ro
|
||||||
|
networks:
|
||||||
|
- videodetect-network
|
||||||
|
depends_on:
|
||||||
|
- prometheus
|
||||||
|
deploy:
|
||||||
|
resources:
|
||||||
|
limits:
|
||||||
|
memory: 512M
|
||||||
|
|
||||||
|
volumes:
|
||||||
|
mariadb_data:
|
||||||
|
driver: local
|
||||||
|
prometheus_data:
|
||||||
|
driver: local
|
||||||
|
grafana_data:
|
||||||
|
driver: local
|
||||||
|
|
||||||
|
networks:
|
||||||
|
videodetect-network:
|
||||||
|
driver: bridge
|
||||||
@@ -0,0 +1,97 @@
|
|||||||
|
{
|
||||||
|
"annotations": { "list": [] },
|
||||||
|
"editable": true,
|
||||||
|
"fiscalYearStartMonth": 0,
|
||||||
|
"graphTooltip": 1,
|
||||||
|
"id": null,
|
||||||
|
"links": [],
|
||||||
|
"panels": [
|
||||||
|
{
|
||||||
|
"datasource": { "type": "prometheus", "uid": "prometheus" },
|
||||||
|
"fieldConfig": { "defaults": { "unit": "short" } },
|
||||||
|
"gridPos": { "h": 8, "w": 8, "x": 0, "y": 0 },
|
||||||
|
"id": 1,
|
||||||
|
"title": "Review Queue Size Over Time",
|
||||||
|
"type": "timeseries",
|
||||||
|
"targets": [
|
||||||
|
{
|
||||||
|
"expr": "queue_depth_review",
|
||||||
|
"legendFormat": "Queue Size"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"datasource": { "type": "prometheus", "uid": "prometheus" },
|
||||||
|
"fieldConfig": { "defaults": { "unit": "short" } },
|
||||||
|
"gridPos": { "h": 8, "w": 8, "x": 8, "y": 0 },
|
||||||
|
"id": 2,
|
||||||
|
"title": "Annotation Rate (Labels/Hour)",
|
||||||
|
"type": "timeseries",
|
||||||
|
"targets": [
|
||||||
|
{
|
||||||
|
"expr": "rate(videos_processed_total{routing_decision=\"REVIEW\"}[1h])",
|
||||||
|
"legendFormat": "Reviews/hour"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"datasource": { "type": "prometheus", "uid": "prometheus" },
|
||||||
|
"fieldConfig": { "defaults": { "unit": "short" } },
|
||||||
|
"gridPos": { "h": 8, "w": 8, "x": 16, "y": 0 },
|
||||||
|
"id": 3,
|
||||||
|
"title": "Model Version Timeline",
|
||||||
|
"type": "state-timeline",
|
||||||
|
"targets": [
|
||||||
|
{
|
||||||
|
"expr": "models{status=\"ACTIVE\"}",
|
||||||
|
"legendFormat": "{{version}} (ACTIVE)"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"expr": "models{status=\"CANDIDATE\"}",
|
||||||
|
"legendFormat": "{{version}} (CANDIDATE)"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"expr": "models{status=\"ARCHIVED\"}",
|
||||||
|
"legendFormat": "{{version}} (ARCHIVED)"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"datasource": { "type": "prometheus", "uid": "prometheus" },
|
||||||
|
"fieldConfig": { "defaults": { "unit": "percentunit", "min": 0, "max": 1 } },
|
||||||
|
"gridPos": { "h": 8, "w": 12, "x": 0, "y": 8 },
|
||||||
|
"id": 4,
|
||||||
|
"title": "F1 Score by Model Version",
|
||||||
|
"type": "timeseries",
|
||||||
|
"targets": [
|
||||||
|
{
|
||||||
|
"expr": "models_f1_score",
|
||||||
|
"legendFormat": "{{version}}"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"datasource": { "type": "prometheus", "uid": "prometheus" },
|
||||||
|
"fieldConfig": { "defaults": { "unit": "percentunit", "max": 1 } },
|
||||||
|
"gridPos": { "h": 8, "w": 12, "x": 12, "y": 8 },
|
||||||
|
"id": 5,
|
||||||
|
"title": "ECE by Model Version",
|
||||||
|
"type": "timeseries",
|
||||||
|
"targets": [
|
||||||
|
{
|
||||||
|
"expr": "models_ece_score",
|
||||||
|
"legendFormat": "{{version}}"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"schemaVersion": 38,
|
||||||
|
"style": "dark",
|
||||||
|
"tags": ["VideoDetect"],
|
||||||
|
"templating": { "list": [] },
|
||||||
|
"time": { "from": "now-24h", "to": "now" },
|
||||||
|
"timepicker": {},
|
||||||
|
"title": "VideoDetect - Active Learning",
|
||||||
|
"uid": "videodetect-active-learning",
|
||||||
|
"version": 1
|
||||||
|
}
|
||||||
@@ -0,0 +1,115 @@
|
|||||||
|
{
|
||||||
|
"annotations": { "list": [] },
|
||||||
|
"editable": true,
|
||||||
|
"fiscalYearStartMonth": 0,
|
||||||
|
"graphTooltip": 1,
|
||||||
|
"id": null,
|
||||||
|
"links": [],
|
||||||
|
"panels": [
|
||||||
|
{
|
||||||
|
"datasource": { "type": "prometheus", "uid": "prometheus" },
|
||||||
|
"fieldConfig": { "defaults": { "unit": "s" } },
|
||||||
|
"gridPos": { "h": 8, "w": 12, "x": 0, "y": 0 },
|
||||||
|
"id": 1,
|
||||||
|
"title": "Processing Duration per Video",
|
||||||
|
"type": "timeseries",
|
||||||
|
"targets": [
|
||||||
|
{
|
||||||
|
"expr": "histogram_quantile(0.5, rate(video_processing_duration_seconds_bucket[5m]))",
|
||||||
|
"legendFormat": "P50"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"expr": "histogram_quantile(0.95, rate(video_processing_duration_seconds_bucket[5m]))",
|
||||||
|
"legendFormat": "P95"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"datasource": { "type": "prometheus", "uid": "prometheus" },
|
||||||
|
"fieldConfig": { "defaults": { "unit": "short" } },
|
||||||
|
"gridPos": { "h": 8, "w": 6, "x": 12, "y": 0 },
|
||||||
|
"id": 2,
|
||||||
|
"title": "Frame Count per Video",
|
||||||
|
"type": "histogram",
|
||||||
|
"targets": [
|
||||||
|
{
|
||||||
|
"expr": "rate(frame_count_per_video[5m])",
|
||||||
|
"legendFormat": "Frames"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"datasource": { "type": "prometheus", "uid": "prometheus" },
|
||||||
|
"fieldConfig": { "defaults": { "unit": "short" } },
|
||||||
|
"gridPos": { "h": 8, "w": 6, "x": 18, "y": 0 },
|
||||||
|
"id": 3,
|
||||||
|
"title": "Face Count per Video",
|
||||||
|
"type": "histogram",
|
||||||
|
"targets": [
|
||||||
|
{
|
||||||
|
"expr": "rate(face_count_per_video[5m])",
|
||||||
|
"legendFormat": "Faces"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"datasource": { "type": "prometheus", "uid": "prometheus" },
|
||||||
|
"fieldConfig": { "defaults": { "unit": "percentunit" } },
|
||||||
|
"gridPos": { "h": 8, "w": 12, "x": 0, "y": 8 },
|
||||||
|
"id": 4,
|
||||||
|
"title": "Confidence Score by Model Version",
|
||||||
|
"type": "barchart",
|
||||||
|
"targets": [
|
||||||
|
{
|
||||||
|
"expr": "avg by (model_version) (confidence_score)",
|
||||||
|
"legendFormat": "{{model_version}}"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"datasource": { "type": "prometheus", "uid": "prometheus" },
|
||||||
|
"fieldConfig": { "defaults": { "unit": "percentunit" } },
|
||||||
|
"gridPos": { "h": 8, "w": 6, "x": 12, "y": 8 },
|
||||||
|
"id": 5,
|
||||||
|
"title": "Routing Decision Distribution",
|
||||||
|
"type": "piechart",
|
||||||
|
"targets": [
|
||||||
|
{
|
||||||
|
"expr": "increase(videos_processed_total{routing_decision=\"MATCH\"}[1h])",
|
||||||
|
"legendFormat": "MATCH"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"expr": "increase(videos_processed_total{routing_decision=\"REVIEW\"}[1h])",
|
||||||
|
"legendFormat": "REVIEW"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"expr": "increase(videos_processed_total{routing_decision=\"SKIP\"}[1h])",
|
||||||
|
"legendFormat": "SKIP"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"datasource": { "type": "prometheus", "uid": "prometheus" },
|
||||||
|
"fieldConfig": { "defaults": { "unit": "reqps" } },
|
||||||
|
"gridPos": { "h": 8, "w": 6, "x": 18, "y": 8 },
|
||||||
|
"id": 6,
|
||||||
|
"title": "Inference Throughput",
|
||||||
|
"type": "timeseries",
|
||||||
|
"targets": [
|
||||||
|
{
|
||||||
|
"expr": "rate(faces_detected_total[5m])",
|
||||||
|
"legendFormat": "Faces/sec"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"schemaVersion": 38,
|
||||||
|
"style": "dark",
|
||||||
|
"tags": ["VideoDetect"],
|
||||||
|
"templating": { "list": [] },
|
||||||
|
"time": { "from": "now-6h", "to": "now" },
|
||||||
|
"timepicker": {},
|
||||||
|
"title": "VideoDetect - Processing Details",
|
||||||
|
"uid": "videodetect-processing-details",
|
||||||
|
"version": 1
|
||||||
|
}
|
||||||
@@ -0,0 +1,108 @@
|
|||||||
|
{
|
||||||
|
"annotations": {
|
||||||
|
"list": []
|
||||||
|
},
|
||||||
|
"editable": true,
|
||||||
|
"fiscalYearStartMonth": 0,
|
||||||
|
"graphTooltip": 1,
|
||||||
|
"id": null,
|
||||||
|
"links": [],
|
||||||
|
"panels": [
|
||||||
|
{
|
||||||
|
"datasource": { "type": "prometheus", "uid": "prometheus" },
|
||||||
|
"fieldConfig": { "defaults": { "color": { "mode": "palette-classic" }, "unit": "videos/h" } },
|
||||||
|
"gridPos": { "h": 8, "w": 12, "x": 0, "y": 0 },
|
||||||
|
"id": 1,
|
||||||
|
"title": "Throughput (Videos/Hour)",
|
||||||
|
"type": "timeseries",
|
||||||
|
"targets": [
|
||||||
|
{
|
||||||
|
"expr": "rate(videos_processed_total[5m]) * 3600",
|
||||||
|
"legendFormat": "{{routing_decision}}"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"datasource": { "type": "prometheus", "uid": "prometheus" },
|
||||||
|
"fieldConfig": { "defaults": { "unit": "short" } },
|
||||||
|
"gridPos": { "h": 8, "w": 6, "x": 12, "y": 0 },
|
||||||
|
"id": 2,
|
||||||
|
"title": "Queue Depth",
|
||||||
|
"type": "stat",
|
||||||
|
"targets": [
|
||||||
|
{ "expr": "queue_depth_pending", "legendFormat": "Pending" },
|
||||||
|
{ "expr": "queue_depth_processing", "legendFormat": "Processing" },
|
||||||
|
{ "expr": "queue_depth_review", "legendFormat": "Review" }
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"datasource": { "type": "prometheus", "uid": "prometheus" },
|
||||||
|
"fieldConfig": { "defaults": { "unit": "percent", "max": 100 } },
|
||||||
|
"gridPos": { "h": 8, "w": 6, "x": 18, "y": 0 },
|
||||||
|
"id": 3,
|
||||||
|
"title": "GPU Utilization",
|
||||||
|
"type": "gauge",
|
||||||
|
"targets": [
|
||||||
|
{ "expr": "gpu_utilization_percent{gpu_id=\"0\"}", "legendFormat": "GPU 0" },
|
||||||
|
{ "expr": "gpu_utilization_percent{gpu_id=\"1\"}", "legendFormat": "GPU 1" }
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"datasource": { "type": "prometheus", "uid": "prometheus" },
|
||||||
|
"fieldConfig": { "defaults": { "unit": "bytes", "max": 24000000000 } },
|
||||||
|
"gridPos": { "h": 8, "w": 12, "x": 0, "y": 8 },
|
||||||
|
"id": 4,
|
||||||
|
"title": "GPU Memory Usage",
|
||||||
|
"type": "timeseries",
|
||||||
|
"targets": [
|
||||||
|
{ "expr": "gpu_memory_used_bytes{gpu_id=\"0\"} / 1073741824", "legendFormat": "GPU 0 (GB)" },
|
||||||
|
{ "expr": "gpu_memory_used_bytes{gpu_id=\"1\"} / 1073741824", "legendFormat": "GPU 1 (GB)" }
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"datasource": { "type": "prometheus", "uid": "prometheus" },
|
||||||
|
"fieldConfig": { "defaults": { "unit": "short" } },
|
||||||
|
"gridPos": { "h": 8, "w": 6, "x": 12, "y": 8 },
|
||||||
|
"id": 5,
|
||||||
|
"title": "Error Rate (per 100 videos)",
|
||||||
|
"type": "stat",
|
||||||
|
"targets": [
|
||||||
|
{
|
||||||
|
"expr": "rate(inference_errors_total[5m]) / (rate(videos_processed_total[5m]) + 0.001) * 100",
|
||||||
|
"legendFormat": "Errors/100"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"datasource": { "type": "prometheus", "uid": "prometheus" },
|
||||||
|
"fieldConfig": { "defaults": { "unit": "percentunit" } },
|
||||||
|
"gridPos": { "h": 8, "w": 6, "x": 18, "y": 8 },
|
||||||
|
"id": 6,
|
||||||
|
"title": "Confidence Distribution",
|
||||||
|
"type": "piechart",
|
||||||
|
"targets": [
|
||||||
|
{
|
||||||
|
"expr": "sum(increase(confidence_score_distribution_bucket{le=\"0.45\"}[1h]))",
|
||||||
|
"legendFormat": "SKIP (<0.45)"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"expr": "sum(increase(confidence_score_distribution_bucket{le=\"0.75\"}[1h])) - sum(increase(confidence_score_distribution_bucket{le=\"0.45\"}[1h]))",
|
||||||
|
"legendFormat": "REVIEW (0.45-0.75)"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"expr": "sum(increase(confidence_score_distribution_bucket{le=\"1\"}[1h])) - sum(increase(confidence_score_distribution_bucket{le=\"0.75\"}[1h]))",
|
||||||
|
"legendFormat": "MATCH (>0.75)"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"schemaVersion": 38,
|
||||||
|
"style": "dark",
|
||||||
|
"tags": ["VideoDetect"],
|
||||||
|
"templating": { "list": [] },
|
||||||
|
"time": { "from": "now-1h", "to": "now" },
|
||||||
|
"timepicker": {},
|
||||||
|
"title": "VideoDetect - System Overview",
|
||||||
|
"uid": "videodetect-system-overview",
|
||||||
|
"version": 1
|
||||||
|
}
|
||||||
@@ -0,0 +1,11 @@
|
|||||||
|
apiVersion: 1
|
||||||
|
|
||||||
|
providers:
|
||||||
|
- name: "default"
|
||||||
|
orgId: 1
|
||||||
|
folder: "VideoDetect"
|
||||||
|
type: file
|
||||||
|
disableDeletion: false
|
||||||
|
editable: true
|
||||||
|
options:
|
||||||
|
path: /etc/grafana/provisioning/dashboards
|
||||||
@@ -0,0 +1,9 @@
|
|||||||
|
apiVersion: 1
|
||||||
|
|
||||||
|
datasources:
|
||||||
|
- name: Prometheus
|
||||||
|
type: prometheus
|
||||||
|
access: proxy
|
||||||
|
url: http://prometheus:9090
|
||||||
|
isDefault: true
|
||||||
|
editable: false
|
||||||
@@ -0,0 +1,26 @@
|
|||||||
|
global:
|
||||||
|
scrape_interval: 15s
|
||||||
|
evaluation_interval: 15s
|
||||||
|
|
||||||
|
scrape_configs:
|
||||||
|
- job_name: "worker"
|
||||||
|
metrics_path: "/metrics"
|
||||||
|
scrape_interval: 10s
|
||||||
|
static_configs:
|
||||||
|
- targets: ["worker:9090"]
|
||||||
|
labels:
|
||||||
|
service: "videodetect-worker"
|
||||||
|
|
||||||
|
- job_name: "ui"
|
||||||
|
metrics_path: "/metrics"
|
||||||
|
scrape_interval: 30s
|
||||||
|
static_configs:
|
||||||
|
- targets: ["ui:5000"]
|
||||||
|
labels:
|
||||||
|
service: "videodetect-ui"
|
||||||
|
|
||||||
|
- job_name: "prometheus"
|
||||||
|
static_configs:
|
||||||
|
- targets: ["localhost:9090"]
|
||||||
|
labels:
|
||||||
|
service: "prometheus"
|
||||||
@@ -0,0 +1,68 @@
|
|||||||
|
#!/usr/bin/perl
|
||||||
|
use strict;
|
||||||
|
use MIME::Base64 qw(encode_base64);
|
||||||
|
use LWP::UserAgent;
|
||||||
|
use JSON;
|
||||||
|
use Data::Dumper;
|
||||||
|
|
||||||
|
my $seconds = 30;
|
||||||
|
my $ollama = 'http://10.0.1.7:11434';
|
||||||
|
my $resolution = "672:672";
|
||||||
|
my $sensitivity = 20;
|
||||||
|
|
||||||
|
my $file = $ARGV[0] || die "Usage: $0 <filename>\n";
|
||||||
|
|
||||||
|
system("ffmpeg -i \"$file\" -vf fps=1/$seconds,scale=$resolution output_%04d.png");
|
||||||
|
my $ua = LWP::UserAgent->new;
|
||||||
|
$ua->timeout(60);
|
||||||
|
|
||||||
|
my $total=0;
|
||||||
|
my $yes=0;
|
||||||
|
$/ = undef;
|
||||||
|
foreach my $file (<*.png>) {
|
||||||
|
print "$file...\n";
|
||||||
|
open(IN, '<:raw', $file) || die "Unable to read $file: $!\n";
|
||||||
|
my $data = <IN>;
|
||||||
|
close(IN);
|
||||||
|
|
||||||
|
my $base64 = encode_base64($data);
|
||||||
|
$data=undef;
|
||||||
|
|
||||||
|
my $ref = {
|
||||||
|
'model' => 'srizon/pixie:latest',
|
||||||
|
'prompt' => "Are there any african american people or in this photo, whether they are shown completely or not? Please answer 'yes' or 'no' only.",
|
||||||
|
'images' => [$base64],
|
||||||
|
};
|
||||||
|
|
||||||
|
my $url = "$ollama/api/generate";
|
||||||
|
my $res = GetRest($url, $ref);
|
||||||
|
print "\t$res\n";
|
||||||
|
$total++;
|
||||||
|
if($res=~/yes/i) {
|
||||||
|
$yes++;
|
||||||
|
}
|
||||||
|
unlink($file);
|
||||||
|
}
|
||||||
|
|
||||||
|
my $percentage = ($yes*100)/$total;
|
||||||
|
print "$yes of $total images gave yes - $percentage\n";
|
||||||
|
if($percentage >= $sensitivity) {
|
||||||
|
print "Probably a black dude, $percentage\n";
|
||||||
|
} else {
|
||||||
|
print "Probably not, $percentage\n";
|
||||||
|
}
|
||||||
|
|
||||||
|
sub GetRest {
|
||||||
|
my($url, $ref)=@_;
|
||||||
|
my $payload = encode_json($ref);
|
||||||
|
|
||||||
|
my $response = $ua->post($url, 'Content_Type' => 'application/json', 'Content' => $payload);
|
||||||
|
if($response->is_success) {
|
||||||
|
my $content = $response->decoded_content;
|
||||||
|
my @parts = split(/\n/, $content);
|
||||||
|
my @refs = map { decode_json($_) } @parts;
|
||||||
|
return join(' ', map { $_->{'response'} } @refs);
|
||||||
|
} else {
|
||||||
|
die "Error: " . $response->status_line;
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,126 @@
|
|||||||
|
"""
|
||||||
|
Configuration loader for VideoDetect.
|
||||||
|
|
||||||
|
Loads config.yaml with environment variable overrides.
|
||||||
|
Provides typed accessors and validates all required fields.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import logging
|
||||||
|
import os
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Any, Dict, Optional
|
||||||
|
|
||||||
|
import yaml
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
# Default config path
|
||||||
|
DEFAULT_CONFIG_PATH = Path(__file__).parent.parent / "config.yaml"
|
||||||
|
|
||||||
|
|
||||||
|
class Config:
|
||||||
|
"""Typed configuration manager with environment variable overrides."""
|
||||||
|
|
||||||
|
def __init__(self, config_path: Optional[str] = None):
|
||||||
|
self._path = Path(config_path) if config_path else DEFAULT_CONFIG_PATH
|
||||||
|
self._data: Dict[str, Any] = {}
|
||||||
|
self._load()
|
||||||
|
|
||||||
|
def _load(self):
|
||||||
|
"""Load config from YAML file."""
|
||||||
|
if not self._path.exists():
|
||||||
|
raise FileNotFoundError(f"Config file not found: {self._path}")
|
||||||
|
|
||||||
|
with open(self._path, "r") as f:
|
||||||
|
self._data = yaml.safe_load(f) or {}
|
||||||
|
|
||||||
|
# Apply environment variable overrides
|
||||||
|
self._apply_env_overrides()
|
||||||
|
logger.info("Config loaded from %s", self._path)
|
||||||
|
|
||||||
|
def _apply_env_overrides(self):
|
||||||
|
"""Override config values with environment variables."""
|
||||||
|
overrides = {
|
||||||
|
"DB_HOST": ("database.host", None),
|
||||||
|
"DB_PORT": ("database.port", int),
|
||||||
|
"DB_NAME": ("database.name", None),
|
||||||
|
"DB_USER": ("database.user", None),
|
||||||
|
"DB_PASSWORD": ("database.password", None),
|
||||||
|
"DB_POOL_SIZE": ("database.pool_size", int),
|
||||||
|
"DB_POOL_MIN": ("database.pool_min", int),
|
||||||
|
"GPU_MAX_MEMORY_GB": ("gpu.max_memory_gb", float),
|
||||||
|
"SAMPLING_INTERVAL_SECONDS": ("sampling.interval_seconds", int),
|
||||||
|
"T_HIGH": ("thresholds.T_high", float),
|
||||||
|
"T_LOW": ("thresholds.T_low", float),
|
||||||
|
"CONFIG_PATH": (None, None), # handled separately
|
||||||
|
}
|
||||||
|
|
||||||
|
for env_key, (config_path, type_fn) in overrides.items():
|
||||||
|
env_val = os.environ.get(env_key)
|
||||||
|
if env_val is not None:
|
||||||
|
if config_path is None:
|
||||||
|
continue
|
||||||
|
if type_fn is not None:
|
||||||
|
env_val = type_fn(env_val)
|
||||||
|
self._set_nested(self._data, config_path, env_val)
|
||||||
|
logger.debug("Config override: %s=%s (from %s)", config_path, env_val, env_key)
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _set_nested(data: Dict, path: str, value):
|
||||||
|
"""Set a value in a nested dict using dot notation."""
|
||||||
|
keys = path.split(".")
|
||||||
|
d = data
|
||||||
|
for key in keys[:-1]:
|
||||||
|
d = d.setdefault(key, {})
|
||||||
|
d[keys[-1]] = value
|
||||||
|
|
||||||
|
def get(self, path: str, default=None):
|
||||||
|
"""Get a config value using dot notation."""
|
||||||
|
keys = path.split(".")
|
||||||
|
d = self._data
|
||||||
|
for key in keys:
|
||||||
|
if isinstance(d, dict):
|
||||||
|
d = d.get(key, default)
|
||||||
|
else:
|
||||||
|
return default
|
||||||
|
return d if d is not None else default
|
||||||
|
|
||||||
|
def get_section(self, section: str) -> Dict:
|
||||||
|
"""Get an entire config section as a dict."""
|
||||||
|
return self._data.get(section, {})
|
||||||
|
|
||||||
|
def validate(self, required_keys: list) -> list:
|
||||||
|
"""Validate that required config keys exist. Returns list of missing keys."""
|
||||||
|
missing = []
|
||||||
|
for key in required_keys:
|
||||||
|
if self.get(key) is None:
|
||||||
|
missing.append(key)
|
||||||
|
if missing:
|
||||||
|
logger.error("Missing required config keys: %s", missing)
|
||||||
|
return missing
|
||||||
|
|
||||||
|
@property
|
||||||
|
def data(self) -> Dict[str, Any]:
|
||||||
|
"""Access raw config data."""
|
||||||
|
return self._data
|
||||||
|
|
||||||
|
def __repr__(self):
|
||||||
|
return f"Config(path={self._path}, sections={list(self._data.keys())})"
|
||||||
|
|
||||||
|
|
||||||
|
# Module-level singleton
|
||||||
|
_config: Optional[Config] = None
|
||||||
|
|
||||||
|
|
||||||
|
def get_config(config_path: Optional[str] = None) -> Config:
|
||||||
|
"""Get or create the global config singleton."""
|
||||||
|
global _config
|
||||||
|
if _config is None:
|
||||||
|
_config = Config(config_path)
|
||||||
|
return _config
|
||||||
|
|
||||||
|
|
||||||
|
def reset_config():
|
||||||
|
"""Reset the config singleton (useful for testing)."""
|
||||||
|
global _config
|
||||||
|
_config = None
|
||||||
@@ -0,0 +1,137 @@
|
|||||||
|
"""
|
||||||
|
Database connection layer with connection pooling for VideoDetect.
|
||||||
|
|
||||||
|
Provides:
|
||||||
|
- Connection pooling via DBUtils + PyMySQL
|
||||||
|
- Automatic reconnection on disconnect
|
||||||
|
- Context manager support
|
||||||
|
- Prepared statements for all queries
|
||||||
|
- Transaction support for atomic state transitions
|
||||||
|
"""
|
||||||
|
|
||||||
|
import contextlib
|
||||||
|
import logging
|
||||||
|
from typing import Optional
|
||||||
|
|
||||||
|
from dbutils.pooled_db import PooledDB
|
||||||
|
import pymysql
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
|
||||||
|
class DBConnector:
|
||||||
|
"""Thread-safe database connection pool manager."""
|
||||||
|
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
host: str = "mariadb",
|
||||||
|
port: int = 3306,
|
||||||
|
database: str = "videodetect",
|
||||||
|
user: str = "videodetect",
|
||||||
|
password: str = "videodetect123",
|
||||||
|
pool_size: int = 20,
|
||||||
|
pool_min: int = 5,
|
||||||
|
pool_recycle: int = 3600,
|
||||||
|
):
|
||||||
|
self._pool = PooledDB(
|
||||||
|
creator=pymysql,
|
||||||
|
maxconnections=pool_size,
|
||||||
|
mincached=pool_min,
|
||||||
|
maxcached=pool_size,
|
||||||
|
maxusage=200,
|
||||||
|
blocking=True,
|
||||||
|
max_idle_time=pool_recycle,
|
||||||
|
connection_timeout=10,
|
||||||
|
charset="utf8mb4",
|
||||||
|
cursorclass=pymysql.cursors.DictCursor,
|
||||||
|
host=host,
|
||||||
|
port=port,
|
||||||
|
database=database,
|
||||||
|
user=user,
|
||||||
|
password=password,
|
||||||
|
read_timeout=30,
|
||||||
|
write_timeout=30,
|
||||||
|
)
|
||||||
|
logger.info(
|
||||||
|
"DB pool initialized: host=%s db=%s pool_size=%d min=%d",
|
||||||
|
host, database, pool_size, pool_min,
|
||||||
|
)
|
||||||
|
|
||||||
|
def get_connection(self):
|
||||||
|
"""Get a connection from the pool."""
|
||||||
|
return self._pool.connection()
|
||||||
|
|
||||||
|
@contextlib.contextmanager
|
||||||
|
def get_cursor(self, transaction: bool = False):
|
||||||
|
"""Context manager for getting a cursor with optional transaction support."""
|
||||||
|
conn = self.get_connection()
|
||||||
|
try:
|
||||||
|
cursor = conn.cursor()
|
||||||
|
if transaction:
|
||||||
|
conn.begin()
|
||||||
|
yield cursor
|
||||||
|
if transaction:
|
||||||
|
conn.commit()
|
||||||
|
except Exception:
|
||||||
|
if transaction:
|
||||||
|
conn.rollback()
|
||||||
|
raise
|
||||||
|
finally:
|
||||||
|
cursor.close()
|
||||||
|
conn.close()
|
||||||
|
|
||||||
|
@contextlib.contextmanager
|
||||||
|
def transaction(self):
|
||||||
|
"""Context manager for a full transaction."""
|
||||||
|
conn = self.get_connection()
|
||||||
|
try:
|
||||||
|
conn.begin()
|
||||||
|
yield conn
|
||||||
|
conn.commit()
|
||||||
|
except Exception:
|
||||||
|
conn.rollback()
|
||||||
|
raise
|
||||||
|
finally:
|
||||||
|
conn.close()
|
||||||
|
|
||||||
|
def execute(self, query: str, params=None, transaction: bool = False):
|
||||||
|
"""Execute a query and return affected rows."""
|
||||||
|
with self.get_cursor(transaction=transaction) as cursor:
|
||||||
|
cursor.execute(query, params or ())
|
||||||
|
return cursor.rowcount
|
||||||
|
|
||||||
|
def fetchone(self, query: str, params=None):
|
||||||
|
"""Execute a query and return one row."""
|
||||||
|
with self.get_cursor() as cursor:
|
||||||
|
cursor.execute(query, params or ())
|
||||||
|
return cursor.fetchone()
|
||||||
|
|
||||||
|
def fetchall(self, query: str, params=None):
|
||||||
|
"""Execute a query and return all rows."""
|
||||||
|
with self.get_cursor() as cursor:
|
||||||
|
cursor.execute(query, params or ())
|
||||||
|
return cursor.fetchall()
|
||||||
|
|
||||||
|
def initialize_schema(self, schema_path: str = "db/schema.sql"):
|
||||||
|
"""Initialize the database schema from SQL file."""
|
||||||
|
with open(schema_path, "r") as f:
|
||||||
|
sql = f.read()
|
||||||
|
# Split on semicolons and execute each statement
|
||||||
|
statements = [s.strip() for s in sql.split(";") if s.strip()]
|
||||||
|
for stmt in statements:
|
||||||
|
if stmt.startswith("--"):
|
||||||
|
continue
|
||||||
|
try:
|
||||||
|
self.execute(stmt)
|
||||||
|
except Exception as e:
|
||||||
|
logger.debug("Schema statement skipped (may already exist): %s", e)
|
||||||
|
logger.info("Schema initialized from %s", schema_path)
|
||||||
|
|
||||||
|
def health_check(self) -> bool:
|
||||||
|
"""Check if the database is reachable."""
|
||||||
|
try:
|
||||||
|
result = self.fetchone("SELECT 1")
|
||||||
|
return result is not None
|
||||||
|
except Exception as e:
|
||||||
|
logger.error("Health check failed: %s", e)
|
||||||
|
return False
|
||||||
@@ -0,0 +1,89 @@
|
|||||||
|
"""
|
||||||
|
Logging configuration for VideoDetect.
|
||||||
|
|
||||||
|
Sets up JSON structured logging via python-json-logger with:
|
||||||
|
- Configurable log levels
|
||||||
|
- Standardized field names
|
||||||
|
- Log rotation
|
||||||
|
- All logs to stdout for Docker capture
|
||||||
|
"""
|
||||||
|
|
||||||
|
import json
|
||||||
|
import logging
|
||||||
|
import logging.handlers
|
||||||
|
import sys
|
||||||
|
from datetime import datetime, timezone
|
||||||
|
|
||||||
|
from python_json_logger import json_formatter
|
||||||
|
|
||||||
|
|
||||||
|
def setup_logging(
|
||||||
|
level: str = "INFO",
|
||||||
|
log_format: str = "json",
|
||||||
|
rotation_max_bytes: int = 104857600, # 100MB
|
||||||
|
rotation_backup_count: int = 10,
|
||||||
|
):
|
||||||
|
"""Configure structured logging for the application."""
|
||||||
|
log_level = getattr(logging, level.upper(), logging.INFO)
|
||||||
|
|
||||||
|
# Root logger
|
||||||
|
root_logger = logging.getLogger()
|
||||||
|
root_logger.setLevel(log_level)
|
||||||
|
|
||||||
|
# Remove existing handlers
|
||||||
|
root_logger.handlers.clear()
|
||||||
|
|
||||||
|
# Handler: stdout (for Docker capture)
|
||||||
|
handler = logging.StreamHandler(sys.stdout)
|
||||||
|
handler.setLevel(log_level)
|
||||||
|
|
||||||
|
if log_format == "json":
|
||||||
|
formatter = JsonFormatter()
|
||||||
|
else:
|
||||||
|
formatter = logging.Formatter(
|
||||||
|
"%(asctime)s %(levelname)s %(name)s %(message)s",
|
||||||
|
datefmt="%Y-%m-%dT%H:%M:%S%z",
|
||||||
|
)
|
||||||
|
|
||||||
|
handler.setFormatter(formatter)
|
||||||
|
root_logger.addHandler(handler)
|
||||||
|
|
||||||
|
# Handler: file rotation (for persistence)
|
||||||
|
file_handler = logging.handlers.RotatingFileHandler(
|
||||||
|
filename="/logs/videodetect.log",
|
||||||
|
maxBytes=rotation_max_bytes,
|
||||||
|
backupCount=rotation_backup_count,
|
||||||
|
encoding="utf-8",
|
||||||
|
)
|
||||||
|
file_handler.setLevel(log_level)
|
||||||
|
file_handler.setFormatter(formatter)
|
||||||
|
root_logger.addHandler(file_handler)
|
||||||
|
|
||||||
|
logging.info("Logging configured: level=%s format=%s", level, log_format)
|
||||||
|
|
||||||
|
|
||||||
|
class JsonFormatter(logging.Formatter):
|
||||||
|
"""JSON log formatter with standardized fields."""
|
||||||
|
|
||||||
|
def format(self, record: logging.LogRecord) -> str:
|
||||||
|
log_data = {
|
||||||
|
"timestamp": datetime.fromtimestamp(record.created, tz=timezone.utc).isoformat(),
|
||||||
|
"level": record.levelname,
|
||||||
|
"logger": record.name,
|
||||||
|
"message": record.getMessage(),
|
||||||
|
"module": record.module,
|
||||||
|
"function": record.funcName,
|
||||||
|
"line": record.lineno,
|
||||||
|
}
|
||||||
|
|
||||||
|
# Add extra fields
|
||||||
|
if hasattr(record, "video_id"):
|
||||||
|
log_data["video_id"] = record.video_id
|
||||||
|
if hasattr(record, "metadata"):
|
||||||
|
log_data["metadata"] = record.metadata
|
||||||
|
|
||||||
|
# Add exception info if present
|
||||||
|
if record.exc_info and record.exc_info[0] is not None:
|
||||||
|
log_data["exception"] = self.formatException(record.exc_info)
|
||||||
|
|
||||||
|
return json.dumps(log_data, default=str)
|
||||||
+99
@@ -0,0 +1,99 @@
|
|||||||
|
"""
|
||||||
|
VideoDetect - Video Classification System
|
||||||
|
|
||||||
|
Main entry point for the worker service.
|
||||||
|
Initializes all components and starts the processing pipeline.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import logging
|
||||||
|
import signal
|
||||||
|
import sys
|
||||||
|
import time
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
# Add src to path
|
||||||
|
sys.path.insert(0, str(Path(__file__).parent))
|
||||||
|
|
||||||
|
from config_loader import get_config
|
||||||
|
from db_connector import DBConnector
|
||||||
|
from logging_config import setup_logging
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
|
||||||
|
def graceful_shutdown(signum, frame):
|
||||||
|
"""Handle shutdown signals gracefully."""
|
||||||
|
logger.info("Received signal %d, initiating graceful shutdown...", signum)
|
||||||
|
sys.exit(0)
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
"""Initialize and start the VideoDetect worker."""
|
||||||
|
# Register signal handlers
|
||||||
|
signal.signal(signal.SIGTERM, graceful_shutdown)
|
||||||
|
signal.signal(signal.SIGINT, graceful_shutdown)
|
||||||
|
|
||||||
|
# Load configuration
|
||||||
|
config = get_config()
|
||||||
|
log_config = config.get_section("logging")
|
||||||
|
setup_logging(
|
||||||
|
level=log_config.get("level", "INFO"),
|
||||||
|
log_format=log_config.get("format", "json"),
|
||||||
|
rotation_max_bytes=log_config.get("rotation_max_bytes", 104857600),
|
||||||
|
rotation_backup_count=log_config.get("rotation_backup_count", 10),
|
||||||
|
)
|
||||||
|
|
||||||
|
logger.info("VideoDetect Worker starting...")
|
||||||
|
logger.info("Config: %s", config)
|
||||||
|
|
||||||
|
# Initialize database connection
|
||||||
|
db_config = config.get_section("database")
|
||||||
|
db = DBConnector(
|
||||||
|
host=db_config.get("host", "mariadb"),
|
||||||
|
port=db_config.get("port", 3306),
|
||||||
|
database=db_config.get("name", "videodetect"),
|
||||||
|
user=db_config.get("user", "videodetect"),
|
||||||
|
password=db_config.get("password", "videodetect123"),
|
||||||
|
pool_size=db_config.get("pool_size", 20),
|
||||||
|
pool_min=db_config.get("pool_min", 5),
|
||||||
|
pool_recycle=db_config.get("pool_recycle", 3600),
|
||||||
|
)
|
||||||
|
|
||||||
|
# Verify database connectivity
|
||||||
|
if not db.health_check():
|
||||||
|
logger.error("Cannot connect to database. Exiting.")
|
||||||
|
sys.exit(1)
|
||||||
|
logger.info("Database connection established.")
|
||||||
|
|
||||||
|
# Initialize schema if needed
|
||||||
|
schema_path = Path(__file__).parent.parent / "db" / "schema.sql"
|
||||||
|
if schema_path.exists():
|
||||||
|
db.initialize_schema(str(schema_path))
|
||||||
|
logger.info("Schema initialized.")
|
||||||
|
|
||||||
|
# Verify GPU availability
|
||||||
|
try:
|
||||||
|
import torch
|
||||||
|
if torch.cuda.is_available():
|
||||||
|
gpu_count = torch.cuda.device_count()
|
||||||
|
gpu_name = torch.cuda.get_device_name(0)
|
||||||
|
logger.info("GPU available: %d GPUs, primary: %s", gpu_count, gpu_name)
|
||||||
|
else:
|
||||||
|
logger.warning("CUDA is not available! Processing will be slow.")
|
||||||
|
except ImportError:
|
||||||
|
logger.warning("PyTorch not installed. GPU features disabled.")
|
||||||
|
|
||||||
|
logger.info("Worker initialization complete. Starting processing loop...")
|
||||||
|
|
||||||
|
# TODO: Start scanner, processor, and monitoring services
|
||||||
|
# This is the skeleton - actual processing logic is in subsequent stories
|
||||||
|
|
||||||
|
try:
|
||||||
|
while True:
|
||||||
|
time.sleep(60) # Main loop placeholder
|
||||||
|
except KeyboardInterrupt:
|
||||||
|
logger.info("Worker shutting down.")
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
@@ -0,0 +1,16 @@
|
|||||||
|
FROM python:3.10-slim
|
||||||
|
|
||||||
|
WORKDIR /app
|
||||||
|
|
||||||
|
RUN apt-get update && apt-get install -y --no-install-recommends \
|
||||||
|
curl \
|
||||||
|
&& rm -rf /var/lib/apt/lists/*
|
||||||
|
|
||||||
|
COPY ui/requirements.txt /app/requirements.txt
|
||||||
|
RUN pip install --no-cache-dir -r requirements.txt
|
||||||
|
|
||||||
|
COPY ui/ /app/ui/
|
||||||
|
|
||||||
|
EXPOSE 5000
|
||||||
|
|
||||||
|
CMD ["gunicorn", "--bind", "0.0.0.0:5000", "--workers", "2", "--timeout", "120", "app:app"]
|
||||||
@@ -0,0 +1,370 @@
|
|||||||
|
"""
|
||||||
|
VideoDetect Review UI - Flask Application
|
||||||
|
|
||||||
|
Lightweight web interface for annotating low-confidence videos.
|
||||||
|
Accessible via http://<server-ip>:5000
|
||||||
|
No authentication or SSL (per TC-06).
|
||||||
|
"""
|
||||||
|
|
||||||
|
import os
|
||||||
|
from flask import Flask, jsonify, request, render_template_string
|
||||||
|
|
||||||
|
app = Flask(__name__)
|
||||||
|
|
||||||
|
# In-memory storage for demo (replace with DB queries in production)
|
||||||
|
review_queue = []
|
||||||
|
|
||||||
|
|
||||||
|
# ------ Routes ------
|
||||||
|
|
||||||
|
@app.route("/")
|
||||||
|
def index():
|
||||||
|
"""Queue page - list of videos awaiting review."""
|
||||||
|
page = request.args.get("page", 1, type=int)
|
||||||
|
per_page = 20
|
||||||
|
|
||||||
|
# Filter unannotated
|
||||||
|
unannotated = [v for v in review_queue if not v.get("annotated")]
|
||||||
|
total = len(unannotated)
|
||||||
|
start = (page - 1) * per_page
|
||||||
|
end = start + per_page
|
||||||
|
videos = unannotated[start:end]
|
||||||
|
|
||||||
|
return render_template_string(
|
||||||
|
INDEX_TEMPLATE,
|
||||||
|
videos=videos,
|
||||||
|
page=page,
|
||||||
|
per_page=per_page,
|
||||||
|
total=total,
|
||||||
|
total_pages=(total + per_page - 1) // per_page,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@app.route("/api/review/queue")
|
||||||
|
def api_queue():
|
||||||
|
"""API: List videos in review queue."""
|
||||||
|
page = request.args.get("page", 1, type=int)
|
||||||
|
per_page = 20
|
||||||
|
|
||||||
|
unannotated = [v for v in review_queue if not v.get("annotated")]
|
||||||
|
total = len(unannotated)
|
||||||
|
start = (page - 1) * per_page
|
||||||
|
end = start + per_page
|
||||||
|
|
||||||
|
return jsonify({
|
||||||
|
"videos": unannotated[start:end],
|
||||||
|
"total": total,
|
||||||
|
"page": page,
|
||||||
|
"per_page": per_page,
|
||||||
|
})
|
||||||
|
|
||||||
|
|
||||||
|
@app.route("/api/review/<int:video_id>")
|
||||||
|
def api_video_details(video_id):
|
||||||
|
"""API: Get video details for annotation."""
|
||||||
|
video = next((v for v in review_queue if v["id"] == video_id), None)
|
||||||
|
if not video:
|
||||||
|
return jsonify({"error": "Video not found"}), 404
|
||||||
|
return jsonify(video)
|
||||||
|
|
||||||
|
|
||||||
|
@app.route("/api/review/<int:video_id>/label", methods=["POST"])
|
||||||
|
def api_label(video_id):
|
||||||
|
"""API: Submit annotation for a video."""
|
||||||
|
data = request.get_json()
|
||||||
|
ground_truth = data.get("ground_truth")
|
||||||
|
notes = data.get("notes", "")
|
||||||
|
|
||||||
|
if ground_truth is None:
|
||||||
|
return jsonify({"error": "ground_truth is required"}), 400
|
||||||
|
|
||||||
|
video = next((v for v in review_queue if v["id"] == video_id), None)
|
||||||
|
if not video:
|
||||||
|
return jsonify({"error": "Video not found"}), 404
|
||||||
|
|
||||||
|
video["annotated"] = True
|
||||||
|
video["ground_truth"] = ground_truth
|
||||||
|
video["notes"] = notes
|
||||||
|
video["annotated_at"] = "2026-08-03T10:30:00Z"
|
||||||
|
|
||||||
|
return jsonify({"status": "annotated", "video_id": video_id, "ground_truth": ground_truth})
|
||||||
|
|
||||||
|
|
||||||
|
@app.route("/api/review/stats")
|
||||||
|
def api_stats():
|
||||||
|
"""API: Review queue statistics."""
|
||||||
|
total = len(review_queue)
|
||||||
|
annotated = sum(1 for v in review_queue if v.get("annotated"))
|
||||||
|
unannotated = total - annotated
|
||||||
|
avg_conf = sum(v.get("confidence_score", 0) for v in review_queue) / max(total, 1)
|
||||||
|
|
||||||
|
dist = {"0.0-0.2": 0, "0.2-0.4": 0, "0.4-0.6": 0, "0.6-0.8": 0, "0.8-1.0": 0}
|
||||||
|
for v in review_queue:
|
||||||
|
c = v.get("confidence_score", 0)
|
||||||
|
if c < 0.2:
|
||||||
|
dist["0.0-0.2"] += 1
|
||||||
|
elif c < 0.4:
|
||||||
|
dist["0.2-0.4"] += 1
|
||||||
|
elif c < 0.6:
|
||||||
|
dist["0.4-0.6"] += 1
|
||||||
|
elif c < 0.8:
|
||||||
|
dist["0.6-0.8"] += 1
|
||||||
|
else:
|
||||||
|
dist["0.8-1.0"] += 1
|
||||||
|
|
||||||
|
return jsonify({
|
||||||
|
"total_in_queue": total,
|
||||||
|
"annotated": annotated,
|
||||||
|
"unannotated": unannotated,
|
||||||
|
"avg_confidence": round(avg_conf, 4),
|
||||||
|
"confidence_distribution": dist,
|
||||||
|
})
|
||||||
|
|
||||||
|
|
||||||
|
# ------ HTML Templates ------
|
||||||
|
|
||||||
|
INDEX_TEMPLATE = """
|
||||||
|
<!DOCTYPE html>
|
||||||
|
<html lang="en">
|
||||||
|
<head>
|
||||||
|
<meta charset="UTF-8">
|
||||||
|
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||||
|
<title>VideoDetect - Review Queue</title>
|
||||||
|
<style>
|
||||||
|
* { margin: 0; padding: 0; box-sizing: border-box; }
|
||||||
|
body { font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif; background: #f5f5f5; color: #333; }
|
||||||
|
.header { background: #1a1a2e; color: white; padding: 20px 40px; display: flex; justify-content: space-between; align-items: center; }
|
||||||
|
.header h1 { font-size: 24px; }
|
||||||
|
.header .stats { font-size: 14px; opacity: 0.8; }
|
||||||
|
.container { max-width: 1200px; margin: 40px auto; padding: 0 20px; }
|
||||||
|
.table { width: 100%; background: white; border-radius: 8px; overflow: hidden; box-shadow: 0 2px 8px rgba(0,0,0,0.1); }
|
||||||
|
table { width: 100%; border-collapse: collapse; }
|
||||||
|
th { background: #f8f9fa; padding: 14px 16px; text-align: left; font-weight: 600; font-size: 13px; text-transform: uppercase; color: #666; border-bottom: 2px solid #e9ecef; }
|
||||||
|
td { padding: 14px 16px; border-bottom: 1px solid #e9ecef; font-size: 14px; }
|
||||||
|
tr:hover { background: #f8f9fa; }
|
||||||
|
.badge { display: inline-block; padding: 4px 10px; border-radius: 12px; font-size: 12px; font-weight: 600; }
|
||||||
|
.badge-match { background: #d4edda; color: #155724; }
|
||||||
|
.badge-review { background: #fff3cd; color: #856404; }
|
||||||
|
.badge-skip { background: #f8d7da; color: #721c24; }
|
||||||
|
.btn { display: inline-block; padding: 8px 16px; background: #007bff; color: white; text-decoration: none; border-radius: 6px; font-size: 14px; border: none; cursor: pointer; }
|
||||||
|
.btn:hover { background: #0056b3; }
|
||||||
|
.pagination { display: flex; justify-content: center; gap: 8px; margin-top: 24px; }
|
||||||
|
.pagination a, .pagination span { padding: 8px 14px; border-radius: 6px; text-decoration: none; font-size: 14px; }
|
||||||
|
.pagination a { background: white; color: #007bff; border: 1px solid #dee2e6; }
|
||||||
|
.pagination a:hover { background: #e9ecef; }
|
||||||
|
.pagination .current { background: #007bff; color: white; }
|
||||||
|
.empty { text-align: center; padding: 60px 20px; color: #999; }
|
||||||
|
.confidence { font-weight: 600; }
|
||||||
|
.confidence-high { color: #28a745; }
|
||||||
|
.confidence-mid { color: #ffc107; }
|
||||||
|
.confidence-low { color: #dc3545; }
|
||||||
|
</style>
|
||||||
|
</head>
|
||||||
|
<body>
|
||||||
|
<div class="header">
|
||||||
|
<h1>VideoDetect</h1>
|
||||||
|
<div class="stats">
|
||||||
|
Review Queue: {{ total }} videos | Page {{ page }} of {{ total_pages }}
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
<div class="container">
|
||||||
|
{% if videos %}
|
||||||
|
<div class="table">
|
||||||
|
<table>
|
||||||
|
<thead>
|
||||||
|
<tr>
|
||||||
|
<th>Video ID</th>
|
||||||
|
<th>File Path</th>
|
||||||
|
<th>Confidence</th>
|
||||||
|
<th>Routing</th>
|
||||||
|
<th>Model</th>
|
||||||
|
<th>Frames</th>
|
||||||
|
<th>Added</th>
|
||||||
|
<th>Action</th>
|
||||||
|
</tr>
|
||||||
|
</thead>
|
||||||
|
<tbody>
|
||||||
|
{% for video in videos %}
|
||||||
|
<tr>
|
||||||
|
<td>{{ video.id }}</td>
|
||||||
|
<td style="max-width: 300px; overflow: hidden; text-overflow: ellipsis; white-space: nowrap;">{{ video.file_path }}</td>
|
||||||
|
<td class="confidence {% if video.confidence_score >= 0.75 %}confidence-high{% elif video.confidence_score >= 0.45 %}confidence-mid{% else %}confidence-low{% endif %}">
|
||||||
|
{{ "%.2f"|format(video.confidence_score) }}
|
||||||
|
</td>
|
||||||
|
<td><span class="badge badge-{{ video.routing_decision|lower }}">{{ video.routing_decision }}</span></td>
|
||||||
|
<td>{{ video.model_version }}</td>
|
||||||
|
<td>{{ video.frame_count }}</td>
|
||||||
|
<td>{{ video.created_at[:10] if video.created_at else 'N/A' }}</td>
|
||||||
|
<td><a href="/review/{{ video.id }}" class="btn">Annotate</a></td>
|
||||||
|
</tr>
|
||||||
|
{% endfor %}
|
||||||
|
</tbody>
|
||||||
|
</table>
|
||||||
|
</div>
|
||||||
|
<div class="pagination">
|
||||||
|
{% if page > 1 %}
|
||||||
|
<a href="?page={{ page - 1 }}">« Prev</a>
|
||||||
|
{% endif %}
|
||||||
|
<span class="current">{{ page }}</span>
|
||||||
|
{% if page < total_pages %}
|
||||||
|
<a href="?page={{ page + 1 }}">Next »</a>
|
||||||
|
{% endif %}
|
||||||
|
</div>
|
||||||
|
{% else %}
|
||||||
|
<div class="empty">
|
||||||
|
<h2>No videos in review queue</h2>
|
||||||
|
<p>All videos have been processed and classified.</p>
|
||||||
|
</div>
|
||||||
|
{% endif %}
|
||||||
|
</div>
|
||||||
|
</body>
|
||||||
|
</html>
|
||||||
|
"""
|
||||||
|
|
||||||
|
REVIEW_TEMPLATE = """
|
||||||
|
<!DOCTYPE html>
|
||||||
|
<html lang="en">
|
||||||
|
<head>
|
||||||
|
<meta charset="UTF-8">
|
||||||
|
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||||
|
<title>VideoDetect - Review #{{ video.id }}</title>
|
||||||
|
<style>
|
||||||
|
* { margin: 0; padding: 0; box-sizing: border-box; }
|
||||||
|
body { font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif; background: #f5f5f5; color: #333; }
|
||||||
|
.header { background: #1a1a2e; color: white; padding: 20px 40px; display: flex; justify-content: space-between; align-items: center; }
|
||||||
|
.header h1 { font-size: 24px; }
|
||||||
|
.header a { color: #adb5bd; text-decoration: none; }
|
||||||
|
.header a:hover { color: white; }
|
||||||
|
.container { max-width: 1000px; margin: 40px auto; padding: 0 20px; }
|
||||||
|
.card { background: white; border-radius: 8px; padding: 24px; margin-bottom: 20px; box-shadow: 0 2px 8px rgba(0,0,0,0.1); }
|
||||||
|
.card h2 { font-size: 18px; margin-bottom: 16px; color: #1a1a2e; }
|
||||||
|
.info-grid { display: grid; grid-template-columns: repeat(3, 1fr); gap: 16px; }
|
||||||
|
.info-item { }
|
||||||
|
.info-label { font-size: 12px; text-transform: uppercase; color: #666; margin-bottom: 4px; }
|
||||||
|
.info-value { font-size: 16px; font-weight: 600; }
|
||||||
|
.video-player { width: 100%; aspect-ratio: 16/9; background: #000; border-radius: 8px; display: flex; align-items: center; justify-content: center; color: white; }
|
||||||
|
.frames-grid { display: grid; grid-template-columns: repeat(5, 1fr); gap: 12px; }
|
||||||
|
.frame-thumb { aspect-ratio: 1; background: #e9ecef; border-radius: 6px; display: flex; flex-direction: column; align-items: center; justify-content: center; font-size: 12px; color: #666; }
|
||||||
|
.frame-thumb .confidence { font-size: 14px; font-weight: 600; color: #007bff; margin-top: 4px; }
|
||||||
|
.label-toggle { display: flex; gap: 12px; margin: 20px 0; }
|
||||||
|
.label-btn { flex: 1; padding: 16px; border: 2px solid #dee2e6; border-radius: 8px; background: white; cursor: pointer; font-size: 16px; font-weight: 600; text-align: center; transition: all 0.2s; }
|
||||||
|
.label-btn:hover { border-color: #007bff; }
|
||||||
|
.label-btn.selected-true { border-color: #28a745; background: #d4edda; color: #155724; }
|
||||||
|
.label-btn.selected-false { border-color: #dc3545; background: #f8d7da; color: #721c24; }
|
||||||
|
.notes { width: 100%; padding: 12px; border: 1px solid #dee2e6; border-radius: 6px; font-size: 14px; resize: vertical; min-height: 80px; }
|
||||||
|
.submit-btn { width: 100%; padding: 14px; background: #007bff; color: white; border: none; border-radius: 8px; font-size: 16px; font-weight: 600; cursor: pointer; margin-top: 16px; }
|
||||||
|
.submit-btn:hover { background: #0056b3; }
|
||||||
|
.submit-btn:disabled { background: #ccc; cursor: not-allowed; }
|
||||||
|
</style>
|
||||||
|
</head>
|
||||||
|
<body>
|
||||||
|
<div class="header">
|
||||||
|
<h1>VideoDetect</h1>
|
||||||
|
<a href="/">« Back to Queue</a>
|
||||||
|
</div>
|
||||||
|
<div class="container">
|
||||||
|
<div class="card">
|
||||||
|
<h2>Video Information</h2>
|
||||||
|
<div class="info-grid">
|
||||||
|
<div class="info-item">
|
||||||
|
<div class="info-label">Video ID</div>
|
||||||
|
<div class="info-value">{{ video.id }}</div>
|
||||||
|
</div>
|
||||||
|
<div class="info-item">
|
||||||
|
<div class="info-label">Confidence</div>
|
||||||
|
<div class="info-value">{{ "%.2f"|format(video.confidence_score) }}</div>
|
||||||
|
</div>
|
||||||
|
<div class="info-item">
|
||||||
|
<div class="info-label">Routing</div>
|
||||||
|
<div class="info-value">{{ video.routing_decision }}</div>
|
||||||
|
</div>
|
||||||
|
<div class="info-item">
|
||||||
|
<div class="info-label">Model</div>
|
||||||
|
<div class="info-value">{{ video.model_version }}</div>
|
||||||
|
</div>
|
||||||
|
<div class="info-item">
|
||||||
|
<div class="info-label">Frames</div>
|
||||||
|
<div class="info-value">{{ video.frame_count }}</div>
|
||||||
|
</div>
|
||||||
|
<div class="info-item">
|
||||||
|
<div class="info-label">Duration</div>
|
||||||
|
<div class="info-value">{{ "%.1f"|format(video.video_duration) if video.video_duration else 'N/A' }}s</div>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div class="card">
|
||||||
|
<h2>Video</h2>
|
||||||
|
<div class="video-player">
|
||||||
|
<video controls style="width: 100%; height: 100%;">
|
||||||
|
<source src="{{ video.file_path }}" type="video/mp4">
|
||||||
|
Video playback not supported in this browser.
|
||||||
|
</video>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div class="card">
|
||||||
|
<h2>Contributing Frames (Top {{ video.top_k_frames|default(5) }})</h2>
|
||||||
|
<div class="frames-grid">
|
||||||
|
{% for frame in video.contributing_frames %}
|
||||||
|
<div class="frame-thumb">
|
||||||
|
<span>⏱ {{ "%.0f"|format(frame.timestamp) }}s</span>
|
||||||
|
<span class="confidence">{{ "%.2f"|format(frame.confidence) }}</span>
|
||||||
|
</div>
|
||||||
|
{% endfor %}
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div class="card">
|
||||||
|
<h2>Annotate</h2>
|
||||||
|
<p style="margin-bottom: 12px; color: #666;">Does this video contain the target demographic?</p>
|
||||||
|
<div class="label-toggle">
|
||||||
|
<button class="label-btn {% if video.ground_truth == True %}selected-true{% endif %}" onclick="selectLabel(true)">✓ Yes (MATCH)</button>
|
||||||
|
<button class="label-btn {% if video.ground_truth == False %}selected-false{% endif %}" onclick="selectLabel(false)">✗ No (NO_MATCH)</button>
|
||||||
|
</div>
|
||||||
|
<textarea class="notes" id="notes" placeholder="Optional notes...">{{ video.notes or '' }}</textarea>
|
||||||
|
<button class="submit-btn" id="submitBtn" onclick="submitLabel()" disabled>Submit Label</button>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<script>
|
||||||
|
let selectedLabel = null;
|
||||||
|
|
||||||
|
function selectLabel(value) {
|
||||||
|
selectedLabel = value;
|
||||||
|
document.querySelectorAll('.label-btn').forEach(btn => {
|
||||||
|
btn.classList.remove('selected-true', 'selected-false');
|
||||||
|
});
|
||||||
|
event.target.classList.add(value ? 'selected-true' : 'selected-false');
|
||||||
|
document.getElementById('submitBtn').disabled = false;
|
||||||
|
}
|
||||||
|
|
||||||
|
async function submitLabel() {
|
||||||
|
const videoId = {{ video.id }};
|
||||||
|
const notes = document.getElementById('notes').value;
|
||||||
|
|
||||||
|
const response = await fetch(`/api/review/${videoId}/label`, {
|
||||||
|
method: 'POST',
|
||||||
|
headers: {'Content-Type': 'application/json'},
|
||||||
|
body: JSON.stringify({ground_truth: selectedLabel, notes: notes})
|
||||||
|
});
|
||||||
|
|
||||||
|
if (response.ok) {
|
||||||
|
alert('Label saved successfully!');
|
||||||
|
window.location.href = '/';
|
||||||
|
} else {
|
||||||
|
alert('Error saving label.');
|
||||||
|
}
|
||||||
|
}
|
||||||
|
</script>
|
||||||
|
</body>
|
||||||
|
</html>
|
||||||
|
"""
|
||||||
|
|
||||||
|
|
||||||
|
# ------ Entry Point ------
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
host = os.environ.get("FLASK_HOST", "0.0.0.0")
|
||||||
|
port = int(os.environ.get("FLASK_PORT", "5000"))
|
||||||
|
app.run(host=host, port=port, debug=False)
|
||||||
@@ -0,0 +1,4 @@
|
|||||||
|
Flask==3.0.0
|
||||||
|
gunicorn==21.2.0
|
||||||
|
PyMySQL==1.1.0
|
||||||
|
Jinja2==3.1.2
|
||||||
@@ -0,0 +1,67 @@
|
|||||||
|
# Base image: CUDA 11.8 runtime on Ubuntu 22.04
|
||||||
|
FROM nvidia/cuda:11.8.0-runtime-ubuntu22.04
|
||||||
|
|
||||||
|
# Avoid interactive prompts during build
|
||||||
|
ENV DEBIAN_FRONTEND=noninteractive
|
||||||
|
ENV TZ=Etc/UTC
|
||||||
|
|
||||||
|
# System dependencies
|
||||||
|
RUN apt-get update && apt-get install -y --no-install-recommends \
|
||||||
|
# Python
|
||||||
|
python3.10 \
|
||||||
|
python3.10-dev \
|
||||||
|
python3.10-venv \
|
||||||
|
python3-pip \
|
||||||
|
# FFmpeg
|
||||||
|
ffmpeg \
|
||||||
|
# OpenCV dependencies
|
||||||
|
libgl1-mesa-glx \
|
||||||
|
libglib2.0-0 \
|
||||||
|
libsm6 \
|
||||||
|
libxext6 \
|
||||||
|
libxrender-dev \
|
||||||
|
# Utilities
|
||||||
|
curl \
|
||||||
|
wget \
|
||||||
|
git \
|
||||||
|
ca-certificates \
|
||||||
|
&& rm -rf /var/lib/apt/lists/*
|
||||||
|
|
||||||
|
# Create non-root user
|
||||||
|
RUN groupadd -g 1000 appuser && \
|
||||||
|
useradd -m -u 1000 -g appuser -s /bin/bash appuser
|
||||||
|
|
||||||
|
# Set working directory
|
||||||
|
WORKDIR /app
|
||||||
|
|
||||||
|
# Create necessary directories
|
||||||
|
RUN mkdir -p /scratch /models /data/training /data/output /logs
|
||||||
|
|
||||||
|
# Copy requirements first for better caching
|
||||||
|
COPY worker/requirements.txt /app/requirements.txt
|
||||||
|
|
||||||
|
# Create and activate virtual environment
|
||||||
|
RUN python3.10 -m venv /opt/venv
|
||||||
|
ENV PATH="/opt/venv/bin:$PATH"
|
||||||
|
|
||||||
|
# Install Python dependencies
|
||||||
|
RUN pip install --no-cache-dir -r requirements.txt
|
||||||
|
|
||||||
|
# Copy application code
|
||||||
|
COPY src/ /app/src/
|
||||||
|
COPY config.yaml /app/config.yaml
|
||||||
|
|
||||||
|
# Set environment variables
|
||||||
|
ENV PYTHONUNBUFFERED=1
|
||||||
|
ENV PATH="/opt/venv/bin:$PATH"
|
||||||
|
ENV LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH
|
||||||
|
|
||||||
|
# Health check
|
||||||
|
HEALTHCHECK --interval=30s --timeout=10s --retries=3 \
|
||||||
|
CMD python3 -c "import torch; assert torch.cuda.is_available()" || exit 1
|
||||||
|
|
||||||
|
# Switch to non-root user
|
||||||
|
USER appuser
|
||||||
|
|
||||||
|
# Default command
|
||||||
|
CMD ["python3", "-m", "src.main"]
|
||||||
@@ -0,0 +1,40 @@
|
|||||||
|
# Core ML Frameworks
|
||||||
|
torch==2.1.0
|
||||||
|
--extra-index-url https://download.pytorch.org/whl/cu118
|
||||||
|
|
||||||
|
# TensorRT (compatible with CUDA 11.8)
|
||||||
|
# Note: Install from NVIDIA repo or wheel for your specific platform
|
||||||
|
# tensorrt==8.6.1 # Uncomment after verifying compatibility
|
||||||
|
|
||||||
|
# ONNX Runtime with GPU support
|
||||||
|
onnxruntime-gpu==1.16.3
|
||||||
|
|
||||||
|
# Computer Vision
|
||||||
|
opencv-python-headless==4.8.1.78
|
||||||
|
Pillow==10.1.0
|
||||||
|
|
||||||
|
# Database
|
||||||
|
PyMySQL==1.1.0
|
||||||
|
DBUtils==3.1.0
|
||||||
|
|
||||||
|
# Data Processing
|
||||||
|
numpy==1.24.4
|
||||||
|
pandas==2.1.4
|
||||||
|
pyarrow==14.0.0
|
||||||
|
|
||||||
|
# Configuration
|
||||||
|
pyyaml==6.0.1
|
||||||
|
|
||||||
|
# Logging
|
||||||
|
python-json-logger==2.0.7
|
||||||
|
|
||||||
|
# Monitoring
|
||||||
|
prometheus-client==0.19.0
|
||||||
|
|
||||||
|
# Utilities
|
||||||
|
tqdm==4.66.1
|
||||||
|
filelock==3.13.1
|
||||||
|
|
||||||
|
# Testing (dev dependencies - install with [dev] extra)
|
||||||
|
# pytest==7.4.3
|
||||||
|
# pytest-cov==4.1.0
|
||||||
Reference in New Issue
Block a user