After updates
This commit is contained in:
Ryan Shpeherd
@@ -82,6 +82,213 @@ open http://localhost:3000 (admin / your_grafana_password)
|
||||
open http://localhost:9090
|
||||
```
|
||||
|
||||
## System Operation
|
||||
|
||||
### How Processes Start
|
||||
|
||||
**Service Initialization:**
|
||||
|
||||
1. **MariaDB** starts first with health check
|
||||
2. **Worker** initializes via `src/main.py`:
|
||||
- Loads `config.yaml`
|
||||
- Sets up JSON logging with rotation
|
||||
- Connects to MariaDB (connection pooling)
|
||||
- Initializes database schema
|
||||
- Verifies GPU availability (CUDA/PyTorch)
|
||||
- Starts the `DirectoryScanner` in a background thread (scans permanent storage in place)
|
||||
- Creates `WorkerPool` with 1 worker thread
|
||||
- Enters job processing loop
|
||||
3. **UI** starts Flask review interface via Gunicorn (2 workers)
|
||||
4. **Monitoring** starts Prometheus and Grafana independently
|
||||
|
||||
### Processing Pipeline
|
||||
|
||||
The worker follows this flow for each video:
|
||||
|
||||
```
|
||||
Pending → Lock → Probe → Sample → Detect → Classify → Aggregate → Route → Persist → Export → Cleanup → Completed
|
||||
```
|
||||
|
||||
**Detailed Steps:**
|
||||
|
||||
1. **Job Queue** - Atomically lock `PENDING` jobs via `UPDATE status = 'PROCESSING'`
|
||||
- Priority: newest files first (`last_scan_time DESC`)
|
||||
- Max concurrent: 1 per GPU
|
||||
|
||||
2. **Probe Video** - Extract metadata via FFprobe
|
||||
- Duration, codec, resolution
|
||||
- Validate against codec whitelist (H.264, H.265, VP8/9, AV1)
|
||||
- Mark `UNSCANNABLE` if invalid
|
||||
|
||||
3. **Sample Frames** - Extract frames at configured interval (default: 30s)
|
||||
- Save as JPEG to `/scratch/{video_id}/frames/`
|
||||
- Quality: 2 (lower=better)
|
||||
|
||||
4. **Detect Faces** - YOLOv8n TensorRT inference (FP32)
|
||||
- Batch size auto-tuned by GPU memory monitor
|
||||
- NMS filtering (IoU: 0.45, confidence: 0.25)
|
||||
- Cap: 10 faces/frame, 100 faces/video
|
||||
|
||||
5. **Extract Crops** - Resize detected faces to 224×224
|
||||
- Save to `/scratch/{video_id}/crops/`
|
||||
|
||||
6. **Classify Crops** - MobileNetV3-Small TensorRT inference
|
||||
- Temperature-scaled softmax (T=1.0)
|
||||
- Returns confidence per crop
|
||||
|
||||
7. **Aggregate Confidence** - Combine crop confidences into video-level score
|
||||
- Strategy: `max` (most conservative)
|
||||
- Alternatives: `weighted_mean`, `top_k_mean`
|
||||
|
||||
8. **Route Decision** - Threshold-based routing:
|
||||
- `C ≥ 0.75` → **MATCH**
|
||||
- `0.45 ≤ C < 0.75` → **REVIEW** (human annotation)
|
||||
- `C < 0.45` → **SKIP**
|
||||
- No faces → **SKIP**
|
||||
|
||||
9. **Persist Results** - Atomic transaction:
|
||||
- Update `videos` table (confidence, routing, status)
|
||||
- Insert `processing_logs` row (audit trail)
|
||||
- State guard: only update if `status='PROCESSING'`
|
||||
|
||||
10. **Export** - Buffer and batch export (default: 100 videos)
|
||||
- Format: Parquet with Snappy compression
|
||||
- Path: `/data/output/{model_version}/`
|
||||
- Fallback: JSONL if Parquet fails
|
||||
|
||||
11. **Cleanup** - Delete `/scratch/{video_id}/` directory
|
||||
- Only after successful persistence
|
||||
- Prevents orphaned scratch files
|
||||
|
||||
**Directory Scanner Service (runs alongside the worker):**
|
||||
- Scans the permanent storage location **in place** (no staging/copy step)
|
||||
- Walks `/data/input` every **2 hours** by default (configurable via `scanner.scan_interval_seconds`)
|
||||
- Detects new, modified, and removed video files by comparing against the DB
|
||||
- Filters to video files by extension
|
||||
- Computes SHA256 hash, probes metadata, validates codec
|
||||
- Queues any video that has not been scanned yet as `PENDING` for the worker pool
|
||||
- **Single-instance guard:** an in-process lock plus a DB lock (with a lease) ensure only one scan runs at a time — a long-running scan never overlaps another, even across multiple worker replicas. The lock lease is refreshed via heartbeats during the scan and is taken over automatically if a scanner crashes.
|
||||
|
||||
### Configuration Reference
|
||||
|
||||
All configuration is in `config.yaml`. Environment variable override format: `VD_<SECTION>_<KEY>` (e.g., `VD_SAMPLING_INTERVAL_SECONDS=60`).
|
||||
|
||||
#### Key Configuration Sections
|
||||
|
||||
**Sampling & Thresholds:**
|
||||
```yaml
|
||||
sampling:
|
||||
interval_seconds: 30 # Frame extraction frequency
|
||||
quality: 2 # JPEG quality (1-31, lower=better)
|
||||
format: jpeg
|
||||
|
||||
thresholds:
|
||||
T_high: 0.75 # MATCH threshold
|
||||
T_low: 0.45 # REVIEW threshold
|
||||
```
|
||||
|
||||
**GPU & Batching:**
|
||||
```yaml
|
||||
gpu:
|
||||
max_memory_gb: 18 # Target VRAM usage
|
||||
batch_size: auto # Auto-tune based on available VRAM
|
||||
|
||||
batching:
|
||||
max_batch_size: 16 # Maximum batch size
|
||||
vram_target_gb: 16 # Target VRAM for batch tuning
|
||||
vram_reduce_threshold_gb: 16 # Reduce batch if above
|
||||
vram_increase_threshold_gb: 10 # Increase batch if below
|
||||
```
|
||||
|
||||
**Storage Paths:**
|
||||
```yaml
|
||||
storage:
|
||||
scratch_path: /scratch # Temporary processing (tmpfs)
|
||||
input_path: /data/input # Source videos (NFS)
|
||||
output_path: /data/output # Results (local/NAS)
|
||||
models_path: /models # TensorRT models
|
||||
training_path: /data/training # Training data
|
||||
```
|
||||
|
||||
**Database:**
|
||||
```yaml
|
||||
database:
|
||||
host: mariadb
|
||||
port: 3306
|
||||
name: videodetect
|
||||
user: videodetect
|
||||
password: videodetect123
|
||||
pool_size: 20 # Connection pool size
|
||||
pool_min: 5
|
||||
pool_recycle: 3600 # Recycle connections after 1h
|
||||
```
|
||||
|
||||
**Face Detection:**
|
||||
```yaml
|
||||
face_detection:
|
||||
model: yolo8n
|
||||
model_path: /models/face_detector/face_detector.trt
|
||||
input_size: 640
|
||||
confidence_threshold: 0.25
|
||||
iou_threshold: 0.45
|
||||
max_faces_per_frame: 10
|
||||
max_faces_per_video: 100
|
||||
```
|
||||
|
||||
**Classification & Aggregation:**
|
||||
```yaml
|
||||
classifier:
|
||||
model: mobilenetv3-small
|
||||
model_path: /models/classifier/classifier.trt
|
||||
input_size: 224
|
||||
temperature: 1.0 # Calibration temperature
|
||||
|
||||
aggregation:
|
||||
strategy: max # max, weighted_mean, top_k_mean
|
||||
alpha: 1.0 # weighted_mean weight for mean
|
||||
beta: 0.1 # weighted_mean weight for variance
|
||||
top_k: 3 # top_k_mean: average top 3 scores
|
||||
```
|
||||
|
||||
**Export:**
|
||||
```yaml
|
||||
export:
|
||||
format: parquet # parquet, jsonl, or both
|
||||
compression: snappy
|
||||
batch_size: 100 # Export after N videos
|
||||
include_frame_confidences: true
|
||||
```
|
||||
|
||||
**Review UI:**
|
||||
```yaml
|
||||
review_ui:
|
||||
host: "0.0.0.0"
|
||||
port: 5000
|
||||
per_page: 20 # Pagination
|
||||
top_k_frames: 5 # Show top-k contributing frames
|
||||
auth_enabled: false # No auth per TC-06
|
||||
ssl_enabled: false # Internal LAN only
|
||||
```
|
||||
|
||||
#### Volume Mounts
|
||||
|
||||
From `docker-compose.yml`:
|
||||
|
||||
- **Input**: NFS mount → `/data/input` (read-only)
|
||||
- **Output**: `./output` → `/data/output`
|
||||
- **Models**: `./models` → `/models`
|
||||
- **Training**: `./training` → `/data/training`
|
||||
- **Scratch**: 100GB tmpfs at `/scratch` (RAM disk)
|
||||
|
||||
### Key Design Principles
|
||||
|
||||
- **Atomic state transitions** - Database locks prevent race conditions
|
||||
- **Crash recovery** - `PROCESSING` jobs automatically requeued on restart
|
||||
- **Idempotent** - Re-running same video produces same result
|
||||
- **Stateless** - Scratch cleanup after each job
|
||||
- **Fail-safe** - 3 retry attempts before marking `ERROR`
|
||||
- **No auth/SSL** - Internal LAN deployment per TC-06
|
||||
|
||||
## Project Structure
|
||||
|
||||
```
|
||||
|
||||
Reference in New Issue
Block a user