VideoDetect/STORY-02.md

STORY-02: Core Ingestion & Codec Handling

Epic

E5: Data Management — As a system, I can scan directories and sync file state to MariaDB. E1: Core Pipeline — As a system, I can detect video resolution and codec.

Related Requirements

ID	Requirement
FR-09	Directory Scanning & Sync: Process to scan input directories, detect new/removed files, and sync state to MariaDB
FR-10	Codec & Resolution Detection & Handling: Detect video properties; handle unsupported codecs by flagging files as UNSCANNABLE
NFR-07	Data Volume Handling: Efficient indexing for ~30TB dataset
NFR-08	Codec Agnosticism: Handle H.264, H.265, VP8, VP9, AV1, MJPEG, etc.
NFR-05	Fault Tolerance: Auto-retry on transient failures; skip & log on fatal errors

Description

Implement the directory scanner, video probing, and robust error handling for codecs and resolutions. This story enables the system to discover new files in the 30TB corpus, extract their metadata, validate codec support, and maintain accurate database state.

Scope

In Scope

• Background directory scanner service that walks /data/input
• File detection and deduplication via SHA-256 hashing
• Video probing via ffprobe (codec, resolution, duration)
• Codec whitelist/blacklist validation
• UNSCANNABLE status for unsupported or corrupt files
• Priority queue / DB-based locking for job assignment
• Atomic state transitions (PENDING → PROCESSING)
• Efficient handling of 30TB directory structure (incremental scanning)

Out of Scope

• Frame extraction (covered in STORY-04)
• Face detection and classification (covered in STORY-04)
• Confidence scoring and routing (covered in STORY-05)
• Review UI (covered in STORY-06)
• Active learning pipeline (covered in STORY-07)
• Monitoring dashboards (covered in STORY-08)

Deliverables

2.1 Directory Scanner Service

File: src/scanner.py

Core components:

• Walker: Recursive directory walker with configurable depth and path filters
• Incremental Sync: Compare current filesystem state against DB last_scan_time — only process new/modified files
• File Detection: Identify files not in DB or with status NEW/PENDING
• Hash Computation: SHA-256 of first 1MB (or full file if < 1MB) for deduplication
• Scan Scheduler: Configurable interval (default: every 60 seconds) via cron-like scheduler
• Concurrency: Multi-threaded walker with configurable worker count (default: 8 threads)
• Error Handling: Per-file error isolation — scanner continues on individual file failures

Key behaviors:

• On scan start: query DB for files with last_scan_time < now() or status IN ('NEW', 'PENDING')
• For each file: compute hash → check DB for duplicate → if new, insert with status PENDING
• For removed files: mark as REMOVED in DB (optional, configurable)
• Update last_scan_time on videos table after successful scan

2.2 Video Probing Module

File: src/prober.py

Features:

• ffprobe wrapper: Execute ffprobe with optimized arguments for speed

  ffprobe -v error -select_streams v:0 -show_entries stream=codec_name,width,height,duration,r_frame_rate -show_entries format=duration -of json <file>
  

• Metadata extraction: Parse JSON output for:
  • codec_name: Video codec identifier
  • width, height: Resolution
  • duration: Video duration in seconds
  • r_frame_rate: Frame rate (for sampling calculations)
• Error handling: Catch ffprobe failures (corrupt files, unsupported formats)
• Timeout: ffprobe execution limited to 10 seconds per file

2.3 Codec Validation

File: src/codec_validator.py

Codec whitelist (supported):

• H.264 (avc1)
• H.265 (hevc)
• VP8 (vp8)
• VP9 (vp9)
• AV1 (av01)
• MJPEG (mjpeg)
• MPEG-4 (mp4v)

Codec blacklist (unsupported):

• Theora
• DivX/Xvid (legacy)
• ProRes (requires special handling)
• Any codec not in whitelist

Behavior:

• If codec not in whitelist → set status UNSCANNABLE, log reason
• If ffprobe fails → set status UNSCANNABLE, log error code
• If file is not a valid video → set status UNSCANNABLE, log reason
• UNSCANNABLE files are excluded from processing pipeline

2.4 Batch Orchestration Skeleton

File: src/orchestrator.py

Components:

• Job Queue: Priority queue based on file modification time (newest first)
• DB-based Locking: Atomic state transition using UPDATE ... WHERE status = 'PENDING'
• Worker Pool: Configurable number of worker processes (default: 2, one per GPU)
• State Machine:

  NEW → PENDING → PROCESSING → COMPLETED
                         → UNSCANNABLE
                         → ERROR
  

• Concurrency Control: Max concurrent processing per GPU (default: 1 video at a time per worker)
• Idempotency: Re-processing a file does not duplicate DB entries or outputs

2.5 Configuration Updates

File: config.yaml (updates to STORY-01)

New fields:

scanner:
  scan_interval_seconds: 60
  walker_threads: 8
  ffprobe_timeout_seconds: 10
  hash_algorithm: sha256
  hash_chunk_size_mb: 1

codec:
  whitelist: [avc1, hevc, vp8, vp9, av01, mjpeg, mp4v]
  default_status_on_error: UNSCANNABLE

queue:
  priority: modification_time  # newest first
  max_concurrent_per_gpu: 1
  lock_timeout_seconds: 300

Acceptance Criteria

Functional

• New files in /data/input appear in DB with correct metadata within 60 seconds of placement
• File hash deduplication prevents re-processing identical files
• ffprobe correctly extracts codec, resolution, and duration for all supported codecs
• Unsupported codec files are marked UNSCANNABLE without crashing the scanner
• Corrupt/unreadable files are marked UNSCANNABLE with appropriate error logged
• Atomic state transitions: PENDING → PROCESSING succeeds only once (no duplicate processing)
• Removed files are detected and marked REMOVED (if configured)
• Scanner handles 30TB directory structure without OOM (memory < 500MB during scan)
• Re-running scanner is idempotent — no duplicate entries or states

Non-Functional

• Scanner completes full 30TB directory walk in < 4 hours (incremental: < 10 minutes for typical day)
• ffprobe timeout (10s) is enforced — does not hang on corrupt files
• Scanner memory usage stays < 500MB regardless of directory depth
• Hash computation for 100MB file completes in < 5 seconds on NVMe
• Worker pool respects GPU count (2 workers for 2 GPUs)

Technical Constraints

• SHA-256 hash is deterministic and reproducible
• Codec detection matches ffprobe output exactly
• Resolution stored as integers (width, height)
• Duration stored as float (seconds)
• All scanner errors are logged with video_id, error_code, and file_path
• Scanner does not modify video files (read-only operation)

Dependencies

• Prerequisites: STORY-01 (Foundation & Infrastructure) — DB schema, connection layer, config
• Depends on: None (runs in parallel with STORY-03 if needed)
• Enables: STORY-04 (Inference Pipeline), STORY-05 (Results & Export)

Risks & Mitigations

Risk	Mitigation
30TB directory walk is extremely slow	Incremental scanning using last_scan_time; only walk new directories
ffprobe hangs on corrupt files	Enforce 10-second timeout via subprocess timeout
Hash computation on slow NAS is bottleneck	Hash only first 1MB; full hash optional for verification
Concurrent scanner + worker conflicts	DB-based locking; scanner only writes PENDING, workers read PENDING
Network mount latency	Cache directory listings; batch DB operations

Estimated Effort

• Sprint: 3-4
• Story Points: 21
• Dependencies: STORY-01