After story 1

STORY-02.md

@@ -0,0 +1,182 @@
+# STORY-02: Core Ingestion & Codec Handling
+
+## Epic
+**E5: Data Management** — As a system, I can scan directories and sync file state to MariaDB.
+**E1: Core Pipeline** — As a system, I can detect video resolution and codec.
+
+## Related Requirements
+| ID | Requirement |
+|----|---------|
+| FR-09 | Directory Scanning & Sync: Process to scan input directories, detect new/removed files, and sync state to MariaDB |
+| FR-10 | Codec & Resolution Detection & Handling: Detect video properties; handle unsupported codecs by flagging files as UNSCANNABLE |
+| NFR-07 | Data Volume Handling: Efficient indexing for ~30TB dataset |
+| NFR-08 | Codec Agnosticism: Handle H.264, H.265, VP8, VP9, AV1, MJPEG, etc. |
+| NFR-05 | Fault Tolerance: Auto-retry on transient failures; skip & log on fatal errors |
+
+## Description
+Implement the directory scanner, video probing, and robust error handling for codecs and resolutions. This story enables the system to discover new files in the 30TB corpus, extract their metadata, validate codec support, and maintain accurate database state.
+
+## Scope
+
+### In Scope
+- Background directory scanner service that walks `/data/input`
+- File detection and deduplication via SHA-256 hashing
+- Video probing via ffprobe (codec, resolution, duration)
+- Codec whitelist/blacklist validation
+- UNSCANNABLE status for unsupported or corrupt files
+- Priority queue / DB-based locking for job assignment
+- Atomic state transitions (PENDING → PROCESSING)
+- Efficient handling of 30TB directory structure (incremental scanning)
+
+### Out of Scope
+- Frame extraction (covered in STORY-04)
+- Face detection and classification (covered in STORY-04)
+- Confidence scoring and routing (covered in STORY-05)
+- Review UI (covered in STORY-06)
+- Active learning pipeline (covered in STORY-07)
+- Monitoring dashboards (covered in STORY-08)
+
+## Deliverables
+
+### 2.1 Directory Scanner Service
+**File:** `src/scanner.py`
+
+Core components:
+- **Walker:** Recursive directory walker with configurable depth and path filters
+- **Incremental Sync:** Compare current filesystem state against DB `last_scan_time` — only process new/modified files
+- **File Detection:** Identify files not in DB or with status NEW/PENDING
+- **Hash Computation:** SHA-256 of first 1MB (or full file if < 1MB) for deduplication
+- **Scan Scheduler:** Configurable interval (default: every 60 seconds) via cron-like scheduler
+- **Concurrency:** Multi-threaded walker with configurable worker count (default: 8 threads)
+- **Error Handling:** Per-file error isolation — scanner continues on individual file failures
+
+Key behaviors:
+- On scan start: query DB for files with `last_scan_time < now()` or `status IN ('NEW', 'PENDING')`
+- For each file: compute hash → check DB for duplicate → if new, insert with status PENDING
+- For removed files: mark as REMOVED in DB (optional, configurable)
+- Update `last_scan_time` on `videos` table after successful scan
+
+### 2.2 Video Probing Module
+**File:** `src/prober.py`
+
+Features:
+- **ffprobe wrapper:** Execute ffprobe with optimized arguments for speed
+  ```bash
+  ffprobe -v error -select_streams v:0 -show_entries stream=codec_name,width,height,duration,r_frame_rate -show_entries format=duration -of json <file>
+  ```
+- **Metadata extraction:** Parse JSON output for:
+  - `codec_name`: Video codec identifier
+  - `width`, `height`: Resolution
+  - `duration`: Video duration in seconds
+  - `r_frame_rate`: Frame rate (for sampling calculations)
+- **Error handling:** Catch ffprobe failures (corrupt files, unsupported formats)
+- **Timeout:** ffprobe execution limited to 10 seconds per file
+
+### 2.3 Codec Validation
+**File:** `src/codec_validator.py`
+
+Codec whitelist (supported):
+- H.264 (avc1)
+- H.265 (hevc)
+- VP8 (vp8)
+- VP9 (vp9)
+- AV1 (av01)
+- MJPEG (mjpeg)
+- MPEG-4 (mp4v)
+
+Codec blacklist (unsupported):
+- Theora
+- DivX/Xvid (legacy)
+- ProRes (requires special handling)
+- Any codec not in whitelist
+
+Behavior:
+- If codec not in whitelist → set status UNSCANNABLE, log reason
+- If ffprobe fails → set status UNSCANNABLE, log error code
+- If file is not a valid video → set status UNSCANNABLE, log reason
+- UNSCANNABLE files are excluded from processing pipeline
+
+### 2.4 Batch Orchestration Skeleton
+**File:** `src/orchestrator.py`
+
+Components:
+- **Job Queue:** Priority queue based on file modification time (newest first)
+- **DB-based Locking:** Atomic state transition using UPDATE ... WHERE status = 'PENDING'
+- **Worker Pool:** Configurable number of worker processes (default: 2, one per GPU)
+- **State Machine:**
+  ```
+  NEW → PENDING → PROCESSING → COMPLETED
+                         → UNSCANNABLE
+                         → ERROR
+  ```
+- **Concurrency Control:** Max concurrent processing per GPU (default: 1 video at a time per worker)
+- **Idempotency:** Re-processing a file does not duplicate DB entries or outputs
+
+### 2.5 Configuration Updates
+**File:** `config.yaml` (updates to STORY-01)
+
+New fields:
+```yaml
+scanner:
+  scan_interval_seconds: 60
+  walker_threads: 8
+  ffprobe_timeout_seconds: 10
+  hash_algorithm: sha256
+  hash_chunk_size_mb: 1
+
+codec:
+  whitelist: [avc1, hevc, vp8, vp9, av01, mjpeg, mp4v]
+  default_status_on_error: UNSCANNABLE
+
+queue:
+  priority: modification_time  # newest first
+  max_concurrent_per_gpu: 1
+  lock_timeout_seconds: 300
+```
+
+## Acceptance Criteria
+
+### Functional
+- [ ] New files in `/data/input` appear in DB with correct metadata within 60 seconds of placement
+- [ ] File hash deduplication prevents re-processing identical files
+- [ ] ffprobe correctly extracts codec, resolution, and duration for all supported codecs
+- [ ] Unsupported codec files are marked UNSCANNABLE without crashing the scanner
+- [ ] Corrupt/unreadable files are marked UNSCANNABLE with appropriate error logged
+- [ ] Atomic state transitions: PENDING → PROCESSING succeeds only once (no duplicate processing)
+- [ ] Removed files are detected and marked REMOVED (if configured)
+- [ ] Scanner handles 30TB directory structure without OOM (memory < 500MB during scan)
+- [ ] Re-running scanner is idempotent — no duplicate entries or states
+
+### Non-Functional
+- [ ] Scanner completes full 30TB directory walk in < 4 hours (incremental: < 10 minutes for typical day)
+- [ ] ffprobe timeout (10s) is enforced — does not hang on corrupt files
+- [ ] Scanner memory usage stays < 500MB regardless of directory depth
+- [ ] Hash computation for 100MB file completes in < 5 seconds on NVMe
+- [ ] Worker pool respects GPU count (2 workers for 2 GPUs)
+
+### Technical Constraints
+- [ ] SHA-256 hash is deterministic and reproducible
+- [ ] Codec detection matches ffprobe output exactly
+- [ ] Resolution stored as integers (width, height)
+- [ ] Duration stored as float (seconds)
+- [ ] All scanner errors are logged with video_id, error_code, and file_path
+- [ ] Scanner does not modify video files (read-only operation)
+
+## Dependencies
+- **Prerequisites:** STORY-01 (Foundation & Infrastructure) — DB schema, connection layer, config
+- **Depends on:** None (runs in parallel with STORY-03 if needed)
+- **Enables:** STORY-04 (Inference Pipeline), STORY-05 (Results & Export)
+
+## Risks & Mitigations
+| Risk | Mitigation |
+|------|--|
+| 30TB directory walk is extremely slow | Incremental scanning using last_scan_time; only walk new directories |
+| ffprobe hangs on corrupt files | Enforce 10-second timeout via subprocess timeout |
+| Hash computation on slow NAS is bottleneck | Hash only first 1MB; full hash optional for verification |
+| Concurrent scanner + worker conflicts | DB-based locking; scanner only writes PENDING, workers read PENDING |
+| Network mount latency | Cache directory listings; batch DB operations |
+
+## Estimated Effort
+- **Sprint:** 3-4
+- **Story Points:** 21
+- **Dependencies:** STORY-01