VideoDetect/STORY-03.md

STORY-03: Frame Sampling

Epic

E1: Core Pipeline — As an engineer, I can configure frame sampling interval and extract frames uniformly.

Related Requirements

ID	Requirement
FR-01	Configurable frame sampling interval (default: 1 frame per 30 seconds); Must support override per job/batch; Uniform temporal sampling preferred
NFR-02	Latency per video: ≤ 45 seconds end-to-end (15-min avg video)
NFR-03	GPU Memory Safety: ≤ 18GB per GPU sustained
NFR-05	Fault Tolerance: Auto-retry on transient failures; skip & log on fatal errors
TC-03	Storage I/O: Fast local NVMe/SSD for temp frame cache

Description

Implement frame extraction from processed videos using FFmpeg/OpenCV. Extract one frame per configurable interval (default: 30 seconds) uniformly across the video duration. Handle variable FPS, high-resolution frames, and manage scratch space efficiently.

Scope

In Scope

• Uniform temporal frame sampling via FFmpeg -ss timestamp extraction
• Configurable sampling interval (default: 30 seconds, override per job)
• Variable FPS handling (calculate correct timestamps)
• Frame extraction to JPEG format for storage efficiency
• Scratch space management (NVMe tmpfs with auto-cleanup)
• High-resolution frame handling (downscale 4K to fit VRAM constraints)
• Frame naming convention: {video_id}_{timestamp}.jpg
• Memory management (delete frames after processing)

Out of Scope

• Face detection (covered in STORY-04)
• Classification (covered in STORY-05)
• Confidence scoring (covered in STORY-05)
• Results export (covered in STORY-06)
• Review UI (covered in STORY-07)

Deliverables

3.1 Frame Sampler Module

File: src/frame_sampler.py

Core functionality:

• Timestamp Calculator: Compute uniform sampling timestamps

  def calculate_timestamps(duration, interval):
      """Return list of timestamps for frame extraction."""
      count = max(1, int(duration / interval))
      step = duration / count
      return [i * step for i in range(count)]
  

• FFmpeg Extraction: Extract frames at computed timestamps

  ffmpeg -ss {timestamp} -i {video_path} -vframes 1 -q:v 2 -f jpeg {output_path}
  

• Variable FPS Handling: Adjust timestamps for videos with variable frame rates
• Quality Control: JPEG quality factor 2 (high quality, reasonable size)
• Error Handling: Per-frame error isolation — skip failed frames, log errors, continue

3.2 Resolution Handling

File: src/frame_sampler.py (resolution handling section)

Features:

• Resolution Detection: Use metadata from prober (STORY-02)
• Downscale Logic: Auto-downscale frames > 1080p to fit VRAM
  • 4K (3840x2160) → 1080p (1920x1080)
  • 2K (2560x1440) → 720p (1280x720)
  • ≤ 1080p → no change
• FFmpeg Scale Filter: Apply during extraction

  ffmpeg -ss {ts} -i {video} -vf "scale=1920:1080:force_original_aspect_ratio=decrease,pad=1920:1080:(ow-iw)/2:(oh-ih)/2" -vframes 1 -q:v 2 -f jpeg {output}
  

• Aspect Ratio Preservation: Letterbox/pillarbox to maintain original aspect ratio

3.3 Scratch Space Management

File: src/scratch_manager.py

Features:

• Directory Structure: /scratch/{video_id}/frames/ for each video
• Naming Convention: {video_id}_{timestamp_ms}.jpg
• Auto-Cleanup: Delete all frames after video processing completes
• Space Monitoring: Alert if scratch usage exceeds 80% of tmpfs capacity
• Cleanup on Error: Ensure frames are cleaned up even if processing fails
• Concurrent Access: Per-video directory isolation (no conflicts between workers)

3.4 Configuration Updates

File: config.yaml (updates to STORY-01/02)

New fields:

sampling:
  interval_seconds: 30
  override_per_job: true
  quality: 2  # JPEG quality (1-31, lower=better)
  format: jpeg

resolution:
  max_height: 1080
  preserve_aspect_ratio: true
  pad_to_square: false  # Will be done in face detection if needed

scratch:
  path: /scratch
  max_usage_percent: 80
  auto_cleanup: true
  cleanup_on_error: true

Acceptance Criteria

Functional

• Frames are extracted at uniform intervals (±1 frame tolerance) across video duration
• Default interval of 30 seconds produces correct number of frames for test videos
• Per-job interval override works (e.g., 10-second interval for specific videos)
• Variable FPS videos produce evenly spaced timestamps (not evenly spaced frames)
• 4K frames are downsampled to 1080p without crashing
• Aspect ratio is preserved during downscaling (no distortion)
• Frames are saved as high-quality JPEGs (quality factor 2)
• Frame naming follows convention: {video_id}_{timestamp_ms}.jpg
• Scratch space is cleaned up after video processing (no leftover frames)
• Failed frame extraction does not stop processing of other frames

Non-Functional

• Frame extraction for a 15-minute video completes in < 10 seconds on NVMe
• Scratch space usage per video < 50MB (typical case)
• Memory usage during extraction < 500MB per worker
• FFmpeg process timeout enforced (30 seconds per frame)
• No frames left in scratch after processing completes (verified by directory check)

Technical Constraints

• FFmpeg uses -ss for accurate timestamp seeking (not frame-by-frame)
• Downscaling uses FFmpeg scale filter (not post-processing resize)
• JPEG quality is consistent across all extracted frames
• Timestamps are in seconds with millisecond precision
• Scratch directory is created per-video and isolated from other workers

Dependencies

• Prerequisites: STORY-01 (Foundation), STORY-02 (Ingestion — provides metadata)
• Depends on: None (runs after ingestion, before STORY-04)
• Enables: STORY-04 (Face Detection — provides frames), STORY-05 (Classification)

Risks & Mitigations

Risk	Mitigation
FFmpeg seeking is inaccurate for some codecs	Use -accurate_seek flag; fall back to frame-by-frame if needed
4K downscaling increases processing time	Use FFmpeg scale filter (GPU-accelerated if available)
Scratch space fills up with many concurrent videos	Monitor usage; implement cleanup queue; alert at 80%
Variable FPS causes uneven frame distribution	Calculate timestamps based on duration, not frame count

Estimated Effort

• Sprint: 5 (first half)
• Story Points: 13
• Dependencies: STORY-01, STORY-02