153 lines
6.4 KiB
Markdown
153 lines
6.4 KiB
Markdown
# STORY-03: Frame Sampling
|
|
|
|
## Epic
|
|
**E1: Core Pipeline** — As an engineer, I can configure frame sampling interval and extract frames uniformly.
|
|
|
|
## Related Requirements
|
|
| ID | Requirement |
|
|
|----|---------|
|
|
| FR-01 | Configurable frame sampling interval (default: 1 frame per 30 seconds); Must support override per job/batch; Uniform temporal sampling preferred |
|
|
| NFR-02 | Latency per video: ≤ 45 seconds end-to-end (15-min avg video) |
|
|
| NFR-03 | GPU Memory Safety: ≤ 18GB per GPU sustained |
|
|
| NFR-05 | Fault Tolerance: Auto-retry on transient failures; skip & log on fatal errors |
|
|
| TC-03 | Storage I/O: Fast local NVMe/SSD for temp frame cache |
|
|
|
|
## Description
|
|
Implement frame extraction from processed videos using FFmpeg/OpenCV. Extract one frame per configurable interval (default: 30 seconds) uniformly across the video duration. Handle variable FPS, high-resolution frames, and manage scratch space efficiently.
|
|
|
|
## Scope
|
|
|
|
### In Scope
|
|
- Uniform temporal frame sampling via FFmpeg `-ss` timestamp extraction
|
|
- Configurable sampling interval (default: 30 seconds, override per job)
|
|
- Variable FPS handling (calculate correct timestamps)
|
|
- Frame extraction to JPEG format for storage efficiency
|
|
- Scratch space management (NVMe tmpfs with auto-cleanup)
|
|
- High-resolution frame handling (downscale 4K to fit VRAM constraints)
|
|
- Frame naming convention: `{video_id}_{timestamp}.jpg`
|
|
- Memory management (delete frames after processing)
|
|
|
|
### Out of Scope
|
|
- Face detection (covered in STORY-04)
|
|
- Classification (covered in STORY-05)
|
|
- Confidence scoring (covered in STORY-05)
|
|
- Results export (covered in STORY-06)
|
|
- Review UI (covered in STORY-07)
|
|
|
|
## Deliverables
|
|
|
|
### 3.1 Frame Sampler Module
|
|
**File:** `src/frame_sampler.py`
|
|
|
|
Core functionality:
|
|
- **Timestamp Calculator:** Compute uniform sampling timestamps
|
|
```python
|
|
def calculate_timestamps(duration, interval):
|
|
"""Return list of timestamps for frame extraction."""
|
|
count = max(1, int(duration / interval))
|
|
step = duration / count
|
|
return [i * step for i in range(count)]
|
|
```
|
|
- **FFmpeg Extraction:** Extract frames at computed timestamps
|
|
```bash
|
|
ffmpeg -ss {timestamp} -i {video_path} -vframes 1 -q:v 2 -f jpeg {output_path}
|
|
```
|
|
- **Variable FPS Handling:** Adjust timestamps for videos with variable frame rates
|
|
- **Quality Control:** JPEG quality factor 2 (high quality, reasonable size)
|
|
- **Error Handling:** Per-frame error isolation — skip failed frames, log errors, continue
|
|
|
|
### 3.2 Resolution Handling
|
|
**File:** `src/frame_sampler.py` (resolution handling section)
|
|
|
|
Features:
|
|
- **Resolution Detection:** Use metadata from prober (STORY-02)
|
|
- **Downscale Logic:** Auto-downscale frames > 1080p to fit VRAM
|
|
- 4K (3840x2160) → 1080p (1920x1080)
|
|
- 2K (2560x1440) → 720p (1280x720)
|
|
- ≤ 1080p → no change
|
|
- **FFmpeg Scale Filter:** Apply during extraction
|
|
```bash
|
|
ffmpeg -ss {ts} -i {video} -vf "scale=1920:1080:force_original_aspect_ratio=decrease,pad=1920:1080:(ow-iw)/2:(oh-ih)/2" -vframes 1 -q:v 2 -f jpeg {output}
|
|
```
|
|
- **Aspect Ratio Preservation:** Letterbox/pillarbox to maintain original aspect ratio
|
|
|
|
### 3.3 Scratch Space Management
|
|
**File:** `src/scratch_manager.py`
|
|
|
|
Features:
|
|
- **Directory Structure:** `/scratch/{video_id}/frames/` for each video
|
|
- **Naming Convention:** `{video_id}_{timestamp_ms}.jpg`
|
|
- **Auto-Cleanup:** Delete all frames after video processing completes
|
|
- **Space Monitoring:** Alert if scratch usage exceeds 80% of tmpfs capacity
|
|
- **Cleanup on Error:** Ensure frames are cleaned up even if processing fails
|
|
- **Concurrent Access:** Per-video directory isolation (no conflicts between workers)
|
|
|
|
### 3.4 Configuration Updates
|
|
**File:** `config.yaml` (updates to STORY-01/02)
|
|
|
|
New fields:
|
|
```yaml
|
|
sampling:
|
|
interval_seconds: 30
|
|
override_per_job: true
|
|
quality: 2 # JPEG quality (1-31, lower=better)
|
|
format: jpeg
|
|
|
|
resolution:
|
|
max_height: 1080
|
|
preserve_aspect_ratio: true
|
|
pad_to_square: false # Will be done in face detection if needed
|
|
|
|
scratch:
|
|
path: /scratch
|
|
max_usage_percent: 80
|
|
auto_cleanup: true
|
|
cleanup_on_error: true
|
|
```
|
|
|
|
## Acceptance Criteria
|
|
|
|
### Functional
|
|
- [ ] Frames are extracted at uniform intervals (±1 frame tolerance) across video duration
|
|
- [ ] Default interval of 30 seconds produces correct number of frames for test videos
|
|
- [ ] Per-job interval override works (e.g., 10-second interval for specific videos)
|
|
- [ ] Variable FPS videos produce evenly spaced timestamps (not evenly spaced frames)
|
|
- [ ] 4K frames are downsampled to 1080p without crashing
|
|
- [ ] Aspect ratio is preserved during downscaling (no distortion)
|
|
- [ ] Frames are saved as high-quality JPEGs (quality factor 2)
|
|
- [ ] Frame naming follows convention: `{video_id}_{timestamp_ms}.jpg`
|
|
- [ ] Scratch space is cleaned up after video processing (no leftover frames)
|
|
- [ ] Failed frame extraction does not stop processing of other frames
|
|
|
|
### Non-Functional
|
|
- [ ] Frame extraction for a 15-minute video completes in < 10 seconds on NVMe
|
|
- [ ] Scratch space usage per video < 50MB (typical case)
|
|
- [ ] Memory usage during extraction < 500MB per worker
|
|
- [ ] FFmpeg process timeout enforced (30 seconds per frame)
|
|
- [ ] No frames left in scratch after processing completes (verified by directory check)
|
|
|
|
### Technical Constraints
|
|
- [ ] FFmpeg uses `-ss` for accurate timestamp seeking (not frame-by-frame)
|
|
- [ ] Downscaling uses FFmpeg scale filter (not post-processing resize)
|
|
- [ ] JPEG quality is consistent across all extracted frames
|
|
- [ ] Timestamps are in seconds with millisecond precision
|
|
- [ ] Scratch directory is created per-video and isolated from other workers
|
|
|
|
## Dependencies
|
|
- **Prerequisites:** STORY-01 (Foundation), STORY-02 (Ingestion — provides metadata)
|
|
- **Depends on:** None (runs after ingestion, before STORY-04)
|
|
- **Enables:** STORY-04 (Face Detection — provides frames), STORY-05 (Classification)
|
|
|
|
## Risks & Mitigations
|
|
| Risk | Mitigation |
|
|
|------|--|
|
|
| FFmpeg seeking is inaccurate for some codecs | Use `-accurate_seek` flag; fall back to frame-by-frame if needed |
|
|
| 4K downscaling increases processing time | Use FFmpeg scale filter (GPU-accelerated if available) |
|
|
| Scratch space fills up with many concurrent videos | Monitor usage; implement cleanup queue; alert at 80% |
|
|
| Variable FPS causes uneven frame distribution | Calculate timestamps based on duration, not frame count |
|
|
|
|
## Estimated Effort
|
|
- **Sprint:** 5 (first half)
|
|
- **Story Points:** 13
|
|
- **Dependencies:** STORY-01, STORY-02
|