After story 1
This commit is contained in:
+181
@@ -0,0 +1,181 @@
|
||||
# STORY-06: Results Persistence & Export
|
||||
|
||||
## Epic
|
||||
**E1: Core Pipeline** — As a system, I can persist processing results and export data for analytics.
|
||||
|
||||
## Related Requirements
|
||||
| ID | Requirement |
|
||||
|----|------|
|
||||
| FR-07 | Metadata logging & audit trail: Stores video ID, timestamps, frame counts, confidence scores, routing decision, model version |
|
||||
| FR-04 | Video-level confidence aggregation & threshold routing: Results saved to Parquet |
|
||||
| NFR-05 | Fault Tolerance: Metadata persisted before cleanup |
|
||||
| NFR-07 | Data Volume Handling: Efficient indexing for ~30TB dataset |
|
||||
| TC-03 | Storage I/O: Shared NAS/SMB for video input/output |
|
||||
|
||||
## Description
|
||||
Implement result persistence, metadata logging, and data export. Update the database with processing results, insert processing log entries, and export summary data in Parquet/JSONL format for analytics. Ensure all persistence happens before scratch space cleanup to prevent data loss.
|
||||
|
||||
## Scope
|
||||
|
||||
### In Scope
|
||||
- Database update for processed videos (status, confidence, routing, timestamps)
|
||||
- Processing log entry insertion (atomic transaction)
|
||||
- Parquet/JSONL export to `/data/output`
|
||||
- Confidence score persistence (frame-level and video-level)
|
||||
- Model version tracking in DB and export
|
||||
- Idempotent result persistence (re-processing doesn't duplicate entries)
|
||||
- Transaction-safe updates (all-or-nothing)
|
||||
|
||||
### Out of Scope
|
||||
- Frame sampling (covered in STORY-03)
|
||||
- Face detection (covered in STORY-04)
|
||||
- Classification (covered in STORY-05)
|
||||
- Review UI (covered in STORY-07)
|
||||
- Active learning pipeline (covered in STORY-08)
|
||||
|
||||
## Deliverables
|
||||
|
||||
### 6.1 Result Updater
|
||||
**File:** `src/result_updater.py`
|
||||
|
||||
Features:
|
||||
- **Video Record Update:** Update `videos` table with processing results
|
||||
```sql
|
||||
UPDATE videos SET
|
||||
status = 'COMPLETED',
|
||||
last_processed_time = NOW(),
|
||||
confidence_score = %s,
|
||||
routing_decision = %s,
|
||||
model_version = %s,
|
||||
frame_count = %s,
|
||||
updated_at = NOW()
|
||||
WHERE id = %s AND status = 'PROCESSING'
|
||||
```
|
||||
- **Atomic Transaction:** Wrap video update + log insert in single transaction
|
||||
- **State Guard:** Only update if status is PROCESSING (prevents duplicate processing)
|
||||
- **Error Handling:** Rollback transaction on any error; re-queue video as PENDING
|
||||
|
||||
### 6.2 Processing Logger
|
||||
**File:** `src/processing_logger.py`
|
||||
|
||||
Features:
|
||||
- **Log Entry Insertion:** Insert record into `processing_logs` table
|
||||
```python
|
||||
def insert_processing_log(video_id, model_version, frame_count,
|
||||
confidence_score, routing_decision):
|
||||
cursor.execute("""
|
||||
INSERT INTO processing_logs
|
||||
(video_id, model_version, frame_count, confidence_score,
|
||||
routing_decision, processed_at)
|
||||
VALUES (%s, %s, %s, %s, %s, NOW())
|
||||
""", (video_id, model_version, frame_count,
|
||||
confidence_score, routing_decision))
|
||||
```
|
||||
- **Frame-Level Confidence Storage:** Store all frame-level confidence scores
|
||||
- Option A: JSON array in processing_logs.confidence_scores (TEXT column)
|
||||
- Option B: Separate `frame_confidences` table (for large datasets)
|
||||
- Default: JSON array in processing_logs (simpler, sufficient for typical frame counts)
|
||||
- **Audit Trail:** All log entries include timestamp, model version, and routing decision
|
||||
|
||||
### 6.3 Data Export
|
||||
**File:** `src/data_export.py`
|
||||
|
||||
Features:
|
||||
- **Parquet Export:** Write summary data to Parquet format
|
||||
```python
|
||||
# Schema:
|
||||
# video_id (int64), file_path (string), file_hash (string),
|
||||
# model_version (string), sample_count (int32),
|
||||
# confidence_scores (list<float>), video_confidence (float),
|
||||
# routing (string), processed_at (timestamp)
|
||||
```
|
||||
- **JSONL Export:** Alternative line-delimited JSON format for streaming analytics
|
||||
```json
|
||||
{"video_id": 12345, "file_path": "/data/input/video.mp4",
|
||||
"model_version": "v1.2.0", "sample_count": 8,
|
||||
"confidence_scores": [0.82, 0.79, 0.85, 0.76, 0.81, 0.83, 0.78, 0.80],
|
||||
"video_confidence": 0.85, "routing": "MATCH", "processed_at": "2026-08-03T10:30:00Z"}
|
||||
```
|
||||
- **Output Directory:** `/data/output/{model_version}/` organized by model version
|
||||
- **File Naming:** `{model_version}_{batch_id}.parquet` and `{model_version}_{batch_id}.jsonl`
|
||||
- **Batch ID:** Sequential counter or timestamp-based (e.g., `v1.2.0_20260803_103000`)
|
||||
- **Compression:** Parquet with Snappy compression
|
||||
- **Export Trigger:** Export after all videos in batch are processed (or periodically)
|
||||
|
||||
### 6.4 Scratch Cleanup
|
||||
**File:** `src/scratch_manager.py` (cleanup section)
|
||||
|
||||
Features:
|
||||
- **Post-Processing Cleanup:** Delete all frames and crops after results are persisted
|
||||
- **Cleanup Order:** Results → Logs → Export → Cleanup (critical ordering)
|
||||
- **Cleanup Verification:** Verify DB update before deleting scratch files
|
||||
- **Cleanup on Error:** If persistence fails, do NOT delete scratch (allow retry)
|
||||
- **Cleanup Logging:** Log cleanup completion with video_id and files removed
|
||||
|
||||
### 6.5 Configuration Updates
|
||||
**File:** `config.yaml` (updates)
|
||||
|
||||
New fields:
|
||||
```yaml
|
||||
results:
|
||||
persist_before_cleanup: true
|
||||
transaction_safe: true
|
||||
state_guard: true # only update PROCESSING → COMPLETED
|
||||
|
||||
export:
|
||||
format: parquet # parquet, jsonl, or both
|
||||
output_path: /data/output
|
||||
compression: snappy
|
||||
batch_size: 100 # export after N videos
|
||||
include_frame_confidences: true
|
||||
```
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
### Functional
|
||||
- [ ] Video status transitions from PROCESSING to COMPLETED after successful processing
|
||||
- [ ] Processing log entry is inserted for every processed video
|
||||
- [ ] Frame-level confidence scores are stored (all p_i values)
|
||||
- [ ] Video-level confidence score is stored correctly
|
||||
- [ ] Routing decision is persisted accurately
|
||||
- [ ] Model version is recorded in both videos and processing_logs tables
|
||||
- [ ] Parquet export produces valid, readable files with correct schema
|
||||
- [ ] JSONL export produces valid JSON lines with correct fields
|
||||
- [ ] Export files are organized by model version in /data/output/
|
||||
- [ ] Scratch space is cleaned up after persistence (verified by directory check)
|
||||
- [ ] Re-processing a video updates existing records (no duplicates)
|
||||
- [ ] Transaction rollback on error prevents partial state updates
|
||||
|
||||
### Non-Functional
|
||||
- [ ] Result persistence completes in < 5 seconds per video
|
||||
- [ ] Parquet export for 100 videos completes in < 10 seconds
|
||||
- [ ] JSONL export for 100 videos completes in < 10 seconds
|
||||
- [ ] Export files are valid (verified by parquet.read_table and jsonl parsing)
|
||||
- [ ] Scratch cleanup frees > 95% of allocated space per video
|
||||
- [ ] No data loss if worker crashes between processing and persistence (PROCESSING state preserved)
|
||||
|
||||
### Technical Constraints
|
||||
- [ ] All DB updates use parameterized queries (no SQL injection)
|
||||
- [ ] Transaction isolation level is READ COMMITTED or higher
|
||||
- [ ] Parquet files use Snappy compression (not gzip, for speed)
|
||||
- [ ] JSONL files are UTF-8 encoded
|
||||
- [ ] Confidence scores are stored with float64 precision
|
||||
- [ ] Timestamps are in UTC (ISO 8601 format)
|
||||
|
||||
## Dependencies
|
||||
- **Prerequisites:** STORY-01 (Foundation — DB schema), STORY-05 (Classification — provides results)
|
||||
- **Depends on:** None (runs after classification)
|
||||
- **Enables:** STORY-07 (Review UI — provides routing decisions), STORY-08 (Active Learning — provides training data)
|
||||
|
||||
## Risks & Mitigations
|
||||
| Risk | Mitigation |
|
||||
|------|--|
|
||||
| NAS write latency slows export | Batch writes; use local tmpfs for export staging |
|
||||
| Parquet library compatibility with CUDA container | Test pyarrow in Worker container early; pin version |
|
||||
| Transaction rollback leaves video in PROCESSING state | DB lock timeout (5 min) prevents permanent lock |
|
||||
| Scratch cleanup before persistence causes data loss | Enforce ordering: persist → verify → cleanup |
|
||||
|
||||
## Estimated Effort
|
||||
- **Sprint:** 5-6 (second half)
|
||||
- **Story Points:** 13
|
||||
- **Dependencies:** STORY-01, STORY-05
|
||||
Reference in New Issue
Block a user