VideoDetect/STORY-06.md

# STORY-06: Results Persistence & Export

## Epic
**E1: Core Pipeline** — As a system, I can persist processing results and export data for analytics.

## Related Requirements
| ID | Requirement |
|----|------|
| FR-07 | Metadata logging & audit trail: Stores video ID, timestamps, frame counts, confidence scores, routing decision, model version |
| FR-04 | Video-level confidence aggregation & threshold routing: Results saved to Parquet |
| NFR-05 | Fault Tolerance: Metadata persisted before cleanup |
| NFR-07 | Data Volume Handling: Efficient indexing for ~30TB dataset |
| TC-03 | Storage I/O: Shared NAS/SMB for video input/output |

## Description
Implement result persistence, metadata logging, and data export. Update the database with processing results, insert processing log entries, and export summary data in Parquet/JSONL format for analytics. Ensure all persistence happens before scratch space cleanup to prevent data loss.

## Scope

### In Scope
- Database update for processed videos (status, confidence, routing, timestamps)
- Processing log entry insertion (atomic transaction)
- Parquet/JSONL export to `/data/output`
- Confidence score persistence (frame-level and video-level)
- Model version tracking in DB and export
- Idempotent result persistence (re-processing doesn't duplicate entries)
- Transaction-safe updates (all-or-nothing)

### Out of Scope
- Frame sampling (covered in STORY-03)
- Face detection (covered in STORY-04)
- Classification (covered in STORY-05)
- Review UI (covered in STORY-07)
- Active learning pipeline (covered in STORY-08)

## Deliverables

### 6.1 Result Updater
**File:** `src/result_updater.py`

Features:
- **Video Record Update:** Update `videos` table with processing results
  ```sql
  UPDATE videos SET
      status = 'COMPLETED',
      last_processed_time = NOW(),
      confidence_score = %s,
      routing_decision = %s,
      model_version = %s,
      frame_count = %s,
      updated_at = NOW()
  WHERE id = %s AND status = 'PROCESSING'
  ```
- **Atomic Transaction:** Wrap video update + log insert in single transaction
- **State Guard:** Only update if status is PROCESSING (prevents duplicate processing)
- **Error Handling:** Rollback transaction on any error; re-queue video as PENDING

### 6.2 Processing Logger
**File:** `src/processing_logger.py`

Features:
- **Log Entry Insertion:** Insert record into `processing_logs` table
  ```python
  def insert_processing_log(video_id, model_version, frame_count,
                            confidence_score, routing_decision):
      cursor.execute("""
          INSERT INTO processing_logs
              (video_id, model_version, frame_count, confidence_score,
               routing_decision, processed_at)
          VALUES (%s, %s, %s, %s, %s, NOW())
      """, (video_id, model_version, frame_count,
            confidence_score, routing_decision))
  ```
- **Frame-Level Confidence Storage:** Store all frame-level confidence scores
  - Option A: JSON array in processing_logs.confidence_scores (TEXT column)
  - Option B: Separate `frame_confidences` table (for large datasets)
  - Default: JSON array in processing_logs (simpler, sufficient for typical frame counts)
- **Audit Trail:** All log entries include timestamp, model version, and routing decision

### 6.3 Data Export
**File:** `src/data_export.py`

Features:
- **Parquet Export:** Write summary data to Parquet format
  ```python
  # Schema:
  # video_id (int64), file_path (string), file_hash (string),
  # model_version (string), sample_count (int32),
  # confidence_scores (list<float>), video_confidence (float),
  # routing (string), processed_at (timestamp)
  ```
- **JSONL Export:** Alternative line-delimited JSON format for streaming analytics
  ```json
  {"video_id": 12345, "file_path": "/data/input/video.mp4",
   "model_version": "v1.2.0", "sample_count": 8,
   "confidence_scores": [0.82, 0.79, 0.85, 0.76, 0.81, 0.83, 0.78, 0.80],
   "video_confidence": 0.85, "routing": "MATCH", "processed_at": "2026-08-03T10:30:00Z"}
  ```
- **Output Directory:** `/data/output/{model_version}/` organized by model version
- **File Naming:** `{model_version}_{batch_id}.parquet` and `{model_version}_{batch_id}.jsonl`
- **Batch ID:** Sequential counter or timestamp-based (e.g., `v1.2.0_20260803_103000`)
- **Compression:** Parquet with Snappy compression
- **Export Trigger:** Export after all videos in batch are processed (or periodically)

### 6.4 Scratch Cleanup
**File:** `src/scratch_manager.py` (cleanup section)

Features:
- **Post-Processing Cleanup:** Delete all frames and crops after results are persisted
- **Cleanup Order:** Results → Logs → Export → Cleanup (critical ordering)
- **Cleanup Verification:** Verify DB update before deleting scratch files
- **Cleanup on Error:** If persistence fails, do NOT delete scratch (allow retry)
- **Cleanup Logging:** Log cleanup completion with video_id and files removed

### 6.5 Configuration Updates
**File:** `config.yaml` (updates)

New fields:
```yaml
results:
  persist_before_cleanup: true
  transaction_safe: true
  state_guard: true  # only update PROCESSING → COMPLETED

export:
  format: parquet  # parquet, jsonl, or both
  output_path: /data/output
  compression: snappy
  batch_size: 100  # export after N videos
  include_frame_confidences: true
```

## Acceptance Criteria

### Functional
- [ ] Video status transitions from PROCESSING to COMPLETED after successful processing
- [ ] Processing log entry is inserted for every processed video
- [ ] Frame-level confidence scores are stored (all p_i values)
- [ ] Video-level confidence score is stored correctly
- [ ] Routing decision is persisted accurately
- [ ] Model version is recorded in both videos and processing_logs tables
- [ ] Parquet export produces valid, readable files with correct schema
- [ ] JSONL export produces valid JSON lines with correct fields
- [ ] Export files are organized by model version in /data/output/
- [ ] Scratch space is cleaned up after persistence (verified by directory check)
- [ ] Re-processing a video updates existing records (no duplicates)
- [ ] Transaction rollback on error prevents partial state updates

### Non-Functional
- [ ] Result persistence completes in < 5 seconds per video
- [ ] Parquet export for 100 videos completes in < 10 seconds
- [ ] JSONL export for 100 videos completes in < 10 seconds
- [ ] Export files are valid (verified by parquet.read_table and jsonl parsing)
- [ ] Scratch cleanup frees > 95% of allocated space per video
- [ ] No data loss if worker crashes between processing and persistence (PROCESSING state preserved)

### Technical Constraints
- [ ] All DB updates use parameterized queries (no SQL injection)
- [ ] Transaction isolation level is READ COMMITTED or higher
- [ ] Parquet files use Snappy compression (not gzip, for speed)
- [ ] JSONL files are UTF-8 encoded
- [ ] Confidence scores are stored with float64 precision
- [ ] Timestamps are in UTC (ISO 8601 format)

## Dependencies
- **Prerequisites:** STORY-01 (Foundation — DB schema), STORY-05 (Classification — provides results)
- **Depends on:** None (runs after classification)
- **Enables:** STORY-07 (Review UI — provides routing decisions), STORY-08 (Active Learning — provides training data)

## Risks & Mitigations
| Risk | Mitigation |
|------|--|
| NAS write latency slows export | Batch writes; use local tmpfs for export staging |
| Parquet library compatibility with CUDA container | Test pyarrow in Worker container early; pin version |
| Transaction rollback leaves video in PROCESSING state | DB lock timeout (5 min) prevents permanent lock |
| Scratch cleanup before persistence causes data loss | Enforce ordering: persist → verify → cleanup |

## Estimated Effort
- **Sprint:** 5-6 (second half)
- **Story Points:** 13
- **Dependencies:** STORY-01, STORY-05