VideoDetect/STORY-06.md

STORY-06: Results Persistence & Export

Epic

E1: Core Pipeline — As a system, I can persist processing results and export data for analytics.

Related Requirements

ID	Requirement
FR-07	Metadata logging & audit trail: Stores video ID, timestamps, frame counts, confidence scores, routing decision, model version
FR-04	Video-level confidence aggregation & threshold routing: Results saved to Parquet
NFR-05	Fault Tolerance: Metadata persisted before cleanup
NFR-07	Data Volume Handling: Efficient indexing for ~30TB dataset
TC-03	Storage I/O: Shared NAS/SMB for video input/output

Description

Implement result persistence, metadata logging, and data export. Update the database with processing results, insert processing log entries, and export summary data in Parquet/JSONL format for analytics. Ensure all persistence happens before scratch space cleanup to prevent data loss.

Scope

In Scope

• Database update for processed videos (status, confidence, routing, timestamps)
• Processing log entry insertion (atomic transaction)
• Parquet/JSONL export to /data/output
• Confidence score persistence (frame-level and video-level)
• Model version tracking in DB and export
• Idempotent result persistence (re-processing doesn't duplicate entries)
• Transaction-safe updates (all-or-nothing)

Out of Scope

• Frame sampling (covered in STORY-03)
• Face detection (covered in STORY-04)
• Classification (covered in STORY-05)
• Review UI (covered in STORY-07)
• Active learning pipeline (covered in STORY-08)

Deliverables

6.1 Result Updater

File: src/result_updater.py

Features:

• Video Record Update: Update videos table with processing results

  UPDATE videos SET
      status = 'COMPLETED',
      last_processed_time = NOW(),
      confidence_score = %s,
      routing_decision = %s,
      model_version = %s,
      frame_count = %s,
      updated_at = NOW()
  WHERE id = %s AND status = 'PROCESSING'
  

• Atomic Transaction: Wrap video update + log insert in single transaction
• State Guard: Only update if status is PROCESSING (prevents duplicate processing)
• Error Handling: Rollback transaction on any error; re-queue video as PENDING

6.2 Processing Logger

File: src/processing_logger.py

Features:

• Log Entry Insertion: Insert record into processing_logs table

  def insert_processing_log(video_id, model_version, frame_count,
                            confidence_score, routing_decision):
      cursor.execute("""
          INSERT INTO processing_logs
              (video_id, model_version, frame_count, confidence_score,
               routing_decision, processed_at)
          VALUES (%s, %s, %s, %s, %s, NOW())
      """, (video_id, model_version, frame_count,
            confidence_score, routing_decision))
  

• Frame-Level Confidence Storage: Store all frame-level confidence scores
  • Option A: JSON array in processing_logs.confidence_scores (TEXT column)
  • Option B: Separate frame_confidences table (for large datasets)
  • Default: JSON array in processing_logs (simpler, sufficient for typical frame counts)
• Audit Trail: All log entries include timestamp, model version, and routing decision

6.3 Data Export

File: src/data_export.py

Features:

• Parquet Export: Write summary data to Parquet format

  # Schema:
  # video_id (int64), file_path (string), file_hash (string),
  # model_version (string), sample_count (int32),
  # confidence_scores (list<float>), video_confidence (float),
  # routing (string), processed_at (timestamp)
  

• JSONL Export: Alternative line-delimited JSON format for streaming analytics

  {"video_id": 12345, "file_path": "/data/input/video.mp4",
   "model_version": "v1.2.0", "sample_count": 8,
   "confidence_scores": [0.82, 0.79, 0.85, 0.76, 0.81, 0.83, 0.78, 0.80],
   "video_confidence": 0.85, "routing": "MATCH", "processed_at": "2026-08-03T10:30:00Z"}
  

• Output Directory: /data/output/{model_version}/ organized by model version
• File Naming: {model_version}_{batch_id}.parquet and {model_version}_{batch_id}.jsonl
• Batch ID: Sequential counter or timestamp-based (e.g., v1.2.0_20260803_103000)
• Compression: Parquet with Snappy compression
• Export Trigger: Export after all videos in batch are processed (or periodically)

6.4 Scratch Cleanup

File: src/scratch_manager.py (cleanup section)

Features:

• Post-Processing Cleanup: Delete all frames and crops after results are persisted
• Cleanup Order: Results → Logs → Export → Cleanup (critical ordering)
• Cleanup Verification: Verify DB update before deleting scratch files
• Cleanup on Error: If persistence fails, do NOT delete scratch (allow retry)
• Cleanup Logging: Log cleanup completion with video_id and files removed

6.5 Configuration Updates

File: config.yaml (updates)

New fields:

results:
  persist_before_cleanup: true
  transaction_safe: true
  state_guard: true  # only update PROCESSING → COMPLETED

export:
  format: parquet  # parquet, jsonl, or both
  output_path: /data/output
  compression: snappy
  batch_size: 100  # export after N videos
  include_frame_confidences: true

Acceptance Criteria

Functional

• Video status transitions from PROCESSING to COMPLETED after successful processing
• Processing log entry is inserted for every processed video
• Frame-level confidence scores are stored (all p_i values)
• Video-level confidence score is stored correctly
• Routing decision is persisted accurately
• Model version is recorded in both videos and processing_logs tables
• Parquet export produces valid, readable files with correct schema
• JSONL export produces valid JSON lines with correct fields
• Export files are organized by model version in /data/output/
• Scratch space is cleaned up after persistence (verified by directory check)
• Re-processing a video updates existing records (no duplicates)
• Transaction rollback on error prevents partial state updates

Non-Functional

• Result persistence completes in < 5 seconds per video
• Parquet export for 100 videos completes in < 10 seconds
• JSONL export for 100 videos completes in < 10 seconds
• Export files are valid (verified by parquet.read_table and jsonl parsing)
• Scratch cleanup frees > 95% of allocated space per video
• No data loss if worker crashes between processing and persistence (PROCESSING state preserved)

Technical Constraints

• All DB updates use parameterized queries (no SQL injection)
• Transaction isolation level is READ COMMITTED or higher
• Parquet files use Snappy compression (not gzip, for speed)
• JSONL files are UTF-8 encoded
• Confidence scores are stored with float64 precision
• Timestamps are in UTC (ISO 8601 format)

Dependencies

• Prerequisites: STORY-01 (Foundation — DB schema), STORY-05 (Classification — provides results)
• Depends on: None (runs after classification)
• Enables: STORY-07 (Review UI — provides routing decisions), STORY-08 (Active Learning — provides training data)

Risks & Mitigations

Risk	Mitigation
NAS write latency slows export	Batch writes; use local tmpfs for export staging
Parquet library compatibility with CUDA container	Test pyarrow in Worker container early; pin version
Transaction rollback leaves video in PROCESSING state	DB lock timeout (5 min) prevents permanent lock
Scratch cleanup before persistence causes data loss	Enforce ordering: persist → verify → cleanup

Estimated Effort

• Sprint: 5-6 (second half)
• Story Points: 13
• Dependencies: STORY-01, STORY-05