# STORY-09: Observability, Monitoring & Hardening

## Epic
**E4: Operations** — As a DevOps engineer, I can schedule, monitor, and resume batch jobs.

## Related Requirements
| ID | Requirement |
|----|------|
| NFR-05 | Fault Tolerance: Auto-retry on transient failures; skip & log on fatal errors |
| NFR-06 | Observability: Prometheus/Grafana metrics + structured logging; Tracks FPS, queue depth, confidence distribution, drift alerts |
| NFR-01 | Throughput: ≥ 30 videos/hour/GPU |
| NFR-03 | GPU Memory Safety: ≤ 18GB per GPU sustained |
| NFR-04 | Determinism & Reproducibility: Config-seeded randomness, versioned models |
| NFR-07 | Data Volume Handling: Efficient indexing for ~30TB dataset |

## Description
Implement comprehensive monitoring, metrics collection, fault tolerance, and system hardening. Expose Prometheus metrics for throughput, GPU utilization, queue depth, and confidence distribution. Build Grafana dashboards for real-time system health. Implement crash recovery, idempotency, retry logic, and drift detection.

## Scope

### In Scope
- Prometheus metrics exposure (videos processed, GPU utilization, queue depth, confidence distribution)
- Grafana dashboard configuration (throughput, error rates, confidence drift)
- Crash recovery with checkpointing
- Idempotent processing guarantees
- Retry logic for transient errors (up to 3 attempts)
- Weekly drift detection job
- Alerting on confidence distribution shifts and review queue growth
- Worker health checks and auto-restart

### Out of Scope
- Frame sampling (covered in STORY-04)
- Face detection (covered in STORY-05)
- Classification (covered in STORY-06)
- Review UI (covered in STORY-08)
- Active learning pipeline (covered in STORY-09)

## Deliverables

### 9.1 Prometheus Metrics
**File:** `src/metrics.py`

Metrics to expose:
- **Counter Metrics:**
  - `videos_processed_total` (label: routing_decision=MATCH|REVIEW|SKIP|UNSCANNABLE|ERROR)
  - `videos_processed_by_model_total` (label: model_version)
  - `frames_extracted_total`
  - `faces_detected_total`
  - `inference_errors_total` (label: error_type)
  - `retry_attempts_total` (label: step=probe|extract|detect|classify)
- **Gauge Metrics:**
  - `gpu_utilization_percent` (label: gpu_id)
  - `gpu_memory_used_bytes` (label: gpu_id)
  - `gpu_memory_free_bytes` (label: gpu_id)
  - `queue_depth_pending`
  - `queue_depth_processing`
  - `queue_depth_review`
  - `scratch_usage_bytes`
  - `scratch_usage_percent`
- **Histogram Metrics:**
  - `video_processing_duration_seconds` (label: routing_decision)
  - `confidence_score_distribution` (buckets: 0.0, 0.1, 0.2, ..., 0.9, 1.0)
  - `frame_count_per_video`
  - `face_count_per_video`
- **Summary Metrics:**
  - `throughput_videos_per_hour` (calculated from counter)
  - `average_confidence` (calculated from histogram)

Implementation:
```python
from prometheus_client import start_http_server, Counter, Gauge, Histogram

# Start metrics HTTP server
start_http_server(9090)  # /metrics endpoint

# Define metrics
videos_processed = Counter(
    'videos_processed_total',
    'Total videos processed',
    ['routing_decision']
)

gpu_memory = Gauge(
    'gpu_memory_used_bytes',
    'GPU memory usage',
    ['gpu_id']
)

confidence_hist = Histogram(
    'confidence_score_distribution',
    'Video confidence scores',
    buckets=[0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0]
)
```

### 9.2 Grafana Dashboards
**File:** `monitoring/grafana/dashboards/`

Dashboard 1: **System Overview**
- Panels:
  - Throughput (videos/hour) — line chart, 1h window
  - Queue depth (pending/processing/review) — stacked bar
  - GPU utilization (both GPUs) — line chart
  - GPU memory usage (both GPUs) — line chart
  - Error rate (errors/100 videos) — bar chart
  - Confidence distribution — histogram

Dashboard 2: **Processing Details**
- Panels:
  - Processing duration per video — scatter plot
  - Frame count per video — histogram
  - Face count per video — histogram
  - Confidence score by model version — box plot
  - Routing decision distribution — pie chart

Dashboard 3: **Active Learning**
- Panels:
  - Review queue size over time — line chart
  - Annotation rate (labels/day) — bar chart
  - Model version timeline — timeline panel
  - F1 score by model version — line chart
  - ECE by model version — line chart

**File:** `monitoring/grafana/dashboards/system_overview.json`
**File:** `monitoring/grafana/dashboards/processing_details.json`
**File:** `monitoring/grafana/dashboards/active_learning.json`

### 9.3 Crash Recovery
**File:** `src/crash_recovery.py`

Features:
- **Checkpointing:** Periodically save processing state
  ```python
  def save_checkpoint(video_id, state, progress):
      checkpoint = {
          'video_id': video_id,
          'state': state,  # 'processing', 'extracting', 'detecting', 'classifying'
          'progress': progress,  # dict of step -> completed
          'timestamp': datetime.utcnow().isoformat()
      }
      with open(f'/scratch/checkpoints/{video_id}.json', 'w') as f:
          json.dump(checkpoint, f)
  ```
- **Recovery on Startup:** Scan for PROCESSING videos and re-queue them
  ```sql
  UPDATE videos SET status = 'PENDING', updated_at = NOW()
  WHERE status = 'PROCESSING' AND updated_at < NOW() - INTERVAL 5 MINUTE
  ```
- **Lock Timeout:** 5 minutes (videos stuck in PROCESSING beyond this are re-queued)
- **Re-queue Logic:** Only re-queue if worker is down (detected via health check)
- **Idempotency:** Re-processing a video produces same results (no duplicates)

### 9.4 Retry Logic
**File:** `src/retry.py`

Features:
- **Retry Decorator:**
  ```python
  def retry(max_attempts=3, delay=1.0, backoff=2.0, exceptions=(Exception,)):
      def decorator(func):
          def wrapper(*args, **kwargs):
              last_exception = None
              for attempt in range(max_attempts):
                  try:
                      return func(*args, **kwargs)
                  except exceptions as e:
                      last_exception = e
                      if attempt < max_attempts - 1:
                          wait = delay * (backoff ** attempt)
                          time.sleep(wait)
              raise last_exception
          return wrapper
      return decorator
  ```
- **Applicable Steps:** ffprobe, frame extraction, face detection, classification, DB writes
- **Transient Errors:** Network timeout, GPU OOM, file lock contention
- **Non-Retryable Errors:** Codec unsupported, corrupt file, invalid path → skip and log
- **Retry Logging:** Log each retry attempt with error type and delay

### 9.5 Drift Detection
**File:** `src/drift_detector.py`

Features:
- **Weekly Job:** Compare current confidence distribution to baseline
  ```python
  def detect_drift(current_confidences, baseline_confidences, threshold=0.10):
      # Compare p(C > 0.5) shift
      current_high = sum(1 for c in current_confidences if c > 0.5) / len(current_confidences)
      baseline_high = sum(1 for c in baseline_confidences if c > 0.5) / len(baseline_confidences)
      shift = abs(current_high - baseline_high)
      
      if shift > threshold:
          alert(f"Confidence drift detected: {shift:.2%} shift in p(C > 0.5)")
          return True
      return False
  ```
- **Baseline:** Stored in DB or config (computed from last training cycle)
- **Alert Conditions:**
  - `p(C > 0.5)` shifts > 10% from baseline
  - Review queue grows unbounded (> 1000 items for > 24 hours)
  - Throughput drops below 20 videos/hour/GPU for > 1 hour
  - Error rate exceeds 5% for any 1-hour window
- **Alert Channels:** Email, Slack webhook, or log entry (configurable)

### 9.6 Worker Health Checks
**File:** `src/health_check.py`

Features:
- **Health Endpoint:** `/health` returns worker status
  ```json
  {
      "status": "healthy",
      "gpu_available": true,
      "gpu_memory_used_gb": 12.5,
      "queue_depth": 42,
      "uptime_seconds": 86400,
      "videos_processed_today": 156,
      "last_error": null
  }
  ```
- **Auto-Restart:** Docker restart policy for worker container
  ```yaml
  restart: unless-stopped
  healthcheck:
    test: ["CMD", "curl", "-f", "http://localhost:9090/health"]
    interval: 30s
    timeout: 10s
    retries: 3
  ```
- **GPU Health:** Monitor GPU temperature and error counts
- **Disk Health:** Monitor scratch space usage and NAS connectivity

### 9.7 Docker Compose Update
**File:** `docker-compose.yml` (update)

Add monitoring services:
```yaml
prometheus:
  image: prom/prometheus:v2.48.0
  volumes:
    - ./monitoring/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
    - prometheus_data:/prometheus
  ports:
    - "9090:9090"
  networks:
    - videodetect-network

grafana:
  image: grafana/grafana:10.2.0
  volumes:
    - grafana_data:/var/lib/grafana
    - ./monitoring/grafana/dashboards:/etc/grafana/provisioning/dashboards
  environment:
    - GF_SECURITY_ADMIN_PASSWORD=${GRAFANA_PASSWORD}
  ports:
    - "3000:3000"
  networks:
    - videodetect-network

volumes:
  prometheus_data:
  grafana_data:
```

### 9.8 Configuration Updates
**File:** `config.yaml` (updates)

New fields:
```yaml
monitoring:
  prometheus:
    enabled: true
    port: 9090
    metrics_path: /metrics
  grafana:
    enabled: true
    port: 3000
  alerts:
    confidence_drift_threshold: 0.10
    review_queue_max_size: 1000
    review_queue_max_age_hours: 24
    throughput_min_videos_per_hour: 20
    throughput_min_duration_hours: 1
    error_rate_threshold: 0.05
    error_rate_window_hours: 1
  drift_detection:
    enabled: true
    schedule: weekly  # cron: 0 2 * * 0 (Sundays at 2 AM)
    baseline_source: db  # db or config
  crash_recovery:
    lock_timeout_minutes: 5
    auto_requeue: true
  retry:
    max_attempts: 3
    initial_delay: 1.0
    backoff_factor: 2.0
    retryable_errors:
      - timeout
      - gpu_oom
      - file_lock
    non_retryable_errors:
      - codec_unsupported
      - file_corrupt
      - invalid_path
```

## Acceptance Criteria

### Functional
- [ ] Prometheus metrics are exposed at `/metrics` endpoint and queryable
- [ ] All required metrics are present (counters, gauges, histograms, summaries)
- [ ] Grafana dashboards load and display correct data
- [ ] System Overview dashboard shows throughput, queue depth, GPU metrics
- [ ] Processing Details dashboard shows duration, frame count, confidence distribution
- [ ] Active Learning dashboard shows review queue, annotation rate, model versions
- [ ] Crash recovery re-queues stuck PROCESSING videos on worker restart
- [ ] Idempotent processing: re-processing produces same results (no duplicates)
- [ ] Retry logic retries transient errors up to 3 times with exponential backoff
- [ ] Non-retryable errors are skipped and logged (no infinite retry)
- [ ] Drift detection runs weekly and alerts on > 10% confidence shift
- [ ] Review queue growth alert triggers when queue > 1000 for > 24 hours
- [ ] Worker health check returns correct status
- [ ] Auto-restart triggers on health check failure

### Non-Functional
- [ ] Metrics collection overhead < 2% of CPU
- [ ] Grafana dashboard loads in < 3 seconds
- [ ] Crash recovery completes in < 30 seconds
- [ ] Drift detection completes in < 5 minutes
- [ ] Alert delivery completes in < 60 seconds
- [ ] Prometheus data retention: 30 days (configurable)
- [ ] Grafana data source refresh: 30 seconds

### Technical Constraints
- [ ] Prometheus metrics follow naming conventions (unit suffixes, proper labels)
- [ ] Grafana dashboards are JSON-exportable and version-controlled
- [ ] Crash recovery is idempotent (running twice produces same result)
- [ ] Retry logic does not retry non-transient errors
- [ ] Drift detection baseline is stored and versioned
- [ ] Health check endpoint responds in < 1 second
- [ ] All alerts are logged with timestamp and context

## Dependencies
- **Prerequisites:** STORY-01 (Foundation), STORY-05 (Classification — provides metrics data), STORY-08 (Active Learning — provides model metrics)
- **Depends on:** None (can be built in parallel with other stories)
- **Enables:** Production deployment and long-term operation

## Risks & Mitigations
| Risk | Mitigation |
|------|--|
| Prometheus metrics cardinality explosion | Limit label cardinality; use histograms instead of individual values |
| Grafana dashboard load time with large datasets | Use Prometheus aggregations; pre-compute panels |
| Crash recovery misses in-flight writes | Use DB transactions; lock timeout prevents permanent locks |
| Drift detection baseline becomes stale | Update baseline with each training cycle |
| Alert fatigue from too many alerts | Tune thresholds; implement alert grouping |

## Estimated Effort
- **Sprint:** 9+
- **Story Points:** 34
- **Dependencies:** STORY-01, STORY-05, STORY-08