After story 1
This commit is contained in:
+360
@@ -0,0 +1,360 @@
|
||||
# STORY-09: Observability, Monitoring & Hardening
|
||||
|
||||
## Epic
|
||||
**E4: Operations** — As a DevOps engineer, I can schedule, monitor, and resume batch jobs.
|
||||
|
||||
## Related Requirements
|
||||
| ID | Requirement |
|
||||
|----|------|
|
||||
| NFR-05 | Fault Tolerance: Auto-retry on transient failures; skip & log on fatal errors |
|
||||
| NFR-06 | Observability: Prometheus/Grafana metrics + structured logging; Tracks FPS, queue depth, confidence distribution, drift alerts |
|
||||
| NFR-01 | Throughput: ≥ 30 videos/hour/GPU |
|
||||
| NFR-03 | GPU Memory Safety: ≤ 18GB per GPU sustained |
|
||||
| NFR-04 | Determinism & Reproducibility: Config-seeded randomness, versioned models |
|
||||
| NFR-07 | Data Volume Handling: Efficient indexing for ~30TB dataset |
|
||||
|
||||
## Description
|
||||
Implement comprehensive monitoring, metrics collection, fault tolerance, and system hardening. Expose Prometheus metrics for throughput, GPU utilization, queue depth, and confidence distribution. Build Grafana dashboards for real-time system health. Implement crash recovery, idempotency, retry logic, and drift detection.
|
||||
|
||||
## Scope
|
||||
|
||||
### In Scope
|
||||
- Prometheus metrics exposure (videos processed, GPU utilization, queue depth, confidence distribution)
|
||||
- Grafana dashboard configuration (throughput, error rates, confidence drift)
|
||||
- Crash recovery with checkpointing
|
||||
- Idempotent processing guarantees
|
||||
- Retry logic for transient errors (up to 3 attempts)
|
||||
- Weekly drift detection job
|
||||
- Alerting on confidence distribution shifts and review queue growth
|
||||
- Worker health checks and auto-restart
|
||||
|
||||
### Out of Scope
|
||||
- Frame sampling (covered in STORY-04)
|
||||
- Face detection (covered in STORY-05)
|
||||
- Classification (covered in STORY-06)
|
||||
- Review UI (covered in STORY-08)
|
||||
- Active learning pipeline (covered in STORY-09)
|
||||
|
||||
## Deliverables
|
||||
|
||||
### 9.1 Prometheus Metrics
|
||||
**File:** `src/metrics.py`
|
||||
|
||||
Metrics to expose:
|
||||
- **Counter Metrics:**
|
||||
- `videos_processed_total` (label: routing_decision=MATCH|REVIEW|SKIP|UNSCANNABLE|ERROR)
|
||||
- `videos_processed_by_model_total` (label: model_version)
|
||||
- `frames_extracted_total`
|
||||
- `faces_detected_total`
|
||||
- `inference_errors_total` (label: error_type)
|
||||
- `retry_attempts_total` (label: step=probe|extract|detect|classify)
|
||||
- **Gauge Metrics:**
|
||||
- `gpu_utilization_percent` (label: gpu_id)
|
||||
- `gpu_memory_used_bytes` (label: gpu_id)
|
||||
- `gpu_memory_free_bytes` (label: gpu_id)
|
||||
- `queue_depth_pending`
|
||||
- `queue_depth_processing`
|
||||
- `queue_depth_review`
|
||||
- `scratch_usage_bytes`
|
||||
- `scratch_usage_percent`
|
||||
- **Histogram Metrics:**
|
||||
- `video_processing_duration_seconds` (label: routing_decision)
|
||||
- `confidence_score_distribution` (buckets: 0.0, 0.1, 0.2, ..., 0.9, 1.0)
|
||||
- `frame_count_per_video`
|
||||
- `face_count_per_video`
|
||||
- **Summary Metrics:**
|
||||
- `throughput_videos_per_hour` (calculated from counter)
|
||||
- `average_confidence` (calculated from histogram)
|
||||
|
||||
Implementation:
|
||||
```python
|
||||
from prometheus_client import start_http_server, Counter, Gauge, Histogram
|
||||
|
||||
# Start metrics HTTP server
|
||||
start_http_server(9090) # /metrics endpoint
|
||||
|
||||
# Define metrics
|
||||
videos_processed = Counter(
|
||||
'videos_processed_total',
|
||||
'Total videos processed',
|
||||
['routing_decision']
|
||||
)
|
||||
|
||||
gpu_memory = Gauge(
|
||||
'gpu_memory_used_bytes',
|
||||
'GPU memory usage',
|
||||
['gpu_id']
|
||||
)
|
||||
|
||||
confidence_hist = Histogram(
|
||||
'confidence_score_distribution',
|
||||
'Video confidence scores',
|
||||
buckets=[0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0]
|
||||
)
|
||||
```
|
||||
|
||||
### 9.2 Grafana Dashboards
|
||||
**File:** `monitoring/grafana/dashboards/`
|
||||
|
||||
Dashboard 1: **System Overview**
|
||||
- Panels:
|
||||
- Throughput (videos/hour) — line chart, 1h window
|
||||
- Queue depth (pending/processing/review) — stacked bar
|
||||
- GPU utilization (both GPUs) — line chart
|
||||
- GPU memory usage (both GPUs) — line chart
|
||||
- Error rate (errors/100 videos) — bar chart
|
||||
- Confidence distribution — histogram
|
||||
|
||||
Dashboard 2: **Processing Details**
|
||||
- Panels:
|
||||
- Processing duration per video — scatter plot
|
||||
- Frame count per video — histogram
|
||||
- Face count per video — histogram
|
||||
- Confidence score by model version — box plot
|
||||
- Routing decision distribution — pie chart
|
||||
|
||||
Dashboard 3: **Active Learning**
|
||||
- Panels:
|
||||
- Review queue size over time — line chart
|
||||
- Annotation rate (labels/day) — bar chart
|
||||
- Model version timeline — timeline panel
|
||||
- F1 score by model version — line chart
|
||||
- ECE by model version — line chart
|
||||
|
||||
**File:** `monitoring/grafana/dashboards/system_overview.json`
|
||||
**File:** `monitoring/grafana/dashboards/processing_details.json`
|
||||
**File:** `monitoring/grafana/dashboards/active_learning.json`
|
||||
|
||||
### 9.3 Crash Recovery
|
||||
**File:** `src/crash_recovery.py`
|
||||
|
||||
Features:
|
||||
- **Checkpointing:** Periodically save processing state
|
||||
```python
|
||||
def save_checkpoint(video_id, state, progress):
|
||||
checkpoint = {
|
||||
'video_id': video_id,
|
||||
'state': state, # 'processing', 'extracting', 'detecting', 'classifying'
|
||||
'progress': progress, # dict of step -> completed
|
||||
'timestamp': datetime.utcnow().isoformat()
|
||||
}
|
||||
with open(f'/scratch/checkpoints/{video_id}.json', 'w') as f:
|
||||
json.dump(checkpoint, f)
|
||||
```
|
||||
- **Recovery on Startup:** Scan for PROCESSING videos and re-queue them
|
||||
```sql
|
||||
UPDATE videos SET status = 'PENDING', updated_at = NOW()
|
||||
WHERE status = 'PROCESSING' AND updated_at < NOW() - INTERVAL 5 MINUTE
|
||||
```
|
||||
- **Lock Timeout:** 5 minutes (videos stuck in PROCESSING beyond this are re-queued)
|
||||
- **Re-queue Logic:** Only re-queue if worker is down (detected via health check)
|
||||
- **Idempotency:** Re-processing a video produces same results (no duplicates)
|
||||
|
||||
### 9.4 Retry Logic
|
||||
**File:** `src/retry.py`
|
||||
|
||||
Features:
|
||||
- **Retry Decorator:**
|
||||
```python
|
||||
def retry(max_attempts=3, delay=1.0, backoff=2.0, exceptions=(Exception,)):
|
||||
def decorator(func):
|
||||
def wrapper(*args, **kwargs):
|
||||
last_exception = None
|
||||
for attempt in range(max_attempts):
|
||||
try:
|
||||
return func(*args, **kwargs)
|
||||
except exceptions as e:
|
||||
last_exception = e
|
||||
if attempt < max_attempts - 1:
|
||||
wait = delay * (backoff ** attempt)
|
||||
time.sleep(wait)
|
||||
raise last_exception
|
||||
return wrapper
|
||||
return decorator
|
||||
```
|
||||
- **Applicable Steps:** ffprobe, frame extraction, face detection, classification, DB writes
|
||||
- **Transient Errors:** Network timeout, GPU OOM, file lock contention
|
||||
- **Non-Retryable Errors:** Codec unsupported, corrupt file, invalid path → skip and log
|
||||
- **Retry Logging:** Log each retry attempt with error type and delay
|
||||
|
||||
### 9.5 Drift Detection
|
||||
**File:** `src/drift_detector.py`
|
||||
|
||||
Features:
|
||||
- **Weekly Job:** Compare current confidence distribution to baseline
|
||||
```python
|
||||
def detect_drift(current_confidences, baseline_confidences, threshold=0.10):
|
||||
# Compare p(C > 0.5) shift
|
||||
current_high = sum(1 for c in current_confidences if c > 0.5) / len(current_confidences)
|
||||
baseline_high = sum(1 for c in baseline_confidences if c > 0.5) / len(baseline_confidences)
|
||||
shift = abs(current_high - baseline_high)
|
||||
|
||||
if shift > threshold:
|
||||
alert(f"Confidence drift detected: {shift:.2%} shift in p(C > 0.5)")
|
||||
return True
|
||||
return False
|
||||
```
|
||||
- **Baseline:** Stored in DB or config (computed from last training cycle)
|
||||
- **Alert Conditions:**
|
||||
- `p(C > 0.5)` shifts > 10% from baseline
|
||||
- Review queue grows unbounded (> 1000 items for > 24 hours)
|
||||
- Throughput drops below 20 videos/hour/GPU for > 1 hour
|
||||
- Error rate exceeds 5% for any 1-hour window
|
||||
- **Alert Channels:** Email, Slack webhook, or log entry (configurable)
|
||||
|
||||
### 9.6 Worker Health Checks
|
||||
**File:** `src/health_check.py`
|
||||
|
||||
Features:
|
||||
- **Health Endpoint:** `/health` returns worker status
|
||||
```json
|
||||
{
|
||||
"status": "healthy",
|
||||
"gpu_available": true,
|
||||
"gpu_memory_used_gb": 12.5,
|
||||
"queue_depth": 42,
|
||||
"uptime_seconds": 86400,
|
||||
"videos_processed_today": 156,
|
||||
"last_error": null
|
||||
}
|
||||
```
|
||||
- **Auto-Restart:** Docker restart policy for worker container
|
||||
```yaml
|
||||
restart: unless-stopped
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "-f", "http://localhost:9090/health"]
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 3
|
||||
```
|
||||
- **GPU Health:** Monitor GPU temperature and error counts
|
||||
- **Disk Health:** Monitor scratch space usage and NAS connectivity
|
||||
|
||||
### 9.7 Docker Compose Update
|
||||
**File:** `docker-compose.yml` (update)
|
||||
|
||||
Add monitoring services:
|
||||
```yaml
|
||||
prometheus:
|
||||
image: prom/prometheus:v2.48.0
|
||||
volumes:
|
||||
- ./monitoring/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
|
||||
- prometheus_data:/prometheus
|
||||
ports:
|
||||
- "9090:9090"
|
||||
networks:
|
||||
- videodetect-network
|
||||
|
||||
grafana:
|
||||
image: grafana/grafana:10.2.0
|
||||
volumes:
|
||||
- grafana_data:/var/lib/grafana
|
||||
- ./monitoring/grafana/dashboards:/etc/grafana/provisioning/dashboards
|
||||
environment:
|
||||
- GF_SECURITY_ADMIN_PASSWORD=${GRAFANA_PASSWORD}
|
||||
ports:
|
||||
- "3000:3000"
|
||||
networks:
|
||||
- videodetect-network
|
||||
|
||||
volumes:
|
||||
prometheus_data:
|
||||
grafana_data:
|
||||
```
|
||||
|
||||
### 9.8 Configuration Updates
|
||||
**File:** `config.yaml` (updates)
|
||||
|
||||
New fields:
|
||||
```yaml
|
||||
monitoring:
|
||||
prometheus:
|
||||
enabled: true
|
||||
port: 9090
|
||||
metrics_path: /metrics
|
||||
grafana:
|
||||
enabled: true
|
||||
port: 3000
|
||||
alerts:
|
||||
confidence_drift_threshold: 0.10
|
||||
review_queue_max_size: 1000
|
||||
review_queue_max_age_hours: 24
|
||||
throughput_min_videos_per_hour: 20
|
||||
throughput_min_duration_hours: 1
|
||||
error_rate_threshold: 0.05
|
||||
error_rate_window_hours: 1
|
||||
drift_detection:
|
||||
enabled: true
|
||||
schedule: weekly # cron: 0 2 * * 0 (Sundays at 2 AM)
|
||||
baseline_source: db # db or config
|
||||
crash_recovery:
|
||||
lock_timeout_minutes: 5
|
||||
auto_requeue: true
|
||||
retry:
|
||||
max_attempts: 3
|
||||
initial_delay: 1.0
|
||||
backoff_factor: 2.0
|
||||
retryable_errors:
|
||||
- timeout
|
||||
- gpu_oom
|
||||
- file_lock
|
||||
non_retryable_errors:
|
||||
- codec_unsupported
|
||||
- file_corrupt
|
||||
- invalid_path
|
||||
```
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
### Functional
|
||||
- [ ] Prometheus metrics are exposed at `/metrics` endpoint and queryable
|
||||
- [ ] All required metrics are present (counters, gauges, histograms, summaries)
|
||||
- [ ] Grafana dashboards load and display correct data
|
||||
- [ ] System Overview dashboard shows throughput, queue depth, GPU metrics
|
||||
- [ ] Processing Details dashboard shows duration, frame count, confidence distribution
|
||||
- [ ] Active Learning dashboard shows review queue, annotation rate, model versions
|
||||
- [ ] Crash recovery re-queues stuck PROCESSING videos on worker restart
|
||||
- [ ] Idempotent processing: re-processing produces same results (no duplicates)
|
||||
- [ ] Retry logic retries transient errors up to 3 times with exponential backoff
|
||||
- [ ] Non-retryable errors are skipped and logged (no infinite retry)
|
||||
- [ ] Drift detection runs weekly and alerts on > 10% confidence shift
|
||||
- [ ] Review queue growth alert triggers when queue > 1000 for > 24 hours
|
||||
- [ ] Worker health check returns correct status
|
||||
- [ ] Auto-restart triggers on health check failure
|
||||
|
||||
### Non-Functional
|
||||
- [ ] Metrics collection overhead < 2% of CPU
|
||||
- [ ] Grafana dashboard loads in < 3 seconds
|
||||
- [ ] Crash recovery completes in < 30 seconds
|
||||
- [ ] Drift detection completes in < 5 minutes
|
||||
- [ ] Alert delivery completes in < 60 seconds
|
||||
- [ ] Prometheus data retention: 30 days (configurable)
|
||||
- [ ] Grafana data source refresh: 30 seconds
|
||||
|
||||
### Technical Constraints
|
||||
- [ ] Prometheus metrics follow naming conventions (unit suffixes, proper labels)
|
||||
- [ ] Grafana dashboards are JSON-exportable and version-controlled
|
||||
- [ ] Crash recovery is idempotent (running twice produces same result)
|
||||
- [ ] Retry logic does not retry non-transient errors
|
||||
- [ ] Drift detection baseline is stored and versioned
|
||||
- [ ] Health check endpoint responds in < 1 second
|
||||
- [ ] All alerts are logged with timestamp and context
|
||||
|
||||
## Dependencies
|
||||
- **Prerequisites:** STORY-01 (Foundation), STORY-05 (Classification — provides metrics data), STORY-08 (Active Learning — provides model metrics)
|
||||
- **Depends on:** None (can be built in parallel with other stories)
|
||||
- **Enables:** Production deployment and long-term operation
|
||||
|
||||
## Risks & Mitigations
|
||||
| Risk | Mitigation |
|
||||
|------|--|
|
||||
| Prometheus metrics cardinality explosion | Limit label cardinality; use histograms instead of individual values |
|
||||
| Grafana dashboard load time with large datasets | Use Prometheus aggregations; pre-compute panels |
|
||||
| Crash recovery misses in-flight writes | Use DB transactions; lock timeout prevents permanent locks |
|
||||
| Drift detection baseline becomes stale | Update baseline with each training cycle |
|
||||
| Alert fatigue from too many alerts | Tune thresholds; implement alert grouping |
|
||||
|
||||
## Estimated Effort
|
||||
- **Sprint:** 9+
|
||||
- **Story Points:** 34
|
||||
- **Dependencies:** STORY-01, STORY-05, STORY-08
|
||||
Reference in New Issue
Block a user