nas-tools/specs/sprint-05-operations-resilience.md

Sprint 05 — Operations & Resilience

Depends on: S04 (CI must be reliable before automating ops) Duration: ~12h Goal: Add health checks, resource limits, log rotation, fix Immich uploads, back up dashboard data, add monitoring endpoint.

---

S05.1 — Add HEALTHCHECK to claude-dev image

• Files: claude-dev/Dockerfile, claude-dev/docker-compose.yml

• Estimate: 1h

• Done means: Dockerfile has HEALTHCHECK --interval=30s --timeout=5s --retries=3 CMD claude --version || exit 1. docker ps shows healthy.

• Verify by:

  1. Deploy claude-dev → docker ps shows "(healthy)"
  2. Break claude binary → container shows "(unhealthy)" after 3 failures

• Risk: claude --version may require network access. Test on NAS first. Alternative: pgrep -f claude or simple curl localhost:<port>.

• S05.1 — Add HEALTHCHECK to claude-dev image

S05.2 — Add resource limits to production containers

• Files: claude-dev/docker-compose.yml, dashboard/docker-compose.yml

• Estimate: 1.5h

• Done means: Production compose files have mem_limit, cpus, and restart_policy. Limits are generous: 2GB dashboard, 1GB claude-dev, 0.5GB sidecars.

• Verify by:

  1. docker stats shows containers respecting limits
  2. Normal operation unaffected
  3. docker compose up -d applies limits without restarting

• Risk: Limits too tight → OOM kills. Start generous, adjust after a week of monitoring.

• S05.2 — Add resource limits to production containers

S05.3 — Implement audit log rotation

• Files: dashboard/backend/main.py:161,171-188

• Estimate: 2h

• Done means: Audit logging uses RotatingFileHandler with max 10MB and 5 backups.

• Verify by:

  1. Write 11MB of audit entries → file rotates, audit.log.1 created
  2. Original audit.log starts fresh
  3. Old backups capped at 5 files

• Risk: Log format change may affect parsing in system.py:82-116. Test the log reader with rotated files.

• S05.3 — Implement audit log rotation

S05.4 — Fix Immich ML model downloads

• Files: immich/docker-compose.yml, immich/.env

• Estimate: 3h (investigation + fix)

• Done means: Immich mobile upload works. ML models download successfully or ML is gracefully disabled.

• Verify by:

  1. Upload photo from phone → succeeds (no timeout)
  2. ssh nas "docker logs immich_machine_learning" → no "Failed to load detection model" errors
  3. If ML disabled: photo upload works, smart search unavailable (acceptable)

• Risk: Most complex item. May require pre-downloading models, configuring mirror, increasing timeouts, or disabling ML temporarily. Environment-specific (China network to modelscope.cn).

• S05.4 — Fix Immich ML model downloads

S05.5 — Add dashboard data to backup script

• Files: backup.sh

• Estimate: 1.5h

• Done means: backup.sh copies opc.db, auth.json, rbac.json, and audit log to backup tarball. Restore procedure documented.

• Verify by:

  1. Run backup.sh → tarball contains dashboard data files
  2. Extract tarball → files are valid (SQLite DB opens, JSON parses)

• Risk: auth.json contains passkey data — ensure backup stored securely.

• S05.5 — Add dashboard data to backup script

S05.6 — Add minimal health check endpoint

• Files: dashboard/backend/main.py or routers/health.py

• Estimate: 1h

• Done means: GET /api/health returns {"status":"ok","db":true,"docker":true,"disk":{"free_gb":123}}. Non-200 triggers optional Telegram notification.

• Verify by:

  1. All services healthy → /api/health returns 200
  2. Docker socket unreachable → returns 503 with docker: false
  3. Cron job calls /api/health every 5 minutes

• Risk: Health endpoint must be lightweight. Cache results for 30 seconds.

• S05.6 — Add minimal health check endpoint

---

Exit criteria

• docker ps shows claude-dev as "(healthy)"
• Production containers have CPU/memory limits in compose
• Audit log rotates at 10MB with 5 backups
• Photo upload from phone succeeds
• backup.sh includes dashboard data files
• GET /api/health returns component statuses
• All backend tests pass
• All frontend tests pass