diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..2ff0752 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,38 @@ +# T YouTube — Text-Based YouTube Digest + +## PDD Workflow + +This project follows **PDD (Product Definition Document)** development. + +**Always read these files before writing code:** +- `docs/PDD.md` — product vision, goals, targets (the north star) +- `docs/specs/active.md` — current sprint spec (what we're building now) + +**Workflow:** +1. PDD → Sprint Spec → Implementation → Validate ACs +2. Scope changes update the sprint spec first, code second +3. Never ship without ACs passing + +## Dev Commands + +- **Build:** `docker compose up -d --build` +- **Test:** `./run_tests.sh` +- **Run locally:** `uvicorn backend.main:app --reload` (backend) + `npm run dev` (frontend) +- **Sync subscriptions:** `python run_sync.py` + +## Architecture + +- **Frontend:** Svelte 5 + Vite (`frontend/`) +- **Backend:** FastAPI + SQLite (`backend/`) +- **Deployment:** Docker container on NAS DS224+, port 8001 +- **Sync:** `run_sync.py` fetches YouTube subscriptions + transcripts via yt-dlp +- **Auth:** Authelia SSO in front, cookie-based session +- **Proxy:** YouTube blocked from China — sync runs via VPS (Oracle Tokyo) or proxy + +## Key Files + +- `docs/architecture.md` — full architecture reference +- `Dockerfile` — multi-stage build (node → python) +- `docker-compose.yml` — deployment config +- `run_sync.py` — subscription sync entry point +- `config/cookies.txt` — YouTube auth cookies (not committed) diff --git a/docs/PDD.md b/docs/PDD.md new file mode 100644 index 0000000..d1629d8 --- /dev/null +++ b/docs/PDD.md @@ -0,0 +1,21 @@ +# Product Definition Document — T YouTube + +## Vision + +A text-first YouTube experience — read transcripts, get AI summaries, never watch a video. + +## Goals + +1. **No-video YouTube** — read subscriptions, watch content as text, never open the YouTube app +2. **Daily digest** — AI-summarized subscription feed delivered to Telegram daily +3. **GFW-proof** — works reliably from China despite YouTube blocks + +## Target Users + +Self (Jimmy Gan). Primary use: morning skim of YouTube subscriptions via text summaries. + +## Success Metrics + +- Daily sync completes without errors +- Telegram digest delivered every morning +- < 5s page load for dashboard diff --git a/docs/configuration.md b/docs/configuration.md new file mode 100644 index 0000000..8473ff5 --- /dev/null +++ b/docs/configuration.md @@ -0,0 +1,152 @@ +# t-youtube Configuration Guide + +## Environment Variables + +### Backend (.env) +```bash +# Database path +DB_DIR=/volume1/docker/t-youtube/backend + +# API Token (for write operations) +API_TOKEN= + +# DeepSeek API (for summarization) +DEEPSEEK_API_KEY= + +# YouTube Data API v3 +GOOGLE_APPLICATION_CREDENTIALS=/path/to/credentials.json +``` + +### Docker Compose +```yaml +version: "3" +services: + t-youtube: + build: . + ports: + - "8001:8001" + environment: + - DB_DIR=/backend/data + - API_TOKEN= + - DEEPSEEK_API_KEY= + volumes: + - ./backend/data:/backend/data +``` + +## API Token Management + +### Generation +```bash +python3 -c "import secrets; print('api-' + secrets.token_urlsafe(32))" +``` + +### Frontend Setup +- First-time users see a token setup modal +- Token stored in localStorage +- Bearer token sent with all API write requests + +### Backend Validation +- Middleware checks `Authorization: Bearer ` for POST/PATCH/DELETE on `/api/*` +- Returns 401 JSON for invalid tokens +- GET endpoints remain open to authenticated frontend users + +## Database Management + +### Location +- **Path**: `/volume1/docker/t-youtube/backend/t_youtube.db` +- **Type**: SQLite (aiosqlite) +- **Backup**: Copy the `.db` file before any schema changes + +### Schema Migrations +- Use `sqlite3` CLI or a migration script +- Always add indexes after schema changes +- Example: + ```bash + sqlite3 t_youtube.db "CREATE INDEX IF NOT EXISTS idx_x ON table(column);" + ``` + +### Daily Sync Configuration + +#### Cron Job +```bash +# Runs every day at 3 AM +0 3 * * * /path/to/run_sync.py >> /var/log/t-youtube-sync.log 2>&1 +``` + +#### Fetcher Settings +- **RSS**: Free, unlimited, ~2000 channels +- **YouTube Data API v3**: ~50 units per sync +- **Rotation**: Fetches in slices to avoid timeouts +- **Retry**: Automatic retry on failure + +#### Summarizer Settings +- **Model**: `deepseek-chat` via DeepSeek API +- **Timeout**: 30 seconds per video +- **Fallback**: Video description if transcript unavailable +- **Language**: Follows content language (Chinese → Chinese summary) + +## Authelia Integration + +### Authentication Flow +1. User accesses protected domain +2. Caddy forwards request to Authelia (`127.0.0.1:9092`) +3. Authelia checks session/2FA +4. If no session → redirect to `auth.jimmygan.com` +5. If session exists → return 200 (allowed) + +### 2FA Methods +- **WebAuthn (Primary)**: Passkey via iCloud Keychain/Bitwarden +- **TOTP (Fallback)**: Time-based one-time password + +### Configuration +- **Address**: `127.0.0.1:9092` +- **Database**: SQLite (`/volume1/docker/authelia/config/db.sqlite3`) +- **LDAP Backend**: LLDAP (`lldap:3890`) +- **Session**: 1 hour expiration, 5 minute inactivity timeout + +### Restart Authelia +```bash +cd /volume1/docker/authelia && /usr/local/bin/docker compose restart authelia +``` + +## Caddy Configuration + +### VPS Caddy (Public) +- **Port**: 443 +- **TLS**: Let's Encrypt (HTTP-01) +- **Authelia**: `forward_auth 100.78.131.124:9092` +- **DNS**: dnsmasq → VPS Tailscale IP `100.109.198.126` + +### NAS Caddy (Local/Tailscale) +- **Port**: 8443 (fallback due to SNI bug) +- **TLS**: Cloudflare DNS-01 +- **Authelia**: `forward_auth 127.0.0.1:9092` +- **SNI Bug**: Per-site `tls` blocks cause cert routing issues +- **Workaround**: Global TLS config + dnsmasq → VPS Tailscale IP + +### Reloading Caddy +```bash +# VPS +ssh caddy-vps "sudo systemctl reload caddy" + +# NAS +cd /volume1/docker/caddy && /usr/local/bin/docker compose restart caddy +``` + +## Monitoring & Health + +### Health Check +```bash +curl -s https://yt.jimmygan.com/api/health +# Expected: {"status": "ok"} +``` + +### Sync Status +- Check `run_sync.py` logs in `/var/log/t-youtube-sync.log` +- Monitor DB for new videos/summaries +- Verify cron job is active: `crontab -l` + +### Database Size +```bash +ls -lh /volume1/docker/t-youtube/backend/t_youtube.db +``` diff --git a/docs/progress.md b/docs/progress.md new file mode 100644 index 0000000..52d4c25 --- /dev/null +++ b/docs/progress.md @@ -0,0 +1,102 @@ +# t-youtube Progress Tracking + +## Session 2026-07-09: Agent Queue, Notes Migration, Architecture Documentation + +### Completed +- [x] **Agent Queue Feature**: + - Added `agent_queue` table to SQLite DB + - Implemented `POST/GET/DELETE` endpoints in `main.py` + - Created `agent_worker.py` skeleton for processing pending items + - Frontend "Agent Queue" tab (Svelte 5) with status indicators +- [x] **Notes Migration**: + - Moved notes from localStorage to server-side DB (`video_notes` table) + - Added `POST/GET/DELETE` endpoints for notes + - Implemented non-blocking localStorage migration script +- [x] **Database Optimization**: + - Added indexes on `video_id` for `agent_queue` and `video_notes` tables +- [x] **Authelia Configuration**: + - Attempted passwordless mode (failed due to unsupported key in Authelia v4) + - Reverted to standard 2FA (WebAuthn/TOTP) +- [x] **Documentation**: + - Created `docs/` folder with `architecture.md` and `progress.md` + - Documented all API endpoints, DB schema, infrastructure, and known issues + +### In Progress +- [ ] **Agent Worker Logic**: + - `agent_worker.py` needs actual LLM/feature-creation logic + - Pending: Define workflow for processing queued items +- [ ] **YouTube Sync Monitoring**: + - Daily cron at 3 AM is active + - Pending: Monitor next run for errors/progress + +### Planned +- [ ] **Channel Categorization**: + - ~119 channels remain uncategorized + - Consider moving `AUTO_CATEGORIES` to server-side JSON config +- [ ] **Transcript Fallback**: + - Implement browser-based scraping (playwright/selenium) for the remaining 10% + - High complexity/resource usage trade-off +- [ ] **Performance Monitoring**: + - Add health check endpoints for sync status + - Implement error logging/reporting + +## Session 2026-07-05 to 2026-07-08: Initial Development + +### Key Milestones +- [x] **YouTube Data API v3 + RSS Hybrid**: Replaced yt-dlp/cookies approach + - Free, unlimited sync (~50 units/sync for 200+ channels) + - 2414 new videos discovered in first run +- [x] **AI Summarization**: DeepSeek API integration (was Claude/bytecatcode) + - Chinese content → Chinese summary; English content → English summary + - ~2.1s per video, ~$0.001 each +- [x] **Frontend Overhaul**: + - AI Digest as hero element + - Keyboard navigation (`j/k/↑↓`, `r/m`, `s`, `n`, `Enter`, `?`) + - Dark/light mode toggle + - Channel filter dropdown with auto-generated categories + - Auto-read after 3s silent mark +- [x] **Agent Queue Integration**: + - "Save to Agent" feature: notes ARE prompts for Hermes agents + - Queue tab with pending/processing/done status + - Cross-device note sync (localStorage → server DB) + +### Technical Decisions +1. **API Token Auth**: Bearer token for API writes, Authelia SSO for frontend +2. **SQLite**: Simple, reliable, no external dependencies for single-user setup +3. **DeepSeek API**: Cost-effective for summarization (~$0.001/video) +4. **Svelte 5**: Reactivity model, keyboard workflow, small bundle size +5. **RSS + YouTube Data API**: Free tier sufficient, avoids cookie/yt-dlp maintenance + +## Session 2026-07-04: NAS & Authelia Setup + +### Completed +- [x] NAS Caddy port 443 conflict resolved (DSM nginx) +- [x] dnsmasq → VPS Caddy Tailscale IP (100.109.198.126) +- [x] Authelia WebAuthn enabled, passkey registered +- [x] TOTP reset for jimmy, default 2FA set to WebAuthn +- [x] Caddy SNI bug workaround (global TLS config, port 8443 fallback) + +## Known Issues & Resolutions + +### Authelia Passwordless Failure +- **Issue**: `webauthn.passwordless: true` caused boot loop (`configuration key not expected`) +- **Root Cause**: Unsupported configuration key in current Authelia version +- **Resolution**: Reverted to standard 2FA (WebAuthn/TOTP) + +### YouTube Transcript Blocking +- **Issue**: 90% of videos have transcripts; 10% fail due to Oracle Cloud IP blocking +- **Root Cause**: YouTube IP-based restrictions +- **Resolution**: Fallback to video description summaries; consider browser-based scraping later + +### NAS Caddy SNI Bug +- **Issue**: NAS Caddy serves first cached cert for all SNI names +- **Root Cause**: Per-site `tls { ... }` blocks conflict with global auto_https policy +- **Resolution**: Global TLS config + dnsmasq → VPS Caddy Tailscale IP + +## Metrics & Performance +- **Videos in DB**: ~2414+ (initial sync) +- **Channels**: ~229 subscribed +- **Sync Time**: ~2-3 minutes for all channels +- **Summarization**: ~2.1s/video, ~$0.001 each via DeepSeek +- **Frontend Load Time**: 0.14-0.22s (API responses) +- **Daily API Usage**: ~50 units (well within 10,000/day free tier) diff --git a/docs/specs/active.md b/docs/specs/active.md new file mode 120000 index 0000000..3a78b09 --- /dev/null +++ b/docs/specs/active.md @@ -0,0 +1 @@ +sprint-001-sync-reliability.md \ No newline at end of file diff --git a/docs/specs/sprint-001-sync-reliability.md b/docs/specs/sprint-001-sync-reliability.md new file mode 100644 index 0000000..d7b2d52 --- /dev/null +++ b/docs/specs/sprint-001-sync-reliability.md @@ -0,0 +1,32 @@ +# Sprint 001 — Sync Reliability & Telegram Delivery + +**Dates:** Active + +## Goals + +- Fix daily sync on NAS +- Verify Telegram delivery pipeline +- Stabilize subscription fetching behind GFW + +## Tasks + +### T1: Fix daily sync cron + +**Description:** +`t-youtube-daily-sync` errored last run. Determine if `run_sync.py` is the active pipeline or if VPS-based `fetch_and_sync.sh` replaced it. Fix whichever is active. + +**ACs:** +- [ ] Daily sync completes without errors +- [ ] New videos appear in dashboard + +**Impl Notes:** +Check NAS cron logs. VPS sync fires every 6h. + +### T2: Verify tg-group-filter delivery + +**Description:** +Cron switched from `deliver: local` to `deliver: all`. Confirm Telegram delivery works for filtered results. + +**ACs:** +- [ ] Telegram receives filtered video digest +- [ ] No duplicate deliveries