diff --git a/docs/architecture.md b/docs/architecture.md new file mode 100644 index 0000000..d9f3cac --- /dev/null +++ b/docs/architecture.md @@ -0,0 +1,200 @@ +# t-youtube — Text-Based YouTube Digest + +## Overview + +A self-hosted YouTube reader that syncs your subscriptions, generates AI summaries, and displays them in a Svelte dashboard. Designed to work around GFW restrictions in China and YouTube IP blocks on cloud providers. + +## Architecture + +``` + ┌─────────────────────────┐ + │ Oracle Tokyo VPS │ + │ (free-amd-caddy) │ + │ │ + │ ┌──────────────────┐ │ + │ │ Caddy (port 443) │ │ + │ │ reverse_proxy to │────┼─── Cloudflare ─── Internet + │ │ NAS via Tailscale │ │ + │ └──────────────────┘ │ + │ │ + │ ┌──────────────────┐ │ + │ │ Cron (every 6h) │ │ + │ │ fetch_and_sync.sh │────┼─── YouTube Data API + │ └──────────────────┘ │ + └──────────┬───────────────┘ + │ Tailscale (DERP relay) + ┌──────────┴───────────────┐ + │ NAS ds224plus │ + │ (100.78.131.124) │ + │ │ + │ ┌───────────────────┐ │ + │ │ Caddy (port 8443) │ │ + │ │ Let's Encrypt via │ │ + │ │ Cloudflare DNS │ │ + │ └───────────────────┘ │ + │ │ + │ ┌───────────────────┐ │ + │ │ Authelia (9092) │ │ + │ │ SSO + 2FA │ │ + │ └───────────────────┘ │ + │ │ + │ ┌───────────────────┐ │ + │ │ t-youtube (8001) │ │ + │ │ FastAPI + Svelte │ │ + │ │ SQLite DB │ │ + │ └───────────────────┘ │ + │ │ + │ ┌───────────────────┐ │ + │ │ Gitea (3300) │ │ + │ │ git.jimmygan.com │ │ + │ └───────────────────┘ │ + └──────────────────────────┘ +``` + +## Data Flow + +1. **Fetch**: VPS cron runs every 6h → YouTube Data API v3 (subscriptions list) + RSS feeds → new video metadata → local SQLite DB +2. **Summarize**: VPS reads new videos from DB → calls Qwen35B on MacBook (port 8085 via Tailscale) → title-only summaries → saves to DB +3. **Sync**: VPS `cat` DB → SSH to NAS → `sudo tee` → overwrites NAS DB → restarts Docker container +4. **Serve**: NAS Docker (FastAPI + Svelte) → serves dashboard to browser → Authelia SSO in front + +## Deployment + +### VPS (Oracle Tokyo) + +Files in `/tmp/yt-fetch/`: + +| File | Purpose | +|------|---------| +| `fetch_and_sync.sh` | Entry point — fetch → summarize → sync | +| `fetcher.py` | YouTube Data API + RSS hybrid fetcher | +| `summarize_new.py` | Title-based summarization via Qwen | +| `db.py` | SQLite schema | +| `auth.py` | OAuth credential management | +| `token.pickle` | YouTube OAuth token | +| `cookies.txt` | YouTube cookies (for transcript fallback) | + +Cron: `0 */6 * * * bash /tmp/yt-fetch/fetch_and_sync.sh` + +### NAS + +Docker container at `/volume1/docker/t-youtube/`: +- `docker-compose.yml` — port 8001:8000, env vars for DeepSeek/Qwen +- `.env` — `DEEPSEEK_API_KEY`, `API_TOKEN` +- `data/t_youtube.db` — SQLite database (synced from VPS) + +### MacBook + +Qwen models run via `llama-server`: +- Port 8080: Qwen3.6-27B-Q8 (27B params, ~30 t/s, 256K ctx) +- Port 8085: Qwen3.6-35B-A3B-Q8 (35B MoE, 3.5B active, faster) +- Both with `--spec-type draft-mtp` for speculative decoding + +## Configuration + +### Environment Variables + +| Variable | Default | Description | +|----------|---------|-------------| +| `DEEPSEEK_API_KEY` | — | DeepSeek API key for summarization | +| `DEEPSEEK_BASE_URL` | `https://api.deepseek.com` | API endpoint | +| `DEEPSEEK_MODEL` | `deepseek-chat` | Model name | +| `API_TOKEN` | — | Bearer token for dashboard write ops | +| `AGENT_ENABLED` | `false` | Enable agent queue worker | +| `QWEN_BASE_URL` | `http://localhost:8080` | Local Qwen endpoint | +| `HTTP_PROXY` | — | Proxy for fetcher/summarizer (China bypass) | +| `AGENT_POLL_INTERVAL` | `30` | Agent queue poll seconds | +| `AGENT_MAX_CONCURRENT` | `3` | Agent queue max workers | + +### Authelia Bypass + +API routes at `/api/*` bypass Authelia via `resources: ['^/api/']` in Authelia `access_control.yaml`. Frontend (`/`) requires SSO login. + +## API Endpoints + +| Endpoint | Method | Auth | Description | +|----------|--------|------|-------------| +| `/api/health` | GET | None | Health check | +| `/api/config` | GET | None | Auth config (token required?) | +| `/api/videos` | GET | None | List videos (status, channel, limit params) | +| `/api/videos/fetch` | POST | Bearer | Trigger subscription sync | +| `/api/videos/{id}/summarize` | POST | Bearer | Trigger single video summary | +| `/api/videos/{id}/status` | POST | Bearer | Update status (pending/read/skipped) | +| `/api/notes` | GET | None | List all notes | +| `/api/notes/{id}` | GET/POST | Bearer (POST) | Get/save note | +| `/api/notes/migrate` | POST | Bearer | Migrate localStorage notes to server | +| `/api/agent-queue` | GET/POST | Bearer (POST) | List/add agent queue items | +| `/api/channels` | GET | None | List distinct channels | +| `/api/auto_categories.json` | GET | None | Channel category mapping | + +## Sprint Plan + +### Sprint 1 ✅ (Current — Deployed) +- Agent queue worker (disabled by default) +- Proxy support for fetcher + summarizer +- Auto-mark-read on navigate away (not timer) +- VPS fetch + summarize cron +- run_tests.sh fixes +- All 2,674 videos summarized (title-based) + +### Sprint 2 — UX Overhaul (Next) +- Full-text search across video titles +- Batch select → mark read/skip +- Infinite scroll / pagination +- Display duration + view count +- Transcript accordion in detail panel + +### Sprint 3 — Polish & Monitoring +- Stats dashboard (videos/day, channel health, sync lag) +- Auto-purge old read videos (configurable TTL) +- Broken channel detection (404 on RSS) +- Cron failure alerts via Telegram bot +- Scheduled DB backup to MacBook + +### Sprint 4 — Code Quality +- Split `main.py` into routers +- Cursor/offset pagination on API +- Rate limiting middleware +- aiosqlite connection pooling + +## Known Issues + +### Transcripts Blocked +YouTube transcripts inaccessible from all our locations: +- **Oracle Cloud VPS**: YouTube blocks cloud provider IPs +- **MacBook (China)**: GFW blocks YouTube API endpoints +- **YouTube Data API captions.download**: 403 Forbidden for non-owned videos + +**Workaround**: Title + description-based summaries via Qwen. Full transcripts need a Japanese residential proxy ($3-5/month). + +### auth.jimmygan.com from LAN +Accessing `auth.jimmygan.com` from LAN shows Immich (photos.jimmygan.com) instead of Authelia. Caused by NAS port 443 routing — DSM nginx still owns port 443 and routes Immich for unmatched domains. + +### Gitea SSL +Uses Let's Encrypt **staging** certificates (acme-staging) — browsers may show untrusted cert warnings. + +## Development + +### Local Setup +```bash +cd ~/repos/t-youtube +# Backend +cd backend && python3 -m venv .venv && source .venv/bin/activate && pip install -r requirements.txt +python3 main.py # runs on :8000 + +# Frontend +cd frontend && npm install && npm run dev # runs on :5173 with API proxy +``` + +### Testing +```bash +cd ~/repos/t-youtube && bash run_tests.sh +# 23 tests across db, fetcher, summarizer, main, agent_worker +``` + +### Deploy to NAS +```bash +cd ~/repos/t-youtube +git add -A && git commit -m "..." && git push gitea main:master +ssh nas 'cd /volume1/docker/t-youtube && git pull origin master && docker compose up -d --build' +```