t-youtube/docs/architecture.md

# t-youtube — Text-Based YouTube Digest

## Overview

A self-hosted YouTube reader that syncs your subscriptions, generates AI summaries, and displays them in a Svelte dashboard. Designed to work around GFW restrictions in China and YouTube IP blocks on cloud providers.

## Architecture

```
                         ┌─────────────────────────┐
                         │   Oracle Tokyo VPS       │
                         │  (free-amd-caddy)        │
                         │                          │
                         │  ┌──────────────────┐    │
                         │  │ Caddy (port 443)  │    │
                         │  │ reverse_proxy to  │────┼─── Cloudflare ─── Internet
                         │  │ NAS via Tailscale │    │
                         │  └──────────────────┘    │
                         │                          │
                         │  ┌──────────────────┐    │
                         │  │ Cron (every 6h)   │    │
                         │  │ fetch_and_sync.sh │────┼─── YouTube Data API
                         │  └──────────────────┘    │
                         └──────────┬───────────────┘
                                    │ Tailscale (DERP relay)
                         ┌──────────┴───────────────┐
                         │   NAS ds224plus           │
                         │  (100.78.131.124)         │
                         │                           │
                         │  ┌───────────────────┐   │
                         │  │ Caddy (port 8443)  │   │
                         │  │ Let's Encrypt via  │   │
                         │  │ Cloudflare DNS     │   │
                         │  └───────────────────┘   │
                         │                           │
                         │  ┌───────────────────┐   │
                         │  │ Authelia (9092)    │   │
                         │  │ SSO + 2FA          │   │
                         │  └───────────────────┘   │
                         │                           │
                         │  ┌───────────────────┐   │
                         │  │ t-youtube (8001)  │   │
                         │  │ FastAPI + Svelte  │   │
                         │  │ SQLite DB          │   │
                         │  └───────────────────┘   │
                         │                           │
                         │  ┌───────────────────┐   │
                         │  │ Gitea (3300)       │   │
                         │  │ git.jimmygan.com   │   │
                         │  └───────────────────┘   │
                         └──────────────────────────┘
```

## Data Flow

1. **Fetch**: VPS cron runs every 6h → YouTube Data API v3 (subscriptions list) + RSS feeds → new video metadata → local SQLite DB
2. **Summarize**: VPS reads new videos from DB → calls Qwen35B on MacBook (port 8085 via Tailscale) → title-only summaries → saves to DB
3. **Sync**: VPS `cat` DB → SSH to NAS → `sudo tee` → overwrites NAS DB → restarts Docker container
4. **Serve**: NAS Docker (FastAPI + Svelte) → serves dashboard to browser → Authelia SSO in front

## Deployment

### VPS (Oracle Tokyo)

Files in `/tmp/yt-fetch/`:

| File | Purpose |
|------|---------|
| `fetch_and_sync.sh` | Entry point — fetch → summarize → sync |
| `fetcher.py` | YouTube Data API + RSS hybrid fetcher |
| `summarize_new.py` | Title-based summarization via Qwen |
| `db.py` | SQLite schema |
| `auth.py` | OAuth credential management |
| `token.pickle` | YouTube OAuth token |
| `cookies.txt` | YouTube cookies (for transcript fallback) |

Cron: `0 */6 * * * bash /tmp/yt-fetch/fetch_and_sync.sh`

### NAS

Docker container at `/volume1/docker/t-youtube/`:
- `docker-compose.yml` — port 8001:8000, env vars for DeepSeek/Qwen
- `.env` — `DEEPSEEK_API_KEY`, `API_TOKEN`
- `data/t_youtube.db` — SQLite database (synced from VPS)

### MacBook

Qwen models run via `llama-server`:
- Port 8080: Qwen3.6-27B-Q8 (27B params, ~30 t/s, 256K ctx)
- Port 8085: Qwen3.6-35B-A3B-Q8 (35B MoE, 3.5B active, faster)
- Both with `--spec-type draft-mtp` for speculative decoding

## Configuration

### Environment Variables

| Variable | Default | Description |
|----------|---------|-------------|
| `DEEPSEEK_API_KEY` | — | DeepSeek API key for summarization |
| `DEEPSEEK_BASE_URL` | `https://api.deepseek.com` | API endpoint |
| `DEEPSEEK_MODEL` | `deepseek-chat` | Model name |
| `API_TOKEN` | — | Bearer token for dashboard write ops |
| `AGENT_ENABLED` | `false` | Enable agent queue worker |
| `QWEN_BASE_URL` | `http://localhost:8080` | Local Qwen endpoint |
| `HTTP_PROXY` | — | Proxy for fetcher/summarizer (China bypass) |
| `AGENT_POLL_INTERVAL` | `30` | Agent queue poll seconds |
| `AGENT_MAX_CONCURRENT` | `3` | Agent queue max workers |

### Authelia Bypass

API routes at `/api/*` bypass Authelia via `resources: ['^/api/']` in Authelia `access_control.yaml`. Frontend (`/`) requires SSO login.

## API Endpoints

| Endpoint | Method | Auth | Description |
|----------|--------|------|-------------|
| `/api/health` | GET | None | Health check |
| `/api/config` | GET | None | Auth config (token required?) |
| `/api/videos` | GET | None | List videos (status, channel, limit params) |
| `/api/videos/fetch` | POST | Bearer | Trigger subscription sync |
| `/api/videos/{id}/summarize` | POST | Bearer | Trigger single video summary |
| `/api/videos/{id}/status` | POST | Bearer | Update status (pending/read/skipped) |
| `/api/notes` | GET | None | List all notes |
| `/api/notes/{id}` | GET/POST | Bearer (POST) | Get/save note |
| `/api/notes/migrate` | POST | Bearer | Migrate localStorage notes to server |
| `/api/agent-queue` | GET/POST | Bearer (POST) | List/add agent queue items |
| `/api/channels` | GET | None | List distinct channels |
| `/api/auto_categories.json` | GET | None | Channel category mapping |

## Sprint Plan

### Sprint 1 ✅ (Current — Deployed)
- Agent queue worker (disabled by default)
- Proxy support for fetcher + summarizer
- Auto-mark-read on navigate away (not timer)
- VPS fetch + summarize cron
- run_tests.sh fixes
- All 2,674 videos summarized (title-based)

### Sprint 2 — UX Overhaul (Next)
- Full-text search across video titles
- Batch select → mark read/skip
- Infinite scroll / pagination
- Display duration + view count
- Transcript accordion in detail panel

### Sprint 3 — Polish & Monitoring
- Stats dashboard (videos/day, channel health, sync lag)
- Auto-purge old read videos (configurable TTL)
- Broken channel detection (404 on RSS)
- Cron failure alerts via Telegram bot
- Scheduled DB backup to MacBook

### Sprint 4 — Code Quality
- Split `main.py` into routers
- Cursor/offset pagination on API
- Rate limiting middleware
- aiosqlite connection pooling

## Known Issues

### Transcripts Blocked
YouTube transcripts inaccessible from all our locations:
- **Oracle Cloud VPS**: YouTube blocks cloud provider IPs
- **MacBook (China)**: GFW blocks YouTube API endpoints
- **YouTube Data API captions.download**: 403 Forbidden for non-owned videos

**Workaround**: Title + description-based summaries via Qwen. Full transcripts need a Japanese residential proxy ($3-5/month).

### auth.jimmygan.com from LAN
Accessing `auth.jimmygan.com` from LAN shows Immich (photos.jimmygan.com) instead of Authelia. Caused by NAS port 443 routing — DSM nginx still owns port 443 and routes Immich for unmatched domains.

### Gitea SSL
Uses Let's Encrypt **staging** certificates (acme-staging) — browsers may show untrusted cert warnings.

## Development

### Local Setup
```bash
cd ~/repos/t-youtube
# Backend
cd backend && python3 -m venv .venv && source .venv/bin/activate && pip install -r requirements.txt
python3 main.py  # runs on :8000

# Frontend
cd frontend && npm install && npm run dev  # runs on :5173 with API proxy
```

### Testing
```bash
cd ~/repos/t-youtube && bash run_tests.sh
# 23 tests across db, fetcher, summarizer, main, agent_worker
```

### Deploy to NAS
```bash
cd ~/repos/t-youtube
git add -A && git commit -m "..." && git push gitea main:master
ssh nas 'cd /volume1/docker/t-youtube && git pull origin master && docker compose up -d --build'
```