t-youtube/docs/architecture.md

t-youtube — Text-Based YouTube Digest

Overview

A self-hosted YouTube reader that syncs your subscriptions, generates AI summaries, and displays them in a Svelte dashboard. Designed to work around GFW restrictions in China and YouTube IP blocks on cloud providers.

Architecture

                         ┌─────────────────────────┐
                         │   Oracle Tokyo VPS       │
                         │  (free-amd-caddy)        │
                         │                          │
                         │  ┌──────────────────┐    │
                         │  │ Caddy (port 443)  │    │
                         │  │ reverse_proxy to  │────┼─── Cloudflare ─── Internet
                         │  │ NAS via Tailscale │    │
                         │  └──────────────────┘    │
                         │                          │
                         │  ┌──────────────────┐    │
                         │  │ Cron (every 6h)   │    │
                         │  │ fetch_and_sync.sh │────┼─── YouTube Data API
                         │  └──────────────────┘    │
                         └──────────┬───────────────┘
                                    │ Tailscale (DERP relay)
                         ┌──────────┴───────────────┐
                         │   NAS ds224plus           │
                         │  (100.78.131.124)         │
                         │                           │
                         │  ┌───────────────────┐   │
                         │  │ Caddy (port 8443)  │   │
                         │  │ Let's Encrypt via  │   │
                         │  │ Cloudflare DNS     │   │
                         │  └───────────────────┘   │
                         │                           │
                         │  ┌───────────────────┐   │
                         │  │ Authelia (9092)    │   │
                         │  │ SSO + 2FA          │   │
                         │  └───────────────────┘   │
                         │                           │
                         │  ┌───────────────────┐   │
                         │  │ t-youtube (8001)  │   │
                         │  │ FastAPI + Svelte  │   │
                         │  │ SQLite DB          │   │
                         │  └───────────────────┘   │
                         │                           │
                         │  ┌───────────────────┐   │
                         │  │ Gitea (3300)       │   │
                         │  │ git.jimmygan.com   │   │
                         │  └───────────────────┘   │
                         └──────────────────────────┘

Data Flow

1. Fetch: VPS cron runs every 6h → YouTube Data API v3 (subscriptions list) + RSS feeds → new video metadata → local SQLite DB
2. Summarize: VPS reads new videos from DB → calls Qwen35B on MacBook (port 8085 via Tailscale) → title-only summaries → saves to DB
3. Sync: VPS cat DB → SSH to NAS → sudo tee → overwrites NAS DB → restarts Docker container
4. Serve: NAS Docker (FastAPI + Svelte) → serves dashboard to browser → Authelia SSO in front

Deployment

VPS (Oracle Tokyo)

Files in /tmp/yt-fetch/:

File	Purpose
fetch_and_sync.sh	Entry point — fetch → summarize → sync
fetcher.py	YouTube Data API + RSS hybrid fetcher
summarize_new.py	Title-based summarization via Qwen
db.py	SQLite schema
auth.py	OAuth credential management
token.pickle	YouTube OAuth token
cookies.txt	YouTube cookies (for transcript fallback)

Cron: 0 */6 * * * bash /tmp/yt-fetch/fetch_and_sync.sh

NAS

Docker container at /volume1/docker/t-youtube/:

• docker-compose.yml — port 8001:8000, env vars for DeepSeek/Qwen
• .env — DEEPSEEK_API_KEY, API_TOKEN
• data/t_youtube.db — SQLite database (synced from VPS)

MacBook

Qwen models run via llama-server:

• Port 8080: Qwen3.6-27B-Q8 (27B params, ~30 t/s, 256K ctx)
• Port 8085: Qwen3.6-35B-A3B-Q8 (35B MoE, 3.5B active, faster)
• Both with --spec-type draft-mtp for speculative decoding

Configuration

Environment Variables

Variable	Default	Description
DEEPSEEK_API_KEY	—	DeepSeek API key for summarization
DEEPSEEK_BASE_URL	https://api.deepseek.com	API endpoint
DEEPSEEK_MODEL	deepseek-chat	Model name
API_TOKEN	—	Bearer token for dashboard write ops
AGENT_ENABLED	false	Enable agent queue worker
QWEN_BASE_URL	http://localhost:8080	Local Qwen endpoint
HTTP_PROXY	—	Proxy for fetcher/summarizer (China bypass)
AGENT_POLL_INTERVAL	30	Agent queue poll seconds
AGENT_MAX_CONCURRENT	3	Agent queue max workers

Authelia Bypass

API routes at /api/* bypass Authelia via resources: ['^/api/'] in Authelia access_control.yaml. Frontend (/) requires SSO login.

API Endpoints

Endpoint	Method	Auth	Description
/api/health	GET	None	Health check
/api/config	GET	None	Auth config (token required?)
/api/videos	GET	None	List videos (status, channel, limit params)
/api/videos/fetch	POST	Bearer	Trigger subscription sync
/api/videos/{id}/summarize	POST	Bearer	Trigger single video summary
/api/videos/{id}/status	POST	Bearer	Update status (pending/read/skipped)
/api/notes	GET	None	List all notes
/api/notes/{id}	GET/POST	Bearer (POST)	Get/save note
/api/notes/migrate	POST	Bearer	Migrate localStorage notes to server
/api/agent-queue	GET/POST	Bearer (POST)	List/add agent queue items
/api/channels	GET	None	List distinct channels
/api/auto_categories.json	GET	None	Channel category mapping

Sprint Plan

Sprint 1 ✅ (Current — Deployed)

• Agent queue worker (disabled by default)
• Proxy support for fetcher + summarizer
• Auto-mark-read on navigate away (not timer)
• VPS fetch + summarize cron
• run_tests.sh fixes
• All 2,674 videos summarized (title-based)

Sprint 2 — UX Overhaul (Next)

• Full-text search across video titles
• Batch select → mark read/skip
• Infinite scroll / pagination
• Display duration + view count
• Transcript accordion in detail panel

Sprint 3 — Polish & Monitoring

• Stats dashboard (videos/day, channel health, sync lag)
• Auto-purge old read videos (configurable TTL)
• Broken channel detection (404 on RSS)
• Cron failure alerts via Telegram bot
• Scheduled DB backup to MacBook

Sprint 4 — Code Quality

• Split main.py into routers
• Cursor/offset pagination on API
• Rate limiting middleware
• aiosqlite connection pooling

Known Issues

Transcripts Blocked

YouTube transcripts inaccessible from all our locations:

• Oracle Cloud VPS: YouTube blocks cloud provider IPs
• MacBook (China): GFW blocks YouTube API endpoints
• YouTube Data API captions.download: 403 Forbidden for non-owned videos

Workaround: Title + description-based summaries via Qwen. Full transcripts need a Japanese residential proxy ($3-5/month).

auth.jimmygan.com from LAN

Accessing auth.jimmygan.com from LAN shows Immich (photos.jimmygan.com) instead of Authelia. Caused by NAS port 443 routing — DSM nginx still owns port 443 and routes Immich for unmatched domains.

Gitea SSL

Uses Let's Encrypt staging certificates (acme-staging) — browsers may show untrusted cert warnings.

Development

Local Setup

cd ~/repos/t-youtube
# Backend
cd backend && python3 -m venv .venv && source .venv/bin/activate && pip install -r requirements.txt
python3 main.py  # runs on :8000

# Frontend
cd frontend && npm install && npm run dev  # runs on :5173 with API proxy

Testing

cd ~/repos/t-youtube && bash run_tests.sh
# 23 tests across db, fetcher, summarizer, main, agent_worker

Deploy to NAS

cd ~/repos/t-youtube
git add -A && git commit -m "..." && git push gitea main:master
ssh nas 'cd /volume1/docker/t-youtube && git pull origin master && docker compose up -d --build'