jimmy/t-youtube
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: architecture, sprints, and deployment guide

Browse Source
This commit is contained in:
Gan, Jimmy
2026-07-09 23:45:41 +08:00
parent cc9a4487ac
commit 2cd0ba13dd
1 changed files with 200 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/architecture.md
+200
View File
@@ -0,0 +1,200 @@
# t-youtube — Text-Based YouTube Digest
## Overview
A self-hosted YouTube reader that syncs your subscriptions, generates AI summaries, and displays them in a Svelte dashboard. Designed to work around GFW restrictions in China and YouTube IP blocks on cloud providers.
## Architecture
```
┌─────────────────────────┐
│ Oracle Tokyo VPS │
│ (free-amd-caddy) │
│ │
│ ┌──────────────────┐ │
│ │ Caddy (port 443) │ │
│ │ reverse_proxy to │────┼─── Cloudflare ─── Internet
│ │ NAS via Tailscale │ │
│ └──────────────────┘ │
│ │
│ ┌──────────────────┐ │
│ │ Cron (every 6h) │ │
│ │ fetch_and_sync.sh │────┼─── YouTube Data API
│ └──────────────────┘ │
└──────────┬───────────────┘
│ Tailscale (DERP relay)
┌──────────┴───────────────┐
│ NAS ds224plus │
│ (100.78.131.124) │
│ │
│ ┌───────────────────┐ │
│ │ Caddy (port 8443) │ │
│ │ Let's Encrypt via │ │
│ │ Cloudflare DNS │ │
│ └───────────────────┘ │
│ │
│ ┌───────────────────┐ │
│ │ Authelia (9092) │ │
│ │ SSO + 2FA │ │
│ └───────────────────┘ │
│ │
│ ┌───────────────────┐ │
│ │ t-youtube (8001) │ │
│ │ FastAPI + Svelte │ │
│ │ SQLite DB │ │
│ └───────────────────┘ │
│ │
│ ┌───────────────────┐ │
│ │ Gitea (3300) │ │
│ │ git.jimmygan.com │ │
│ └───────────────────┘ │
└──────────────────────────┘
```
## Data Flow
1. **Fetch**: VPS cron runs every 6h → YouTube Data API v3 (subscriptions list) + RSS feeds → new video metadata → local SQLite DB
2. **Summarize**: VPS reads new videos from DB → calls Qwen35B on MacBook (port 8085 via Tailscale) → title-only summaries → saves to DB
3. **Sync**: VPS `cat` DB → SSH to NAS → `sudo tee` → overwrites NAS DB → restarts Docker container
4. **Serve**: NAS Docker (FastAPI + Svelte) → serves dashboard to browser → Authelia SSO in front
## Deployment
### VPS (Oracle Tokyo)
Files in `/tmp/yt-fetch/`:
| File | Purpose |
|------|---------|
| `fetch_and_sync.sh` | Entry point — fetch → summarize → sync |
| `fetcher.py` | YouTube Data API + RSS hybrid fetcher |
| `summarize_new.py` | Title-based summarization via Qwen |
| `db.py` | SQLite schema |
| `auth.py` | OAuth credential management |
| `token.pickle` | YouTube OAuth token |
| `cookies.txt` | YouTube cookies (for transcript fallback) |
Cron: `0 */6 * * * bash /tmp/yt-fetch/fetch_and_sync.sh`
### NAS
Docker container at `/volume1/docker/t-youtube/`:
- `docker-compose.yml` — port 8001:8000, env vars for DeepSeek/Qwen
- `.env` — `DEEPSEEK_API_KEY`, `API_TOKEN`
- `data/t_youtube.db` — SQLite database (synced from VPS)
### MacBook
Qwen models run via `llama-server`:
- Port 8080: Qwen3.6-27B-Q8 (27B params, ~30 t/s, 256K ctx)
- Port 8085: Qwen3.6-35B-A3B-Q8 (35B MoE, 3.5B active, faster)
- Both with `--spec-type draft-mtp` for speculative decoding
## Configuration
### Environment Variables
| Variable | Default | Description |
|----------|---------|-------------|
| `DEEPSEEK_API_KEY` | — | DeepSeek API key for summarization |
| `DEEPSEEK_BASE_URL` | `https://api.deepseek.com` | API endpoint |
| `DEEPSEEK_MODEL` | `deepseek-chat` | Model name |
| `API_TOKEN` | — | Bearer token for dashboard write ops |
| `AGENT_ENABLED` | `false` | Enable agent queue worker |
| `QWEN_BASE_URL` | `http://localhost:8080` | Local Qwen endpoint |
| `HTTP_PROXY` | — | Proxy for fetcher/summarizer (China bypass) |
| `AGENT_POLL_INTERVAL` | `30` | Agent queue poll seconds |
| `AGENT_MAX_CONCURRENT` | `3` | Agent queue max workers |
### Authelia Bypass
API routes at `/api/*` bypass Authelia via `resources: ['^/api/']` in Authelia `access_control.yaml`. Frontend (`/`) requires SSO login.
## API Endpoints
| Endpoint | Method | Auth | Description |
|----------|--------|------|-------------|
| `/api/health` | GET | None | Health check |
| `/api/config` | GET | None | Auth config (token required?) |
| `/api/videos` | GET | None | List videos (status, channel, limit params) |
| `/api/videos/fetch` | POST | Bearer | Trigger subscription sync |
| `/api/videos/{id}/summarize` | POST | Bearer | Trigger single video summary |
| `/api/videos/{id}/status` | POST | Bearer | Update status (pending/read/skipped) |
| `/api/notes` | GET | None | List all notes |
| `/api/notes/{id}` | GET/POST | Bearer (POST) | Get/save note |
| `/api/notes/migrate` | POST | Bearer | Migrate localStorage notes to server |
| `/api/agent-queue` | GET/POST | Bearer (POST) | List/add agent queue items |
| `/api/channels` | GET | None | List distinct channels |
| `/api/auto_categories.json` | GET | None | Channel category mapping |
## Sprint Plan
### Sprint 1 ✅ (Current — Deployed)
- Agent queue worker (disabled by default)
- Proxy support for fetcher + summarizer
- Auto-mark-read on navigate away (not timer)
- VPS fetch + summarize cron
- run_tests.sh fixes
- All 2,674 videos summarized (title-based)
### Sprint 2 — UX Overhaul (Next)
- Full-text search across video titles
- Batch select → mark read/skip
- Infinite scroll / pagination
- Display duration + view count
- Transcript accordion in detail panel
### Sprint 3 — Polish & Monitoring
- Stats dashboard (videos/day, channel health, sync lag)
- Auto-purge old read videos (configurable TTL)
- Broken channel detection (404 on RSS)
- Cron failure alerts via Telegram bot
- Scheduled DB backup to MacBook
### Sprint 4 — Code Quality
- Split `main.py` into routers
- Cursor/offset pagination on API
- Rate limiting middleware
- aiosqlite connection pooling
## Known Issues
### Transcripts Blocked
YouTube transcripts inaccessible from all our locations:
- **Oracle Cloud VPS**: YouTube blocks cloud provider IPs
- **MacBook (China)**: GFW blocks YouTube API endpoints
- **YouTube Data API captions.download**: 403 Forbidden for non-owned videos
**Workaround**: Title + description-based summaries via Qwen. Full transcripts need a Japanese residential proxy ($3-5/month).
### auth.jimmygan.com from LAN
Accessing `auth.jimmygan.com` from LAN shows Immich (photos.jimmygan.com) instead of Authelia. Caused by NAS port 443 routing — DSM nginx still owns port 443 and routes Immich for unmatched domains.
### Gitea SSL
Uses Let's Encrypt **staging** certificates (acme-staging) — browsers may show untrusted cert warnings.
## Development
### Local Setup
```bash
cd ~/repos/t-youtube
# Backend
cd backend && python3 -m venv .venv && source .venv/bin/activate && pip install -r requirements.txt
python3 main.py # runs on :8000
# Frontend
cd frontend && npm install && npm run dev # runs on :5173 with API proxy
```
### Testing
```bash
cd ~/repos/t-youtube && bash run_tests.sh
# 23 tests across db, fetcher, summarizer, main, agent_worker
```
### Deploy to NAS
```bash
cd ~/repos/t-youtube
git add -A && git commit -m "..." && git push gitea main:master
ssh nas 'cd /volume1/docker/t-youtube && git pull origin master && docker compose up -d --build'
```