Compare commits
7 Commits
cc9a4487ac
...
main
| Author | SHA1 | Date | |
|---|---|---|---|
 Gan, Jimmy
|
43977205b6 | ||
|
Gan, Jimmy
|
37cc8bfec6 | ||
|
Gan, Jimmy
|
5e8ac62672 | ||
|
Gan, Jimmy
|
1e1eec1ecd | ||
|
Gan, Jimmy
|
9c81befaa2 | ||
|
Gan, Jimmy
|
31bb2496a7 | ||
|
Gan, Jimmy
|
2cd0ba13dd |
@@ -23,11 +23,16 @@ jobs:
|
||||
export DOCKER_API_VERSION=1.43
|
||||
docker build -t t-youtube:latest /volume1/docker/t-youtube-build 2>&1
|
||||
|
||||
- name: Deploy to NAS
|
||||
- name: Save & Load image on host
|
||||
run: |
|
||||
docker stop t-youtube 2>/dev/null || true
|
||||
docker rm t-youtube 2>/dev/null || true
|
||||
docker run -d --name t-youtube --restart unless-stopped \
|
||||
export DOCKER_API_VERSION=1.43
|
||||
docker save t-youtube:latest | /volume1/@appstore/ContainerManager/usr/bin/docker load 2>&1
|
||||
|
||||
- name: Swap container on host
|
||||
run: |
|
||||
/volume1/@appstore/ContainerManager/usr/bin/docker stop t-youtube 2>/dev/null || true
|
||||
/volume1/@appstore/ContainerManager/usr/bin/docker rm t-youtube 2>/dev/null || true
|
||||
/volume1/@appstore/ContainerManager/usr/bin/docker run -d --name t-youtube --restart unless-stopped \
|
||||
-p 8001:8000 \
|
||||
--env-file /volume1/docker/t-youtube/.env \
|
||||
-v /volume1/docker/t-youtube/data:/app/data \
|
||||
@@ -36,6 +41,7 @@ jobs:
|
||||
|
||||
- name: Verify
|
||||
run: |
|
||||
export DOCKER_API_VERSION=1.43
|
||||
sleep 3
|
||||
docker ps --filter name=t-youtube --format "{{.Status}}"
|
||||
echo "OK: deploy complete"
|
||||
|
||||
@@ -0,0 +1,38 @@
|
||||
# T YouTube — Text-Based YouTube Digest
|
||||
|
||||
## PDD Workflow
|
||||
|
||||
This project follows **PDD (Product Definition Document)** development.
|
||||
|
||||
**Always read these files before writing code:**
|
||||
- `docs/PDD.md` — product vision, goals, targets (the north star)
|
||||
- `docs/specs/active.md` — current sprint spec (what we're building now)
|
||||
|
||||
**Workflow:**
|
||||
1. PDD → Sprint Spec → Implementation → Validate ACs
|
||||
2. Scope changes update the sprint spec first, code second
|
||||
3. Never ship without ACs passing
|
||||
|
||||
## Dev Commands
|
||||
|
||||
- **Build:** `docker compose up -d --build`
|
||||
- **Test:** `./run_tests.sh`
|
||||
- **Run locally:** `uvicorn backend.main:app --reload` (backend) + `npm run dev` (frontend)
|
||||
- **Sync subscriptions:** `python run_sync.py`
|
||||
|
||||
## Architecture
|
||||
|
||||
- **Frontend:** Svelte 5 + Vite (`frontend/`)
|
||||
- **Backend:** FastAPI + SQLite (`backend/`)
|
||||
- **Deployment:** Docker container on NAS DS224+, port 8001
|
||||
- **Sync:** `run_sync.py` fetches YouTube subscriptions + transcripts via yt-dlp
|
||||
- **Auth:** Authelia SSO in front, cookie-based session
|
||||
- **Proxy:** YouTube blocked from China — sync runs via VPS (Oracle Tokyo) or proxy
|
||||
|
||||
## Key Files
|
||||
|
||||
- `docs/architecture.md` — full architecture reference
|
||||
- `Dockerfile` — multi-stage build (node → python)
|
||||
- `docker-compose.yml` — deployment config
|
||||
- `run_sync.py` — subscription sync entry point
|
||||
- `config/cookies.txt` — YouTube auth cookies (not committed)
|
||||
@@ -63,4 +63,44 @@ t-youtube/
|
||||
* **Key Takeaways (Bullet list)**
|
||||
* **Actionable Verdict**
|
||||
3. **Read/Skip State**: Mark videos as **Read** or **Skip** to clear them from your pending queue, allowing you to only keep track of what's unread.
|
||||
4. **View Full Transcript**: Expand the accordion at the bottom of the digest to read the raw transcripts if needed.
|
||||
4. **View Full Transcript**: Expand the accordion at the bottom of the digest to read the raw transcripts if needed.
|
||||
5. **Agent Queue**: Save videos with notes to a queue for Hermes agents to process (create skills, open issues, etc.)
|
||||
|
||||
---
|
||||
|
||||
## 📚 Documentation
|
||||
|
||||
Comprehensive documentation for this project is available in the `docs/` folder:
|
||||
|
||||
- **[Architecture](docs/architecture.md)** - System overview, components, API endpoints, database schema
|
||||
- **[Progress](docs/progress.md)** - Session history, completed tasks, planned features, metrics
|
||||
- **[Configuration](docs/configuration.md)** - Environment variables, API tokens, database, Authelia, Caddy setup
|
||||
- **[Troubleshooting](docs/troubleshooting.md)** - Common issues, recovery procedures, monitoring
|
||||
|
||||
---
|
||||
|
||||
## 🚀 Setup Instructions
|
||||
|
||||
### 1. Extract your YouTube browser cookies
|
||||
Since YouTube restricts subscription feeds to logged-in users, you need to export your logged-in cookies so `yt-dlp` can request the feed on your behalf:
|
||||
1. Install a browser extension like **Get cookies.txt LOCALLY** (Chrome/Edge) or **cookies.txt** (Firefox).
|
||||
2. Open YouTube in your browser and ensure you are logged in to your account.
|
||||
3. Click the extension icon and export the cookies for `youtube.com` as a Netscape-formatted text file.
|
||||
4. Create a folder named `config` in your project root, rename the exported file to `cookies.txt`, and save it inside:
|
||||
```text
|
||||
t-youtube/config/cookies.txt
|
||||
```
|
||||
|
||||
### 2. Configure Environment Variables
|
||||
Create a `.env` file in the root directory:
|
||||
```env
|
||||
# Optional: Override the fallback OpenAI / Anthropic key
|
||||
OPENAI_API_KEY=sk-Kq5...rOlx
|
||||
```
|
||||
|
||||
### 3. Build & Deploy using Docker Compose
|
||||
Run the following command to build the multi-stage image (compiles frontend static assets using Node, then packages backend with Python) and start the container:
|
||||
```bash
|
||||
docker compose up -d --build
|
||||
```
|
||||
The server will boot up and listen on port `8000`.
|
||||
|
||||
+21
@@ -0,0 +1,21 @@
|
||||
# Product Definition Document — T YouTube
|
||||
|
||||
## Vision
|
||||
|
||||
A text-first YouTube experience — read transcripts, get AI summaries, never watch a video.
|
||||
|
||||
## Goals
|
||||
|
||||
1. **No-video YouTube** — read subscriptions, watch content as text, never open the YouTube app
|
||||
2. **Daily digest** — AI-summarized subscription feed delivered to Telegram daily
|
||||
3. **GFW-proof** — works reliably from China despite YouTube blocks
|
||||
|
||||
## Target Users
|
||||
|
||||
Self (Jimmy Gan). Primary use: morning skim of YouTube subscriptions via text summaries.
|
||||
|
||||
## Success Metrics
|
||||
|
||||
- Daily sync completes without errors
|
||||
- Telegram digest delivered every morning
|
||||
- < 5s page load for dashboard
|
||||
@@ -0,0 +1,200 @@
|
||||
# t-youtube — Text-Based YouTube Digest
|
||||
|
||||
## Overview
|
||||
|
||||
A self-hosted YouTube reader that syncs your subscriptions, generates AI summaries, and displays them in a Svelte dashboard. Designed to work around GFW restrictions in China and YouTube IP blocks on cloud providers.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌─────────────────────────┐
|
||||
│ Oracle Tokyo VPS │
|
||||
│ (free-amd-caddy) │
|
||||
│ │
|
||||
│ ┌──────────────────┐ │
|
||||
│ │ Caddy (port 443) │ │
|
||||
│ │ reverse_proxy to │────┼─── Cloudflare ─── Internet
|
||||
│ │ NAS via Tailscale │ │
|
||||
│ └──────────────────┘ │
|
||||
│ │
|
||||
│ ┌──────────────────┐ │
|
||||
│ │ Cron (every 6h) │ │
|
||||
│ │ fetch_and_sync.sh │────┼─── YouTube Data API
|
||||
│ └──────────────────┘ │
|
||||
└──────────┬───────────────┘
|
||||
│ Tailscale (DERP relay)
|
||||
┌──────────┴───────────────┐
|
||||
│ NAS ds224plus │
|
||||
│ (100.78.131.124) │
|
||||
│ │
|
||||
│ ┌───────────────────┐ │
|
||||
│ │ Caddy (port 8443) │ │
|
||||
│ │ Let's Encrypt via │ │
|
||||
│ │ Cloudflare DNS │ │
|
||||
│ └───────────────────┘ │
|
||||
│ │
|
||||
│ ┌───────────────────┐ │
|
||||
│ │ Authelia (9092) │ │
|
||||
│ │ SSO + 2FA │ │
|
||||
│ └───────────────────┘ │
|
||||
│ │
|
||||
│ ┌───────────────────┐ │
|
||||
│ │ t-youtube (8001) │ │
|
||||
│ │ FastAPI + Svelte │ │
|
||||
│ │ SQLite DB │ │
|
||||
│ └───────────────────┘ │
|
||||
│ │
|
||||
│ ┌───────────────────┐ │
|
||||
│ │ Gitea (3300) │ │
|
||||
│ │ git.jimmygan.com │ │
|
||||
│ └───────────────────┘ │
|
||||
└──────────────────────────┘
|
||||
```
|
||||
|
||||
## Data Flow
|
||||
|
||||
1. **Fetch**: VPS cron runs every 6h → YouTube Data API v3 (subscriptions list) + RSS feeds → new video metadata → local SQLite DB
|
||||
2. **Summarize**: VPS reads new videos from DB → calls Qwen35B on MacBook (port 8085 via Tailscale) → title-only summaries → saves to DB
|
||||
3. **Sync**: VPS `cat` DB → SSH to NAS → `sudo tee` → overwrites NAS DB → restarts Docker container
|
||||
4. **Serve**: NAS Docker (FastAPI + Svelte) → serves dashboard to browser → Authelia SSO in front
|
||||
|
||||
## Deployment
|
||||
|
||||
### VPS (Oracle Tokyo)
|
||||
|
||||
Files in `/tmp/yt-fetch/`:
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `fetch_and_sync.sh` | Entry point — fetch → summarize → sync |
|
||||
| `fetcher.py` | YouTube Data API + RSS hybrid fetcher |
|
||||
| `summarize_new.py` | Title-based summarization via Qwen |
|
||||
| `db.py` | SQLite schema |
|
||||
| `auth.py` | OAuth credential management |
|
||||
| `token.pickle` | YouTube OAuth token |
|
||||
| `cookies.txt` | YouTube cookies (for transcript fallback) |
|
||||
|
||||
Cron: `0 */6 * * * bash /tmp/yt-fetch/fetch_and_sync.sh`
|
||||
|
||||
### NAS
|
||||
|
||||
Docker container at `/volume1/docker/t-youtube/`:
|
||||
- `docker-compose.yml` — port 8001:8000, env vars for DeepSeek/Qwen
|
||||
- `.env` — `DEEPSEEK_API_KEY`, `API_TOKEN`
|
||||
- `data/t_youtube.db` — SQLite database (synced from VPS)
|
||||
|
||||
### MacBook
|
||||
|
||||
Qwen models run via `llama-server`:
|
||||
- Port 8080: Qwen3.6-27B-Q8 (27B params, ~30 t/s, 256K ctx)
|
||||
- Port 8085: Qwen3.6-35B-A3B-Q8 (35B MoE, 3.5B active, faster)
|
||||
- Both with `--spec-type draft-mtp` for speculative decoding
|
||||
|
||||
## Configuration
|
||||
|
||||
### Environment Variables
|
||||
|
||||
| Variable | Default | Description |
|
||||
|----------|---------|-------------|
|
||||
| `DEEPSEEK_API_KEY` | — | DeepSeek API key for summarization |
|
||||
| `DEEPSEEK_BASE_URL` | `https://api.deepseek.com` | API endpoint |
|
||||
| `DEEPSEEK_MODEL` | `deepseek-chat` | Model name |
|
||||
| `API_TOKEN` | — | Bearer token for dashboard write ops |
|
||||
| `AGENT_ENABLED` | `false` | Enable agent queue worker |
|
||||
| `QWEN_BASE_URL` | `http://localhost:8080` | Local Qwen endpoint |
|
||||
| `HTTP_PROXY` | — | Proxy for fetcher/summarizer (China bypass) |
|
||||
| `AGENT_POLL_INTERVAL` | `30` | Agent queue poll seconds |
|
||||
| `AGENT_MAX_CONCURRENT` | `3` | Agent queue max workers |
|
||||
|
||||
### Authelia Bypass
|
||||
|
||||
API routes at `/api/*` bypass Authelia via `resources: ['^/api/']` in Authelia `access_control.yaml`. Frontend (`/`) requires SSO login.
|
||||
|
||||
## API Endpoints
|
||||
|
||||
| Endpoint | Method | Auth | Description |
|
||||
|----------|--------|------|-------------|
|
||||
| `/api/health` | GET | None | Health check |
|
||||
| `/api/config` | GET | None | Auth config (token required?) |
|
||||
| `/api/videos` | GET | None | List videos (status, channel, limit params) |
|
||||
| `/api/videos/fetch` | POST | Bearer | Trigger subscription sync |
|
||||
| `/api/videos/{id}/summarize` | POST | Bearer | Trigger single video summary |
|
||||
| `/api/videos/{id}/status` | POST | Bearer | Update status (pending/read/skipped) |
|
||||
| `/api/notes` | GET | None | List all notes |
|
||||
| `/api/notes/{id}` | GET/POST | Bearer (POST) | Get/save note |
|
||||
| `/api/notes/migrate` | POST | Bearer | Migrate localStorage notes to server |
|
||||
| `/api/agent-queue` | GET/POST | Bearer (POST) | List/add agent queue items |
|
||||
| `/api/channels` | GET | None | List distinct channels |
|
||||
| `/api/auto_categories.json` | GET | None | Channel category mapping |
|
||||
|
||||
## Sprint Plan
|
||||
|
||||
### Sprint 1 ✅ (Current — Deployed)
|
||||
- Agent queue worker (disabled by default)
|
||||
- Proxy support for fetcher + summarizer
|
||||
- Auto-mark-read on navigate away (not timer)
|
||||
- VPS fetch + summarize cron
|
||||
- run_tests.sh fixes
|
||||
- All 2,674 videos summarized (title-based)
|
||||
|
||||
### Sprint 2 — UX Overhaul (Next)
|
||||
- Full-text search across video titles
|
||||
- Batch select → mark read/skip
|
||||
- Infinite scroll / pagination
|
||||
- Display duration + view count
|
||||
- Transcript accordion in detail panel
|
||||
|
||||
### Sprint 3 — Polish & Monitoring
|
||||
- Stats dashboard (videos/day, channel health, sync lag)
|
||||
- Auto-purge old read videos (configurable TTL)
|
||||
- Broken channel detection (404 on RSS)
|
||||
- Cron failure alerts via Telegram bot
|
||||
- Scheduled DB backup to MacBook
|
||||
|
||||
### Sprint 4 — Code Quality
|
||||
- Split `main.py` into routers
|
||||
- Cursor/offset pagination on API
|
||||
- Rate limiting middleware
|
||||
- aiosqlite connection pooling
|
||||
|
||||
## Known Issues
|
||||
|
||||
### Transcripts Blocked
|
||||
YouTube transcripts inaccessible from all our locations:
|
||||
- **Oracle Cloud VPS**: YouTube blocks cloud provider IPs
|
||||
- **MacBook (China)**: GFW blocks YouTube API endpoints
|
||||
- **YouTube Data API captions.download**: 403 Forbidden for non-owned videos
|
||||
|
||||
**Workaround**: Title + description-based summaries via Qwen. Full transcripts need a Japanese residential proxy ($3-5/month).
|
||||
|
||||
### auth.jimmygan.com from LAN
|
||||
Accessing `auth.jimmygan.com` from LAN shows Immich (photos.jimmygan.com) instead of Authelia. Caused by NAS port 443 routing — DSM nginx still owns port 443 and routes Immich for unmatched domains.
|
||||
|
||||
### Gitea SSL
|
||||
Uses Let's Encrypt **staging** certificates (acme-staging) — browsers may show untrusted cert warnings.
|
||||
|
||||
## Development
|
||||
|
||||
### Local Setup
|
||||
```bash
|
||||
cd ~/repos/t-youtube
|
||||
# Backend
|
||||
cd backend && python3 -m venv .venv && source .venv/bin/activate && pip install -r requirements.txt
|
||||
python3 main.py # runs on :8000
|
||||
|
||||
# Frontend
|
||||
cd frontend && npm install && npm run dev # runs on :5173 with API proxy
|
||||
```
|
||||
|
||||
### Testing
|
||||
```bash
|
||||
cd ~/repos/t-youtube && bash run_tests.sh
|
||||
# 23 tests across db, fetcher, summarizer, main, agent_worker
|
||||
```
|
||||
|
||||
### Deploy to NAS
|
||||
```bash
|
||||
cd ~/repos/t-youtube
|
||||
git add -A && git commit -m "..." && git push gitea main:master
|
||||
ssh nas 'cd /volume1/docker/t-youtube && git pull origin master && docker compose up -d --build'
|
||||
```
|
||||
@@ -0,0 +1,152 @@
|
||||
# t-youtube Configuration Guide
|
||||
|
||||
## Environment Variables
|
||||
|
||||
### Backend (.env)
|
||||
```bash
|
||||
# Database path
|
||||
DB_DIR=/volume1/docker/t-youtube/backend
|
||||
|
||||
# API Token (for write operations)
|
||||
API_TOKEN=<your-api-token>
|
||||
|
||||
# DeepSeek API (for summarization)
|
||||
DEEPSEEK_API_KEY=<your-key>
|
||||
|
||||
# YouTube Data API v3
|
||||
GOOGLE_APPLICATION_CREDENTIALS=/path/to/credentials.json
|
||||
```
|
||||
|
||||
### Docker Compose
|
||||
```yaml
|
||||
version: "3"
|
||||
services:
|
||||
t-youtube:
|
||||
build: .
|
||||
ports:
|
||||
- "8001:8001"
|
||||
environment:
|
||||
- DB_DIR=/backend/data
|
||||
- API_TOKEN=<your-token>
|
||||
- DEEPSEEK_API_KEY=<your-key>
|
||||
volumes:
|
||||
- ./backend/data:/backend/data
|
||||
```
|
||||
|
||||
## API Token Management
|
||||
|
||||
### Generation
|
||||
```bash
|
||||
python3 -c "import secrets; print('api-' + secrets.token_urlsafe(32))"
|
||||
```
|
||||
|
||||
### Frontend Setup
|
||||
- First-time users see a token setup modal
|
||||
- Token stored in localStorage
|
||||
- Bearer token sent with all API write requests
|
||||
|
||||
### Backend Validation
|
||||
- Middleware checks `Authorization: Bearer <token>` for POST/PATCH/DELETE on `/api/*`
|
||||
- Returns 401 JSON for invalid tokens
|
||||
- GET endpoints remain open to authenticated frontend users
|
||||
|
||||
## Database Management
|
||||
|
||||
### Location
|
||||
- **Path**: `/volume1/docker/t-youtube/backend/t_youtube.db`
|
||||
- **Type**: SQLite (aiosqlite)
|
||||
- **Backup**: Copy the `.db` file before any schema changes
|
||||
|
||||
### Schema Migrations
|
||||
- Use `sqlite3` CLI or a migration script
|
||||
- Always add indexes after schema changes
|
||||
- Example:
|
||||
```bash
|
||||
sqlite3 t_youtube.db "CREATE INDEX IF NOT EXISTS idx_x ON table(column);"
|
||||
```
|
||||
|
||||
### Daily Sync Configuration
|
||||
|
||||
#### Cron Job
|
||||
```bash
|
||||
# Runs every day at 3 AM
|
||||
0 3 * * * /path/to/run_sync.py >> /var/log/t-youtube-sync.log 2>&1
|
||||
```
|
||||
|
||||
#### Fetcher Settings
|
||||
- **RSS**: Free, unlimited, ~2000 channels
|
||||
- **YouTube Data API v3**: ~50 units per sync
|
||||
- **Rotation**: Fetches in slices to avoid timeouts
|
||||
- **Retry**: Automatic retry on failure
|
||||
|
||||
#### Summarizer Settings
|
||||
- **Model**: `deepseek-chat` via DeepSeek API
|
||||
- **Timeout**: 30 seconds per video
|
||||
- **Fallback**: Video description if transcript unavailable
|
||||
- **Language**: Follows content language (Chinese → Chinese summary)
|
||||
|
||||
## Authelia Integration
|
||||
|
||||
### Authentication Flow
|
||||
1. User accesses protected domain
|
||||
2. Caddy forwards request to Authelia (`127.0.0.1:9092`)
|
||||
3. Authelia checks session/2FA
|
||||
4. If no session → redirect to `auth.jimmygan.com`
|
||||
5. If session exists → return 200 (allowed)
|
||||
|
||||
### 2FA Methods
|
||||
- **WebAuthn (Primary)**: Passkey via iCloud Keychain/Bitwarden
|
||||
- **TOTP (Fallback)**: Time-based one-time password
|
||||
|
||||
### Configuration
|
||||
- **Address**: `127.0.0.1:9092`
|
||||
- **Database**: SQLite (`/volume1/docker/authelia/config/db.sqlite3`)
|
||||
- **LDAP Backend**: LLDAP (`lldap:3890`)
|
||||
- **Session**: 1 hour expiration, 5 minute inactivity timeout
|
||||
|
||||
### Restart Authelia
|
||||
```bash
|
||||
cd /volume1/docker/authelia && /usr/local/bin/docker compose restart authelia
|
||||
```
|
||||
|
||||
## Caddy Configuration
|
||||
|
||||
### VPS Caddy (Public)
|
||||
- **Port**: 443
|
||||
- **TLS**: Let's Encrypt (HTTP-01)
|
||||
- **Authelia**: `forward_auth 100.78.131.124:9092`
|
||||
- **DNS**: dnsmasq → VPS Tailscale IP `100.109.198.126`
|
||||
|
||||
### NAS Caddy (Local/Tailscale)
|
||||
- **Port**: 8443 (fallback due to SNI bug)
|
||||
- **TLS**: Cloudflare DNS-01
|
||||
- **Authelia**: `forward_auth 127.0.0.1:9092`
|
||||
- **SNI Bug**: Per-site `tls` blocks cause cert routing issues
|
||||
- **Workaround**: Global TLS config + dnsmasq → VPS Tailscale IP
|
||||
|
||||
### Reloading Caddy
|
||||
```bash
|
||||
# VPS
|
||||
ssh caddy-vps "sudo systemctl reload caddy"
|
||||
|
||||
# NAS
|
||||
cd /volume1/docker/caddy && /usr/local/bin/docker compose restart caddy
|
||||
```
|
||||
|
||||
## Monitoring & Health
|
||||
|
||||
### Health Check
|
||||
```bash
|
||||
curl -s https://yt.jimmygan.com/api/health
|
||||
# Expected: {"status": "ok"}
|
||||
```
|
||||
|
||||
### Sync Status
|
||||
- Check `run_sync.py` logs in `/var/log/t-youtube-sync.log`
|
||||
- Monitor DB for new videos/summaries
|
||||
- Verify cron job is active: `crontab -l`
|
||||
|
||||
### Database Size
|
||||
```bash
|
||||
ls -lh /volume1/docker/t-youtube/backend/t_youtube.db
|
||||
```
|
||||
@@ -0,0 +1,102 @@
|
||||
# t-youtube Progress Tracking
|
||||
|
||||
## Session 2026-07-09: Agent Queue, Notes Migration, Architecture Documentation
|
||||
|
||||
### Completed
|
||||
- [x] **Agent Queue Feature**:
|
||||
- Added `agent_queue` table to SQLite DB
|
||||
- Implemented `POST/GET/DELETE` endpoints in `main.py`
|
||||
- Created `agent_worker.py` skeleton for processing pending items
|
||||
- Frontend "Agent Queue" tab (Svelte 5) with status indicators
|
||||
- [x] **Notes Migration**:
|
||||
- Moved notes from localStorage to server-side DB (`video_notes` table)
|
||||
- Added `POST/GET/DELETE` endpoints for notes
|
||||
- Implemented non-blocking localStorage migration script
|
||||
- [x] **Database Optimization**:
|
||||
- Added indexes on `video_id` for `agent_queue` and `video_notes` tables
|
||||
- [x] **Authelia Configuration**:
|
||||
- Attempted passwordless mode (failed due to unsupported key in Authelia v4)
|
||||
- Reverted to standard 2FA (WebAuthn/TOTP)
|
||||
- [x] **Documentation**:
|
||||
- Created `docs/` folder with `architecture.md` and `progress.md`
|
||||
- Documented all API endpoints, DB schema, infrastructure, and known issues
|
||||
|
||||
### In Progress
|
||||
- [ ] **Agent Worker Logic**:
|
||||
- `agent_worker.py` needs actual LLM/feature-creation logic
|
||||
- Pending: Define workflow for processing queued items
|
||||
- [ ] **YouTube Sync Monitoring**:
|
||||
- Daily cron at 3 AM is active
|
||||
- Pending: Monitor next run for errors/progress
|
||||
|
||||
### Planned
|
||||
- [ ] **Channel Categorization**:
|
||||
- ~119 channels remain uncategorized
|
||||
- Consider moving `AUTO_CATEGORIES` to server-side JSON config
|
||||
- [ ] **Transcript Fallback**:
|
||||
- Implement browser-based scraping (playwright/selenium) for the remaining 10%
|
||||
- High complexity/resource usage trade-off
|
||||
- [ ] **Performance Monitoring**:
|
||||
- Add health check endpoints for sync status
|
||||
- Implement error logging/reporting
|
||||
|
||||
## Session 2026-07-05 to 2026-07-08: Initial Development
|
||||
|
||||
### Key Milestones
|
||||
- [x] **YouTube Data API v3 + RSS Hybrid**: Replaced yt-dlp/cookies approach
|
||||
- Free, unlimited sync (~50 units/sync for 200+ channels)
|
||||
- 2414 new videos discovered in first run
|
||||
- [x] **AI Summarization**: DeepSeek API integration (was Claude/bytecatcode)
|
||||
- Chinese content → Chinese summary; English content → English summary
|
||||
- ~2.1s per video, ~$0.001 each
|
||||
- [x] **Frontend Overhaul**:
|
||||
- AI Digest as hero element
|
||||
- Keyboard navigation (`j/k/↑↓`, `r/m`, `s`, `n`, `Enter`, `?`)
|
||||
- Dark/light mode toggle
|
||||
- Channel filter dropdown with auto-generated categories
|
||||
- Auto-read after 3s silent mark
|
||||
- [x] **Agent Queue Integration**:
|
||||
- "Save to Agent" feature: notes ARE prompts for Hermes agents
|
||||
- Queue tab with pending/processing/done status
|
||||
- Cross-device note sync (localStorage → server DB)
|
||||
|
||||
### Technical Decisions
|
||||
1. **API Token Auth**: Bearer token for API writes, Authelia SSO for frontend
|
||||
2. **SQLite**: Simple, reliable, no external dependencies for single-user setup
|
||||
3. **DeepSeek API**: Cost-effective for summarization (~$0.001/video)
|
||||
4. **Svelte 5**: Reactivity model, keyboard workflow, small bundle size
|
||||
5. **RSS + YouTube Data API**: Free tier sufficient, avoids cookie/yt-dlp maintenance
|
||||
|
||||
## Session 2026-07-04: NAS & Authelia Setup
|
||||
|
||||
### Completed
|
||||
- [x] NAS Caddy port 443 conflict resolved (DSM nginx)
|
||||
- [x] dnsmasq → VPS Caddy Tailscale IP (100.109.198.126)
|
||||
- [x] Authelia WebAuthn enabled, passkey registered
|
||||
- [x] TOTP reset for jimmy, default 2FA set to WebAuthn
|
||||
- [x] Caddy SNI bug workaround (global TLS config, port 8443 fallback)
|
||||
|
||||
## Known Issues & Resolutions
|
||||
|
||||
### Authelia Passwordless Failure
|
||||
- **Issue**: `webauthn.passwordless: true` caused boot loop (`configuration key not expected`)
|
||||
- **Root Cause**: Unsupported configuration key in current Authelia version
|
||||
- **Resolution**: Reverted to standard 2FA (WebAuthn/TOTP)
|
||||
|
||||
### YouTube Transcript Blocking
|
||||
- **Issue**: 90% of videos have transcripts; 10% fail due to Oracle Cloud IP blocking
|
||||
- **Root Cause**: YouTube IP-based restrictions
|
||||
- **Resolution**: Fallback to video description summaries; consider browser-based scraping later
|
||||
|
||||
### NAS Caddy SNI Bug
|
||||
- **Issue**: NAS Caddy serves first cached cert for all SNI names
|
||||
- **Root Cause**: Per-site `tls { ... }` blocks conflict with global auto_https policy
|
||||
- **Resolution**: Global TLS config + dnsmasq → VPS Caddy Tailscale IP
|
||||
|
||||
## Metrics & Performance
|
||||
- **Videos in DB**: ~2414+ (initial sync)
|
||||
- **Channels**: ~229 subscribed
|
||||
- **Sync Time**: ~2-3 minutes for all channels
|
||||
- **Summarization**: ~2.1s/video, ~$0.001 each via DeepSeek
|
||||
- **Frontend Load Time**: 0.14-0.22s (API responses)
|
||||
- **Daily API Usage**: ~50 units (well within 10,000/day free tier)
|
||||
Symlink
+1
@@ -0,0 +1 @@
|
||||
sprint-001-sync-reliability.md
|
||||
@@ -0,0 +1,32 @@
|
||||
# Sprint 001 — Sync Reliability & Telegram Delivery
|
||||
|
||||
**Dates:** Active
|
||||
|
||||
## Goals
|
||||
|
||||
- Fix daily sync on NAS
|
||||
- Verify Telegram delivery pipeline
|
||||
- Stabilize subscription fetching behind GFW
|
||||
|
||||
## Tasks
|
||||
|
||||
### T1: Fix daily sync cron
|
||||
|
||||
**Description:**
|
||||
`t-youtube-daily-sync` errored last run. Determine if `run_sync.py` is the active pipeline or if VPS-based `fetch_and_sync.sh` replaced it. Fix whichever is active.
|
||||
|
||||
**ACs:**
|
||||
- [ ] Daily sync completes without errors
|
||||
- [ ] New videos appear in dashboard
|
||||
|
||||
**Impl Notes:**
|
||||
Check NAS cron logs. VPS sync fires every 6h.
|
||||
|
||||
### T2: Verify tg-group-filter delivery
|
||||
|
||||
**Description:**
|
||||
Cron switched from `deliver: local` to `deliver: all`. Confirm Telegram delivery works for filtered results.
|
||||
|
||||
**ACs:**
|
||||
- [ ] Telegram receives filtered video digest
|
||||
- [ ] No duplicate deliveries
|
||||
@@ -0,0 +1,194 @@
|
||||
# t-youtube Troubleshooting Guide
|
||||
|
||||
## Common Issues
|
||||
|
||||
### 1. Site Down / 502 Bad Gateway
|
||||
- **Cause**: t-youtube container not running or Authelia down
|
||||
- **Check**:
|
||||
```bash
|
||||
ssh nas 'docker ps | grep t-youtube'
|
||||
ssh nas 'docker ps | grep authelia'
|
||||
```
|
||||
- **Fix**:
|
||||
```bash
|
||||
cd /volume1/docker/t-youtube && /usr/local/bin/docker compose up -d
|
||||
cd /volume1/docker/authelia && /usr/local/bin/docker compose restart authelia
|
||||
```
|
||||
|
||||
### 2. Authelia Loop / Login Fails
|
||||
- **Cause**: WebAuthn/TOTP misconfiguration or session expired
|
||||
- **Check Authelia logs**:
|
||||
```bash
|
||||
ssh nas 'docker logs authelia --tail 20'
|
||||
```
|
||||
- **Fix**:
|
||||
- Clear browser cache/cookies
|
||||
- Reset WebAuthn via Authelia DB (see below)
|
||||
- Check dnsmasq config (should point to VPS Tailscale IP)
|
||||
|
||||
### 3. YouTube Data API Quota Exceeded
|
||||
- **Symptom**: Fetcher returns empty results or 403 errors
|
||||
- **Cause**: Daily quota limit reached (10,000 units)
|
||||
- **Check**:
|
||||
```bash
|
||||
ssh nas 'docker logs t-youtube 2>&1 | grep -i "quota\|403"'
|
||||
```
|
||||
- **Fix**:
|
||||
- Wait until next day (quota resets at 00:00 UTC)
|
||||
- Reduce number of channels or increase sync interval
|
||||
- Monitor usage in Google Cloud Console
|
||||
|
||||
### 4. Summarization Failing
|
||||
- **Symptom**: Videos show "No summary" or generic descriptions
|
||||
- **Cause**: DeepSeek API key invalid or rate limited
|
||||
- **Check**:
|
||||
```bash
|
||||
ssh nas 'docker logs t-youtube 2>&1 | grep -i "summariz\|deepseek"'
|
||||
```
|
||||
- **Fix**:
|
||||
- Verify `DEEPSEEK_API_KEY` in `.env`
|
||||
- Test API key manually:
|
||||
```bash
|
||||
curl -s -X POST https://api.deepseek.com/chat/completions \
|
||||
-H "Authorization: Bearer ***
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"model": "deepseek-chat", "messages": [{"role": "user", "content": "test"}]}'
|
||||
```
|
||||
|
||||
### 5. Database Locked/Corrupted
|
||||
- **Symptom**: API returns 500 errors or hangs
|
||||
- **Cause**: Concurrent write operations or unexpected shutdown
|
||||
- **Fix**:
|
||||
```bash
|
||||
cd /volume1/docker/t-youtube/backend
|
||||
cp t_youtube.db t_youtube.db.backup
|
||||
sqlite3 t_youtube.db "PRAGMA integrity_check;"
|
||||
# If corrupted, restore from backup or recreate
|
||||
```
|
||||
|
||||
### 6. Caddy Config Issues
|
||||
- **Symptom**: 502/503 errors, wrong cert, or domain unreachable
|
||||
- **Check**:
|
||||
```bash
|
||||
ssh caddy-vps "sudo caddy validate --config /etc/caddy/Caddyfile"
|
||||
ssh nas 'docker exec caddy caddy validate --config /etc/caddy/Caddyfile'
|
||||
```
|
||||
- **Fix**:
|
||||
- Validate config
|
||||
- Reload/restart Caddy
|
||||
- Check TLS certs:
|
||||
```bash
|
||||
echo | openssl s_client -connect 161.33.182.109:443 -servername yt.jimmygan.com 2>&1 | openssl x509 -noout -subject -dates
|
||||
```
|
||||
|
||||
## API Debugging
|
||||
|
||||
### Check Video List
|
||||
```bash
|
||||
curl -s https://yt.jimmygan.com/api/videos | python3 -m json.tool | head -20
|
||||
```
|
||||
|
||||
### Check Agent Queue
|
||||
```bash
|
||||
curl -s -H "Authorization: Bearer *** https://yt.jimmygan.com/api/agent-queue
|
||||
```
|
||||
|
||||
### Check Notes
|
||||
```bash
|
||||
curl -s -H "Authorization: Bearer *** https://yt.jimmygan.com/api/notes
|
||||
```
|
||||
|
||||
### Check Channel List
|
||||
```bash
|
||||
curl -s -H "Authorization: Bearer *** https://yt.jimmygan.com/api/channels
|
||||
```
|
||||
|
||||
### Test Summarization
|
||||
```bash
|
||||
curl -s -X POST -H "Authorization: Bearer ***
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"video_id": "<video_id>"}' \
|
||||
https://yt.jimmygan.com/api/agent-queue/<video_id>/summarize
|
||||
```
|
||||
|
||||
## Database Queries
|
||||
|
||||
### Count Videos by Status
|
||||
```sql
|
||||
SELECT status, COUNT(*) FROM videos GROUP BY status;
|
||||
```
|
||||
|
||||
### Count Agent Queue Items by Status
|
||||
```sql
|
||||
SELECT status, COUNT(*) FROM agent_queue GROUP BY status;
|
||||
```
|
||||
|
||||
### Find Videos Without Summaries
|
||||
```sql
|
||||
SELECT video_id, title FROM videos WHERE summary IS NULL OR summary = '';
|
||||
```
|
||||
|
||||
### Check DB Size
|
||||
```bash
|
||||
ls -lh /volume1/docker/t-youtube/backend/t_youtube.db
|
||||
```
|
||||
|
||||
## Logs
|
||||
|
||||
### t-youtube Logs
|
||||
```bash
|
||||
ssh nas 'docker logs t-youtube --tail 50'
|
||||
ssh nas 'docker logs t-youtube --tail 50 --since 1h'
|
||||
```
|
||||
|
||||
### Authelia Logs
|
||||
```bash
|
||||
ssh nas 'docker logs authelia --tail 50'
|
||||
```
|
||||
|
||||
### Sync Logs
|
||||
```bash
|
||||
ssh nas 'cat /var/log/t-youtube-sync.log'
|
||||
ssh nas 'tail -100 /var/log/t-youtube-sync.log'
|
||||
```
|
||||
|
||||
### Caddy Logs (VPS)
|
||||
```bash
|
||||
ssh caddy-vps "sudo journalctl -u caddy --since '1 hour ago'"
|
||||
ssh caddy-vps "sudo journalctl -u caddy --no-pager -n 30"
|
||||
```
|
||||
|
||||
## Recovery Procedures
|
||||
|
||||
### Full System Restart
|
||||
```bash
|
||||
# 1. Restart services
|
||||
cd /volume1/docker/authelia && /usr/local/bin/docker compose restart authelia
|
||||
cd /volume1/docker/t-youtube && /usr/local/bin/docker compose up -d
|
||||
|
||||
# 2. Verify
|
||||
curl -s https://yt.jimmygan.com/api/health
|
||||
curl -s https://auth.jimmygan.com/api/health
|
||||
```
|
||||
|
||||
### Database Backup & Restore
|
||||
```bash
|
||||
# Backup
|
||||
cp /volume1/docker/t-youtube/backend/t_youtube.db /tmp/t-youtube-backup-$(date +%Y%m%d).db
|
||||
|
||||
# Restore
|
||||
cp /tmp/t-youtube-backup-20260709.db /volume1/docker/t-youtube/backend/t_youtube.db
|
||||
cd /volume1/docker/t-youtube && /usr/local/bin/docker compose restart t-youtube
|
||||
```
|
||||
|
||||
### Authelia User Reset (if locked out)
|
||||
```bash
|
||||
# Stop Authelia
|
||||
ssh nas 'docker stop authelia'
|
||||
|
||||
# Reset TOTP for jimmy
|
||||
ssh nas 'sqlite3 /volume1/docker/authelia/config/db.sqlite3 "DELETE FROM totp_configurations WHERE username='\'jimmy'\'';"'
|
||||
|
||||
# Restart Authelia
|
||||
ssh nas 'docker start authelia'
|
||||
```
|
||||
Reference in New Issue
Block a user