refactor: reformat tg_filter.py and clean up

.gitea/workflows/deploy.yml

@@ -23,22 +23,25 @@ jobs:
           export DOCKER_API_VERSION=1.43
           docker build -t t-youtube:latest /volume1/docker/t-youtube-build 2>&1
 
-      - name: Deploy to NAS
+      - name: Save & Load image on host
         run: |
-          set -x
+          export DOCKER_API_VERSION=1.43
-          docker stop t-youtube 2>/dev/null || echo "no container to stop"
+          docker save t-youtube:latest | /volume1/@appstore/ContainerManager/usr/bin/docker load 2>&1
-          docker rm t-youtube 2>/dev/null || echo "no container to remove"
+
-          ls -la /volume1/docker/t-youtube/.env 2>&1
+      - name: Swap container on host
-          docker run -d --name t-youtube --restart unless-stopped \
+        run: |
+          /volume1/@appstore/ContainerManager/usr/bin/docker stop t-youtube 2>/dev/null || true
+          /volume1/@appstore/ContainerManager/usr/bin/docker rm t-youtube 2>/dev/null || true
+          /volume1/@appstore/ContainerManager/usr/bin/docker run -d --name t-youtube --restart unless-stopped \
             -p 8001:8000 \
             --env-file /volume1/docker/t-youtube/.env \
             -v /volume1/docker/t-youtube/data:/app/data \
             -v /volume1/docker/t-youtube/config:/app/config \
             t-youtube:latest 2>&1
-          echo "deploy exit: $?"
 
       - name: Verify
         run: |
+          export DOCKER_API_VERSION=1.43
           sleep 3
           docker ps --filter name=t-youtube --format "{{.Status}}"
           echo "OK: deploy complete"

CLAUDE.md

@@ -0,0 +1,38 @@
+# T YouTube — Text-Based YouTube Digest
+
+## PDD Workflow
+
+This project follows **PDD (Product Definition Document)** development.
+
+**Always read these files before writing code:**
+- `docs/PDD.md` — product vision, goals, targets (the north star)
+- `docs/specs/active.md` — current sprint spec (what we're building now)
+
+**Workflow:**
+1. PDD → Sprint Spec → Implementation → Validate ACs
+2. Scope changes update the sprint spec first, code second
+3. Never ship without ACs passing
+
+## Dev Commands
+
+- **Build:** `docker compose up -d --build`
+- **Test:** `./run_tests.sh`
+- **Run locally:** `uvicorn backend.main:app --reload` (backend) + `npm run dev` (frontend)
+- **Sync subscriptions:** `python run_sync.py`
+
+## Architecture
+
+- **Frontend:** Svelte 5 + Vite (`frontend/`)
+- **Backend:** FastAPI + SQLite (`backend/`)
+- **Deployment:** Docker container on NAS DS224+, port 8001
+- **Sync:** `run_sync.py` fetches YouTube subscriptions + transcripts via yt-dlp
+- **Auth:** Authelia SSO in front, cookie-based session
+- **Proxy:** YouTube blocked from China — sync runs via VPS (Oracle Tokyo) or proxy
+
+## Key Files
+
+- `docs/architecture.md` — full architecture reference
+- `Dockerfile` — multi-stage build (node → python)
+- `docker-compose.yml` — deployment config
+- `run_sync.py` — subscription sync entry point
+- `config/cookies.txt` — YouTube auth cookies (not committed)

README.md

@@ -64,3 +64,43 @@ t-youtube/
     *   **Actionable Verdict**
 3.  **Read/Skip State**: Mark videos as **Read** or **Skip** to clear them from your pending queue, allowing you to only keep track of what's unread.
 4. **View Full Transcript**: Expand the accordion at the bottom of the digest to read the raw transcripts if needed.
+5. **Agent Queue**: Save videos with notes to a queue for Hermes agents to process (create skills, open issues, etc.)
+
+---
+
+## 📚 Documentation
+
+Comprehensive documentation for this project is available in the `docs/` folder:
+
+- **[Architecture](docs/architecture.md)** - System overview, components, API endpoints, database schema
+- **[Progress](docs/progress.md)** - Session history, completed tasks, planned features, metrics
+- **[Configuration](docs/configuration.md)** - Environment variables, API tokens, database, Authelia, Caddy setup
+- **[Troubleshooting](docs/troubleshooting.md)** - Common issues, recovery procedures, monitoring
+
+---
+
+## 🚀 Setup Instructions
+
+### 1. Extract your YouTube browser cookies
+Since YouTube restricts subscription feeds to logged-in users, you need to export your logged-in cookies so `yt-dlp` can request the feed on your behalf:
+1. Install a browser extension like **Get cookies.txt LOCALLY** (Chrome/Edge) or **cookies.txt** (Firefox).
+2. Open YouTube in your browser and ensure you are logged in to your account.
+3. Click the extension icon and export the cookies for `youtube.com` as a Netscape-formatted text file.
+4. Create a folder named `config` in your project root, rename the exported file to `cookies.txt`, and save it inside:
+   ```text
+   t-youtube/config/cookies.txt
+   ```
+
+### 2. Configure Environment Variables
+Create a `.env` file in the root directory:
+```env
+# Optional: Override the fallback OpenAI / Anthropic key
+OPENAI_API_KEY=sk-Kq5...rOlx
+```
+
+### 3. Build & Deploy using Docker Compose
+Run the following command to build the multi-stage image (compiles frontend static assets using Node, then packages backend with Python) and start the container:
+```bash
+docker compose up -d --build
+```
+The server will boot up and listen on port `8000`.

backend/tg_filter.py

@@ -1,9 +1,7 @@
 """
 Telegram Group Filter — scours 100+ groups for interesting messages.
-Runs as: t-youtube venv + Pyrogram session.
+Rotates through all groups across cron cycles to avoid rate limits.
-
+Resets processed tracking per batch so each cycle gets fresh messages.
-Scoring: mentions, links, code blocks, media, keywords → score ≥ threshold → saved.
-Delivers to: hermes_telegram_bot or stdout.
 """
 
 import asyncio
@@ -32,15 +30,19 @@ WORKDIR = "/Users/jimmyg"
 DB_PATH = os.path.join(WORKDIR, ".hermes", "tg_filter.db")
 
 # Scoring
-SCORE_MENTION = 30        # @username or reply to user
+SCORE_MENTION = 30
-SCORE_KEYWORD = 15        # Interesting keyword match
+SCORE_KEYWORD = 15
-SCORE_LINK = 10           # Contains URL
+SCORE_LINK = 10
-SCORE_MEDIA = 8           # Photo/video/file
+SCORE_MEDIA = 8
-SCORE_CODE = 12           # Code block
+SCORE_CODE = 12
-SCORE_LONG_TEXT = 5       # ≥200 chars of substance
+SCORE_LONG_TEXT = 5
-THRESHOLD = 20            # Minimum score to forward
+THRESHOLD = 40
 
-# Keywords that signal interesting content
+# Scan config
+MSGS_PER_GROUP = 500
+GROUPS_PER_CYCLE = 56
+
+# Keywords
 INTERESTING_KEYWORDS = [
     "ai", "llm", "gpt", "deepseek", "claude", "gemini", "qwen", "openai",
     "docker", "kubernetes", "k8s", "tailscale", "wireguard", "vps", "nas",
@@ -53,101 +55,64 @@ INTERESTING_KEYWORDS = [
     "startup", "saas", "mvp", "产品", "产品发布",
 ]
 
-# Groups to always ignore (IDs or title substrings)
+IGNORE_PATTERNS = ["合租社群", "黄油派对"]
-IGNORE_PATTERNS = [
+
-    "合租社群",
-    "黄油派对",
-]
 
 def score_message(msg: Message) -> int:
-    """Score a message for interestingness."""
     score = 0
     text = msg.text or msg.caption or ""
-
-    # Mentions
     if msg.entities:
         for e in msg.entities:
-            if e.type in ("mention", "text_mention"):
+            if e.type in ("mention", "text_mention"): score += SCORE_MENTION
-                score += SCORE_MENTION
+            if e.type == "url": score += SCORE_LINK
-            if e.type == "url":
+            if e.type == "code" or e.type == "pre": score += SCORE_CODE
-                score += SCORE_LINK
+    if msg.photo or msg.video or msg.document or msg.audio: score += SCORE_MEDIA
-            if e.type == "code" or e.type == "pre":
+    if re.search(r'https?://[^\s]+', text): score += SCORE_LINK
-                score += SCORE_CODE
-
-    # Media
-    if msg.photo or msg.video or msg.document or msg.audio:
-        score += SCORE_MEDIA
-
-    # Links in plain text
-    if re.search(r'https?://[^\s]+', text):
-        score += SCORE_LINK
-
-    # Keyword matches
     text_lower = text.lower()
     for kw in INTERESTING_KEYWORDS:
-        if kw in text_lower:
+        if kw in text_lower: score += SCORE_KEYWORD
-            score += SCORE_KEYWORD
+    if len(text.strip()) >= 200: score += SCORE_LONG_TEXT
-
+    if msg.forward_from or msg.forward_from_chat: score += 5
-    # Long substantive text
+    if msg.reply_to_message_id: score += 3
-    if len(text.strip()) >= 200:
-        score += SCORE_LONG_TEXT
-
-    # Forwarded messages are often interesting
-    if msg.forward_from or msg.forward_from_chat:
-        score += 5
-
-    # Reply to user's own messages (engagement)
-    if msg.reply_to_message_id:
-        score += 3
-
     return score
 
 
 def should_ignore(chat_title: str) -> bool:
-    """Check if a chat should be ignored entirely."""
+    if not chat_title: return False
-    if not chat_title:
-        return False
     for pat in IGNORE_PATTERNS:
-        if pat in chat_title:
+        if pat in chat_title: return True
-            return True
     return False
 
 
 def init_db():
-    """Create DB + table for tracking processed messages."""
     os.makedirs(os.path.dirname(DB_PATH), exist_ok=True)
     db = sqlite3.connect(DB_PATH)
-    db.execute("""
+    db.execute("CREATE TABLE IF NOT EXISTS processed (chat_id INTEGER, message_id INTEGER, PRIMARY KEY (chat_id, message_id))")
-        CREATE TABLE IF NOT EXISTS processed (
+    db.execute("CREATE TABLE IF NOT EXISTS interesting (id INTEGER PRIMARY KEY AUTOINCREMENT, chat_id INTEGER, chat_title TEXT, message_id INTEGER, sender TEXT, text TEXT, score INTEGER, captured_at TEXT, delivered INTEGER DEFAULT 0)")
-            chat_id INTEGER,
+    db.execute("CREATE TABLE IF NOT EXISTS group_stats (chat_id INTEGER PRIMARY KEY, chat_title TEXT, total_scanned INTEGER DEFAULT 0, interesting_found INTEGER DEFAULT 0, last_interesting_at TEXT, last_scan_at TEXT)")
-            message_id INTEGER,
-            PRIMARY KEY (chat_id, message_id)
-        )
-    """)
-    db.execute("""
-        CREATE TABLE IF NOT EXISTS interesting (
-            id INTEGER PRIMARY KEY AUTOINCREMENT,
-            chat_id INTEGER,
-            chat_title TEXT,
-            message_id INTEGER,
-            sender TEXT,
-            text TEXT,
-            score INTEGER,
-            captured_at TEXT,
-            delivered INTEGER DEFAULT 0
-        )
-    """)
     db.commit()
     return db
 
 
-async def scan_and_report(dry_run: bool = False) -> list:
+def update_group_stats(db, chat_id: int, chat_title: str, found_interesting: int, now: str):
-    """Scan dialogs for interesting messages. Returns list of interesting items."""
+    cur = db.execute("SELECT total_scanned, interesting_found FROM group_stats WHERE chat_id = ?", (chat_id,))
-    db = init_db()
+    row = cur.fetchone()
-    processed_ids = set()
+    if row:
-    for row in db.execute("SELECT chat_id, message_id FROM processed"):
+        db.execute(
-        processed_ids.add((row[0], row[1]))
+            "UPDATE group_stats SET total_scanned = total_scanned + 1, interesting_found = interesting_found + ?, "
+            "last_interesting_at = CASE WHEN ? > 0 THEN ? ELSE last_interesting_at END, last_scan_at = ? WHERE chat_id = ?",
+            (found_interesting, found_interesting, now if found_interesting > 0 else None, now, chat_id)
+        )
+    else:
+        db.execute(
+            "INSERT INTO group_stats (chat_id, chat_title, total_scanned, interesting_found, last_interesting_at, last_scan_at) "
+            "VALUES (?, ?, 1, ?, ?, ?)",
+            (chat_id, chat_title, found_interesting, now if found_interesting > 0 else None, now)
+        )
 
+
+async def scan_and_report(dry_run: bool = False) -> list:
+    db = init_db()
     c = Client("pyro_session", API_ID, API_HASH, workdir=WORKDIR)
     await c.start()
     me = await c.get_me()
@@ -156,78 +121,104 @@ async def scan_and_report(dry_run: bool = False) -> list:
     interesting = []
     total_read = 0
     total_skipped = 0
+    total_groups = 0
+    now = datetime.now(timezone.utc).isoformat()
 
+    # Collect all eligible dialogs
+    candidates = []
     async for dialog in c.get_dialogs():
         chat = dialog.chat
         chat_id = chat.id
         chat_title = chat.title or chat.first_name or "Unknown"
-
         if should_ignore(chat_title):
             continue
+        candidates.append((chat_id, chat_title))
 
-        if not dialog.unread_messages_count:
+    log.info(f"Total eligible groups: {len(candidates)}")
-            continue
 
-        # Read recent messages (up to 50 per chat)
+    # Sort by last_scan_at (NULLs first) to rotate
-        async for msg in c.get_chat_history(chat_id, limit=50):
+    sorted_candidates = []
-            if (chat_id, msg.id) in processed_ids:
+    for cid, title in candidates:
-                continue
+        cur = db.execute("SELECT last_scan_at FROM group_stats WHERE chat_id = ?", (cid,))
+        row = cur.fetchone()
+        if row and row[0]:
+            sorted_candidates.append((cid, title, row[0]))
+        else:
+            sorted_candidates.append((cid, title, ""))
+    sorted_candidates.sort(key=lambda x: (x[2] == "", x[2]))
+    sorted_candidates = sorted_candidates[:GROUPS_PER_CYCLE]
+
+    log.info(f"Scanning {len(sorted_candidates)} groups this cycle")
+
+    # Clear processed entries for these groups so we get fresh messages
+    group_ids = [cid for cid, _, _ in sorted_candidates]
+    db.execute("DELETE FROM processed WHERE chat_id IN ({})".format(
+        ",".join("?" * len(group_ids))
+    ), group_ids)
+    db.commit()
+
+    for chat_id, chat_title, _ in sorted_candidates:
+        total_groups += 1
+        group_found = 0
+
+        async for msg in c.get_chat_history(chat_id, limit=MSGS_PER_GROUP):
             total_read += 1
-
             score = score_message(msg)
+
             if score >= THRESHOLD:
                 sender = None
                 if msg.from_user:
                     sender = msg.from_user.first_name or msg.from_user.username or str(msg.from_user.id)
                 text = (msg.text or msg.caption or "")[:2000]
-                
                 item = {
-                    "chat_id": chat_id,
+                    "chat_id": chat_id, "chat_title": chat_title, "message_id": msg.id,
-                    "chat_title": chat_title,
+                    "sender": sender, "text": text, "score": score,
-                    "message_id": msg.id,
-                    "sender": sender,
-                    "text": text,
-                    "score": score,
                     "has_media": bool(msg.photo or msg.video or msg.document),
                     "url": f"https://t.me/c/{str(chat_id).replace('-100', '')}/{msg.id}" if chat_id < 0 else None,
                 }
                 interesting.append(item)
-
+                group_found += 1
-                # Save to DB
                 db.execute(
                     "INSERT OR IGNORE INTO interesting (chat_id, chat_title, message_id, sender, text, score, captured_at) VALUES (?, ?, ?, ?, ?, ?, ?)",
-                    (chat_id, chat_title, msg.id, sender, text, score, datetime.now(timezone.utc).isoformat()),
+                    (chat_id, chat_title, msg.id, sender, text, score, now),
                 )
             else:
                 total_skipped += 1
 
-            # Mark as processed
+        update_group_stats(db, chat_id, chat_title, group_found, now)
-            db.execute("INSERT OR IGNORE INTO processed (chat_id, message_id) VALUES (?, ?)", (chat_id, msg.id))
-
         db.commit()
 
     await c.stop()
     db.close()
 
-    log.info(f"Read {total_read} messages, skipped {total_skipped}, found {len(interesting)} interesting")
+    log.info(f"Scanned {total_groups} groups, read {total_read} msgs, found {len(interesting)} interesting")
 
     if not dry_run and interesting:
         for item in interesting:
             _print_item(item)
 
+    # Group quality summary
+    print(f"\n{'='*50}")
+    print(f"📊 GROUP QUALITY SUMMARY")
+    print(f"{'='*50}")
+    db2 = sqlite3.connect(DB_PATH)
+    rows = db2.execute(
+        "SELECT chat_title, total_scanned, interesting_found, last_interesting_at FROM group_stats ORDER BY interesting_found DESC LIMIT 50"
+    ).fetchall()
+    for title, scanned, found, last_interest in rows:
+        status = "🟢" if found >= 1 else "🟡" if scanned >= 3 else "⚪"
+        print(f"{status} {title} — {scanned} scans, {found} interesting, last signal: {last_interest or 'never'}")
+    db2.close()
+
     return interesting
 
 
 def _print_item(item: dict):
-    """Print an interesting item for delivery."""
     print(f"\n{'='*50}")
     print(f"📌 {item['chat_title']} [{item['score']}pts]")
-    if item['sender']:
+    if item['sender']: print(f"👤 {item['sender']}")
-        print(f"👤 {item['sender']}")
+    if item['has_media']: print(f"📎 [has media]")
-    if item['has_media']:
+    if item['url']: print(f"🔗 {item['url']}")
-        print(f"📎 [has media]")
-    if item['url']:
-        print(f"🔗 {item['url']}")
     print(f"\n{item['text'][:500]}")
     print(f"{'='*50}")
 

docs/PDD.md

@@ -0,0 +1,21 @@
+# Product Definition Document — T YouTube
+
+## Vision
+
+A text-first YouTube experience — read transcripts, get AI summaries, never watch a video.
+
+## Goals
+
+1. **No-video YouTube** — read subscriptions, watch content as text, never open the YouTube app
+2. **Daily digest** — AI-summarized subscription feed delivered to Telegram daily
+3. **GFW-proof** — works reliably from China despite YouTube blocks
+
+## Target Users
+
+Self (Jimmy Gan). Primary use: morning skim of YouTube subscriptions via text summaries.
+
+## Success Metrics
+
+- Daily sync completes without errors
+- Telegram digest delivered every morning
+- < 5s page load for dashboard

docs/configuration.md

@@ -0,0 +1,152 @@
+# t-youtube Configuration Guide
+
+## Environment Variables
+
+### Backend (.env)
+```bash
+# Database path
+DB_DIR=/volume1/docker/t-youtube/backend
+
+# API Token (for write operations)
+API_TOKEN=<your-api-token>
+
+# DeepSeek API (for summarization)
+DEEPSEEK_API_KEY=<your-key>
+
+# YouTube Data API v3
+GOOGLE_APPLICATION_CREDENTIALS=/path/to/credentials.json
+```
+
+### Docker Compose
+```yaml
+version: "3"
+services:
+  t-youtube:
+    build: .
+    ports:
+      - "8001:8001"
+    environment:
+      - DB_DIR=/backend/data
+      - API_TOKEN=<your-token>
+      - DEEPSEEK_API_KEY=<your-key>
+    volumes:
+      - ./backend/data:/backend/data
+```
+
+## API Token Management
+
+### Generation
+```bash
+python3 -c "import secrets; print('api-' + secrets.token_urlsafe(32))"
+```
+
+### Frontend Setup
+- First-time users see a token setup modal
+- Token stored in localStorage
+- Bearer token sent with all API write requests
+
+### Backend Validation
+- Middleware checks `Authorization: Bearer <token>` for POST/PATCH/DELETE on `/api/*`
+- Returns 401 JSON for invalid tokens
+- GET endpoints remain open to authenticated frontend users
+
+## Database Management
+
+### Location
+- **Path**: `/volume1/docker/t-youtube/backend/t_youtube.db`
+- **Type**: SQLite (aiosqlite)
+- **Backup**: Copy the `.db` file before any schema changes
+
+### Schema Migrations
+- Use `sqlite3` CLI or a migration script
+- Always add indexes after schema changes
+- Example:
+  ```bash
+  sqlite3 t_youtube.db "CREATE INDEX IF NOT EXISTS idx_x ON table(column);"
+  ```
+
+### Daily Sync Configuration
+
+#### Cron Job
+```bash
+# Runs every day at 3 AM
+0 3 * * * /path/to/run_sync.py >> /var/log/t-youtube-sync.log 2>&1
+```
+
+#### Fetcher Settings
+- **RSS**: Free, unlimited, ~2000 channels
+- **YouTube Data API v3**: ~50 units per sync
+- **Rotation**: Fetches in slices to avoid timeouts
+- **Retry**: Automatic retry on failure
+
+#### Summarizer Settings
+- **Model**: `deepseek-chat` via DeepSeek API
+- **Timeout**: 30 seconds per video
+- **Fallback**: Video description if transcript unavailable
+- **Language**: Follows content language (Chinese → Chinese summary)
+
+## Authelia Integration
+
+### Authentication Flow
+1. User accesses protected domain
+2. Caddy forwards request to Authelia (`127.0.0.1:9092`)
+3. Authelia checks session/2FA
+4. If no session → redirect to `auth.jimmygan.com`
+5. If session exists → return 200 (allowed)
+
+### 2FA Methods
+- **WebAuthn (Primary)**: Passkey via iCloud Keychain/Bitwarden
+- **TOTP (Fallback)**: Time-based one-time password
+
+### Configuration
+- **Address**: `127.0.0.1:9092`
+- **Database**: SQLite (`/volume1/docker/authelia/config/db.sqlite3`)
+- **LDAP Backend**: LLDAP (`lldap:3890`)
+- **Session**: 1 hour expiration, 5 minute inactivity timeout
+
+### Restart Authelia
+```bash
+cd /volume1/docker/authelia && /usr/local/bin/docker compose restart authelia
+```
+
+## Caddy Configuration
+
+### VPS Caddy (Public)
+- **Port**: 443
+- **TLS**: Let's Encrypt (HTTP-01)
+- **Authelia**: `forward_auth 100.78.131.124:9092`
+- **DNS**: dnsmasq → VPS Tailscale IP `100.109.198.126`
+
+### NAS Caddy (Local/Tailscale)
+- **Port**: 8443 (fallback due to SNI bug)
+- **TLS**: Cloudflare DNS-01
+- **Authelia**: `forward_auth 127.0.0.1:9092`
+- **SNI Bug**: Per-site `tls` blocks cause cert routing issues
+- **Workaround**: Global TLS config + dnsmasq → VPS Tailscale IP
+
+### Reloading Caddy
+```bash
+# VPS
+ssh caddy-vps "sudo systemctl reload caddy"
+
+# NAS
+cd /volume1/docker/caddy && /usr/local/bin/docker compose restart caddy
+```
+
+## Monitoring & Health
+
+### Health Check
+```bash
+curl -s https://yt.jimmygan.com/api/health
+# Expected: {"status": "ok"}
+```
+
+### Sync Status
+- Check `run_sync.py` logs in `/var/log/t-youtube-sync.log`
+- Monitor DB for new videos/summaries
+- Verify cron job is active: `crontab -l`
+
+### Database Size
+```bash
+ls -lh /volume1/docker/t-youtube/backend/t_youtube.db
+```

docs/progress.md

@@ -0,0 +1,102 @@
+# t-youtube Progress Tracking
+
+## Session 2026-07-09: Agent Queue, Notes Migration, Architecture Documentation
+
+### Completed
+- [x] **Agent Queue Feature**:
+  - Added `agent_queue` table to SQLite DB
+  - Implemented `POST/GET/DELETE` endpoints in `main.py`
+  - Created `agent_worker.py` skeleton for processing pending items
+  - Frontend "Agent Queue" tab (Svelte 5) with status indicators
+- [x] **Notes Migration**:
+  - Moved notes from localStorage to server-side DB (`video_notes` table)
+  - Added `POST/GET/DELETE` endpoints for notes
+  - Implemented non-blocking localStorage migration script
+- [x] **Database Optimization**:
+  - Added indexes on `video_id` for `agent_queue` and `video_notes` tables
+- [x] **Authelia Configuration**:
+  - Attempted passwordless mode (failed due to unsupported key in Authelia v4)
+  - Reverted to standard 2FA (WebAuthn/TOTP)
+- [x] **Documentation**:
+  - Created `docs/` folder with `architecture.md` and `progress.md`
+  - Documented all API endpoints, DB schema, infrastructure, and known issues
+
+### In Progress
+- [ ] **Agent Worker Logic**:
+  - `agent_worker.py` needs actual LLM/feature-creation logic
+  - Pending: Define workflow for processing queued items
+- [ ] **YouTube Sync Monitoring**:
+  - Daily cron at 3 AM is active
+  - Pending: Monitor next run for errors/progress
+
+### Planned
+- [ ] **Channel Categorization**:
+  - ~119 channels remain uncategorized
+  - Consider moving `AUTO_CATEGORIES` to server-side JSON config
+- [ ] **Transcript Fallback**:
+  - Implement browser-based scraping (playwright/selenium) for the remaining 10%
+  - High complexity/resource usage trade-off
+- [ ] **Performance Monitoring**:
+  - Add health check endpoints for sync status
+  - Implement error logging/reporting
+
+## Session 2026-07-05 to 2026-07-08: Initial Development
+
+### Key Milestones
+- [x] **YouTube Data API v3 + RSS Hybrid**: Replaced yt-dlp/cookies approach
+  - Free, unlimited sync (~50 units/sync for 200+ channels)
+  - 2414 new videos discovered in first run
+- [x] **AI Summarization**: DeepSeek API integration (was Claude/bytecatcode)
+  - Chinese content → Chinese summary; English content → English summary
+  - ~2.1s per video, ~$0.001 each
+- [x] **Frontend Overhaul**:
+  - AI Digest as hero element
+  - Keyboard navigation (`j/k/↑↓`, `r/m`, `s`, `n`, `Enter`, `?`)
+  - Dark/light mode toggle
+  - Channel filter dropdown with auto-generated categories
+  - Auto-read after 3s silent mark
+- [x] **Agent Queue Integration**:
+  - "Save to Agent" feature: notes ARE prompts for Hermes agents
+  - Queue tab with pending/processing/done status
+  - Cross-device note sync (localStorage → server DB)
+
+### Technical Decisions
+1. **API Token Auth**: Bearer token for API writes, Authelia SSO for frontend
+2. **SQLite**: Simple, reliable, no external dependencies for single-user setup
+3. **DeepSeek API**: Cost-effective for summarization (~$0.001/video)
+4. **Svelte 5**: Reactivity model, keyboard workflow, small bundle size
+5. **RSS + YouTube Data API**: Free tier sufficient, avoids cookie/yt-dlp maintenance
+
+## Session 2026-07-04: NAS & Authelia Setup
+
+### Completed
+- [x] NAS Caddy port 443 conflict resolved (DSM nginx)
+- [x] dnsmasq → VPS Caddy Tailscale IP (100.109.198.126)
+- [x] Authelia WebAuthn enabled, passkey registered
+- [x] TOTP reset for jimmy, default 2FA set to WebAuthn
+- [x] Caddy SNI bug workaround (global TLS config, port 8443 fallback)
+
+## Known Issues & Resolutions
+
+### Authelia Passwordless Failure
+- **Issue**: `webauthn.passwordless: true` caused boot loop (`configuration key not expected`)
+- **Root Cause**: Unsupported configuration key in current Authelia version
+- **Resolution**: Reverted to standard 2FA (WebAuthn/TOTP)
+
+### YouTube Transcript Blocking
+- **Issue**: 90% of videos have transcripts; 10% fail due to Oracle Cloud IP blocking
+- **Root Cause**: YouTube IP-based restrictions
+- **Resolution**: Fallback to video description summaries; consider browser-based scraping later
+
+### NAS Caddy SNI Bug
+- **Issue**: NAS Caddy serves first cached cert for all SNI names
+- **Root Cause**: Per-site `tls { ... }` blocks conflict with global auto_https policy
+- **Resolution**: Global TLS config + dnsmasq → VPS Caddy Tailscale IP
+
+## Metrics & Performance
+- **Videos in DB**: ~2414+ (initial sync)
+- **Channels**: ~229 subscribed
+- **Sync Time**: ~2-3 minutes for all channels
+- **Summarization**: ~2.1s/video, ~$0.001 each via DeepSeek
+- **Frontend Load Time**: 0.14-0.22s (API responses)
+- **Daily API Usage**: ~50 units (well within 10,000/day free tier)

docs/specs/active.md

@@ -0,0 +1 @@
+sprint-001-sync-reliability.md

docs/specs/sprint-001-sync-reliability.md

@@ -0,0 +1,32 @@
+# Sprint 001 — Sync Reliability & Telegram Delivery
+
+**Dates:** Active
+
+## Goals
+
+- Fix daily sync on NAS
+- Verify Telegram delivery pipeline
+- Stabilize subscription fetching behind GFW
+
+## Tasks
+
+### T1: Fix daily sync cron
+
+**Description:**
+`t-youtube-daily-sync` errored last run. Determine if `run_sync.py` is the active pipeline or if VPS-based `fetch_and_sync.sh` replaced it. Fix whichever is active.
+
+**ACs:**
+- [ ] Daily sync completes without errors
+- [ ] New videos appear in dashboard
+
+**Impl Notes:**
+Check NAS cron logs. VPS sync fires every 6h.
+
+### T2: Verify tg-group-filter delivery
+
+**Description:**
+Cron switched from `deliver: local` to `deliver: all`. Confirm Telegram delivery works for filtered results.
+
+**ACs:**
+- [ ] Telegram receives filtered video digest
+- [ ] No duplicate deliveries

docs/troubleshooting.md

@@ -0,0 +1,194 @@
+# t-youtube Troubleshooting Guide
+
+## Common Issues
+
+### 1. Site Down / 502 Bad Gateway
+- **Cause**: t-youtube container not running or Authelia down
+- **Check**:
+  ```bash
+  ssh nas 'docker ps | grep t-youtube'
+  ssh nas 'docker ps | grep authelia'
+  ```
+- **Fix**:
+  ```bash
+  cd /volume1/docker/t-youtube && /usr/local/bin/docker compose up -d
+  cd /volume1/docker/authelia && /usr/local/bin/docker compose restart authelia
+  ```
+
+### 2. Authelia Loop / Login Fails
+- **Cause**: WebAuthn/TOTP misconfiguration or session expired
+- **Check Authelia logs**:
+  ```bash
+  ssh nas 'docker logs authelia --tail 20'
+  ```
+- **Fix**:
+  - Clear browser cache/cookies
+  - Reset WebAuthn via Authelia DB (see below)
+  - Check dnsmasq config (should point to VPS Tailscale IP)
+
+### 3. YouTube Data API Quota Exceeded
+- **Symptom**: Fetcher returns empty results or 403 errors
+- **Cause**: Daily quota limit reached (10,000 units)
+- **Check**:
+  ```bash
+  ssh nas 'docker logs t-youtube 2>&1 | grep -i "quota\|403"'
+  ```
+- **Fix**:
+  - Wait until next day (quota resets at 00:00 UTC)
+  - Reduce number of channels or increase sync interval
+  - Monitor usage in Google Cloud Console
+
+### 4. Summarization Failing
+- **Symptom**: Videos show "No summary" or generic descriptions
+- **Cause**: DeepSeek API key invalid or rate limited
+- **Check**:
+  ```bash
+  ssh nas 'docker logs t-youtube 2>&1 | grep -i "summariz\|deepseek"'
+  ```
+- **Fix**:
+  - Verify `DEEPSEEK_API_KEY` in `.env`
+  - Test API key manually:
+    ```bash
+    curl -s -X POST https://api.deepseek.com/chat/completions \
+      -H "Authorization: Bearer ***
+      -H "Content-Type: application/json" \
+      -d '{"model": "deepseek-chat", "messages": [{"role": "user", "content": "test"}]}'
+    ```
+
+### 5. Database Locked/Corrupted
+- **Symptom**: API returns 500 errors or hangs
+- **Cause**: Concurrent write operations or unexpected shutdown
+- **Fix**:
+  ```bash
+  cd /volume1/docker/t-youtube/backend
+  cp t_youtube.db t_youtube.db.backup
+  sqlite3 t_youtube.db "PRAGMA integrity_check;"
+  # If corrupted, restore from backup or recreate
+  ```
+
+### 6. Caddy Config Issues
+- **Symptom**: 502/503 errors, wrong cert, or domain unreachable
+- **Check**:
+  ```bash
+  ssh caddy-vps "sudo caddy validate --config /etc/caddy/Caddyfile"
+  ssh nas 'docker exec caddy caddy validate --config /etc/caddy/Caddyfile'
+  ```
+- **Fix**:
+  - Validate config
+  - Reload/restart Caddy
+  - Check TLS certs:
+    ```bash
+    echo | openssl s_client -connect 161.33.182.109:443 -servername yt.jimmygan.com 2>&1 | openssl x509 -noout -subject -dates
+    ```
+
+## API Debugging
+
+### Check Video List
+```bash
+curl -s https://yt.jimmygan.com/api/videos | python3 -m json.tool | head -20
+```
+
+### Check Agent Queue
+```bash
+curl -s -H "Authorization: Bearer *** https://yt.jimmygan.com/api/agent-queue
+```
+
+### Check Notes
+```bash
+curl -s -H "Authorization: Bearer *** https://yt.jimmygan.com/api/notes
+```
+
+### Check Channel List
+```bash
+curl -s -H "Authorization: Bearer *** https://yt.jimmygan.com/api/channels
+```
+
+### Test Summarization
+```bash
+curl -s -X POST -H "Authorization: Bearer ***
+  -H "Content-Type: application/json" \
+  -d '{"video_id": "<video_id>"}' \
+  https://yt.jimmygan.com/api/agent-queue/<video_id>/summarize
+```
+
+## Database Queries
+
+### Count Videos by Status
+```sql
+SELECT status, COUNT(*) FROM videos GROUP BY status;
+```
+
+### Count Agent Queue Items by Status
+```sql
+SELECT status, COUNT(*) FROM agent_queue GROUP BY status;
+```
+
+### Find Videos Without Summaries
+```sql
+SELECT video_id, title FROM videos WHERE summary IS NULL OR summary = '';
+```
+
+### Check DB Size
+```bash
+ls -lh /volume1/docker/t-youtube/backend/t_youtube.db
+```
+
+## Logs
+
+### t-youtube Logs
+```bash
+ssh nas 'docker logs t-youtube --tail 50'
+ssh nas 'docker logs t-youtube --tail 50 --since 1h'
+```
+
+### Authelia Logs
+```bash
+ssh nas 'docker logs authelia --tail 50'
+```
+
+### Sync Logs
+```bash
+ssh nas 'cat /var/log/t-youtube-sync.log'
+ssh nas 'tail -100 /var/log/t-youtube-sync.log'
+```
+
+### Caddy Logs (VPS)
+```bash
+ssh caddy-vps "sudo journalctl -u caddy --since '1 hour ago'"
+ssh caddy-vps "sudo journalctl -u caddy --no-pager -n 30"
+```
+
+## Recovery Procedures
+
+### Full System Restart
+```bash
+# 1. Restart services
+cd /volume1/docker/authelia && /usr/local/bin/docker compose restart authelia
+cd /volume1/docker/t-youtube && /usr/local/bin/docker compose up -d
+
+# 2. Verify
+curl -s https://yt.jimmygan.com/api/health
+curl -s https://auth.jimmygan.com/api/health
+```
+
+### Database Backup & Restore
+```bash
+# Backup
+cp /volume1/docker/t-youtube/backend/t_youtube.db /tmp/t-youtube-backup-$(date +%Y%m%d).db
+
+# Restore
+cp /tmp/t-youtube-backup-20260709.db /volume1/docker/t-youtube/backend/t_youtube.db
+cd /volume1/docker/t-youtube && /usr/local/bin/docker compose restart t-youtube
+```
+
+### Authelia User Reset (if locked out)
+```bash
+# Stop Authelia
+ssh nas 'docker stop authelia'
+
+# Reset TOTP for jimmy
+ssh nas 'sqlite3 /volume1/docker/authelia/config/db.sqlite3 "DELETE FROM totp_configurations WHERE username='\'jimmy'\'';"'
+
+# Restart Authelia
+ssh nas 'docker start authelia'
+```