ci: revert clone URL to http://gitea:3000 now that container network is properly configured

- Add video_notes table and agent_queue table for persistent notes and AI agent tasks
- Add API endpoints: POST/GET /api/notes/{id}, POST /api/notes/migrate, agent queue endpoints
- Rebuild frontend: fix duplicate loadVideos(), non-blocking localStorage migration, notes UI
- Switch pip install to Tsinghua mirror for faster downloads in China
- Add backend/sync_rotation.json to .gitignore (runtime state)

.gitea/workflows/deploy.yml

@@ -0,0 +1,43 @@
+name: Deploy t-youtube
+on:
+  push:
+    branches: [main]
+    paths:
+      - 'backend/**'
+      - 'frontend/**'
+      - 'Dockerfile'
+      - '.gitea/workflows/deploy.yml'
+  workflow_dispatch:
+
+jobs:
+  build-and-deploy:
+    name: Build & Deploy
+    runs-on: nas
+    container:
+      image: docker:cli
+      network: gitea_gitea
+      volumes:
+        - /volume1:/volume1
+    steps:
+      - name: Checkout
+        run: |
+          apk add --no-cache git
+          rm -rf /volume1/docker/t-youtube-build && git clone --depth=1 http://gitea:3000/jimmy/t-youtube.git /volume1/docker/t-youtube-build 2>&1
+
+      - name: Build Docker image
+        run: |
+          export DOCKER_API_VERSION=1.43
+          docker build -t t-youtube:latest /volume1/docker/t-youtube-build 2>&1
+
+      - name: Swap container on host
+        run: |
+          docker stop t-youtube 2>/dev/null || true
+          docker rm t-youtube 2>/dev/null || true
+          docker run -d --name t-youtube --restart unless-stopped             -p 8001:8000             --env-file /volume1/docker/t-youtube/.env             -v /volume1/docker/t-youtube/data:/app/data             -v /volume1/docker/t-youtube/config:/app/config             t-youtube:latest 2>&1
+
+      - name: Verify
+        run: |
+          export DOCKER_API_VERSION=1.43
+          sleep 3
+          docker ps --filter name=t-youtube --format "{{.Status}}"
+          echo "OK: deploy complete"

.gitignore

@@ -22,3 +22,5 @@ config/cookies.txt
 token.pickle
 config/client_secret.json
 config/token.pickle
+backend/sync_rotation.json
+._*

CLAUDE.md

@@ -0,0 +1,38 @@
+# T YouTube — Text-Based YouTube Digest
+
+## PDD Workflow
+
+This project follows **PDD (Product Definition Document)** development.
+
+**Always read these files before writing code:**
+- `docs/PDD.md` — product vision, goals, targets (the north star)
+- `docs/specs/active.md` — current sprint spec (what we're building now)
+
+**Workflow:**
+1. PDD → Sprint Spec → Implementation → Validate ACs
+2. Scope changes update the sprint spec first, code second
+3. Never ship without ACs passing
+
+## Dev Commands
+
+- **Build:** `docker compose up -d --build`
+- **Test:** `./run_tests.sh`
+- **Run locally:** `uvicorn backend.main:app --reload` (backend) + `npm run dev` (frontend)
+- **Sync subscriptions:** `python run_sync.py`
+
+## Architecture
+
+- **Frontend:** Svelte 5 + Vite (`frontend/`)
+- **Backend:** FastAPI + SQLite (`backend/`)
+- **Deployment:** Docker container on NAS DS224+, port 8001
+- **Sync:** `run_sync.py` fetches YouTube subscriptions + transcripts via yt-dlp
+- **Auth:** Authelia SSO in front, cookie-based session
+- **Proxy:** YouTube blocked from China — sync runs via VPS (Oracle Tokyo) or proxy
+
+## Key Files
+
+- `docs/architecture.md` — full architecture reference
+- `Dockerfile` — multi-stage build (node → python)
+- `docker-compose.yml` — deployment config
+- `run_sync.py` — subscription sync entry point
+- `config/cookies.txt` — YouTube auth cookies (not committed)

Dockerfile

@@ -13,16 +13,12 @@ WORKDIR /app
 # Install system dependencies
 RUN apt-get update && apt-get install -y --no-install-recommends \
     curl \
-    ffmpeg \
     && rm -rf /var/lib/apt/lists/*
 
-# Download latest official yt-dlp to ensure subscription scrapers do not break
-RUN curl -L https://github.com/yt-dlp/yt-dlp/releases/latest/download/yt-dlp -o /usr/local/bin/yt-dlp && \
-    chmod a+rx /usr/local/bin/yt-dlp
 
 # Install python packages
 COPY backend/requirements.txt ./
-RUN pip install --no-cache-dir -r requirements.txt
+RUN pip install --no-cache-dir -i https://pypi.tuna.tsinghua.edu.cn/simple -r requirements.txt
 
 # Copy built static frontend
 COPY --from=frontend-builder /app/frontend/dist ./frontend/dist

README.md

@@ -63,4 +63,44 @@ t-youtube/
     *   **Key Takeaways (Bullet list)**
     *   **Actionable Verdict**
 3.  **Read/Skip State**: Mark videos as **Read** or **Skip** to clear them from your pending queue, allowing you to only keep track of what's unread.
-4.  **View Full Transcript**: Expand the accordion at the bottom of the digest to read the raw transcripts if needed.
+4. **View Full Transcript**: Expand the accordion at the bottom of the digest to read the raw transcripts if needed.
+5. **Agent Queue**: Save videos with notes to a queue for Hermes agents to process (create skills, open issues, etc.)
+
+---
+
+## 📚 Documentation
+
+Comprehensive documentation for this project is available in the `docs/` folder:
+
+- **[Architecture](docs/architecture.md)** - System overview, components, API endpoints, database schema
+- **[Progress](docs/progress.md)** - Session history, completed tasks, planned features, metrics
+- **[Configuration](docs/configuration.md)** - Environment variables, API tokens, database, Authelia, Caddy setup
+- **[Troubleshooting](docs/troubleshooting.md)** - Common issues, recovery procedures, monitoring
+
+---
+
+## 🚀 Setup Instructions
+
+### 1. Extract your YouTube browser cookies
+Since YouTube restricts subscription feeds to logged-in users, you need to export your logged-in cookies so `yt-dlp` can request the feed on your behalf:
+1. Install a browser extension like **Get cookies.txt LOCALLY** (Chrome/Edge) or **cookies.txt** (Firefox).
+2. Open YouTube in your browser and ensure you are logged in to your account.
+3. Click the extension icon and export the cookies for `youtube.com` as a Netscape-formatted text file.
+4. Create a folder named `config` in your project root, rename the exported file to `cookies.txt`, and save it inside:
+   ```text
+   t-youtube/config/cookies.txt
+   ```
+
+### 2. Configure Environment Variables
+Create a `.env` file in the root directory:
+```env
+# Optional: Override the fallback OpenAI / Anthropic key
+OPENAI_API_KEY=sk-Kq5...rOlx
+```
+
+### 3. Build & Deploy using Docker Compose
+Run the following command to build the multi-stage image (compiles frontend static assets using Node, then packages backend with Python) and start the container:
+```bash
+docker compose up -d --build
+```
+The server will boot up and listen on port `8000`.

backend/agent_worker.py

@@ -0,0 +1,166 @@
+"""
+Agent Queue Worker — polls agent_queue for pending items and
+processes them against the local Qwen27B API (:8080).
+"""
+import asyncio
+import json
+import logging
+import os
+import aiosqlite
+import httpx
+
+import db
+
+log = logging.getLogger(__name__)
+
+QWEN_BASE_URL = os.environ.get("QWEN_BASE_URL", "http://localhost:8080")
+QWEN_MODEL = os.environ.get("QWEN_MODEL", "current")
+POLL_INTERVAL = int(os.environ.get("AGENT_POLL_INTERVAL", "30"))
+MAX_CONCURRENT = int(os.environ.get("AGENT_MAX_CONCURRENT", "3"))
+SYSTEM_PROMPT = os.environ.get(
+    "AGENT_SYSTEM_PROMPT",
+    "You are a helpful AI assistant. The user has watched a YouTube video and "
+    "left you instructions. Read the video info below and respond to their "
+    "prompt. Be concise and actionable."
+)
+
+semaphore = asyncio.Semaphore(MAX_CONCURRENT)
+
+
+async def _call_qwen(prompt: str, video_title: str, video_context: str) -> str:
+    """Call the local Qwen27B API with video context + user prompt."""
+    messages = [
+        {"role": "system", "content": SYSTEM_PROMPT},
+        {
+            "role": "user",
+            "content": (
+                f"## Video Title\n{video_title}\n\n"
+                f"## Video Context\n{video_context}\n\n"
+                f"## User's Request\n{prompt}"
+            ),
+        },
+    ]
+
+    http_proxy = os.environ.get("HTTP_PROXY") or os.environ.get("http_proxy")
+    https_proxy = os.environ.get("HTTPS_PROXY") or os.environ.get("https_proxy")
+    proxies = http_proxy or https_proxy
+
+    async with httpx.AsyncClient(proxy=proxies, timeout=180) as client:
+        resp = await client.post(
+            f"{QWEN_BASE_URL}/v1/chat/completions",
+            json={
+                "model": QWEN_MODEL,
+                "messages": messages,
+                "max_tokens": 2048,
+                "temperature": 0.3,
+            },
+        )
+        resp.raise_for_status()
+        data = resp.json()
+        return data["choices"][0]["message"]["content"]
+
+
+async def _process_item(item: dict) -> None:
+    """Process a single agent_queue item."""
+    video_id = item["video_id"]
+    user_prompt = item["user_prompt"]
+
+    async with aiosqlite.connect(db.DB_PATH) as conn:
+        conn.row_factory = aiosqlite.Row
+
+        # Mark as processing
+        await conn.execute(
+            "UPDATE agent_queue SET status = 'processing' WHERE id = ?",
+            (item["id"],),
+        )
+        await conn.commit()
+
+        # Get video info
+        cursor = await conn.execute(
+            "SELECT title, channel_name, summary, transcript FROM videos WHERE video_id = ?",
+            (video_id,),
+        )
+        video = await cursor.fetchone()
+
+    if not video:
+        async with aiosqlite.connect(db.DB_PATH) as conn:
+            await conn.execute(
+                "UPDATE agent_queue SET status = 'failed', result = 'Video not found in DB' WHERE id = ?",
+                (item["id"],),
+            )
+            await conn.commit()
+        return
+
+    # Build video context from whatever we have
+    context_parts = []
+    if video["channel_name"]:
+        context_parts.append(f"Channel: {video['channel_name']}")
+    if video["summary"] and not video["summary"].startswith("📺 Short clip"):
+        context_parts.append(f"Summary:\n{video['summary']}")
+    if video["transcript"] and video["transcript"] != "No transcript available":
+        context_parts.append(f"Transcript (first 4000 chars):\n{video['transcript'][:4000]}")
+    video_context = "\n\n".join(context_parts) if context_parts else "No additional context available."
+
+    try:
+        result = await _call_qwen(user_prompt, video["title"], video_context)
+        status = "done"
+    except Exception as e:
+        log.warning(f"Qwen API call failed for item {item['id']}: {e}")
+        result = f"Error: {e}"
+        status = "failed"
+
+    async with aiosqlite.connect(db.DB_PATH) as conn:
+        await conn.execute(
+            "UPDATE agent_queue SET status = ?, result = ?, processed_at = CURRENT_TIMESTAMP WHERE id = ?",
+            (status, result, item["id"]),
+        )
+        await conn.commit()
+
+
+async def process_queue_loop() -> None:
+    """Main polling loop — runs forever."""
+    log.info(
+        f"Agent worker started: Qwen at {QWEN_BASE_URL}, "
+        f"poll every {POLL_INTERVAL}s, max {MAX_CONCURRENT} concurrent"
+    )
+
+    while True:
+        try:
+            async with aiosqlite.connect(db.DB_PATH) as conn:
+                conn.row_factory = aiosqlite.Row
+                cursor = await conn.execute(
+                    "SELECT id, video_id, user_prompt FROM agent_queue WHERE status = 'pending' ORDER BY created_at ASC LIMIT ?",
+                    (MAX_CONCURRENT,),
+                )
+                items = await cursor.fetchall()
+
+            if items:
+                log.info(f"Agent queue: processing {len(items)} item(s)")
+                tasks = [_process_item(dict(item)) for item in items]
+                await asyncio.gather(*tasks)
+
+        except Exception as e:
+            log.error(f"Agent worker loop error: {e}")
+
+        await asyncio.sleep(POLL_INTERVAL)
+
+
+async def start_worker():
+    """Entry point — called from main.py startup."""
+    task = asyncio.create_task(process_queue_loop())
+    log.info("Agent queue worker scheduled")
+    return task
+
+
+if __name__ == "__main__":
+    logging.basicConfig(
+        level=logging.INFO, format="%(asctime)s - %(levelname)s - %(message)s"
+    )
+
+    async def main():
+        from db import init_db
+        await init_db()
+        log.info("Agent worker running in foreground. Ctrl+C to stop.")
+        await process_queue_loop()
+
+    asyncio.run(main())

backend/db.py

@@ -17,12 +17,70 @@ async def init_db():
                 thumbnail_url TEXT,
                 publish_date TEXT,
                 duration INTEGER,
+                view_count INTEGER DEFAULT 0,
                 transcript TEXT,
                 summary TEXT,
                 status TEXT DEFAULT 'pending',
                 created_at TEXT DEFAULT CURRENT_TIMESTAMP
             )
         """)
+        await db.execute("""
+            CREATE TABLE IF NOT EXISTS video_notes (
+                video_id TEXT PRIMARY KEY,
+                note_text TEXT NOT NULL,
+                updated_at TEXT DEFAULT CURRENT_TIMESTAMP,
+                FOREIGN KEY (video_id) REFERENCES videos(video_id)
+            )
+        """)
+        await db.execute("""
+            CREATE TABLE IF NOT EXISTS agent_queue (
+                id INTEGER PRIMARY KEY AUTOINCREMENT,
+                video_id TEXT NOT NULL,
+                user_prompt TEXT NOT NULL,
+                status TEXT DEFAULT 'pending',
+                result TEXT,
+                created_at TEXT DEFAULT CURRENT_TIMESTAMP,
+                processed_at TEXT,
+                FOREIGN KEY (video_id) REFERENCES videos(video_id)
+            )
+        """)
+        # Migration: Add result column if it doesn't exist (since the DB might already exist)
+        try:
+            async with db.execute("PRAGMA table_info(agent_queue)") as cursor:
+                columns = [row[1] for row in await cursor.fetchall()]
+                if "result" not in columns:
+                    await db.execute("ALTER TABLE agent_queue ADD COLUMN result TEXT")
+        except Exception as e:
+            # Table might not exist yet if not created, but CREATE TABLE IF NOT EXISTS handles it
+            pass
+        # FTS5 virtual table for full-text search on title + summary
+        await db.execute("""
+            CREATE VIRTUAL TABLE IF NOT EXISTS videos_fts USING fts5(
+                title, summary,
+                content='videos',
+                content_rowid='rowid'
+            )
+        """)
+        # Trigger to keep FTS5 index in sync
+        await db.execute("""
+            CREATE TRIGGER IF NOT EXISTS videos_fts_insert AFTER INSERT ON videos
+            BEGIN
+                INSERT INTO videos_fts(rowid, title, summary) VALUES (new.rowid, new.title, new.summary);
+            END;
+        """)
+        await db.execute("""
+            CREATE TRIGGER IF NOT EXISTS videos_fts_update AFTER UPDATE ON videos
+            BEGIN
+                INSERT INTO videos_fts(videos_fts, rowid, title, summary) VALUES ('delete', old.rowid, old.title, old.summary);
+                INSERT INTO videos_fts(rowid, title, summary) VALUES (new.rowid, new.title, new.summary);
+            END;
+        """)
+        await db.execute("""
+            CREATE TRIGGER IF NOT EXISTS videos_fts_delete AFTER DELETE ON videos
+            BEGIN
+                INSERT INTO videos_fts(videos_fts, rowid, title, summary) VALUES ('delete', old.rowid, old.title, old.summary);
+            END;
+        """)
         await db.commit()
 
 async def get_db():

backend/fetcher.py

@@ -1,6 +1,12 @@
 import asyncio
+import os
+import json
+import html
 import logging
 import aiosqlite
+import httpx
+import xml.etree.ElementTree as ET
+import re
 from googleapiclient.discovery import build
 from googleapiclient.errors import HttpError
 from auth import get_credentials
@@ -8,28 +14,86 @@ from db import DB_PATH
 
 log = logging.getLogger(__name__)
 
+ATOM_NS = "http://www.w3.org/2005/Atom"
+
+CHANNELS_PER_SYNC = 50  # Rotate through all channels — N per sync
+_ROTATION_FILE = os.path.join(os.environ.get("DB_DIR", os.path.dirname(os.path.abspath(__file__))), "sync_rotation.json")
+
+
+def _get_rotation_slice(total: int) -> tuple[int, int]:
+    """Return (start, end) indices for this sync's channel slice."""
+    state = {"offset": 0, "total": total}
+    if os.path.exists(_ROTATION_FILE):
+        try:
+            with open(_ROTATION_FILE) as f:
+                state = json.load(f)
+        except (json.JSONDecodeError, OSError):
+            pass
+    # If total changed (subscribed/unsubscribed), reset
+    if state.get("total") != total:
+        state = {"offset": 0, "total": total}
+    start = state["offset"]
+    end = min(start + CHANNELS_PER_SYNC, total)
+    return start, end
+
+
+def _save_rotation_slice(total: int, start: int, end: int):
+    """Persist next sync offset."""
+    next_offset = end  # Continue from where we left off
+    if next_offset >= total:
+        next_offset = 0  # Wrap around — full cycle complete
+    os.makedirs(os.path.dirname(_ROTATION_FILE), exist_ok=True)
+    with open(_ROTATION_FILE, "w") as f:
+        json.dump({"offset": next_offset, "total": total}, f)
+
+async def _fetch_rss(client, channel_id):
+    """Fetch latest videos from a channel's RSS feed (free). Returns list of (video_id, title)."""
+    url = f"https://www.youtube.com/feeds/videos.xml?channel_id={channel_id}"
+    try:
+        resp = await client.get(url, timeout=15)
+        if resp.status_code != 200:
+            return []
+        root = ET.fromstring(resp.text)
+        entries = []
+        for entry in root.findall(f"{{{ATOM_NS}}}entry"):
+            vid = entry.find(f"{{{ATOM_NS}}}link").get("href").split("=")[-1]
+            title_el = entry.find(f"{{{ATOM_NS}}}title")
+            title = html.unescape(title_el.text) if title_el is not None and title_el.text else "Untitled"
+            entries.append((vid, title))
+        return entries
+    except Exception as e:
+        log.warning(f"RSS error for {channel_id}: {e}")
+        return []
+
+
+
+def parse_duration(duration_str: str) -> int:
+    """Parse ISO 8601 duration string (e.g., PT1H2M10S) to seconds."""
+    if not duration_str:
+        return 0
+    pattern = re.compile(r'PT(?:(\d+)H)?(?:(\d+)M)?(?:(\d+)S)?')
+    match = pattern.match(duration_str)
+    if not match:
+        return 0
+    hours = int(match.group(1)) if match.group(1) else 0
+    minutes = int(match.group(2)) if match.group(2) else 0
+    seconds = int(match.group(3)) if match.group(3) else 0
+    return hours * 3600 + minutes * 60 + seconds
+
 async def fetch_subscriptions():
-    """Fetch latest videos from subscribed channels using YouTube Data API v3."""
+    """Fetch latest videos from subscribed channels using RSS + YouTube Data API v3."""
     try:
         creds = get_credentials()
-    except FileNotFoundError as e:
-        log.error(f"OAuth setup incomplete: {e}")
-        return 0
     except Exception as e:
         log.error(f"OAuth credential error: {e}")
         return 0
 
     youtube = build("youtube", "v3", credentials=creds)
 
-    # Step 1: Get subscribed channel IDs
+    # Step 1: Get subscribed channel IDs (1 unit)
     channel_ids = []
     try:
-        request = youtube.subscriptions().list(
+        request = youtube.subscriptions().list(part="snippet", mine=True, maxResults=50, order="alphabetical")
-            part="snippet",
-            mine=True,
-            maxResults=50,
-            order="alphabetical"
-        )
         while request:
             response = request.execute()
             for item in response.get("items", []):
@@ -38,61 +102,102 @@ async def fetch_subscriptions():
                     channel_ids.append(cid)
             request = youtube.subscriptions().list_next(request, response)
     except HttpError as e:
-        log.error(f"YouTube API error fetching subscriptions: {e}")
+        log.error(f"YouTube API error: {e}")
         return 0
 
     log.info(f"Found {len(channel_ids)} subscribed channels")
 
-    if not channel_ids:
+    # Step 2: Rotate — only process CHANNELS_PER_SYNC channels per sync
-        log.warning("No subscribed channels found. Make sure the authenticated account has subscriptions.")
+    start, end = _get_rotation_slice(len(channel_ids))
+    slice_ids = channel_ids[start:end]
+    log.info(f"Processing channels {start+1}–{end} of {len(channel_ids)} ({len(slice_ids)} this sync)")
+
+    # Step 3: Fetch RSS from slice channels (free, parallel)
+    rss_videos = {}
+    async with httpx.AsyncClient(
+        proxy=os.environ.get("HTTP_PROXY") or os.environ.get("http_proxy") or
+              os.environ.get("HTTPS_PROXY") or os.environ.get("https_proxy") or None,
+        timeout=20
+    ) as client:
+        tasks = [_fetch_rss(client, cid) for cid in slice_ids]
+        results = await asyncio.gather(*tasks)
+        for cid, entries in zip(slice_ids, results):
+            if entries:
+                rss_videos[cid] = entries
+
+    all_video_ids = []
+    rss_map = {}
+    for cid, entries in rss_videos.items():
+        for vid, title in entries:
+            if vid not in all_video_ids:
+                all_video_ids.append(vid)
+            rss_map[vid] = (cid, title)
+
+    log.info(f"RSS discovered {len(all_video_ids)} videos from {len(rss_videos)} channels")
+
+    if not all_video_ids:
+        _save_rotation_slice(len(channel_ids), start, end)
         return 0
 
-    # Step 2: For each channel, get latest videos
+    # Step 3: Check which videos are new (in SQLite)
+    async with aiosqlite.connect(DB_PATH) as db:
+        new_ids = []
+        for vid in all_video_ids:
+            cursor = await db.execute("SELECT 1 FROM videos WHERE video_id = ?", (vid,))
+            if not await cursor.fetchone():
+                new_ids.append(vid)
+
+    log.info(f"{len(new_ids)} new videos to fetch metadata for")
+
+    if not new_ids:
+        _save_rotation_slice(len(channel_ids), start, end)
+        return 0
+
+    # Step 4: Fetch metadata via videos.list (1 unit per 50 videos)
     new_count = 0
     async with aiosqlite.connect(DB_PATH) as db:
-        for cid in channel_ids[:20]:
+        for i in range(0, len(new_ids), 50):
+            batch = new_ids[i:i+50]
             try:
-                req = youtube.search().list(
+                resp = youtube.videos().list(part="snippet,contentDetails,statistics", id=",".join(batch)).execute()
-                    part="snippet",
-                    channelId=cid,
-                    maxResults=10,
-                    order="date",
-                    type="video"
-                )
-                resp = req.execute()
                 for item in resp.get("items", []):
-                    vid = item["id"]["videoId"]
+                    vid = item["id"]
                     snippet = item["snippet"]
+                    cid, rss_title = rss_map.get(vid, (snippet.get("channelId", ""), snippet.get("title", "")))
 
-                    cursor = await db.execute(
+                    content_details = item.get("contentDetails", {})
-                        "SELECT 1 FROM videos WHERE video_id = ?", (vid,)
+                    statistics = item.get("statistics", {})
-                    )
+                    duration_str = content_details.get("duration", "")
-                    if await cursor.fetchone():
+                    duration_secs = parse_duration(duration_str)
-                        continue
+                    view_count = int(statistics.get("viewCount", 0))
 
-                    pub_time = snippet.get("publishTime", "")
                     await db.execute("""
                         INSERT INTO videos
-                        (video_id, title, channel_name, channel_id, url, thumbnail_url, publish_date)
+                        (video_id, title, channel_name, channel_id, url, thumbnail_url, publish_date, duration, view_count)
-                        VALUES (?, ?, ?, ?, ?, ?, ?)
+                        VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)
                     """, (
                         vid,
-                        snippet.get("title", "Untitled"),
+                        html.unescape(snippet.get("title", rss_title)),
-                        snippet.get("channelTitle", "Unknown"),
+                        html.unescape(snippet.get("channelTitle", "Unknown")),
                         snippet.get("channelId", cid),
                         f"https://www.youtube.com/watch?v={vid}",
                         snippet.get("thumbnails", {}).get("high", {}).get("url", ""),
-                        pub_time[:10] if pub_time else None
+                        (snippet.get("publishTime") or "")[:10],
+                        duration_secs,
+                        view_count
                     ))
                     new_count += 1
-                    log.debug(f"New video: {snippet.get('title', 'Untitled')[:50]}")
             except HttpError as e:
-                log.warning(f"Channel {cid} fetch error: {e}")
+                log.warning(f"videos.list error: {e}")
                 continue
 
         await db.commit()
 
     log.info(f"Ingestion completed. Found {new_count} new videos.")
+
+    # Save rotation state for next sync
+    _save_rotation_slice(len(channel_ids), start, end)
+
     return new_count
 
 

backend/main.py

@@ -1,6 +1,9 @@
 import os
+import json
 import logging
-from fastapi import FastAPI, BackgroundTasks, Depends, HTTPException, Query
+from fastapi import FastAPI, BackgroundTasks, Depends, HTTPException, Query, Response
+from fastapi import Request
+from fastapi.responses import JSONResponse
 from fastapi.middleware.cors import CORSMiddleware
 from fastapi.staticfiles import StaticFiles
 from pydantic import BaseModel
@@ -10,33 +13,91 @@ import aiosqlite
 from db import init_db, get_db
 from fetcher import fetch_subscriptions
 from summarizer import summarize_video, summarize_all_pending
+from agent_worker import start_worker
 
 logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')
 log = logging.getLogger(__name__)
 
 app = FastAPI(title="T YouTube", description="Text-based YouTube Summarizer")
 
-# Enable CORS for development
 app.add_middleware(
     CORSMiddleware,
     allow_origins=["*"],
     allow_credentials=True,
     allow_methods=["*"],
     allow_headers=["*"],
+
 )
 
+# API token validation — disabled. Authelia (Caddy forward_auth) handles auth upstream.
+API_TOKEN=""
+
+async def validate_api_token(request: Request, call_next):
+    path = request.url.path
+    method = request.method
+    # Only check auth for write operations on /api/*
+    # Skip token check if Authelia/Caddy already authenticated the user
+    if path.startswith("/api/") and method in ("POST", "PATCH", "DELETE") and API_TOKEN:
+        # Check if request came through Authelia (Remote-User header set by Caddy forward_auth)
+        if request.headers.get("Remote-User"):
+            return await call_next(request)
+        auth_header = request.headers.get("Authorization", "")
+        if not auth_header.startswith("Bearer ") or auth_header[7:] != API_TOKEN:
+            return JSONResponse(status_code=401, content={"detail": "Unauthorized"})
+    return await call_next(request)
+
+app.middleware("http")(validate_api_token)
+
+
 class StatusUpdate(BaseModel):
     status: str
 
-# Database Startup Hook
+class NoteUpdate(BaseModel):
+    note_text: str
+    queue_to_agent: bool = False
+
+class NoteMigrate(BaseModel):
+    notes: dict
+
+class AgentQueueItem(BaseModel):
+    video_id: str
+    user_prompt: str
+
+class AgentQueueUpdate(BaseModel):
+    status: str
+
 @app.on_event("startup")
 async def startup():
     await init_db()
+    if os.environ.get("AGENT_ENABLED", "").lower() == "true":
+        await start_worker()
+        log.info("Agent queue worker enabled")
+    else:
+        log.info("Agent queue worker disabled (set AGENT_ENABLED=true to enable)")
 
 @app.get("/api/health")
 async def health():
     return {"status": "ok"}
 
+@app.get("/api/config")
+async def api_config():
+    """Return API config without exposing the token."""
+    return {"auth_enabled": bool(API_TOKEN)}
+
+@app.get("/auto_categories.json")
+async def auto_categories():
+    path = os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), "frontend", "src", "auto_categories.json")
+    if os.path.exists(path):
+        with open(path) as f:
+            return json.load(f)
+    return {}
+
+@app.get("/api/channels")
+async def get_channels(db: aiosqlite.Connection = Depends(get_db)):
+    cursor = await db.execute("SELECT DISTINCT channel_name, channel_id FROM videos ORDER BY channel_name")
+    rows = await cursor.fetchall()
+    return [dict(row) for row in rows]
+
 @app.get("/")
 async def root():
     from fastapi.responses import FileResponse
@@ -46,10 +107,51 @@ async def root():
 async def get_videos(
     status: Optional[str] = "pending",
     limit: int = 50,
+    offset: int = 0,
+    channel: Optional[str] = None,
+    search: Optional[str] = None,
     db: aiosqlite.Connection = Depends(get_db)
 ):
-    query = "SELECT * FROM videos WHERE status = ? ORDER BY publish_date DESC, created_at DESC LIMIT ?"
+    """Get videos with optional search filter and pagination."""
-    cursor = await db.execute(query, (status, limit))
+    if search and len(search) >= 2:
+        # Use search endpoint logic
+        q = search.replace('"', '\\"')
+        base_query = "SELECT v.* FROM videos_fts f JOIN videos v ON f.rowid = v.rowid WHERE videos_fts MATCH ?"
+        params = [q]
+        if status:
+            base_query += " AND v.status = ?"
+            params.append(status)
+        if channel:
+            base_query += " AND v.channel_name = ?"
+            params.append(channel)
+        base_query += " ORDER BY rank LIMIT ? OFFSET ?"
+        params.extend([limit, offset])
+    else:
+        if channel:
+            query = "SELECT * FROM videos WHERE status = ? AND channel_name = ? ORDER BY publish_date DESC, created_at DESC LIMIT ? OFFSET ?"
+            cursor = await db.execute(query, (status, channel, limit, offset))
+        else:
+            query = "SELECT * FROM videos WHERE status = ? ORDER BY publish_date DESC, created_at DESC LIMIT ? OFFSET ?"
+            cursor = await db.execute(query, (status, limit, offset))
+        rows = await cursor.fetchall()
+        return [dict(row) for row in rows]
+    
+    try:
+        cursor = await db.execute(base_query, params)
+    except Exception:
+        # Fallback to LIKE search
+        like_query = "SELECT * FROM videos WHERE title LIKE ?"
+        params = [f"%{search}%"]
+        if status:
+            like_query += " AND status = ?"
+            params.append(status)
+        if channel:
+            like_query += " AND channel_name = ?"
+            params.append(channel)
+        like_query += " ORDER BY publish_date DESC LIMIT ? OFFSET ?"
+        params.extend([limit, offset])
+        cursor = await db.execute(like_query, params)
+    
     rows = await cursor.fetchall()
     return [dict(row) for row in rows]
 
@@ -73,7 +175,6 @@ async def trigger_summarize(video_id: str, background_tasks: BackgroundTasks, db
     exists = await cursor.fetchone()
     if not exists:
         raise HTTPException(status_code=404, detail="Video not found")
-
     background_tasks.add_task(summarize_video, video_id)
     return {"message": f"Summarization for video {video_id} queued."}
 
@@ -85,17 +186,285 @@ async def update_status(
 ):
     if update.status not in ["pending", "read", "skipped"]:
         raise HTTPException(status_code=400, detail="Invalid status value")
-
     cursor = await db.execute("SELECT 1 FROM videos WHERE video_id = ?", (video_id,))
     exists = await cursor.fetchone()
     if not exists:
         raise HTTPException(status_code=404, detail="Video not found")
-
     await db.execute("UPDATE videos SET status = ? WHERE video_id = ?", (update.status, video_id))
     await db.commit()
     return {"message": f"Video {video_id} status updated to {update.status}."}
 
-# Mount frontend built files for production if they exist
+# ---- Notes (server-side, replaces localStorage) ----
+
+
+
+@app.get("/api/notes")
+async def list_notes(db: aiosqlite.Connection = Depends(get_db)):
+    cursor = await db.execute("SELECT video_id, note_text, updated_at FROM video_notes ORDER BY updated_at DESC")
+    rows = await cursor.fetchall()
+    return [dict(row) for row in rows]
+
+@app.get("/api/notes/{video_id}")
+async def get_note(video_id: str, db: aiosqlite.Connection = Depends(get_db)):
+    cursor = await db.execute("SELECT note_text, updated_at FROM video_notes WHERE video_id = ?", (video_id,))
+    row = await cursor.fetchone()
+    if not row:
+        return {"note_text": "", "updated_at": None}
+    return dict(row)
+
+@app.post("/api/notes/{video_id}")
+async def save_note(video_id: str, update: NoteUpdate, db: aiosqlite.Connection = Depends(get_db)):
+    await db.execute("""
+        INSERT INTO video_notes (video_id, note_text, updated_at)
+        VALUES (?, ?, CURRENT_TIMESTAMP)
+        ON CONFLICT(video_id) DO UPDATE SET
+            note_text = excluded.note_text,
+            updated_at = CURRENT_TIMESTAMP
+    """, (video_id, update.note_text))
+    await db.commit()
+
+    if update.queue_to_agent and update.note_text.strip():
+        await db.execute("""
+            INSERT INTO agent_queue (video_id, user_prompt, status)
+            VALUES (?, ?, 'pending')
+        """, (video_id, update.note_text.strip()))
+        await db.commit()
+        log.info(f"Queued video {video_id} for agent: {update.note_text[:80]}")
+
+    return {"message": "Note saved"}
+
+@app.post("/api/notes/migrate")
+async def migrate_notes(migration: NoteMigrate, db: aiosqlite.Connection = Depends(get_db)):
+    count = 0
+    for video_id, note_text in migration.notes.items():
+        if not note_text.strip():
+            continue
+        await db.execute("""
+            INSERT INTO video_notes (video_id, note_text, updated_at)
+            VALUES (?, ?, CURRENT_TIMESTAMP)
+            ON CONFLICT(video_id) DO UPDATE SET
+                note_text = excluded.note_text,
+                updated_at = CURRENT_TIMESTAMP
+        """, (video_id, note_text))
+        count += 1
+    await db.commit()
+    return {"message": f"{count} notes migrated"}
+
+# ---- Agent Queue ----
+
+@app.get("/api/agent-queue")
+async def list_agent_queue(
+    status: Optional[str] = None,
+    limit: int = 50,
+    db: aiosqlite.Connection = Depends(get_db)
+):
+    if status:
+        query = """
+            SELECT q.id, q.video_id, q.user_prompt, q.status, q.created_at, q.processed_at,
+                   v.title, v.channel_name, v.url, v.summary
+            FROM agent_queue q
+            LEFT JOIN videos v ON q.video_id = v.video_id
+            WHERE q.status = ?
+            ORDER BY q.created_at DESC LIMIT ?
+        """
+        cursor = await db.execute(query, (status, limit))
+    else:
+        query = """
+            SELECT q.id, q.video_id, q.user_prompt, q.status, q.created_at, q.processed_at,
+                   v.title, v.channel_name, v.url, v.summary
+            FROM agent_queue q
+            LEFT JOIN videos v ON q.video_id = v.video_id
+            ORDER BY q.created_at DESC LIMIT ?
+        """
+        cursor = await db.execute(query, (limit,))
+    rows = await cursor.fetchall()
+    return [dict(row) for row in rows]
+
+@app.post("/api/agent-queue")
+async def add_to_agent_queue(item: AgentQueueItem, db: aiosqlite.Connection = Depends(get_db)):
+    cursor = await db.execute("SELECT 1 FROM videos WHERE video_id = ?", (item.video_id,))
+    exists = await cursor.fetchone()
+    if not exists:
+        raise HTTPException(status_code=404, detail="Video not found")
+    await db.execute("""
+        INSERT INTO agent_queue (video_id, user_prompt, status)
+        VALUES (?, ?, 'pending')
+    """, (item.video_id, item.user_prompt))
+    await db.commit()
+    return {"message": "Item queued for agent"}
+
+@app.patch("/api/agent-queue/{item_id}")
+async def update_agent_queue(item_id: int, update: AgentQueueUpdate, db: aiosqlite.Connection = Depends(get_db)):
+    valid_statuses = ["pending", "processing", "done", "failed"]
+    if update.status not in valid_statuses:
+        raise HTTPException(status_code=400, detail=f"Invalid status. Must be one of: {valid_statuses}")
+    cursor = await db.execute("SELECT 1 FROM agent_queue WHERE id = ?", (item_id,))
+    exists = await cursor.fetchone()
+    if not exists:
+        raise HTTPException(status_code=404, detail="Queue item not found")
+    if update.status in ("done", "failed"):
+        await db.execute("""
+            UPDATE agent_queue SET status = ?, processed_at = CURRENT_TIMESTAMP WHERE id = ?
+        """, (update.status, item_id))
+    else:
+        await db.execute("UPDATE agent_queue SET status = ? WHERE id = ?", (update.status, item_id))
+    await db.commit()
+    return {"message": f"Queue item {item_id} updated to {update.status}"}
+
+@app.delete("/api/agent-queue/{item_id}")
+async def remove_from_agent_queue(item_id: int, db: aiosqlite.Connection = Depends(get_db)):
+    cursor = await db.execute("SELECT 1 FROM agent_queue WHERE id = ?", (item_id,))
+    exists = await cursor.fetchone()
+    if not exists:
+        raise HTTPException(status_code=404, detail="Queue item not found")
+    await db.execute("DELETE FROM agent_queue WHERE id = ?", (item_id,))
+    await db.commit()
+    return {"message": f"Queue item {item_id} removed"}
+
+# ---- Search (FTS5) ----
+
+@app.get("/api/videos/search")
+async def search_videos(
+    q: str = "",
+    status: Optional[str] = None,
+    limit: int = 50,
+    channel: Optional[str] = None,
+    db: aiosqlite.Connection = Depends(get_db)
+):
+    """Full-text search across video titles and summaries using FTS5."""
+    if not q or len(q) < 2:
+        return []
+    
+    # Try FTS5 first
+    fts_query = q.replace('"', '\"')
+    base_query = "SELECT v.* FROM videos_fts f JOIN videos v ON f.rowid = v.rowid WHERE videos_fts MATCH ?"
+    params = [fts_query]
+    
+    if status:
+        base_query += " AND v.status = ?"
+        params.append(status)
+    if channel:
+        base_query += " AND v.channel_name = ?"
+        params.append(channel)
+    base_query += " ORDER BY rank LIMIT ?"
+    params.append(limit)
+    
+    try:
+        cursor = await db.execute(base_query, params)
+    except Exception:
+        # Fallback to LIKE if FTS5 not available
+        like_query = "SELECT * FROM videos WHERE title LIKE ?"
+        params = [f"%{q}%"]
+        if status:
+            like_query += " AND status = ?"
+            params.append(status)
+        if channel:
+            like_query += " AND channel_name = ?"
+            params.append(channel)
+        like_query += " LIMIT ?"
+        params.append(limit)
+        cursor = await db.execute(like_query, params)
+    
+    rows = await cursor.fetchall()
+    return [dict(row) for row in rows]
+
+# ---- Batch Operations ----
+
+class BatchUpdate(BaseModel):
+    video_ids: list[str]
+    status: str
+
+@app.post("/api/videos/batch")
+async def batch_update(
+    batch: BatchUpdate,
+    db: aiosqlite.Connection = Depends(get_db)
+):
+    """Batch update video statuses."""
+    if not batch.video_ids:
+        return {"message": "No video IDs provided"}
+    if batch.status not in ["pending", "read", "skipped"]:
+        raise HTTPException(status_code=400, detail="Invalid status")
+    
+    placeholders = ",".join(["?" for _ in batch.video_ids])
+    await db.execute(f"UPDATE videos SET status = ? WHERE video_id IN ({placeholders})", 
+                     [batch.status] + batch.video_ids)
+    await db.commit()
+    
+    return {"message": f"Updated {len(batch.video_ids)} videos"}
+
+# ---- Stats Dashboard ----
+
+@app.get("/api/stats")
+async def get_stats(db: aiosqlite.Connection = Depends(get_db)):
+    """Return dashboard statistics."""
+    stats = {}
+    
+    # Total videos
+    cursor = await db.execute("SELECT COUNT(*) FROM videos")
+    stats["total"] = (await cursor.fetchone())[0]
+    
+    # Status breakdown
+    cursor = await db.execute("SELECT status, COUNT(*) FROM videos GROUP BY status")
+    stats["status_breakdown"] = {row[0]: row[1] for row in await cursor.fetchall()}
+    
+    # Channels
+    cursor = await db.execute("SELECT COUNT(DISTINCT channel_name) FROM videos")
+    stats["channels"] = (await cursor.fetchone())[0]
+    
+    # Videos with summaries vs without
+    cursor = await db.execute("SELECT COUNT(*) FROM videos WHERE summary IS NOT NULL AND summary != ''")
+    stats["with_summary"] = (await cursor.fetchone())[0]
+    
+    # Recent activity (last 7 days)
+    cursor = await db.execute("SELECT COUNT(*) FROM videos WHERE created_at >= datetime('now', '-7 days')")
+    stats["last_7_days"] = (await cursor.fetchone())[0]
+    
+    # Top channels by video count
+    cursor = await db.execute("SELECT channel_name, COUNT(*) as cnt FROM videos GROUP BY channel_name ORDER BY cnt DESC LIMIT 10")
+    stats["top_channels"] = [{"name": row[0], "count": row[1]} for row in await cursor.fetchall()]
+    
+    # Summary rate
+    if stats["total"] > 0:
+        stats["summary_rate"] = round(stats["with_summary"] / stats["total"] * 100, 1)
+    else:
+        stats["summary_rate"] = 0
+    
+    return stats
+
+# ---- Duration + View Count Update ----
+
+class VideoMetaUpdate(BaseModel):
+    duration: Optional[int] = None
+    view_count: Optional[int] = None
+
+@app.patch("/api/videos/{video_id}/meta")
+async def update_video_meta(
+    video_id: str,
+    meta: VideoMetaUpdate,
+    db: aiosqlite.Connection = Depends(get_db)
+):
+    """Update video metadata (duration, view count)."""
+    updates = []
+    params = []
+    if meta.duration is not None:
+        updates.append("duration = ?")
+        params.append(meta.duration)
+    if meta.view_count is not None:
+        updates.append("view_count = ?")
+        params.append(meta.view_count)
+    
+    if not updates:
+        return {"message": "No updates provided"}
+    
+    params.append(video_id)
+    await db.execute(f"UPDATE videos SET {', '.join(updates)} WHERE video_id = ?", params)
+    await db.commit()
+    return {"message": "Metadata updated"}
+
+# ---- Include view_count in video list ----
+
+# ---- Static frontend ----
+
 frontend_build_path = os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), "frontend/dist")
 if os.path.exists(frontend_build_path):
     app.mount("/", StaticFiles(directory=frontend_build_path, html=True), name="frontend")

backend/requirements.txt

@@ -1,6 +1,5 @@
 fastapi>=0.100.0
 uvicorn>=0.20.0
-yt-dlp>=2023.0.0
 youtube-transcript-api>=0.6.0
 aiosqlite>=0.18.0
 httpx>=0.24.0

backend/run_sync.sh

@@ -0,0 +1,25 @@
+#!/bin/bash
+set -e
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
+cd "$SCRIPT_DIR/.."
+source "$SCRIPT_DIR/../.env"
+"$SCRIPT_DIR/.venv/bin/python3" -c "
+import asyncio, sys, logging
+sys.path.insert(0, '$SCRIPT_DIR')
+from db import init_db
+from fetcher import fetch_subscriptions
+from summarizer import summarize_all_pending
+
+logging.basicConfig(level=logging.INFO, format='%(asctime)s %(levelname)s %(message)s')
+
+async def main():
+    await init_db()
+    count = await fetch_subscriptions()
+    print(f'Fetched {count} new videos')
+    if count > 0:
+        print(f'Summarizing {count} videos...')
+        await summarize_all_pending()
+    print('Done')
+
+asyncio.run(main())
+"

backend/summarizer.py

@@ -9,6 +9,11 @@ from db import DB_PATH
 
 log = logging.getLogger(__name__)
 
+def proxy_from_env():
+    """Return proxy URL from HTTP_PROXY/HTTPS_PROXY env vars, or None."""
+    return os.environ.get("HTTP_PROXY") or os.environ.get("http_proxy") or \
+           os.environ.get("HTTPS_PROXY") or os.environ.get("https_proxy") or None
+
 # Fallback pattern to retrieve the correct bytecatcode credentials
 API_KEY = os.environ.get("DEEPSEEK_API_KEY", os.environ.get("OPENAI_API_KEY", ""))
 BASE_URL = os.environ.get("DEEPSEEK_BASE_URL", "https://api.deepseek.com")
@@ -39,6 +44,10 @@ for p in [os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__)
 def download_transcript(video_id: str) -> str:
     try:
         session = requests.Session()
+        # Respect HTTP_PROXY/HTTPS_PROXY env vars
+        proxy = proxy_from_env()
+        if proxy:
+            session.proxies = {"http": proxy, "https": proxy}
         session.headers.update({
             "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36",
             "Accept-Language": "en-US,en;q=0.9"
@@ -62,9 +71,9 @@ def download_transcript(video_id: str) -> str:
         log.warning(f"Failed to fetch transcript for {video_id}: {e}")
         return None
 
-async def summarize_video(video_id: str) -> str:
+async def summarize_video(video_id: str) -> str | None:
-    # 1. Try transcript, fall back to description
+    # 1. Try transcript, fall back to description (offload sync call to thread)
-    transcript = download_transcript(video_id)
+    transcript = await asyncio.to_thread(download_transcript, video_id)
     
     if not transcript:
         # Fallback: use video description from YouTube API
@@ -108,7 +117,7 @@ async def summarize_video(video_id: str) -> str:
         ]
     }
 
-    async with httpx.AsyncClient(timeout=120) as client:
+    async with httpx.AsyncClient(proxy=proxy_from_env(), timeout=120) as client:
         try:
             resp = await client.post(
                 f"{BASE_URL}/v1/chat/completions",
@@ -120,7 +129,7 @@ async def summarize_video(video_id: str) -> str:
             summary = data["choices"][0]["message"]["content"]
         except Exception as e:
             log.error(f"DeepSeek API request failed for {video_id}: {e}")
-            summary = f"Error generating summary: {e}"
+            return None  # Leave summary NULL for retry on next sync
 
     # 3. Save transcript & summary to db
     async with aiosqlite.connect(DB_PATH) as db:

backend/tests/test_agent_worker.py

@@ -0,0 +1,112 @@
+"""Tests for agent_worker.py — mocks Qwen API."""
+import pytest
+import os
+import json
+import aiosqlite
+import httpx
+from unittest.mock import patch, AsyncMock, MagicMock
+
+
+@pytest.mark.asyncio
+async def test_agent_worker_polls_and_processes(test_db, monkeypatch):
+    """Worker picks up a pending item, calls Qwen, marks it done."""
+    from agent_worker import _process_item
+
+    # Seed a pending queue item + video
+    await test_db.execute("""
+        INSERT INTO videos (video_id, title, channel_name, url, summary, status)
+        VALUES ('ag_vid', 'Test Video', 'Test Channel', 'https://youtube.com/watch?v=ag_vid',
+                '**Summary**\n- Point 1\n- Point 2\nVerdict.', 'pending')
+    """)
+    await test_db.execute("""
+        INSERT INTO agent_queue (video_id, user_prompt, status)
+        VALUES ('ag_vid', 'Save this as a skill', 'pending')
+    """)
+    await test_db.commit()
+
+    # Get the item
+    cursor = await test_db.execute(
+        "SELECT id, video_id, user_prompt FROM agent_queue WHERE status = 'pending'"
+    )
+    item = dict(await cursor.fetchone())
+
+    # Mock Qwen API
+    mock_response = MagicMock(spec=httpx.Response)
+    mock_response.status_code = 200
+    mock_response.json.return_value = {
+        "choices": [{"message": {"content": "I've saved the skill. Done."}}]
+    }
+
+    with patch("httpx.AsyncClient.post", new_callable=AsyncMock, return_value=mock_response):
+        await _process_item(item)
+
+    # Verify
+    cursor = await test_db.execute(
+        "SELECT status, result FROM agent_queue WHERE id = ?", (item["id"],)
+    )
+    row = await cursor.fetchone()
+    assert row["status"] == "done"
+    assert "I've saved the skill" in row["result"]
+
+
+@pytest.mark.asyncio
+async def test_agent_worker_handles_missing_video(test_db):
+    """When video doesn't exist, marks as failed gracefully."""
+    from agent_worker import _process_item
+
+    await test_db.execute("""
+        INSERT INTO agent_queue (video_id, user_prompt, status)
+        VALUES ('missing_vid', 'Do something', 'pending')
+    """)
+    await test_db.commit()
+
+    cursor = await test_db.execute(
+        "SELECT id, video_id, user_prompt FROM agent_queue WHERE status = 'pending'"
+    )
+    item = dict(await cursor.fetchone())
+
+    await _process_item(item)
+
+    cursor = await test_db.execute(
+        "SELECT status, result FROM agent_queue WHERE id = ?", (item["id"],)
+    )
+    row = await cursor.fetchone()
+    assert row["status"] == "failed"
+    assert "not found" in row["result"].lower()
+
+
+@pytest.mark.asyncio
+async def test_agent_worker_qwen_error_marks_failed(test_db):
+    """Qwen API failure should mark as failed, not crash."""
+    from agent_worker import _process_item
+
+    await test_db.execute("""
+        INSERT INTO videos (video_id, title, channel_name, url, status)
+        VALUES ('err_vid', 'Error Video', 'Test', 'https://youtube.com/watch?v=err_vid', 'pending')
+    """)
+    await test_db.execute("""
+        INSERT INTO agent_queue (video_id, user_prompt, status)
+        VALUES ('err_vid', 'Do it', 'pending')
+    """)
+    await test_db.commit()
+
+    cursor = await test_db.execute(
+        "SELECT id, video_id, user_prompt FROM agent_queue WHERE status = 'pending'"
+    )
+    item = dict(await cursor.fetchone())
+
+    # Mock HTTP error
+    mock_response = MagicMock(spec=httpx.Response)
+    mock_response.status_code = 503
+    mock_response.raise_for_status.side_effect = httpx.HTTPStatusError(
+        "503 Service Unavailable", request=MagicMock(), response=mock_response
+    )
+
+    with patch("httpx.AsyncClient.post", new_callable=AsyncMock, return_value=mock_response):
+        await _process_item(item)
+
+    cursor = await test_db.execute(
+        "SELECT status FROM agent_queue WHERE id = ?", (item["id"],)
+    )
+    row = await cursor.fetchone()
+    assert row["status"] == "failed"

backend/tests/test_fetcher.py

@@ -1,112 +1,200 @@
+"""Tests for fetcher.py — mocks YouTube Data API v3 + RSS."""
 import pytest
 import os
 import json
 import aiosqlite
 import fetcher
-import asyncio
+from unittest.mock import patch, AsyncMock, MagicMock
-from unittest.mock import patch, AsyncMock
 
-class MockProcess:
-    def __init__(self, stdout_data, returncode=0, stderr_data=b""):
-        self.stdout_data = stdout_data
-        self.stderr_data = stderr_data
-        self.returncode = returncode
 
-    async def communicate(self):
+def _clean_rotation_file():
-        return self.stdout_data, self.stderr_data
+    """Remove rotation state file if it exists."""
+    path = getattr(fetcher, "_ROTATION_FILE", None)
+    if path and os.path.exists(path):
+        os.remove(path)
+
+
+@pytest.fixture(autouse=True)
+def auto_clean_rotation():
+    _clean_rotation_file()
+    yield
+    _clean_rotation_file()
+
+
+def _mock_youtube_build(sub_channels=None, video_data=None):
+    """Build a mock YouTube API service that returns canned responses."""
+    sub_channels = sub_channels or [
+        {"snippet": {"resourceId": {"channelId": "chan001"}}},
+        {"snippet": {"resourceId": {"channelId": "chan002"}}},
+    ]
+    video_data = video_data or {}
+
+    def mock_subscriptions_list(**kwargs):
+        return MagicMock(execute=MagicMock(return_value={"items": sub_channels}))
+
+    def mock_videos_list(**kwargs):
+        vid_str = kwargs.get("id", "")
+        ids = [v for v in vid_str.split(",") if v]
+        items = []
+        for vid in ids:
+            data = video_data.get(vid, {
+                "title": f"Video {vid}",
+                "channelTitle": "Test Channel",
+                "channelId": "chan001",
+                "thumbnails": {"high": {"url": f"https://img.youtube.com/vi/{vid}/hqdefault.jpg"}},
+                "publishTime": "2026-06-01T00:00:00Z",
+                "contentDetails": {"duration": "PT5M30S"},
+                "statistics": {"viewCount": "1250"},
+            })
+            snippet = {k: v for k, v in data.items() if k not in ("contentDetails", "statistics")}
+            content_details = data.get("contentDetails", {"duration": "PT5M30S"})
+            statistics = data.get("statistics", {"viewCount": "1250"})
+            items.append({
+                "id": vid,
+                "snippet": snippet,
+                "contentDetails": content_details,
+                "statistics": statistics
+            })
+        return MagicMock(execute=MagicMock(return_value={"items": items}))
+
+    svc = MagicMock()
+    svc.subscriptions = MagicMock(return_value=MagicMock(list=mock_subscriptions_list, list_next=MagicMock(return_value=None)))
+    svc.videos = MagicMock(return_value=MagicMock(list=mock_videos_list))
+    return svc
+
+
+def _mock_rss_response(videos):
+    """Build RSS XML string for a list of (video_id, title) tuples."""
+    entries = []
+    for vid, title in videos:
+        entries.append(f"""<entry>
+      <id>yt:video:{vid}</id>
+      <link href="https://www.youtube.com/watch?v={vid}"/>
+      <title>{title}</title>
+    </entry>""")
+    return f"""<?xml version="1.0" encoding="UTF-8"?>
+<feed xmlns="http://www.w3.org/2005/Atom">
+  {''.join(entries)}
+</feed>"""
+
 
 @pytest.mark.asyncio
-async def test_fetch_subscriptions_no_cookies(tmp_path, monkeypatch):
+async def test_fetch_zero_channels(test_db):
-    monkeypatch.setattr(fetcher, "COOKIES_PATH", "/non/existent/path/cookies.txt")
+    """When no subscriptions exist, should return 0."""
-    count = await fetcher.fetch_subscriptions()
+    mock_svc = _mock_youtube_build(sub_channels=[])
-    assert count == 0
+    with patch("fetcher.build", return_value=mock_svc), \
-
+         patch("fetcher.get_credentials", return_value="mock_creds"):
-@pytest.mark.asyncio
-async def test_fetch_subscriptions_success(test_db, tmp_path, monkeypatch):
-    cookies_file = os.path.join(tmp_path, "mock_cookies.txt")
-    with open(cookies_file, "w") as f:
-        f.write("# Netscape HTTP Cookie File")
-    monkeypatch.setattr(fetcher, "COOKIES_PATH", cookies_file)
-
-    video1 = {
-        "id": "vid123",
-        "title": "Mock Video 1",
-        "uploader": "Uploader One",
-        "uploader_id": "chan123",
-        "upload_date": "20260601",
-        "duration": 360,
-        "thumbnail": "https://img.youtube.com/vi/vid123/hqdefault.jpg"
-    }
-    video2 = {
-        "id": "vid456",
-        "title": "Mock Video 2",
-        "uploader": "Uploader Two",
-        "uploader_id": "chan456",
-        "upload_date": "20260602",
-        "duration": 720
-    }
-    stdout_bytes = (json.dumps(video1) + "\n" + json.dumps(video2) + "\n").encode("utf-8")
-
-    mock_process = MockProcess(stdout_data=stdout_bytes, returncode=0)
-
-    with patch("asyncio.create_subprocess_exec", return_value=mock_process) as mock_exec:
         count = await fetcher.fetch_subscriptions()
+        assert count == 0
 
-        mock_exec.assert_called_once()
+
-        args = mock_exec.call_args[0]
+@pytest.mark.asyncio
-        assert "yt-dlp" in args
+async def test_fetch_rss_empty(test_db):
-        assert "https://www.youtube.com/feed/subscriptions" in args
+    """When RSS returns nothing, count should be 0."""
-        assert cookies_file in args
+    mock_svc = _mock_youtube_build()
+    with patch("fetcher.build", return_value=mock_svc), \
+         patch("fetcher.get_credentials", return_value="mock_creds"), \
+         patch("httpx.AsyncClient.get", AsyncMock(return_value=MagicMock(status_code=404))):
+        count = await fetcher.fetch_subscriptions()
+        assert count == 0
+
+
+@pytest.mark.asyncio
+async def test_fetch_new_videos(test_db):
+    """New videos from RSS should be inserted into DB."""
+    mock_svc = _mock_youtube_build(video_data={
+        "vid001": {"title": "First Video", "channelTitle": "Chan One",
+                    "channelId": "chan001", "thumbnails": {"high": {"url": ""}},
+                    "publishTime": "2026-06-01T00:00:00Z"},
+        "vid002": {"title": "Second Video", "channelTitle": "Chan Two",
+                    "channelId": "chan002", "thumbnails": {"high": {"url": ""}},
+                    "publishTime": "2026-06-02T00:00:00Z"},
+    })
+    rss_xml = _mock_rss_response([("vid001", "First Video"), ("vid002", "Second Video")])
+    mock_response = MagicMock(status_code=200, text=rss_xml)
+
+    with patch("fetcher.build", return_value=mock_svc), \
+         patch("fetcher.get_credentials", return_value="mock_creds"), \
+         patch("httpx.AsyncClient.get", AsyncMock(return_value=mock_response)):
+        count = await fetcher.fetch_subscriptions()
         assert count == 2
 
-    # Query db to verify values
+    # Verify DB
     cursor = await test_db.execute("SELECT * FROM videos ORDER BY video_id")
-    videos = await cursor.fetchall()
+    rows = await cursor.fetchall()
-    assert len(videos) == 2
+    assert len(rows) == 2
+    assert rows[0]["video_id"] == "vid001"
+    assert rows[0]["title"] == "First Video"
+    assert rows[0]["duration"] == 330  # PT5M30S = 330s
+    assert rows[0]["view_count"] == 1250
+    assert rows[1]["video_id"] == "vid002"
+    assert rows[1]["title"] == "Second Video"
+    assert rows[1]["duration"] == 330
+    assert rows[1]["view_count"] == 1250
 
-    assert videos[0]["video_id"] == "vid123"
-    assert videos[0]["title"] == "Mock Video 1"
-    assert videos[0]["channel_name"] == "Uploader One"
-    assert videos[0]["channel_id"] == "chan123"
-    assert videos[0]["url"] == "https://www.youtube.com/watch?v=vid123"
-    assert videos[0]["thumbnail_url"] == "https://img.youtube.com/vi/vid123/hqdefault.jpg"
-    assert videos[0]["publish_date"] == "2026-06-01"
-    assert videos[0]["duration"] == 360
-    assert videos[0]["status"] == "pending"
-
-    assert videos[1]["video_id"] == "vid456"
-    assert videos[1]["title"] == "Mock Video 2"
-    assert videos[1]["publish_date"] == "2026-06-02"
-    assert videos[1]["duration"] == 720
 
 @pytest.mark.asyncio
-async def test_fetch_subscriptions_skip_duplicates(test_db, tmp_path, monkeypatch):
+async def test_fetch_skip_duplicates(test_db):
-    cookies_file = os.path.join(tmp_path, "mock_cookies.txt")
+    """Existing videos should be skipped."""
-    with open(cookies_file, "w") as f:
+    # Pre-insert one video
-        f.write("# Netscape HTTP Cookie File")
+    await test_db.execute(
-    monkeypatch.setattr(fetcher, "COOKIES_PATH", cookies_file)
+        "INSERT INTO videos (video_id, title, url, status) VALUES ('vid001', 'Old', 'https://youtube.com/watch?v=vid001', 'pending')"
-
+    )
-    # Insert an existing video record first
-    await test_db.execute("""
-        INSERT INTO videos (video_id, title, url, status)
-        VALUES ('vid123', 'Mock Video 1', 'https://www.youtube.com/watch?v=vid123', 'pending')
-    """)
     await test_db.commit()
 
-    video1 = {
+    mock_svc = _mock_youtube_build(video_data={
-        "id": "vid123",
+        "vid001": {"title": "First Video (updated)", "channelTitle": "Chan One",
-        "title": "Mock Video 1",
+                    "channelId": "chan001"},
-        "upload_date": "20260601"
+        "vid002": {"title": "Second Video", "channelTitle": "Chan Two",
-    }
+                    "channelId": "chan002"},
-    video2 = {
+    })
-        "id": "vid456",
+    rss_xml = _mock_rss_response([("vid001", "First Video"), ("vid002", "Second Video")])
-        "title": "Mock Video 2",
+    mock_response = MagicMock(status_code=200, text=rss_xml)
-        "upload_date": "20260602"
-    }
-    stdout_bytes = (json.dumps(video1) + "\n" + json.dumps(video2) + "\n").encode("utf-8")
 
-    mock_process = MockProcess(stdout_data=stdout_bytes, returncode=0)
+    with patch("fetcher.build", return_value=mock_svc), \
-
+         patch("fetcher.get_credentials", return_value="mock_creds"), \
-    with patch("asyncio.create_subprocess_exec", return_value=mock_process):
+         patch("httpx.AsyncClient.get", AsyncMock(return_value=mock_response)):
         count = await fetcher.fetch_subscriptions()
-        assert count == 1 # Only 1 new video should be added (vid456)
+        assert count == 1  # Only vid002 is new
+
+    # Verify DB — vid001 still has original title (not updated)
+    cursor = await test_db.execute("SELECT title FROM videos WHERE video_id='vid001'")
+    row = await cursor.fetchone()
+    assert row["title"] == "Old"
+
+
+@pytest.mark.asyncio
+async def test_rotation_advances(test_db):
+    """Rotation slice should advance after each sync."""
+    # Mock 150 channels
+    sub_channels = [{"snippet": {"resourceId": {"channelId": f"chan{i:03d}"}}} for i in range(150)]
+    mock_svc = _mock_youtube_build(sub_channels=sub_channels)
+
+    rss_xml = _mock_rss_response([])  # No videos, just test rotation
+    mock_response = MagicMock(status_code=200, text=rss_xml)
+
+    with patch("fetcher.build", return_value=mock_svc), \
+         patch("fetcher.get_credentials", return_value="mock_creds"), \
+         patch("httpx.AsyncClient.get", AsyncMock(return_value=mock_response)):
+        # First sync — should process chan000-chan049
+        count = await fetcher.fetch_subscriptions()
+        # Second sync — chan050-chan099
+        count2 = await fetcher.fetch_subscriptions()
+        # Third sync — chan100-chan149
+        count3 = await fetcher.fetch_subscriptions()
+        # Fourth sync — wrap around, chan000-chan049 again
+        count4 = await fetcher.fetch_subscriptions()
+
+    # Verify rotation file
+    with open(fetcher._ROTATION_FILE) as f:
+        state = json.load(f)
+    assert state["offset"] == 50  # After 4th sync, offset = 50 (wrapped from 150)
+    assert state["total"] == 150
+
+
+@pytest.mark.asyncio
+async def test_fetch_quietly_handles_missing_credentials(test_db):
+    """When OAuth is missing, should return 0 gracefully."""
+    with patch("fetcher.get_credentials", side_effect=FileNotFoundError("No client_secret")):
+        count = await fetcher.fetch_subscriptions()
+        assert count == 0

backend/tests/test_summarizer.py

@@ -1,3 +1,4 @@
+"""Tests for summarizer.py — mocks transcript API + DeepSeek API."""
 import pytest
 import os
 import httpx
@@ -5,6 +6,7 @@ import aiosqlite
 import summarizer
 from unittest.mock import patch, MagicMock, AsyncMock
 
+
 class MockTranscript:
     def fetch(self):
         return [
@@ -25,6 +27,17 @@ class MockTranscriptList:
     def __iter__(self):
         return iter([self.transcript])
 
+
+def _mock_deepseek_response(content: str) -> MagicMock:
+    """Build a DeepSeek/OpenAI-compatible API response."""
+    r = MagicMock(spec=httpx.Response)
+    r.status_code = 200
+    r.json.return_value = {
+        "choices": [{"message": {"content": content}}]
+    }
+    return r
+
+
 @pytest.mark.asyncio
 async def test_download_transcript_success():
     mock_trans = MockTranscript()
@@ -34,9 +47,9 @@ async def test_download_transcript_success():
         text = summarizer.download_transcript("test_vid")
         assert text == "Welcome to this video tutorial. Today we are learning about Svelte 5 and runes."
 
+
 @pytest.mark.asyncio
 async def test_summarize_video_success(test_db):
-    # Insert a pending video in db first
     await test_db.execute("""
         INSERT INTO videos (video_id, title, url, status)
         VALUES ('summarize_vid', 'Video to Summarize', 'https://www.youtube.com/watch?v=summarize_vid', 'pending')
@@ -45,19 +58,8 @@ async def test_summarize_video_success(test_db):
 
     mock_trans = MockTranscript()
     mock_list = MockTranscriptList(mock_trans)
+    mock_response = _mock_deepseek_response("**Mock Core Hook**\n- Takeaway 1\n- Takeaway 2\nVerdict line.")
 
-    # Mock Claude HTTP response
-    mock_response = MagicMock(spec=httpx.Response)
-    mock_response.status_code = 200
-    mock_response.json.return_value = {
-        "content": [
-            {
-                "text": "**Mock Core Hook**\n- Takeaway 1\n- Takeaway 2\nVerdict line."
-            }
-        ]
-    }
-
-    # Patch both transcript api and httpx AsyncClient post call
     with patch("youtube_transcript_api.YouTubeTranscriptApi.list", return_value=mock_list), \
          patch("httpx.AsyncClient.post", new_callable=AsyncMock, return_value=mock_response) as mock_post:
 
@@ -69,12 +71,12 @@ async def test_summarize_video_success(test_db):
         assert summarizer.BASE_URL in url
 
         headers = mock_post.call_args[1]["headers"]
-        assert "x-api-key" in headers
+        assert "Authorization" in headers
         assert headers["User-Agent"] is not None
 
         payload = mock_post.call_args[1]["json"]
-        assert payload["model"] == "claude-haiku-4-5-20251001"
+        assert payload["model"] == "deepseek-chat"
-        assert len(payload["messages"]) == 1
+        assert len(payload["messages"]) == 2  # system + user
 
         assert summary == "**Mock Core Hook**\n- Takeaway 1\n- Takeaway 2\nVerdict line."
 
@@ -85,9 +87,40 @@ async def test_summarize_video_success(test_db):
     assert video["transcript"] == "Welcome to this video tutorial. Today we are learning about Svelte 5 and runes."
     assert video["summary"] == "**Mock Core Hook**\n- Takeaway 1\n- Takeaway 2\nVerdict line."
 
+
+@pytest.mark.asyncio
+async def test_summarize_video_deepseek_error_leaves_null(test_db):
+    """When DeepSeek fails, summary should stay NULL for retry."""
+    await test_db.execute("""
+        INSERT INTO videos (video_id, title, url, status)
+        VALUES ('err_vid', 'Error Video', 'https://youtube.com/watch?v=err_vid', 'pending')
+    """)
+    await test_db.commit()
+
+    mock_trans = MockTranscript()
+    mock_list = MockTranscriptList(mock_trans)
+    mock_response = MagicMock(spec=httpx.Response)
+    mock_response.status_code = 429
+    # raise_for_status on 429 should raise HTTPStatusError
+    mock_response.raise_for_status.side_effect = httpx.HTTPStatusError(
+        "429 Rate Limited", request=MagicMock(), response=mock_response
+    )
+
+    with patch("youtube_transcript_api.YouTubeTranscriptApi.list", return_value=mock_list), \
+         patch("httpx.AsyncClient.post", new_callable=AsyncMock, return_value=mock_response):
+
+        result = await summarizer.summarize_video("err_vid")
+
+        assert result is None  # Returns None instead of an error string
+
+    # Verify DB — summary should remain NULL
+    cursor = await test_db.execute("SELECT summary FROM videos WHERE video_id = 'err_vid'")
+    row = await cursor.fetchone()
+    assert row["summary"] is None
+
+
 @pytest.mark.asyncio
 async def test_summarize_all_pending(test_db):
-    # Insert multiple videos, one pending and one already summarized
     await test_db.execute("""
         INSERT INTO videos (video_id, title, url, summary, status)
         VALUES
@@ -98,28 +131,21 @@ async def test_summarize_all_pending(test_db):
 
     mock_trans = MockTranscript()
     mock_list = MockTranscriptList(mock_trans)
-
+    mock_response = _mock_deepseek_response("Fresh summary")
-    mock_response = MagicMock(spec=httpx.Response)
-    mock_response.status_code = 200
-    mock_response.json.return_value = {
-        "content": [{"text": "Fresh summary"}]
-    }
 
     with patch("youtube_transcript_api.YouTubeTranscriptApi.list", return_value=mock_list), \
          patch("httpx.AsyncClient.post", new_callable=AsyncMock, return_value=mock_response) as mock_post:
 
         await summarizer.summarize_all_pending()
 
-        # Should only call API once for 'pending_vid'
+        # Should only call API once for 'pending_vid' (done_vid already has summary)
         mock_post.assert_called_once()
-        payload = mock_post.call_args[1]["json"]
-        assert "Welcome to this video tutorial." in payload["messages"][0]["content"]
 
     # Verify db states
     cursor = await test_db.execute("SELECT video_id, summary FROM videos ORDER BY video_id")
     videos = await cursor.fetchall()
     assert videos[0]["video_id"] == "done_vid"
-    assert videos[0]["summary"] == "Existing summary" # Untouched
+    assert videos[0]["summary"] == "Existing summary"  # Untouched
 
     assert videos[1]["video_id"] == "pending_vid"
-    assert videos[1]["summary"] == "Fresh summary" # Updated
+    assert videos[1]["summary"] == "Fresh summary"  # Updated

backend/tg_filter.py

@@ -0,0 +1,229 @@
+"""
+Telegram Group Filter — scours 100+ groups for interesting messages.
+Rotates through all groups across cron cycles to avoid rate limits.
+Resets processed tracking per batch so each cycle gets fresh messages.
+"""
+
+import asyncio
+import os
+import re
+import json
+import sqlite3
+import logging
+from datetime import datetime, timezone, timedelta
+from pyrogram import Client
+from pyrogram.types import Message
+from typing import Optional
+
+logging.basicConfig(
+    level=logging.INFO,
+    format="%(asctime)s [%(levelname)s] %(message)s",
+    datefmt="%H:%M:%S",
+)
+log = logging.getLogger("tg_filter")
+
+# === CONFIG ===
+PHONE = open("/tmp/tg_phone.txt").read().strip()
+API_ID = 2040
+API_HASH = "b18441a1ff607e10a989891a5462e627"
+WORKDIR = "/Users/jimmyg"
+DB_PATH = os.path.join(WORKDIR, ".hermes", "tg_filter.db")
+
+# Scoring
+SCORE_MENTION = 30
+SCORE_KEYWORD = 15
+SCORE_LINK = 10
+SCORE_MEDIA = 8
+SCORE_CODE = 12
+SCORE_LONG_TEXT = 5
+THRESHOLD = 40
+
+# Scan config
+MSGS_PER_GROUP = 500
+GROUPS_PER_CYCLE = 56
+
+# Keywords
+INTERESTING_KEYWORDS = [
+    "ai", "llm", "gpt", "deepseek", "claude", "gemini", "qwen", "openai",
+    "docker", "kubernetes", "k8s", "tailscale", "wireguard", "vps", "nas",
+    "self-host", "homelab", "server", "deploy", "ci/cd", "gitea", "git",
+    "api", "automation", "script", "python", "rust", "go", "typescript",
+    "svelte", "react", "nextjs", "fastapi", "sveltekit",
+    "tutorial", "guide", "how to", "free", "open source", "github",
+    "security", "vulnerability", "cve", "patch", "update",
+    "opportunity", "合作", "项目", "招聘", "job", "remote",
+    "startup", "saas", "mvp", "产品", "产品发布",
+]
+
+IGNORE_PATTERNS = ["合租社群", "黄油派对"]
+
+
+def score_message(msg: Message) -> int:
+    score = 0
+    text = msg.text or msg.caption or ""
+    if msg.entities:
+        for e in msg.entities:
+            if e.type in ("mention", "text_mention"): score += SCORE_MENTION
+            if e.type == "url": score += SCORE_LINK
+            if e.type == "code" or e.type == "pre": score += SCORE_CODE
+    if msg.photo or msg.video or msg.document or msg.audio: score += SCORE_MEDIA
+    if re.search(r'https?://[^\s]+', text): score += SCORE_LINK
+    text_lower = text.lower()
+    for kw in INTERESTING_KEYWORDS:
+        if kw in text_lower: score += SCORE_KEYWORD
+    if len(text.strip()) >= 200: score += SCORE_LONG_TEXT
+    if msg.forward_from or msg.forward_from_chat: score += 5
+    if msg.reply_to_message_id: score += 3
+    return score
+
+
+def should_ignore(chat_title: str) -> bool:
+    if not chat_title: return False
+    for pat in IGNORE_PATTERNS:
+        if pat in chat_title: return True
+    return False
+
+
+def init_db():
+    os.makedirs(os.path.dirname(DB_PATH), exist_ok=True)
+    db = sqlite3.connect(DB_PATH)
+    db.execute("CREATE TABLE IF NOT EXISTS processed (chat_id INTEGER, message_id INTEGER, PRIMARY KEY (chat_id, message_id))")
+    db.execute("CREATE TABLE IF NOT EXISTS interesting (id INTEGER PRIMARY KEY AUTOINCREMENT, chat_id INTEGER, chat_title TEXT, message_id INTEGER, sender TEXT, text TEXT, score INTEGER, captured_at TEXT, delivered INTEGER DEFAULT 0)")
+    db.execute("CREATE TABLE IF NOT EXISTS group_stats (chat_id INTEGER PRIMARY KEY, chat_title TEXT, total_scanned INTEGER DEFAULT 0, interesting_found INTEGER DEFAULT 0, last_interesting_at TEXT, last_scan_at TEXT)")
+    db.commit()
+    return db
+
+
+def update_group_stats(db, chat_id: int, chat_title: str, found_interesting: int, now: str):
+    cur = db.execute("SELECT total_scanned, interesting_found FROM group_stats WHERE chat_id = ?", (chat_id,))
+    row = cur.fetchone()
+    if row:
+        db.execute(
+            "UPDATE group_stats SET total_scanned = total_scanned + 1, interesting_found = interesting_found + ?, "
+            "last_interesting_at = CASE WHEN ? > 0 THEN ? ELSE last_interesting_at END, last_scan_at = ? WHERE chat_id = ?",
+            (found_interesting, found_interesting, now if found_interesting > 0 else None, now, chat_id)
+        )
+    else:
+        db.execute(
+            "INSERT INTO group_stats (chat_id, chat_title, total_scanned, interesting_found, last_interesting_at, last_scan_at) "
+            "VALUES (?, ?, 1, ?, ?, ?)",
+            (chat_id, chat_title, found_interesting, now if found_interesting > 0 else None, now)
+        )
+
+
+async def scan_and_report(dry_run: bool = False) -> list:
+    db = init_db()
+    c = Client("pyro_session", API_ID, API_HASH, workdir=WORKDIR)
+    await c.start()
+    me = await c.get_me()
+    log.info(f"Scanned as: {me.first_name}")
+
+    interesting = []
+    total_read = 0
+    total_skipped = 0
+    total_groups = 0
+    now = datetime.now(timezone.utc).isoformat()
+
+    # Collect all eligible dialogs
+    candidates = []
+    async for dialog in c.get_dialogs():
+        chat = dialog.chat
+        chat_id = chat.id
+        chat_title = chat.title or chat.first_name or "Unknown"
+        if should_ignore(chat_title):
+            continue
+        candidates.append((chat_id, chat_title))
+
+    log.info(f"Total eligible groups: {len(candidates)}")
+
+    # Sort by last_scan_at (NULLs first) to rotate
+    sorted_candidates = []
+    for cid, title in candidates:
+        cur = db.execute("SELECT last_scan_at FROM group_stats WHERE chat_id = ?", (cid,))
+        row = cur.fetchone()
+        if row and row[0]:
+            sorted_candidates.append((cid, title, row[0]))
+        else:
+            sorted_candidates.append((cid, title, ""))
+    sorted_candidates.sort(key=lambda x: (x[2] == "", x[2]))
+    sorted_candidates = sorted_candidates[:GROUPS_PER_CYCLE]
+
+    log.info(f"Scanning {len(sorted_candidates)} groups this cycle")
+
+    # Clear processed entries for these groups so we get fresh messages
+    group_ids = [cid for cid, _, _ in sorted_candidates]
+    db.execute("DELETE FROM processed WHERE chat_id IN ({})".format(
+        ",".join("?" * len(group_ids))
+    ), group_ids)
+    db.commit()
+
+    for chat_id, chat_title, _ in sorted_candidates:
+        total_groups += 1
+        group_found = 0
+
+        async for msg in c.get_chat_history(chat_id, limit=MSGS_PER_GROUP):
+            total_read += 1
+            score = score_message(msg)
+
+            if score >= THRESHOLD:
+                sender = None
+                if msg.from_user:
+                    sender = msg.from_user.first_name or msg.from_user.username or str(msg.from_user.id)
+                text = (msg.text or msg.caption or "")[:2000]
+                item = {
+                    "chat_id": chat_id, "chat_title": chat_title, "message_id": msg.id,
+                    "sender": sender, "text": text, "score": score,
+                    "has_media": bool(msg.photo or msg.video or msg.document),
+                    "url": f"https://t.me/c/{str(chat_id).replace('-100', '')}/{msg.id}" if chat_id < 0 else None,
+                }
+                interesting.append(item)
+                group_found += 1
+                db.execute(
+                    "INSERT OR IGNORE INTO interesting (chat_id, chat_title, message_id, sender, text, score, captured_at) VALUES (?, ?, ?, ?, ?, ?, ?)",
+                    (chat_id, chat_title, msg.id, sender, text, score, now),
+                )
+            else:
+                total_skipped += 1
+
+        update_group_stats(db, chat_id, chat_title, group_found, now)
+        db.commit()
+
+    await c.stop()
+    db.close()
+
+    log.info(f"Scanned {total_groups} groups, read {total_read} msgs, found {len(interesting)} interesting")
+
+    if not dry_run and interesting:
+        for item in interesting:
+            _print_item(item)
+
+    # Group quality summary
+    print(f"\n{'='*50}")
+    print(f"📊 GROUP QUALITY SUMMARY")
+    print(f"{'='*50}")
+    db2 = sqlite3.connect(DB_PATH)
+    rows = db2.execute(
+        "SELECT chat_title, total_scanned, interesting_found, last_interesting_at FROM group_stats ORDER BY interesting_found DESC LIMIT 50"
+    ).fetchall()
+    for title, scanned, found, last_interest in rows:
+        status = "🟢" if found >= 1 else "🟡" if scanned >= 3 else "⚪"
+        print(f"{status} {title} — {scanned} scans, {found} interesting, last signal: {last_interest or 'never'}")
+    db2.close()
+
+    return interesting
+
+
+def _print_item(item: dict):
+    print(f"\n{'='*50}")
+    print(f"📌 {item['chat_title']} [{item['score']}pts]")
+    if item['sender']: print(f"👤 {item['sender']}")
+    if item['has_media']: print(f"📎 [has media]")
+    if item['url']: print(f"🔗 {item['url']}")
+    print(f"\n{item['text'][:500]}")
+    print(f"{'='*50}")
+
+
+if __name__ == "__main__":
+    import sys
+    dry_run = "--dry-run" in sys.argv
+    asyncio.run(scan_and_report(dry_run=dry_run))

backend/tg_filter.sh

@@ -0,0 +1,6 @@
+#!/bin/bash
+set -e
+cd ~/repos/t-youtube/backend
+export PYROGRAM_WORKDIR=/Users/jimmyg
+
+~/repos/t-youtube/backend/.venv/bin/python3 tg_filter.py 2>&1

docker-compose.yml

@@ -7,14 +7,15 @@ services:
       dockerfile: Dockerfile
     container_name: t-youtube
     restart: unless-stopped
-    network_mode: host
+    ports:
+      - "8001:8000"
+    env_file:
+      - .env
     environment:
-      - OPENAI_API_KEY=${OPENAI_API_KEY:-sk-Kq5JbgmnEEkdzTqNAQ6bV6nztQ0ngeHTHF9Vg8nUglQvrOlx}
+      - DEEPSEEK_API_KEY=${DEEPSEEK_API_KEY}
-      - ANTHROPIC_API_KEY=${ANTHROPIC_API_KEY}
+      - DEEPSEEK_BASE_URL=https://api.deepseek.com
-      - ANTHROPIC_BASE_URL=https://www.bytecatcode.org
+      - DEEPSEEK_MODEL=deepseek-chat
-      - HTTP_PROXY=http://100.70.115.1:8118
+      - DB_DIR=/app/data
-      - HTTPS_PROXY=http://100.70.115.1:8118
-      - NO_PROXY=localhost,127.0.0.1,100.0.0.0/8,192.168.0.0/16
     volumes:
       - ./data:/app/data
       - ./config:/app/config

docs/PDD.md

@@ -0,0 +1,21 @@
+# Product Definition Document — T YouTube
+
+## Vision
+
+A text-first YouTube experience — read transcripts, get AI summaries, never watch a video.
+
+## Goals
+
+1. **No-video YouTube** — read subscriptions, watch content as text, never open the YouTube app
+2. **Daily digest** — AI-summarized subscription feed delivered to Telegram daily
+3. **GFW-proof** — works reliably from China despite YouTube blocks
+
+## Target Users
+
+Self (Jimmy Gan). Primary use: morning skim of YouTube subscriptions via text summaries.
+
+## Success Metrics
+
+- Daily sync completes without errors
+- Telegram digest delivered every morning
+- < 5s page load for dashboard

docs/PLAN.md

@@ -0,0 +1,61 @@
+# Remaining Tasks
+
+## P0 — Immediate Follow-ups
+
+### 1. Deploy Sprint 1 code to NAS (Docker rebuild)
+Sprint 1 fixes (`575fdab`+, agent_worker.py, proxy support, scroll-based mark-read) are committed and pushed to Gitea, but **NAS container is running old code** — `agent_worker.py` missing. 
+- Rebuild Docker image with latest code
+- Restart container
+- Verify `AGENT_ENABLED=true` and `QWEN_BASE_URL=http://localhost:8080` in container env
+
+### 2. Fix daily sync cron
+`t-youtube-daily-sync` errored last run (Jul 9 03:55). Check the sync script at `run_sync.py` for the failure cause. The VPS-based `fetch_and_sync.sh` may have replaced the local sync cron — verify which is the active pipeline.
+
+### 3. Verify tg-group-filter first delivery
+Cron was fixed from `deliver: local` → `deliver: all`. First run at ~22:18 should deliver to Telegram. Check that the filter output reaches the user.
+
+---
+
+## P1 — High Value
+
+### 4. nas-tools main deploy (sidebar refactor)
+Dev CI is green. Sidebar now includes t-youtube, Media Management, Hermes. Just needs:
+```bash
+git checkout main
+git merge dev
+git push origin main
+```
+Then verify production deploy on `nas.jimmygan.com`.
+
+### 5. Connect Google AI Pro to Hermes
+User has Google AI Pro plan accessed via `ccg` → Antigravity proxy (`localhost:8081`). Configure Hermes to use it (Option 1 chosen: custom provider pointing at the proxy). This unlocks Gemini models at no extra cost, improving summarization and agent quality.
+
+---
+
+## P2 — Product & Brand
+
+### 6. t-youtube Sprint 2 (UX Overhaul)
+- Full-text search across video titles
+- Batch select / mark multiple videos read
+- Infinite scroll (cursor-based pagination)
+- Duration + view count in list
+- Transcript accordion in detail view
+
+### 7. t-youtube → public asset (brand play)
+Already built: vim-keyboard YouTube reader with AI summaries. Could be:
+- Twitter/X thread: "I built a better YouTube reader"
+- Open-source README with screenshots
+- Blog post about architecture (FastAPI + Svelte 5 + DeepSeek)
+
+---
+
+## P3 — Infrastructure
+
+### 8. Proxy env support in n8n / other services
+For stable operation from China.
+
+### 9. t-youtube DB backup to MBP
+Single source of truth. Add cron: `ssh nas "cat /volume1/docker/t-youtube/data/t_youtube.db" > ~/backups/t_youtube/$(date +%F).db`
+
+### 10. Cleanup old Gmail labels
+Old labels (Bank, Brokers, Fund, Login Notice, jobs, 购物) still in sidebar — not deleted, just archived.

docs/architecture.md

@@ -0,0 +1,200 @@
+# t-youtube — Text-Based YouTube Digest
+
+## Overview
+
+A self-hosted YouTube reader that syncs your subscriptions, generates AI summaries, and displays them in a Svelte dashboard. Designed to work around GFW restrictions in China and YouTube IP blocks on cloud providers.
+
+## Architecture
+
+```
+                         ┌─────────────────────────┐
+                         │   Oracle Tokyo VPS       │
+                         │  (free-amd-caddy)        │
+                         │                          │
+                         │  ┌──────────────────┐    │
+                         │  │ Caddy (port 443)  │    │
+                         │  │ reverse_proxy to  │────┼─── Cloudflare ─── Internet
+                         │  │ NAS via Tailscale │    │
+                         │  └──────────────────┘    │
+                         │                          │
+                         │  ┌──────────────────┐    │
+                         │  │ Cron (every 6h)   │    │
+                         │  │ fetch_and_sync.sh │────┼─── YouTube Data API
+                         │  └──────────────────┘    │
+                         └──────────┬───────────────┘
+                                    │ Tailscale (DERP relay)
+                         ┌──────────┴───────────────┐
+                         │   NAS ds224plus           │
+                         │  (100.78.131.124)         │
+                         │                           │
+                         │  ┌───────────────────┐   │
+                         │  │ Caddy (port 8443)  │   │
+                         │  │ Let's Encrypt via  │   │
+                         │  │ Cloudflare DNS     │   │
+                         │  └───────────────────┘   │
+                         │                           │
+                         │  ┌───────────────────┐   │
+                         │  │ Authelia (9092)    │   │
+                         │  │ SSO + 2FA          │   │
+                         │  └───────────────────┘   │
+                         │                           │
+                         │  ┌───────────────────┐   │
+                         │  │ t-youtube (8001)  │   │
+                         │  │ FastAPI + Svelte  │   │
+                         │  │ SQLite DB          │   │
+                         │  └───────────────────┘   │
+                         │                           │
+                         │  ┌───────────────────┐   │
+                         │  │ Gitea (3300)       │   │
+                         │  │ git.jimmygan.com   │   │
+                         │  └───────────────────┘   │
+                         └──────────────────────────┘
+```
+
+## Data Flow
+
+1. **Fetch**: VPS cron runs every 6h → YouTube Data API v3 (subscriptions list) + RSS feeds → new video metadata → local SQLite DB
+2. **Summarize**: VPS reads new videos from DB → calls Qwen35B on MacBook (port 8085 via Tailscale) → title-only summaries → saves to DB
+3. **Sync**: VPS `cat` DB → SSH to NAS → `sudo tee` → overwrites NAS DB → restarts Docker container
+4. **Serve**: NAS Docker (FastAPI + Svelte) → serves dashboard to browser → Authelia SSO in front
+
+## Deployment
+
+### VPS (Oracle Tokyo)
+
+Files in `/tmp/yt-fetch/`:
+
+| File | Purpose |
+|------|---------|
+| `fetch_and_sync.sh` | Entry point — fetch → summarize → sync |
+| `fetcher.py` | YouTube Data API + RSS hybrid fetcher |
+| `summarize_new.py` | Title-based summarization via Qwen |
+| `db.py` | SQLite schema |
+| `auth.py` | OAuth credential management |
+| `token.pickle` | YouTube OAuth token |
+| `cookies.txt` | YouTube cookies (for transcript fallback) |
+
+Cron: `0 */6 * * * bash /tmp/yt-fetch/fetch_and_sync.sh`
+
+### NAS
+
+Docker container at `/volume1/docker/t-youtube/`:
+- `docker-compose.yml` — port 8001:8000, env vars for DeepSeek/Qwen
+- `.env` — `DEEPSEEK_API_KEY`, `API_TOKEN`
+- `data/t_youtube.db` — SQLite database (synced from VPS)
+
+### MacBook
+
+Qwen models run via `llama-server`:
+- Port 8080: Qwen3.6-27B-Q8 (27B params, ~30 t/s, 256K ctx)
+- Port 8085: Qwen3.6-35B-A3B-Q8 (35B MoE, 3.5B active, faster)
+- Both with `--spec-type draft-mtp` for speculative decoding
+
+## Configuration
+
+### Environment Variables
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `DEEPSEEK_API_KEY` | — | DeepSeek API key for summarization |
+| `DEEPSEEK_BASE_URL` | `https://api.deepseek.com` | API endpoint |
+| `DEEPSEEK_MODEL` | `deepseek-chat` | Model name |
+| `API_TOKEN` | — | Bearer token for dashboard write ops |
+| `AGENT_ENABLED` | `false` | Enable agent queue worker |
+| `QWEN_BASE_URL` | `http://localhost:8080` | Local Qwen endpoint |
+| `HTTP_PROXY` | — | Proxy for fetcher/summarizer (China bypass) |
+| `AGENT_POLL_INTERVAL` | `30` | Agent queue poll seconds |
+| `AGENT_MAX_CONCURRENT` | `3` | Agent queue max workers |
+
+### Authelia Bypass
+
+API routes at `/api/*` bypass Authelia via `resources: ['^/api/']` in Authelia `access_control.yaml`. Frontend (`/`) requires SSO login.
+
+## API Endpoints
+
+| Endpoint | Method | Auth | Description |
+|----------|--------|------|-------------|
+| `/api/health` | GET | None | Health check |
+| `/api/config` | GET | None | Auth config (token required?) |
+| `/api/videos` | GET | None | List videos (status, channel, limit params) |
+| `/api/videos/fetch` | POST | Bearer | Trigger subscription sync |
+| `/api/videos/{id}/summarize` | POST | Bearer | Trigger single video summary |
+| `/api/videos/{id}/status` | POST | Bearer | Update status (pending/read/skipped) |
+| `/api/notes` | GET | None | List all notes |
+| `/api/notes/{id}` | GET/POST | Bearer (POST) | Get/save note |
+| `/api/notes/migrate` | POST | Bearer | Migrate localStorage notes to server |
+| `/api/agent-queue` | GET/POST | Bearer (POST) | List/add agent queue items |
+| `/api/channels` | GET | None | List distinct channels |
+| `/api/auto_categories.json` | GET | None | Channel category mapping |
+
+## Sprint Plan
+
+### Sprint 1 ✅ (Current — Deployed)
+- Agent queue worker (disabled by default)
+- Proxy support for fetcher + summarizer
+- Auto-mark-read on navigate away (not timer)
+- VPS fetch + summarize cron
+- run_tests.sh fixes
+- All 2,674 videos summarized (title-based)
+
+### Sprint 2 — UX Overhaul (Next)
+- Full-text search across video titles
+- Batch select → mark read/skip
+- Infinite scroll / pagination
+- Display duration + view count
+- Transcript accordion in detail panel
+
+### Sprint 3 — Polish & Monitoring
+- Stats dashboard (videos/day, channel health, sync lag)
+- Auto-purge old read videos (configurable TTL)
+- Broken channel detection (404 on RSS)
+- Cron failure alerts via Telegram bot
+- Scheduled DB backup to MacBook
+
+### Sprint 4 — Code Quality
+- Split `main.py` into routers
+- Cursor/offset pagination on API
+- Rate limiting middleware
+- aiosqlite connection pooling
+
+## Known Issues
+
+### Transcripts Blocked
+YouTube transcripts inaccessible from all our locations:
+- **Oracle Cloud VPS**: YouTube blocks cloud provider IPs
+- **MacBook (China)**: GFW blocks YouTube API endpoints
+- **YouTube Data API captions.download**: 403 Forbidden for non-owned videos
+
+**Workaround**: Title + description-based summaries via Qwen. Full transcripts need a Japanese residential proxy ($3-5/month).
+
+### auth.jimmygan.com from LAN
+Accessing `auth.jimmygan.com` from LAN shows Immich (photos.jimmygan.com) instead of Authelia. Caused by NAS port 443 routing — DSM nginx still owns port 443 and routes Immich for unmatched domains.
+
+### Gitea SSL
+Uses Let's Encrypt **staging** certificates (acme-staging) — browsers may show untrusted cert warnings.
+
+## Development
+
+### Local Setup
+```bash
+cd ~/repos/t-youtube
+# Backend
+cd backend && python3 -m venv .venv && source .venv/bin/activate && pip install -r requirements.txt
+python3 main.py  # runs on :8000
+
+# Frontend
+cd frontend && npm install && npm run dev  # runs on :5173 with API proxy
+```
+
+### Testing
+```bash
+cd ~/repos/t-youtube && bash run_tests.sh
+# 23 tests across db, fetcher, summarizer, main, agent_worker
+```
+
+### Deploy to NAS
+```bash
+cd ~/repos/t-youtube
+git add -A && git commit -m "..." && git push gitea main:master
+ssh nas 'cd /volume1/docker/t-youtube && git pull origin master && docker compose up -d --build'
+```

docs/configuration.md

@@ -0,0 +1,152 @@
+# t-youtube Configuration Guide
+
+## Environment Variables
+
+### Backend (.env)
+```bash
+# Database path
+DB_DIR=/volume1/docker/t-youtube/backend
+
+# API Token (for write operations)
+API_TOKEN=<your-api-token>
+
+# DeepSeek API (for summarization)
+DEEPSEEK_API_KEY=<your-key>
+
+# YouTube Data API v3
+GOOGLE_APPLICATION_CREDENTIALS=/path/to/credentials.json
+```
+
+### Docker Compose
+```yaml
+version: "3"
+services:
+  t-youtube:
+    build: .
+    ports:
+      - "8001:8001"
+    environment:
+      - DB_DIR=/backend/data
+      - API_TOKEN=<your-token>
+      - DEEPSEEK_API_KEY=<your-key>
+    volumes:
+      - ./backend/data:/backend/data
+```
+
+## API Token Management
+
+### Generation
+```bash
+python3 -c "import secrets; print('api-' + secrets.token_urlsafe(32))"
+```
+
+### Frontend Setup
+- First-time users see a token setup modal
+- Token stored in localStorage
+- Bearer token sent with all API write requests
+
+### Backend Validation
+- Middleware checks `Authorization: Bearer <token>` for POST/PATCH/DELETE on `/api/*`
+- Returns 401 JSON for invalid tokens
+- GET endpoints remain open to authenticated frontend users
+
+## Database Management
+
+### Location
+- **Path**: `/volume1/docker/t-youtube/backend/t_youtube.db`
+- **Type**: SQLite (aiosqlite)
+- **Backup**: Copy the `.db` file before any schema changes
+
+### Schema Migrations
+- Use `sqlite3` CLI or a migration script
+- Always add indexes after schema changes
+- Example:
+  ```bash
+  sqlite3 t_youtube.db "CREATE INDEX IF NOT EXISTS idx_x ON table(column);"
+  ```
+
+### Daily Sync Configuration
+
+#### Cron Job
+```bash
+# Runs every day at 3 AM
+0 3 * * * /path/to/run_sync.py >> /var/log/t-youtube-sync.log 2>&1
+```
+
+#### Fetcher Settings
+- **RSS**: Free, unlimited, ~2000 channels
+- **YouTube Data API v3**: ~50 units per sync
+- **Rotation**: Fetches in slices to avoid timeouts
+- **Retry**: Automatic retry on failure
+
+#### Summarizer Settings
+- **Model**: `deepseek-chat` via DeepSeek API
+- **Timeout**: 30 seconds per video
+- **Fallback**: Video description if transcript unavailable
+- **Language**: Follows content language (Chinese → Chinese summary)
+
+## Authelia Integration
+
+### Authentication Flow
+1. User accesses protected domain
+2. Caddy forwards request to Authelia (`127.0.0.1:9092`)
+3. Authelia checks session/2FA
+4. If no session → redirect to `auth.jimmygan.com`
+5. If session exists → return 200 (allowed)
+
+### 2FA Methods
+- **WebAuthn (Primary)**: Passkey via iCloud Keychain/Bitwarden
+- **TOTP (Fallback)**: Time-based one-time password
+
+### Configuration
+- **Address**: `127.0.0.1:9092`
+- **Database**: SQLite (`/volume1/docker/authelia/config/db.sqlite3`)
+- **LDAP Backend**: LLDAP (`lldap:3890`)
+- **Session**: 1 hour expiration, 5 minute inactivity timeout
+
+### Restart Authelia
+```bash
+cd /volume1/docker/authelia && /usr/local/bin/docker compose restart authelia
+```
+
+## Caddy Configuration
+
+### VPS Caddy (Public)
+- **Port**: 443
+- **TLS**: Let's Encrypt (HTTP-01)
+- **Authelia**: `forward_auth 100.78.131.124:9092`
+- **DNS**: dnsmasq → VPS Tailscale IP `100.109.198.126`
+
+### NAS Caddy (Local/Tailscale)
+- **Port**: 8443 (fallback due to SNI bug)
+- **TLS**: Cloudflare DNS-01
+- **Authelia**: `forward_auth 127.0.0.1:9092`
+- **SNI Bug**: Per-site `tls` blocks cause cert routing issues
+- **Workaround**: Global TLS config + dnsmasq → VPS Tailscale IP
+
+### Reloading Caddy
+```bash
+# VPS
+ssh caddy-vps "sudo systemctl reload caddy"
+
+# NAS
+cd /volume1/docker/caddy && /usr/local/bin/docker compose restart caddy
+```
+
+## Monitoring & Health
+
+### Health Check
+```bash
+curl -s https://yt.jimmygan.com/api/health
+# Expected: {"status": "ok"}
+```
+
+### Sync Status
+- Check `run_sync.py` logs in `/var/log/t-youtube-sync.log`
+- Monitor DB for new videos/summaries
+- Verify cron job is active: `crontab -l`
+
+### Database Size
+```bash
+ls -lh /volume1/docker/t-youtube/backend/t_youtube.db
+```

docs/progress.md

@@ -0,0 +1,102 @@
+# t-youtube Progress Tracking
+
+## Session 2026-07-09: Agent Queue, Notes Migration, Architecture Documentation
+
+### Completed
+- [x] **Agent Queue Feature**:
+  - Added `agent_queue` table to SQLite DB
+  - Implemented `POST/GET/DELETE` endpoints in `main.py`
+  - Created `agent_worker.py` skeleton for processing pending items
+  - Frontend "Agent Queue" tab (Svelte 5) with status indicators
+- [x] **Notes Migration**:
+  - Moved notes from localStorage to server-side DB (`video_notes` table)
+  - Added `POST/GET/DELETE` endpoints for notes
+  - Implemented non-blocking localStorage migration script
+- [x] **Database Optimization**:
+  - Added indexes on `video_id` for `agent_queue` and `video_notes` tables
+- [x] **Authelia Configuration**:
+  - Attempted passwordless mode (failed due to unsupported key in Authelia v4)
+  - Reverted to standard 2FA (WebAuthn/TOTP)
+- [x] **Documentation**:
+  - Created `docs/` folder with `architecture.md` and `progress.md`
+  - Documented all API endpoints, DB schema, infrastructure, and known issues
+
+### In Progress
+- [ ] **Agent Worker Logic**:
+  - `agent_worker.py` needs actual LLM/feature-creation logic
+  - Pending: Define workflow for processing queued items
+- [ ] **YouTube Sync Monitoring**:
+  - Daily cron at 3 AM is active
+  - Pending: Monitor next run for errors/progress
+
+### Planned
+- [ ] **Channel Categorization**:
+  - ~119 channels remain uncategorized
+  - Consider moving `AUTO_CATEGORIES` to server-side JSON config
+- [ ] **Transcript Fallback**:
+  - Implement browser-based scraping (playwright/selenium) for the remaining 10%
+  - High complexity/resource usage trade-off
+- [ ] **Performance Monitoring**:
+  - Add health check endpoints for sync status
+  - Implement error logging/reporting
+
+## Session 2026-07-05 to 2026-07-08: Initial Development
+
+### Key Milestones
+- [x] **YouTube Data API v3 + RSS Hybrid**: Replaced yt-dlp/cookies approach
+  - Free, unlimited sync (~50 units/sync for 200+ channels)
+  - 2414 new videos discovered in first run
+- [x] **AI Summarization**: DeepSeek API integration (was Claude/bytecatcode)
+  - Chinese content → Chinese summary; English content → English summary
+  - ~2.1s per video, ~$0.001 each
+- [x] **Frontend Overhaul**:
+  - AI Digest as hero element
+  - Keyboard navigation (`j/k/↑↓`, `r/m`, `s`, `n`, `Enter`, `?`)
+  - Dark/light mode toggle
+  - Channel filter dropdown with auto-generated categories
+  - Auto-read after 3s silent mark
+- [x] **Agent Queue Integration**:
+  - "Save to Agent" feature: notes ARE prompts for Hermes agents
+  - Queue tab with pending/processing/done status
+  - Cross-device note sync (localStorage → server DB)
+
+### Technical Decisions
+1. **API Token Auth**: Bearer token for API writes, Authelia SSO for frontend
+2. **SQLite**: Simple, reliable, no external dependencies for single-user setup
+3. **DeepSeek API**: Cost-effective for summarization (~$0.001/video)
+4. **Svelte 5**: Reactivity model, keyboard workflow, small bundle size
+5. **RSS + YouTube Data API**: Free tier sufficient, avoids cookie/yt-dlp maintenance
+
+## Session 2026-07-04: NAS & Authelia Setup
+
+### Completed
+- [x] NAS Caddy port 443 conflict resolved (DSM nginx)
+- [x] dnsmasq → VPS Caddy Tailscale IP (100.109.198.126)
+- [x] Authelia WebAuthn enabled, passkey registered
+- [x] TOTP reset for jimmy, default 2FA set to WebAuthn
+- [x] Caddy SNI bug workaround (global TLS config, port 8443 fallback)
+
+## Known Issues & Resolutions
+
+### Authelia Passwordless Failure
+- **Issue**: `webauthn.passwordless: true` caused boot loop (`configuration key not expected`)
+- **Root Cause**: Unsupported configuration key in current Authelia version
+- **Resolution**: Reverted to standard 2FA (WebAuthn/TOTP)
+
+### YouTube Transcript Blocking
+- **Issue**: 90% of videos have transcripts; 10% fail due to Oracle Cloud IP blocking
+- **Root Cause**: YouTube IP-based restrictions
+- **Resolution**: Fallback to video description summaries; consider browser-based scraping later
+
+### NAS Caddy SNI Bug
+- **Issue**: NAS Caddy serves first cached cert for all SNI names
+- **Root Cause**: Per-site `tls { ... }` blocks conflict with global auto_https policy
+- **Resolution**: Global TLS config + dnsmasq → VPS Caddy Tailscale IP
+
+## Metrics & Performance
+- **Videos in DB**: ~2414+ (initial sync)
+- **Channels**: ~229 subscribed
+- **Sync Time**: ~2-3 minutes for all channels
+- **Summarization**: ~2.1s/video, ~$0.001 each via DeepSeek
+- **Frontend Load Time**: 0.14-0.22s (API responses)
+- **Daily API Usage**: ~50 units (well within 10,000/day free tier)

docs/specs/active.md

@@ -0,0 +1 @@
+sprint-002-ux-search-metadata.md

docs/specs/sprint-001-sync-reliability.md

@@ -0,0 +1,34 @@
+# Sprint 002 — UX Overhaul, Full-Text Search & Metadata Enrichment
+
+**Goal:** Overhaul the t-youtube reading experience by completing the search workflow, adding transcript visibility, fetching duration/views metadata, and implementing cursor-based/offset pagination (infinite scroll).
+
+---
+
+## T2.1 — Full-Text Search Frontend Integration
+- **Files:** `frontend/src/routes/YoutubeDashboard.svelte`
+- **Done means:** Search queries directly populate the main video list in the sidebar (rather than a temporary dropdown), and clearing the search resets the view.
+- **Verify by:** 
+  1. Type search query -> video list dynamically updates with FTS results from backend.
+  2. Click a search result -> detail view loads it.
+  3. Clear query/blur search -> list reverts to standard tab view.
+
+## T2.2 — Transcript Accordion in Details View
+- **Files:** `frontend/src/routes/YoutubeDashboard.svelte`
+- **Done means:** Add an expandable accordion below the "AI Digest" section to view the raw video transcript. If transcript is long, render in scrollable container.
+- **Verify by:**
+  1. Click video with transcript -> "Show Full Transcript" accordion appears.
+  2. Click to expand -> raw transcript displays correctly.
+
+## T2.3 — Ingest Duration and View Count Metadata
+- **Files:** `backend/fetcher.py`, `backend/tests/test_fetcher.py`
+- **Done means:** Update `fetcher.py` videos.list API call to request `part="snippet,contentDetails,statistics"`. Parse ISO 8601 duration into integer seconds, and store duration + view count in SQLite db. Update unit tests to include mock response fields.
+- **Verify by:**
+  1. Run `./run_tests.sh` to ensure fetcher tests pass.
+  2. Sync subscriptions -> inspect database to confirm `duration` and `view_count` are populated for new videos.
+
+## T2.4 — Cursor/Offset Pagination (Infinite Scroll)
+- **Files:** `backend/main.py`, `frontend/src/routes/YoutubeDashboard.svelte`
+- **Done means:** Add pagination queries to `/api/videos` backend endpoint (e.g. `offset` and `limit`). In Svelte frontend, implement scroll listener on the sidebar video list to trigger next page load when reaching bottom.
+- **Verify by:**
+  1. Load pending tab -> initially loads 50 videos.
+  2. Scroll to bottom of sidebar -> next 50 videos append automatically.

docs/specs/sprint-002-ux-search-metadata.md

@@ -0,0 +1,34 @@
+# Sprint 002 — UX Overhaul, Full-Text Search & Metadata Enrichment
+
+**Goal:** Overhaul the t-youtube reading experience by completing the search workflow, adding transcript visibility, fetching duration/views metadata, and implementing cursor-based/offset pagination (infinite scroll).
+
+---
+
+## [x] T2.1 — Full-Text Search Frontend Integration
+- **Files:** `frontend/src/routes/YoutubeDashboard.svelte`
+- **Done means:** Search queries directly populate the main video list in the sidebar (rather than a temporary dropdown), and clearing the search resets the view.
+- **Verify by:** 
+  1. Type search query -> video list dynamically updates with FTS results from backend.
+  2. Click a search result -> detail view loads it.
+  3. Clear query/blur search -> list reverts to standard tab view.
+
+## [x] T2.2 — Transcript Accordion in Details View
+- **Files:** `frontend/src/routes/YoutubeDashboard.svelte`
+- **Done means:** Add an expandable accordion below the "AI Digest" section to view the raw video transcript. If transcript is long, render in scrollable container.
+- **Verify by:**
+  1. Click video with transcript -> "Show Full Transcript" accordion appears.
+  2. Click to expand -> raw transcript displays correctly.
+
+## [x] T2.3 — Ingest Duration and View Count Metadata
+- **Files:** `backend/fetcher.py`, `backend/tests/test_fetcher.py`
+- **Done means:** Update `fetcher.py` videos.list API call to request `part="snippet,contentDetails,statistics"`. Parse ISO 8601 duration into integer seconds, and store duration + view count in SQLite db. Update unit tests to include mock response fields.
+- **Verify by:**
+  1. Run `./run_tests.sh` to ensure fetcher tests pass.
+  2. Sync subscriptions -> inspect database to confirm `duration` and `view_count` are populated for new videos.
+
+## [x] T2.4 — Cursor/Offset Pagination (Infinite Scroll)
+- **Files:** `backend/main.py`, `frontend/src/routes/YoutubeDashboard.svelte`
+- **Done means:** Add pagination queries to `/api/videos` backend endpoint (e.g. `offset` and `limit`). In Svelte frontend, implement scroll listener on the sidebar video list to trigger next page load when reaching bottom.
+- **Verify by:**
+  1. Load pending tab -> initially loads 50 videos.
+  2. Scroll to bottom of sidebar -> next 50 videos append automatically.

docs/troubleshooting.md

@@ -0,0 +1,194 @@
+# t-youtube Troubleshooting Guide
+
+## Common Issues
+
+### 1. Site Down / 502 Bad Gateway
+- **Cause**: t-youtube container not running or Authelia down
+- **Check**:
+  ```bash
+  ssh nas 'docker ps | grep t-youtube'
+  ssh nas 'docker ps | grep authelia'
+  ```
+- **Fix**:
+  ```bash
+  cd /volume1/docker/t-youtube && /usr/local/bin/docker compose up -d
+  cd /volume1/docker/authelia && /usr/local/bin/docker compose restart authelia
+  ```
+
+### 2. Authelia Loop / Login Fails
+- **Cause**: WebAuthn/TOTP misconfiguration or session expired
+- **Check Authelia logs**:
+  ```bash
+  ssh nas 'docker logs authelia --tail 20'
+  ```
+- **Fix**:
+  - Clear browser cache/cookies
+  - Reset WebAuthn via Authelia DB (see below)
+  - Check dnsmasq config (should point to VPS Tailscale IP)
+
+### 3. YouTube Data API Quota Exceeded
+- **Symptom**: Fetcher returns empty results or 403 errors
+- **Cause**: Daily quota limit reached (10,000 units)
+- **Check**:
+  ```bash
+  ssh nas 'docker logs t-youtube 2>&1 | grep -i "quota\|403"'
+  ```
+- **Fix**:
+  - Wait until next day (quota resets at 00:00 UTC)
+  - Reduce number of channels or increase sync interval
+  - Monitor usage in Google Cloud Console
+
+### 4. Summarization Failing
+- **Symptom**: Videos show "No summary" or generic descriptions
+- **Cause**: DeepSeek API key invalid or rate limited
+- **Check**:
+  ```bash
+  ssh nas 'docker logs t-youtube 2>&1 | grep -i "summariz\|deepseek"'
+  ```
+- **Fix**:
+  - Verify `DEEPSEEK_API_KEY` in `.env`
+  - Test API key manually:
+    ```bash
+    curl -s -X POST https://api.deepseek.com/chat/completions \
+      -H "Authorization: Bearer ***
+      -H "Content-Type: application/json" \
+      -d '{"model": "deepseek-chat", "messages": [{"role": "user", "content": "test"}]}'
+    ```
+
+### 5. Database Locked/Corrupted
+- **Symptom**: API returns 500 errors or hangs
+- **Cause**: Concurrent write operations or unexpected shutdown
+- **Fix**:
+  ```bash
+  cd /volume1/docker/t-youtube/backend
+  cp t_youtube.db t_youtube.db.backup
+  sqlite3 t_youtube.db "PRAGMA integrity_check;"
+  # If corrupted, restore from backup or recreate
+  ```
+
+### 6. Caddy Config Issues
+- **Symptom**: 502/503 errors, wrong cert, or domain unreachable
+- **Check**:
+  ```bash
+  ssh caddy-vps "sudo caddy validate --config /etc/caddy/Caddyfile"
+  ssh nas 'docker exec caddy caddy validate --config /etc/caddy/Caddyfile'
+  ```
+- **Fix**:
+  - Validate config
+  - Reload/restart Caddy
+  - Check TLS certs:
+    ```bash
+    echo | openssl s_client -connect 161.33.182.109:443 -servername yt.jimmygan.com 2>&1 | openssl x509 -noout -subject -dates
+    ```
+
+## API Debugging
+
+### Check Video List
+```bash
+curl -s https://yt.jimmygan.com/api/videos | python3 -m json.tool | head -20
+```
+
+### Check Agent Queue
+```bash
+curl -s -H "Authorization: Bearer *** https://yt.jimmygan.com/api/agent-queue
+```
+
+### Check Notes
+```bash
+curl -s -H "Authorization: Bearer *** https://yt.jimmygan.com/api/notes
+```
+
+### Check Channel List
+```bash
+curl -s -H "Authorization: Bearer *** https://yt.jimmygan.com/api/channels
+```
+
+### Test Summarization
+```bash
+curl -s -X POST -H "Authorization: Bearer ***
+  -H "Content-Type: application/json" \
+  -d '{"video_id": "<video_id>"}' \
+  https://yt.jimmygan.com/api/agent-queue/<video_id>/summarize
+```
+
+## Database Queries
+
+### Count Videos by Status
+```sql
+SELECT status, COUNT(*) FROM videos GROUP BY status;
+```
+
+### Count Agent Queue Items by Status
+```sql
+SELECT status, COUNT(*) FROM agent_queue GROUP BY status;
+```
+
+### Find Videos Without Summaries
+```sql
+SELECT video_id, title FROM videos WHERE summary IS NULL OR summary = '';
+```
+
+### Check DB Size
+```bash
+ls -lh /volume1/docker/t-youtube/backend/t_youtube.db
+```
+
+## Logs
+
+### t-youtube Logs
+```bash
+ssh nas 'docker logs t-youtube --tail 50'
+ssh nas 'docker logs t-youtube --tail 50 --since 1h'
+```
+
+### Authelia Logs
+```bash
+ssh nas 'docker logs authelia --tail 50'
+```
+
+### Sync Logs
+```bash
+ssh nas 'cat /var/log/t-youtube-sync.log'
+ssh nas 'tail -100 /var/log/t-youtube-sync.log'
+```
+
+### Caddy Logs (VPS)
+```bash
+ssh caddy-vps "sudo journalctl -u caddy --since '1 hour ago'"
+ssh caddy-vps "sudo journalctl -u caddy --no-pager -n 30"
+```
+
+## Recovery Procedures
+
+### Full System Restart
+```bash
+# 1. Restart services
+cd /volume1/docker/authelia && /usr/local/bin/docker compose restart authelia
+cd /volume1/docker/t-youtube && /usr/local/bin/docker compose up -d
+
+# 2. Verify
+curl -s https://yt.jimmygan.com/api/health
+curl -s https://auth.jimmygan.com/api/health
+```
+
+### Database Backup & Restore
+```bash
+# Backup
+cp /volume1/docker/t-youtube/backend/t_youtube.db /tmp/t-youtube-backup-$(date +%Y%m%d).db
+
+# Restore
+cp /tmp/t-youtube-backup-20260709.db /volume1/docker/t-youtube/backend/t_youtube.db
+cd /volume1/docker/t-youtube && /usr/local/bin/docker compose restart t-youtube
+```
+
+### Authelia User Reset (if locked out)
+```bash
+# Stop Authelia
+ssh nas 'docker stop authelia'
+
+# Reset TOTP for jimmy
+ssh nas 'sqlite3 /volume1/docker/authelia/config/db.sqlite3 "DELETE FROM totp_configurations WHERE username='\'jimmy'\'';"'
+
+# Restart Authelia
+ssh nas 'docker start authelia'
+```

frontend/src/auto_categories.js

@@ -0,0 +1 @@
+export default {"AI Engineer": "AI & Tech", "AI-Fan AI研究室-帆哥": "AI & Tech", "AI大模型": "AI & Tech", "AI探索与发现": "AI & Tech", "Aurora Tech": "AI & Tech", "Claude": "AI & Tech", "ColdFusion": "AI & Tech", "Data Is Beautiful": "AI & Tech", "DeepLearningAI": "AI & Tech", "Google": "AI & Tech", "Google Cloud Tech": "AI & Tech", "Lex Fridman": "AI & Tech", "NiceKate AI": "AI & Tech", "PC Sim": "AI & Tech", "Stanford Online": "AI & Tech", "The Linux Foundation": "AI & Tech", "回到Axton": "AI & Tech", "回形针PaperClip": "AI & Tech", "林亦LYi": "AI & Tech", "硅谷101播客": "AI & Tech", "老石谈芯": "AI & Tech", "鲲鹏Talk": "AI & Tech", "3D Beast": "3D & Design", "Arrimus3D Modeling and Design": "3D & Design", "Blender": "3D & Design", "Blender Guru": "3D & Design", "Creators 3D": "3D & Design", "Designfusion": "3D & Design", "Foundry": "3D & Design", "Grant Abbitt (Gabbitt)": "3D & Design", "Gravity Sketch": "3D & Design", "Hyperganic": "3D & Design", "MSLattice": "3D & Design", "Manolo Remiddi": "3D & Design", "Rhino 3D (Rhinoceros3d official)": "3D & Design", "Shaper3d": "3D & Design", "nTop": "3D & Design", "spherene AG": "3D & Design", "魔界造物": "3D & Design", "Chuck Severance": "Programming", "Coursera": "Programming", "freeCodeCamp.org": "Programming", "Kevin Stratvert": "Programming", "Leila Gharani": "Programming", "LinkedIn Learning": "Programming", "Metabase": "Programming", "n8n": "Programming", "The Coding Gopher": "Programming", "Brian Casel": "Programming", "code秘密花园": "Programming", "cs tech": "Programming", "fequalsf": "Programming", "openscreencast": "Programming", "技术爬爬虾  TechShrimp": "Programming", "林本兔Limbuntu": "Programming", "Best Choice Review": "Hardware & Reviews", "Marques Brownlee": "Hardware & Reviews", "TESTV": "Hardware & Reviews", "先看评测": "Hardware & Reviews", "极客湾Geekerwan": "Hardware & Reviews", "硬核拆解": "Hardware & Reviews", "笔吧评测室": "Hardware & Reviews", "差评硬件部": "Hardware & Reviews", "充电头网": "Hardware & Reviews", "Apple Developer": "Apple Dev", "Made by Google": "Apple Dev", "Allen的分享": "Apple Dev", "Sypnotix": "Apple Dev", "8K World": "Photography", "Can Fiona": "Photography", "小天fotos": "Photography", "影像极客Fotogeeker": "Photography", "摄影师PHiL": "Photography", "虚空光影CosmosFilm": "Photography", "路易斯 LouisDrone": "Photography", "Bruno Mars": "Music", "GraceLeeMusic": "Music", "Katie Melua": "Music", "The Weeknd": "Music", "TheWeekndVEVO": "Music", "杰威爾歌詞MV頻道JVR Lyric MV": "Music", "薛汀哲": "Music", "Salah Trainer": "Sports", "Skills N Talents (swimming)": "Sports", "TotalSports TV": "Sports", "WorldSBK": "Sports", "Everyday Cycling": "Sports", "爱羽客羽毛球网": "Sports", "adidas": "Sports", "Sneaks & Feet极客鞋谈": "Automotive", "TOP的遥控玩具": "Automotive", "悟空的日常": "Automotive"}

frontend/src/auto_categories.json

@@ -0,0 +1 @@
+{"AI Engineer": "AI & Tech", "AI-Fan AI\u7814\u7a76\u5ba4-\u5e06\u54e5": "AI & Tech", "AI\u5927\u6a21\u578b": "AI & Tech", "AI\u63a2\u7d22\u4e0e\u53d1\u73b0": "AI & Tech", "Aurora Tech": "AI & Tech", "Claude": "AI & Tech", "ColdFusion": "AI & Tech", "Data Is Beautiful": "AI & Tech", "DeepLearningAI": "AI & Tech", "Google": "AI & Tech", "Google Cloud Tech": "AI & Tech", "Lex Fridman": "AI & Tech", "NiceKate AI": "AI & Tech", "PC Sim": "AI & Tech", "Stanford Online": "AI & Tech", "The Linux Foundation": "AI & Tech", "\u56de\u5230Axton": "AI & Tech", "\u56de\u5f62\u9488PaperClip": "AI & Tech", "\u6797\u4ea6LYi": "AI & Tech", "\u7845\u8c37101\u64ad\u5ba2": "AI & Tech", "\u8001\u77f3\u8c08\u82af": "AI & Tech", "\u9cb2\u9e4fTalk": "AI & Tech", "3D Beast": "3D & Design", "Arrimus3D Modeling and Design": "3D & Design", "Blender": "3D & Design", "Blender Guru": "3D & Design", "Creators 3D": "3D & Design", "Designfusion": "3D & Design", "Foundry": "3D & Design", "Grant Abbitt (Gabbitt)": "3D & Design", "Gravity Sketch": "3D & Design", "Hyperganic": "3D & Design", "MSLattice": "3D & Design", "Manolo Remiddi": "3D & Design", "Rhino 3D (Rhinoceros3d official)": "3D & Design", "Shaper3d": "3D & Design", "nTop": "3D & Design", "spherene AG": "3D & Design", "\u9b54\u754c\u9020\u7269": "3D & Design", "Chuck Severance": "Programming", "Coursera": "Programming", "freeCodeCamp.org": "Programming", "Kevin Stratvert": "Programming", "Leila Gharani": "Programming", "LinkedIn Learning": "Programming", "Metabase": "Programming", "n8n": "Programming", "The Coding Gopher": "Programming", "Brian Casel": "Programming", "code\u79d8\u5bc6\u82b1\u56ed": "Programming", "cs tech": "Programming", "fequalsf": "Programming", "openscreencast": "Programming", "\u6280\u672f\u722c\u722c\u867e  TechShrimp": "Programming", "\u6797\u672c\u5154Limbuntu": "Programming", "Best Choice Review": "Hardware & Reviews", "Marques Brownlee": "Hardware & Reviews", "TESTV": "Hardware & Reviews", "\u5148\u770b\u8bc4\u6d4b": "Hardware & Reviews", "\u6781\u5ba2\u6e7eGeekerwan": "Hardware & Reviews", "\u786c\u6838\u62c6\u89e3": "Hardware & Reviews", "\u7b14\u5427\u8bc4\u6d4b\u5ba4": "Hardware & Reviews", "\u5dee\u8bc4\u786c\u4ef6\u90e8": "Hardware & Reviews", "\u5145\u7535\u5934\u7f51": "Hardware & Reviews", "Apple Developer": "Apple Dev", "Made by Google": "Apple Dev", "Allen\u7684\u5206\u4eab": "Apple Dev", "Sypnotix": "Apple Dev", "8K World": "Photography", "Can Fiona": "Photography", "\u5c0f\u5929fotos": "Photography", "\u5f71\u50cf\u6781\u5ba2Fotogeeker": "Photography", "\u6444\u5f71\u5e08PHiL": "Photography", "\u865a\u7a7a\u5149\u5f71CosmosFilm": "Photography", "\u8def\u6613\u65af LouisDrone": "Photography", "Bruno Mars": "Music", "GraceLeeMusic": "Music", "Katie Melua": "Music", "The Weeknd": "Music", "TheWeekndVEVO": "Music", "\u6770\u5a01\u723e\u6b4c\u8a5eMV\u983b\u9053JVR Lyric MV": "Music", "\u859b\u6c40\u54f2": "Music", "Salah Trainer": "Sports", "Skills N Talents (swimming)": "Sports", "TotalSports TV": "Sports", "WorldSBK": "Sports", "Everyday Cycling": "Sports", "\u7231\u7fbd\u5ba2\u7fbd\u6bdb\u7403\u7f51": "Sports", "adidas": "Sports", "Sneaks & Feet\u6781\u5ba2\u978b\u8c08": "Automotive", "TOP\u7684\u9065\u63a7\u73a9\u5177": "Automotive", "\u609f\u7a7a\u7684\u65e5\u5e38": "Automotive"}

run_sync.py

@@ -0,0 +1,61 @@
+#!/usr/bin/env python3
+import asyncio, os, sys, logging
+
+REPO_ROOT = os.path.dirname(os.path.abspath(__file__))
+BACKEND = os.path.join(REPO_ROOT, "backend")
+ENV_PATH = os.path.join(REPO_ROOT, ".env")
+if os.path.exists(ENV_PATH):
+    with open(ENV_PATH) as f:
+        for line in f:
+            line = line.strip()
+            if line and not line.startswith("#") and "=" in line:
+                key, _, value = line.partition("=")
+                value = value.strip().strip("'\"")
+                os.environ[key] = value
+
+sys.path.insert(0, BACKEND)
+os.chdir(BACKEND)
+
+from db import init_db, DB_PATH
+from fetcher import fetch_subscriptions
+from summarizer import summarize_video
+import aiosqlite
+
+BATCH_SIZE = int(os.environ.get("SUMMARIZE_BATCH", "20"))
+BATCH_DELAY = float(os.environ.get("SUMMARIZE_BATCH_DELAY", "2.0"))
+
+logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
+
+async def main():
+    await init_db()
+    
+    count = await fetch_subscriptions()
+    print(f"Fetched {count} new videos")
+    
+    async with aiosqlite.connect(DB_PATH) as db:
+        cur = await db.execute("SELECT COUNT(*) FROM videos WHERE summary IS NULL OR summary = ''")
+        total = (await cur.fetchone())[0]
+    
+    if total == 0:
+        print("No pending videos to summarize.")
+        return
+    
+    print(f"Processing batch of {BATCH_SIZE} (of {total} pending)...")
+    
+    async with aiosqlite.connect(DB_PATH) as db:
+        db.row_factory = aiosqlite.Row
+        cur = await db.execute("SELECT video_id FROM videos WHERE summary IS NULL OR summary = '' LIMIT ?", (BATCH_SIZE,))
+        rows = await cur.fetchall()
+    
+    done = 0
+    for row in rows:
+        await summarize_video(row["video_id"])
+        done += 1
+        if done < len(rows):
+            await asyncio.sleep(BATCH_DELAY)
+    
+    remaining = total - done
+    print(f"Batch done: {done}/{len(rows)}, {remaining} pending")
+
+if __name__ == "__main__":
+    asyncio.run(main())

run_tests.sh

@@ -4,15 +4,20 @@ set -e
 DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 VIDEO_ID="jNQXAC9IVRw"
 
-# 1. Auto-detect and configure SOCKS5 proxy if active on 7890
+# 1. Auto-detect and configure proxy
+PROXY_DETECTED=false
 if nc -z 127.0.0.1 7890 >/dev/null 2>&1; then
-    echo "=== Auto-detected local proxy on port 7890. Enabling proxy... ==="
+    echo "=== Auto-detected local proxy on port 7890. Enabling SOCKS5 proxy... ==="
     export http_proxy=socks5://127.0.0.1:7890
     export https_proxy=socks5://127.0.0.1:7890
     export all_proxy=socks5://127.0.0.1:7890
     export HTTP_PROXY=socks5://127.0.0.1:7890
     export HTTPS_PROXY=socks5://127.0.0.1:7890
     export ALL_PROXY=socks5://127.0.0.1:7890
+    PROXY_DETECTED=true
+elif [ -n "$HTTP_PROXY" ] || [ -n "$http_proxy" ]; then
+    echo "=== Using existing HTTP_PROXY env var: ${HTTP_PROXY:-$http_proxy} ==="
+    PROXY_DETECTED=true
 fi
 
 # 2. Set up virtual environment and install requirements if not set up
@@ -25,9 +30,11 @@ fi
 echo "=== Step 1: Running Mock Unit Tests ==="
 "$DIR/backend/.venv/bin/python3" -m pytest "$DIR/backend/tests/"
 
-echo ""
+# Only run live integration tests if a proxy is available (needed from China)
-echo "=== Step 2: Seeding DB for Live Integration Test ==="
+if [ "$PROXY_DETECTED" = true ]; then
-"$DIR/backend/.venv/bin/python3" -c "import asyncio, aiosqlite, sys; sys.path.insert(0, '$DIR/backend'); import db;
+    echo ""
+    echo "=== Step 2: Seeding DB for Live Integration Test ==="
+    "$DIR/backend/.venv/bin/python3" -c "import asyncio, aiosqlite, sys; sys.path.insert(0, '$DIR/backend'); import db;
 async def run():
     await db.init_db()
     async with aiosqlite.connect(db.DB_PATH) as conn:
@@ -35,14 +42,15 @@ async def run():
         await conn.commit()
 asyncio.run(run())"
 
-echo "=== Step 3: Running Live Ingestion & AI Summarization ==="
+    echo "=== Step 3: Running Live Ingestion & AI Summarization ==="
-set +e
+    set +e
-"$DIR/backend/.venv/bin/python3" "$DIR/backend/summarizer.py" "$VIDEO_ID"
+    "$DIR/backend/.venv/bin/python3" "$DIR/backend/summarizer.py" "$VIDEO_ID"
-set -e
+    SUMMARIZER_EXIT=$?
+    set -e
 
-echo ""
+    echo ""
-echo "=== Step 4: Verifying SQLite Database Storage ==="
+    echo "=== Step 4: Verifying SQLite Database Storage ==="
-"$DIR/backend/.venv/bin/python3" -c "import asyncio, aiosqlite, sys; sys.path.insert(0, '$DIR/backend'); import db;
+    "$DIR/backend/.venv/bin/python3" -c "import asyncio, aiosqlite, sys; sys.path.insert(0, '$DIR/backend'); import db;
 async def run():
     async with aiosqlite.connect(db.DB_PATH) as conn:
         conn.row_factory = aiosqlite.Row
@@ -57,9 +65,17 @@ async def run():
             print()
             print('=============================================')
             print('WARNING: LIVE INTEGRATION TEST COULD NOT COMPLETE')
-            print('YouTube blocked this request or cookies.txt was missing.')
+            print('YouTube blocked this request or DeepSeek is unreachable.')
             print('Please ensure your proxy is active on port 7890 or export')
-            print('your youtube cookies to config/cookies.txt.')
+            print('HTTP_PROXY before running.')
             print('This warning is expected and did not break the build.')
             print('=============================================')
 asyncio.run(run())"
+else
+    echo ""
+    echo "=== Skipping Live Integration Test (no proxy detected) ==="
+    echo "From China, YouTube API and DeepSeek require a proxy."
+    echo "Export HTTP_PROXY or SOCKS5 on port 7890 to enable."
+    echo ""
+    echo "=== Mock tests passed — live test skipped (OK) ==="
+fi