From 43977205b66c7556059323194eeff6b81a6338fd Mon Sep 17 00:00:00 2001
From: "Gan, Jimmy" <jimmy.gan@adidas.com>
Date: Fri, 10 Jul 2026 00:18:50 +0800
Subject: [PATCH] docs: add comprehensive project docs (architecture, progress,
 configuration, troubleshooting)

---
 README.md               |  42 ++++++++-
 docs/troubleshooting.md | 194 ++++++++++++++++++++++++++++++++++++++++
 2 files changed, 235 insertions(+), 1 deletion(-)
 create mode 100644 docs/troubleshooting.md

diff --git a/README.md b/README.md
index 76ed38d..f00bb7f 100644
--- a/README.md
+++ b/README.md
@@ -63,4 +63,44 @@ t-youtube/
     *   **Key Takeaways (Bullet list)**
     *   **Actionable Verdict**
 3.  **Read/Skip State**: Mark videos as **Read** or **Skip** to clear them from your pending queue, allowing you to only keep track of what's unread.
-4.  **View Full Transcript**: Expand the accordion at the bottom of the digest to read the raw transcripts if needed.
+4. **View Full Transcript**: Expand the accordion at the bottom of the digest to read the raw transcripts if needed.
+5. **Agent Queue**: Save videos with notes to a queue for Hermes agents to process (create skills, open issues, etc.)
+
+---
+
+## 📚 Documentation
+
+Comprehensive documentation for this project is available in the `docs/` folder:
+
+- **[Architecture](docs/architecture.md)** - System overview, components, API endpoints, database schema
+- **[Progress](docs/progress.md)** - Session history, completed tasks, planned features, metrics
+- **[Configuration](docs/configuration.md)** - Environment variables, API tokens, database, Authelia, Caddy setup
+- **[Troubleshooting](docs/troubleshooting.md)** - Common issues, recovery procedures, monitoring
+
+---
+
+## 🚀 Setup Instructions
+
+### 1. Extract your YouTube browser cookies
+Since YouTube restricts subscription feeds to logged-in users, you need to export your logged-in cookies so `yt-dlp` can request the feed on your behalf:
+1. Install a browser extension like **Get cookies.txt LOCALLY** (Chrome/Edge) or **cookies.txt** (Firefox).
+2. Open YouTube in your browser and ensure you are logged in to your account.
+3. Click the extension icon and export the cookies for `youtube.com` as a Netscape-formatted text file.
+4. Create a folder named `config` in your project root, rename the exported file to `cookies.txt`, and save it inside:
+   ```text
+   t-youtube/config/cookies.txt
+   ```
+
+### 2. Configure Environment Variables
+Create a `.env` file in the root directory:
+```env
+# Optional: Override the fallback OpenAI / Anthropic key
+OPENAI_API_KEY=sk-Kq5...rOlx
+```
+
+### 3. Build & Deploy using Docker Compose
+Run the following command to build the multi-stage image (compiles frontend static assets using Node, then packages backend with Python) and start the container:
+```bash
+docker compose up -d --build
+```
+The server will boot up and listen on port `8000`.
diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md
new file mode 100644
index 0000000..b48a053
--- /dev/null
+++ b/docs/troubleshooting.md
@@ -0,0 +1,194 @@
+# t-youtube Troubleshooting Guide
+
+## Common Issues
+
+### 1. Site Down / 502 Bad Gateway
+- **Cause**: t-youtube container not running or Authelia down
+- **Check**:
+  ```bash
+  ssh nas 'docker ps | grep t-youtube'
+  ssh nas 'docker ps | grep authelia'
+  ```
+- **Fix**:
+  ```bash
+  cd /volume1/docker/t-youtube && /usr/local/bin/docker compose up -d
+  cd /volume1/docker/authelia && /usr/local/bin/docker compose restart authelia
+  ```
+
+### 2. Authelia Loop / Login Fails
+- **Cause**: WebAuthn/TOTP misconfiguration or session expired
+- **Check Authelia logs**:
+  ```bash
+  ssh nas 'docker logs authelia --tail 20'
+  ```
+- **Fix**:
+  - Clear browser cache/cookies
+  - Reset WebAuthn via Authelia DB (see below)
+  - Check dnsmasq config (should point to VPS Tailscale IP)
+
+### 3. YouTube Data API Quota Exceeded
+- **Symptom**: Fetcher returns empty results or 403 errors
+- **Cause**: Daily quota limit reached (10,000 units)
+- **Check**:
+  ```bash
+  ssh nas 'docker logs t-youtube 2>&1 | grep -i "quota\|403"'
+  ```
+- **Fix**:
+  - Wait until next day (quota resets at 00:00 UTC)
+  - Reduce number of channels or increase sync interval
+  - Monitor usage in Google Cloud Console
+
+### 4. Summarization Failing
+- **Symptom**: Videos show "No summary" or generic descriptions
+- **Cause**: DeepSeek API key invalid or rate limited
+- **Check**:
+  ```bash
+  ssh nas 'docker logs t-youtube 2>&1 | grep -i "summariz\|deepseek"'
+  ```
+- **Fix**:
+  - Verify `DEEPSEEK_API_KEY` in `.env`
+  - Test API key manually:
+    ```bash
+    curl -s -X POST https://api.deepseek.com/chat/completions \
+      -H "Authorization: Bearer ***
+      -H "Content-Type: application/json" \
+      -d '{"model": "deepseek-chat", "messages": [{"role": "user", "content": "test"}]}'
+    ```
+
+### 5. Database Locked/Corrupted
+- **Symptom**: API returns 500 errors or hangs
+- **Cause**: Concurrent write operations or unexpected shutdown
+- **Fix**:
+  ```bash
+  cd /volume1/docker/t-youtube/backend
+  cp t_youtube.db t_youtube.db.backup
+  sqlite3 t_youtube.db "PRAGMA integrity_check;"
+  # If corrupted, restore from backup or recreate
+  ```
+
+### 6. Caddy Config Issues
+- **Symptom**: 502/503 errors, wrong cert, or domain unreachable
+- **Check**:
+  ```bash
+  ssh caddy-vps "sudo caddy validate --config /etc/caddy/Caddyfile"
+  ssh nas 'docker exec caddy caddy validate --config /etc/caddy/Caddyfile'
+  ```
+- **Fix**:
+  - Validate config
+  - Reload/restart Caddy
+  - Check TLS certs:
+    ```bash
+    echo | openssl s_client -connect 161.33.182.109:443 -servername yt.jimmygan.com 2>&1 | openssl x509 -noout -subject -dates
+    ```
+
+## API Debugging
+
+### Check Video List
+```bash
+curl -s https://yt.jimmygan.com/api/videos | python3 -m json.tool | head -20
+```
+
+### Check Agent Queue
+```bash
+curl -s -H "Authorization: Bearer *** https://yt.jimmygan.com/api/agent-queue
+```
+
+### Check Notes
+```bash
+curl -s -H "Authorization: Bearer *** https://yt.jimmygan.com/api/notes
+```
+
+### Check Channel List
+```bash
+curl -s -H "Authorization: Bearer *** https://yt.jimmygan.com/api/channels
+```
+
+### Test Summarization
+```bash
+curl -s -X POST -H "Authorization: Bearer ***
+  -H "Content-Type: application/json" \
+  -d '{"video_id": "<video_id>"}' \
+  https://yt.jimmygan.com/api/agent-queue/<video_id>/summarize
+```
+
+## Database Queries
+
+### Count Videos by Status
+```sql
+SELECT status, COUNT(*) FROM videos GROUP BY status;
+```
+
+### Count Agent Queue Items by Status
+```sql
+SELECT status, COUNT(*) FROM agent_queue GROUP BY status;
+```
+
+### Find Videos Without Summaries
+```sql
+SELECT video_id, title FROM videos WHERE summary IS NULL OR summary = '';
+```
+
+### Check DB Size
+```bash
+ls -lh /volume1/docker/t-youtube/backend/t_youtube.db
+```
+
+## Logs
+
+### t-youtube Logs
+```bash
+ssh nas 'docker logs t-youtube --tail 50'
+ssh nas 'docker logs t-youtube --tail 50 --since 1h'
+```
+
+### Authelia Logs
+```bash
+ssh nas 'docker logs authelia --tail 50'
+```
+
+### Sync Logs
+```bash
+ssh nas 'cat /var/log/t-youtube-sync.log'
+ssh nas 'tail -100 /var/log/t-youtube-sync.log'
+```
+
+### Caddy Logs (VPS)
+```bash
+ssh caddy-vps "sudo journalctl -u caddy --since '1 hour ago'"
+ssh caddy-vps "sudo journalctl -u caddy --no-pager -n 30"
+```
+
+## Recovery Procedures
+
+### Full System Restart
+```bash
+# 1. Restart services
+cd /volume1/docker/authelia && /usr/local/bin/docker compose restart authelia
+cd /volume1/docker/t-youtube && /usr/local/bin/docker compose up -d
+
+# 2. Verify
+curl -s https://yt.jimmygan.com/api/health
+curl -s https://auth.jimmygan.com/api/health
+```
+
+### Database Backup & Restore
+```bash
+# Backup
+cp /volume1/docker/t-youtube/backend/t_youtube.db /tmp/t-youtube-backup-$(date +%Y%m%d).db
+
+# Restore
+cp /tmp/t-youtube-backup-20260709.db /volume1/docker/t-youtube/backend/t_youtube.db
+cd /volume1/docker/t-youtube && /usr/local/bin/docker compose restart t-youtube
+```
+
+### Authelia User Reset (if locked out)
+```bash
+# Stop Authelia
+ssh nas 'docker stop authelia'
+
+# Reset TOTP for jimmy
+ssh nas 'sqlite3 /volume1/docker/authelia/config/db.sqlite3 "DELETE FROM totp_configurations WHERE username='\'jimmy'\'';"'
+
+# Restart Authelia
+ssh nas 'docker start authelia'
+```