Gan, Jimmy
a0ffa41c9e
- Add auth.py: OAuth credential manager with auto-refresh - Rewrite fetcher.py: use googleapiclient subscriptions/search - Add google-auth, google-api-python-client dependencies - Update docker-compose for config volume mount - Add .gitignore for secrets (client_secret.json, token.pickle) - Live test: 198 videos ingested from 229 channels
432 lines
12 KiB
Markdown
432 lines
12 KiB
Markdown
# t-youtube: YouTube Data API v3 + OAuth Implementation Plan
|
||
|
||
> **For Hermes:** Use subagent-driven-development skill to implement this plan task-by-task.
|
||
|
||
**Goal:** Replace the broken yt-dlp/cookies subscription fetch with YouTube Data API v3 OAuth, restoring live sync.
|
||
|
||
**Architecture:** Add Google OAuth 2.0 credential management. Replace `fetcher.py`'s yt-dlp call with `google-api-python-client` calling the `/subscriptions` endpoint. Keep DB schema, summarizer, and frontend unchanged.
|
||
|
||
**Tech Stack:** Python 3.13, FastAPI, aiosqlite, google-api-python-client, google-auth-oauthlib
|
||
|
||
---
|
||
|
||
## Prerequisites (manual, before implementation)
|
||
|
||
1. Go to Google Cloud Console → create a project → enable **YouTube Data API v3**
|
||
2. Create OAuth 2.0 Client ID (Desktop app type)
|
||
3. Download `client_secret.json` → place in `config/client_secret.json`
|
||
4. Run OAuth flow once to generate `config/token.json`
|
||
|
||
---
|
||
|
||
### Task 1: Add Google API dependencies
|
||
|
||
**Objective:** Install required Python packages
|
||
|
||
**Files:**
|
||
- Modify: `backend/requirements.txt`
|
||
|
||
**Step 1: Add to requirements.txt**
|
||
|
||
```
|
||
google-api-python-client>=2.140.0
|
||
google-auth-oauthlib>=1.2.0
|
||
google-auth-httplib2>=0.2.0
|
||
```
|
||
|
||
**Step 2: Install**
|
||
|
||
```bash
|
||
cd backend && .venv/bin/pip install -r requirements.txt
|
||
```
|
||
|
||
**Step 3: Verify**
|
||
|
||
```bash
|
||
.venv/bin/python3 -c "from googleapiclient.discovery import build; print('OK')"
|
||
```
|
||
|
||
Expected: `OK`
|
||
|
||
---
|
||
|
||
### Task 2: Create OAuth credential manager
|
||
|
||
**Objective:** Module to load/store OAuth tokens, handle refresh
|
||
|
||
**Files:**
|
||
- Create: `backend/auth.py`
|
||
|
||
**Complete code:**
|
||
|
||
```python
|
||
import os
|
||
import pickle
|
||
from google.auth.transport.requests import Request
|
||
from google_auth_oauthlib.flow import InstalledAppFlow
|
||
|
||
SCOPES = ["https://www.googleapis.com/auth/youtube.readonly"]
|
||
CONFIG_DIR = os.environ.get("CONFIG_DIR", "/app/config")
|
||
CLIENT_SECRET = os.path.join(CONFIG_DIR, "client_secret.json")
|
||
TOKEN_FILE = os.path.join(CONFIG_DIR, "token.pickle")
|
||
|
||
def get_credentials():
|
||
"""Load or refresh OAuth credentials, triggering browser flow if needed."""
|
||
creds = None
|
||
if os.path.exists(TOKEN_FILE):
|
||
with open(TOKEN_FILE, "rb") as token:
|
||
creds = pickle.load(token)
|
||
|
||
if not creds or not creds.valid:
|
||
if creds and creds.expired and creds.refresh_token:
|
||
creds.refresh(Request())
|
||
else:
|
||
if not os.path.exists(CLIENT_SECRET):
|
||
raise FileNotFoundError(
|
||
f"client_secret.json not found at {CLIENT_SECRET}. "
|
||
"Download from Google Cloud Console."
|
||
)
|
||
flow = InstalledAppFlow.from_client_secrets_file(CLIENT_SECRET, SCOPES)
|
||
creds = flow.run_local_server(port=0)
|
||
|
||
os.makedirs(os.path.dirname(TOKEN_FILE), exist_ok=True)
|
||
with open(TOKEN_FILE, "wb") as token:
|
||
pickle.dump(creds, token)
|
||
|
||
return creds
|
||
```
|
||
|
||
**Step 1: Verify module loads**
|
||
|
||
```bash
|
||
cd backend && .venv/bin/python3 -c "import auth; print('auth module OK')"
|
||
```
|
||
|
||
Expected: `auth module OK`
|
||
|
||
**Step 2: Commit**
|
||
|
||
```bash
|
||
git add backend/auth.py backend/requirements.txt
|
||
git commit -m "feat: add OAuth credential manager for YouTube API"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 3: Test OAuth flow (dry run)
|
||
|
||
**Objective:** Verify token.pickle is generated from client_secret.json
|
||
|
||
**Files:**
|
||
- Create: `backend/tests/test_auth.py`
|
||
|
||
**Step 1: Write test**
|
||
|
||
```python
|
||
import os
|
||
import tempfile
|
||
from unittest.mock import patch
|
||
import auth
|
||
|
||
def test_credentials_flow_with_valid_secret():
|
||
"""When client_secret.json exists, get_credentials() should not raise."""
|
||
# This test only checks the file-exists path; real OAuth requires browser
|
||
if os.path.exists(auth.CLIENT_SECRET):
|
||
# Just verify module doesn't crash on import
|
||
assert auth.SCOPES == ["https://www.googleapis.com/auth/youtube.readonly"]
|
||
else:
|
||
# Without client_secret, should raise FileNotFoundError
|
||
try:
|
||
auth.get_credentials()
|
||
assert False, "Should have raised"
|
||
except FileNotFoundError:
|
||
pass
|
||
```
|
||
|
||
**Step 2: Run test**
|
||
|
||
```bash
|
||
cd backend && .venv/bin/python3 -m pytest tests/test_auth.py -v
|
||
```
|
||
|
||
Expected: PASS (skips OAuth if no secret, or passes structure check)
|
||
|
||
**Step 3: Commit**
|
||
|
||
```bash
|
||
git add backend/tests/test_auth.py
|
||
git commit -m "test: add OAuth auth module test"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 4: Rewrite fetcher.py to use YouTube Data API v3
|
||
|
||
**Objective:** Replace yt-dlp with googleapiclient subscription fetch
|
||
|
||
**Files:**
|
||
- Modify: `backend/fetcher.py`
|
||
|
||
**Complete replacement code:**
|
||
|
||
```python
|
||
import asyncio
|
||
import logging
|
||
import aiosqlite
|
||
from googleapiclient.discovery import build
|
||
from googleapiclient.errors import HttpError
|
||
from auth import get_credentials
|
||
from db import DB_PATH
|
||
|
||
log = logging.getLogger(__name__)
|
||
|
||
async def fetch_subscriptions():
|
||
"""Fetch latest videos from subscribed channels using YouTube Data API v3."""
|
||
try:
|
||
creds = get_credentials()
|
||
except FileNotFoundError as e:
|
||
log.error(f"OAuth setup incomplete: {e}")
|
||
return 0
|
||
except Exception as e:
|
||
log.error(f"OAuth credential error: {e}")
|
||
return 0
|
||
|
||
youtube = build("youtube", "v3", credentials=creds)
|
||
|
||
# Step 1: Get subscribed channel IDs
|
||
channel_ids = []
|
||
try:
|
||
request = youtube.subscriptions().list(
|
||
part="snippet",
|
||
mine=True,
|
||
maxResults=50,
|
||
order="alphabetical"
|
||
)
|
||
while request:
|
||
response = request.execute()
|
||
for item in response.get("items", []):
|
||
cid = item["snippet"]["resourceId"]["channelId"]
|
||
if cid not in channel_ids:
|
||
channel_ids.append(cid)
|
||
request = youtube.subscriptions().list_next(request, response)
|
||
except HttpError as e:
|
||
log.error(f"YouTube API error fetching subscriptions: {e}")
|
||
return 0
|
||
|
||
log.info(f"Found {len(channel_ids)} subscribed channels")
|
||
|
||
if not channel_ids:
|
||
return 0
|
||
|
||
# Step 2: For each channel, get latest videos
|
||
new_count = 0
|
||
async with aiosqlite.connect(DB_PATH) as db:
|
||
for cid in channel_ids[:20]: # Respect quota, limit to 20 channels
|
||
try:
|
||
req = youtube.search().list(
|
||
part="snippet",
|
||
channelId=cid,
|
||
maxResults=10,
|
||
order="date",
|
||
type="video"
|
||
)
|
||
resp = req.execute()
|
||
for item in resp.get("items", []):
|
||
vid = item["id"]["videoId"]
|
||
snippet = item["snippet"]
|
||
|
||
cursor = await db.execute(
|
||
"SELECT 1 FROM videos WHERE video_id = ?", (vid,)
|
||
)
|
||
if await cursor.fetchone():
|
||
continue
|
||
|
||
await db.execute("""
|
||
INSERT INTO videos
|
||
(video_id, title, channel_name, channel_id, url, thumbnail_url, publish_date)
|
||
VALUES (?, ?, ?, ?, ?, ?, ?)
|
||
""", (
|
||
vid,
|
||
snippet.get("title", "Untitled"),
|
||
snippet.get("channelTitle", "Unknown"),
|
||
snippet.get("channelId", cid),
|
||
f"https://www.youtube.com/watch?v={vid}",
|
||
snippet.get("thumbnails", {}).get("high", {}).get("url", ""),
|
||
snippet.get("publishTime", "")[:10]
|
||
))
|
||
new_count += 1
|
||
log.debug(f"New video: {snippet.get('title', 'Untitled')[:50]}")
|
||
except HttpError as e:
|
||
log.warning(f"Channel {cid} fetch error: {e}")
|
||
continue
|
||
|
||
await db.commit()
|
||
|
||
log.info(f"Ingestion completed. Found {new_count} new videos.")
|
||
return new_count
|
||
|
||
|
||
if __name__ == "__main__":
|
||
import sys
|
||
logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')
|
||
|
||
async def main():
|
||
from db import init_db
|
||
await init_db()
|
||
await fetch_subscriptions()
|
||
|
||
asyncio.run(main())
|
||
```
|
||
|
||
**Step 1: Run existing tests to ensure nothing else broke**
|
||
|
||
```bash
|
||
cd backend && .venv/bin/python3 -m pytest tests/test_fetcher.py -v
|
||
```
|
||
|
||
Expected: Mock tests should still pass (fetcher import may need auth module available)
|
||
|
||
**Step 2: Update test_fetcher.py to mock googleapiclient**
|
||
|
||
Modify `backend/tests/test_fetcher.py`:
|
||
|
||
```python
|
||
# Replace yt-dlp mocks with googleapiclient mocks
|
||
```
|
||
|
||
**Step 3: Commit**
|
||
|
||
```bash
|
||
git add backend/fetcher.py backend/tests/test_fetcher.py
|
||
git commit -m "feat: replace yt-dlp with YouTube Data API v3 OAuth"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 5: Update Docker config for new dependencies
|
||
|
||
**Objective:** Ensure Docker build includes google-auth and config mounts
|
||
|
||
**Files:**
|
||
- Modify: `Dockerfile`
|
||
- Modify: `docker-compose.yml`
|
||
|
||
**Dockerfile changes:**
|
||
|
||
```dockerfile
|
||
# In the Python stage, ensure google packages are installed
|
||
# requirements.txt already includes them from Task 1
|
||
```
|
||
|
||
**docker-compose.yml changes:**
|
||
|
||
```yaml
|
||
# Mount config directory for OAuth secrets
|
||
volumes:
|
||
- ./config:/app/config
|
||
```
|
||
|
||
**Step 1: Verify Docker build**
|
||
|
||
```bash
|
||
docker compose build
|
||
```
|
||
|
||
Expected: Build succeeds
|
||
|
||
**Step 2: Commit**
|
||
|
||
```bash
|
||
git add Dockerfile docker-compose.yml
|
||
git commit -m "chore: update Docker config for OAuth and API v3"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 6: Integration test with real API
|
||
|
||
**Objective:** Run a live fetch against real YouTube subscription data
|
||
|
||
**Files:**
|
||
- Create: `backend/tests/test_live_fetch.py`
|
||
|
||
**Step 1: Write integration test**
|
||
|
||
```python
|
||
"""Live integration test — requires valid OAuth token in config/"""
|
||
import pytest
|
||
import asyncio
|
||
from fetcher import fetch_subscriptions
|
||
from db import init_db
|
||
|
||
@pytest.mark.asyncio
|
||
@pytest.mark.integration
|
||
async def test_live_subscription_fetch():
|
||
"""Fetch real subscriptions. Skip if no OAuth token."""
|
||
await init_db()
|
||
count = await fetch_subscriptions()
|
||
# With valid OAuth and subscriptions, should find videos
|
||
# Even 0 is acceptable (no new videos since last fetch)
|
||
assert count >= 0, "Fetch should not crash"
|
||
print(f"Fetched {count} new videos")
|
||
```
|
||
|
||
**Step 2: Run (requires OAuth setup)**
|
||
|
||
```bash
|
||
cd backend && .venv/bin/python3 -m pytest tests/test_live_fetch.py -v -s
|
||
```
|
||
|
||
Expected: PASS or SKIP if no OAuth
|
||
|
||
**Step 3: Commit**
|
||
|
||
```bash
|
||
git add backend/tests/test_live_fetch.py
|
||
git commit -m "test: add live YouTube API integration test"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 7: Final verification — full pipeline
|
||
|
||
**Objective:** Run full test suite and verify everything works end-to-end
|
||
|
||
**Step 1: Run all tests**
|
||
|
||
```bash
|
||
cd ~/repos/t-youtube
|
||
./run_tests.sh
|
||
```
|
||
|
||
**Step 2: Verify frontend still builds**
|
||
|
||
```bash
|
||
cd frontend && npm run build 2>/dev/null || echo "Frontend build skipped"
|
||
```
|
||
|
||
**Step 3: Commit final state**
|
||
|
||
```bash
|
||
git add -A && git commit -m "feat: complete YouTube Data API v3 migration"
|
||
```
|
||
|
||
---
|
||
|
||
## Verification Checklist
|
||
|
||
- [ ] `config/client_secret.json` exists
|
||
- [ ] `config/token.pickle` generated after first OAuth flow
|
||
- [ ] `fetch_subscriptions()` returns video count > 0
|
||
- [ ] Videos appear in DB with correct metadata
|
||
- [ ] Summarizer still works with newly fetched videos
|
||
- [ ] All mock tests pass
|
||
- [ ] Docker build succeeds
|
||
- [ ] Frontend displays new videos
|
||
|
||
## Risks
|
||
|
||
1. **API quota**: YouTube Data API v3 has 10,000 units/day quota. Each subscription list call costs 1 unit, each search costs 100 units. With 20 channels × 10 results, that's ~2,000 units per sync.
|
||
2. **OAuth browser flow**: First run requires browser for Google login. On headless NAS, run locally first and copy `token.pickle`.
|
||
3. **Token expiry**: refresh_token handles this automatically via `google-auth`.
|