# t-youtube: YouTube Data API v3 + OAuth Implementation Plan

> **For Hermes:** Use subagent-driven-development skill to implement this plan task-by-task.

**Goal:** Replace the broken yt-dlp/cookies subscription fetch with YouTube Data API v3 OAuth, restoring live sync.

**Architecture:** Add Google OAuth 2.0 credential management. Replace `fetcher.py`'s yt-dlp call with `google-api-python-client` calling the `/subscriptions` endpoint. Keep DB schema, summarizer, and frontend unchanged.

**Tech Stack:** Python 3.13, FastAPI, aiosqlite, google-api-python-client, google-auth-oauthlib

---

## Prerequisites (manual, before implementation)

1. Go to Google Cloud Console → create a project → enable **YouTube Data API v3**
2. Create OAuth 2.0 Client ID (Desktop app type)
3. Download `client_secret.json` → place in `config/client_secret.json`
4. Run OAuth flow once to generate `config/token.json`

---

### Task 1: Add Google API dependencies

**Objective:** Install required Python packages

**Files:**
- Modify: `backend/requirements.txt`

**Step 1: Add to requirements.txt**

```
google-api-python-client>=2.140.0
google-auth-oauthlib>=1.2.0
google-auth-httplib2>=0.2.0
```

**Step 2: Install**

```bash
cd backend && .venv/bin/pip install -r requirements.txt
```

**Step 3: Verify**

```bash
.venv/bin/python3 -c "from googleapiclient.discovery import build; print('OK')"
```

Expected: `OK`

---

### Task 2: Create OAuth credential manager

**Objective:** Module to load/store OAuth tokens, handle refresh

**Files:**
- Create: `backend/auth.py`

**Complete code:**

```python
import os
import pickle
from google.auth.transport.requests import Request
from google_auth_oauthlib.flow import InstalledAppFlow

SCOPES = ["https://www.googleapis.com/auth/youtube.readonly"]
CONFIG_DIR = os.environ.get("CONFIG_DIR", "/app/config")
CLIENT_SECRET = os.path.join(CONFIG_DIR, "client_secret.json")
TOKEN_FILE = os.path.join(CONFIG_DIR, "token.pickle")

def get_credentials():
    """Load or refresh OAuth credentials, triggering browser flow if needed."""
    creds = None
    if os.path.exists(TOKEN_FILE):
        with open(TOKEN_FILE, "rb") as token:
            creds = pickle.load(token)

    if not creds or not creds.valid:
        if creds and creds.expired and creds.refresh_token:
            creds.refresh(Request())
        else:
            if not os.path.exists(CLIENT_SECRET):
                raise FileNotFoundError(
                    f"client_secret.json not found at {CLIENT_SECRET}. "
                    "Download from Google Cloud Console."
                )
            flow = InstalledAppFlow.from_client_secrets_file(CLIENT_SECRET, SCOPES)
            creds = flow.run_local_server(port=0)

        os.makedirs(os.path.dirname(TOKEN_FILE), exist_ok=True)
        with open(TOKEN_FILE, "wb") as token:
            pickle.dump(creds, token)

    return creds
```

**Step 1: Verify module loads**

```bash
cd backend && .venv/bin/python3 -c "import auth; print('auth module OK')"
```

Expected: `auth module OK`

**Step 2: Commit**

```bash
git add backend/auth.py backend/requirements.txt
git commit -m "feat: add OAuth credential manager for YouTube API"
```

---

### Task 3: Test OAuth flow (dry run)

**Objective:** Verify token.pickle is generated from client_secret.json

**Files:**
- Create: `backend/tests/test_auth.py`

**Step 1: Write test**

```python
import os
import tempfile
from unittest.mock import patch
import auth

def test_credentials_flow_with_valid_secret():
    """When client_secret.json exists, get_credentials() should not raise."""
    # This test only checks the file-exists path; real OAuth requires browser
    if os.path.exists(auth.CLIENT_SECRET):
        # Just verify module doesn't crash on import
        assert auth.SCOPES == ["https://www.googleapis.com/auth/youtube.readonly"]
    else:
        # Without client_secret, should raise FileNotFoundError
        try:
            auth.get_credentials()
            assert False, "Should have raised"
        except FileNotFoundError:
            pass
```

**Step 2: Run test**

```bash
cd backend && .venv/bin/python3 -m pytest tests/test_auth.py -v
```

Expected: PASS (skips OAuth if no secret, or passes structure check)

**Step 3: Commit**

```bash
git add backend/tests/test_auth.py
git commit -m "test: add OAuth auth module test"
```

---

### Task 4: Rewrite fetcher.py to use YouTube Data API v3

**Objective:** Replace yt-dlp with googleapiclient subscription fetch

**Files:**
- Modify: `backend/fetcher.py`

**Complete replacement code:**

```python
import asyncio
import logging
import aiosqlite
from googleapiclient.discovery import build
from googleapiclient.errors import HttpError
from auth import get_credentials
from db import DB_PATH

log = logging.getLogger(__name__)

async def fetch_subscriptions():
    """Fetch latest videos from subscribed channels using YouTube Data API v3."""
    try:
        creds = get_credentials()
    except FileNotFoundError as e:
        log.error(f"OAuth setup incomplete: {e}")
        return 0
    except Exception as e:
        log.error(f"OAuth credential error: {e}")
        return 0

    youtube = build("youtube", "v3", credentials=creds)

    # Step 1: Get subscribed channel IDs
    channel_ids = []
    try:
        request = youtube.subscriptions().list(
            part="snippet",
            mine=True,
            maxResults=50,
            order="alphabetical"
        )
        while request:
            response = request.execute()
            for item in response.get("items", []):
                cid = item["snippet"]["resourceId"]["channelId"]
                if cid not in channel_ids:
                    channel_ids.append(cid)
            request = youtube.subscriptions().list_next(request, response)
    except HttpError as e:
        log.error(f"YouTube API error fetching subscriptions: {e}")
        return 0

    log.info(f"Found {len(channel_ids)} subscribed channels")

    if not channel_ids:
        return 0

    # Step 2: For each channel, get latest videos
    new_count = 0
    async with aiosqlite.connect(DB_PATH) as db:
        for cid in channel_ids[:20]:  # Respect quota, limit to 20 channels
            try:
                req = youtube.search().list(
                    part="snippet",
                    channelId=cid,
                    maxResults=10,
                    order="date",
                    type="video"
                )
                resp = req.execute()
                for item in resp.get("items", []):
                    vid = item["id"]["videoId"]
                    snippet = item["snippet"]

                    cursor = await db.execute(
                        "SELECT 1 FROM videos WHERE video_id = ?", (vid,)
                    )
                    if await cursor.fetchone():
                        continue

                    await db.execute("""
                        INSERT INTO videos
                        (video_id, title, channel_name, channel_id, url, thumbnail_url, publish_date)
                        VALUES (?, ?, ?, ?, ?, ?, ?)
                    """, (
                        vid,
                        snippet.get("title", "Untitled"),
                        snippet.get("channelTitle", "Unknown"),
                        snippet.get("channelId", cid),
                        f"https://www.youtube.com/watch?v={vid}",
                        snippet.get("thumbnails", {}).get("high", {}).get("url", ""),
                        snippet.get("publishTime", "")[:10]
                    ))
                    new_count += 1
                    log.debug(f"New video: {snippet.get('title', 'Untitled')[:50]}")
            except HttpError as e:
                log.warning(f"Channel {cid} fetch error: {e}")
                continue

        await db.commit()

    log.info(f"Ingestion completed. Found {new_count} new videos.")
    return new_count


if __name__ == "__main__":
    import sys
    logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')

    async def main():
        from db import init_db
        await init_db()
        await fetch_subscriptions()

    asyncio.run(main())
```

**Step 1: Run existing tests to ensure nothing else broke**

```bash
cd backend && .venv/bin/python3 -m pytest tests/test_fetcher.py -v
```

Expected: Mock tests should still pass (fetcher import may need auth module available)

**Step 2: Update test_fetcher.py to mock googleapiclient**

Modify `backend/tests/test_fetcher.py`:

```python
# Replace yt-dlp mocks with googleapiclient mocks
```

**Step 3: Commit**

```bash
git add backend/fetcher.py backend/tests/test_fetcher.py
git commit -m "feat: replace yt-dlp with YouTube Data API v3 OAuth"
```

---

### Task 5: Update Docker config for new dependencies

**Objective:** Ensure Docker build includes google-auth and config mounts

**Files:**
- Modify: `Dockerfile`
- Modify: `docker-compose.yml`

**Dockerfile changes:**

```dockerfile
# In the Python stage, ensure google packages are installed
# requirements.txt already includes them from Task 1
```

**docker-compose.yml changes:**

```yaml
# Mount config directory for OAuth secrets
volumes:
  - ./config:/app/config
```

**Step 1: Verify Docker build**

```bash
docker compose build
```

Expected: Build succeeds

**Step 2: Commit**

```bash
git add Dockerfile docker-compose.yml
git commit -m "chore: update Docker config for OAuth and API v3"
```

---

### Task 6: Integration test with real API

**Objective:** Run a live fetch against real YouTube subscription data

**Files:**
- Create: `backend/tests/test_live_fetch.py`

**Step 1: Write integration test**

```python
"""Live integration test — requires valid OAuth token in config/"""
import pytest
import asyncio
from fetcher import fetch_subscriptions
from db import init_db

@pytest.mark.asyncio
@pytest.mark.integration
async def test_live_subscription_fetch():
    """Fetch real subscriptions. Skip if no OAuth token."""
    await init_db()
    count = await fetch_subscriptions()
    # With valid OAuth and subscriptions, should find videos
    # Even 0 is acceptable (no new videos since last fetch)
    assert count >= 0, "Fetch should not crash"
    print(f"Fetched {count} new videos")
```

**Step 2: Run (requires OAuth setup)**

```bash
cd backend && .venv/bin/python3 -m pytest tests/test_live_fetch.py -v -s
```

Expected: PASS or SKIP if no OAuth

**Step 3: Commit**

```bash
git add backend/tests/test_live_fetch.py
git commit -m "test: add live YouTube API integration test"
```

---

### Task 7: Final verification — full pipeline

**Objective:** Run full test suite and verify everything works end-to-end

**Step 1: Run all tests**

```bash
cd ~/repos/t-youtube
./run_tests.sh
```

**Step 2: Verify frontend still builds**

```bash
cd frontend && npm run build 2>/dev/null || echo "Frontend build skipped"
```

**Step 3: Commit final state**

```bash
git add -A && git commit -m "feat: complete YouTube Data API v3 migration"
```

---

## Verification Checklist

- [ ] `config/client_secret.json` exists
- [ ] `config/token.pickle` generated after first OAuth flow
- [ ] `fetch_subscriptions()` returns video count > 0
- [ ] Videos appear in DB with correct metadata
- [ ] Summarizer still works with newly fetched videos
- [ ] All mock tests pass
- [ ] Docker build succeeds
- [ ] Frontend displays new videos

## Risks

1. **API quota**: YouTube Data API v3 has 10,000 units/day quota. Each subscription list call costs 1 unit, each search costs 100 units. With 20 channels × 10 results, that's ~2,000 units per sync.
2. **OAuth browser flow**: First run requires browser for Google login. On headless NAS, run locally first and copy `token.pickle`.
3. **Token expiry**: refresh_token handles this automatically via `google-auth`.