feat: use yt-dlp as primary transcript fetcher, fallback to youtube-transcript-api

- Add _ytdlp_transcript() — fetch transcripts via yt-dlp Python API
  (supports json3/srv3/vtt/srt subtitle formats, cookies.txt auth)
- Add _parse_subtitles() — parse subtitle formats into plain text
- Keep _yttranscript_api_transcript() as fallback
- download_transcript() tries yt-dlp first, then youtube-transcript-api
- Add yt-dlp to requirements.txt
- Update tests to mock both paths (all 23 pass)
- Document Google Data API v3 caption limitations in architecture.md
  (captions.list works, captions.download=403 for non-owned videos)
This commit is contained in:
Gan, Jimmy
2026-07-12 01:36:53 +08:00
parent fd16217076
commit f1ada7d277
4 changed files with 147 additions and 11 deletions
+32 -1
View File
@@ -165,7 +165,38 @@ YouTube transcripts inaccessible from all our locations:
- **MacBook (China)**: GFW blocks YouTube API endpoints
- **YouTube Data API captions.download**: 403 Forbidden for non-owned videos
**Workaround**: Title + description-based summaries via Qwen. Full transcripts need a Japanese residential proxy ($3-5/month).
#### Google Data API v3 Transcript Support (Investigation 2025-07-11)
| Endpoint | Works? | Notes |
|----------|--------|-------|
| `captions.list` (part=snippet, videoId=...) | ✅ Yes | Lists available caption tracks with language, kind (standard/asr), status |
| `captions.download` (id=..., tfmt='srt') | ❌ **403 Forbidden** | Only works for videos **you own**. Returns error: *"permissions associated with the request are not sufficient to download the caption track"* |
| `videos.list` (part=snippet) | ✅ Yes | Returns video description (~200-300 chars typical), but NOT transcript/captions |
**Conclusion**: There is no Google official API to download captions of other creators' videos. The `captions.download` endpoint is strictly limited to video owners. Third-party tools (`youtube-transcript-api`, `yt-dlp`) reverse-engineer YouTube's internal innertube API to extract captions, which is why they get blocked.
#### Transcript Retrieval Strategy
The only reliable way to get transcripts is **fresh YouTube session cookies + yt-dlp**:
```
Cookies (exported from browser) → yt-dlp (write-subs) → HTML/JSON → transcript text
```
yt-dlp is preferred over `youtube-transcript-api` because:
1. More actively maintained and widely used
2. Better at evading bot detection (more complete browser fingerprinting)
3. Supports `--cookies-from-browser` for direct cookie extraction
4. Has built-in subtitle extraction (`--write-subs`)
**Current blockers** (verified 2025-07-11):
1. `config/cookies.txt` is **expired** (last exported 2024-06-24) — YouTube session cookies expire quickly
2. macOS encrypts Chrome cookies via Keychain → `browser_cookie3` and `yt-dlp --cookies-from-browser` fail in non-interactive sessions
3. `youtube-transcript-api` returns `RequestBlocked` from this environment
**Fix path**: Re-export cookies.txt from browser → `summarizer.py` uses yt-dlp with that cookies file.
**Workaround** (current, degraded): Title + description-based summaries via Qwen. Description provides ~200-300 chars of useful text.
### auth.jimmygan.com from LAN
Accessing `auth.jimmygan.com` from LAN shows Immich (photos.jimmygan.com) instead of Authelia. Caused by NAS port 443 routing — DSM nginx still owns port 443 and routes Immich for unmatched domains.