feat: use yt-dlp as primary transcript fetcher, fallback to youtube-transcript-api
- Add _ytdlp_transcript() — fetch transcripts via yt-dlp Python API (supports json3/srv3/vtt/srt subtitle formats, cookies.txt auth) - Add _parse_subtitles() — parse subtitle formats into plain text - Keep _yttranscript_api_transcript() as fallback - download_transcript() tries yt-dlp first, then youtube-transcript-api - Add yt-dlp to requirements.txt - Update tests to mock both paths (all 23 pass) - Document Google Data API v3 caption limitations in architecture.md (captions.list works, captions.download=403 for non-owned videos)
This commit is contained in:
+32
-1
@@ -165,7 +165,38 @@ YouTube transcripts inaccessible from all our locations:
|
||||
- **MacBook (China)**: GFW blocks YouTube API endpoints
|
||||
- **YouTube Data API captions.download**: 403 Forbidden for non-owned videos
|
||||
|
||||
**Workaround**: Title + description-based summaries via Qwen. Full transcripts need a Japanese residential proxy ($3-5/month).
|
||||
#### Google Data API v3 Transcript Support (Investigation 2025-07-11)
|
||||
|
||||
| Endpoint | Works? | Notes |
|
||||
|----------|--------|-------|
|
||||
| `captions.list` (part=snippet, videoId=...) | ✅ Yes | Lists available caption tracks with language, kind (standard/asr), status |
|
||||
| `captions.download` (id=..., tfmt='srt') | ❌ **403 Forbidden** | Only works for videos **you own**. Returns error: *"permissions associated with the request are not sufficient to download the caption track"* |
|
||||
| `videos.list` (part=snippet) | ✅ Yes | Returns video description (~200-300 chars typical), but NOT transcript/captions |
|
||||
|
||||
**Conclusion**: There is no Google official API to download captions of other creators' videos. The `captions.download` endpoint is strictly limited to video owners. Third-party tools (`youtube-transcript-api`, `yt-dlp`) reverse-engineer YouTube's internal innertube API to extract captions, which is why they get blocked.
|
||||
|
||||
#### Transcript Retrieval Strategy
|
||||
|
||||
The only reliable way to get transcripts is **fresh YouTube session cookies + yt-dlp**:
|
||||
|
||||
```
|
||||
Cookies (exported from browser) → yt-dlp (write-subs) → HTML/JSON → transcript text
|
||||
```
|
||||
|
||||
yt-dlp is preferred over `youtube-transcript-api` because:
|
||||
1. More actively maintained and widely used
|
||||
2. Better at evading bot detection (more complete browser fingerprinting)
|
||||
3. Supports `--cookies-from-browser` for direct cookie extraction
|
||||
4. Has built-in subtitle extraction (`--write-subs`)
|
||||
|
||||
**Current blockers** (verified 2025-07-11):
|
||||
1. `config/cookies.txt` is **expired** (last exported 2024-06-24) — YouTube session cookies expire quickly
|
||||
2. macOS encrypts Chrome cookies via Keychain → `browser_cookie3` and `yt-dlp --cookies-from-browser` fail in non-interactive sessions
|
||||
3. `youtube-transcript-api` returns `RequestBlocked` from this environment
|
||||
|
||||
**Fix path**: Re-export cookies.txt from browser → `summarizer.py` uses yt-dlp with that cookies file.
|
||||
|
||||
**Workaround** (current, degraded): Title + description-based summaries via Qwen. Description provides ~200-300 chars of useful text.
|
||||
|
||||
### auth.jimmygan.com from LAN
|
||||
Accessing `auth.jimmygan.com` from LAN shows Immich (photos.jimmygan.com) instead of Authelia. Caused by NAS port 443 routing — DSM nginx still owns port 443 and routes Immich for unmatched domains.
|
||||
|
||||
Reference in New Issue
Block a user