diff --git a/README.md b/README.md index ac684eb..8d2be29 100644 --- a/README.md +++ b/README.md @@ -80,6 +80,11 @@ Trade-off: ~3-4s model load time when switching models. ## Contents - `docs/` — Full optimization journey with benchmarks at each step + - `01-problem.md` — Why two models on one GPU is slow + - `02-initial-setup.md` — The naive two-server setup + - `03-optimization-journey.md` — Every optimization we tried + - `04-lessons-learned.md` — Hard-won insights and discoveries + - `06-router-mode.md` — Router mode setup and config - `configs/` — Launchd plist, INI preset, utility scripts - `benchmarks/` — Before/after speed comparison data diff --git a/docs/04-lessons-learned.md b/docs/04-lessons-learned.md new file mode 100644 index 0000000..eab186a --- /dev/null +++ b/docs/04-lessons-learned.md @@ -0,0 +1,160 @@ +# Lessons Learned + +The discoveries and hard-won insights from optimizing two LLMs on a single M5 Max GPU. + +## 1. Launchd Management — Don't Kill, Unload + +**The hard way:** Running `kill -9` on a launchd-managed process is futile. `KeepAlive=true` restarts it instantly with the old config. + +**The right way:** +```bash +# Before editing the plist +launchctl unload ~/Library/LaunchAgents/com.example.plist + +# After editing the plist +launchctl load ~/Library/LaunchAgents/com.example.plist +``` + +**Why it matters:** If you kill the process without unloading, launchd respawns it before you finish editing the plist. You end up fighting a zombie that keeps coming back. + +**Tell-tale sign:** You see "couldn't bind HTTP server socket" when starting a replacement — the old process auto-restarted on the same port. + +## 2. Dell Hub Power — 90W Is Not Enough + +Your MacBook Pro wants **140W**. The Dell U2723 monitor's USB-C port delivers 63-90W, depending on the connection state. + +**Symptoms of undervolt:** +- Battery slowly drains even while "plugged in" +- "Not charging" status despite AC power detected +- System runs fine but can't top off the battery under GPU load + +**The fix is simple:** Plug the 140W adapter directly into the Mac. Use the Dell hub only for display and peripherals. Two cables. + +**Measured power delivery:** +| Source | Wattage | Can charge? | +|--------|---------|-------------| +| Apple 140W direct | 140W | Yes — full speed | +| Dell U2723 USB-C | 63-90W | Unreliable — drains under GPU load | + +## 3. Batch Size — Smaller Is Faster for Single-User + +**Counter-intuitive finding:** `--batch-size 2048 --ubatch-size 512` (default-adjacent for throughput servers) **hurts** single-user generation speed. + +**Why:** During autoregressive generation, you produce 1 token at a time. Large batch sizes add overhead in GPU scheduling and memory management for zero benefit. + +**The fix:** +```bash +--batch-size 512 --ubatch-size 128 +``` + +**Impact:** +| Model | 2048/512 | 512/128 | Gain | +|-------|----------|---------|------| +| 35B MoE | ~23 t/s | ~25 t/s | ~10% | +| 27B dense | ~12 t/s | ~16 t/s | **+33%** | + +Dense models benefit more because they're bandwidth-bound — less dispatch overhead means more time reading weights. + +## 4. Thread Count — Per-Architecture Tuning + +**Myth:** More CPU threads = faster generation. +**Reality:** Optimal thread count depends on model architecture. + +**Testing on M5 Max (18 CPU cores):** +| Threads | 35B MoE | 27B dense | +|---------|---------|-----------| +| 8 | — | 45.5 t/s | +| 10 | 128.6 t/s | **48.0 t/s** ✅ | +| 14 | **132 t/s** ✅ | 43 t/s | +| Auto (18) | ~120 t/s | ~38 t/s | + +**Rule of thumb:** +- **MoE models** (low active params, compute-bound attention) → t=14 +- **Dense models** (high active params, bandwidth-bound) → t=10 +- Too many threads adds dispatch overhead without bandwidth benefit + +**Implementation:** Use `--models-preset` with INI file for per-model thread counts. + +## 5. KV Cache Quantization — Q4_0 Over Q8_0 + +**Lesson:** KV cache is a **caching layer**, not model weights. Its precision has negligible effect on output quality. + +```bash +--cache-type-k q4_0 --cache-type-v q4_0 +``` + +**Why Q4_0 > Q8_0:** +- Halves memory bandwidth per token (less data to move = faster) +- Negligible quality loss in cached attention states +- With router mode and one model loaded, RAM headroom is ample anyway + +**What Q8_0 would cost:** 2x memory bandwidth for cached K/V reads during generation. No measurable quality benefit. + +## 6. MTP Speculative Decoding — Free Speed + +Qwen models have built-in Multi-Token Prediction heads. Enable them: + +```bash +--spec-type draft-mtp --spec-draft-n-max 3 +``` + +**Measured performance:** + +| Model | Acceptance | Mean Length | Effective Speedup | +|-------|-----------|-------------|-------------------| +| 35B MoE | 74% | 2.84 | ~2.1x | +| 27B dense | 82% | 3.62 | ~2.6x | + +Each forward pass produces 3 draft tokens. The acceptance rate determines how many are kept. At 80%+ acceptance, you get ~2.5 tokens per forward pass instead of 1. + +**Why MTP is better than draft models:** No separate draft model to load. Qwen's MTP heads are part of the model weights — negligible extra compute. + +## 7. Flash Attention — Critical for Long Contexts + +Without `--flash-attn on`, generation speed degrades as the KV cache grows (O(n²) attention). + +**Test results (35B MoE):** +| Context Size | No Flash Attn | With Flash Attn | +|-------------|---------------|-----------------| +| 100 tokens | ~22 t/s | ~22 t/s | +| 10K tokens | ~12 t/s | ~20 t/s | +| 30K tokens | ~9 t/s | ~18 t/s | + +Flash attention's benefit scales with context length. For agent sessions that accumulate 20K+ tokens, it's essential. + +## 8. Router Mode vs Two Servers — The Real Cost + +**Two separate servers:** Both models permanently loaded. Even when idle, the second model consumes GPU scheduler time and memory bandwidth overhead. + +**Router mode (--models-max 1):** Active model gets full bandwidth. Idle model is completely evicted from GPU memory. + +**The real cost of keeping both loaded:** +| Model | Solo | With Other Loaded | Penalty | +|-------|------|-------------------|---------| +| 35B MoE | 94 t/s | 23 t/s | **-75%** | +| 27B dense | 12 t/s | 12 t/s | ~0% (already saturated) | + +The 27B doesn't care because it's reading 27 GB/token — it saturates bandwidth regardless. The 35B MoE (3 GB/token) has massive headroom that disappears the moment it competes. + +**Router mode model switch delay:** ~3-4 seconds. This is model weights loading from SSD to GPU memory — not context processing. `--slot-save-path` would help but doesn't work with router mode (child processes killed on eviction). + +## 9. Context Window Sizing — 128K Is the Sweet Spot + +**Why not 262K:** The KV cache for 262K on the 35B MoE consumed ~15 GB of RSS — a third of the model's memory footprint. For agent sessions, typical usage tops out at ~65K tokens. + +**Why not 32K:** Agent sessions with tool calls, file reads, and long reasoning chains can exceed 32K. 65K was measured in active use. + +**Verdict:** 128K gives 2x headroom over observed usage, saves ~8 GB RAM vs 262K. + +## 10. Swap Monitoring — Know When You're at the Wall + +A simple watchdog script catches memory creep before it slows everything: + +```bash +# ~/.hermes/scripts/swap_watch.sh +# Quiet unless swap > 2GB +``` + +Set up as a cron job: every 30 minutes, no_agent mode, silent when healthy. + +**Why:** You can have 32 GB free and still have 2 GB of stale swap that macOS hasn't purged. The monitor tells you if swap is growing (real pressure) or just leftover (harmless).