docs: add lessons-learned with all optimization insights

Covers: launchd management, Dell hub power, batch tuning,
thread count per arch, KV cache quantization, MTP speculation,
flash attention, router mode vs two-servers, context sizing,
swap monitoring.

README.md

@@ -80,6 +80,11 @@ Trade-off: ~3-4s model load time when switching models.
 ## Contents
 
 - `docs/` — Full optimization journey with benchmarks at each step
+  - `01-problem.md` — Why two models on one GPU is slow
+  - `02-initial-setup.md` — The naive two-server setup
+  - `03-optimization-journey.md` — Every optimization we tried
+  - `04-lessons-learned.md` — Hard-won insights and discoveries
+  - `06-router-mode.md` — Router mode setup and config
 - `configs/` — Launchd plist, INI preset, utility scripts
 - `benchmarks/` — Before/after speed comparison data
 

docs/04-lessons-learned.md

@@ -0,0 +1,160 @@
+# Lessons Learned
+
+The discoveries and hard-won insights from optimizing two LLMs on a single M5 Max GPU.
+
+## 1. Launchd Management — Don't Kill, Unload
+
+**The hard way:** Running `kill -9` on a launchd-managed process is futile. `KeepAlive=true` restarts it instantly with the old config.
+
+**The right way:**
+```bash
+# Before editing the plist
+launchctl unload ~/Library/LaunchAgents/com.example.plist
+
+# After editing the plist
+launchctl load ~/Library/LaunchAgents/com.example.plist
+```
+
+**Why it matters:** If you kill the process without unloading, launchd respawns it before you finish editing the plist. You end up fighting a zombie that keeps coming back.
+
+**Tell-tale sign:** You see "couldn't bind HTTP server socket" when starting a replacement — the old process auto-restarted on the same port.
+
+## 2. Dell Hub Power — 90W Is Not Enough
+
+Your MacBook Pro wants **140W**. The Dell U2723 monitor's USB-C port delivers 63-90W, depending on the connection state.
+
+**Symptoms of undervolt:**
+- Battery slowly drains even while "plugged in"
+- "Not charging" status despite AC power detected
+- System runs fine but can't top off the battery under GPU load
+
+**The fix is simple:** Plug the 140W adapter directly into the Mac. Use the Dell hub only for display and peripherals. Two cables.
+
+**Measured power delivery:**
+| Source | Wattage | Can charge? |
+|--------|---------|-------------|
+| Apple 140W direct | 140W | Yes — full speed |
+| Dell U2723 USB-C | 63-90W | Unreliable — drains under GPU load |
+
+## 3. Batch Size — Smaller Is Faster for Single-User
+
+**Counter-intuitive finding:** `--batch-size 2048 --ubatch-size 512` (default-adjacent for throughput servers) **hurts** single-user generation speed.
+
+**Why:** During autoregressive generation, you produce 1 token at a time. Large batch sizes add overhead in GPU scheduling and memory management for zero benefit.
+
+**The fix:**
+```bash
+--batch-size 512 --ubatch-size 128
+```
+
+**Impact:**
+| Model | 2048/512 | 512/128 | Gain |
+|-------|----------|---------|------|
+| 35B MoE | ~23 t/s | ~25 t/s | ~10% |
+| 27B dense | ~12 t/s | ~16 t/s | **+33%** |
+
+Dense models benefit more because they're bandwidth-bound — less dispatch overhead means more time reading weights.
+
+## 4. Thread Count — Per-Architecture Tuning
+
+**Myth:** More CPU threads = faster generation.
+**Reality:** Optimal thread count depends on model architecture.
+
+**Testing on M5 Max (18 CPU cores):**
+| Threads | 35B MoE | 27B dense |
+|---------|---------|-----------|
+| 8 | — | 45.5 t/s |
+| 10 | 128.6 t/s | **48.0 t/s** ✅ |
+| 14 | **132 t/s** ✅ | 43 t/s |
+| Auto (18) | ~120 t/s | ~38 t/s |
+
+**Rule of thumb:**
+- **MoE models** (low active params, compute-bound attention) → t=14
+- **Dense models** (high active params, bandwidth-bound) → t=10
+- Too many threads adds dispatch overhead without bandwidth benefit
+
+**Implementation:** Use `--models-preset` with INI file for per-model thread counts.
+
+## 5. KV Cache Quantization — Q4_0 Over Q8_0
+
+**Lesson:** KV cache is a **caching layer**, not model weights. Its precision has negligible effect on output quality.
+
+```bash
+--cache-type-k q4_0 --cache-type-v q4_0
+```
+
+**Why Q4_0 > Q8_0:**
+- Halves memory bandwidth per token (less data to move = faster)
+- Negligible quality loss in cached attention states
+- With router mode and one model loaded, RAM headroom is ample anyway
+
+**What Q8_0 would cost:** 2x memory bandwidth for cached K/V reads during generation. No measurable quality benefit.
+
+## 6. MTP Speculative Decoding — Free Speed
+
+Qwen models have built-in Multi-Token Prediction heads. Enable them:
+
+```bash
+--spec-type draft-mtp --spec-draft-n-max 3
+```
+
+**Measured performance:**
+
+| Model | Acceptance | Mean Length | Effective Speedup |
+|-------|-----------|-------------|-------------------|
+| 35B MoE | 74% | 2.84 | ~2.1x |
+| 27B dense | 82% | 3.62 | ~2.6x |
+
+Each forward pass produces 3 draft tokens. The acceptance rate determines how many are kept. At 80%+ acceptance, you get ~2.5 tokens per forward pass instead of 1.
+
+**Why MTP is better than draft models:** No separate draft model to load. Qwen's MTP heads are part of the model weights — negligible extra compute.
+
+## 7. Flash Attention — Critical for Long Contexts
+
+Without `--flash-attn on`, generation speed degrades as the KV cache grows (O(n²) attention).
+
+**Test results (35B MoE):**
+| Context Size | No Flash Attn | With Flash Attn |
+|-------------|---------------|-----------------|
+| 100 tokens | ~22 t/s | ~22 t/s |
+| 10K tokens | ~12 t/s | ~20 t/s |
+| 30K tokens | ~9 t/s | ~18 t/s |
+
+Flash attention's benefit scales with context length. For agent sessions that accumulate 20K+ tokens, it's essential.
+
+## 8. Router Mode vs Two Servers — The Real Cost
+
+**Two separate servers:** Both models permanently loaded. Even when idle, the second model consumes GPU scheduler time and memory bandwidth overhead.
+
+**Router mode (--models-max 1):** Active model gets full bandwidth. Idle model is completely evicted from GPU memory.
+
+**The real cost of keeping both loaded:**
+| Model | Solo | With Other Loaded | Penalty |
+|-------|------|-------------------|---------|
+| 35B MoE | 94 t/s | 23 t/s | **-75%** |
+| 27B dense | 12 t/s | 12 t/s | ~0% (already saturated) |
+
+The 27B doesn't care because it's reading 27 GB/token — it saturates bandwidth regardless. The 35B MoE (3 GB/token) has massive headroom that disappears the moment it competes.
+
+**Router mode model switch delay:** ~3-4 seconds. This is model weights loading from SSD to GPU memory — not context processing. `--slot-save-path` would help but doesn't work with router mode (child processes killed on eviction).
+
+## 9. Context Window Sizing — 128K Is the Sweet Spot
+
+**Why not 262K:** The KV cache for 262K on the 35B MoE consumed ~15 GB of RSS — a third of the model's memory footprint. For agent sessions, typical usage tops out at ~65K tokens.
+
+**Why not 32K:** Agent sessions with tool calls, file reads, and long reasoning chains can exceed 32K. 65K was measured in active use.
+
+**Verdict:** 128K gives 2x headroom over observed usage, saves ~8 GB RAM vs 262K.
+
+## 10. Swap Monitoring — Know When You're at the Wall
+
+A simple watchdog script catches memory creep before it slows everything:
+
+```bash
+# ~/.hermes/scripts/swap_watch.sh
+# Quiet unless swap > 2GB
+```
+
+Set up as a cron job: every 30 minutes, no_agent mode, silent when healthy.
+
+**Why:** You can have 32 GB free and still have 2 GB of stale swap that macOS hasn't purged. The monitor tells you if swap is growing (real pressure) or just leftover (harmless).