# Lessons Learned

The discoveries and hard-won insights from optimizing two LLMs on a single M5 Max GPU.

## 1. Launchd Management — Don't Kill, Unload

**The hard way:** Running `kill -9` on a launchd-managed process is futile. `KeepAlive=true` restarts it instantly with the old config.

**The right way:**
```bash
# Before editing the plist
launchctl unload ~/Library/LaunchAgents/com.example.plist

# After editing the plist
launchctl load ~/Library/LaunchAgents/com.example.plist
```

**Why it matters:** If you kill the process without unloading, launchd respawns it before you finish editing the plist. You end up fighting a zombie that keeps coming back.

**Tell-tale sign:** You see "couldn't bind HTTP server socket" when starting a replacement — the old process auto-restarted on the same port.

## 2. Dell Hub Power — 90W Is Not Enough

Your MacBook Pro wants **140W**. The Dell U2723 monitor's USB-C port delivers 63-90W, depending on the connection state.

**Symptoms of undervolt:**
- Battery slowly drains even while "plugged in"
- "Not charging" status despite AC power detected
- System runs fine but can't top off the battery under GPU load

**The fix is simple:** Plug the 140W adapter directly into the Mac. Use the Dell hub only for display and peripherals. Two cables.

**Measured power delivery:**
| Source | Wattage | Can charge? |
|--------|---------|-------------|
| Apple 140W direct | 140W | Yes — full speed |
| Dell U2723 USB-C | 63-90W | Unreliable — drains under GPU load |

## 3. Batch Size — Smaller Is Faster for Single-User

**Counter-intuitive finding:** `--batch-size 2048 --ubatch-size 512` (default-adjacent for throughput servers) **hurts** single-user generation speed.

**Why:** During autoregressive generation, you produce 1 token at a time. Large batch sizes add overhead in GPU scheduling and memory management for zero benefit.

**The fix:**
```bash
--batch-size 512 --ubatch-size 128
```

**Impact:**
| Model | 2048/512 | 512/128 | Gain |
|-------|----------|---------|------|
| 35B MoE | ~23 t/s | ~25 t/s | ~10% |
| 27B dense | ~12 t/s | ~16 t/s | **+33%** |

Dense models benefit more because they're bandwidth-bound — less dispatch overhead means more time reading weights.

## 4. Thread Count — Per-Architecture Tuning

**Myth:** More CPU threads = faster generation.
**Reality:** Optimal thread count depends on model architecture.

**Testing on M5 Max (18 CPU cores):**
| Threads | 35B MoE | 27B dense |
|---------|---------|-----------|
| 8 | — | 45.5 t/s |
| 10 | 128.6 t/s | **48.0 t/s** ✅ |
| 14 | **132 t/s** ✅ | 43 t/s |
| Auto (18) | ~120 t/s | ~38 t/s |

**Rule of thumb:**
- **MoE models** (low active params, compute-bound attention) → t=14
- **Dense models** (high active params, bandwidth-bound) → t=10
- Too many threads adds dispatch overhead without bandwidth benefit

**Implementation:** Use `--models-preset` with INI file for per-model thread counts.

## 5. KV Cache Quantization — Q4_0 Over Q8_0

**Lesson:** KV cache is a **caching layer**, not model weights. Its precision has negligible effect on output quality.

```bash
--cache-type-k q4_0 --cache-type-v q4_0
```

**Why Q4_0 > Q8_0:**
- Halves memory bandwidth per token (less data to move = faster)
- Negligible quality loss in cached attention states
- With router mode and one model loaded, RAM headroom is ample anyway

**What Q8_0 would cost:** 2x memory bandwidth for cached K/V reads during generation. No measurable quality benefit.

## 6. MTP Speculative Decoding — Free Speed

Qwen models have built-in Multi-Token Prediction heads. Enable them:

```bash
--spec-type draft-mtp --spec-draft-n-max 3
```

**Measured performance:**

| Model | Acceptance | Mean Length | Effective Speedup |
|-------|-----------|-------------|-------------------|
| 35B MoE | 74% | 2.84 | ~2.1x |
| 27B dense | 82% | 3.62 | ~2.6x |

Each forward pass produces 3 draft tokens. The acceptance rate determines how many are kept. At 80%+ acceptance, you get ~2.5 tokens per forward pass instead of 1.

**Why MTP is better than draft models:** No separate draft model to load. Qwen's MTP heads are part of the model weights — negligible extra compute.

## 7. Flash Attention — Critical for Long Contexts

Without `--flash-attn on`, generation speed degrades as the KV cache grows (O(n²) attention).

**Test results (35B MoE):**
| Context Size | No Flash Attn | With Flash Attn |
|-------------|---------------|-----------------|
| 100 tokens | ~22 t/s | ~22 t/s |
| 10K tokens | ~12 t/s | ~20 t/s |
| 30K tokens | ~9 t/s | ~18 t/s |

Flash attention's benefit scales with context length. For agent sessions that accumulate 20K+ tokens, it's essential.

## 8. Router Mode vs Two Servers — The Real Cost

**Two separate servers:** Both models permanently loaded. Even when idle, the second model consumes GPU scheduler time and memory bandwidth overhead.

**Router mode (--models-max 1):** Active model gets full bandwidth. Idle model is completely evicted from GPU memory.

**The real cost of keeping both loaded:**
| Model | Solo | With Other Loaded | Penalty |
|-------|------|-------------------|---------|
| 35B MoE | 94 t/s | 23 t/s | **-75%** |
| 27B dense | 12 t/s | 12 t/s | ~0% (already saturated) |

The 27B doesn't care because it's reading 27 GB/token — it saturates bandwidth regardless. The 35B MoE (3 GB/token) has massive headroom that disappears the moment it competes.

**Router mode model switch delay:** ~3-4 seconds. This is model weights loading from SSD to GPU memory — not context processing. `--slot-save-path` would help but doesn't work with router mode (child processes killed on eviction).

## 9. Context Window Sizing — 128K Is the Sweet Spot

**Why not 262K:** The KV cache for 262K on the 35B MoE consumed ~15 GB of RSS — a third of the model's memory footprint. For agent sessions, typical usage tops out at ~65K tokens.

**Why not 32K:** Agent sessions with tool calls, file reads, and long reasoning chains can exceed 32K. 65K was measured in active use.

**Verdict:** 128K gives 2x headroom over observed usage, saves ~8 GB RAM vs 262K.

## 10. Swap Monitoring — Know When You're at the Wall

A simple watchdog script catches memory creep before it slows everything:

```bash
# ~/.hermes/scripts/swap_watch.sh
# Quiet unless swap > 2GB
```

Set up as a cron job: every 30 minutes, no_agent mode, silent when healthy.

**Why:** You can have 32 GB free and still have 2 GB of stale swap that macOS hasn't purged. The monitor tells you if swap is growing (real pressure) or just leftover (harmless).