Initial: llama.cpp router mode optimization guide for Apple Silicon

Full documentation of optimizing two LLMs on a single M5 Max GPU:
- KV cache quantization (Q4_0)
- Flash attention and batch tuning
- Router mode with --models-max 1
- Per-model thread optimization via INI presets
- Before/after benchmarks (12→48 t/s on 27B, 23→132 t/s on 35B)

docs/03-optimization-journey.md

@@ -0,0 +1,99 @@
+# The Optimization Journey
+
+## Overview
+
+Starting from the baseline (two servers, ~20 t/s on 35B, ~12 t/s on 27B), each optimization improved speed by addressing specific bottlenecks.
+
+## Optimization 1: KV Cache Quantization
+
+**Problem:** Full-precision (f16) KV cache consumes massive RAM. For 128K context on the 35B model, the KV cache alone uses ~15 GB.
+
+**Fix:** Switch to Q4_0 KV cache.
+
+```bash
+--cache-type-k q4_0 --cache-type-v q4_0
+```
+
+**Impact:** -50% KV cache memory. Negligible quality loss — KV cache is a caching layer, not model weights. The precision of cached attention states has minimal effect on output quality.
+
+## Optimization 2: Context Window Tuning
+
+**Problem:** The 35B used 262K context (way too much for chat). The 27B used 131K. Combined KV caches consumed ~32 GB.
+
+**Fix:** Both models at 128K. Sufficient for agent sessions (typical usage tops out at ~65K tokens).
+
+```bash
+-c 131072
+```
+
+**Impact:** Saved ~16 GB RAM. Freed headroom for the rest of the system.
+
+## Optimization 3: Flash Attention
+
+**Problem:** Without flash attention, generation speed degrades as KV cache grows (O(n²) attention cost).
+
+**Fix:** Enable flash attention.
+
+```bash
+--flash-attn on
+```
+
+**Impact:** Maintains generation speed even with large contexts. With FA, generation stays at ~20 t/s regardless of context size. Without it, speed drops to ~10 t/s when context reaches 20K+ tokens.
+
+## Optimization 4: Batch Size Tuning
+
+**Problem:** Default batch sizes (2048/512) are optimized for server throughput (many concurrent requests), not single-user generation latency.
+
+**Fix:** Smaller batch sizes for single-user use.
+
+```bash
+--batch-size 512 --ubatch-size 128
+```
+
+**Impact:** +25-50% generation speed for dense models. The 27B jumped from 12 to 16 t/s. The 35B MoE was less affected (3B active params already efficient).
+
+**Why:** During generation (1 token at a time), the batch size affects how the GPU schedules compute. Smaller batches reduce latency per decode step.
+
+## Optimization 5: Thread Count Tuning
+
+**Problem:** Auto-detected thread count uses all CPU cores (18), causing dispatch overhead without benefit.
+
+**Fix:** Explicit thread count optimized per architecture.
+
+```bash
+-t 14    # General fallback
+```
+
+**Impact:** MoE models benefit from more threads (attention compute-heavy). Dense models benefit from fewer threads (bandwidth-bound, less dispatch overhead).
+
+| Model | Optimal Threads | Speed |
+|-------|----------------|-------|
+| 35B MoE (3B active) | 14 | 132 t/s |
+| 27B dense (27B active) | 10 | 48 t/s |
+
+## Optimization 6: Speculative Decoding (MTP)
+
+**Fix:** Increase speculation from 1 to 3.
+
+```bash
+--spec-draft-n-max 3
+```
+
+**Impact:** With 82-93% MTP acceptance rate, each forward pass produces ~2.5-3 tokens instead of 1. This effectively doubles generation speed for free.
+
+**Key insight:** Qwen models have built-in Multi-Token Prediction heads. The `--spec-type draft-mtp` flag uses these native heads rather than a separate draft model, adding negligible overhead.
+
+## The Big One: Router Mode
+
+The above optimizations improved speed, but the fundamental problem remained: **two processes competing for GPU bandwidth.**
+
+**Fix:** Replace two `llama-server` processes with one router-mode server.
+
+```bash
+llama-server \
+  --models-dir ~/.hermes/models-router \
+  --models-max 1 \
+  --models-preset ~/.hermes/llama-models.ini
+```
+
+**Impact:** +5.7x on 35B, +3.0x on 27B. See `06-router-mode.md` for full details.