Initial: llama.cpp router mode optimization guide for Apple Silicon

Full documentation of optimizing two LLMs on a single M5 Max GPU:
- KV cache quantization (Q4_0)
- Flash attention and batch tuning
- Router mode with --models-max 1
- Per-model thread optimization via INI presets
- Before/after benchmarks (12→48 t/s on 27B, 23→132 t/s on 35B)

README.md

@@ -0,0 +1,88 @@
+# llama.cpp Router Mode on Apple Silicon
+
+**Single GPU, two models, zero contention — from 12 t/s to 132 t/s**
+
+A practical guide to running multiple LLMs on one Apple Silicon Mac using `llama-server` router mode, achieving full GPU bandwidth for each model with automatic LRU eviction.
+
+## The Problem
+
+Running two `llama-server` processes on the same GPU causes severe bandwidth contention:
+
+```
+Two separate servers (before):
+  35B MoE:  23 t/s  ← -75% from potential
+  27B dense: 16 t/s  ← -63% from potential
+```
+
+Both models stay permanently loaded, permanently competing for GPU memory bandwidth.
+
+## The Solution
+
+A single `llama-server` with `--models-max 1` evicts the idle model from GPU memory, giving full bandwidth to whichever model is active:
+
+```
+Router mode (after):
+  35B MoE:  132 t/s  ← 5.7x faster
+  27B dense:  48 t/s  ← 3.0x faster
+```
+
+## Quick Start
+
+```bash
+# 1. Create model symlinks
+mkdir -p ~/.hermes/models-router
+ln -sf /path/to/model1.gguf ~/.hermes/models-router/
+ln -sf /path/to/model2.gguf ~/.hermes/models-router/
+
+# 2. Create per-model INI preset
+cat > ~/.hermes/llama-models.ini << 'EOF'
+[*]
+flash-attn = on
+cache-type-k = q4_0
+cache-type-v = q4_0
+batch-size = 512
+ubatch-size = 128
+spec-type = draft-mtp
+spec-draft-n-max = 3
+
+[model.Model1]
+threads = 14
+
+[model.Model2]
+threads = 10
+EOF
+
+# 3. Launch router
+llama-server \
+  --models-dir ~/.hermes/models-router \
+  --models-max 1 \
+  --models-preset ~/.hermes/llama-models.ini \
+  --host 0.0.0.0 --port 8085 \
+  -ngl 99 -c 131072 --mlock \
+  --metrics
+```
+
+## Speed Results (M5 Max 40-core, 128GB)
+
+| Model | Two Servers | Router Mode | Gain |
+|-------|-------------|-------------|------|
+| Qwen3.6-35B-A3B-Q8_0 (MoE) | 23 t/s | 132 t/s | **5.7x** |
+| Qwen3.6-27B-Q8_0 (dense) | 16 t/s | 48 t/s | **3.0x** |
+
+Trade-off: ~3-4s model load time when switching models.
+
+## Hardware
+
+- **Chip:** Apple M5 Max (40 GPU cores, 614 GB/s bandwidth)
+- **RAM:** 128 GB Unified Memory
+- **Software:** llama.cpp b9910+, launchd, Hermes Agent
+
+## Contents
+
+- `docs/` — Full optimization journey with benchmarks at each step
+- `configs/` — Launchd plist, INI preset, utility scripts
+- `benchmarks/` — Before/after speed comparison data
+
+## License
+
+MIT — use it, share it, productize it.