Initial: llama.cpp router mode optimization guide for Apple Silicon

Full documentation of optimizing two LLMs on a single M5 Max GPU:
- KV cache quantization (Q4_0)
- Flash attention and batch tuning
- Router mode with --models-max 1
- Per-model thread optimization via INI presets
- Before/after benchmarks (12→48 t/s on 27B, 23→132 t/s on 35B)

benchmarks/results.md

@@ -0,0 +1,87 @@
+# Benchmark Results
+
+## Hardware
+
+| Component | Spec |
+|-----------|------|
+| Chip | Apple M5 Max |
+| GPU Cores | 40 |
+| RAM | 128 GB Unified Memory |
+| Memory Bandwidth | 614 GB/s |
+| OS | macOS 26.3.2 |
+| llama.cpp | b9910 (f5525f7e7) |
+
+## Models
+
+| Model | Architecture | Size | Quant |
+|-------|-------------|------|-------|
+| Qwen3.6-35B-A3B | MoE (3B active) | ~35 GB | Q8_0 |
+| Qwen3.6-27B | Dense | ~27 GB | Q8_0 |
+
+## Config
+
+```
+-c 131072, --cache-type-k q4_0, --cache-type-v q4_0
+--flash-attn on, --spec-type draft-mtp, --spec-draft-n-max 3
+--batch-size 512, --ubatch-size 128
+```
+
+## Results
+
+### Two Separate Servers (Before)
+
+**Both loaded, both generating:**
+
+| Model | Gen Speed | Notes |
+|-------|-----------|-------|
+| 35B MoE | 23 t/s | -75% from solo potential |
+| 27B dense | 12 t/s | Already bandwidth-saturated |
+
+**Solo (only one running):**
+
+| Model | Gen Speed | Bandwidth Efficiency |
+|-------|-----------|---------------------|
+| 35B MoE | 94 t/s | 3.2 GB/step × 94 = 49% of 614 GB/s |
+| 27B dense | 12 t/s | 30.4 GB/step × 12 = 59% of 614 GB/s |
+
+### Router Mode (After)
+
+**--models-max 1, auto-swap on request:**
+
+| Model | Gen Speed (cold) | Gen Speed (warm) | Gain vs Before |
+|-------|-----------------|------------------|----------------|
+| 35B MoE | 132 t/s | 148 t/s | **5.7x** |
+| 27B dense | 38 t/s | 48 t/s | **3.0x** |
+
+**Per-model thread tuning:**
+
+| Model | Optimal Threads | Speed |
+|-------|----------------|-------|
+| 35B MoE (attention-compute bound) | 14 | 148 t/s |
+| 27B dense (bandwidth bound) | 10 | 48 t/s |
+
+### Model Switch Latency
+
+| Transition | Time |
+|-----------|------|
+| 35B → 27B (cold start) | ~4s |
+| 27B → 35B (reload) | ~3s |
+
+The 3-4s delay is model weights loading from SSD to GPU memory via Metal. Smaller batch sizes and Q4_0 KV cache help minimize this.
+
+## MTP Speculative Decoding
+
+| Model | Acceptance Rate | Mean Draft Length | Effective Tokens/Step |
+|-------|----------------|-------------------|----------------------|
+| 35B MoE | 74% | 2.84 | 2.1 |
+| 27B dense | 82% | 3.62 | 2.6 |
+
+MTP (Multi-Token Prediction) uses the model's built-in speculative heads. Each forward pass produces 3 draft tokens; acceptance rate determines how many are kept. With ~2.5 tokens per forward pass, effective generation speed is ~2.5x the raw decode speed.
+
+## Methodology
+
+- Tested via llama.cpp `/v1/completions` API
+- Fixed prompt: 7-11 tokens
+- Generation: 200 tokens, temperature 0
+- Timings from response body `timings.predicted_per_second`
+- Warm = model already loaded in GPU, cold = first request after model load