Initial: llama.cpp router mode optimization guide for Apple Silicon

Full documentation of optimizing two LLMs on a single M5 Max GPU:
- KV cache quantization (Q4_0)
- Flash attention and batch tuning
- Router mode with --models-max 1
- Per-model thread optimization via INI presets
- Before/after benchmarks (12→48 t/s on 27B, 23→132 t/s on 35B)
This commit is contained in:
Gan, Jimmy
2026-07-10 02:27:36 +08:00
commit 3994e29cb0
8 changed files with 630 additions and 0 deletions
+14
View File
@@ -0,0 +1,14 @@
[*]
flash-attn = on
cache-type-k = q4_0
cache-type-v = q4_0
batch-size = 512
ubatch-size = 128
spec-type = draft-mtp
spec-draft-n-max = 3
[model.Qwen3.6-35B-A3B-Q8_0]
threads = 14
[model.Qwen3.6-27B-Q8_0]
threads = 10