Initial: llama.cpp router mode optimization guide for Apple Silicon
Full documentation of optimizing two LLMs on a single M5 Max GPU: - KV cache quantization (Q4_0) - Flash attention and batch tuning - Router mode with --models-max 1 - Per-model thread optimization via INI presets - Before/after benchmarks (12→48 t/s on 27B, 23→132 t/s on 35B)
This commit is contained in:
@@ -0,0 +1,14 @@
|
||||
[*]
|
||||
flash-attn = on
|
||||
cache-type-k = q4_0
|
||||
cache-type-v = q4_0
|
||||
batch-size = 512
|
||||
ubatch-size = 128
|
||||
spec-type = draft-mtp
|
||||
spec-draft-n-max = 3
|
||||
|
||||
[model.Qwen3.6-35B-A3B-Q8_0]
|
||||
threads = 14
|
||||
|
||||
[model.Qwen3.6-27B-Q8_0]
|
||||
threads = 10
|
||||
Reference in New Issue
Block a user