Gan, Jimmy
3994e29cb0
Full documentation of optimizing two LLMs on a single M5 Max GPU: - KV cache quantization (Q4_0) - Flash attention and batch tuning - Router mode with --models-max 1 - Per-model thread optimization via INI presets - Before/after benchmarks (12→48 t/s on 27B, 23→132 t/s on 35B)
15 lines
220 B
INI
15 lines
220 B
INI
[*]
|
|
flash-attn = on
|
|
cache-type-k = q4_0
|
|
cache-type-v = q4_0
|
|
batch-size = 512
|
|
ubatch-size = 128
|
|
spec-type = draft-mtp
|
|
spec-draft-n-max = 3
|
|
|
|
[model.Qwen3.6-35B-A3B-Q8_0]
|
|
threads = 14
|
|
|
|
[model.Qwen3.6-27B-Q8_0]
|
|
threads = 10
|