Files

T

Gan, Jimmy 3994e29cb0 Initial: llama.cpp router mode optimization guide for Apple Silicon

Full documentation of optimizing two LLMs on a single M5 Max GPU:
- KV cache quantization (Q4_0)
- Flash attention and batch tuning
- Router mode with --models-max 1
- Per-model thread optimization via INI presets
- Before/after benchmarks (12→48 t/s on 27B, 23→132 t/s on 35B)

2026-07-10 02:27:36 +08:00

2.5 KiB

Raw Blame History

Benchmark Results

Hardware

Component	Spec
Chip	Apple M5 Max
GPU Cores	40
RAM	128 GB Unified Memory
Memory Bandwidth	614 GB/s
OS	macOS 26.3.2
llama.cpp	b9910 (f5525f7e7)

Models

Model	Architecture	Size	Quant
Qwen3.6-35B-A3B	MoE (3B active)	~35 GB	Q8_0
Qwen3.6-27B	Dense	~27 GB	Q8_0

Config

-c 131072, --cache-type-k q4_0, --cache-type-v q4_0
--flash-attn on, --spec-type draft-mtp, --spec-draft-n-max 3
--batch-size 512, --ubatch-size 128

Results

Two Separate Servers (Before)

Both loaded, both generating:

Model	Gen Speed	Notes
35B MoE	23 t/s	-75% from solo potential
27B dense	12 t/s	Already bandwidth-saturated

Solo (only one running):

Model	Gen Speed	Bandwidth Efficiency
35B MoE	94 t/s	3.2 GB/step × 94 = 49% of 614 GB/s
27B dense	12 t/s	30.4 GB/step × 12 = 59% of 614 GB/s

Router Mode (After)

--models-max 1, auto-swap on request:

Model	Gen Speed (cold)	Gen Speed (warm)	Gain vs Before
35B MoE	132 t/s	148 t/s	5.7x
27B dense	38 t/s	48 t/s	3.0x

Per-model thread tuning:

Model	Optimal Threads	Speed
35B MoE (attention-compute bound)	14	148 t/s
27B dense (bandwidth bound)	10	48 t/s

Model Switch Latency

Transition	Time
35B → 27B (cold start)	~4s
27B → 35B (reload)	~3s

The 3-4s delay is model weights loading from SSD to GPU memory via Metal. Smaller batch sizes and Q4_0 KV cache help minimize this.

MTP Speculative Decoding

Model	Acceptance Rate	Mean Draft Length	Effective Tokens/Step
35B MoE	74%	2.84	2.1
27B dense	82%	3.62	2.6

MTP (Multi-Token Prediction) uses the model's built-in speculative heads. Each forward pass produces 3 draft tokens; acceptance rate determines how many are kept. With ~2.5 tokens per forward pass, effective generation speed is ~2.5x the raw decode speed.

Methodology

Tested via llama.cpp /v1/completions API
Fixed prompt: 7-11 tokens
Generation: 200 tokens, temperature 0
Timings from response body timings.predicted_per_second
Warm = model already loaded in GPU, cold = first request after model load

2.5 KiB Raw Blame History Unescape Escape