Gan, Jimmy
3994e29cb0
Full documentation of optimizing two LLMs on a single M5 Max GPU: - KV cache quantization (Q4_0) - Flash attention and batch tuning - Router mode with --models-max 1 - Per-model thread optimization via INI presets - Before/after benchmarks (12→48 t/s on 27B, 23→132 t/s on 35B)
3.4 KiB
The Optimization Journey
Overview
Starting from the baseline (two servers, ~20 t/s on 35B, ~12 t/s on 27B), each optimization improved speed by addressing specific bottlenecks.
Optimization 1: KV Cache Quantization
Problem: Full-precision (f16) KV cache consumes massive RAM. For 128K context on the 35B model, the KV cache alone uses ~15 GB.
Fix: Switch to Q4_0 KV cache.
--cache-type-k q4_0 --cache-type-v q4_0
Impact: -50% KV cache memory. Negligible quality loss — KV cache is a caching layer, not model weights. The precision of cached attention states has minimal effect on output quality.
Optimization 2: Context Window Tuning
Problem: The 35B used 262K context (way too much for chat). The 27B used 131K. Combined KV caches consumed ~32 GB.
Fix: Both models at 128K. Sufficient for agent sessions (typical usage tops out at ~65K tokens).
-c 131072
Impact: Saved ~16 GB RAM. Freed headroom for the rest of the system.
Optimization 3: Flash Attention
Problem: Without flash attention, generation speed degrades as KV cache grows (O(n²) attention cost).
Fix: Enable flash attention.
--flash-attn on
Impact: Maintains generation speed even with large contexts. With FA, generation stays at ~20 t/s regardless of context size. Without it, speed drops to ~10 t/s when context reaches 20K+ tokens.
Optimization 4: Batch Size Tuning
Problem: Default batch sizes (2048/512) are optimized for server throughput (many concurrent requests), not single-user generation latency.
Fix: Smaller batch sizes for single-user use.
--batch-size 512 --ubatch-size 128
Impact: +25-50% generation speed for dense models. The 27B jumped from 12 to 16 t/s. The 35B MoE was less affected (3B active params already efficient).
Why: During generation (1 token at a time), the batch size affects how the GPU schedules compute. Smaller batches reduce latency per decode step.
Optimization 5: Thread Count Tuning
Problem: Auto-detected thread count uses all CPU cores (18), causing dispatch overhead without benefit.
Fix: Explicit thread count optimized per architecture.
-t 14 # General fallback
Impact: MoE models benefit from more threads (attention compute-heavy). Dense models benefit from fewer threads (bandwidth-bound, less dispatch overhead).
| Model | Optimal Threads | Speed |
|---|---|---|
| 35B MoE (3B active) | 14 | 132 t/s |
| 27B dense (27B active) | 10 | 48 t/s |
Optimization 6: Speculative Decoding (MTP)
Fix: Increase speculation from 1 to 3.
--spec-draft-n-max 3
Impact: With 82-93% MTP acceptance rate, each forward pass produces ~2.5-3 tokens instead of 1. This effectively doubles generation speed for free.
Key insight: Qwen models have built-in Multi-Token Prediction heads. The --spec-type draft-mtp flag uses these native heads rather than a separate draft model, adding negligible overhead.
The Big One: Router Mode
The above optimizations improved speed, but the fundamental problem remained: two processes competing for GPU bandwidth.
Fix: Replace two llama-server processes with one router-mode server.
llama-server \
--models-dir ~/.hermes/models-router \
--models-max 1 \
--models-preset ~/.hermes/llama-models.ini
Impact: +5.7x on 35B, +3.0x on 27B. See 06-router-mode.md for full details.