Gan, Jimmy
3994e29cb0
Full documentation of optimizing two LLMs on a single M5 Max GPU: - KV cache quantization (Q4_0) - Flash attention and batch tuning - Router mode with --models-max 1 - Per-model thread optimization via INI presets - Before/after benchmarks (12→48 t/s on 27B, 23→132 t/s on 35B)
1.9 KiB
The Problem: Two Models, One GPU
Why Two Models?
Different model architectures excel at different tasks:
- MoE (Mixture of Experts): Fast generation (~3B active params), good for chat. Example: Qwen3.6-35B-A3B
- Dense: Better reasoning on coding tasks, all parameters active. Example: Qwen3.6-27B
Running both gives you the best of both worlds — but on a single GPU, they compete.
The Naive Approach
The obvious solution: run two llama-server processes on different ports.
# Server 1: 35B chat model on :8085
llama-server -m qwen35b.gguf --port 8085 -ngl 99
# Server 2: 27B coding model on :8080
llama-server -m qwen27b.gguf --port 8080 -ngl 99
Result: Both models stay permanently loaded in GPU memory, permanently competing for memory bandwidth.
Why It's Slow
On Apple Silicon's Unified Memory architecture, GPU and CPU share the same memory pool. When two processes both use Metal GPU acceleration:
- Memory bandwidth is shared — Both models' weights (75 GB combined) compete for the ~614 GB/s memory bus
- GPU scheduler splits time — macOS Metal driver context-switches between processes
- KV cache doubles the tax — Both models maintain large KV caches for their context windows
The result: each model gets roughly half the GPU bandwidth it could achieve alone.
Measured Impact
| Model | Solo Speed | With Other Loaded | Penalty |
|---|---|---|---|
| 35B MoE | 94 t/s | 23 t/s | -75% |
| 27B dense | 12 t/s | 12 t/s | ~0% (already bandwidth-saturated) |
The 27B barely notices because it's already reading 27 GB of weights per token — its 12 t/s saturates the available bandwidth regardless. The 35B MoE (3B active params) has huge headroom but loses most of it to contention.
The Goal
Find a way to give each model full GPU bandwidth without having to manually kill and restart servers.