Gan, Jimmy
3994e29cb0
Full documentation of optimizing two LLMs on a single M5 Max GPU: - KV cache quantization (Q4_0) - Flash attention and batch tuning - Router mode with --models-max 1 - Per-model thread optimization via INI presets - Before/after benchmarks (12→48 t/s on 27B, 23→132 t/s on 35B)
89 lines
2.2 KiB
Markdown
89 lines
2.2 KiB
Markdown
# llama.cpp Router Mode on Apple Silicon
|
|
|
|
**Single GPU, two models, zero contention — from 12 t/s to 132 t/s**
|
|
|
|
A practical guide to running multiple LLMs on one Apple Silicon Mac using `llama-server` router mode, achieving full GPU bandwidth for each model with automatic LRU eviction.
|
|
|
|
## The Problem
|
|
|
|
Running two `llama-server` processes on the same GPU causes severe bandwidth contention:
|
|
|
|
```
|
|
Two separate servers (before):
|
|
35B MoE: 23 t/s ← -75% from potential
|
|
27B dense: 16 t/s ← -63% from potential
|
|
```
|
|
|
|
Both models stay permanently loaded, permanently competing for GPU memory bandwidth.
|
|
|
|
## The Solution
|
|
|
|
A single `llama-server` with `--models-max 1` evicts the idle model from GPU memory, giving full bandwidth to whichever model is active:
|
|
|
|
```
|
|
Router mode (after):
|
|
35B MoE: 132 t/s ← 5.7x faster
|
|
27B dense: 48 t/s ← 3.0x faster
|
|
```
|
|
|
|
## Quick Start
|
|
|
|
```bash
|
|
# 1. Create model symlinks
|
|
mkdir -p ~/.hermes/models-router
|
|
ln -sf /path/to/model1.gguf ~/.hermes/models-router/
|
|
ln -sf /path/to/model2.gguf ~/.hermes/models-router/
|
|
|
|
# 2. Create per-model INI preset
|
|
cat > ~/.hermes/llama-models.ini << 'EOF'
|
|
[*]
|
|
flash-attn = on
|
|
cache-type-k = q4_0
|
|
cache-type-v = q4_0
|
|
batch-size = 512
|
|
ubatch-size = 128
|
|
spec-type = draft-mtp
|
|
spec-draft-n-max = 3
|
|
|
|
[model.Model1]
|
|
threads = 14
|
|
|
|
[model.Model2]
|
|
threads = 10
|
|
EOF
|
|
|
|
# 3. Launch router
|
|
llama-server \
|
|
--models-dir ~/.hermes/models-router \
|
|
--models-max 1 \
|
|
--models-preset ~/.hermes/llama-models.ini \
|
|
--host 0.0.0.0 --port 8085 \
|
|
-ngl 99 -c 131072 --mlock \
|
|
--metrics
|
|
```
|
|
|
|
## Speed Results (M5 Max 40-core, 128GB)
|
|
|
|
| Model | Two Servers | Router Mode | Gain |
|
|
|-------|-------------|-------------|------|
|
|
| Qwen3.6-35B-A3B-Q8_0 (MoE) | 23 t/s | 132 t/s | **5.7x** |
|
|
| Qwen3.6-27B-Q8_0 (dense) | 16 t/s | 48 t/s | **3.0x** |
|
|
|
|
Trade-off: ~3-4s model load time when switching models.
|
|
|
|
## Hardware
|
|
|
|
- **Chip:** Apple M5 Max (40 GPU cores, 614 GB/s bandwidth)
|
|
- **RAM:** 128 GB Unified Memory
|
|
- **Software:** llama.cpp b9910+, launchd, Hermes Agent
|
|
|
|
## Contents
|
|
|
|
- `docs/` — Full optimization journey with benchmarks at each step
|
|
- `configs/` — Launchd plist, INI preset, utility scripts
|
|
- `benchmarks/` — Before/after speed comparison data
|
|
|
|
## License
|
|
|
|
MIT — use it, share it, productize it.
|