Gan, Jimmy
3994e29cb0
Full documentation of optimizing two LLMs on a single M5 Max GPU: - KV cache quantization (Q4_0) - Flash attention and batch tuning - Router mode with --models-max 1 - Per-model thread optimization via INI presets - Before/after benchmarks (12→48 t/s on 27B, 23→132 t/s on 35B)
48 lines
1.9 KiB
Markdown
48 lines
1.9 KiB
Markdown
# The Problem: Two Models, One GPU
|
|
|
|
## Why Two Models?
|
|
|
|
Different model architectures excel at different tasks:
|
|
|
|
- **MoE (Mixture of Experts):** Fast generation (~3B active params), good for chat. Example: Qwen3.6-35B-A3B
|
|
- **Dense:** Better reasoning on coding tasks, all parameters active. Example: Qwen3.6-27B
|
|
|
|
Running both gives you the best of both worlds — but on a single GPU, they compete.
|
|
|
|
## The Naive Approach
|
|
|
|
The obvious solution: run two `llama-server` processes on different ports.
|
|
|
|
```bash
|
|
# Server 1: 35B chat model on :8085
|
|
llama-server -m qwen35b.gguf --port 8085 -ngl 99
|
|
|
|
# Server 2: 27B coding model on :8080
|
|
llama-server -m qwen27b.gguf --port 8080 -ngl 99
|
|
```
|
|
|
|
**Result:** Both models stay permanently loaded in GPU memory, permanently competing for memory bandwidth.
|
|
|
|
## Why It's Slow
|
|
|
|
On Apple Silicon's Unified Memory architecture, GPU and CPU share the same memory pool. When two processes both use Metal GPU acceleration:
|
|
|
|
1. **Memory bandwidth is shared** — Both models' weights (75 GB combined) compete for the ~614 GB/s memory bus
|
|
2. **GPU scheduler splits time** — macOS Metal driver context-switches between processes
|
|
3. **KV cache doubles the tax** — Both models maintain large KV caches for their context windows
|
|
|
|
The result: each model gets roughly **half** the GPU bandwidth it could achieve alone.
|
|
|
|
## Measured Impact
|
|
|
|
| Model | Solo Speed | With Other Loaded | Penalty |
|
|
|-------|-----------|-------------------|---------|
|
|
| 35B MoE | 94 t/s | 23 t/s | **-75%** |
|
|
| 27B dense | 12 t/s | 12 t/s | **~0%** (already bandwidth-saturated) |
|
|
|
|
The 27B barely notices because it's already reading 27 GB of weights per token — its 12 t/s saturates the available bandwidth regardless. The 35B MoE (3B active params) has huge headroom but loses most of it to contention.
|
|
|
|
## The Goal
|
|
|
|
Find a way to give each model **full GPU bandwidth** without having to manually kill and restart servers.
|