Gan, Jimmy
3994e29cb0
Full documentation of optimizing two LLMs on a single M5 Max GPU: - KV cache quantization (Q4_0) - Flash attention and batch tuning - Router mode with --models-max 1 - Per-model thread optimization via INI presets - Before/after benchmarks (12→48 t/s on 27B, 23→132 t/s on 35B)
88 lines
2.5 KiB
Markdown
88 lines
2.5 KiB
Markdown
# Benchmark Results
|
||
|
||
## Hardware
|
||
|
||
| Component | Spec |
|
||
|-----------|------|
|
||
| Chip | Apple M5 Max |
|
||
| GPU Cores | 40 |
|
||
| RAM | 128 GB Unified Memory |
|
||
| Memory Bandwidth | 614 GB/s |
|
||
| OS | macOS 26.3.2 |
|
||
| llama.cpp | b9910 (f5525f7e7) |
|
||
|
||
## Models
|
||
|
||
| Model | Architecture | Size | Quant |
|
||
|-------|-------------|------|-------|
|
||
| Qwen3.6-35B-A3B | MoE (3B active) | ~35 GB | Q8_0 |
|
||
| Qwen3.6-27B | Dense | ~27 GB | Q8_0 |
|
||
|
||
## Config
|
||
|
||
```
|
||
-c 131072, --cache-type-k q4_0, --cache-type-v q4_0
|
||
--flash-attn on, --spec-type draft-mtp, --spec-draft-n-max 3
|
||
--batch-size 512, --ubatch-size 128
|
||
```
|
||
|
||
## Results
|
||
|
||
### Two Separate Servers (Before)
|
||
|
||
**Both loaded, both generating:**
|
||
|
||
| Model | Gen Speed | Notes |
|
||
|-------|-----------|-------|
|
||
| 35B MoE | 23 t/s | -75% from solo potential |
|
||
| 27B dense | 12 t/s | Already bandwidth-saturated |
|
||
|
||
**Solo (only one running):**
|
||
|
||
| Model | Gen Speed | Bandwidth Efficiency |
|
||
|-------|-----------|---------------------|
|
||
| 35B MoE | 94 t/s | 3.2 GB/step × 94 = 49% of 614 GB/s |
|
||
| 27B dense | 12 t/s | 30.4 GB/step × 12 = 59% of 614 GB/s |
|
||
|
||
### Router Mode (After)
|
||
|
||
**--models-max 1, auto-swap on request:**
|
||
|
||
| Model | Gen Speed (cold) | Gen Speed (warm) | Gain vs Before |
|
||
|-------|-----------------|------------------|----------------|
|
||
| 35B MoE | 132 t/s | 148 t/s | **5.7x** |
|
||
| 27B dense | 38 t/s | 48 t/s | **3.0x** |
|
||
|
||
**Per-model thread tuning:**
|
||
|
||
| Model | Optimal Threads | Speed |
|
||
|-------|----------------|-------|
|
||
| 35B MoE (attention-compute bound) | 14 | 148 t/s |
|
||
| 27B dense (bandwidth bound) | 10 | 48 t/s |
|
||
|
||
### Model Switch Latency
|
||
|
||
| Transition | Time |
|
||
|-----------|------|
|
||
| 35B → 27B (cold start) | ~4s |
|
||
| 27B → 35B (reload) | ~3s |
|
||
|
||
The 3-4s delay is model weights loading from SSD to GPU memory via Metal. Smaller batch sizes and Q4_0 KV cache help minimize this.
|
||
|
||
## MTP Speculative Decoding
|
||
|
||
| Model | Acceptance Rate | Mean Draft Length | Effective Tokens/Step |
|
||
|-------|----------------|-------------------|----------------------|
|
||
| 35B MoE | 74% | 2.84 | 2.1 |
|
||
| 27B dense | 82% | 3.62 | 2.6 |
|
||
|
||
MTP (Multi-Token Prediction) uses the model's built-in speculative heads. Each forward pass produces 3 draft tokens; acceptance rate determines how many are kept. With ~2.5 tokens per forward pass, effective generation speed is ~2.5x the raw decode speed.
|
||
|
||
## Methodology
|
||
|
||
- Tested via llama.cpp `/v1/completions` API
|
||
- Fixed prompt: 7-11 tokens
|
||
- Generation: 200 tokens, temperature 0
|
||
- Timings from response body `timings.predicted_per_second`
|
||
- Warm = model already loaded in GPU, cold = first request after model load
|