Gan, Jimmy
3994e29cb0
Full documentation of optimizing two LLMs on a single M5 Max GPU: - KV cache quantization (Q4_0) - Flash attention and batch tuning - Router mode with --models-max 1 - Per-model thread optimization via INI presets - Before/after benchmarks (12→48 t/s on 27B, 23→132 t/s on 35B)
100 lines
3.4 KiB
Markdown
100 lines
3.4 KiB
Markdown
# The Optimization Journey
|
|
|
|
## Overview
|
|
|
|
Starting from the baseline (two servers, ~20 t/s on 35B, ~12 t/s on 27B), each optimization improved speed by addressing specific bottlenecks.
|
|
|
|
## Optimization 1: KV Cache Quantization
|
|
|
|
**Problem:** Full-precision (f16) KV cache consumes massive RAM. For 128K context on the 35B model, the KV cache alone uses ~15 GB.
|
|
|
|
**Fix:** Switch to Q4_0 KV cache.
|
|
|
|
```bash
|
|
--cache-type-k q4_0 --cache-type-v q4_0
|
|
```
|
|
|
|
**Impact:** -50% KV cache memory. Negligible quality loss — KV cache is a caching layer, not model weights. The precision of cached attention states has minimal effect on output quality.
|
|
|
|
## Optimization 2: Context Window Tuning
|
|
|
|
**Problem:** The 35B used 262K context (way too much for chat). The 27B used 131K. Combined KV caches consumed ~32 GB.
|
|
|
|
**Fix:** Both models at 128K. Sufficient for agent sessions (typical usage tops out at ~65K tokens).
|
|
|
|
```bash
|
|
-c 131072
|
|
```
|
|
|
|
**Impact:** Saved ~16 GB RAM. Freed headroom for the rest of the system.
|
|
|
|
## Optimization 3: Flash Attention
|
|
|
|
**Problem:** Without flash attention, generation speed degrades as KV cache grows (O(n²) attention cost).
|
|
|
|
**Fix:** Enable flash attention.
|
|
|
|
```bash
|
|
--flash-attn on
|
|
```
|
|
|
|
**Impact:** Maintains generation speed even with large contexts. With FA, generation stays at ~20 t/s regardless of context size. Without it, speed drops to ~10 t/s when context reaches 20K+ tokens.
|
|
|
|
## Optimization 4: Batch Size Tuning
|
|
|
|
**Problem:** Default batch sizes (2048/512) are optimized for server throughput (many concurrent requests), not single-user generation latency.
|
|
|
|
**Fix:** Smaller batch sizes for single-user use.
|
|
|
|
```bash
|
|
--batch-size 512 --ubatch-size 128
|
|
```
|
|
|
|
**Impact:** +25-50% generation speed for dense models. The 27B jumped from 12 to 16 t/s. The 35B MoE was less affected (3B active params already efficient).
|
|
|
|
**Why:** During generation (1 token at a time), the batch size affects how the GPU schedules compute. Smaller batches reduce latency per decode step.
|
|
|
|
## Optimization 5: Thread Count Tuning
|
|
|
|
**Problem:** Auto-detected thread count uses all CPU cores (18), causing dispatch overhead without benefit.
|
|
|
|
**Fix:** Explicit thread count optimized per architecture.
|
|
|
|
```bash
|
|
-t 14 # General fallback
|
|
```
|
|
|
|
**Impact:** MoE models benefit from more threads (attention compute-heavy). Dense models benefit from fewer threads (bandwidth-bound, less dispatch overhead).
|
|
|
|
| Model | Optimal Threads | Speed |
|
|
|-------|----------------|-------|
|
|
| 35B MoE (3B active) | 14 | 132 t/s |
|
|
| 27B dense (27B active) | 10 | 48 t/s |
|
|
|
|
## Optimization 6: Speculative Decoding (MTP)
|
|
|
|
**Fix:** Increase speculation from 1 to 3.
|
|
|
|
```bash
|
|
--spec-draft-n-max 3
|
|
```
|
|
|
|
**Impact:** With 82-93% MTP acceptance rate, each forward pass produces ~2.5-3 tokens instead of 1. This effectively doubles generation speed for free.
|
|
|
|
**Key insight:** Qwen models have built-in Multi-Token Prediction heads. The `--spec-type draft-mtp` flag uses these native heads rather than a separate draft model, adding negligible overhead.
|
|
|
|
## The Big One: Router Mode
|
|
|
|
The above optimizations improved speed, but the fundamental problem remained: **two processes competing for GPU bandwidth.**
|
|
|
|
**Fix:** Replace two `llama-server` processes with one router-mode server.
|
|
|
|
```bash
|
|
llama-server \
|
|
--models-dir ~/.hermes/models-router \
|
|
--models-max 1 \
|
|
--models-preset ~/.hermes/llama-models.ini
|
|
```
|
|
|
|
**Impact:** +5.7x on 35B, +3.0x on 27B. See `06-router-mode.md` for full details.
|