commit 3994e29cb055bb91bc6ae852182d6d84f18c4a44 Author: Gan, Jimmy Date: Fri Jul 10 02:27:36 2026 +0800 Initial: llama.cpp router mode optimization guide for Apple Silicon Full documentation of optimizing two LLMs on a single M5 Max GPU: - KV cache quantization (Q4_0) - Flash attention and batch tuning - Router mode with --models-max 1 - Per-model thread optimization via INI presets - Before/after benchmarks (12→48 t/s on 27B, 23→132 t/s on 35B) diff --git a/README.md b/README.md new file mode 100644 index 0000000..ac684eb --- /dev/null +++ b/README.md @@ -0,0 +1,88 @@ +# llama.cpp Router Mode on Apple Silicon + +**Single GPU, two models, zero contention — from 12 t/s to 132 t/s** + +A practical guide to running multiple LLMs on one Apple Silicon Mac using `llama-server` router mode, achieving full GPU bandwidth for each model with automatic LRU eviction. + +## The Problem + +Running two `llama-server` processes on the same GPU causes severe bandwidth contention: + +``` +Two separate servers (before): + 35B MoE: 23 t/s ← -75% from potential + 27B dense: 16 t/s ← -63% from potential +``` + +Both models stay permanently loaded, permanently competing for GPU memory bandwidth. + +## The Solution + +A single `llama-server` with `--models-max 1` evicts the idle model from GPU memory, giving full bandwidth to whichever model is active: + +``` +Router mode (after): + 35B MoE: 132 t/s ← 5.7x faster + 27B dense: 48 t/s ← 3.0x faster +``` + +## Quick Start + +```bash +# 1. Create model symlinks +mkdir -p ~/.hermes/models-router +ln -sf /path/to/model1.gguf ~/.hermes/models-router/ +ln -sf /path/to/model2.gguf ~/.hermes/models-router/ + +# 2. Create per-model INI preset +cat > ~/.hermes/llama-models.ini << 'EOF' +[*] +flash-attn = on +cache-type-k = q4_0 +cache-type-v = q4_0 +batch-size = 512 +ubatch-size = 128 +spec-type = draft-mtp +spec-draft-n-max = 3 + +[model.Model1] +threads = 14 + +[model.Model2] +threads = 10 +EOF + +# 3. Launch router +llama-server \ + --models-dir ~/.hermes/models-router \ + --models-max 1 \ + --models-preset ~/.hermes/llama-models.ini \ + --host 0.0.0.0 --port 8085 \ + -ngl 99 -c 131072 --mlock \ + --metrics +``` + +## Speed Results (M5 Max 40-core, 128GB) + +| Model | Two Servers | Router Mode | Gain | +|-------|-------------|-------------|------| +| Qwen3.6-35B-A3B-Q8_0 (MoE) | 23 t/s | 132 t/s | **5.7x** | +| Qwen3.6-27B-Q8_0 (dense) | 16 t/s | 48 t/s | **3.0x** | + +Trade-off: ~3-4s model load time when switching models. + +## Hardware + +- **Chip:** Apple M5 Max (40 GPU cores, 614 GB/s bandwidth) +- **RAM:** 128 GB Unified Memory +- **Software:** llama.cpp b9910+, launchd, Hermes Agent + +## Contents + +- `docs/` — Full optimization journey with benchmarks at each step +- `configs/` — Launchd plist, INI preset, utility scripts +- `benchmarks/` — Before/after speed comparison data + +## License + +MIT — use it, share it, productize it. diff --git a/benchmarks/results.md b/benchmarks/results.md new file mode 100644 index 0000000..c3aec3c --- /dev/null +++ b/benchmarks/results.md @@ -0,0 +1,87 @@ +# Benchmark Results + +## Hardware + +| Component | Spec | +|-----------|------| +| Chip | Apple M5 Max | +| GPU Cores | 40 | +| RAM | 128 GB Unified Memory | +| Memory Bandwidth | 614 GB/s | +| OS | macOS 26.3.2 | +| llama.cpp | b9910 (f5525f7e7) | + +## Models + +| Model | Architecture | Size | Quant | +|-------|-------------|------|-------| +| Qwen3.6-35B-A3B | MoE (3B active) | ~35 GB | Q8_0 | +| Qwen3.6-27B | Dense | ~27 GB | Q8_0 | + +## Config + +``` +-c 131072, --cache-type-k q4_0, --cache-type-v q4_0 +--flash-attn on, --spec-type draft-mtp, --spec-draft-n-max 3 +--batch-size 512, --ubatch-size 128 +``` + +## Results + +### Two Separate Servers (Before) + +**Both loaded, both generating:** + +| Model | Gen Speed | Notes | +|-------|-----------|-------| +| 35B MoE | 23 t/s | -75% from solo potential | +| 27B dense | 12 t/s | Already bandwidth-saturated | + +**Solo (only one running):** + +| Model | Gen Speed | Bandwidth Efficiency | +|-------|-----------|---------------------| +| 35B MoE | 94 t/s | 3.2 GB/step × 94 = 49% of 614 GB/s | +| 27B dense | 12 t/s | 30.4 GB/step × 12 = 59% of 614 GB/s | + +### Router Mode (After) + +**--models-max 1, auto-swap on request:** + +| Model | Gen Speed (cold) | Gen Speed (warm) | Gain vs Before | +|-------|-----------------|------------------|----------------| +| 35B MoE | 132 t/s | 148 t/s | **5.7x** | +| 27B dense | 38 t/s | 48 t/s | **3.0x** | + +**Per-model thread tuning:** + +| Model | Optimal Threads | Speed | +|-------|----------------|-------| +| 35B MoE (attention-compute bound) | 14 | 148 t/s | +| 27B dense (bandwidth bound) | 10 | 48 t/s | + +### Model Switch Latency + +| Transition | Time | +|-----------|------| +| 35B → 27B (cold start) | ~4s | +| 27B → 35B (reload) | ~3s | + +The 3-4s delay is model weights loading from SSD to GPU memory via Metal. Smaller batch sizes and Q4_0 KV cache help minimize this. + +## MTP Speculative Decoding + +| Model | Acceptance Rate | Mean Draft Length | Effective Tokens/Step | +|-------|----------------|-------------------|----------------------| +| 35B MoE | 74% | 2.84 | 2.1 | +| 27B dense | 82% | 3.62 | 2.6 | + +MTP (Multi-Token Prediction) uses the model's built-in speculative heads. Each forward pass produces 3 draft tokens; acceptance rate determines how many are kept. With ~2.5 tokens per forward pass, effective generation speed is ~2.5x the raw decode speed. + +## Methodology + +- Tested via llama.cpp `/v1/completions` API +- Fixed prompt: 7-11 tokens +- Generation: 200 tokens, temperature 0 +- Timings from response body `timings.predicted_per_second` +- Warm = model already loaded in GPU, cold = first request after model load diff --git a/configs/com.llama-server-router.plist b/configs/com.llama-server-router.plist new file mode 100644 index 0000000..cad688e --- /dev/null +++ b/configs/com.llama-server-router.plist @@ -0,0 +1,39 @@ + + + + + KeepAlive + Labelcom.example.llama-server-router + ProgramArguments + + /opt/homebrew/bin/llama-server + --models-dir + /Users/username/.hermes/models-router + --models-max1 + --models-preset + /Users/username/.hermes/llama-models.ini + --host0.0.0.0 + --port8085 + -ngl99 + -c131072 + --mlock + --spec-typedraft-mtp + --spec-draft-n-max3 + --batch-size512 + --ubatch-size128 + --cache-type-kq4_0 + --cache-type-vq4_0 + --flash-attnon + --metrics + + RunAtLoad + StandardOutPath + /tmp/llama-server-router.log + StandardErrorPath + /tmp/llama-server-router.log + ThrottleInterval + 5 + WorkingDirectory + /Users/username + + diff --git a/configs/llama-models.ini b/configs/llama-models.ini new file mode 100644 index 0000000..c0c2ae4 --- /dev/null +++ b/configs/llama-models.ini @@ -0,0 +1,14 @@ +[*] +flash-attn = on +cache-type-k = q4_0 +cache-type-v = q4_0 +batch-size = 512 +ubatch-size = 128 +spec-type = draft-mtp +spec-draft-n-max = 3 + +[model.Qwen3.6-35B-A3B-Q8_0] +threads = 14 + +[model.Qwen3.6-27B-Q8_0] +threads = 10 diff --git a/docs/01-problem.md b/docs/01-problem.md new file mode 100644 index 0000000..af43f3d --- /dev/null +++ b/docs/01-problem.md @@ -0,0 +1,47 @@ +# The Problem: Two Models, One GPU + +## Why Two Models? + +Different model architectures excel at different tasks: + +- **MoE (Mixture of Experts):** Fast generation (~3B active params), good for chat. Example: Qwen3.6-35B-A3B +- **Dense:** Better reasoning on coding tasks, all parameters active. Example: Qwen3.6-27B + +Running both gives you the best of both worlds — but on a single GPU, they compete. + +## The Naive Approach + +The obvious solution: run two `llama-server` processes on different ports. + +```bash +# Server 1: 35B chat model on :8085 +llama-server -m qwen35b.gguf --port 8085 -ngl 99 + +# Server 2: 27B coding model on :8080 +llama-server -m qwen27b.gguf --port 8080 -ngl 99 +``` + +**Result:** Both models stay permanently loaded in GPU memory, permanently competing for memory bandwidth. + +## Why It's Slow + +On Apple Silicon's Unified Memory architecture, GPU and CPU share the same memory pool. When two processes both use Metal GPU acceleration: + +1. **Memory bandwidth is shared** — Both models' weights (75 GB combined) compete for the ~614 GB/s memory bus +2. **GPU scheduler splits time** — macOS Metal driver context-switches between processes +3. **KV cache doubles the tax** — Both models maintain large KV caches for their context windows + +The result: each model gets roughly **half** the GPU bandwidth it could achieve alone. + +## Measured Impact + +| Model | Solo Speed | With Other Loaded | Penalty | +|-------|-----------|-------------------|---------| +| 35B MoE | 94 t/s | 23 t/s | **-75%** | +| 27B dense | 12 t/s | 12 t/s | **~0%** (already bandwidth-saturated) | + +The 27B barely notices because it's already reading 27 GB of weights per token — its 12 t/s saturates the available bandwidth regardless. The 35B MoE (3B active params) has huge headroom but loses most of it to contention. + +## The Goal + +Find a way to give each model **full GPU bandwidth** without having to manually kill and restart servers. diff --git a/docs/02-initial-setup.md b/docs/02-initial-setup.md new file mode 100644 index 0000000..c79eac2 --- /dev/null +++ b/docs/02-initial-setup.md @@ -0,0 +1,90 @@ +# Step 1: Two Separate Servers + +The initial setup for running both model servers as persistent macOS services via `launchd`. + +## Launchd Plist for 35B Chat Model + +```xml + + + + KeepAlive + ProgramArguments + + /opt/homebrew/bin/llama-server + -m + /Users/jimmyg/models/Qwen3.6-35B-A3B-Q8_0.gguf + --host0.0.0.0 + --port8085 + -ngl99 + -c262144 + --mlock + + RunAtLoad + StandardOutPath + /Users/jimmyg/.hermes/logs/llama-server-qwen35b.log + StandardErrorPath + /Users/jimmyg/.hermes/logs/llama-server-qwen35b.log + + +``` + +## Launchd Plist for 27B Coding Model + +```xml + + + + KeepAlive + ProgramArguments + + /opt/homebrew/bin/llama-server + -m + /Users/jimmyg/models/Qwen3.6-27B-Q8_0.gguf + --host0.0.0.0 + --port8080 + -ngl99 + -c131072 + + RunAtLoad + + +``` + +## Initial Flags Explained + +| Flag | Value | Purpose | +|------|-------|---------| +| `-ngl 99` | All layers on GPU | Full Metal acceleration | +| `-c` | 262K / 131K | Context window size | +| `--mlock` | - | Prevent model from being swapped | +| `--spec-type draft-mtp` | - | Multi-Token Prediction speculation | + +## Loading the Services + +```bash +launchctl load ~/Library/LaunchAgents/com.jimmyg.llama-server-qwen35b.plist +launchctl load ~/Library/LaunchAgents/com.jimmyg.llama-server.plist +``` + +## Hermes Agent Integration + +Two custom providers in `~/.hermes/config.yaml`: + +```yaml +custom_providers: + - base_url: http://localhost:8085/v1 + model: /Users/jimmyg/models/Qwen3.6-35B-A3B-Q8_0.gguf + name: qwen35b + - base_url: http://localhost:8080/v1 + model: Qwen3.6-27B-Q8_0 + name: qwen27b +``` + +## Baseline Performance + +At this point with default settings: +- 35B MoE: ~20 t/s +- 27B dense: ~12 t/s + +Plenty of room for optimization. diff --git a/docs/03-optimization-journey.md b/docs/03-optimization-journey.md new file mode 100644 index 0000000..e901b70 --- /dev/null +++ b/docs/03-optimization-journey.md @@ -0,0 +1,99 @@ +# The Optimization Journey + +## Overview + +Starting from the baseline (two servers, ~20 t/s on 35B, ~12 t/s on 27B), each optimization improved speed by addressing specific bottlenecks. + +## Optimization 1: KV Cache Quantization + +**Problem:** Full-precision (f16) KV cache consumes massive RAM. For 128K context on the 35B model, the KV cache alone uses ~15 GB. + +**Fix:** Switch to Q4_0 KV cache. + +```bash +--cache-type-k q4_0 --cache-type-v q4_0 +``` + +**Impact:** -50% KV cache memory. Negligible quality loss — KV cache is a caching layer, not model weights. The precision of cached attention states has minimal effect on output quality. + +## Optimization 2: Context Window Tuning + +**Problem:** The 35B used 262K context (way too much for chat). The 27B used 131K. Combined KV caches consumed ~32 GB. + +**Fix:** Both models at 128K. Sufficient for agent sessions (typical usage tops out at ~65K tokens). + +```bash +-c 131072 +``` + +**Impact:** Saved ~16 GB RAM. Freed headroom for the rest of the system. + +## Optimization 3: Flash Attention + +**Problem:** Without flash attention, generation speed degrades as KV cache grows (O(n²) attention cost). + +**Fix:** Enable flash attention. + +```bash +--flash-attn on +``` + +**Impact:** Maintains generation speed even with large contexts. With FA, generation stays at ~20 t/s regardless of context size. Without it, speed drops to ~10 t/s when context reaches 20K+ tokens. + +## Optimization 4: Batch Size Tuning + +**Problem:** Default batch sizes (2048/512) are optimized for server throughput (many concurrent requests), not single-user generation latency. + +**Fix:** Smaller batch sizes for single-user use. + +```bash +--batch-size 512 --ubatch-size 128 +``` + +**Impact:** +25-50% generation speed for dense models. The 27B jumped from 12 to 16 t/s. The 35B MoE was less affected (3B active params already efficient). + +**Why:** During generation (1 token at a time), the batch size affects how the GPU schedules compute. Smaller batches reduce latency per decode step. + +## Optimization 5: Thread Count Tuning + +**Problem:** Auto-detected thread count uses all CPU cores (18), causing dispatch overhead without benefit. + +**Fix:** Explicit thread count optimized per architecture. + +```bash +-t 14 # General fallback +``` + +**Impact:** MoE models benefit from more threads (attention compute-heavy). Dense models benefit from fewer threads (bandwidth-bound, less dispatch overhead). + +| Model | Optimal Threads | Speed | +|-------|----------------|-------| +| 35B MoE (3B active) | 14 | 132 t/s | +| 27B dense (27B active) | 10 | 48 t/s | + +## Optimization 6: Speculative Decoding (MTP) + +**Fix:** Increase speculation from 1 to 3. + +```bash +--spec-draft-n-max 3 +``` + +**Impact:** With 82-93% MTP acceptance rate, each forward pass produces ~2.5-3 tokens instead of 1. This effectively doubles generation speed for free. + +**Key insight:** Qwen models have built-in Multi-Token Prediction heads. The `--spec-type draft-mtp` flag uses these native heads rather than a separate draft model, adding negligible overhead. + +## The Big One: Router Mode + +The above optimizations improved speed, but the fundamental problem remained: **two processes competing for GPU bandwidth.** + +**Fix:** Replace two `llama-server` processes with one router-mode server. + +```bash +llama-server \ + --models-dir ~/.hermes/models-router \ + --models-max 1 \ + --models-preset ~/.hermes/llama-models.ini +``` + +**Impact:** +5.7x on 35B, +3.0x on 27B. See `06-router-mode.md` for full details. diff --git a/docs/06-router-mode.md b/docs/06-router-mode.md new file mode 100644 index 0000000..56aeee7 --- /dev/null +++ b/docs/06-router-mode.md @@ -0,0 +1,166 @@ +# Router Mode Setup + +## Architecture + +Instead of running two separate `llama-server` processes, the router mode runs a single server that: + +1. **Discovers models** in a directory (symlinks to actual GGUF files) +2. **Loads on demand** — only loads a model when a request comes in +3. **Evicts by LRU** — when `--models-max 1`, switches models atomically + +``` +┌─────────────────┐ ┌──────────────────┐ +│ Chat Request │────▶│ llama-server │ +│ 35B MoE │ │ Router Mode │ +└─────────────────┘ │ :8085 │ + │ │ +┌─────────────────┐ │ ┌────────────┐ │ +│ Coding Subagent │────▶│ │ 35B MoE │ │ +│ 27B dense │ │ │ (loaded) │ │ +└─────────────────┘ │ └────────────┘ │ + │ │ + │ ┌────────────┐ │ + │ │ 27B dense │ │ + │ │ (evicted) │ │ + │ └────────────┘ │ + └──────────────────┘ +``` + +## Models Directory + +Create a directory with symlinks to your GGUF files: + +```bash +mkdir -p ~/.hermes/models-router +ln -sf /path/to/Qwen3.6-35B-A3B-Q8_0.gguf ~/.hermes/models-router/ +ln -sf /path/to/Qwen3.6-27B-Q8_0.gguf ~/.hermes/models-router/ +``` + +The server uses the GGUF filename (without extension) as the model ID. + +## Launch Command + +```bash +llama-server \ + --models-dir ~/.hermes/models-router \ + --models-max 1 \ + --models-preset ~/.hermes/llama-models.ini \ + --host 0.0.0.0 --port 8085 \ + -ngl 99 -c 131072 --mlock \ + --spec-type draft-mtp --spec-draft-n-max 3 \ + --batch-size 512 --ubatch-size 128 \ + --cache-type-k q4_0 --cache-type-v q4_0 \ + --flash-attn on \ + --metrics +``` + +## Per-Model INI Preset + +Model-specific settings (like thread count) go in the INI file: + +```ini +[*] +flash-attn = on +cache-type-k = q4_0 +cache-type-v = q4_0 +batch-size = 512 +ubatch-size = 128 +spec-type = draft-mtp +spec-draft-n-max = 3 + +[model.Qwen3.6-35B-A3B-Q8_0] +threads = 14 + +[model.Qwen3.6-27B-Q8_0] +threads = 10 +``` + +The `[*]` section applies to all models. Model-specific sections override for individual models. + +## Launchd Service (Persistent) + +Create a launchd plist for automatic startup: + +```xml + + + + + KeepAlive + Labelcom.jimmyg.llama-server-router + ProgramArguments + + /opt/homebrew/bin/llama-server + --models-dir + /Users/jimmyg/.hermes/models-router + --models-max1 + --models-preset + /Users/jimmyg/.hermes/llama-models.ini + --host0.0.0.0 + --port8085 + -ngl99 + -c131072 + --mlock + --spec-typedraft-mtp + --spec-draft-n-max3 + --batch-size512 + --ubatch-size128 + --cache-type-kq4_0 + --cache-type-vq4_0 + --flash-attnon + --metrics + + RunAtLoad + StandardOutPath + /Users/jimmyg/.hermes/logs/llama-server-router.log + StandardErrorPath + /Users/jimmyg/.hermes/logs/llama-server-router.log + + +``` + +Load it: + +```bash +launchctl load ~/Library/LaunchAgents/com.jimmyg.llama-server-router.plist +``` + +## Hermes Agent Config + +Both providers now point to the same server: + +```yaml +custom_providers: + - base_url: http://localhost:8085/v1 + model: Qwen3.6-35B-A3B-Q8_0 + name: qwen35b + - base_url: http://localhost:8085/v1 + model: Qwen3.6-27B-Q8_0 + name: qwen27b +``` + +## API Usage + +Send requests with the model name (GGUF filename without extension): + +```bash +# 35B chat +curl -X POST http://localhost:8085/v1/completions \ + -d '{"model":"Qwen3.6-35B-A3B-Q8_0","prompt":"Hello"}' + +# 27B coding +curl -X POST http://localhost:8085/v1/completions \ + -d '{"model":"Qwen3.6-27B-Q8_0","prompt":"def quicksort:"}' +``` + +## Monitoring + +The `--metrics` flag exposes Prometheus-formatted metrics: + +```bash +curl http://localhost:8085/metrics | grep predicted_tokens_seconds +``` + +## Known Limitation + +`--slot-save-path` (KV cache persistence) does not work with router mode in current versions. Child processes are killed on eviction, so KV cache is lost. This is a feature gap, not a bug — router mode wasn't designed for context persistence across model switches.