jimmy/local-agent
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Initial: llama.cpp router mode optimization guide for Apple Silicon

Browse Source 
Full documentation of optimizing two LLMs on a single M5 Max GPU:
- KV cache quantization (Q4_0)
- Flash attention and batch tuning
- Router mode with --models-max 1
- Per-model thread optimization via INI presets
- Before/after benchmarks (12→48 t/s on 27B, 23→132 t/s on 35B)
This commit is contained in:
Gan, Jimmy
2026-07-10 02:27:36 +08:00
commit 3994e29cb0
8 changed files with 630 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+88
View File
@@ -0,0 +1,88 @@
# llama.cpp Router Mode on Apple Silicon
**Single GPU, two models, zero contention — from 12 t/s to 132 t/s**
A practical guide to running multiple LLMs on one Apple Silicon Mac using `llama-server` router mode, achieving full GPU bandwidth for each model with automatic LRU eviction.
## The Problem
Running two `llama-server` processes on the same GPU causes severe bandwidth contention:
```
Two separate servers (before):
35B MoE: 23 t/s ← -75% from potential
27B dense: 16 t/s ← -63% from potential
```
Both models stay permanently loaded, permanently competing for GPU memory bandwidth.
## The Solution
A single `llama-server` with `--models-max 1` evicts the idle model from GPU memory, giving full bandwidth to whichever model is active:
```
Router mode (after):
35B MoE: 132 t/s ← 5.7x faster
27B dense: 48 t/s ← 3.0x faster
```
## Quick Start
```bash
# 1. Create model symlinks
mkdir -p ~/.hermes/models-router
ln -sf /path/to/model1.gguf ~/.hermes/models-router/
ln -sf /path/to/model2.gguf ~/.hermes/models-router/
# 2. Create per-model INI preset
cat > ~/.hermes/llama-models.ini << 'EOF'
[*]
flash-attn = on
cache-type-k = q4_0
cache-type-v = q4_0
batch-size = 512
ubatch-size = 128
spec-type = draft-mtp
spec-draft-n-max = 3
[model.Model1]
threads = 14
[model.Model2]
threads = 10
EOF
# 3. Launch router
llama-server \
--models-dir ~/.hermes/models-router \
--models-max 1 \
--models-preset ~/.hermes/llama-models.ini \
--host 0.0.0.0 --port 8085 \
-ngl 99 -c 131072 --mlock \
--metrics
```
## Speed Results (M5 Max 40-core, 128GB)
| Model | Two Servers | Router Mode | Gain |
|-------|-------------|-------------|------|
| Qwen3.6-35B-A3B-Q8_0 (MoE) | 23 t/s | 132 t/s | **5.7x** |
| Qwen3.6-27B-Q8_0 (dense) | 16 t/s | 48 t/s | **3.0x** |
Trade-off: ~3-4s model load time when switching models.
## Hardware
- **Chip:** Apple M5 Max (40 GPU cores, 614 GB/s bandwidth)
- **RAM:** 128 GB Unified Memory
- **Software:** llama.cpp b9910+, launchd, Hermes Agent
## Contents
- `docs/` — Full optimization journey with benchmarks at each step
- `configs/` — Launchd plist, INI preset, utility scripts
- `benchmarks/` — Before/after speed comparison data
## License
MIT — use it, share it, productize it.
benchmarks/results.md
+87
View File
@@ -0,0 +1,87 @@
# Benchmark Results
## Hardware
| Component | Spec |
|-----------|------|
| Chip | Apple M5 Max |
| GPU Cores | 40 |
| RAM | 128 GB Unified Memory |
| Memory Bandwidth | 614 GB/s |
| OS | macOS 26.3.2 |
| llama.cpp | b9910 (f5525f7e7) |
## Models
| Model | Architecture | Size | Quant |
|-------|-------------|------|-------|
| Qwen3.6-35B-A3B | MoE (3B active) | ~35 GB | Q8_0 |
| Qwen3.6-27B | Dense | ~27 GB | Q8_0 |
## Config
```
-c 131072, --cache-type-k q4_0, --cache-type-v q4_0
--flash-attn on, --spec-type draft-mtp, --spec-draft-n-max 3
--batch-size 512, --ubatch-size 128
```
## Results
### Two Separate Servers (Before)
**Both loaded, both generating:**
| Model | Gen Speed | Notes |
|-------|-----------|-------|
| 35B MoE | 23 t/s | -75% from solo potential |
| 27B dense | 12 t/s | Already bandwidth-saturated |
**Solo (only one running):**
| Model | Gen Speed | Bandwidth Efficiency |
|-------|-----------|---------------------|
| 35B MoE | 94 t/s | 3.2 GB/step × 94 = 49% of 614 GB/s |
| 27B dense | 12 t/s | 30.4 GB/step × 12 = 59% of 614 GB/s |
### Router Mode (After)
**--models-max 1, auto-swap on request:**
| Model | Gen Speed (cold) | Gen Speed (warm) | Gain vs Before |
|-------|-----------------|------------------|----------------|
| 35B MoE | 132 t/s | 148 t/s | **5.7x** |
| 27B dense | 38 t/s | 48 t/s | **3.0x** |
**Per-model thread tuning:**
| Model | Optimal Threads | Speed |
|-------|----------------|-------|
| 35B MoE (attention-compute bound) | 14 | 148 t/s |
| 27B dense (bandwidth bound) | 10 | 48 t/s |
### Model Switch Latency
| Transition | Time |
|-----------|------|
| 35B → 27B (cold start) | ~4s |
| 27B → 35B (reload) | ~3s |
The 3-4s delay is model weights loading from SSD to GPU memory via Metal. Smaller batch sizes and Q4_0 KV cache help minimize this.
## MTP Speculative Decoding
| Model | Acceptance Rate | Mean Draft Length | Effective Tokens/Step |
|-------|----------------|-------------------|----------------------|
| 35B MoE | 74% | 2.84 | 2.1 |
| 27B dense | 82% | 3.62 | 2.6 |
MTP (Multi-Token Prediction) uses the model's built-in speculative heads. Each forward pass produces 3 draft tokens; acceptance rate determines how many are kept. With ~2.5 tokens per forward pass, effective generation speed is ~2.5x the raw decode speed.
## Methodology
- Tested via llama.cpp `/v1/completions` API
- Fixed prompt: 7-11 tokens
- Generation: 200 tokens, temperature 0
- Timings from response body `timings.predicted_per_second`
- Warm = model already loaded in GPU, cold = first request after model load
configs/com.llama-server-router.plist
+39
View File
@@ -0,0 +1,39 @@
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>KeepAlive</key><true/>
<key>Label</key><string>com.example.llama-server-router</string>
<key>ProgramArguments</key>
<array>
<string>/opt/homebrew/bin/llama-server</string>
<string>--models-dir</string>
<string>/Users/username/.hermes/models-router</string>
<string>--models-max</string><string>1</string>
<string>--models-preset</string>
<string>/Users/username/.hermes/llama-models.ini</string>
<string>--host</string><string>0.0.0.0</string>
<string>--port</string><string>8085</string>
<string>-ngl</string><string>99</string>
<string>-c</string><string>131072</string>
<string>--mlock</string>
<string>--spec-type</string><string>draft-mtp</string>
<string>--spec-draft-n-max</string><string>3</string>
<string>--batch-size</string><string>512</string>
<string>--ubatch-size</string><string>128</string>
<string>--cache-type-k</string><string>q4_0</string>
<string>--cache-type-v</string><string>q4_0</string>
<string>--flash-attn</string><string>on</string>
<string>--metrics</string>
</array>
<key>RunAtLoad</key><true/>
<key>StandardOutPath</key>
<string>/tmp/llama-server-router.log</string>
<key>StandardErrorPath</key>
<string>/tmp/llama-server-router.log</string>
<key>ThrottleInterval</key>
<integer>5</integer>
<key>WorkingDirectory</key>
<string>/Users/username</string>
</dict>
</plist>
configs/llama-models.ini
+14
View File
@@ -0,0 +1,14 @@
[*]
flash-attn = on
cache-type-k = q4_0
cache-type-v = q4_0
batch-size = 512
ubatch-size = 128
spec-type = draft-mtp
spec-draft-n-max = 3
[model.Qwen3.6-35B-A3B-Q8_0]
threads = 14
[model.Qwen3.6-27B-Q8_0]
threads = 10
docs/01-problem.md
+47
View File
@@ -0,0 +1,47 @@
# The Problem: Two Models, One GPU
## Why Two Models?
Different model architectures excel at different tasks:
- **MoE (Mixture of Experts):** Fast generation (~3B active params), good for chat. Example: Qwen3.6-35B-A3B
- **Dense:** Better reasoning on coding tasks, all parameters active. Example: Qwen3.6-27B
Running both gives you the best of both worlds — but on a single GPU, they compete.
## The Naive Approach
The obvious solution: run two `llama-server` processes on different ports.
```bash
# Server 1: 35B chat model on :8085
llama-server -m qwen35b.gguf --port 8085 -ngl 99
# Server 2: 27B coding model on :8080
llama-server -m qwen27b.gguf --port 8080 -ngl 99
```
**Result:** Both models stay permanently loaded in GPU memory, permanently competing for memory bandwidth.
## Why It's Slow
On Apple Silicon's Unified Memory architecture, GPU and CPU share the same memory pool. When two processes both use Metal GPU acceleration:
1. **Memory bandwidth is shared** — Both models' weights (75 GB combined) compete for the ~614 GB/s memory bus
2. **GPU scheduler splits time** — macOS Metal driver context-switches between processes
3. **KV cache doubles the tax** — Both models maintain large KV caches for their context windows
The result: each model gets roughly **half** the GPU bandwidth it could achieve alone.
## Measured Impact
| Model | Solo Speed | With Other Loaded | Penalty |
|-------|-----------|-------------------|---------|
| 35B MoE | 94 t/s | 23 t/s | **-75%** |
| 27B dense | 12 t/s | 12 t/s | **~0%** (already bandwidth-saturated) |
The 27B barely notices because it's already reading 27 GB of weights per token — its 12 t/s saturates the available bandwidth regardless. The 35B MoE (3B active params) has huge headroom but loses most of it to contention.
## The Goal
Find a way to give each model **full GPU bandwidth** without having to manually kill and restart servers.
docs/02-initial-setup.md
+90
View File
@@ -0,0 +1,90 @@
# Step 1: Two Separate Servers
The initial setup for running both model servers as persistent macOS services via `launchd`.
## Launchd Plist for 35B Chat Model
```xml
<!-- ~/Library/LaunchAgents/com.jimmyg.llama-server-qwen35b.plist -->
<plist>
<dict>
<key>KeepAlive</key><true/>
<key>ProgramArguments</key>
<array>
<string>/opt/homebrew/bin/llama-server</string>
<string>-m</string>
<string>/Users/jimmyg/models/Qwen3.6-35B-A3B-Q8_0.gguf</string>
<string>--host</string><string>0.0.0.0</string>
<string>--port</string><string>8085</string>
<string>-ngl</string><string>99</string>
<string>-c</string><string>262144</string>
<string>--mlock</string>
</array>
<key>RunAtLoad</key><true/>
<key>StandardOutPath</key>
<string>/Users/jimmyg/.hermes/logs/llama-server-qwen35b.log</string>
<key>StandardErrorPath</key>
<string>/Users/jimmyg/.hermes/logs/llama-server-qwen35b.log</string>
</dict>
</plist>
```
## Launchd Plist for 27B Coding Model
```xml
<!-- ~/Library/LaunchAgents/com.jimmyg.llama-server.plist -->
<plist>
<dict>
<key>KeepAlive</key><true/>
<key>ProgramArguments</key>
<array>
<string>/opt/homebrew/bin/llama-server</string>
<string>-m</string>
<string>/Users/jimmyg/models/Qwen3.6-27B-Q8_0.gguf</string>
<string>--host</string><string>0.0.0.0</string>
<string>--port</string><string>8080</string>
<string>-ngl</string><string>99</string>
<string>-c</string><string>131072</string>
</array>
<key>RunAtLoad</key><true/>
</dict>
</plist>
```
## Initial Flags Explained
| Flag | Value | Purpose |
|------|-------|---------|
| `-ngl 99` | All layers on GPU | Full Metal acceleration |
| `-c` | 262K / 131K | Context window size |
| `--mlock` | - | Prevent model from being swapped |
| `--spec-type draft-mtp` | - | Multi-Token Prediction speculation |
## Loading the Services
```bash
launchctl load ~/Library/LaunchAgents/com.jimmyg.llama-server-qwen35b.plist
launchctl load ~/Library/LaunchAgents/com.jimmyg.llama-server.plist
```
## Hermes Agent Integration
Two custom providers in `~/.hermes/config.yaml`:
```yaml
custom_providers:
- base_url: http://localhost:8085/v1
model: /Users/jimmyg/models/Qwen3.6-35B-A3B-Q8_0.gguf
name: qwen35b
- base_url: http://localhost:8080/v1
model: Qwen3.6-27B-Q8_0
name: qwen27b
```
## Baseline Performance
At this point with default settings:
- 35B MoE: ~20 t/s
- 27B dense: ~12 t/s
Plenty of room for optimization.
docs/03-optimization-journey.md
+99
View File
@@ -0,0 +1,99 @@
# The Optimization Journey
## Overview
Starting from the baseline (two servers, ~20 t/s on 35B, ~12 t/s on 27B), each optimization improved speed by addressing specific bottlenecks.
## Optimization 1: KV Cache Quantization
**Problem:** Full-precision (f16) KV cache consumes massive RAM. For 128K context on the 35B model, the KV cache alone uses ~15 GB.
**Fix:** Switch to Q4_0 KV cache.
```bash
--cache-type-k q4_0 --cache-type-v q4_0
```
**Impact:** -50% KV cache memory. Negligible quality loss — KV cache is a caching layer, not model weights. The precision of cached attention states has minimal effect on output quality.
## Optimization 2: Context Window Tuning
**Problem:** The 35B used 262K context (way too much for chat). The 27B used 131K. Combined KV caches consumed ~32 GB.
**Fix:** Both models at 128K. Sufficient for agent sessions (typical usage tops out at ~65K tokens).
```bash
-c 131072
```
**Impact:** Saved ~16 GB RAM. Freed headroom for the rest of the system.
## Optimization 3: Flash Attention
**Problem:** Without flash attention, generation speed degrades as KV cache grows (O(n²) attention cost).
**Fix:** Enable flash attention.
```bash
--flash-attn on
```
**Impact:** Maintains generation speed even with large contexts. With FA, generation stays at ~20 t/s regardless of context size. Without it, speed drops to ~10 t/s when context reaches 20K+ tokens.
## Optimization 4: Batch Size Tuning
**Problem:** Default batch sizes (2048/512) are optimized for server throughput (many concurrent requests), not single-user generation latency.
**Fix:** Smaller batch sizes for single-user use.
```bash
--batch-size 512 --ubatch-size 128
```
**Impact:** +25-50% generation speed for dense models. The 27B jumped from 12 to 16 t/s. The 35B MoE was less affected (3B active params already efficient).
**Why:** During generation (1 token at a time), the batch size affects how the GPU schedules compute. Smaller batches reduce latency per decode step.
## Optimization 5: Thread Count Tuning
**Problem:** Auto-detected thread count uses all CPU cores (18), causing dispatch overhead without benefit.
**Fix:** Explicit thread count optimized per architecture.
```bash
-t 14 # General fallback
```
**Impact:** MoE models benefit from more threads (attention compute-heavy). Dense models benefit from fewer threads (bandwidth-bound, less dispatch overhead).
| Model | Optimal Threads | Speed |
|-------|----------------|-------|
| 35B MoE (3B active) | 14 | 132 t/s |
| 27B dense (27B active) | 10 | 48 t/s |
## Optimization 6: Speculative Decoding (MTP)
**Fix:** Increase speculation from 1 to 3.
```bash
--spec-draft-n-max 3
```
**Impact:** With 82-93% MTP acceptance rate, each forward pass produces ~2.5-3 tokens instead of 1. This effectively doubles generation speed for free.
**Key insight:** Qwen models have built-in Multi-Token Prediction heads. The `--spec-type draft-mtp` flag uses these native heads rather than a separate draft model, adding negligible overhead.
## The Big One: Router Mode
The above optimizations improved speed, but the fundamental problem remained: **two processes competing for GPU bandwidth.**
**Fix:** Replace two `llama-server` processes with one router-mode server.
```bash
llama-server \
--models-dir ~/.hermes/models-router \
--models-max 1 \
--models-preset ~/.hermes/llama-models.ini
```
**Impact:** +5.7x on 35B, +3.0x on 27B. See `06-router-mode.md` for full details.
docs/06-router-mode.md
+166
View File
@@ -0,0 +1,166 @@
# Router Mode Setup
## Architecture
Instead of running two separate `llama-server` processes, the router mode runs a single server that:
1. **Discovers models** in a directory (symlinks to actual GGUF files)
2. **Loads on demand** — only loads a model when a request comes in
3. **Evicts by LRU** — when `--models-max 1`, switches models atomically
```
┌─────────────────┐ ┌──────────────────┐
│ Chat Request │────▶│ llama-server │
│ 35B MoE │ │ Router Mode │
└─────────────────┘ │ :8085 │
│ │
┌─────────────────┐ │ ┌────────────┐ │
│ Coding Subagent │────▶│ │ 35B MoE │ │
│ 27B dense │ │ │ (loaded) │ │
└─────────────────┘ │ └────────────┘ │
│ │
│ ┌────────────┐ │
│ │ 27B dense │ │
│ │ (evicted) │ │
│ └────────────┘ │
└──────────────────┘
```
## Models Directory
Create a directory with symlinks to your GGUF files:
```bash
mkdir -p ~/.hermes/models-router
ln -sf /path/to/Qwen3.6-35B-A3B-Q8_0.gguf ~/.hermes/models-router/
ln -sf /path/to/Qwen3.6-27B-Q8_0.gguf ~/.hermes/models-router/
```
The server uses the GGUF filename (without extension) as the model ID.
## Launch Command
```bash
llama-server \
--models-dir ~/.hermes/models-router \
--models-max 1 \
--models-preset ~/.hermes/llama-models.ini \
--host 0.0.0.0 --port 8085 \
-ngl 99 -c 131072 --mlock \
--spec-type draft-mtp --spec-draft-n-max 3 \
--batch-size 512 --ubatch-size 128 \
--cache-type-k q4_0 --cache-type-v q4_0 \
--flash-attn on \
--metrics
```
## Per-Model INI Preset
Model-specific settings (like thread count) go in the INI file:
```ini
[*]
flash-attn = on
cache-type-k = q4_0
cache-type-v = q4_0
batch-size = 512
ubatch-size = 128
spec-type = draft-mtp
spec-draft-n-max = 3
[model.Qwen3.6-35B-A3B-Q8_0]
threads = 14
[model.Qwen3.6-27B-Q8_0]
threads = 10
```
The `[*]` section applies to all models. Model-specific sections override for individual models.
## Launchd Service (Persistent)
Create a launchd plist for automatic startup:
```xml
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>KeepAlive</key><true/>
<key>Label</key><string>com.jimmyg.llama-server-router</string>
<key>ProgramArguments</key>
<array>
<string>/opt/homebrew/bin/llama-server</string>
<string>--models-dir</string>
<string>/Users/jimmyg/.hermes/models-router</string>
<string>--models-max</string><string>1</string>
<string>--models-preset</string>
<string>/Users/jimmyg/.hermes/llama-models.ini</string>
<string>--host</string><string>0.0.0.0</string>
<string>--port</string><string>8085</string>
<string>-ngl</string><string>99</string>
<string>-c</string><string>131072</string>
<string>--mlock</string>
<string>--spec-type</string><string>draft-mtp</string>
<string>--spec-draft-n-max</string><string>3</string>
<string>--batch-size</string><string>512</string>
<string>--ubatch-size</string><string>128</string>
<string>--cache-type-k</string><string>q4_0</string>
<string>--cache-type-v</string><string>q4_0</string>
<string>--flash-attn</string><string>on</string>
<string>--metrics</string>
</array>
<key>RunAtLoad</key><true/>
<key>StandardOutPath</key>
<string>/Users/jimmyg/.hermes/logs/llama-server-router.log</string>
<key>StandardErrorPath</key>
<string>/Users/jimmyg/.hermes/logs/llama-server-router.log</string>
</dict>
</plist>
```
Load it:
```bash
launchctl load ~/Library/LaunchAgents/com.jimmyg.llama-server-router.plist
```
## Hermes Agent Config
Both providers now point to the same server:
```yaml
custom_providers:
- base_url: http://localhost:8085/v1
model: Qwen3.6-35B-A3B-Q8_0
name: qwen35b
- base_url: http://localhost:8085/v1
model: Qwen3.6-27B-Q8_0
name: qwen27b
```
## API Usage
Send requests with the model name (GGUF filename without extension):
```bash
# 35B chat
curl -X POST http://localhost:8085/v1/completions \
-d '{"model":"Qwen3.6-35B-A3B-Q8_0","prompt":"Hello"}'
# 27B coding
curl -X POST http://localhost:8085/v1/completions \
-d '{"model":"Qwen3.6-27B-Q8_0","prompt":"def quicksort:"}'
```
## Monitoring
The `--metrics` flag exposes Prometheus-formatted metrics:
```bash
curl http://localhost:8085/metrics | grep predicted_tokens_seconds
```
## Known Limitation
`--slot-save-path` (KV cache persistence) does not work with router mode in current versions. Child processes are killed on eviction, so KV cache is lost. This is a feature gap, not a bug — router mode wasn't designed for context persistence across model switches.