llama.cpp Router Mode on Apple Silicon

Single GPU, two models, zero contention — from 12 t/s to 132 t/s

A practical guide to running multiple LLMs on one Apple Silicon Mac using llama-server router mode, achieving full GPU bandwidth for each model with automatic LRU eviction.

The Problem

Running two llama-server processes on the same GPU causes severe bandwidth contention:

Two separate servers (before):
  35B MoE:  23 t/s  ← -75% from potential
  27B dense: 16 t/s  ← -63% from potential

Both models stay permanently loaded, permanently competing for GPU memory bandwidth.

The Solution

A single llama-server with --models-max 1 evicts the idle model from GPU memory, giving full bandwidth to whichever model is active:

Router mode (after):
  35B MoE:  132 t/s  ← 5.7x faster
  27B dense:  48 t/s  ← 3.0x faster

Quick Start

# 1. Create model symlinks
mkdir -p ~/.hermes/models-router
ln -sf /path/to/model1.gguf ~/.hermes/models-router/
ln -sf /path/to/model2.gguf ~/.hermes/models-router/

# 2. Create per-model INI preset
cat > ~/.hermes/llama-models.ini << 'EOF'
[*]
flash-attn = on
cache-type-k = q4_0
cache-type-v = q4_0
batch-size = 512
ubatch-size = 128
spec-type = draft-mtp
spec-draft-n-max = 3

[model.Model1]
threads = 14

[model.Model2]
threads = 10
EOF

# 3. Launch router
llama-server \
  --models-dir ~/.hermes/models-router \
  --models-max 1 \
  --models-preset ~/.hermes/llama-models.ini \
  --host 0.0.0.0 --port 8085 \
  -ngl 99 -c 131072 --mlock \
  --metrics

Speed Results (M5 Max 40-core, 128GB)

Model	Two Servers	Router Mode	Gain
Qwen3.6-35B-A3B-Q8_0 (MoE)	23 t/s	132 t/s	5.7x
Qwen3.6-27B-Q8_0 (dense)	16 t/s	48 t/s	3.0x

Trade-off: ~3-4s model load time when switching models.

Hardware

• Chip: Apple M5 Max (40 GPU cores, 614 GB/s bandwidth)
• RAM: 128 GB Unified Memory
• Software: llama.cpp b9910+, launchd, Hermes Agent

Contents

• docs/ — Full optimization journey with benchmarks at each step
  • 01-problem.md — Why two models on one GPU is slow
  • 02-initial-setup.md — The naive two-server setup
  • 03-optimization-journey.md — Every optimization we tried
  • 04-lessons-learned.md — Hard-won insights and discoveries
  • 06-router-mode.md — Router mode setup and config
• configs/ — Launchd plist, INI preset, utility scripts
• benchmarks/ — Before/after speed comparison data

License

MIT — use it, share it, productize it.