Files

T

Gan, Jimmy 3994e29cb0 Initial: llama.cpp router mode optimization guide for Apple Silicon

Full documentation of optimizing two LLMs on a single M5 Max GPU:
- KV cache quantization (Q4_0)
- Flash attention and batch tuning
- Router mode with --models-max 1
- Per-model thread optimization via INI presets
- Before/after benchmarks (12→48 t/s on 27B, 23→132 t/s on 35B)

2026-07-10 02:27:36 +08:00

5.2 KiB

Raw Blame History

Router Mode Setup

Architecture

Instead of running two separate llama-server processes, the router mode runs a single server that:

Discovers models in a directory (symlinks to actual GGUF files)
Loads on demand — only loads a model when a request comes in
Evicts by LRU — when --models-max 1, switches models atomically

┌─────────────────┐     ┌──────────────────┐
│  Chat Request   │────▶│  llama-server    │
│  35B MoE        │     │  Router Mode     │
└─────────────────┘     │  :8085           │
                        │                  │
┌─────────────────┐     │  ┌────────────┐  │
│ Coding Subagent │────▶│  │  35B MoE   │  │
│  27B dense      │     │  │  (loaded)  │  │
└─────────────────┘     │  └────────────┘  │
                        │                  │
                        │  ┌────────────┐  │
                        │  │  27B dense │  │
                        │  │  (evicted) │  │
                        │  └────────────┘  │
                        └──────────────────┘

Models Directory

Create a directory with symlinks to your GGUF files:

mkdir -p ~/.hermes/models-router
ln -sf /path/to/Qwen3.6-35B-A3B-Q8_0.gguf ~/.hermes/models-router/
ln -sf /path/to/Qwen3.6-27B-Q8_0.gguf ~/.hermes/models-router/

The server uses the GGUF filename (without extension) as the model ID.

Launch Command

llama-server \
  --models-dir ~/.hermes/models-router \
  --models-max 1 \
  --models-preset ~/.hermes/llama-models.ini \
  --host 0.0.0.0 --port 8085 \
  -ngl 99 -c 131072 --mlock \
  --spec-type draft-mtp --spec-draft-n-max 3 \
  --batch-size 512 --ubatch-size 128 \
  --cache-type-k q4_0 --cache-type-v q4_0 \
  --flash-attn on \
  --metrics

Per-Model INI Preset

Model-specific settings (like thread count) go in the INI file:

[*]
flash-attn = on
cache-type-k = q4_0
cache-type-v = q4_0
batch-size = 512
ubatch-size = 128
spec-type = draft-mtp
spec-draft-n-max = 3

[model.Qwen3.6-35B-A3B-Q8_0]
threads = 14

[model.Qwen3.6-27B-Q8_0]
threads = 10

The [*] section applies to all models. Model-specific sections override for individual models.

Launchd Service (Persistent)

Create a launchd plist for automatic startup:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
  <key>KeepAlive</key><true/>
  <key>Label</key><string>com.jimmyg.llama-server-router</string>
  <key>ProgramArguments</key>
  <array>
    <string>/opt/homebrew/bin/llama-server</string>
    <string>--models-dir</string>
    <string>/Users/jimmyg/.hermes/models-router</string>
    <string>--models-max</string><string>1</string>
    <string>--models-preset</string>
    <string>/Users/jimmyg/.hermes/llama-models.ini</string>
    <string>--host</string><string>0.0.0.0</string>
    <string>--port</string><string>8085</string>
    <string>-ngl</string><string>99</string>
    <string>-c</string><string>131072</string>
    <string>--mlock</string>
    <string>--spec-type</string><string>draft-mtp</string>
    <string>--spec-draft-n-max</string><string>3</string>
    <string>--batch-size</string><string>512</string>
    <string>--ubatch-size</string><string>128</string>
    <string>--cache-type-k</string><string>q4_0</string>
    <string>--cache-type-v</string><string>q4_0</string>
    <string>--flash-attn</string><string>on</string>
    <string>--metrics</string>
  </array>
  <key>RunAtLoad</key><true/>
  <key>StandardOutPath</key>
  <string>/Users/jimmyg/.hermes/logs/llama-server-router.log</string>
  <key>StandardErrorPath</key>
  <string>/Users/jimmyg/.hermes/logs/llama-server-router.log</string>
</dict>
</plist>

Load it:

launchctl load ~/Library/LaunchAgents/com.jimmyg.llama-server-router.plist

Hermes Agent Config

Both providers now point to the same server:

custom_providers:
  - base_url: http://localhost:8085/v1
    model: Qwen3.6-35B-A3B-Q8_0
    name: qwen35b
  - base_url: http://localhost:8085/v1
    model: Qwen3.6-27B-Q8_0
    name: qwen27b

API Usage

Send requests with the model name (GGUF filename without extension):

# 35B chat
curl -X POST http://localhost:8085/v1/completions \
  -d '{"model":"Qwen3.6-35B-A3B-Q8_0","prompt":"Hello"}'

# 27B coding
curl -X POST http://localhost:8085/v1/completions \
  -d '{"model":"Qwen3.6-27B-Q8_0","prompt":"def quicksort:"}'

Monitoring

The --metrics flag exposes Prometheus-formatted metrics:

curl http://localhost:8085/metrics | grep predicted_tokens_seconds

Known Limitation

--slot-save-path (KV cache persistence) does not work with router mode in current versions. Child processes are killed on eviction, so KV cache is lost. This is a feature gap, not a bug — router mode wasn't designed for context persistence across model switches.

5.2 KiB Raw Blame History