Initial: llama.cpp router mode optimization guide for Apple Silicon
Full documentation of optimizing two LLMs on a single M5 Max GPU: - KV cache quantization (Q4_0) - Flash attention and batch tuning - Router mode with --models-max 1 - Per-model thread optimization via INI presets - Before/after benchmarks (12→48 t/s on 27B, 23→132 t/s on 35B)
This commit is contained in:
Gan, Jimmy
@@ -0,0 +1,166 @@
|
||||
# Router Mode Setup
|
||||
|
||||
## Architecture
|
||||
|
||||
Instead of running two separate `llama-server` processes, the router mode runs a single server that:
|
||||
|
||||
1. **Discovers models** in a directory (symlinks to actual GGUF files)
|
||||
2. **Loads on demand** — only loads a model when a request comes in
|
||||
3. **Evicts by LRU** — when `--models-max 1`, switches models atomically
|
||||
|
||||
```
|
||||
┌─────────────────┐ ┌──────────────────┐
|
||||
│ Chat Request │────▶│ llama-server │
|
||||
│ 35B MoE │ │ Router Mode │
|
||||
└─────────────────┘ │ :8085 │
|
||||
│ │
|
||||
┌─────────────────┐ │ ┌────────────┐ │
|
||||
│ Coding Subagent │────▶│ │ 35B MoE │ │
|
||||
│ 27B dense │ │ │ (loaded) │ │
|
||||
└─────────────────┘ │ └────────────┘ │
|
||||
│ │
|
||||
│ ┌────────────┐ │
|
||||
│ │ 27B dense │ │
|
||||
│ │ (evicted) │ │
|
||||
│ └────────────┘ │
|
||||
└──────────────────┘
|
||||
```
|
||||
|
||||
## Models Directory
|
||||
|
||||
Create a directory with symlinks to your GGUF files:
|
||||
|
||||
```bash
|
||||
mkdir -p ~/.hermes/models-router
|
||||
ln -sf /path/to/Qwen3.6-35B-A3B-Q8_0.gguf ~/.hermes/models-router/
|
||||
ln -sf /path/to/Qwen3.6-27B-Q8_0.gguf ~/.hermes/models-router/
|
||||
```
|
||||
|
||||
The server uses the GGUF filename (without extension) as the model ID.
|
||||
|
||||
## Launch Command
|
||||
|
||||
```bash
|
||||
llama-server \
|
||||
--models-dir ~/.hermes/models-router \
|
||||
--models-max 1 \
|
||||
--models-preset ~/.hermes/llama-models.ini \
|
||||
--host 0.0.0.0 --port 8085 \
|
||||
-ngl 99 -c 131072 --mlock \
|
||||
--spec-type draft-mtp --spec-draft-n-max 3 \
|
||||
--batch-size 512 --ubatch-size 128 \
|
||||
--cache-type-k q4_0 --cache-type-v q4_0 \
|
||||
--flash-attn on \
|
||||
--metrics
|
||||
```
|
||||
|
||||
## Per-Model INI Preset
|
||||
|
||||
Model-specific settings (like thread count) go in the INI file:
|
||||
|
||||
```ini
|
||||
[*]
|
||||
flash-attn = on
|
||||
cache-type-k = q4_0
|
||||
cache-type-v = q4_0
|
||||
batch-size = 512
|
||||
ubatch-size = 128
|
||||
spec-type = draft-mtp
|
||||
spec-draft-n-max = 3
|
||||
|
||||
[model.Qwen3.6-35B-A3B-Q8_0]
|
||||
threads = 14
|
||||
|
||||
[model.Qwen3.6-27B-Q8_0]
|
||||
threads = 10
|
||||
```
|
||||
|
||||
The `[*]` section applies to all models. Model-specific sections override for individual models.
|
||||
|
||||
## Launchd Service (Persistent)
|
||||
|
||||
Create a launchd plist for automatic startup:
|
||||
|
||||
```xml
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
|
||||
<plist version="1.0">
|
||||
<dict>
|
||||
<key>KeepAlive</key><true/>
|
||||
<key>Label</key><string>com.jimmyg.llama-server-router</string>
|
||||
<key>ProgramArguments</key>
|
||||
<array>
|
||||
<string>/opt/homebrew/bin/llama-server</string>
|
||||
<string>--models-dir</string>
|
||||
<string>/Users/jimmyg/.hermes/models-router</string>
|
||||
<string>--models-max</string><string>1</string>
|
||||
<string>--models-preset</string>
|
||||
<string>/Users/jimmyg/.hermes/llama-models.ini</string>
|
||||
<string>--host</string><string>0.0.0.0</string>
|
||||
<string>--port</string><string>8085</string>
|
||||
<string>-ngl</string><string>99</string>
|
||||
<string>-c</string><string>131072</string>
|
||||
<string>--mlock</string>
|
||||
<string>--spec-type</string><string>draft-mtp</string>
|
||||
<string>--spec-draft-n-max</string><string>3</string>
|
||||
<string>--batch-size</string><string>512</string>
|
||||
<string>--ubatch-size</string><string>128</string>
|
||||
<string>--cache-type-k</string><string>q4_0</string>
|
||||
<string>--cache-type-v</string><string>q4_0</string>
|
||||
<string>--flash-attn</string><string>on</string>
|
||||
<string>--metrics</string>
|
||||
</array>
|
||||
<key>RunAtLoad</key><true/>
|
||||
<key>StandardOutPath</key>
|
||||
<string>/Users/jimmyg/.hermes/logs/llama-server-router.log</string>
|
||||
<key>StandardErrorPath</key>
|
||||
<string>/Users/jimmyg/.hermes/logs/llama-server-router.log</string>
|
||||
</dict>
|
||||
</plist>
|
||||
```
|
||||
|
||||
Load it:
|
||||
|
||||
```bash
|
||||
launchctl load ~/Library/LaunchAgents/com.jimmyg.llama-server-router.plist
|
||||
```
|
||||
|
||||
## Hermes Agent Config
|
||||
|
||||
Both providers now point to the same server:
|
||||
|
||||
```yaml
|
||||
custom_providers:
|
||||
- base_url: http://localhost:8085/v1
|
||||
model: Qwen3.6-35B-A3B-Q8_0
|
||||
name: qwen35b
|
||||
- base_url: http://localhost:8085/v1
|
||||
model: Qwen3.6-27B-Q8_0
|
||||
name: qwen27b
|
||||
```
|
||||
|
||||
## API Usage
|
||||
|
||||
Send requests with the model name (GGUF filename without extension):
|
||||
|
||||
```bash
|
||||
# 35B chat
|
||||
curl -X POST http://localhost:8085/v1/completions \
|
||||
-d '{"model":"Qwen3.6-35B-A3B-Q8_0","prompt":"Hello"}'
|
||||
|
||||
# 27B coding
|
||||
curl -X POST http://localhost:8085/v1/completions \
|
||||
-d '{"model":"Qwen3.6-27B-Q8_0","prompt":"def quicksort:"}'
|
||||
```
|
||||
|
||||
## Monitoring
|
||||
|
||||
The `--metrics` flag exposes Prometheus-formatted metrics:
|
||||
|
||||
```bash
|
||||
curl http://localhost:8085/metrics | grep predicted_tokens_seconds
|
||||
```
|
||||
|
||||
## Known Limitation
|
||||
|
||||
`--slot-save-path` (KV cache persistence) does not work with router mode in current versions. Child processes are killed on eviction, so KV cache is lost. This is a feature gap, not a bug — router mode wasn't designed for context persistence across model switches.
|
||||
Reference in New Issue
Block a user