Gan, Jimmy
3994e29cb0
Full documentation of optimizing two LLMs on a single M5 Max GPU: - KV cache quantization (Q4_0) - Flash attention and batch tuning - Router mode with --models-max 1 - Per-model thread optimization via INI presets - Before/after benchmarks (12→48 t/s on 27B, 23→132 t/s on 35B)
167 lines
5.2 KiB
Markdown
167 lines
5.2 KiB
Markdown
# Router Mode Setup
|
|
|
|
## Architecture
|
|
|
|
Instead of running two separate `llama-server` processes, the router mode runs a single server that:
|
|
|
|
1. **Discovers models** in a directory (symlinks to actual GGUF files)
|
|
2. **Loads on demand** — only loads a model when a request comes in
|
|
3. **Evicts by LRU** — when `--models-max 1`, switches models atomically
|
|
|
|
```
|
|
┌─────────────────┐ ┌──────────────────┐
|
|
│ Chat Request │────▶│ llama-server │
|
|
│ 35B MoE │ │ Router Mode │
|
|
└─────────────────┘ │ :8085 │
|
|
│ │
|
|
┌─────────────────┐ │ ┌────────────┐ │
|
|
│ Coding Subagent │────▶│ │ 35B MoE │ │
|
|
│ 27B dense │ │ │ (loaded) │ │
|
|
└─────────────────┘ │ └────────────┘ │
|
|
│ │
|
|
│ ┌────────────┐ │
|
|
│ │ 27B dense │ │
|
|
│ │ (evicted) │ │
|
|
│ └────────────┘ │
|
|
└──────────────────┘
|
|
```
|
|
|
|
## Models Directory
|
|
|
|
Create a directory with symlinks to your GGUF files:
|
|
|
|
```bash
|
|
mkdir -p ~/.hermes/models-router
|
|
ln -sf /path/to/Qwen3.6-35B-A3B-Q8_0.gguf ~/.hermes/models-router/
|
|
ln -sf /path/to/Qwen3.6-27B-Q8_0.gguf ~/.hermes/models-router/
|
|
```
|
|
|
|
The server uses the GGUF filename (without extension) as the model ID.
|
|
|
|
## Launch Command
|
|
|
|
```bash
|
|
llama-server \
|
|
--models-dir ~/.hermes/models-router \
|
|
--models-max 1 \
|
|
--models-preset ~/.hermes/llama-models.ini \
|
|
--host 0.0.0.0 --port 8085 \
|
|
-ngl 99 -c 131072 --mlock \
|
|
--spec-type draft-mtp --spec-draft-n-max 3 \
|
|
--batch-size 512 --ubatch-size 128 \
|
|
--cache-type-k q4_0 --cache-type-v q4_0 \
|
|
--flash-attn on \
|
|
--metrics
|
|
```
|
|
|
|
## Per-Model INI Preset
|
|
|
|
Model-specific settings (like thread count) go in the INI file:
|
|
|
|
```ini
|
|
[*]
|
|
flash-attn = on
|
|
cache-type-k = q4_0
|
|
cache-type-v = q4_0
|
|
batch-size = 512
|
|
ubatch-size = 128
|
|
spec-type = draft-mtp
|
|
spec-draft-n-max = 3
|
|
|
|
[model.Qwen3.6-35B-A3B-Q8_0]
|
|
threads = 14
|
|
|
|
[model.Qwen3.6-27B-Q8_0]
|
|
threads = 10
|
|
```
|
|
|
|
The `[*]` section applies to all models. Model-specific sections override for individual models.
|
|
|
|
## Launchd Service (Persistent)
|
|
|
|
Create a launchd plist for automatic startup:
|
|
|
|
```xml
|
|
<?xml version="1.0" encoding="UTF-8"?>
|
|
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
|
|
<plist version="1.0">
|
|
<dict>
|
|
<key>KeepAlive</key><true/>
|
|
<key>Label</key><string>com.jimmyg.llama-server-router</string>
|
|
<key>ProgramArguments</key>
|
|
<array>
|
|
<string>/opt/homebrew/bin/llama-server</string>
|
|
<string>--models-dir</string>
|
|
<string>/Users/jimmyg/.hermes/models-router</string>
|
|
<string>--models-max</string><string>1</string>
|
|
<string>--models-preset</string>
|
|
<string>/Users/jimmyg/.hermes/llama-models.ini</string>
|
|
<string>--host</string><string>0.0.0.0</string>
|
|
<string>--port</string><string>8085</string>
|
|
<string>-ngl</string><string>99</string>
|
|
<string>-c</string><string>131072</string>
|
|
<string>--mlock</string>
|
|
<string>--spec-type</string><string>draft-mtp</string>
|
|
<string>--spec-draft-n-max</string><string>3</string>
|
|
<string>--batch-size</string><string>512</string>
|
|
<string>--ubatch-size</string><string>128</string>
|
|
<string>--cache-type-k</string><string>q4_0</string>
|
|
<string>--cache-type-v</string><string>q4_0</string>
|
|
<string>--flash-attn</string><string>on</string>
|
|
<string>--metrics</string>
|
|
</array>
|
|
<key>RunAtLoad</key><true/>
|
|
<key>StandardOutPath</key>
|
|
<string>/Users/jimmyg/.hermes/logs/llama-server-router.log</string>
|
|
<key>StandardErrorPath</key>
|
|
<string>/Users/jimmyg/.hermes/logs/llama-server-router.log</string>
|
|
</dict>
|
|
</plist>
|
|
```
|
|
|
|
Load it:
|
|
|
|
```bash
|
|
launchctl load ~/Library/LaunchAgents/com.jimmyg.llama-server-router.plist
|
|
```
|
|
|
|
## Hermes Agent Config
|
|
|
|
Both providers now point to the same server:
|
|
|
|
```yaml
|
|
custom_providers:
|
|
- base_url: http://localhost:8085/v1
|
|
model: Qwen3.6-35B-A3B-Q8_0
|
|
name: qwen35b
|
|
- base_url: http://localhost:8085/v1
|
|
model: Qwen3.6-27B-Q8_0
|
|
name: qwen27b
|
|
```
|
|
|
|
## API Usage
|
|
|
|
Send requests with the model name (GGUF filename without extension):
|
|
|
|
```bash
|
|
# 35B chat
|
|
curl -X POST http://localhost:8085/v1/completions \
|
|
-d '{"model":"Qwen3.6-35B-A3B-Q8_0","prompt":"Hello"}'
|
|
|
|
# 27B coding
|
|
curl -X POST http://localhost:8085/v1/completions \
|
|
-d '{"model":"Qwen3.6-27B-Q8_0","prompt":"def quicksort:"}'
|
|
```
|
|
|
|
## Monitoring
|
|
|
|
The `--metrics` flag exposes Prometheus-formatted metrics:
|
|
|
|
```bash
|
|
curl http://localhost:8085/metrics | grep predicted_tokens_seconds
|
|
```
|
|
|
|
## Known Limitation
|
|
|
|
`--slot-save-path` (KV cache persistence) does not work with router mode in current versions. Child processes are killed on eviction, so KV cache is lost. This is a feature gap, not a bug — router mode wasn't designed for context persistence across model switches.
|