# Router Mode Setup

## Architecture

Instead of running two separate `llama-server` processes, the router mode runs a single server that:

1. **Discovers models** in a directory (symlinks to actual GGUF files)
2. **Loads on demand** — only loads a model when a request comes in
3. **Evicts by LRU** — when `--models-max 1`, switches models atomically

```
┌─────────────────┐     ┌──────────────────┐
│  Chat Request   │────▶│  llama-server    │
│  35B MoE        │     │  Router Mode     │
└─────────────────┘     │  :8085           │
                        │                  │
┌─────────────────┐     │  ┌────────────┐  │
│ Coding Subagent │────▶│  │  35B MoE   │  │
│  27B dense      │     │  │  (loaded)  │  │
└─────────────────┘     │  └────────────┘  │
                        │                  │
                        │  ┌────────────┐  │
                        │  │  27B dense │  │
                        │  │  (evicted) │  │
                        │  └────────────┘  │
                        └──────────────────┘
```

## Models Directory

Create a directory with symlinks to your GGUF files:

```bash
mkdir -p ~/.hermes/models-router
ln -sf /path/to/Qwen3.6-35B-A3B-Q8_0.gguf ~/.hermes/models-router/
ln -sf /path/to/Qwen3.6-27B-Q8_0.gguf ~/.hermes/models-router/
```

The server uses the GGUF filename (without extension) as the model ID.

## Launch Command

```bash
llama-server \
  --models-dir ~/.hermes/models-router \
  --models-max 1 \
  --models-preset ~/.hermes/llama-models.ini \
  --host 0.0.0.0 --port 8085 \
  -ngl 99 -c 131072 --mlock \
  --spec-type draft-mtp --spec-draft-n-max 3 \
  --batch-size 512 --ubatch-size 128 \
  --cache-type-k q4_0 --cache-type-v q4_0 \
  --flash-attn on \
  --metrics
```

## Per-Model INI Preset

Model-specific settings (like thread count) go in the INI file:

```ini
[*]
flash-attn = on
cache-type-k = q4_0
cache-type-v = q4_0
batch-size = 512
ubatch-size = 128
spec-type = draft-mtp
spec-draft-n-max = 3

[model.Qwen3.6-35B-A3B-Q8_0]
threads = 14

[model.Qwen3.6-27B-Q8_0]
threads = 10
```

The `[*]` section applies to all models. Model-specific sections override for individual models.

## Launchd Service (Persistent)

Create a launchd plist for automatic startup:

```xml
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
  <key>KeepAlive</key><true/>
  <key>Label</key><string>com.jimmyg.llama-server-router</string>
  <key>ProgramArguments</key>
  <array>
    <string>/opt/homebrew/bin/llama-server</string>
    <string>--models-dir</string>
    <string>/Users/jimmyg/.hermes/models-router</string>
    <string>--models-max</string><string>1</string>
    <string>--models-preset</string>
    <string>/Users/jimmyg/.hermes/llama-models.ini</string>
    <string>--host</string><string>0.0.0.0</string>
    <string>--port</string><string>8085</string>
    <string>-ngl</string><string>99</string>
    <string>-c</string><string>131072</string>
    <string>--mlock</string>
    <string>--spec-type</string><string>draft-mtp</string>
    <string>--spec-draft-n-max</string><string>3</string>
    <string>--batch-size</string><string>512</string>
    <string>--ubatch-size</string><string>128</string>
    <string>--cache-type-k</string><string>q4_0</string>
    <string>--cache-type-v</string><string>q4_0</string>
    <string>--flash-attn</string><string>on</string>
    <string>--metrics</string>
  </array>
  <key>RunAtLoad</key><true/>
  <key>StandardOutPath</key>
  <string>/Users/jimmyg/.hermes/logs/llama-server-router.log</string>
  <key>StandardErrorPath</key>
  <string>/Users/jimmyg/.hermes/logs/llama-server-router.log</string>
</dict>
</plist>
```

Load it:

```bash
launchctl load ~/Library/LaunchAgents/com.jimmyg.llama-server-router.plist
```

## Hermes Agent Config

Both providers now point to the same server:

```yaml
custom_providers:
  - base_url: http://localhost:8085/v1
    model: Qwen3.6-35B-A3B-Q8_0
    name: qwen35b
  - base_url: http://localhost:8085/v1
    model: Qwen3.6-27B-Q8_0
    name: qwen27b
```

## API Usage

Send requests with the model name (GGUF filename without extension):

```bash
# 35B chat
curl -X POST http://localhost:8085/v1/completions \
  -d '{"model":"Qwen3.6-35B-A3B-Q8_0","prompt":"Hello"}'

# 27B coding
curl -X POST http://localhost:8085/v1/completions \
  -d '{"model":"Qwen3.6-27B-Q8_0","prompt":"def quicksort:"}'
```

## Monitoring

The `--metrics` flag exposes Prometheus-formatted metrics:

```bash
curl http://localhost:8085/metrics | grep predicted_tokens_seconds
```

## Known Limitation

`--slot-save-path` (KV cache persistence) does not work with router mode in current versions. Child processes are killed on eviction, so KV cache is lost. This is a feature gap, not a bug — router mode wasn't designed for context persistence across model switches.