# local-agent

Optimization guide for running multiple LLMs on a single Apple Silicon GPU using llama.cpp router mode.

## Key Configs

- **Router:** `~/.hermes/models-router/` with symlinks, managed via launchd
- **Plist:** `~/Library/LaunchAgents/com.jimmyg.llama-server-router.plist`
- **INI:** `~/.hermes/llama-models.ini` (per-model threads: 35B t=14, 27B t=10)
- **Old plists** disabled at `*.plist.disabled`
- **Hermes providers** both point to `http://localhost:8085/v1`

## Speeds

| Model | Router Mode |
|-------|-------------|
| 35B MoE | 132 t/s |
| 27B dense | 48 t/s |

Model switch delay: ~3-4s. Slot-saving doesn't work with router mode.

## Key Learnings

- Don't `kill -9` launchd services — use `launchctl unload/load`
- Batch: `512/128` not `2048/512` for single-user speed
- Threads: MoE → 14, dense → 10
- KV cache: Q4_0 > Q8_0 for bandwidth
- MTP speculation: `--spec-draft-n-max 3` (~2.5x speedup)
- Full lessons in `docs/04-lessons-learned.md`