Gan, Jimmy
30 lines
976 B
Markdown
30 lines
976 B
Markdown
# local-agent
|
|
|
|
Optimization guide for running multiple LLMs on a single Apple Silicon GPU using llama.cpp router mode.
|
|
|
|
## Key Configs
|
|
|
|
- **Router:** `~/.hermes/models-router/` with symlinks, managed via launchd
|
|
- **Plist:** `~/Library/LaunchAgents/com.jimmyg.llama-server-router.plist`
|
|
- **INI:** `~/.hermes/llama-models.ini` (per-model threads: 35B t=14, 27B t=10)
|
|
- **Old plists** disabled at `*.plist.disabled`
|
|
- **Hermes providers** both point to `http://localhost:8085/v1`
|
|
|
|
## Speeds
|
|
|
|
| Model | Router Mode |
|
|
|-------|-------------|
|
|
| 35B MoE | 132 t/s |
|
|
| 27B dense | 48 t/s |
|
|
|
|
Model switch delay: ~3-4s. Slot-saving doesn't work with router mode.
|
|
|
|
## Key Learnings
|
|
|
|
- Don't `kill -9` launchd services — use `launchctl unload/load`
|
|
- Batch: `512/128` not `2048/512` for single-user speed
|
|
- Threads: MoE → 14, dense → 10
|
|
- KV cache: Q4_0 > Q8_0 for bandwidth
|
|
- MTP speculation: `--spec-draft-n-max 3` (~2.5x speedup)
|
|
- Full lessons in `docs/04-lessons-learned.md`
|