diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..0668efb --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,29 @@ +# local-agent + +Optimization guide for running multiple LLMs on a single Apple Silicon GPU using llama.cpp router mode. + +## Key Configs + +- **Router:** `~/.hermes/models-router/` with symlinks, managed via launchd +- **Plist:** `~/Library/LaunchAgents/com.jimmyg.llama-server-router.plist` +- **INI:** `~/.hermes/llama-models.ini` (per-model threads: 35B t=14, 27B t=10) +- **Old plists** disabled at `*.plist.disabled` +- **Hermes providers** both point to `http://localhost:8085/v1` + +## Speeds + +| Model | Router Mode | +|-------|-------------| +| 35B MoE | 132 t/s | +| 27B dense | 48 t/s | + +Model switch delay: ~3-4s. Slot-saving doesn't work with router mode. + +## Key Learnings + +- Don't `kill -9` launchd services — use `launchctl unload/load` +- Batch: `512/128` not `2048/512` for single-user speed +- Threads: MoE → 14, dense → 10 +- KV cache: Q4_0 > Q8_0 for bandwidth +- MTP speculation: `--spec-draft-n-max 3` (~2.5x speedup) +- Full lessons in `docs/04-lessons-learned.md`