local-agent/CLAUDE.md

local-agent

Optimization guide for running multiple LLMs on a single Apple Silicon GPU using llama.cpp router mode.

Key Configs

• Router: ~/.hermes/models-router/ with symlinks, managed via launchd
• Plist: ~/Library/LaunchAgents/com.jimmyg.llama-server-router.plist
• INI: ~/.hermes/llama-models.ini (per-model threads: 35B t=14, 27B t=10)
• Old plists disabled at *.plist.disabled
• Hermes providers both point to http://localhost:8085/v1

Speeds

Model	Router Mode
35B MoE	132 t/s
27B dense	48 t/s

Model switch delay: ~3-4s. Slot-saving doesn't work with router mode.

Key Learnings

• Don't kill -9 launchd services — use launchctl unload/load
• Batch: 512/128 not 2048/512 for single-user speed
• Threads: MoE → 14, dense → 10
• KV cache: Q4_0 > Q8_0 for bandwidth
• MTP speculation: --spec-draft-n-max 3 (~2.5x speedup)
• Full lessons in docs/04-lessons-learned.md