local-agent/docs/04-lessons-learned.md

Lessons Learned

The discoveries and hard-won insights from optimizing two LLMs on a single M5 Max GPU.

1. Launchd Management — Don't Kill, Unload

The hard way: Running kill -9 on a launchd-managed process is futile. KeepAlive=true restarts it instantly with the old config.

The right way:

# Before editing the plist
launchctl unload ~/Library/LaunchAgents/com.example.plist

# After editing the plist
launchctl load ~/Library/LaunchAgents/com.example.plist

Why it matters: If you kill the process without unloading, launchd respawns it before you finish editing the plist. You end up fighting a zombie that keeps coming back.

Tell-tale sign: You see "couldn't bind HTTP server socket" when starting a replacement — the old process auto-restarted on the same port.

2. Dell Hub Power — 90W Is Not Enough

Your MacBook Pro wants 140W. The Dell U2723 monitor's USB-C port delivers 63-90W, depending on the connection state.

Symptoms of undervolt:

• Battery slowly drains even while "plugged in"
• "Not charging" status despite AC power detected
• System runs fine but can't top off the battery under GPU load

The fix is simple: Plug the 140W adapter directly into the Mac. Use the Dell hub only for display and peripherals. Two cables.

Measured power delivery:

Source	Wattage	Can charge?
Apple 140W direct	140W	Yes — full speed
Dell U2723 USB-C	63-90W	Unreliable — drains under GPU load

3. Batch Size — Smaller Is Faster for Single-User

Counter-intuitive finding: --batch-size 2048 --ubatch-size 512 (default-adjacent for throughput servers) hurts single-user generation speed.

Why: During autoregressive generation, you produce 1 token at a time. Large batch sizes add overhead in GPU scheduling and memory management for zero benefit.

The fix:

--batch-size 512 --ubatch-size 128

Impact:

Model	2048/512	512/128	Gain
35B MoE	~23 t/s	~25 t/s	~10%
27B dense	~12 t/s	~16 t/s	+33%

Dense models benefit more because they're bandwidth-bound — less dispatch overhead means more time reading weights.

4. Thread Count — Per-Architecture Tuning

Myth: More CPU threads = faster generation. Reality: Optimal thread count depends on model architecture.

Testing on M5 Max (18 CPU cores):

Threads	35B MoE	27B dense
8	—	45.5 t/s
10	128.6 t/s	48.0 t/s ✅
14	132 t/s ✅	43 t/s
Auto (18)	~120 t/s	~38 t/s

Rule of thumb:

• MoE models (low active params, compute-bound attention) → t=14
• Dense models (high active params, bandwidth-bound) → t=10
• Too many threads adds dispatch overhead without bandwidth benefit

Implementation: Use --models-preset with INI file for per-model thread counts.

5. KV Cache Quantization — Q4_0 Over Q8_0

Lesson: KV cache is a caching layer, not model weights. Its precision has negligible effect on output quality.

--cache-type-k q4_0 --cache-type-v q4_0

Why Q4_0 > Q8_0:

• Halves memory bandwidth per token (less data to move = faster)
• Negligible quality loss in cached attention states
• With router mode and one model loaded, RAM headroom is ample anyway

What Q8_0 would cost: 2x memory bandwidth for cached K/V reads during generation. No measurable quality benefit.

6. MTP Speculative Decoding — Free Speed

Qwen models have built-in Multi-Token Prediction heads. Enable them:

--spec-type draft-mtp --spec-draft-n-max 3

Measured performance:

Model	Acceptance	Mean Length	Effective Speedup
35B MoE	74%	2.84	~2.1x
27B dense	82%	3.62	~2.6x

Each forward pass produces 3 draft tokens. The acceptance rate determines how many are kept. At 80%+ acceptance, you get ~2.5 tokens per forward pass instead of 1.

Why MTP is better than draft models: No separate draft model to load. Qwen's MTP heads are part of the model weights — negligible extra compute.

7. Flash Attention — Critical for Long Contexts

Without --flash-attn on, generation speed degrades as the KV cache grows (O(n²) attention).

Test results (35B MoE):

Context Size	No Flash Attn	With Flash Attn
100 tokens	~22 t/s	~22 t/s
10K tokens	~12 t/s	~20 t/s
30K tokens	~9 t/s	~18 t/s

Flash attention's benefit scales with context length. For agent sessions that accumulate 20K+ tokens, it's essential.

8. Router Mode vs Two Servers — The Real Cost

Two separate servers: Both models permanently loaded. Even when idle, the second model consumes GPU scheduler time and memory bandwidth overhead.

Router mode (--models-max 1): Active model gets full bandwidth. Idle model is completely evicted from GPU memory.

The real cost of keeping both loaded:

Model	Solo	With Other Loaded	Penalty
35B MoE	94 t/s	23 t/s	-75%
27B dense	12 t/s	12 t/s	~0% (already saturated)

The 27B doesn't care because it's reading 27 GB/token — it saturates bandwidth regardless. The 35B MoE (3 GB/token) has massive headroom that disappears the moment it competes.

Router mode model switch delay: ~3-4 seconds. This is model weights loading from SSD to GPU memory — not context processing. --slot-save-path would help but doesn't work with router mode (child processes killed on eviction).

9. Context Window Sizing — 128K Is the Sweet Spot

Why not 262K: The KV cache for 262K on the 35B MoE consumed ~15 GB of RSS — a third of the model's memory footprint. For agent sessions, typical usage tops out at ~65K tokens.

Why not 32K: Agent sessions with tool calls, file reads, and long reasoning chains can exceed 32K. 65K was measured in active use.

Verdict: 128K gives 2x headroom over observed usage, saves ~8 GB RAM vs 262K.

10. Swap Monitoring — Know When You're at the Wall

A simple watchdog script catches memory creep before it slows everything:

# ~/.hermes/scripts/swap_watch.sh
# Quiet unless swap > 2GB

Set up as a cron job: every 30 minutes, no_agent mode, silent when healthy.

Why: You can have 32 GB free and still have 2 GB of stale swap that macOS hasn't purged. The monitor tells you if swap is growing (real pressure) or just leftover (harmless).