Gan, Jimmy
cc10540e9e
Covers: launchd management, Dell hub power, batch tuning, thread count per arch, KV cache quantization, MTP speculation, flash attention, router mode vs two-servers, context sizing, swap monitoring.
161 lines
6.4 KiB
Markdown
161 lines
6.4 KiB
Markdown
# Lessons Learned
|
|
|
|
The discoveries and hard-won insights from optimizing two LLMs on a single M5 Max GPU.
|
|
|
|
## 1. Launchd Management — Don't Kill, Unload
|
|
|
|
**The hard way:** Running `kill -9` on a launchd-managed process is futile. `KeepAlive=true` restarts it instantly with the old config.
|
|
|
|
**The right way:**
|
|
```bash
|
|
# Before editing the plist
|
|
launchctl unload ~/Library/LaunchAgents/com.example.plist
|
|
|
|
# After editing the plist
|
|
launchctl load ~/Library/LaunchAgents/com.example.plist
|
|
```
|
|
|
|
**Why it matters:** If you kill the process without unloading, launchd respawns it before you finish editing the plist. You end up fighting a zombie that keeps coming back.
|
|
|
|
**Tell-tale sign:** You see "couldn't bind HTTP server socket" when starting a replacement — the old process auto-restarted on the same port.
|
|
|
|
## 2. Dell Hub Power — 90W Is Not Enough
|
|
|
|
Your MacBook Pro wants **140W**. The Dell U2723 monitor's USB-C port delivers 63-90W, depending on the connection state.
|
|
|
|
**Symptoms of undervolt:**
|
|
- Battery slowly drains even while "plugged in"
|
|
- "Not charging" status despite AC power detected
|
|
- System runs fine but can't top off the battery under GPU load
|
|
|
|
**The fix is simple:** Plug the 140W adapter directly into the Mac. Use the Dell hub only for display and peripherals. Two cables.
|
|
|
|
**Measured power delivery:**
|
|
| Source | Wattage | Can charge? |
|
|
|--------|---------|-------------|
|
|
| Apple 140W direct | 140W | Yes — full speed |
|
|
| Dell U2723 USB-C | 63-90W | Unreliable — drains under GPU load |
|
|
|
|
## 3. Batch Size — Smaller Is Faster for Single-User
|
|
|
|
**Counter-intuitive finding:** `--batch-size 2048 --ubatch-size 512` (default-adjacent for throughput servers) **hurts** single-user generation speed.
|
|
|
|
**Why:** During autoregressive generation, you produce 1 token at a time. Large batch sizes add overhead in GPU scheduling and memory management for zero benefit.
|
|
|
|
**The fix:**
|
|
```bash
|
|
--batch-size 512 --ubatch-size 128
|
|
```
|
|
|
|
**Impact:**
|
|
| Model | 2048/512 | 512/128 | Gain |
|
|
|-------|----------|---------|------|
|
|
| 35B MoE | ~23 t/s | ~25 t/s | ~10% |
|
|
| 27B dense | ~12 t/s | ~16 t/s | **+33%** |
|
|
|
|
Dense models benefit more because they're bandwidth-bound — less dispatch overhead means more time reading weights.
|
|
|
|
## 4. Thread Count — Per-Architecture Tuning
|
|
|
|
**Myth:** More CPU threads = faster generation.
|
|
**Reality:** Optimal thread count depends on model architecture.
|
|
|
|
**Testing on M5 Max (18 CPU cores):**
|
|
| Threads | 35B MoE | 27B dense |
|
|
|---------|---------|-----------|
|
|
| 8 | — | 45.5 t/s |
|
|
| 10 | 128.6 t/s | **48.0 t/s** ✅ |
|
|
| 14 | **132 t/s** ✅ | 43 t/s |
|
|
| Auto (18) | ~120 t/s | ~38 t/s |
|
|
|
|
**Rule of thumb:**
|
|
- **MoE models** (low active params, compute-bound attention) → t=14
|
|
- **Dense models** (high active params, bandwidth-bound) → t=10
|
|
- Too many threads adds dispatch overhead without bandwidth benefit
|
|
|
|
**Implementation:** Use `--models-preset` with INI file for per-model thread counts.
|
|
|
|
## 5. KV Cache Quantization — Q4_0 Over Q8_0
|
|
|
|
**Lesson:** KV cache is a **caching layer**, not model weights. Its precision has negligible effect on output quality.
|
|
|
|
```bash
|
|
--cache-type-k q4_0 --cache-type-v q4_0
|
|
```
|
|
|
|
**Why Q4_0 > Q8_0:**
|
|
- Halves memory bandwidth per token (less data to move = faster)
|
|
- Negligible quality loss in cached attention states
|
|
- With router mode and one model loaded, RAM headroom is ample anyway
|
|
|
|
**What Q8_0 would cost:** 2x memory bandwidth for cached K/V reads during generation. No measurable quality benefit.
|
|
|
|
## 6. MTP Speculative Decoding — Free Speed
|
|
|
|
Qwen models have built-in Multi-Token Prediction heads. Enable them:
|
|
|
|
```bash
|
|
--spec-type draft-mtp --spec-draft-n-max 3
|
|
```
|
|
|
|
**Measured performance:**
|
|
|
|
| Model | Acceptance | Mean Length | Effective Speedup |
|
|
|-------|-----------|-------------|-------------------|
|
|
| 35B MoE | 74% | 2.84 | ~2.1x |
|
|
| 27B dense | 82% | 3.62 | ~2.6x |
|
|
|
|
Each forward pass produces 3 draft tokens. The acceptance rate determines how many are kept. At 80%+ acceptance, you get ~2.5 tokens per forward pass instead of 1.
|
|
|
|
**Why MTP is better than draft models:** No separate draft model to load. Qwen's MTP heads are part of the model weights — negligible extra compute.
|
|
|
|
## 7. Flash Attention — Critical for Long Contexts
|
|
|
|
Without `--flash-attn on`, generation speed degrades as the KV cache grows (O(n²) attention).
|
|
|
|
**Test results (35B MoE):**
|
|
| Context Size | No Flash Attn | With Flash Attn |
|
|
|-------------|---------------|-----------------|
|
|
| 100 tokens | ~22 t/s | ~22 t/s |
|
|
| 10K tokens | ~12 t/s | ~20 t/s |
|
|
| 30K tokens | ~9 t/s | ~18 t/s |
|
|
|
|
Flash attention's benefit scales with context length. For agent sessions that accumulate 20K+ tokens, it's essential.
|
|
|
|
## 8. Router Mode vs Two Servers — The Real Cost
|
|
|
|
**Two separate servers:** Both models permanently loaded. Even when idle, the second model consumes GPU scheduler time and memory bandwidth overhead.
|
|
|
|
**Router mode (--models-max 1):** Active model gets full bandwidth. Idle model is completely evicted from GPU memory.
|
|
|
|
**The real cost of keeping both loaded:**
|
|
| Model | Solo | With Other Loaded | Penalty |
|
|
|-------|------|-------------------|---------|
|
|
| 35B MoE | 94 t/s | 23 t/s | **-75%** |
|
|
| 27B dense | 12 t/s | 12 t/s | ~0% (already saturated) |
|
|
|
|
The 27B doesn't care because it's reading 27 GB/token — it saturates bandwidth regardless. The 35B MoE (3 GB/token) has massive headroom that disappears the moment it competes.
|
|
|
|
**Router mode model switch delay:** ~3-4 seconds. This is model weights loading from SSD to GPU memory — not context processing. `--slot-save-path` would help but doesn't work with router mode (child processes killed on eviction).
|
|
|
|
## 9. Context Window Sizing — 128K Is the Sweet Spot
|
|
|
|
**Why not 262K:** The KV cache for 262K on the 35B MoE consumed ~15 GB of RSS — a third of the model's memory footprint. For agent sessions, typical usage tops out at ~65K tokens.
|
|
|
|
**Why not 32K:** Agent sessions with tool calls, file reads, and long reasoning chains can exceed 32K. 65K was measured in active use.
|
|
|
|
**Verdict:** 128K gives 2x headroom over observed usage, saves ~8 GB RAM vs 262K.
|
|
|
|
## 10. Swap Monitoring — Know When You're at the Wall
|
|
|
|
A simple watchdog script catches memory creep before it slows everything:
|
|
|
|
```bash
|
|
# ~/.hermes/scripts/swap_watch.sh
|
|
# Quiet unless swap > 2GB
|
|
```
|
|
|
|
Set up as a cron job: every 30 minutes, no_agent mode, silent when healthy.
|
|
|
|
**Why:** You can have 32 GB free and still have 2 GB of stale swap that macOS hasn't purged. The monitor tells you if swap is growing (real pressure) or just leftover (harmless).
|