jimmy/local-agent
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Initial: llama.cpp router mode optimization guide for Apple Silicon

Browse Source 
Full documentation of optimizing two LLMs on a single M5 Max GPU:
- KV cache quantization (Q4_0)
- Flash attention and batch tuning
- Router mode with --models-max 1
- Per-model thread optimization via INI presets
- Before/after benchmarks (12→48 t/s on 27B, 23→132 t/s on 35B)
This commit is contained in:
Gan, Jimmy
2026-07-10 02:27:36 +08:00
commit 3994e29cb0
8 changed files with 630 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/02-initial-setup.md
+90
View File
@@ -0,0 +1,90 @@
# Step 1: Two Separate Servers
The initial setup for running both model servers as persistent macOS services via `launchd`.
## Launchd Plist for 35B Chat Model
```xml
<!-- ~/Library/LaunchAgents/com.jimmyg.llama-server-qwen35b.plist -->
<plist>
<dict>
<key>KeepAlive</key><true/>
<key>ProgramArguments</key>
<array>
<string>/opt/homebrew/bin/llama-server</string>
<string>-m</string>
<string>/Users/jimmyg/models/Qwen3.6-35B-A3B-Q8_0.gguf</string>
<string>--host</string><string>0.0.0.0</string>
<string>--port</string><string>8085</string>
<string>-ngl</string><string>99</string>
<string>-c</string><string>262144</string>
<string>--mlock</string>
</array>
<key>RunAtLoad</key><true/>
<key>StandardOutPath</key>
<string>/Users/jimmyg/.hermes/logs/llama-server-qwen35b.log</string>
<key>StandardErrorPath</key>
<string>/Users/jimmyg/.hermes/logs/llama-server-qwen35b.log</string>
</dict>
</plist>
```
## Launchd Plist for 27B Coding Model
```xml
<!-- ~/Library/LaunchAgents/com.jimmyg.llama-server.plist -->
<plist>
<dict>
<key>KeepAlive</key><true/>
<key>ProgramArguments</key>
<array>
<string>/opt/homebrew/bin/llama-server</string>
<string>-m</string>
<string>/Users/jimmyg/models/Qwen3.6-27B-Q8_0.gguf</string>
<string>--host</string><string>0.0.0.0</string>
<string>--port</string><string>8080</string>
<string>-ngl</string><string>99</string>
<string>-c</string><string>131072</string>
</array>
<key>RunAtLoad</key><true/>
</dict>
</plist>
```
## Initial Flags Explained
| Flag | Value | Purpose |
|------|-------|---------|
| `-ngl 99` | All layers on GPU | Full Metal acceleration |
| `-c` | 262K / 131K | Context window size |
| `--mlock` | - | Prevent model from being swapped |
| `--spec-type draft-mtp` | - | Multi-Token Prediction speculation |
## Loading the Services
```bash
launchctl load ~/Library/LaunchAgents/com.jimmyg.llama-server-qwen35b.plist
launchctl load ~/Library/LaunchAgents/com.jimmyg.llama-server.plist
```
## Hermes Agent Integration
Two custom providers in `~/.hermes/config.yaml`:
```yaml
custom_providers:
- base_url: http://localhost:8085/v1
model: /Users/jimmyg/models/Qwen3.6-35B-A3B-Q8_0.gguf
name: qwen35b
- base_url: http://localhost:8080/v1
model: Qwen3.6-27B-Q8_0
name: qwen27b
```
## Baseline Performance
At this point with default settings:
- 35B MoE: ~20 t/s
- 27B dense: ~12 t/s
Plenty of room for optimization.