Gan, Jimmy
3994e29cb0
Full documentation of optimizing two LLMs on a single M5 Max GPU: - KV cache quantization (Q4_0) - Flash attention and batch tuning - Router mode with --models-max 1 - Per-model thread optimization via INI presets - Before/after benchmarks (12→48 t/s on 27B, 23→132 t/s on 35B)
91 lines
2.4 KiB
Markdown
91 lines
2.4 KiB
Markdown
# Step 1: Two Separate Servers
|
|
|
|
The initial setup for running both model servers as persistent macOS services via `launchd`.
|
|
|
|
## Launchd Plist for 35B Chat Model
|
|
|
|
```xml
|
|
<!-- ~/Library/LaunchAgents/com.jimmyg.llama-server-qwen35b.plist -->
|
|
<plist>
|
|
<dict>
|
|
<key>KeepAlive</key><true/>
|
|
<key>ProgramArguments</key>
|
|
<array>
|
|
<string>/opt/homebrew/bin/llama-server</string>
|
|
<string>-m</string>
|
|
<string>/Users/jimmyg/models/Qwen3.6-35B-A3B-Q8_0.gguf</string>
|
|
<string>--host</string><string>0.0.0.0</string>
|
|
<string>--port</string><string>8085</string>
|
|
<string>-ngl</string><string>99</string>
|
|
<string>-c</string><string>262144</string>
|
|
<string>--mlock</string>
|
|
</array>
|
|
<key>RunAtLoad</key><true/>
|
|
<key>StandardOutPath</key>
|
|
<string>/Users/jimmyg/.hermes/logs/llama-server-qwen35b.log</string>
|
|
<key>StandardErrorPath</key>
|
|
<string>/Users/jimmyg/.hermes/logs/llama-server-qwen35b.log</string>
|
|
</dict>
|
|
</plist>
|
|
```
|
|
|
|
## Launchd Plist for 27B Coding Model
|
|
|
|
```xml
|
|
<!-- ~/Library/LaunchAgents/com.jimmyg.llama-server.plist -->
|
|
<plist>
|
|
<dict>
|
|
<key>KeepAlive</key><true/>
|
|
<key>ProgramArguments</key>
|
|
<array>
|
|
<string>/opt/homebrew/bin/llama-server</string>
|
|
<string>-m</string>
|
|
<string>/Users/jimmyg/models/Qwen3.6-27B-Q8_0.gguf</string>
|
|
<string>--host</string><string>0.0.0.0</string>
|
|
<string>--port</string><string>8080</string>
|
|
<string>-ngl</string><string>99</string>
|
|
<string>-c</string><string>131072</string>
|
|
</array>
|
|
<key>RunAtLoad</key><true/>
|
|
</dict>
|
|
</plist>
|
|
```
|
|
|
|
## Initial Flags Explained
|
|
|
|
| Flag | Value | Purpose |
|
|
|------|-------|---------|
|
|
| `-ngl 99` | All layers on GPU | Full Metal acceleration |
|
|
| `-c` | 262K / 131K | Context window size |
|
|
| `--mlock` | - | Prevent model from being swapped |
|
|
| `--spec-type draft-mtp` | - | Multi-Token Prediction speculation |
|
|
|
|
## Loading the Services
|
|
|
|
```bash
|
|
launchctl load ~/Library/LaunchAgents/com.jimmyg.llama-server-qwen35b.plist
|
|
launchctl load ~/Library/LaunchAgents/com.jimmyg.llama-server.plist
|
|
```
|
|
|
|
## Hermes Agent Integration
|
|
|
|
Two custom providers in `~/.hermes/config.yaml`:
|
|
|
|
```yaml
|
|
custom_providers:
|
|
- base_url: http://localhost:8085/v1
|
|
model: /Users/jimmyg/models/Qwen3.6-35B-A3B-Q8_0.gguf
|
|
name: qwen35b
|
|
- base_url: http://localhost:8080/v1
|
|
model: Qwen3.6-27B-Q8_0
|
|
name: qwen27b
|
|
```
|
|
|
|
## Baseline Performance
|
|
|
|
At this point with default settings:
|
|
- 35B MoE: ~20 t/s
|
|
- 27B dense: ~12 t/s
|
|
|
|
Plenty of room for optimization.
|