Files

T

Gan, Jimmy 3994e29cb0 Initial: llama.cpp router mode optimization guide for Apple Silicon

Full documentation of optimizing two LLMs on a single M5 Max GPU:
- KV cache quantization (Q4_0)
- Flash attention and batch tuning
- Router mode with --models-max 1
- Per-model thread optimization via INI presets
- Before/after benchmarks (12→48 t/s on 27B, 23→132 t/s on 35B)

2026-07-10 02:27:36 +08:00

2.4 KiB

Raw Permalink Blame History

Step 1: Two Separate Servers

The initial setup for running both model servers as persistent macOS services via launchd.

Launchd Plist for 35B Chat Model

<!-- ~/Library/LaunchAgents/com.jimmyg.llama-server-qwen35b.plist -->
<plist>
<dict>
  <key>KeepAlive</key><true/>
  <key>ProgramArguments</key>
  <array>
    <string>/opt/homebrew/bin/llama-server</string>
    <string>-m</string>
    <string>/Users/jimmyg/models/Qwen3.6-35B-A3B-Q8_0.gguf</string>
    <string>--host</string><string>0.0.0.0</string>
    <string>--port</string><string>8085</string>
    <string>-ngl</string><string>99</string>
    <string>-c</string><string>262144</string>
    <string>--mlock</string>
  </array>
  <key>RunAtLoad</key><true/>
  <key>StandardOutPath</key>
  <string>/Users/jimmyg/.hermes/logs/llama-server-qwen35b.log</string>
  <key>StandardErrorPath</key>
  <string>/Users/jimmyg/.hermes/logs/llama-server-qwen35b.log</string>
</dict>
</plist>

Launchd Plist for 27B Coding Model

<!-- ~/Library/LaunchAgents/com.jimmyg.llama-server.plist -->
<plist>
<dict>
  <key>KeepAlive</key><true/>
  <key>ProgramArguments</key>
  <array>
    <string>/opt/homebrew/bin/llama-server</string>
    <string>-m</string>
    <string>/Users/jimmyg/models/Qwen3.6-27B-Q8_0.gguf</string>
    <string>--host</string><string>0.0.0.0</string>
    <string>--port</string><string>8080</string>
    <string>-ngl</string><string>99</string>
    <string>-c</string><string>131072</string>
  </array>
  <key>RunAtLoad</key><true/>
</dict>
</plist>

Initial Flags Explained

Flag	Value	Purpose
`-ngl 99`	All layers on GPU	Full Metal acceleration
`-c`	262K / 131K	Context window size
`--mlock`	-	Prevent model from being swapped
`--spec-type draft-mtp`	-	Multi-Token Prediction speculation

Loading the Services

launchctl load ~/Library/LaunchAgents/com.jimmyg.llama-server-qwen35b.plist
launchctl load ~/Library/LaunchAgents/com.jimmyg.llama-server.plist

Hermes Agent Integration

Two custom providers in ~/.hermes/config.yaml:

custom_providers:
  - base_url: http://localhost:8085/v1
    model: /Users/jimmyg/models/Qwen3.6-35B-A3B-Q8_0.gguf
    name: qwen35b
  - base_url: http://localhost:8080/v1
    model: Qwen3.6-27B-Q8_0
    name: qwen27b

Baseline Performance

At this point with default settings:

35B MoE: ~20 t/s
27B dense: ~12 t/s

Plenty of room for optimization.

2.4 KiB Raw Permalink Blame History