# Benchmark Results

## Hardware

| Component | Spec |
|-----------|------|
| Chip | Apple M5 Max |
| GPU Cores | 40 |
| RAM | 128 GB Unified Memory |
| Memory Bandwidth | 614 GB/s |
| OS | macOS 26.3.2 |
| llama.cpp | b9910 (f5525f7e7) |

## Models

| Model | Architecture | Size | Quant |
|-------|-------------|------|-------|
| Qwen3.6-35B-A3B | MoE (3B active) | ~35 GB | Q8_0 |
| Qwen3.6-27B | Dense | ~27 GB | Q8_0 |

## Config

```
-c 131072, --cache-type-k q4_0, --cache-type-v q4_0
--flash-attn on, --spec-type draft-mtp, --spec-draft-n-max 3
--batch-size 512, --ubatch-size 128
```

## Results

### Two Separate Servers (Before)

**Both loaded, both generating:**

| Model | Gen Speed | Notes |
|-------|-----------|-------|
| 35B MoE | 23 t/s | -75% from solo potential |
| 27B dense | 12 t/s | Already bandwidth-saturated |

**Solo (only one running):**

| Model | Gen Speed | Bandwidth Efficiency |
|-------|-----------|---------------------|
| 35B MoE | 94 t/s | 3.2 GB/step × 94 = 49% of 614 GB/s |
| 27B dense | 12 t/s | 30.4 GB/step × 12 = 59% of 614 GB/s |

### Router Mode (After)

**--models-max 1, auto-swap on request:**

| Model | Gen Speed (cold) | Gen Speed (warm) | Gain vs Before |
|-------|-----------------|------------------|----------------|
| 35B MoE | 132 t/s | 148 t/s | **5.7x** |
| 27B dense | 38 t/s | 48 t/s | **3.0x** |

**Per-model thread tuning:**

| Model | Optimal Threads | Speed |
|-------|----------------|-------|
| 35B MoE (attention-compute bound) | 14 | 148 t/s |
| 27B dense (bandwidth bound) | 10 | 48 t/s |

### Model Switch Latency

| Transition | Time |
|-----------|------|
| 35B → 27B (cold start) | ~4s |
| 27B → 35B (reload) | ~3s |

The 3-4s delay is model weights loading from SSD to GPU memory via Metal. Smaller batch sizes and Q4_0 KV cache help minimize this.

## MTP Speculative Decoding

| Model | Acceptance Rate | Mean Draft Length | Effective Tokens/Step |
|-------|----------------|-------------------|----------------------|
| 35B MoE | 74% | 2.84 | 2.1 |
| 27B dense | 82% | 3.62 | 2.6 |

MTP (Multi-Token Prediction) uses the model's built-in speculative heads. Each forward pass produces 3 draft tokens; acceptance rate determines how many are kept. With ~2.5 tokens per forward pass, effective generation speed is ~2.5x the raw decode speed.

## Methodology

- Tested via llama.cpp `/v1/completions` API
- Fixed prompt: 7-11 tokens
- Generation: 200 tokens, temperature 0
- Timings from response body `timings.predicted_per_second`
- Warm = model already loaded in GPU, cold = first request after model load