# The Optimization Journey

## Overview

Starting from the baseline (two servers, ~20 t/s on 35B, ~12 t/s on 27B), each optimization improved speed by addressing specific bottlenecks.

## Optimization 1: KV Cache Quantization

**Problem:** Full-precision (f16) KV cache consumes massive RAM. For 128K context on the 35B model, the KV cache alone uses ~15 GB.

**Fix:** Switch to Q4_0 KV cache.

```bash
--cache-type-k q4_0 --cache-type-v q4_0
```

**Impact:** -50% KV cache memory. Negligible quality loss — KV cache is a caching layer, not model weights. The precision of cached attention states has minimal effect on output quality.

## Optimization 2: Context Window Tuning

**Problem:** The 35B used 262K context (way too much for chat). The 27B used 131K. Combined KV caches consumed ~32 GB.

**Fix:** Both models at 128K. Sufficient for agent sessions (typical usage tops out at ~65K tokens).

```bash
-c 131072
```

**Impact:** Saved ~16 GB RAM. Freed headroom for the rest of the system.

## Optimization 3: Flash Attention

**Problem:** Without flash attention, generation speed degrades as KV cache grows (O(n²) attention cost).

**Fix:** Enable flash attention.

```bash
--flash-attn on
```

**Impact:** Maintains generation speed even with large contexts. With FA, generation stays at ~20 t/s regardless of context size. Without it, speed drops to ~10 t/s when context reaches 20K+ tokens.

## Optimization 4: Batch Size Tuning

**Problem:** Default batch sizes (2048/512) are optimized for server throughput (many concurrent requests), not single-user generation latency.

**Fix:** Smaller batch sizes for single-user use.

```bash
--batch-size 512 --ubatch-size 128
```

**Impact:** +25-50% generation speed for dense models. The 27B jumped from 12 to 16 t/s. The 35B MoE was less affected (3B active params already efficient).

**Why:** During generation (1 token at a time), the batch size affects how the GPU schedules compute. Smaller batches reduce latency per decode step.

## Optimization 5: Thread Count Tuning

**Problem:** Auto-detected thread count uses all CPU cores (18), causing dispatch overhead without benefit.

**Fix:** Explicit thread count optimized per architecture.

```bash
-t 14    # General fallback
```

**Impact:** MoE models benefit from more threads (attention compute-heavy). Dense models benefit from fewer threads (bandwidth-bound, less dispatch overhead).

| Model | Optimal Threads | Speed |
|-------|----------------|-------|
| 35B MoE (3B active) | 14 | 132 t/s |
| 27B dense (27B active) | 10 | 48 t/s |

## Optimization 6: Speculative Decoding (MTP)

**Fix:** Increase speculation from 1 to 3.

```bash
--spec-draft-n-max 3
```

**Impact:** With 82-93% MTP acceptance rate, each forward pass produces ~2.5-3 tokens instead of 1. This effectively doubles generation speed for free.

**Key insight:** Qwen models have built-in Multi-Token Prediction heads. The `--spec-type draft-mtp` flag uses these native heads rather than a separate draft model, adding negligible overhead.

## The Big One: Router Mode

The above optimizations improved speed, but the fundamental problem remained: **two processes competing for GPU bandwidth.**

**Fix:** Replace two `llama-server` processes with one router-mode server.

```bash
llama-server \
  --models-dir ~/.hermes/models-router \
  --models-max 1 \
  --models-preset ~/.hermes/llama-models.ini
```

**Impact:** +5.7x on 35B, +3.0x on 27B. See `06-router-mode.md` for full details.