Measured on this card

NVIDIA GeForce RTX 3060

12 GB · Ampere sm_86 · 360 GB/s

Saluki 27B IQ2-mix: token generation

Typical app Our engine t/s faster 1.6×

Model: Saluki 27B IQ2-mix
File: 7.9 GB
Memory used: 9.6 GB / 12 GB
A 500-word answer: 24.4 s → 15.3 s

Memory: 12GB
Bandwidth: 360GB/s
Power limit: 170W
Architecture: Ampere sm_86
PCIe: gen3 x16

What fits

12 GB of memory. Each block is a model loaded by our engine, with its context and KV cache.

Saluki 27B IQ2-mix

9.6 GB

file 7.9 GB used 9.6 GB

Bonsai 2 27B Q2_0 (PQ2_0 repack)

9.4 GB

used 9.4 GB

Models for this card

Every measured model that fits in the card’s memory. The best one first.

Underdog Saluki 27B

IQ2-mix 7.9 GB Licence: Licence not verified yet

Our pick for this card
Writing speed

1.6× faster

Same card, same model, same script. Only the software changes.

Honestly: the details

After a 16k context: 22.5 → 35.5 t/s. Without speculation (llama-bench tg128) our engine is 2.4% slower here (24.2 → 23.6): the whole gain on this card comes from the MTP head.

Typical app 24.0 t/s

Our engine 38.2 t/s

Typical app: llama.cpp with default settings, like a typical app (on the RTX 3090 the same as Ollama) · Our engine: our engine with the same flags plus our patches · spec_sweep through the server, the same script on both sides: chat, 6 prompts × 3 seeds, T=1, fresh server, median; ABBA order, mean of the two passes /; :168

A 500-word answer
derived from t/s × tokens per word /; :168 × 2.69 tok/słowo (raport 06 §2.1.2)/; :168 × 1.17 tok/słowo (raport 06 §2.1.2)

in Polish

in English

Replay at real speed

The same answer, written out at the measured speed.

Typical app: 24.0 t/s, Our engine: 38.2 t/s. spec_sweep through the server, the same script on both sides: chat, 6 prompts × 3 seeds, T=1, fresh server, median; ABBA order, mean of the two passes

Typical app IQ2-mix

24.0 t/s

First I collect the sensor data and check which readings are reliable. Then I compare them with yesterday’s run, flag the outliers and suggest three fixes. I describe each one briefly, with its risk and cost. Finally I write a summary for the team and a list of questions worth answering before the rollout. If something does not add up, I go back to the raw data, check the calibration and compare the results with last week. I note what I changed and why, so anyone can repeat the steps. Only then do I send the report, with a chart and a short recommendation for what to do tomorrow morning. Each figure links to its source, and every assumption is written down next to it, so nothing has to be taken on trust.

156 / 156 tokens done ·6.5 s
Our engine IQ2-mix

38.2 t/s

First I collect the sensor data and check which readings are reliable. Then I compare them with yesterday’s run, flag the outliers and suggest three fixes. I describe each one briefly, with its risk and cost. Finally I write a summary for the team and a list of questions worth answering before the rollout. If something does not add up, I go back to the raw data, check the calibration and compare the results with last week. I note what I changed and why, so anyone can repeat the steps. Only then do I send the report, with a chart and a short recommendation for what to do tomorrow morning. Each figure links to its source, and every assumption is written down next to it, so nothing has to be taken on trust.

156 / 156 tokens done ·4.1 s 1.6×
Conditions

spec_sweep through the server, the same script on both sides: chat, 6 prompts × 3 seeds, T=1, fresh server, median; ABBA order, mean of the two passes
- ours: the Kujira MTP head n=2 (64k vocabulary, IQ4_XS)
- lookup W16
- GPU verify
- KV q8_0
- 20k context (-c 20480)
- A0: plain -m (-fit: 34k context, KV f16)
- no speculation
- card at 170 W
spec_sweep through the server, the same script on both sides: chat, 6 prompts × 3 seeds, T=1, fresh server, median; ABBA order, mean of the two passes · 170 W (constantly at the limit), PCIe gen3 x16, 360 GB/s, Xeon E5-2698 v4, driver 595.71.05.
About the model

A 27B model (qwen35 architecture) quantised to ~2 bits with standard IQ types: a 7.9 GB file. It has no MTP head of its own; our engine adds the Kujira MTP head.

Better quality (perplexity 7.67 vs 8.62), and 1.6× faster with the MTP head.
- low memory
- quality on 12 GB
All measurements
Instruments

Typical app → Our engine
Tokens per step: tokens accepted per model step (MTP speculation) · Step time: The step is longer because it yields ~2.2 tokens. Per token: 41.7 → 26.3 ms (derived: step time ÷ tokens per step). · Prefill 512: reading the prompt, tokens per second · Prefill 4096: reading the prompt, tokens per second · Memory used: ours: the Kujira MTP head n=2 (64k vocabulary, IQ4_XS), lookup W16, GPU verify, KV q8_0, 20k context (-c 20480); A0: plain -m (-fit: 34k context, KV f16), no speculation; card at 170 W (the source gives GiB; shown as GB, the way the vendor labels the card memory) /; :170/; :169/; :187 (10313 MiB)/; :187 (9811 MiB)

Tokens per step

1.00 2.22

↗ better
Step time

41.7 58.4 ms

↘ longer step, ↗ more tokens
Prefill 512

581 564 t/s

≈ no change
Prefill 4096

576 559 t/s

≈ no change
Memory used

10.1 9.6 GB

↗ better
Same model, same quality: measured

Honestly

Saluki 27B

KLD ours vs upstream 0.00065

upstream noise 0.00061
Bonsai 2 27B

KLD ours vs upstream 0.00032

upstream noise 0.00021

In plain mode our engine does not produce bit-identical text to upstream (patches 0001/0002, plus 0013 for Qwen). So we measure quality with KLD: on the RTX 3060 the difference stays within upstream’s own noise.
Ternary Bonsai 2 27B

Q2_0 (PQ2_0 repack) Licence: Licence not verified yet
Writing speed

Another fork is faster here.

Same card, same model, same script. Only the software changes.

Honestly: the details

Honestly: the Prism fork with its PTQ1_0 format is faster than us here (44.8 t/s). Our engine speeds up upstream on the same Q2_0 file by 7%.

Prism PQ2_0 (default) 36.8 t/s

Prism PTQ1_0 (fastest) 44.8 t/s

upstream, Q2_0 file 33.1 t/s

our engine, Q2_0 file 35.5 t/s

Prism PQ2_0 (default) · Prism PTQ1_0 (fastest) · upstream, Q2_0 file · our engine, Q2_0 file · spec_sweep through the server: 6 prompts × 3 seeds, T=1, 400-token answers, short context, median (A0: 36.77) (A1: 44.77) (A1q: 33.15) (A2t: 35.51)

A 500-word answer
The bars run in real time: this is how long the answer takes at the measured speed. spec_sweep through the server: 6 prompts × 3 seeds, T=1, 400-token answers, short context, median

in Polish

in English

Replay at real speed

The same answer, written out at the measured speed.

upstream, Q2_0 file: 33.1 t/s, our engine, Q2_0 file: 35.5 t/s, Prism PTQ1_0 (fastest): 44.8 t/s. spec_sweep through the server: 6 prompts × 3 seeds, T=1, 400-token answers, short context, median

upstream, Q2_0 file Q2_0 (PQ2_0 repack)

33.1 t/s

First I collect the sensor data and check which readings are reliable. Then I compare them with yesterday’s run, flag the outliers and suggest three fixes. I describe each one briefly, with its risk and cost. Finally I write a summary for the team and a list of questions worth answering before the rollout. If something does not add up, I go back to the raw data, check the calibration and compare the results with last week. I note what I changed and why, so anyone can repeat the steps. Only then do I send the report, with a chart and a short recommendation for what to do tomorrow morning. Each figure links to its source, and every assumption is written down next to it, so nothing has to be taken on trust.

156 / 156 tokens done ·4.7 s
our engine, Q2_0 file Q2_0 (PQ2_0 repack)

35.5 t/s

First I collect the sensor data and check which readings are reliable. Then I compare them with yesterday’s run, flag the outliers and suggest three fixes. I describe each one briefly, with its risk and cost. Finally I write a summary for the team and a list of questions worth answering before the rollout. If something does not add up, I go back to the raw data, check the calibration and compare the results with last week. I note what I changed and why, so anyone can repeat the steps. Only then do I send the report, with a chart and a short recommendation for what to do tomorrow morning. Each figure links to its source, and every assumption is written down next to it, so nothing has to be taken on trust.

156 / 156 tokens done ·4.4 s 1.07×
Prism PTQ1_0 (fastest)

44.8 t/s

First I collect the sensor data and check which readings are reliable. Then I compare them with yesterday’s run, flag the outliers and suggest three fixes. I describe each one briefly, with its risk and cost. Finally I write a summary for the team and a list of questions worth answering before the rollout. If something does not add up, I go back to the raw data, check the calibration and compare the results with last week. I note what I changed and why, so anyone can repeat the steps. Only then do I send the report, with a chart and a short recommendation for what to do tomorrow morning. Each figure links to its source, and every assumption is written down next to it, so nothing has to be taken on trust.

156 / 156 tokens done ·3.5 s
Conditions

spec_sweep through the server: 6 prompts × 3 seeds, T=1, 400-token answers, short context, median
- 32k context
- KV f16
- -ub 1024
- no speculation (no MTP head)
spec_sweep through the server: 6 prompts × 3 seeds, T=1, 400-token answers, short context, median · 170 W (constantly at the limit), PCIe gen3 x16, 360 GB/s, Xeon E5-2698 v4, driver 595.71.05. :109-113
About the model

A 27B model with ternary weights (−1, 0, +1) in the Prism fork formats. No MTP head. We repack it into upstream Q2_0 losslessly.
- speed on 12 GB
All measurements
Instruments

Prism PQ2_0 (default) → our engine, Q2_0 file
Prefill 4096: reading the prompt, tokens per second · First token: ~2k-token prompt, no cache · Memory used: 32k context, KV f16, -ub 1024; no speculation (no MTP head)

Prefill 4096

601 622 t/s

↗ better
First token

3.4 3.4 s

≈ no change
Memory used

10.1 9.4 GB

↗ better
Long context
Generation as the conversation grows (llama-bench, no speculation). The gain here is small, and we show it as it is.

our engine upstream, Q2_0 file context length (linear scale) →

Same model, same quality: measured

Honestly

Saluki 27B

KLD ours vs upstream 0.00065

upstream noise 0.00061
Bonsai 2 27B

KLD ours vs upstream 0.00032

upstream noise 0.00021

In plain mode our engine does not produce bit-identical text to upstream (patches 0001/0002, plus 0013 for Qwen). So we measure quality with KLD: on the RTX 3060 the difference stays within upstream’s own noise.

What next

Compare with another card Check your hardware

NVIDIA GeForce RTX 3060

Writing speed

A 500-word answer

Replay at real speed

Conditions

About the model

Instruments

Honestly