Anton, chapter 10: Making Gemma fly
The local model on the Spark was Nemotron when this began.
It produced around 58 tokens per second and depended on two unmerged patches I had backported by hand just to load correctly.
Then Gemma 4 arrived.
It was multimodal, capable of native tool use, and much closer to the model I wanted Anton to grow around. I moved local inference back to Gemma and landed in the high forties.
That should have felt like progress.
Instead, it reminded me of an unfinished question from the spring: was that the limit of the hardware, or merely the limit of the first configuration that worked?
I decided to find out with measurements rather than intuition.
Quantisation was not the lever
The obvious idea was better quantisation.
A new quantisation-aware checkpoint promised near-full quality at four-bit weight precision. The theory was attractive: smaller weights, less memory traffic, more speed.
I sent several research agents after the question from different angles.
They returned with the same answer.
On this workload, the Spark was constrained primarily by memory bandwidth. Any four-bit format moved roughly the same amount of weight data per generated token. A different checkpoint would not change that basic limit.
Worse, the existing path used a low-precision format without receiving the full native-compute benefit from the hardware and software combination. I was paying for patched loading and a specialised kernel without finding a meaningful new speed lever.
Quantisation had delivered the storage win.
It was not where the next latency win lived.
The spec sheet is not the benchmark
I looked at other machines I already owned.
My work Mac and an idle laptop advertised more memory bandwidth than the Spark. If decode speed were determined by bandwidth alone, they should have won.
They did not.
The same model ran slower.
The architecture was mixture-of-experts, and the efficiency of the available kernels mattered enormously. A theoretical bandwidth ceiling says little about how close the software stack gets to it.
That detour produced a rule worth the time it cost:
The spec sheet is not the benchmark.
Hardware figures describe possibility.
The application experiences kernels, schedulers, caches, frameworks, and actual request shapes.
Several tokens for one weight read
The remaining lever was speculative decoding.
A small drafter predicts several likely next tokens. The larger model verifies them together. When the workload is memory-bound, the benefit is structural: one expensive pass over the main model's weights can validate more than one generated token.
I followed a published recipe and initially achieved roughly half its headline result.
The command was the same. My baseline matched theirs.
The difference was the measurement.
Their benchmark isolated decode throughput after prefill. Mine measured wall-clock time as a caller experienced it, including the parts around generation.
Both numbers were valid.
They answered different questions.
The honest user-facing improvement was around 1.4 to 1.9 times, depending on how predictable the output was.
That was less dramatic than the headline and more useful than repeating it.
Test the way Anton is used
Single-request throughput still was not the most important measurement.
Anton repeatedly sends similar prefixes: the same system prompt, tool schemas, conversation history, and agent instructions. In an active conversation, much of that prefix is already represented in the KV cache.
So I tested the real router prompt, real conversation structure, and multiple concurrent requests.
The result that mattered was latency on the common path.
With the prefix cached, an answer that had taken around five seconds could return in roughly half a second.
That improvement did not come from speculative decoding alone.
It came from two independent mechanisms:
- Speculative decoding reduced the cost of producing new tokens.
- Prefix caching avoided recomputing what the model had already seen.
I had spent days looking at quantisation because it was visible and concrete.
The larger win came from understanding the request lifecycle.
Throughput is not one number
Then I tried to break the serving stack.
Five hundred requests arrived at once.
They queued. None were dropped. The machine remained stable, and the thermal warnings I had heard about did not reproduce.
Aggregate throughput climbed above a thousand tokens per second across the workload.
That number should not be confused with the speed one person sees. Single-stream latency, decode throughput, time to first token, cached-turn latency, and aggregate throughput are different measurements.
The chapter's real subject became measurement itself.
Performance claims are often disagreements between denominators.
A benchmark can remove prefill, warm the cache, run one stream, batch many streams, or measure only steady-state decoding. Each result may be correct while describing a different system.
The relevant benchmark is the one that resembles the workload you actually care about.
The ceiling of one idea
By the end of the weekend, local Gemma could answer a live cached conversation in around half a second, accelerate ordinary generation through speculation, sustain heavy concurrent traffic, and fail over to another machine if the Spark stumbled.
The original question had an answer.
The high-forties result had not been the ceiling of the hardware.
It had been the ceiling of one idea: straightforward autoregressive decoding on a single request without fully exploiting the repeated structure around it.
Most of the climb came not from a more exotic model or a more aggressive quantisation.
It came from measuring honestly enough to notice where time was actually being spent.
The lesson is easy to state and hard to practise:
Do not optimise the number that is easiest to measure. Optimise the experience the system actually delivers.