Anton, chapter 8: Local survivability
The morning began with everything broken.
Anthropic rejected every request because of a billing problem. LiteLLM had no fallback configured for the affected model group, so the error propagated through the entire system.
Interactive conversations failed. Scheduled jobs stopped. The heartbeat went quiet.
The card issue itself took minutes to fix.
The real bug was that one provider's billing state could take Anton offline while a capable local model sat idle on the same machine.
The plan became obvious:
- Route every paid provider toward the local model as a final fallback.
- Exercise the fallback path until it actually worked.
- Use the disruption to improve local inference.
In that order.
Survivability had to land before optimisation.
A fallback that has never run
Adding the LiteLLM fallback chain was easy.
Testing it was more revealing.
The paid request failed as expected, LiteLLM routed to Gustav (the local Gemma), and vLLM rejected the tool call because the server had never been launched with automatic tool choice and the appropriate parsers.
The local model had been "available" for months, but the path used by real agents had never reached it.
The configuration error was latent because nothing routinely exercised the fallback.
That produced the first lesson of the day:
A fallback that is not tested is only a theory.
I enabled automatic tool choice and selected the Gemma tool-call and reasoning parsers available in the running image. A smoke test through LiteLLM returned a proper structured tool call.
Only then did the chain exist end to end.
Benchmark before change
With the system stable, I was ready to change the local model configuration.
Then I noticed I had no baseline.
Without one, an improvement would be anecdotal and a regression might go unnoticed.
I ran a fixed prompt three times with deterministic settings.
The result was steady: 23.4 tokens per second.
That number mattered less than the act of recording it.
The temptation to skip a benchmark is strongest when the change feels obviously beneficial. That is precisely when discipline matters.
Benchmark first. Change second.
Read before touching production
I investigated three possible paths.
Runtime low-precision quantisation for the Gemma mixture-of-experts model had unresolved loader problems in vLLM. A pre-quantised NVFP4 checkpoint was available and small enough to free substantial memory. That checkpoint required a patched Gemma model file because vLLM's expert mapping did not yet handle several NVFP4 scale-key suffixes.
The patch was not theoretical. It corresponded to an open upstream issue.
This changed the deployment plan.
The new model, image, and patched file were downloaded while the existing BF16 container continued serving traffic. The launch script contained the new configuration and a commented rollback block directly beneath it.
The disruption window should contain only the swap.
Everything else can happen beforehand.
The swap
The new container used the pre-quantised model through vLLM's ModelOpt path and selected the Marlin mixture-of-experts backend.
The served model name remained unchanged, so LiteLLM required no corresponding edit.
That is another benefit of indirection: the implementation can change while its contract stays still.
The container restarted in roughly ninety seconds.
The log confirmed that the Marlin NVFP4 MoE backend had been selected. Tool calling still worked.
Then I repeated the same benchmark.
The average rose to 43.5 tokens per second, or 1.86 times the original throughput.
Model weight memory fell from roughly 52 GB to 16.5 GB. The freed memory expanded the available KV-cache budget and allowed the configured context window to grow from 32K to 128K.
| Before | After | |
|---|---|---|
| Single-request throughput | 23.4 tok/s | 43.5 tok/s |
| Weight memory | ~52 GB | ~16.5 GB |
| Maximum context | 32,768 | 131,072 |
The new throughput varied more from run to run than BF16. The path depended on a software backend rather than a perfectly native execution route, and the published result on similar hardware remained somewhat higher.
That was acceptable.
The useful result was not winning a benchmark. It was nearly doubling throughput, cutting model weight memory by more than two thirds, preserving tool calling, and making local fallback capable of handling real agent traffic.
Rollback is part of deployment
The BF16 configuration stayed in the launch script, commented and ready.
Reverting required a container recreation, not a data restore or another research session.
A rollback plan written after a deployment fails is not a rollback plan.
It is incident response.
By the evening, the system was harder to take down than it had been that morning.
Any paid provider could fail because of billing, rate limits, or an API incident, and traffic would continue toward the machine on my desk.
The local model was faster, smaller, and capable of handling much longer contexts.
The day produced three rules worth keeping:
Exercise fallback paths before you need them.
Benchmark before changing the thing you intend to improve.
Stage everything possible while the current service is still running.
The morning began with a trivial billing problem revealing a structural weakness.
The evening ended with local inference becoming part of Anton's reliability model rather than merely an interesting capability.