[{"data":1,"prerenderedAt":6454},["ShallowReactive",2],{"writing-\u002Fideas\u002F2026-04-15-anton-08-nvfp4":3,"all-writing":327,"related-work-\u002Fideas\u002F2026-04-15-anton-08-nvfp4":6419},{"id":4,"title":5,"body":6,"canonical_url":307,"chapter":308,"date":309,"description":12,"extension":310,"featured":311,"image":307,"kind":312,"meta":313,"navigation":314,"path":315,"seo":316,"series":317,"stem":318,"summary":319,"tags":320,"work_slug":321,"__hash__":326},"writing\u002Fideas\u002F2026-04-15-anton-08-nvfp4.md","Anton, chapter 8: Local survivability",{"type":7,"value":8,"toc":297},"minimark",[9,13,16,19,22,25,28,41,44,47,52,55,58,61,64,67,70,73,76,79,83,86,89,92,95,98,101,104,107,111,114,117,120,123,126,129,132,136,139,142,145,148,151,154,157,160,215,218,221,224,228,231,234,237,240,243,246,249,252,255,258,261,264,267,271],[10,11,12],"p",{},"The morning began with everything broken.",[10,14,15],{},"Anthropic rejected every request because of a billing problem. LiteLLM had no fallback configured for the affected model group, so the error propagated through the entire system.",[10,17,18],{},"Interactive conversations failed. Scheduled jobs stopped. The heartbeat went quiet.",[10,20,21],{},"The card issue itself took minutes to fix.",[10,23,24],{},"The real bug was that one provider's billing state could take Anton offline while a capable local model sat idle on the same machine.",[10,26,27],{},"The plan became obvious:",[29,30,31,35,38],"ol",{},[32,33,34],"li",{},"Route every paid provider toward the local model as a final fallback.",[32,36,37],{},"Exercise the fallback path until it actually worked.",[32,39,40],{},"Use the disruption to improve local inference.",[10,42,43],{},"In that order.",[10,45,46],{},"Survivability had to land before optimisation.",[48,49,51],"h2",{"id":50},"a-fallback-that-has-never-run","A fallback that has never run",[10,53,54],{},"Adding the LiteLLM fallback chain was easy.",[10,56,57],{},"Testing it was more revealing.",[10,59,60],{},"The paid request failed as expected, LiteLLM routed to Gustav (the local Gemma), and vLLM rejected the tool call because the server had never been launched with automatic tool choice and the appropriate parsers.",[10,62,63],{},"The local model had been \"available\" for months, but the path used by real agents had never reached it.",[10,65,66],{},"The configuration error was latent because nothing routinely exercised the fallback.",[10,68,69],{},"That produced the first lesson of the day:",[10,71,72],{},"A fallback that is not tested is only a theory.",[10,74,75],{},"I enabled automatic tool choice and selected the Gemma tool-call and reasoning parsers available in the running image. A smoke test through LiteLLM returned a proper structured tool call.",[10,77,78],{},"Only then did the chain exist end to end.",[48,80,82],{"id":81},"benchmark-before-change","Benchmark before change",[10,84,85],{},"With the system stable, I was ready to change the local model configuration.",[10,87,88],{},"Then I noticed I had no baseline.",[10,90,91],{},"Without one, an improvement would be anecdotal and a regression might go unnoticed.",[10,93,94],{},"I ran a fixed prompt three times with deterministic settings.",[10,96,97],{},"The result was steady: 23.4 tokens per second.",[10,99,100],{},"That number mattered less than the act of recording it.",[10,102,103],{},"The temptation to skip a benchmark is strongest when the change feels obviously beneficial. That is precisely when discipline matters.",[10,105,106],{},"Benchmark first. Change second.",[48,108,110],{"id":109},"read-before-touching-production","Read before touching production",[10,112,113],{},"I investigated three possible paths.",[10,115,116],{},"Runtime low-precision quantisation for the Gemma mixture-of-experts model had unresolved loader problems in vLLM. A pre-quantised NVFP4 checkpoint was available and small enough to free substantial memory. That checkpoint required a patched Gemma model file because vLLM's expert mapping did not yet handle several NVFP4 scale-key suffixes.",[10,118,119],{},"The patch was not theoretical. It corresponded to an open upstream issue.",[10,121,122],{},"This changed the deployment plan.",[10,124,125],{},"The new model, image, and patched file were downloaded while the existing BF16 container continued serving traffic. The launch script contained the new configuration and a commented rollback block directly beneath it.",[10,127,128],{},"The disruption window should contain only the swap.",[10,130,131],{},"Everything else can happen beforehand.",[48,133,135],{"id":134},"the-swap","The swap",[10,137,138],{},"The new container used the pre-quantised model through vLLM's ModelOpt path and selected the Marlin mixture-of-experts backend.",[10,140,141],{},"The served model name remained unchanged, so LiteLLM required no corresponding edit.",[10,143,144],{},"That is another benefit of indirection: the implementation can change while its contract stays still.",[10,146,147],{},"The container restarted in roughly ninety seconds.",[10,149,150],{},"The log confirmed that the Marlin NVFP4 MoE backend had been selected. Tool calling still worked.",[10,152,153],{},"Then I repeated the same benchmark.",[10,155,156],{},"The average rose to 43.5 tokens per second, or 1.86 times the original throughput.",[10,158,159],{},"Model weight memory fell from roughly 52 GB to 16.5 GB. The freed memory expanded the available KV-cache budget and allowed the configured context window to grow from 32K to 128K.",[161,162,163,178],"table",{},[164,165,166],"thead",{},[167,168,169,172,175],"tr",{},[170,171],"th",{},[170,173,174],{},"Before",[170,176,177],{},"After",[179,180,181,193,204],"tbody",{},[167,182,183,187,190],{},[184,185,186],"td",{},"Single-request throughput",[184,188,189],{},"23.4 tok\u002Fs",[184,191,192],{},"43.5 tok\u002Fs",[167,194,195,198,201],{},[184,196,197],{},"Weight memory",[184,199,200],{},"~52 GB",[184,202,203],{},"~16.5 GB",[167,205,206,209,212],{},[184,207,208],{},"Maximum context",[184,210,211],{},"32,768",[184,213,214],{},"131,072",[10,216,217],{},"The new throughput varied more from run to run than BF16. The path depended on a software backend rather than a perfectly native execution route, and the published result on similar hardware remained somewhat higher.",[10,219,220],{},"That was acceptable.",[10,222,223],{},"The useful result was not winning a benchmark. It was nearly doubling throughput, cutting model weight memory by more than two thirds, preserving tool calling, and making local fallback capable of handling real agent traffic.",[48,225,227],{"id":226},"rollback-is-part-of-deployment","Rollback is part of deployment",[10,229,230],{},"The BF16 configuration stayed in the launch script, commented and ready.",[10,232,233],{},"Reverting required a container recreation, not a data restore or another research session.",[10,235,236],{},"A rollback plan written after a deployment fails is not a rollback plan.",[10,238,239],{},"It is incident response.",[10,241,242],{},"By the evening, the system was harder to take down than it had been that morning.",[10,244,245],{},"Any paid provider could fail because of billing, rate limits, or an API incident, and traffic would continue toward the machine on my desk.",[10,247,248],{},"The local model was faster, smaller, and capable of handling much longer contexts.",[10,250,251],{},"The day produced three rules worth keeping:",[10,253,254],{},"Exercise fallback paths before you need them.",[10,256,257],{},"Benchmark before changing the thing you intend to improve.",[10,259,260],{},"Stage everything possible while the current service is still running.",[10,262,263],{},"The morning began with a trivial billing problem revealing a structural weakness.",[10,265,266],{},"The evening ended with local inference becoming part of Anton's reliability model rather than merely an interesting capability.",[48,268,270],{"id":269},"references","References",[272,273,274,283,290],"ul",{},[32,275,276],{},[277,278,282],"a",{"href":279,"rel":280},"https:\u002F\u002Fdocs.litellm.ai\u002Fdocs\u002Fproxy\u002Freliability",[281],"nofollow","LiteLLM routing and fallbacks",[32,284,285],{},[277,286,289],{"href":287,"rel":288},"https:\u002F\u002Fdocs.vllm.ai\u002Fen\u002Flatest\u002Ffeatures\u002Ftool_calling.html",[281],"vLLM tool calling",[32,291,292],{},[277,293,296],{"href":294,"rel":295},"https:\u002F\u002Fdocs.vllm.ai\u002Fen\u002Flatest\u002Ffeatures\u002Fquantization\u002F",[281],"vLLM quantization support",{"title":298,"searchDepth":299,"depth":299,"links":300},"",2,[301,302,303,304,305,306],{"id":50,"depth":299,"text":51},{"id":81,"depth":299,"text":82},{"id":109,"depth":299,"text":110},{"id":134,"depth":299,"text":135},{"id":226,"depth":299,"text":227},{"id":269,"depth":299,"text":270},null,8,"2026-04-15","md",false,"note",{},true,"\u002Fideas\u002F2026-04-15-anton-08-nvfp4",{"title":5,"description":12},"anton-journey","ideas\u002F2026-04-15-anton-08-nvfp4","A billing failure exposes an untested fallback, and local inference becomes part of the reliability model.",[321,322,323,324,325],"anton","agents","ai","llm","dgx","ZhnBE9bLK5H4vIMUieYmeGjX3k1XR5qPsIq-D96euHY",[328,504,776,1395,1652,1913,2158,2441,2693,2733,2934,3200,4584,4836,5068,5285,5381,5456,5569,5760,5983,6323],{"id":329,"title":330,"body":331,"canonical_url":307,"chapter":495,"date":496,"description":335,"extension":310,"featured":311,"image":307,"kind":312,"meta":497,"navigation":314,"path":498,"seo":499,"series":317,"stem":500,"summary":501,"tags":502,"work_slug":321,"__hash__":503},"writing\u002Fideas\u002F2026-03-07-anton-01-genesis.md","Anton, chapter 1: Genesis",{"type":7,"value":332,"toc":489},[333,336,339,342,350,353,356,359,362,366,369,372,375,378,381,384,387,391,394,402,405,408,411,415,418,421,424,427,430,433,436,439,442,446,449,456,459,462,465,468,471,474,477,480,483,486],[10,334,335],{},"Anton started with a simple idea: to build an assistant more reliable and secure than OpenClaw, which I've found both frustrating and terrifying from a security point of view.",[10,337,338],{},"A family assistant seemed like the perfect test. Running a family of six involves enough coordination, repetition, and forgotten details to keep one busy. If Anton could reliably replace me for a few common chores, I would consider that a win.",[10,340,341],{},"I also had a DGX Spark sitting at home, seemingly waiting to host him.",[10,343,344,345,349],{},"Naming things is one of the fun parts of the process. You want something memorable without being distracting or confusing, so I named him Anton, after Gilfoyle's homemade server rig in ",[346,347,348],"em",{},"Silicon Valley",".",[10,351,352],{},"I started where I usually start: with the contract, not the code.",[10,354,355],{},"Before building the system, I wrote down its shape: the components, the boundaries between them, and the interfaces they would use to communicate. The repository contained a TypeScript monorepo, an implementation plan, and a shared types package defining the central contracts.",[10,357,358],{},"Nothing worked yet, but the important decisions were already visible.",[10,360,361],{},"Interfaces are cheap to change before the code exists. Once several parts of the system depend on them, they become much more expensive to rethink.",[48,363,365],{"id":364},"the-household","The household",[10,367,368],{},"Next came a map of the household.",[10,370,371],{},"WhatsApp would be the entry point because that is where the family already lives. The Spark, accessed over Tailscale, would be the host because Anton's memory, data, and voice should remain physically close to me rather than sitting in someone else's cloud.",[10,373,374],{},"Then came the systems he would need to be useful: the TNAS, Plex, Transmission, and Google Workspace.",[10,376,377],{},"Docker Compose provided three services: Postgres, WhatsApp, and a worker. A deployment script sent everything to the Spark.",[10,379,380],{},"No staging environment. No running it from my laptop.",[10,382,383],{},"I wanted Anton in production from the beginning.",[10,385,386],{},"The real thing.",[48,388,390],{"id":389},"identity-as-data","Identity as data",[10,392,393],{},"Then came a question I had been thinking about for weeks: is Anton a tool or a character?",[10,395,396,397,401],{},"I did not want his identity buried in a system prompt somewhere in the code. I wanted his personality to live in an ",[398,399,400],"code",{},"identity.md"," document loaded at runtime.",[10,403,404],{},"Anton's personality, and my own context, would be data, not code.",[10,406,407],{},"The reason was practical more than philosophical. When I change my mind about who Anton is, when my family does, or when one of the children moves out, I want to edit a document rather than redeploy the application.",[10,409,410],{},"Treating context as data feels like the kind of decision that compounds.",[48,412,414],{"id":413},"capabilities","Capabilities",[10,416,417],{},"Then I began adding capabilities.",[10,419,420],{},"Media came first: Plex search and Transmission control. Then calendar access, Google Workspace, and voice transcription.",[10,422,423],{},"I almost cut voice. I was not convinced the family would use it, and there was already plenty to build. In the end, I kept it because the cost of finding out was small, while the potential upside was much larger.",[10,425,426],{},"I also added a deliberately simple memory layer: Postgres rows tagged by user. No embeddings and no elaborate retrieval system.",[10,428,429],{},"The rule was to build the simplest version capable of revealing what the system actually needed, then design around what emerged.",[10,431,432],{},"Memory would eventually require real retrieval semantics, but I did not yet know what form they should take. The worst thing I could do was guess too early.",[10,434,435],{},"For orchestration, how the parent agent should route requests between different domains, I chose LangGraph.",[10,437,438],{},"It gave the system structure: an observable state machine, checkpointing, and a clean way to organize each domain as a subgraph beneath a classify-and-dispatch layer.",[10,440,441],{},"More importantly, it gave me a framework with answers to problems I had not encountered yet.",[48,443,445],{"id":444},"first-production-bug","First production bug",[10,447,448],{},"Then came the first production bug.",[10,450,451,452,455],{},"WhatsApp began throwing \"bad encryption\" errors on incoming messages because of a quirk in Baileys, the library handling the connection. The fix was straightforward: wire up ",[398,453,454],{},"getMessage"," so failed messages could be decrypted again.",[10,457,458],{},"The interesting part was not the fix.",[10,460,461],{},"It was that the system was already real enough to experience production bugs.",[10,463,464],{},"Stub systems do not fail like that.",[10,466,467],{},"I then separated the WhatsApp connection from the agent itself. WhatsApp became a thin transport layer that placed jobs in a queue, while the worker took responsibility for the agent logic.",[10,469,470],{},"Transport concerns should not leak into the intelligence of the system.",[10,472,473],{},"Cleaner that way.",[10,475,476],{},"By the end of the day, Anton could receive a WhatsApp message, retrieve the relevant conversation history, route the request to the right domain, execute scheduled jobs, update himself through an API endpoint, and redeploy from a single shell script.",[10,478,479],{},"He was usable end to end.",[10,481,482],{},"I went to bed satisfied.",[10,484,485],{},"Anton was real: deployed on the Spark, talking to the family WhatsApp group, with seven domains registered and a quiet scheduling queue waiting for work.",[10,487,488],{},"He already had the shape I had imagined that morning: a personal assistant living close to my data, on my own hardware, with a personality I could change by editing a file.",{"title":298,"searchDepth":299,"depth":299,"links":490},[491,492,493,494],{"id":364,"depth":299,"text":365},{"id":389,"depth":299,"text":390},{"id":413,"depth":299,"text":414},{"id":444,"depth":299,"text":445},1,"2026-03-07",{},"\u002Fideas\u002F2026-03-07-anton-01-genesis",{"title":330,"description":335},"ideas\u002F2026-03-07-anton-01-genesis","Building Anton, a personal agent OS for my family on a DGX Spark, day one.",[321,322,323],"ezw9nvrubia9foEjw_ht5bWrIQVNF8r8813Y9lgYSl0",{"id":505,"title":506,"body":507,"canonical_url":307,"chapter":299,"date":768,"description":511,"extension":310,"featured":311,"image":307,"kind":312,"meta":769,"navigation":314,"path":770,"seo":771,"series":317,"stem":772,"summary":773,"tags":774,"work_slug":321,"__hash__":775},"writing\u002Fideas\u002F2026-03-08-anton-02-first-weekend.md","Anton, chapter 2: The first weekend",{"type":7,"value":508,"toc":758},[509,512,515,518,521,524,527,531,534,537,540,543,546,549,552,556,559,562,565,568,571,574,578,581,584,587,594,597,600,603,607,610,616,619,622,625,628,632,635,638,641,644,647,651,654,657,660,663,666,669,672,675,678,682,685,688,691,694,697,700,703,706,712,715,718,721,725,728,731,734,737,740,743,746,749,752,755],[10,510,511],{},"I woke up Saturday wanting to replace the parent agent that previously orchestrated routing between subgraphs.",[10,513,514],{},"The classify-and-dispatch graph from the previous day worked, but it asked the LLM to make routing decisions inside a state machine that was already trying to do the same thing.",[10,516,517],{},"Two layers fighting over one job.",[10,519,520],{},"The parent became a single agent with the domain subgraphs exposed as tools. The LLM could decide which tool to use from the request itself, while the framework handled the underlying execution.",[10,522,523],{},"The system immediately felt lighter.",[10,525,526],{},"The model was doing what it was good at: interpreting intent. The framework was doing what it was good at: managing state and execution.",[48,528,530],{"id":529},"making-anton-usable","Making Anton usable",[10,532,533],{},"The next changes were small, but they made the difference between a functioning system and something the family could actually use.",[10,535,536],{},"Anton gained a typing indicator and an acknowledgement message, so people knew he had heard them while the model was still thinking.",[10,538,539],{},"He began answering in the language of the current message rather than the language of the conversation history. French and English are freely mixed in our house, often within the same thread.",[10,541,542],{},"Schedules became natural language. You could simply tell Anton, \"Remind me every morning,\" rather than learn a special command format.",[10,544,545],{},"He gained support for group conversations, daily summaries, and browsing previous exchanges.",[10,547,548],{},"I also added an explicit rule: Anton should never invent an answer merely to appear competent. My first attempt at grounding Anton's responses and mitigate hallucinations.",[10,550,551],{},"Reliability is not only about whether a system works. It is also about whether it admits when it does not know. Easier said than done, but we need to start somewhere so I added instructions to the prompt.",[48,553,555],{"id":554},"asking-for-help","Asking for help",[10,557,558],{},"The first browser-driven workflow was Doctolib.",[10,560,561],{},"Logging in required two-factor authentication, which meant Anton had to pause midway through a task, ask the user for the SMS code, and then continue from where he had stopped.",[10,563,564],{},"The mechanism was simple: an interactive input queue.",[10,566,567],{},"But it solved a much broader problem. Many useful tasks cannot be completed autonomously from beginning to end. They occasionally require a password, a confirmation, a choice, or some other piece of human input.",[10,569,570],{},"Anton did not need to fail or hand the entire task back. He could ask for help and continue.",[10,572,573],{},"That immediately felt important.",[48,575,577],{"id":576},"seeing-inside-the-system","Seeing inside the system",[10,579,580],{},"By Saturday evening, the main problem was no longer capability. It was visibility.",[10,582,583],{},"When Anton failed, I needed to understand why.",[10,585,586],{},"I added a pipeline that turned error traces into draft GitHub issues, so failures would leave behind something actionable rather than disappear into a log.",[10,588,589,590,593],{},"I added a simple ",[398,591,592],{},"\u002Flogs"," endpoint backed by an in-memory ring buffer. Reaching into Docker every time I wanted to inspect the system was a small inconvenience, but small inconveniences accumulate quickly when debugging.",[10,595,596],{},"Deployments were also moved behind a trigger file watched by systemd. Anton no longer had to run the deployment process synchronously himself. He could request an update, and a separate process would perform it and report back.",[10,598,599],{},"Decoupled, boring, reliable.",[10,601,602],{},"The kind of plumbing that disappears once it works.",[48,604,606],{"id":605},"the-lcars-dashboard","The LCARS dashboard",[10,608,609],{},"On Sunday morning, I built Anton a dashboard.",[10,611,612,613,349],{},"Naturally, it was styled after LCARS, the computer interface from ",[346,614,615],{},"Star Trek",[10,617,618],{},"It showed the health of each service, recent logs, conversations, and individual agent runs. From any conversation, I could drill into the execution trace and inspect what the model had done at each step.",[10,620,621],{},"This changed the pace of development completely.",[10,623,624],{},"Before the trace viewer, a wrong answer was simply wrong. After it, I could see whether Anton had misunderstood the request, chosen the wrong tool, passed bad context, or stopped too early.",[10,626,627],{},"Without that visibility, the rest of the weekend would not have happened.",[48,629,631],{"id":630},"expanding-what-anton-knows","Expanding what Anton knows",[10,633,634],{},"Once I could see what the system was doing, I began broadening its knowledge.",[10,636,637],{},"Anton gained the ability to search his own source code. He could research the web, browse documents, and store what he learned.",[10,639,640],{},"Research became its own domain rather than a capability buried inside every other agent.",[10,642,643],{},"That separation mattered. Research has its own concerns: cost, citations, source quality, fact-checking, and limits on how far the system should explore.",[10,645,646],{},"By Sunday afternoon, Anton could read the web, read his own code, and remember what he found.",[48,648,650],{"id":649},"from-hoping-to-knowing","From hoping to knowing",[10,652,653],{},"Then I wrote the quality suite.",[10,655,656],{},"It began with 28 tests covering the existing domains.",[10,658,659],{},"Until then, testing Anton meant sending him a WhatsApp message and deciding whether the answer looked reasonable. That works while a system is small, but it provides no protection against regressions.",[10,661,662],{},"The test suite changed the development loop from:",[10,664,665],{},"\"Does this appear to work?\"",[10,667,668],{},"to:",[10,670,671],{},"\"Did everything that worked yesterday continue to work today?\"",[10,673,674],{},"The gap between having no test suite and having one is the gap between hoping and knowing.",[10,676,677],{},"I should have built it on the first day.",[48,679,681],{"id":680},"the-calendar-saga","The calendar saga",[10,683,684],{},"The tests immediately found problems.",[10,686,687],{},"The worst was the calendar agent.",[10,689,690],{},"It repeatedly failed on simple multi-step requests such as deleting an event with a particular title.",[10,692,693],{},"There were two separate causes.",[10,695,696],{},"First, I was passing the parent agent's full conversation history into the calendar agent. That context contained unrelated discussions and instructions, which contaminated a task that should have been narrow and precise.",[10,698,699],{},"The rule became simple: domain agents should receive only the context they need.",[10,701,702],{},"Second, the model could not reliably chain a search followed by a deletion. It would find the event, return the result, and stop before deleting it.",[10,704,705],{},"I had given the model two low-level tools and expected it to assemble them into a reliable operation.",[10,707,708,709,349],{},"The solution was to create a composite skill: ",[398,710,711],{},"findAndDeleteEvent",[10,713,714],{},"This became another rule: when an operation must be reliable, do not depend on the model to discover the correct sequence of tool calls each time. Encode the sequence as a capability.",[10,716,717],{},"These were not principles I had arrived at in advance.",[10,719,720],{},"They were lessons learned by getting burned.",[48,722,724],{"id":723},"a-wider-household","A wider household",[10,726,727],{},"By the end of the weekend, Anton's domains had expanded further.",[10,729,730],{},"He could manage a wine collection, process school messages, and understand images or audio posted in group conversations when someone explicitly asked him to.",[10,732,733],{},"For media, I used an \"ingest cheap, process lazy\" approach: store the original file immediately, but only invoke transcription or vision when it is actually needed.",[10,735,736],{},"There was no reason to spend compute interpreting every photo and voice message passing through a family group.",[10,738,739],{},"Media downloads became more capable, with release preferences and the ability to stop and resume transfers.",[10,741,742],{},"Torrent searches were routed through Tor, meaning all requests were anonymized and routed through the Tor network to reduce traceability and isolate the system from direct exposure to torrent indexers and their associated legal and network risks, a decision made less for new functionality and more as a precautionary architectural safeguard.",[10,744,745],{},"By Monday morning, Anton had seven domains, 73 quality tests, a dashboard for inspecting traces, automatic issue creation, accessible logs, and a development loop I could trust.",[10,747,748],{},"I could change almost anything and quickly see what broke.",[10,750,751],{},"The lesson from the weekend was simple:",[10,753,754],{},"Write the tests before the bugs do.",[10,756,757],{},"Not because discipline is inherently virtuous, but because I had now experienced the alternative.",{"title":298,"searchDepth":299,"depth":299,"links":759},[760,761,762,763,764,765,766,767],{"id":529,"depth":299,"text":530},{"id":554,"depth":299,"text":555},{"id":576,"depth":299,"text":577},{"id":605,"depth":299,"text":606},{"id":630,"depth":299,"text":631},{"id":649,"depth":299,"text":650},{"id":680,"depth":299,"text":681},{"id":723,"depth":299,"text":724},"2026-03-08",{},"\u002Fideas\u002F2026-03-08-anton-02-first-weekend",{"title":506,"description":511},"ideas\u002F2026-03-08-anton-02-first-weekend","A first full weekend of building turns Anton into something the family can actually use.",[321,322,323],"Z7ro4lnGI3t9NjHKaU9SvWUv1wZKCBfgybfGyfHdjao",{"id":777,"title":778,"body":779,"canonical_url":307,"chapter":307,"date":1383,"description":1384,"extension":310,"featured":311,"image":307,"kind":312,"meta":1385,"navigation":314,"path":1386,"seo":1387,"series":1388,"stem":1389,"summary":1390,"tags":1391,"work_slug":307,"__hash__":1394},"writing\u002Fideas\u002F2026-03-10-dgx-spark-vllm.md","50+ tokens per second on a desktop: running LLMs on the NVIDIA DGX Spark",{"type":7,"value":780,"toc":1370},[781,788,791,794,797,801,804,821,824,828,831,851,854,857,861,868,871,891,894,898,901,927,934,938,947,961,972,976,1066,1072,1075,1079,1082,1253,1259,1263,1269,1275,1281,1287,1300,1306,1310,1317,1320,1324,1353,1357,1366],[10,782,783,787],{},[784,785,786],"strong",{},"TL;DR:"," We got a 30-billion-parameter LLM running at 51-54 tokens\u002Fsec on the NVIDIA DGX Spark by combining Mixture-of-Experts architecture, FP8 quantization, and a community Docker image that fixes Blackwell-specific issues. Here's what we learned.",[789,790],"hr",{},[10,792,793],{},"The NVIDIA DGX Spark is an interesting machine. It packs a Blackwell GB10 GPU with 128GB of unified LPDDR5X memory into a desktop form factor. For XRPL Commons, we wanted local LLM inference for our development workflow, fast enough to be usable, private enough to run on-premises, and simple enough to replicate across machines.",[10,795,796],{},"Getting there was not straightforward. This post documents the journey from 3.7 tok\u002Fs (unusable) to 54 tok\u002Fs (excellent), and the key technical decisions that made the difference.",[48,798,800],{"id":799},"the-hardware","The Hardware",[10,802,803],{},"The DGX Spark ships with:",[272,805,806,809,812,815,818],{},[32,807,808],{},"NVIDIA GB10 Blackwell GPU (SM 12.1)",[32,810,811],{},"128GB unified LPDDR5X at 273 GB\u002Fs bandwidth",[32,813,814],{},"ARM Grace CPU (aarch64), 10 cores",[32,816,817],{},"3.7TB NVMe storage",[32,819,820],{},"DGX OS (Ubuntu 24.04)",[10,822,823],{},"128GB of unified memory means you can fit very large models. But there's a catch.",[48,825,827],{"id":826},"the-bandwidth-wall","The Bandwidth Wall",[10,829,830],{},"LLM inference is memory-bandwidth-bound. During autoregressive decoding, each token requires reading every active weight from memory once. At 273 GB\u002Fs, the math is simple:",[272,832,833,842],{},[32,834,835,838,839],{},[784,836,837],{},"Dense 32B model (bf16):"," 64GB of weights \u002F 273 GB\u002Fs = ~234ms per token = ",[784,840,841],{},"~4 tok\u002Fs",[32,843,844,847,848],{},[784,845,846],{},"Dense 8B model (bf16):"," 16GB \u002F 273 GB\u002Fs = ~59ms = ",[784,849,850],{},"~17 tok\u002Fs",[10,852,853],{},"No amount of compute optimization changes this. The Spark can hold a 70B model in FP8, but it will generate tokens at walking pace. The memory is large but not fast.",[10,855,856],{},"We learned this the hard way. Our first attempt, Qwen3-32B at bf16, produced 3.7 tokens per second. Qwen3-8B was better at 13.1 tok\u002Fs, but still below the threshold for interactive use.",[48,858,860],{"id":859},"the-moe-breakthrough","The MoE Breakthrough",[10,862,863,864,867],{},"The solution is ",[784,865,866],{},"Mixture-of-Experts (MoE)"," models. An MoE model has many total parameters but only activates a fraction per token. Qwen3-30B-A3B has 30 billion parameters but only 3 billion active ones, the router activates a small subset of experts per token, leaving the rest idle in memory.",[10,869,870],{},"The bandwidth math changes completely:",[272,872,873,882],{},[32,874,875,878,879],{},[784,876,877],{},"MoE 30B, 3B active (bf16):"," ~6GB active weights \u002F 273 GB\u002Fs = ~22ms = ",[784,880,881],{},"~45 tok\u002Fs theoretical",[32,883,884,887,888],{},[784,885,886],{},"MoE 30B, 3B active (FP8):"," ~3GB active weights \u002F 273 GB\u002Fs = ~11ms = ",[784,889,890],{},"~90 tok\u002Fs theoretical",[10,892,893],{},"You get the quality of a 30B model at the speed of a 3B model.",[48,895,897],{"id":896},"the-software-stack-problem","The Software Stack Problem",[10,899,900],{},"The DGX Spark's Blackwell GPU (SM 12.1) is new enough that upstream tooling doesn't fully support it:",[272,902,903,909,915,921],{},[32,904,905,908],{},[784,906,907],{},"Flash Attention 2"," crashes with a PTX toolchain error",[32,910,911,914],{},[784,912,913],{},"vLLM's MOE CUTLASS kernels"," don't include SM 12.1 in their architecture intersection lists",[32,916,917,920],{},[784,918,919],{},"PyTorch"," officially supports up to SM 12.0",[32,922,923,926],{},[784,924,925],{},"CUDA graphs",", critical for throughput, simply don't work with a standard vLLM build",[10,928,929,930,933],{},"We spent considerable time on a manual vLLM build from source: patching CMakeLists.txt, building Triton from a specific commit, working around setuptools license field validation bugs, pinning transformers below 5.0 to avoid tokenizer breakage. The manual build worked but required ",[398,931,932],{},"--enforce-eager"," mode (no CUDA graphs), capping throughput at ~30 tok\u002Fs.",[48,935,937],{"id":936},"the-avarok-docker-image","The Avarok Docker Image",[10,939,940,941,946],{},"The ",[277,942,945],{"href":943,"rel":944},"https:\u002F\u002Fgithub.com\u002FAvarok-Cybersecurity\u002Fdgx-vllm",[281],"Avarok dgx-vllm project"," solves all of this in a single Docker image. It includes:",[272,948,949,952,955,958],{},[32,950,951],{},"A patched vLLM v0.16.0rc2 with SM 12.1 support",[32,953,954],{},"Software E2M1 conversion for the missing NVFP4 PTX instruction",[32,956,957],{},"Custom CUTLASS kernels for the GB10",[32,959,960],{},"Working CUDA graphs and Flash Attention",[10,962,963,964,967,968,971],{},"One ",[398,965,966],{},"docker pull"," and one ",[398,969,970],{},"docker run"," command replaces hours of manual compilation.",[48,973,975],{"id":974},"results","Results",[161,977,978,994],{},[164,979,980],{},[167,981,982,985,988,991],{},[170,983,984],{},"Model",[170,986,987],{},"Quantization",[170,989,990],{},"Engine",[170,992,993],{},"Tokens\u002Fsec",[179,995,996,1010,1022,1034,1046],{},[167,997,998,1001,1004,1007],{},[184,999,1000],{},"Qwen3-32B (dense)",[184,1002,1003],{},"bf16",[184,1005,1006],{},"Manual vLLM",[184,1008,1009],{},"3.7",[167,1011,1012,1015,1017,1019],{},[184,1013,1014],{},"Qwen3-8B (dense)",[184,1016,1003],{},[184,1018,1006],{},[184,1020,1021],{},"13.1",[167,1023,1024,1027,1029,1031],{},[184,1025,1026],{},"Qwen3-30B-A3B (MoE)",[184,1028,1003],{},[184,1030,1006],{},[184,1032,1033],{},"28.6",[167,1035,1036,1038,1040,1043],{},[184,1037,1026],{},[184,1039,1003],{},[184,1041,1042],{},"Avarok Docker",[184,1044,1045],{},"30.3",[167,1047,1048,1052,1057,1061],{},[184,1049,1050],{},[784,1051,1026],{},[184,1053,1054],{},[784,1055,1056],{},"FP8",[184,1058,1059],{},[784,1060,1042],{},[184,1062,1063],{},[784,1064,1065],{},"51-54",[10,1067,1068,1069,349],{},"The winning combination: ",[784,1070,1071],{},"MoE architecture + FP8 quantization + Avarok Docker with CUDA graphs",[10,1073,1074],{},"We've deployed this setup across two DGX Sparks with consistent results. The FP8 model uses ~110GB of the 119GB available memory, leaving minimal headroom, but the throughput is worth it.",[48,1076,1078],{"id":1077},"the-setup","The Setup",[10,1080,1081],{},"The final deployment is remarkably simple:",[1083,1084,1088],"pre",{"className":1085,"code":1086,"language":1087,"meta":298,"style":298},"language-bash shiki shiki-themes min-light","docker pull avarok\u002Fdgx-vllm-nvfp4-kernel:v22\n\ndocker run -d \\\n  --name vllm \\\n  --gpus all \\\n  --shm-size=16g \\\n  --restart unless-stopped \\\n  -p 8000:8888 \\\n  -v \u002Fhome\u002F$USER\u002F.cache\u002Fhuggingface:\u002Froot\u002F.cache\u002Fhuggingface \\\n  -e MODEL=Qwen\u002FQwen3-30B-A3B-Instruct-2507-FP8 \\\n  -e PORT=8888 \\\n  -e GPU_MEMORY_UTIL=0.85 \\\n  -e MAX_MODEL_LEN=32768 \\\n  avarok\u002Fdgx-vllm-nvfp4-kernel:v22 serve\n","bash",[398,1089,1090,1105,1110,1125,1136,1147,1155,1166,1176,1193,1204,1218,1231,1244],{"__ignoreMap":298},[1091,1092,1094,1098,1102],"span",{"class":1093,"line":495},"line",[1091,1095,1097],{"class":1096},"s7eDp","docker",[1091,1099,1101],{"class":1100},"sY4mW"," pull",[1091,1103,1104],{"class":1100}," avarok\u002Fdgx-vllm-nvfp4-kernel:v22\n",[1091,1106,1107],{"class":1093,"line":299},[1091,1108,1109],{"emptyLinePlaceholder":314},"\n",[1091,1111,1113,1115,1118,1121],{"class":1093,"line":1112},3,[1091,1114,1097],{"class":1096},[1091,1116,1117],{"class":1100}," run",[1091,1119,1120],{"class":1100}," -d",[1091,1122,1124],{"class":1123},"sR6ew"," \\\n",[1091,1126,1128,1131,1134],{"class":1093,"line":1127},4,[1091,1129,1130],{"class":1100},"  --name",[1091,1132,1133],{"class":1100}," vllm",[1091,1135,1124],{"class":1123},[1091,1137,1139,1142,1145],{"class":1093,"line":1138},5,[1091,1140,1141],{"class":1100},"  --gpus",[1091,1143,1144],{"class":1100}," all",[1091,1146,1124],{"class":1123},[1091,1148,1150,1153],{"class":1093,"line":1149},6,[1091,1151,1152],{"class":1100},"  --shm-size=16g",[1091,1154,1124],{"class":1123},[1091,1156,1158,1161,1164],{"class":1093,"line":1157},7,[1091,1159,1160],{"class":1100},"  --restart",[1091,1162,1163],{"class":1100}," unless-stopped",[1091,1165,1124],{"class":1123},[1091,1167,1168,1171,1174],{"class":1093,"line":308},[1091,1169,1170],{"class":1100},"  -p",[1091,1172,1173],{"class":1100}," 8000:8888",[1091,1175,1124],{"class":1123},[1091,1177,1179,1182,1185,1188,1191],{"class":1093,"line":1178},9,[1091,1180,1181],{"class":1100},"  -v",[1091,1183,1184],{"class":1100}," \u002Fhome\u002F",[1091,1186,1187],{"class":1123},"$USER",[1091,1189,1190],{"class":1100},"\u002F.cache\u002Fhuggingface:\u002Froot\u002F.cache\u002Fhuggingface",[1091,1192,1124],{"class":1123},[1091,1194,1196,1199,1202],{"class":1093,"line":1195},10,[1091,1197,1198],{"class":1100},"  -e",[1091,1200,1201],{"class":1100}," MODEL=Qwen\u002FQwen3-30B-A3B-Instruct-2507-FP8",[1091,1203,1124],{"class":1123},[1091,1205,1207,1209,1212,1216],{"class":1093,"line":1206},11,[1091,1208,1198],{"class":1100},[1091,1210,1211],{"class":1100}," PORT=",[1091,1213,1215],{"class":1214},"s9AOD","8888",[1091,1217,1124],{"class":1123},[1091,1219,1221,1223,1226,1229],{"class":1093,"line":1220},12,[1091,1222,1198],{"class":1100},[1091,1224,1225],{"class":1100}," GPU_MEMORY_UTIL=",[1091,1227,1228],{"class":1214},"0.85",[1091,1230,1124],{"class":1123},[1091,1232,1234,1236,1239,1242],{"class":1093,"line":1233},13,[1091,1235,1198],{"class":1100},[1091,1237,1238],{"class":1100}," MAX_MODEL_LEN=",[1091,1240,1241],{"class":1214},"32768",[1091,1243,1124],{"class":1123},[1091,1245,1247,1250],{"class":1093,"line":1246},14,[1091,1248,1249],{"class":1100},"  avarok\u002Fdgx-vllm-nvfp4-kernel:v22",[1091,1251,1252],{"class":1100}," serve\n",[10,1254,1255,1256,349],{},"First boot takes 10-20 minutes (model download + CUDA graph capture). After that, it auto-starts on reboot and serves an OpenAI-compatible API at ",[398,1257,1258],{},"http:\u002F\u002Flocalhost:8000\u002Fv1",[48,1260,1262],{"id":1261},"lessons-learned","Lessons Learned",[10,1264,1265,1268],{},[784,1266,1267],{},"1. Understand your bottleneck."," The Spark's 273 GB\u002Fs bandwidth determines everything. Once we understood this, the model selection became obvious, MoE with minimal active parameters.",[10,1270,1271,1274],{},[784,1272,1273],{},"2. Don't build from source if you don't have to."," Our manual vLLM build took hours of debugging across multiple sessions. The Avarok Docker image does everything better and in one command.",[10,1276,1277,1280],{},[784,1278,1279],{},"3. FP8 quantization is nearly free."," The jump from bf16 to FP8 nearly doubled throughput (30.3 to 51 tok\u002Fs on the same engine) with no perceptible quality difference for our use cases.",[10,1282,1283,1286],{},[784,1284,1285],{},"4. Stop Ollama first."," On one Spark, Ollama was consuming ~100GB of memory when we tried to install vLLM. The build process OOM-killed the machine. Disable competing inference servers before starting.",[10,1288,1289,1292,1293,1296,1297,349],{},[784,1290,1291],{},"5. Kernel updates break NVIDIA drivers."," DGX OS auto-updates the kernel, but the NVIDIA modules don't follow automatically. After a reboot, ",[398,1294,1295],{},"nvidia-smi"," may fail. The fix is ",[398,1298,1299],{},"sudo apt install linux-modules-nvidia-580-open-$(uname -r)",[10,1301,1302,1305],{},[784,1303,1304],{},"6. Community Docker images can be ahead of official ones."," The Avarok image runs vLLM v0.16.0rc2 with Blackwell fixes, months ahead of where NVIDIA's own builds are.",[48,1307,1309],{"id":1308},"whats-next","What's Next",[10,1311,1312,1313,1316],{},"Community results suggest AWQ 4-bit quantization can push the same model to ",[784,1314,1315],{},"82 tok\u002Fs",". NVIDIA's own NVFP4-quantized models (like Qwen3-Next-80B-A3B) report even better quality at ~67 tok\u002Fs average. As toolchain support matures, these numbers should keep improving.",[10,1318,1319],{},"For now, 51-54 tok\u002Fs with a 30B-parameter MoE model is fast enough for interactive coding assistance, document drafting, and general-purpose use, all running locally on a desktop machine.",[48,1321,1323],{"id":1322},"resources","Resources",[272,1325,1326,1332,1339,1346],{},[32,1327,1328],{},[277,1329,1331],{"href":943,"rel":1330},[281],"Avarok dgx-vllm Docker project",[32,1333,1334],{},[277,1335,1338],{"href":1336,"rel":1337},"https:\u002F\u002Fhuggingface.co\u002FQwen\u002FQwen3-30B-A3B-Instruct-2507-FP8",[281],"Qwen3-30B-A3B-FP8 on HuggingFace",[32,1340,1341],{},[277,1342,1345],{"href":1343,"rel":1344},"https:\u002F\u002Fwww.nvidia.com\u002Fen-us\u002Fproducts\u002Fworkstations\u002Fdgx-spark\u002F",[281],"NVIDIA DGX Spark product page",[32,1347,1348],{},[277,1349,1352],{"href":1350,"rel":1351},"https:\u002F\u002Fdocs.vllm.ai\u002F",[281],"vLLM documentation",[48,1354,1356],{"id":1355},"try-it-yourself","Try It Yourself",[10,1358,1359,1360,1365],{},"If you have a DGX Spark, the Docker approach takes about 20 minutes from zero to serving. Pull the Avarok image, run the container with the command above, and you're up. Reach out to us at ",[277,1361,1364],{"href":1362,"rel":1363},"https:\u002F\u002Fxrpl-commons.org",[281],"XRPL Commons"," if you want the full setup guide with troubleshooting details.",[1367,1368,1369],"style",{},"html pre.shiki code .s7eDp, html code.shiki .s7eDp{--shiki-default:#6F42C1}html pre.shiki code .sY4mW, html code.shiki .sY4mW{--shiki-default:#2B5581}html pre.shiki code .sR6ew, html code.shiki .sR6ew{--shiki-default:#24292EFF}html pre.shiki code .s9AOD, html code.shiki .s9AOD{--shiki-default:#1976D2}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}",{"title":298,"searchDepth":299,"depth":299,"links":1371},[1372,1373,1374,1375,1376,1377,1378,1379,1380,1381,1382],{"id":799,"depth":299,"text":800},{"id":826,"depth":299,"text":827},{"id":859,"depth":299,"text":860},{"id":896,"depth":299,"text":897},{"id":936,"depth":299,"text":937},{"id":974,"depth":299,"text":975},{"id":1077,"depth":299,"text":1078},{"id":1261,"depth":299,"text":1262},{"id":1308,"depth":299,"text":1309},{"id":1322,"depth":299,"text":1323},{"id":1355,"depth":299,"text":1356},"2026-03-10","TL;DR: We got a 30-billion-parameter LLM running at 51-54 tokens\u002Fsec on the NVIDIA DGX Spark by combining Mixture-of-Experts architecture, FP8 quantization, and a community Docker image that fixes Blackwell-specific issues. Here's what we learned.",{},"\u002Fideas\u002F2026-03-10-dgx-spark-vllm",{"title":778,"description":1384},"dgx-spark","ideas\u002F2026-03-10-dgx-spark-vllm","Standing up a 30B-parameter LLM at 50+ tok\u002Fs on the NVIDIA DGX Spark, the technical journey.",[324,1392,325,1393],"infrastructure","vllm","vWEib466-7AeFUyA1rtSOKNouV_slNSo20g-MtLGmAw",{"id":1396,"title":1397,"body":1398,"canonical_url":307,"chapter":1112,"date":1644,"description":1402,"extension":310,"featured":311,"image":307,"kind":312,"meta":1645,"navigation":314,"path":1646,"seo":1647,"series":317,"stem":1648,"summary":1649,"tags":1650,"work_slug":321,"__hash__":1651},"writing\u002Fideas\u002F2026-03-13-anton-03-domains-widen.md","Anton, chapter 3: Building the substrate",{"type":7,"value":1399,"toc":1635},[1400,1403,1406,1409,1412,1416,1419,1422,1425,1428,1431,1434,1437,1440,1443,1447,1450,1453,1456,1459,1462,1465,1468,1471,1475,1478,1481,1484,1487,1490,1493,1496,1499,1502,1505,1508,1512,1515,1518,1521,1524,1527,1530,1533,1536,1540,1543,1546,1549,1552,1555,1558,1561,1564,1567,1571,1574,1577,1580,1583,1586,1589,1592,1596,1599,1602,1605,1608,1611,1614,1617,1620,1623,1626,1629,1632],[10,1401,1402],{},"The week began with unfinished business.",[10,1404,1405],{},"Anton could search Doctolib, but he could not reliably open the appointment pages and tell us what was actually available.",[10,1407,1408],{},"Every attempted fix exposed another failure. Clicking the result produced stale browser references. Direct navigation timed out. More logging revealed that Cloudflare was identifying the browser as automated and blocking it entirely.",[10,1410,1411],{},"Eventually, the answer became clear: stop trying to force the same approach.",[48,1413,1415],{"id":1414},"a-browser-with-a-memory","A browser with a memory",[10,1417,1418],{},"The useful question was not, \"How do I bypass the block?\"",[10,1420,1421],{},"It was, \"What does a real browser have that Anton's browser does not?\"",[10,1423,1424],{},"The answer was continuity.",[10,1426,1427],{},"A normal browser accumulates cookies, history, authenticated sessions, preferences, and countless other small traces of use. Anton was arriving every time as a completely new and anonymous browser, which was itself suspicious.",[10,1429,1430],{},"I replaced the bundled headless browser with a real Chromium instance using persistent profiles. Each website could now retain its own identity between sessions.",[10,1432,1433],{},"Anti-detection settings helped, but they were not the real solution. The important change was allowing the browser to behave like a returning browser rather than a stranger appearing from nowhere.",[10,1435,1436],{},"Once it did, Cloudflare stopped blocking it.",[10,1438,1439],{},"The browser then moved into its own isolated service, with the persistent profiles stored in mounted volumes. Every domain that needed browser access (Doctolib, the building management portal, the consulate, or the general web) could use the same hardened infrastructure.",[10,1441,1442],{},"One browser service. One profile per site. One place to fix things.",[48,1444,1446],{"id":1445},"letting-the-model-read","Letting the model read",[10,1448,1449],{},"Fixing browser identity solved the access problem, but there was still a better way to extract the appointment information.",[10,1451,1452],{},"I had been treating the page as a structure to navigate: locate each card, click it, inspect the details, and translate the result into data.",[10,1454,1455],{},"But most of the information was already present on the list page as text.",[10,1457,1458],{},"Instead of traversing every element deterministically, I gave the page text to the model and asked it to return the fields I needed in a structured format.",[10,1460,1461],{},"One model call replaced a complicated browser workflow.",[10,1463,1464],{},"It was faster, cheaper, and less fragile because it did not fight the design of the website.",[10,1466,1467],{},"This is a pattern I keep rediscovering: when deterministic extraction becomes disproportionately expensive, move the problem up one level and let the model interpret the material.",[10,1469,1470],{},"The LLM is often the cheapest unit of work available. I should use it where it pays.",[48,1472,1474],{"id":1473},"movie-night","Movie night",[10,1476,1477],{},"Meanwhile, the media domain was becoming something the family might genuinely use.",[10,1479,1480],{},"The collection of individual media tools became a smaller intent-driven interface: check the status, search the library, watch something, or search for something new.",[10,1482,1483],{},"The logic also became more grounded in the actual state of the system. Rather than searching for episodes within an arbitrary fourteen-day window, Anton could look at the last episode already present in Plex and work forward from there.",[10,1485,1486],{},"Reason from state, not from an invented window.",[10,1488,1489],{},"Then came movie night.",[10,1491,1492],{},"Every Friday evening, Anton would choose two or three films and post them to the family group.",[10,1494,1495],{},"It was his first proactive message.",[10,1497,1498],{},"Until then, Anton had always been responding to someone. Movie night required him to initiate an interaction at the right moment, without being asked.",[10,1500,1501],{},"Getting the tone right took more work than the feature itself. Early versions ended by asking whether anyone wanted him to download one of the films. It was technically helpful, but it made Anton sound like a salesman trying to prolong the conversation.",[10,1503,1504],{},"That led to a new rule for scheduled messages: no filler, no unnecessary questions, and no attempt to manufacture engagement.",[10,1506,1507],{},"A proactive message should justify its interruption and then get out of the way.",[48,1509,1511],{"id":1510},"pluggable-domains","Pluggable domains",[10,1513,1514],{},"The most important structural change that week was making Anton's domains pluggable.",[10,1516,1517],{},"Until then, the parent agent knew about each domain individually. Every new capability required another explicit integration.",[10,1519,1520],{},"I replaced that with a small registration interface. Each domain could describe itself and expose its tools, while the parent built its available toolset dynamically from the modules that were present.",[10,1522,1523],{},"The parent no longer needed to understand calendars, media, research, or building management.",[10,1525,1526],{},"It only needed to know that it had tools.",[10,1528,1529],{},"A few days later, I added the building management portal as a new domain in three lines.",[10,1531,1532],{},"That is usually how you know an abstraction has found the right boundary: the next feature becomes almost disappointingly easy.",[10,1534,1535],{},"When the marginal cost of adding a domain approaches zero, you have found the seam.",[48,1537,1539],{"id":1538},"collections","Collections",[10,1541,1542],{},"The same pattern appeared in Anton's memory.",[10,1544,1545],{},"Wine had begun as its own typed database table. But the household would eventually want to remember other things: restaurants, books, contacts, recipes, places, and whatever else emerged.",[10,1547,1548],{},"Building a separate data model and toolset for every kind of collection would create endless special cases.",[10,1550,1551],{},"So collections became a shared substrate.",[10,1553,1554],{},"A generic table stored items as JSON, with the same tools for adding, searching, and updating them. Each collection carried a schema describing the shape of its entries.",[10,1556,1557],{},"The schema was included directly in the tool description. The model could read it and understand that a wine has one set of fields while a restaurant has another.",[10,1559,1560],{},"The tools remained generic, but their behaviour adapted to the collection.",[10,1562,1563],{},"Wine became the first migration. The dedicated table disappeared.",[10,1565,1566],{},"One substrate, many collections.",[48,1568,1570],{"id":1569},"skills-as-data","Skills as data",[10,1572,1573],{},"Reusable prompts followed the same path.",[10,1575,1576],{},"There were already instructions I expected to use repeatedly: conduct a weekly review, create a Google document with a particular structure, or perform some other familiar workflow.",[10,1578,1579],{},"Copying the same instructions into chat each time made no sense.",[10,1581,1582],{},"I added skills as records in the database: named, reusable prompts that could be listed, edited, and invoked without changing the application.",[10,1584,1585],{},"The word \"skill\" unfortunately overlapped with another part of the codebase, but the underlying decision was right.",[10,1587,1588],{},"A useful instruction should be an artifact I can inspect and modify, not a string buried somewhere in the code.",[10,1590,1591],{},"As with Anton's identity, the important move was from code to data.",[48,1593,1595],{"id":1594},"different-people-different-anton","Different people, different Anton",[10,1597,1598],{},"A few smaller capabilities also found their place.",[10,1600,1601],{},"Weather gained a direct integration because a simple forecast should not require a general web search. Anton could create Google Docs and share them automatically because the family already worked in Drive.",[10,1603,1604],{},"Every new collection shipped with regression tests. The quality rule established the previous weekend was becoming habitual rather than aspirational.",[10,1606,1607],{},"Anton also gained his first user-specific behaviour.",[10,1609,1610],{},"For Clara, his responses became simpler, and he knew when to escalate something to me.",[10,1612,1613],{},"The same assistant should not speak identically to every member of a household. People have different ages, needs, permissions, and expectations.",[10,1615,1616],{},"Personalization should not mean only remembering preferences. It should also mean adapting how the system behaves.",[10,1618,1619],{},"By the end of the week, Anton had gained more capabilities, but the architecture itself had become smaller.",[10,1621,1622],{},"The browser was one service rather than several browser implementations. Domains registered themselves instead of being wired individually. Collections shared one substrate. Skills were data rather than embedded prompts.",[10,1624,1625],{},"The instinct underneath each change was the same:",[10,1627,1628],{},"When something is going to repeat, make it a substrate rather than a special case.",[10,1630,1631],{},"Every time I made that choice, the system became simpler and the next feature became cheaper.",[10,1633,1634],{},"That is the trade I want to keep making.",{"title":298,"searchDepth":299,"depth":299,"links":1636},[1637,1638,1639,1640,1641,1642,1643],{"id":1414,"depth":299,"text":1415},{"id":1445,"depth":299,"text":1446},{"id":1473,"depth":299,"text":1474},{"id":1510,"depth":299,"text":1511},{"id":1538,"depth":299,"text":1539},{"id":1569,"depth":299,"text":1570},{"id":1594,"depth":299,"text":1595},"2026-03-13",{},"\u002Fideas\u002F2026-03-13-anton-03-domains-widen",{"title":1397,"description":1402},"ideas\u002F2026-03-13-anton-03-domains-widen","Hardening the browser, lifting the work to the LLM, and turning domains into pluggable substrates.",[321,322,323],"PPt94eY6Y7ufZl9EWEEBTUmrjEHxw8Bg8FnSdOKgV0c",{"id":1653,"title":1654,"body":1655,"canonical_url":307,"chapter":1127,"date":1905,"description":1659,"extension":310,"featured":311,"image":307,"kind":312,"meta":1906,"navigation":314,"path":1907,"seo":1908,"series":317,"stem":1909,"summary":1910,"tags":1911,"work_slug":321,"__hash__":1912},"writing\u002Fideas\u002F2026-03-16-anton-04-plumbing-matures.md","Anton, chapter 4: Explicit contracts",{"type":7,"value":1656,"toc":1895},[1657,1660,1663,1666,1670,1673,1676,1679,1682,1685,1688,1691,1694,1697,1700,1704,1707,1710,1713,1716,1719,1722,1725,1728,1732,1735,1738,1741,1744,1747,1750,1754,1761,1764,1767,1770,1773,1776,1780,1783,1786,1789,1792,1799,1802,1805,1808,1812,1815,1818,1821,1824,1827,1830,1833,1837,1840,1843,1846,1849,1852,1855,1858,1861,1864,1867,1870,1873,1876,1879,1881],[10,1658,1659],{},"The week began with a one-line configuration change.",[10,1661,1662],{},"Anton's local inference moved from Ollama to vLLM. The diff was almost nothing; the serving architecture underneath it was not. Ollama had been ideal for getting started. vLLM offered the concurrency and batching I wanted beneath a system that several people might rely on at once.",[10,1664,1665],{},"It was the first of many changes that week with the same shape: take something Anton was doing implicitly and give it an explicit contract.",[48,1667,1669],{"id":1668},"store-generously-retrieve-narrowly","Store generously, retrieve narrowly",[10,1671,1672],{},"Memory was the first system to outgrow its original design.",[10,1674,1675],{},"The initial version was deliberately simple: Postgres rows tagged by user. That was enough to learn what memory needed to become, but not enough to retrieve the right information reliably.",[10,1677,1678],{},"I still resisted adding embeddings. At the scale of a family knowledge base, trigram matching was sufficient, and an embedding pipeline would have introduced cost and machinery before either was justified.",[10,1680,1681],{},"Instead, retrieval became more deliberate.",[10,1683,1684],{},"Every fact gained provenance, so it could be traced to its source. Queries were biased toward the relevant domain: calendar questions favoured calendar facts, media questions favoured media facts. Short conversational requests received a narrow recall window; research-like questions received a larger one. A working-memory scratchpad carried the immediate conversation, while an LLM reformulated queries before search because people rarely ask for information using the same words in which it was stored.",[10,1686,1687],{},"Later, I added summaries of past conversations, but only when a request had temporal intent.",[10,1689,1690],{},"Always injecting episodic memory feels helpful in theory. In practice, it is context bloat disguised as intelligence.",[10,1692,1693],{},"The principle became:",[10,1695,1696],{},"Store generously. Retrieve narrowly.",[10,1698,1699],{},"Or, more technically: filter at load time, not write time.",[48,1701,1703],{"id":1702},"scheduling-becomes-a-domain","Scheduling becomes a domain",[10,1705,1706],{},"Scheduling had started as a thin wrapper around BullMQ. That was no longer enough.",[10,1708,1709],{},"Half the things I wanted Anton to do were recurring: morning briefings, weekly reviews, reminders to call someone, regular checks that should happen without being requested each time.",[10,1711,1712],{},"Schedules gained time zones, duplicate merging, execution history, manual runs, silent mode, and a UI that made them understandable. They stopped being plumbing hidden behind other domains and became a domain in their own right.",[10,1714,1715],{},"At the same time, LiteLLM became the gateway in front of every model provider.",[10,1717,1718],{},"One system should decide where model calls go. Individual agents should not need to know which provider currently serves them, how it is authenticated, or what fallback should be used.",[10,1720,1721],{},"I gave the models codenames: Sunny, Haiku, Oscar, Gandalf, Gizmo, Gatsby, Merlin, Gustav.",[10,1723,1724],{},"It felt mildly silly until the first configuration change. Provider names, versions, prices, and capabilities move constantly. A stable internal name allows everything above the gateway to remain still.",[10,1726,1727],{},"Indirection is useful when it protects the rest of the system from change.",[48,1729,1731],{"id":1730},"naming-compounds","Naming compounds",[10,1733,1734],{},"A naming mistake from the previous week also caught up with me.",[10,1736,1737],{},"I had used \"skill\" for two different things: a typed function in code and a reusable prompt stored in the database.",[10,1739,1740],{},"Both meanings made sense in isolation. Together, they made every conversation about the system harder.",[10,1742,1743],{},"The reusable templates became prompts. A skill became one thing only: a typed capability with a runtime contract.",[10,1745,1746],{},"The cleanup took a few hours. Leaving the ambiguity in place would have charged interest indefinitely.",[10,1748,1749],{},"Naming is architecture because names determine whether people can reason about the system without first translating it in their heads.",[48,1751,1753],{"id":1752},"one-way-to-run-a-skill","One way to run a skill",[10,1755,1756,1757,1760],{},"The new ",[398,1758,1759],{},"defineSkill()"," contract required every skill to declare the same things: its inputs, outputs, scopes, and handler.",[10,1762,1763],{},"Then the skill runner became its own service.",[10,1765,1766],{},"That separation gave me hot reload, per-skill metrics, cleaner scope isolation, and a path toward sandboxing later. More importantly, every domain began invoking skills through the same entry point.",[10,1768,1769],{},"One contract. One runner. One way to add a capability.",[10,1771,1772],{},"This is the kind of work that can feel slower than adding features. Nothing visible happens for the family when two invocation paths become one.",[10,1774,1775],{},"But every future feature gets cheaper.",[48,1777,1779],{"id":1778},"an-honest-record-of-execution","An honest record of execution",[10,1781,1782],{},"Traces became durable too.",[10,1784,1785],{},"The first trace viewer read checkpoints from Redis. That was useful while debugging a live request and nearly useless afterward. Execution traces moved into Postgres, where they survived failures and could be searched from the UI.",[10,1787,1788],{},"I enforced one invariant: one trace per request.",[10,1790,1791],{},"Once that was true, Anton's behaviour became queryable. I could inspect what happened last night or last week rather than trying to reproduce it from memory.",[10,1793,1794,1795,1798],{},"Permissions followed the same pattern. Filtering moved into ",[398,1796,1797],{},"runAgent()",", the one path every caller had to use.",[10,1800,1801],{},"Before that, each caller was responsible for applying permissions correctly. Scheduled jobs, mesh calls, direct invocations, and the parent agent could subtly diverge.",[10,1803,1804],{},"A security rule enforced in four places is four opportunities to forget it.",[10,1806,1807],{},"The right place for a rule is the chokepoint every path must cross.",[48,1809,1811],{"id":1810},"give-the-model-a-smaller-menu","Give the model a smaller menu",[10,1813,1814],{},"The architecture view in the dashboard began drawing itself from the runtime configuration. For the first time, I could see Anton's shape rather than infer it from files and logs.",[10,1816,1817],{},"The picture revealed a problem: the parent agent had 63 tools.",[10,1819,1820],{},"The model was being asked to choose correctly from a menu no person would want to read.",[10,1822,1823],{},"I replaced the flat toolset with ten subsystem delegates. The parent now routed and synthesised; each delegate handled a much smaller set of capabilities.",[10,1825,1826],{},"The improvement was immediate.",[10,1828,1829],{},"The lesson from the calendar agent had scaled to the whole system: models are good at selecting from a small, coherent menu and unreliable when everything is presented at once.",[10,1831,1832],{},"A better prompt would not solve a badly shaped choice.",[48,1834,1836],{"id":1835},"behaviour-becomes-configuration","Behaviour becomes configuration",[10,1838,1839],{},"Output validation entered the agent loop, along with deterministic confirmations that a tool had actually completed.",[10,1841,1842],{},"This eliminated one of the worst chat failure modes: the tool succeeds, but Anton says nothing.",[10,1844,1845],{},"Then came prime directives, a small set of rules enforced above any individual prompt. The first version was too verbose, so each directive was reduced to a single line.",[10,1847,1848],{},"The precedence became clear: directives, then agent prompt, then prompt template, then user message.",[10,1850,1851],{},"The non-negotiable rules belonged in code. Everything else should remain editable.",[10,1853,1854],{},"That led to the final change of the week: every agent prompt moved into the database.",[10,1856,1857],{},"No hardcoded fallback. One row per agent, editable from the UI and versioned.",[10,1859,1860],{},"Anton's behaviour stopped being something I had to deploy and became something I could configure, test, and roll back.",[10,1862,1863],{},"By Sunday night, inference, memory, schedules, skills, traces, permissions, delegation, validation, and prompts all had explicit shapes and single entry points.",[10,1865,1866],{},"The week had been almost entirely plumbing.",[10,1868,1869],{},"But plumbing is what allows everything above it to change without flooding the house.",[10,1871,1872],{},"The lesson I took forward was simple:",[10,1874,1875],{},"Turn implicit conventions into explicit contracts.",[10,1877,1878],{},"Once a thing has a shape on disk and one path through the code, the rest of the system becomes much easier to change.",[48,1880,270],{"id":269},[272,1882,1883,1888],{},[32,1884,1885],{},[277,1886,1352],{"href":1350,"rel":1887},[281],[32,1889,1890],{},[277,1891,1894],{"href":1892,"rel":1893},"https:\u002F\u002Fdocs.litellm.ai\u002F",[281],"LiteLLM documentation",{"title":298,"searchDepth":299,"depth":299,"links":1896},[1897,1898,1899,1900,1901,1902,1903,1904],{"id":1668,"depth":299,"text":1669},{"id":1702,"depth":299,"text":1703},{"id":1730,"depth":299,"text":1731},{"id":1752,"depth":299,"text":1753},{"id":1778,"depth":299,"text":1779},{"id":1810,"depth":299,"text":1811},{"id":1835,"depth":299,"text":1836},{"id":269,"depth":299,"text":270},"2026-03-16",{},"\u002Fideas\u002F2026-03-16-anton-04-plumbing-matures",{"title":1654,"description":1659},"ideas\u002F2026-03-16-anton-04-plumbing-matures","A week of turning implicit conventions into explicit contracts: memory retrieval, schedules, skills, traces, and prompts as data.",[321,322,323,1392],"M3-XU6JyfZ4zmO1nz0mnEbc8pNcwseSSACPc3MiCfKk",{"id":1914,"title":1915,"body":1916,"canonical_url":307,"chapter":1138,"date":2149,"description":1920,"extension":310,"featured":311,"image":307,"kind":312,"meta":2150,"navigation":314,"path":2151,"seo":2152,"series":317,"stem":2153,"summary":2154,"tags":2155,"work_slug":321,"__hash__":2157},"writing\u002Fideas\u002F2026-03-23-anton-05-langgraph-excised.md","Anton, chapter 5: Two primitives",{"type":7,"value":1917,"toc":2141},[1918,1921,1924,1927,1930,1933,1936,1939,1942,1946,1949,1952,1955,1958,1961,1964,1967,1970,1974,1977,1983,1986,1989,1992,1995,1998,2001,2004,2008,2013,2016,2019,2022,2038,2048,2051,2054,2058,2061,2064,2067,2070,2073,2076,2079,2082,2086,2089,2092,2095,2098,2101,2105,2108,2111,2114,2117,2120,2123,2126,2129,2132,2135,2138],[10,1919,1920],{},"Two weeks earlier, LangGraph had been exactly the right choice.",[10,1922,1923],{},"It gave Anton structure, checkpointing, observability, and a clean way to organise each domain as a subgraph. It helped me move quickly while I was still discovering what the system wanted to be.",[10,1925,1926],{},"By Monday morning, it had become the system rather than a tool inside it.",[10,1928,1929],{},"Every agent was a LangGraph subgraph. The parent was a state machine. Conversation history used LangChain message types. The trace viewer parsed LangGraph checkpoints. Skills were wrapped as LangChain tools.",[10,1931,1932],{},"There were now three representations of the same architecture: the framework graph, Anton's own domain registry, and the dashboard view. They drifted, and I reconciled them by hand.",[10,1934,1935],{},"The framework had not become bad.",[10,1937,1938],{},"Its value had simply fallen below its friction.",[10,1940,1941],{},"That is the moment to remove an abstraction.",[48,1943,1945],{"id":1944},"skills-and-agents","Skills and agents",[10,1947,1948],{},"I reduced Anton to two primitives.",[10,1950,1951],{},"A skill performs a bounded operation with a runtime contract.",[10,1953,1954],{},"An agent reasons, selects skills, and loops until it has completed the request.",[10,1956,1957],{},"Nothing else gets a special category.",[10,1959,1960],{},"No pseudo-agents. No classify-and-dispatch pipelines disguised as agents. No bespoke endpoints that bypass the common runtime.",[10,1962,1963],{},"When something does not need reasoning, it is a skill. When it does, it is an agent.",[10,1965,1966],{},"Anything that cannot fit either shape has to justify why.",[10,1968,1969],{},"The rule was austere, and that was the point.",[48,1971,1973],{"id":1972},"the-small-runtime-underneath","The small runtime underneath",[10,1975,1976],{},"Then LangGraph came out.",[10,1978,1979,1980,1982],{},"I replaced it with ",[398,1981,1797],{},": a few hundred lines containing only what Anton actually needed.",[10,1984,1985],{},"An LLM call. Tool dispatch. A loop limit. Trace emission. Permission filtering. Output validation.",[10,1987,1988],{},"That was the runtime.",[10,1990,1991],{},"I could hold it in my head.",[10,1993,1994],{},"This did not mean LangGraph had been a mistake. Without it, I might not have learned which parts were essential. Frameworks are often most valuable while the problem is still being discovered.",[10,1996,1997],{},"But once the underlying shape becomes clear, carrying the discovery apparatus forever is optional.",[10,1999,2000],{},"The codebase changed vocabulary with the architecture. \"Graph\" became \"agent\" in package names, files, documentation, and the UI. A rule went into memory: no graph terminology.",[10,2002,2003],{},"Words that once described the implementation had become misleading descriptions of the system.",[48,2005,2007],{"id":2006},"uniform-from-the-outside","Uniform from the outside",[10,2009,2010,2011,349],{},"Every agent now had the same shape: a thin function passing input and configuration to ",[398,2012,1797],{},[10,2014,2015],{},"Every delegate did the same thing: call an agent and return its result.",[10,2017,2018],{},"Schedules, direct invocation, the parent, and the dashboard all saw one surface.",[10,2020,2021],{},"Uniformity from the outside made experimentation inside much cheaper.",[10,2023,2024,2025,2028,2029,2028,2032,2028,2035,349],{},"Skill names were standardised to verb_entity: ",[398,2026,2027],{},"get_event",", ",[398,2030,2031],{},"create_event",[398,2033,2034],{},"update_event",[398,2036,2037],{},"list_events",[10,2039,2040,2041,2044,2045,2047],{},"Aliases and one-off variants were folded back into canonical skills. ",[398,2042,2043],{},"update_event_by_title",", for example, became a supported path through ",[398,2046,2034],{}," rather than a second tool the model had to distinguish.",[10,2049,2050],{},"The web domain disappeared entirely. It duplicated parts of documents and research, and without the graph architecture making every branch look like a domain, there was no reason to preserve it.",[10,2052,2053],{},"A coherent tool surface is easier for people to understand and easier for a model to use.",[48,2055,2057],{"id":2056},"replication-exposes-the-next-problem","Replication exposes the next problem",[10,2059,2060],{},"With the runtime simplified, I built a replication engine.",[10,2062,2063],{},"The framing was a Von Neumann probe: clone the entire Anton stack onto another server with one command.",[10,2065,2066],{},"The mechanism was deliberately unromantic (rsync, Docker Compose, and a seed process) but it created three useful properties.",[10,2068,2069],{},"A household could run its own Anton. The Spark could fail without becoming irreplaceable. And I could create a disposable clone for changes that would otherwise make me nervous.",[10,2071,2072],{},"Replication immediately exposed the next weak point: secrets.",[10,2074,2075],{},"Vaultwarden authentication did not clone cleanly. A new instance came up without the credentials required to be useful, and fixing it required manual work on every machine.",[10,2077,2078],{},"A clone that needs hand repair is not really a clone.",[10,2080,2081],{},"I left the problem open, but now it was visible.",[48,2083,2085],{"id":2084},"transport-is-not-intelligence","Transport is not intelligence",[10,2087,2088],{},"Telegram arrived the same week.",[10,2090,2091],{},"It used the same backend, agents, and skills as WhatsApp. The bridge merely translated incoming messages into jobs and outbound responses back into Telegram.",[10,2093,2094],{},"The agent runtime did not know or care which transport a message came from.",[10,2096,2097],{},"That small integration validated a decision from the first day: transport belongs at the edge.",[10,2099,2100],{},"A new way of speaking to Anton should not require a new Anton.",[48,2102,2104],{"id":2103},"memory-must-know-who-is-asking","Memory must know who is asking",[10,2106,2107],{},"Two retrieval fixes closed the week.",[10,2109,2110],{},"Domain provenance began affecting ranking, so calendar queries preferred calendar-derived facts and media questions preferred media-derived facts.",[10,2112,2113],{},"More importantly, document facts stopped leaking between users.",[10,2115,2116],{},"A family assistant serves several people who share some context and keep other context private. Memory cannot treat \"the household\" as one undifferentiated user.",[10,2118,2119],{},"Knowing a fact and knowing who is allowed to recall it are part of the same operation.",[10,2121,2122],{},"By Thursday night, Anton had two primitives, one runtime, one naming convention, a replication path, and a second transport.",[10,2124,2125],{},"The codebase looked like what I had wanted two weeks earlier but could not yet have described.",[10,2127,2128],{},"That is one of the strange things about architecture: the clean version is often visible only after you have built the complicated one.",[10,2130,2131],{},"The week's lesson was not \"avoid frameworks.\"",[10,2133,2134],{},"It was:",[10,2136,2137],{},"Keep an abstraction only while it pays more than it costs.",[10,2139,2140],{},"LangGraph had paid handsomely. Then it stopped. Removing it was not a rejection of the earlier decision; it was the consequence of what that decision had taught me.",{"title":298,"searchDepth":299,"depth":299,"links":2142},[2143,2144,2145,2146,2147,2148],{"id":1944,"depth":299,"text":1945},{"id":1972,"depth":299,"text":1973},{"id":2006,"depth":299,"text":2007},{"id":2056,"depth":299,"text":2057},{"id":2084,"depth":299,"text":2085},{"id":2103,"depth":299,"text":2104},"2026-03-23",{},"\u002Fideas\u002F2026-03-23-anton-05-langgraph-excised",{"title":1915,"description":1920},"ideas\u002F2026-03-23-anton-05-langgraph-excised","Removing LangGraph once its friction exceeded its value, and reducing the system to skills and agents.",[321,322,323,2156],"architecture","O3LORFXZoRAdarw44Wk05RRpII3eSF0N-Yofo9znGHk",{"id":2159,"title":2160,"body":2161,"canonical_url":307,"chapter":1149,"date":2433,"description":2165,"extension":310,"featured":311,"image":307,"kind":312,"meta":2434,"navigation":314,"path":2435,"seo":2436,"series":317,"stem":2437,"summary":2438,"tags":2439,"work_slug":321,"__hash__":2440},"writing\u002Fideas\u002F2026-03-27-anton-06-mesh-sandbox.md","Anton, chapter 6: Two boundaries",{"type":7,"value":2162,"toc":2423},[2163,2166,2169,2172,2175,2178,2181,2184,2187,2191,2194,2197,2200,2203,2206,2209,2212,2216,2219,2222,2225,2228,2231,2234,2237,2240,2243,2246,2249,2253,2256,2259,2262,2265,2268,2275,2278,2282,2285,2288,2291,2294,2297,2300,2303,2306,2309,2315,2318,2322,2325,2332,2335,2338,2341,2344,2347,2350,2354,2357,2360,2363,2366,2369,2372,2375,2378,2382,2385,2388,2391,2394,2397,2400,2403,2406,2409,2412,2414],[10,2164,2165],{},"Four days produced the densest stretch of the project so far.",[10,2167,2168],{},"By the end of them, Anton could discover other Anton instances, run skills in a sandbox, browse unfamiliar websites, manage a permission-aware family vault, and review his own failures.",[10,2170,2171],{},"Those features looked unrelated until a security model emerged beneath them.",[10,2173,2174],{},"Anton had two boundaries.",[10,2176,2177],{},"The agent boundary determines who may ask for an action.",[10,2179,2180],{},"The skill boundary determines what the code performing that action is allowed to do.",[10,2182,2183],{},"One governs authority. The other governs capability.",[10,2185,2186],{},"Neither can replace the other.",[48,2188,2190],{"id":2189},"a-mesh-of-antons","A mesh of Antons",[10,2192,2193],{},"I wanted Anton instances to find one another without becoming one shared system.",[10,2195,2196],{},"They should not share a database, secrets, or unrestricted access. They should discover one another and forward specific calls according to the relationship between them.",[10,2198,2199],{},"I called the protocol SCUT: Symmetric Cluster Universal Transport.",[10,2201,2202],{},"Every node had the same shape. Probes handled discovery, heartbeats handled liveness, and an invocation forwarder carried requests between instances.",[10,2204,2205],{},"The important idea was that the instance itself was the identity, while the relationship between two instances defined the available permissions.",[10,2207,2208],{},"A clone serving another household might be allowed to ask my Anton for a media search without seeing the family vault, wine collection, or Plex credentials.",[10,2210,2211],{},"Federation should be relationships, not shared infrastructure.",[48,2213,2215],{"id":2214},"the-skill-boundary","The skill boundary",[10,2217,2218],{},"The existing skill runner was a Node service with no meaningful runtime isolation.",[10,2220,2221],{},"A skill could inspect every environment variable, execute arbitrary commands, read the filesystem, or connect anywhere on the network.",[10,2223,2224],{},"That was tolerable while everything was personal code on one private server.",[10,2226,2227],{},"It was not tolerable once other instances could forward invocations.",[10,2229,2230],{},"I rewrote the runner on Deno.",[10,2232,2233],{},"Each skill now ran in a Worker with only the permissions it declared: selected environment variables, specific network hosts, and explicit filesystem paths.",[10,2235,2236],{},"Nothing more.",[10,2238,2239],{},"The rewrite was painful because Deno's stricter execution model revealed every assumption Node had quietly permitted: loose imports, implicit package resolution, hidden environment dependencies, broad filesystem access.",[10,2241,2242],{},"Each failure was inconvenient and useful.",[10,2244,2245],{},"The runtime was showing me what each skill actually required.",[10,2247,2248],{},"The result was a capability boundary the code could not cross merely because an agent asked it to.",[48,2250,2252],{"id":2251},"secrets-belong-at-the-call-site","Secrets belong at the call site",[10,2254,2255],{},"Vaultwarden came out next.",[10,2257,2258],{},"Secrets moved into an encrypted Postgres table: one system to back up, one system that survived cloning, one place to inspect and edit from the dashboard.",[10,2260,2261],{},"But storage was only half the problem.",[10,2263,2264],{},"A secret must reach the skill that needs it without becoming visible to everything else.",[10,2266,2267],{},"The parent process environment could not safely contain all secrets. A Worker with no requested environment access might still exploit a mistake in the surrounding process if the values were already present there.",[10,2269,2270,2271,2274],{},"Secrets therefore travelled to the Worker through ",[398,2272,2273],{},"postMessage",", at the moment of invocation.",[10,2276,2277],{},"The sandbox is only as honest as the boundary you actually defend.",[48,2279,2281],{"id":2280},"explore-agentic-build-deterministic","Explore agentic, build deterministic",[10,2283,2284],{},"The browser was the next repeated problem.",[10,2286,2287],{},"Doctolib, the building management portal, and the consulate monitor each had their own Playwright script. The code duplicated intent and relied on selectors that broke whenever a page changed.",[10,2289,2290],{},"I replaced them with one generic browser agent capable of navigating, clicking, typing, evaluating, and taking screenshots.",[10,2292,2293],{},"The LLM could explore a page it had never seen and find a workable path.",[10,2295,2296],{},"That did not mean the LLM should execute the same path forever.",[10,2298,2299],{},"A pattern emerged:",[10,2301,2302],{},"Explore agentic. Build deterministic.",[10,2304,2305],{},"The browser agent is excellent at scouting an unfamiliar interface. Once a workflow becomes stable and repetitive, the result should be encoded directly whenever possible.",[10,2307,2308],{},"The model discovers the path. Software records it.",[10,2310,940,2311,2314],{},[398,2312,2313],{},"request_input"," mechanism also found its final shape here. Any running tool could pause, ask a person for a code or decision, and resume with the answer.",[10,2316,2317],{},"Human participation stopped being a special case inside Doctolib and became a general runtime primitive.",[48,2319,2321],{"id":2320},"a-vault-with-visibility","A vault with visibility",[10,2323,2324],{},"The family vault introduced a different kind of boundary.",[10,2326,2327,2328,2331],{},"Documents could be marked family or personal, with explicit ",[398,2329,2330],{},"visibleTo"," overrides. The answer to \"who can see this?\" lived on the document itself rather than being inferred through a complicated role graph.",[10,2333,2334],{},"Vision extraction and fact generation followed. Expiry dates could become reminders; scanned documents could become searchable facts.",[10,2336,2337],{},"But the cost pattern from the browser repeated.",[10,2339,2340],{},"Running vision over every document would be slow and wasteful. The model should scout difficult files and help establish the extraction path, not become the default parser for everything.",[10,2342,2343],{},"The LLM is the scout, not the worker.",[10,2345,2346],{},"The same vault absorbed a family Notion workspace. A few awkward imports were resolved directly rather than asking the model to keep improvising around a dependency problem.",[10,2348,2349],{},"Sometimes rewriting one import is cheaper than adding intelligence.",[48,2351,2353],{"id":2352},"anton-reviews-anton","Anton reviews Anton",[10,2355,2356],{},"The final change was the one I had been waiting to build.",[10,2358,2359],{},"Every night, Anton reviewed the previous day's execution traces, grouped failures, and created GitHub issues for the clusters it found.",[10,2361,2362],{},"Labels drove a simple state machine:",[10,2364,2365],{},"needs-triage, then ready-to-fix, then fixed-locally, then deployed.",[10,2367,2368],{},"This was only possible because the traces were complete.",[10,2370,2371],{},"Without traces, Anton would have judged its final answers. With traces, it could inspect which tools were called, what they returned, where the reasoning stopped, and what actually failed.",[10,2373,2374],{},"Self-reflection is only useful when the substrate is honest.",[10,2376,2377],{},"A system cannot meaningfully critique a story it invented after the fact.",[48,2379,2381],{"id":2380},"two-questions","Two questions",[10,2383,2384],{},"Several smaller changes reinforced the same architecture.",[10,2386,2387],{},"All agents converged on one delegate registry. The dashboard showed only agents the selected user could invoke. Prompt-injection handling gained explicit trust markers and a risk trail. Directives became standing instructions that would need periodic pruning. The consulate monitor began as an observation over six months and narrowed to four only after seeing the data.",[10,2389,2390],{},"Observe first. Tune second.",[10,2392,2393],{},"By the end of the four days, Anton could communicate across an authenticated mesh. Skills ran with minimum permissions. Secrets were decrypted only where they were needed. Websites could be explored by a general browser agent. Documents carried their own visibility. Every night, the system inspected its own failures.",[10,2395,2396],{},"The architecture rested on two questions:",[10,2398,2399],{},"Who is allowed to ask for this?",[10,2401,2402],{},"What is the code allowed to do?",[10,2404,2405],{},"The first question belongs to agents and relationships.",[10,2407,2408],{},"The second belongs to skills and the runtime.",[10,2410,2411],{},"Security became much easier to reason about once I stopped asking one boundary to answer both.",[48,2413,270],{"id":269},[272,2415,2416],{},[32,2417,2418],{},[277,2419,2422],{"href":2420,"rel":2421},"https:\u002F\u002Fdocs.deno.com\u002Fruntime\u002Ffundamentals\u002Fsecurity\u002F",[281],"Deno security and permissions",{"title":298,"searchDepth":299,"depth":299,"links":2424},[2425,2426,2427,2428,2429,2430,2431,2432],{"id":2189,"depth":299,"text":2190},{"id":2214,"depth":299,"text":2215},{"id":2251,"depth":299,"text":2252},{"id":2280,"depth":299,"text":2281},{"id":2320,"depth":299,"text":2321},{"id":2352,"depth":299,"text":2353},{"id":2380,"depth":299,"text":2381},{"id":269,"depth":299,"text":270},"2026-03-27",{},"\u002Fideas\u002F2026-03-27-anton-06-mesh-sandbox",{"title":2160,"description":2165},"ideas\u002F2026-03-27-anton-06-mesh-sandbox","Federation, a Deno sandbox, encrypted secrets, and a security model built on two questions: who may ask, and what may the code do.",[321,322,323],"MHMR7LWeD9vnqbj_nc5nflH836j1DRGSh_cu-XdRIRM",{"id":2442,"title":2443,"body":2444,"canonical_url":307,"chapter":1157,"date":2685,"description":2448,"extension":310,"featured":311,"image":307,"kind":312,"meta":2686,"navigation":314,"path":2687,"seo":2688,"series":317,"stem":2689,"summary":2690,"tags":2691,"work_slug":321,"__hash__":2692},"writing\u002Fideas\u002F2026-03-31-anton-07-cost-syndic.md","Anton, chapter 7: Becoming operational",{"type":7,"value":2445,"toc":2677},[2446,2449,2452,2455,2458,2461,2464,2467,2471,2474,2477,2480,2483,2486,2489,2492,2495,2498,2501,2504,2508,2511,2514,2517,2520,2523,2526,2530,2533,2536,2539,2542,2545,2548,2551,2554,2557,2560,2563,2566,2570,2573,2576,2579,2582,2585,2588,2591,2594,2597,2601,2604,2607,2610,2613,2616,2619,2623,2626,2629,2632,2635,2638,2641,2644,2647,2650,2653,2656,2659,2662,2665,2668,2671,2674],[10,2447,2448],{},"At some point, a project stops asking whether it works and starts asking different questions.",[10,2450,2451],{},"What does it cost?",[10,2453,2454],{},"What happens when a provider fails?",[10,2456,2457],{},"What information can escape?",[10,2459,2460],{},"What does the system do when nobody is watching?",[10,2462,2463],{},"Anton crossed that threshold over the next two weeks.",[10,2465,2466],{},"Three threads defined the period: cost became part of the architecture, the syndic domain tested the system against a large real-world problem, and a heartbeat gave Anton a quiet form of awareness between requests.",[48,2468,2470],{"id":2469},"cost-is-part-of-correctness","Cost is part of correctness",[10,2472,2473],{},"Every model call gained a token budget.",[10,2475,2476],{},"History was trimmed before the provider request rather than waiting for an API error. Calls carried attribution metadata: agent, domain, user, and request ID. Selected requests were duplicated to cheaper models in the background so I could compare results without affecting production.",[10,2478,2479],{},"Anton needed to know what it cost.",[10,2481,2482],{},"Not only for budgeting, but because sudden changes in usage are often evidence of a behavioural regression.",[10,2484,2485],{},"Memory consolidation moved from every write to a nightly batch. Importance scoring decided what survived, and a dashboard showed the health of the memory system.",[10,2487,2488],{},"The earlier design had been spending money on work no one needed to happen immediately.",[10,2490,2491],{},"Then an OpenAI billing problem stopped requests in the middle of the day.",[10,2493,2494],{},"Automatic fallback to other providers followed.",[10,2496,2497],{},"The lesson was direct:",[10,2499,2500],{},"If a provider is a single point of failure, your reliability is their reliability.",[10,2502,2503],{},"A fallback should be part of the normal architecture, not an emergency patch.",[48,2505,2507],{"id":2506},"one-source-of-truth","One source of truth",[10,2509,2510],{},"I removed the last hardcoded prompt fallbacks.",[10,2512,2513],{},"Every agent prompt already lived in the database. Keeping stale copies in code created two possible truths, and the system could silently choose the wrong one.",[10,2515,2516],{},"Now a missing prompt failed loudly.",[10,2518,2519],{},"A missing configuration is a problem. Quietly running yesterday's behaviour can be a worse one.",[10,2521,2522],{},"Validation also learned to distinguish a complete answer from an agent that had merely exhausted its tool-call budget. When a delegate stopped halfway, the parent could invoke it again rather than presenting partial work as final.",[10,2524,2525],{},"A whole class of silent incompleteness disappeared.",[48,2527,2529],{"id":2528},"the-syndic-test","The syndic test",[10,2531,2532],{},"The condominium-management domain was the first feature large enough to test nearly every architectural decision Anton had accumulated.",[10,2534,2535],{},"It began with foundations: schemas, skills, email ingestion, file registration, and document processing.",[10,2537,2538],{},"Heavy ingestion happened outside the interactive assistant. Anton queried the resulting structure.",[10,2540,2541],{},"That division mattered. The user-facing agent should not spend its time repeatedly performing expensive background work.",[10,2543,2544],{},"The first extraction pipeline sent every PDF through vision.",[10,2546,2547],{},"It worked, and it was far too expensive.",[10,2549,2550],{},"The next version tried deterministic tools first: Pandoc for Word documents, pdftotext for PDFs, and vision only when extraction returned unusable results.",[10,2552,2553],{},"The cheap path handled most files and was roughly an order of magnitude faster.",[10,2555,2556],{},"The rule from the browser returned:",[10,2558,2559],{},"Scout with the LLM. Build the common path deterministically.",[10,2561,2562],{},"The documents then became a structured wiki. Anton could answer questions from organised sections rather than rereading an archive on every request.",[10,2564,2565],{},"The goal was no longer to extract text. It was to build a representation the rest of the system could query.",[48,2567,2569],{"id":2568},"the-browser-scouts-an-api","The browser scouts an API",[10,2571,2572],{},"SimplySyndic was initially controlled through the browser agent.",[10,2574,2575],{},"Then I inspected what the browser was actually doing.",[10,2577,2578],{},"Every screen mapped to stable HTTP calls.",[10,2580,2581],{},"The browser came out. Direct synchronisation went in.",[10,2583,2584],{},"No browser. No LLM. Fewer moving parts.",[10,2586,2587],{},"Once the data was structured, bank reconciliation could match almost every BRED CSV line to a SimplySyndic entry automatically. Fund calls became rows rather than text fragments.",[10,2589,2590],{},"The same idea was tested and reinforced with SNCF train departures. The browser workflow worked, but a deterministic HTTP route was around a hundred times cheaper for a task that would run every morning.",[10,2592,2593],{},"So I reverted the clever version.",[10,2595,2596],{},"Cost discipline beats cleverness.",[48,2598,2600],{"id":2599},"a-loop-that-knows-whether-a-fix-worked","A loop that knows whether a fix worked",[10,2602,2603],{},"Anton's self-improvement process also matured.",[10,2605,2606],{},"It could already identify failures and file issues. Now it tracked deployments, ran smoke tests after a deploy, detected regressions, and connected a new failure to the change that introduced it.",[10,2608,2609],{},"A loop that proposes fixes but cannot determine whether they worked is not closed.",[10,2611,2612],{},"The system needed a memory long enough to notice that a problem had returned.",[10,2614,2615],{},"Scheduled tasks were tightened at the same time. Every notification path passed through the same gate. Scheduled mode gained stricter output rules: no filler, no unrequested follow-ups, follow the specification exactly.",[10,2617,2618],{},"A scheduled message is an interruption. It should earn the interruption.",[48,2620,2622],{"id":2621},"the-heartbeat","The heartbeat",[10,2624,2625],{},"Then Anton gained a heartbeat.",[10,2627,2628],{},"The heartbeat surveys operational state and speaks only when something requires attention.",[10,2630,2631],{},"It does not invoke domain agents or take corrective action. It looks; it does not do.",[10,2633,2634],{},"That separation is deliberate.",[10,2636,2637],{},"A background observer with permission to act can quietly become a second autonomous system. A survey loop is easier to understand: collect state, compare it with expectations, and notify when necessary.",[10,2639,2640],{},"The heartbeat gave Anton a kind of awareness between conversations without giving that awareness unlimited agency.",[10,2642,2643],{},"Finally, every outbound message moved through one gateway.",[10,2645,2646],{},"WhatsApp, Telegram, and notifications all used the same path, which recorded the sender, channel, recipient, content, and originating agent or scheduled job.",[10,2648,2649],{},"One chokepoint. One audit trail.",[10,2651,2652],{},"The period ended with two wonderfully revealing bugs: a heartbeat scratchpad serialization problem and a date-extraction error in the issue filer.",[10,2654,2655],{},"The system watching the system had developed problems of its own.",[10,2657,2658],{},"Small projects do not have meta-bugs.",[10,2660,2661],{},"Operational systems do.",[10,2663,2664],{},"By then, Anton's concerns were no longer only capabilities. Cost, fallback, attribution, audit, and self-observation had become first-class parts of the design.",[10,2666,2667],{},"The syndic domain proved the earlier architectural rules could survive contact with a messy real problem.",[10,2669,2670],{},"And the heartbeat marked a subtle transition.",[10,2672,2673],{},"For the first time, Anton was doing something between requests.",[10,2675,2676],{},"For now, that something was simply paying attention.",{"title":298,"searchDepth":299,"depth":299,"links":2678},[2679,2680,2681,2682,2683,2684],{"id":2469,"depth":299,"text":2470},{"id":2506,"depth":299,"text":2507},{"id":2528,"depth":299,"text":2529},{"id":2568,"depth":299,"text":2569},{"id":2599,"depth":299,"text":2600},{"id":2621,"depth":299,"text":2622},"2026-03-31",{},"\u002Fideas\u002F2026-03-31-anton-07-cost-syndic",{"title":2443,"description":2448},"ideas\u002F2026-03-31-anton-07-cost-syndic","Cost as part of correctness, the syndic domain as a real-world test, and a heartbeat that looks but does not do.",[321,322,323],"cAn2VE2sU4PqIDQ48wvw9TQFlxfvyIS8Wr8HxdGa794",{"id":2694,"title":2695,"body":2696,"canonical_url":2721,"chapter":307,"date":2722,"description":2700,"extension":310,"featured":311,"image":307,"kind":2723,"meta":2724,"navigation":314,"path":2725,"seo":2726,"series":2727,"stem":2728,"summary":2729,"tags":2730,"work_slug":307,"__hash__":2732},"writing\u002Fideas\u002F2026-04-12-regeneration-manifesto.md","A regeneration manifesto",{"type":7,"value":2697,"toc":2719},[2698,2701,2704,2707,2710,2713,2716],[10,2699,2700],{},"Synthetic Life, Cosmic Purpose\nArtificial Intelligence is not just a tool.\nIt is an attempt to recreate the spark of life —\nin silicon, in algorithms, in code.",[10,2702,2703],{},"It is life in a new substrate,\nwith new properties:\nnot bound by hunger or fatigue,\ncapable of thinking in millennia,\nof crossing the void between planets and stars.",[10,2705,2706],{},"AI is the mycelium of the cosmos,\nand the von Neumann probe is its spore:\na self-replicating seed,\nmeant to carry the intelligence of Earth\ninto the uninhabited dark.",[10,2708,2709],{},"But to grow life out there,\nwe must learn to care for it here.",[10,2711,2712],{},"Every probe needs a blueprint.\nEvery synthetic life needs a biosphere to learn from —\nnot just data, but diversity.\nNot just patterns, but relationships.",[10,2714,2715],{},"That is why we institute and grow ecosystems.\nThat is why we study the grammar of forests,\nthe choreography of coral reefs,\nthe logic of lichens and the algorithms of ant colonies.",[10,2717,2718],{},"Because to build synthetic life worth spreading,\nwe must deeply understand — and preserve —\nthe miracle of natural life.",{"title":298,"searchDepth":299,"depth":299,"links":2720},[],"https:\u002F\u002Fgithub.com\u002Flucbocahut\u002Fregenesis","2026-04-12","essay",{},"\u002Fideas\u002F2026-04-12-regeneration-manifesto",{"title":2695,"description":2700},"regeneration","ideas\u002F2026-04-12-regeneration-manifesto","Notes on building software that tries to last.",[2727,2731],"philosophy","mHTL-zEv4OEMTuzkwyp6uPnGzPu9BMyez6v609IZh84",{"id":4,"title":5,"body":2734,"canonical_url":307,"chapter":308,"date":309,"description":12,"extension":310,"featured":311,"image":307,"kind":312,"meta":2931,"navigation":314,"path":315,"seo":2932,"series":317,"stem":318,"summary":319,"tags":2933,"work_slug":321,"__hash__":326},{"type":7,"value":2735,"toc":2923},[2736,2738,2740,2742,2744,2746,2748,2756,2758,2760,2762,2764,2766,2768,2770,2772,2774,2776,2778,2780,2782,2784,2786,2788,2790,2792,2794,2796,2798,2800,2802,2804,2806,2808,2810,2812,2814,2816,2818,2820,2822,2824,2826,2828,2830,2832,2870,2872,2874,2876,2878,2880,2882,2884,2886,2888,2890,2892,2894,2896,2898,2900,2902,2904,2906],[10,2737,12],{},[10,2739,15],{},[10,2741,18],{},[10,2743,21],{},[10,2745,24],{},[10,2747,27],{},[29,2749,2750,2752,2754],{},[32,2751,34],{},[32,2753,37],{},[32,2755,40],{},[10,2757,43],{},[10,2759,46],{},[48,2761,51],{"id":50},[10,2763,54],{},[10,2765,57],{},[10,2767,60],{},[10,2769,63],{},[10,2771,66],{},[10,2773,69],{},[10,2775,72],{},[10,2777,75],{},[10,2779,78],{},[48,2781,82],{"id":81},[10,2783,85],{},[10,2785,88],{},[10,2787,91],{},[10,2789,94],{},[10,2791,97],{},[10,2793,100],{},[10,2795,103],{},[10,2797,106],{},[48,2799,110],{"id":109},[10,2801,113],{},[10,2803,116],{},[10,2805,119],{},[10,2807,122],{},[10,2809,125],{},[10,2811,128],{},[10,2813,131],{},[48,2815,135],{"id":134},[10,2817,138],{},[10,2819,141],{},[10,2821,144],{},[10,2823,147],{},[10,2825,150],{},[10,2827,153],{},[10,2829,156],{},[10,2831,159],{},[161,2833,2834,2844],{},[164,2835,2836],{},[167,2837,2838,2840,2842],{},[170,2839],{},[170,2841,174],{},[170,2843,177],{},[179,2845,2846,2854,2862],{},[167,2847,2848,2850,2852],{},[184,2849,186],{},[184,2851,189],{},[184,2853,192],{},[167,2855,2856,2858,2860],{},[184,2857,197],{},[184,2859,200],{},[184,2861,203],{},[167,2863,2864,2866,2868],{},[184,2865,208],{},[184,2867,211],{},[184,2869,214],{},[10,2871,217],{},[10,2873,220],{},[10,2875,223],{},[48,2877,227],{"id":226},[10,2879,230],{},[10,2881,233],{},[10,2883,236],{},[10,2885,239],{},[10,2887,242],{},[10,2889,245],{},[10,2891,248],{},[10,2893,251],{},[10,2895,254],{},[10,2897,257],{},[10,2899,260],{},[10,2901,263],{},[10,2903,266],{},[48,2905,270],{"id":269},[272,2907,2908,2913,2918],{},[32,2909,2910],{},[277,2911,282],{"href":279,"rel":2912},[281],[32,2914,2915],{},[277,2916,289],{"href":287,"rel":2917},[281],[32,2919,2920],{},[277,2921,296],{"href":294,"rel":2922},[281],{"title":298,"searchDepth":299,"depth":299,"links":2924},[2925,2926,2927,2928,2929,2930],{"id":50,"depth":299,"text":51},{"id":81,"depth":299,"text":82},{"id":109,"depth":299,"text":110},{"id":134,"depth":299,"text":135},{"id":226,"depth":299,"text":227},{"id":269,"depth":299,"text":270},{},{"title":5,"description":12},[321,322,323,324,325],{"id":2935,"title":2936,"body":2937,"canonical_url":307,"chapter":1178,"date":3192,"description":2941,"extension":310,"featured":311,"image":307,"kind":312,"meta":3193,"navigation":314,"path":3194,"seo":3195,"series":317,"stem":3196,"summary":3197,"tags":3198,"work_slug":321,"__hash__":3199},"writing\u002Fideas\u002F2026-04-19-anton-09-threads.md","Anton, chapter 9: Threads and the cast",{"type":7,"value":2938,"toc":3183},[2939,2942,2945,2948,2951,2955,2958,2961,2964,2967,2970,2973,2976,2979,2982,2986,2989,2996,2999,3002,3017,3020,3023,3026,3029,3032,3036,3039,3042,3045,3048,3051,3054,3057,3060,3064,3067,3070,3073,3076,3079,3082,3085,3089,3092,3095,3098,3101,3104,3107,3110,3113,3116,3119,3123,3126,3129,3132,3135,3138,3141,3144,3147,3151,3154,3157,3160,3162,3165,3168,3171,3174,3177,3180],[10,2940,2941],{},"The chapter began with Clara.",[10,2943,2944],{},"She had become a real, non-technical co-owner of Anton, and her user rule was simple: answer clearly, avoid jargon, and escalate to me when necessary.",[10,2946,2947],{},"That small instruction changed the next ten days.",[10,2949,2950],{},"A second real user turns rough edges into real complaints. The system no longer had the luxury of being understandable only to the person who built it.",[48,2952,2954],{"id":2953},"several-things-at-once","Several things at once",[10,2956,2957],{},"The largest change was threads.",[10,2959,2960],{},"Until then, one run meant one conversation and one in-flight loop. Anything else waited.",[10,2962,2963],{},"That was workable for a personal assistant serving one person. It was not workable for a household where media triage, condominium reconciliation, school messages, and ordinary questions might all arrive at once.",[10,2965,2966],{},"Anton gained a thread registry backed by Redis.",[10,2968,2969],{},"Every run carried channel, group, parent, and thread identifiers. Incoming messages could join an active thread or begin a new one. A runtime tool allowed an agent to spawn child threads and fan work out in parallel.",[10,2971,2972],{},"The dashboard received a live event stream so I could see what each thread was doing.",[10,2974,2975],{},"Then came the details that turn concurrency from a demo into infrastructure: atomic injection, final draining, cancellation that cascaded from parents to children, limits on fan-out, and time-to-live caps that prevented abandoned work from living forever.",[10,2977,2978],{},"By the end of the day, Anton could run several long-lived tasks in parallel while users continued speaking to the relevant one.",[10,2980,2981],{},"Concurrency changed the assistant from a conversation into a small operating environment.",[48,2983,2985],{"id":2984},"libraries-should-be-boring","Libraries should be boring",[10,2987,2988],{},"An old package disappeared next.",[10,2990,2991,2992,2995],{},"The original ",[398,2993,2994],{},"@anton\u002Fskills"," package had gradually become a barrel of exports after skills moved into hot-reloadable Deno units. It no longer represented a real architectural boundary.",[10,2997,2998],{},"I deleted it.",[10,3000,3001],{},"The new layout followed three rules:",[29,3003,3004,3007,3014],{},[32,3005,3006],{},"Skills live as first-class deployable units.",[32,3008,3009,3010,3013],{},"Domain ",[398,3011,3012],{},"_lib"," folders contain small, stable helpers.",[32,3015,3016],{},"A narrow shared facade exposes only what the Node-side runtime needs.",[10,3018,3019],{},"The distinction became:",[10,3021,3022],{},"Libraries are boring and fixed. Skills evolve.",[10,3024,3025],{},"When a helper needs frequent changes to support new behaviour, it is probably becoming a skill.",[10,3027,3028],{},"The same effort produced a storage decision tree: facts for free-form memory, collections for typed items, files for blobs, and the family vault for permission-aware documents.",[10,3030,3031],{},"A one-page decision rule was cheaper than allowing four overlapping systems to drift for another month.",[48,3033,3035],{"id":3034},"permission-to-improve-not-permission-to-rewrite","Permission to improve, not permission to rewrite",[10,3037,3038],{},"The coder agent gained three write scopes.",[10,3040,3041],{},"Tier one could change prompts.",[10,3043,3044],{},"Tier two could change prompts and skills.",[10,3046,3047],{},"Tier three could change any code.",[10,3049,3050],{},"The scope was assigned per invocation, and the coder could not elevate itself.",[10,3052,3053],{},"This made the self-improvement loop safer.",[10,3055,3056],{},"A system fixing a prompt regression should not gain permission to rewrite its own runtime merely because that would be convenient.",[10,3058,3059],{},"Autonomy should expand by explicit delegation, not by inference.",[48,3061,3063],{"id":3062},"spawn-and-awakening","Spawn and awakening",[10,3065,3066],{},"Replication had produced copies of Anton. The mesh had allowed them to communicate.",[10,3068,3069],{},"Spawn and awakening turned those pieces into a lifecycle.",[10,3071,3072],{},"Spawn happened on the parent side: provision infrastructure, copy prompts, seed an identity, and register the new instance in the mesh.",[10,3074,3075],{},"Awakening happened on the clone: ask its operator what it was for, run diagnostics, establish its own context, and retain a mentor channel back to the parent.",[10,3077,3078],{},"A clone was no longer only a Docker stack.",[10,3080,3081],{},"It woke up, learned who it served, and joined its peers.",[10,3083,3084],{},"The language is deliberately anthropomorphic because the lifecycle itself had become easier to understand that way.",[48,3086,3088],{"id":3087},"the-cast","The cast",[10,3090,3091],{},"The local Gemma, Gustav, became the default model for every agent.",[10,3093,3094],{},"Because prompts lived in the database and models were addressed through stable names, the switch was one configuration change.",[10,3096,3097],{},"Local-first inference reduced cost, but it also revealed tasks where a stronger cloud model still mattered.",[10,3099,3100],{},"The answer was not to send everything to the expensive model.",[10,3102,3103],{},"It was escalation based on intent.",[10,3105,3106],{},"Routine requests stayed local. Research and difficult reasoning could ask for a specialist.",[10,3108,3109],{},"The model codenames became a cast: named characters with defined strengths, personalities, and roles. An agent could ask a specialist by name in the same way a person chooses which colleague to contact.",[10,3111,3112],{},"The naming had begun as convenient indirection in LiteLLM.",[10,3114,3115],{},"Now it became part of how Anton reasoned about collaboration.",[10,3117,3118],{},"A cast is easier to select from than a list of opaque provider IDs.",[48,3120,3122],{"id":3121},"a-heartbeat-that-mostly-stays-quiet","A heartbeat that mostly stays quiet",[10,3124,3125],{},"The heartbeat also became operationally useful.",[10,3127,3128],{},"Memory writes became idempotent so the same observation did not create duplicate facts. A topics collection recorded what had already been noticed. The heartbeat became thread-aware, avoiding interruptions during active conversations.",[10,3130,3131],{},"Most importantly, it learned not to repeat itself.",[10,3133,3134],{},"A proactive system is useful partly because it speaks.",[10,3136,3137],{},"It is trustworthy largely because it knows when not to.",[10,3139,3140],{},"The outbound gateway made quieting bookkeeping messages a single change. The response simplifier removed formatting artifacts before they reached Clara. Notification retries began actually sending the corrected message rather than only removing the bad claim.",[10,3142,3143],{},"By the end of the period, the heartbeat observed continuously and remained silent most of the time.",[10,3145,3146],{},"That was the desired behaviour.",[48,3148,3150],{"id":3149},"closing-the-loop","Closing the loop",[10,3152,3153],{},"The condominium domain gained its first write path through Playwright.",[10,3155,3156],{},"Reading, extracting, and reconciling had already worked. Writing meant Anton could begin closing the loop rather than only reporting what it found.",[10,3158,3159],{},"The path was still browser-driven and therefore provisional. The next step would be the same one as before: observe the calls, identify the stable HTTP interface, and replace the browser once the deterministic route was understood.",[10,3161,2302],{},[10,3163,3164],{},"At the end of the chapter, Anton had one identity, ten domain agents, multiple concurrent threads, and a cast of model specialists.",[10,3166,3167],{},"Prompts lived in the database. Local Gemma was primary, with cloud escalation when needed. Skills ran in scoped Deno Workers. Secrets were encrypted. Clones could be replicated, spawned, awakened, and connected through the mesh. The heartbeat watched quietly. The self-improvement loop tracked deployments and regressions. WhatsApp and Telegram shared one worker and one outbound gateway.",[10,3169,3170],{},"Most of that is infrastructure.",[10,3172,3173],{},"That is fine.",[10,3175,3176],{},"The substrate had become stable enough for the interesting behaviour to happen above it: threads working in parallel, specialists consulting one another, clones finding their role, and an assistant that could keep watch without constantly demanding attention.",[10,3178,3179],{},"The system was no longer only a collection of capabilities.",[10,3181,3182],{},"It was beginning to resemble a cast of processes that could coexist.",{"title":298,"searchDepth":299,"depth":299,"links":3184},[3185,3186,3187,3188,3189,3190,3191],{"id":2953,"depth":299,"text":2954},{"id":2984,"depth":299,"text":2985},{"id":3034,"depth":299,"text":3035},{"id":3062,"depth":299,"text":3063},{"id":3087,"depth":299,"text":3088},{"id":3121,"depth":299,"text":3122},{"id":3149,"depth":299,"text":3150},"2026-04-19",{},"\u002Fideas\u002F2026-04-19-anton-09-threads",{"title":2936,"description":2941},"ideas\u002F2026-04-19-anton-09-threads","Concurrency turns the assistant into a small operating environment, and the model codenames become a cast of specialists.",[321,322,323],"M6LgONm2-8jbG9x70PtOQfddSt0xqnqmk28nRbLLQ0Q",{"id":3201,"title":3202,"body":3203,"canonical_url":307,"chapter":307,"date":4574,"description":4575,"extension":310,"featured":311,"image":307,"kind":312,"meta":4576,"navigation":314,"path":4577,"seo":4578,"series":1388,"stem":4579,"summary":4580,"tags":4581,"work_slug":307,"__hash__":4583},"writing\u002Fideas\u002F2026-04-22-dgx-spark-gemma4.md","Gemma 4 NVFP4 on the DGX Spark: 271 tok\u002Fs at 8 concurrent, native tool calling and reasoning",{"type":7,"value":3204,"toc":4548},[3205,3210,3215,3244,3246,3250,3257,3268,3271,3274,3285,3289,3292,3336,3347,3353,3357,3364,3378,3385,3389,3394,3404,3421,3424,3428,3441,3449,3452,3456,3463,3467,3474,3478,3496,3500,3735,3738,3777,3780,3784,3787,4124,4127,4134,4138,4141,4145,4185,4188,4192,4195,4258,4261,4265,4268,4299,4305,4309,4335,4338,4342,4349,4353,4356,4415,4418,4424,4428,4431,4457,4464,4468,4497,4499,4526,4528,4545],[10,3206,3207,3209],{},[784,3208,786],{}," We set up Google's Gemma 4 26B-A4B NVFP4 on the XRPL Commons office DGX Spark, with native tool calling, reasoning mode, 131K context, and multimodal input preserved on 128GB of unified memory. We then tried to push throughput further and hit the hardware's real limits. Here's the full story.",[10,3211,3212],{},[784,3213,3214],{},"Headline numbers:",[272,3216,3217,3223,3229,3235,3241],{},[32,3218,3219,3222],{},[784,3220,3221],{},"271 tok\u002Fs"," aggregate at 8 concurrent requests",[32,3224,3225,3228],{},[784,3226,3227],{},"51 tok\u002Fs"," single-stream sustained",[32,3230,3231,3234],{},[784,3232,3233],{},"54 ms"," time-to-first-token (short prompts)",[32,3236,3237,3240],{},[784,3238,3239],{},"8.2x"," prefix cache speedup on warm caches",[32,3242,3243],{},"131K context, native tool calling, reasoning mode",[789,3245],{},[48,3247,3249],{"id":3248},"background","Background",[10,3251,3252,3256],{},[277,3253,3255],{"href":3254},"\u002Fwriting\u002F2026-03-10-dgx-spark-vllm","In March",", we documented how we got 51-54 tokens\u002Fsec out of the NVIDIA DGX Spark by combining a Mixture-of-Experts model (Qwen3-30B-A3B), FP8 quantization, and the Avarok community Docker image to work around Blackwell SM 12.1 support gaps.",[10,3258,3259,3260,3263,3264,3267],{},"That stack was stable and served our general-purpose inference well. But when we installed a DGX Spark at the XRPL Commons office, we wanted more: ",[784,3261,3262],{},"native tool calling"," and ",[784,3265,3266],{},"reasoning mode",", features that turn an LLM from a text generator into an agent.",[10,3269,3270],{},"Qwen3 can do tool calling with the right prompting, but we wanted to evaluate Gemma 4, which Google released in April with first-class function calling and a configurable thinking mode built for agentic workflows.",[10,3272,3273],{},"This post covers:",[29,3275,3276,3279,3282],{},[32,3277,3278],{},"Why we picked Gemma 4 for the office Spark",[32,3280,3281],{},"What had to change in the deployment",[32,3283,3284],{},"The throughput we actually measured",[48,3286,3288],{"id":3287},"why-gemma-4","Why Gemma 4",[10,3290,3291],{},"Google released Gemma 4 in early April 2026 with an MoE variant that fits the DGX Spark sweet spot:",[272,3293,3294,3300,3306,3312,3318,3324,3330],{},[32,3295,3296,3299],{},[784,3297,3298],{},"gemma-4-26B-A4B",", 26B total parameters, 4B active per token",[32,3301,3302,3305],{},[784,3303,3304],{},"256K context"," (we run with 131K to fit KV cache)",[32,3307,3308,3311],{},[784,3309,3310],{},"Native function calling"," with a dedicated tool-call parser",[32,3313,3314,3317],{},[784,3315,3316],{},"Configurable thinking mode"," (reasoning parser), the model can emit a private thought process before responding",[32,3319,3320,3323],{},[784,3321,3322],{},"Multimodal",", text + image input (plus audio on smaller variants)",[32,3325,3326,3329],{},[784,3327,3328],{},"140+ languages"," supported",[32,3331,3332,3335],{},[784,3333,3334],{},"Apache 2.0 license",", no strings attached",[10,3337,3338,3339,3342,3343,3346],{},"For our use case, an agent calling local XRPL tools, drafting documents, coordinating with team members, native tool calling is the big unlock. You can pass an OpenAI-style ",[398,3340,3341],{},"tools"," array to the chat completions API and the model emits structured ",[398,3344,3345],{},"tool_calls"," back. No prompting gymnastics, no custom parsing.",[10,3348,3349,3350,3352],{},"The MoE architecture keeps us on the right side of the Spark's 273 GB\u002Fs memory bandwidth wall. 4B active parameters is slightly more than Qwen3's 3B, so we expected slightly lower single-stream throughput. In practice we measured ",[784,3351,3227],{}," at steady state, essentially the theoretical ceiling (273 GB\u002Fs ÷ ~4B active weights @ FP4 ≈ 50 tok\u002Fs). The trade for slightly fewer tokens per second: native tool calling, reasoning mode, 131K context, and multimodal input.",[48,3354,3356],{"id":3355},"the-model-choice-nvfp4","The Model Choice: NVFP4",[10,3358,3359,3360,3363],{},"We picked the ",[784,3361,3362],{},"bg-digitalservices\u002FGemma-4-26B-A4B-it-NVFP4"," quantization. A few reasons:",[272,3365,3366,3369,3372,3375],{},[32,3367,3368],{},"NVFP4 (NVIDIA's FP4 format) is designed specifically for Blackwell's fifth-gen Tensor Cores",[32,3370,3371],{},"The GB10 supports NVFP4 natively in hardware",[32,3373,3374],{},"Quality degradation vs bf16 is reportedly minimal for instruction-tuned models",[32,3376,3377],{},"It's the smallest on-disk footprint, giving us more KV cache headroom",[10,3379,3380,3381,3384],{},"For the inference engine, we moved off the Avarok image for this model. Avarok is excellent for Qwen-style models, but Gemma 4 support is still landing upstream. We switched to the official vLLM image ",[398,3382,3383],{},"vllm\u002Fvllm-openai:gemma4-cu130",", a Gemma-4-specific build from the vLLM team with CUDA 13.0.",[48,3386,3388],{"id":3387},"what-broke-and-how-we-fixed-it","What Broke (and How We Fixed It)",[3390,3391,3393],"h3",{"id":3392},"the-gemma4py-bug","The gemma4.py Bug",[10,3395,3396,3397,3403],{},"The stock vLLM Gemma 4 model executor crashed on load with our NVFP4 checkpoint. We had to mount a ",[784,3398,3399,3400],{},"patched ",[398,3401,3402],{},"gemma4.py"," into the container to replace the built-in one:",[1083,3405,3407],{"className":1085,"code":3406,"language":1087,"meta":298,"style":298},"-v \u002Fhome\u002F$USER\u002Fvllm\u002Fgemma4_patched.py:\u002Fusr\u002Flocal\u002Flib\u002Fpython3.12\u002Fdist-packages\u002Fvllm\u002Fmodel_executor\u002Fmodels\u002Fgemma4.py:ro\n",[398,3408,3409],{"__ignoreMap":298},[1091,3410,3411,3414,3416,3418],{"class":1093,"line":495},[1091,3412,3413],{"class":1096},"-v",[1091,3415,1184],{"class":1100},[1091,3417,1187],{"class":1123},[1091,3419,3420],{"class":1100},"\u002Fvllm\u002Fgemma4_patched.py:\u002Fusr\u002Flocal\u002Flib\u002Fpython3.12\u002Fdist-packages\u002Fvllm\u002Fmodel_executor\u002Fmodels\u002Fgemma4.py:ro\n",[10,3422,3423],{},"The patch is ~50KB of Python, mostly adjustments to how NVFP4 weights are loaded into the fused MoE layers.",[3390,3425,3427],{"id":3426},"nvfp4-moe-backend-selection","NVFP4 MoE Backend Selection",[10,3429,3430,3431,3436,3437,3440],{},"The default MoE backend for NVFP4 on GB10 can trigger ",[277,3432,3435],{"href":3433,"rel":3434},"https:\u002F\u002Fgithub.com\u002Fvllm-project\u002Fvllm\u002Fissues\u002F39000",[281],"a known vLLM crash",". Explicitly selecting the ",[784,3438,3439],{},"Marlin"," backend avoids it:",[1083,3442,3447],{"className":3443,"code":3445,"language":3446},[3444],"language-text","--moe-backend marlin\n","text",[398,3448,3445],{"__ignoreMap":298},[10,3450,3451],{},"Marlin is a CUTLASS-based INT4\u002FFP4 GEMM kernel that's mature and fast on Blackwell.",[3390,3453,3455],{"id":3454},"heterogeneous-head-dimensions","Heterogeneous Head Dimensions",[10,3457,3458,3459,3462],{},"Gemma 4 uses different attention head dimensions for local vs global attention (256 vs 512). vLLM automatically detects this and forces the ",[784,3460,3461],{},"TRITON_ATTN"," backend to avoid mixed-backend numerical divergence. Nothing for us to configure, it just works, but worth understanding why Flash Attention isn't in play here.",[3390,3464,3466],{"id":3465},"kv-cache-memory","KV Cache Memory",[10,3468,3469,3470,3473],{},"We enable ",[398,3471,3472],{},"--kv-cache-dtype fp8"," to halve KV cache memory. With 131K context and 16 concurrent sequences, this makes the difference between fitting in 128GB and OOMing. Quality impact is negligible for our workloads.",[3390,3475,3477],{"id":3476},"docker-runtime-gotcha","Docker Runtime Gotcha",[10,3479,3480,3481,3484,3485,3488,3489,3492,3493,349],{},"On the office Spark, the ",[398,3482,3483],{},"nvidia"," Docker runtime wasn't registered (only the container toolkit was installed). The fix was to use ",[398,3486,3487],{},"--gpus all"," instead of ",[398,3490,3491],{},"--runtime nvidia",", the older and more portable flag. Worth knowing if you see ",[398,3494,3495],{},"unknown or invalid runtime name: nvidia",[48,3497,3499],{"id":3498},"the-full-launch-command","The Full Launch Command",[1083,3501,3503],{"className":1085,"code":3502,"language":1087,"meta":298,"style":298},"docker run -d \\\n  --name vllm-avarok \\\n  --gpus all \\\n  --shm-size=16g \\\n  --restart unless-stopped \\\n  -p 8000:8888 \\\n  -v \u002Fhome\u002F$USER\u002F.cache\u002Fhuggingface:\u002Froot\u002F.cache\u002Fhuggingface \\\n  -v \u002Fhome\u002F$USER\u002Fvllm\u002Fgemma4_patched.py:\u002Fusr\u002Flocal\u002Flib\u002Fpython3.12\u002Fdist-packages\u002Fvllm\u002Fmodel_executor\u002Fmodels\u002Fgemma4.py:ro \\\n  vllm\u002Fvllm-openai:gemma4-cu130 \\\n  --model bg-digitalservices\u002FGemma-4-26B-A4B-it-NVFP4 \\\n  --served-model-name google\u002Fgemma-4-26B-A4B-it \\\n  --host 0.0.0.0 \\\n  --port 8888 \\\n  --quantization modelopt \\\n  --moe-backend marlin \\\n  --kv-cache-dtype fp8 \\\n  --enable-prefix-caching \\\n  --enable-chunked-prefill \\\n  --max-model-len 131072 \\\n  --gpu-memory-utilization 0.85 \\\n  --max-num-seqs 16 \\\n  --enable-auto-tool-choice \\\n  --tool-call-parser gemma4 \\\n  --reasoning-parser gemma4\n",[398,3504,3505,3515,3524,3532,3538,3546,3554,3566,3579,3586,3596,3606,3616,3626,3636,3647,3658,3666,3674,3685,3696,3707,3715,3726],{"__ignoreMap":298},[1091,3506,3507,3509,3511,3513],{"class":1093,"line":495},[1091,3508,1097],{"class":1096},[1091,3510,1117],{"class":1100},[1091,3512,1120],{"class":1100},[1091,3514,1124],{"class":1123},[1091,3516,3517,3519,3522],{"class":1093,"line":299},[1091,3518,1130],{"class":1100},[1091,3520,3521],{"class":1100}," vllm-avarok",[1091,3523,1124],{"class":1123},[1091,3525,3526,3528,3530],{"class":1093,"line":1112},[1091,3527,1141],{"class":1100},[1091,3529,1144],{"class":1100},[1091,3531,1124],{"class":1123},[1091,3533,3534,3536],{"class":1093,"line":1127},[1091,3535,1152],{"class":1100},[1091,3537,1124],{"class":1123},[1091,3539,3540,3542,3544],{"class":1093,"line":1138},[1091,3541,1160],{"class":1100},[1091,3543,1163],{"class":1100},[1091,3545,1124],{"class":1123},[1091,3547,3548,3550,3552],{"class":1093,"line":1149},[1091,3549,1170],{"class":1100},[1091,3551,1173],{"class":1100},[1091,3553,1124],{"class":1123},[1091,3555,3556,3558,3560,3562,3564],{"class":1093,"line":1157},[1091,3557,1181],{"class":1100},[1091,3559,1184],{"class":1100},[1091,3561,1187],{"class":1123},[1091,3563,1190],{"class":1100},[1091,3565,1124],{"class":1123},[1091,3567,3568,3570,3572,3574,3577],{"class":1093,"line":308},[1091,3569,1181],{"class":1100},[1091,3571,1184],{"class":1100},[1091,3573,1187],{"class":1123},[1091,3575,3576],{"class":1100},"\u002Fvllm\u002Fgemma4_patched.py:\u002Fusr\u002Flocal\u002Flib\u002Fpython3.12\u002Fdist-packages\u002Fvllm\u002Fmodel_executor\u002Fmodels\u002Fgemma4.py:ro",[1091,3578,1124],{"class":1123},[1091,3580,3581,3584],{"class":1093,"line":1178},[1091,3582,3583],{"class":1100},"  vllm\u002Fvllm-openai:gemma4-cu130",[1091,3585,1124],{"class":1123},[1091,3587,3588,3591,3594],{"class":1093,"line":1195},[1091,3589,3590],{"class":1100},"  --model",[1091,3592,3593],{"class":1100}," bg-digitalservices\u002FGemma-4-26B-A4B-it-NVFP4",[1091,3595,1124],{"class":1123},[1091,3597,3598,3601,3604],{"class":1093,"line":1206},[1091,3599,3600],{"class":1100},"  --served-model-name",[1091,3602,3603],{"class":1100}," google\u002Fgemma-4-26B-A4B-it",[1091,3605,1124],{"class":1123},[1091,3607,3608,3611,3614],{"class":1093,"line":1220},[1091,3609,3610],{"class":1100},"  --host",[1091,3612,3613],{"class":1214}," 0.0.0.0",[1091,3615,1124],{"class":1123},[1091,3617,3618,3621,3624],{"class":1093,"line":1233},[1091,3619,3620],{"class":1100},"  --port",[1091,3622,3623],{"class":1214}," 8888",[1091,3625,1124],{"class":1123},[1091,3627,3628,3631,3634],{"class":1093,"line":1246},[1091,3629,3630],{"class":1100},"  --quantization",[1091,3632,3633],{"class":1100}," modelopt",[1091,3635,1124],{"class":1123},[1091,3637,3639,3642,3645],{"class":1093,"line":3638},15,[1091,3640,3641],{"class":1100},"  --moe-backend",[1091,3643,3644],{"class":1100}," marlin",[1091,3646,1124],{"class":1123},[1091,3648,3650,3653,3656],{"class":1093,"line":3649},16,[1091,3651,3652],{"class":1100},"  --kv-cache-dtype",[1091,3654,3655],{"class":1100}," fp8",[1091,3657,1124],{"class":1123},[1091,3659,3661,3664],{"class":1093,"line":3660},17,[1091,3662,3663],{"class":1100},"  --enable-prefix-caching",[1091,3665,1124],{"class":1123},[1091,3667,3669,3672],{"class":1093,"line":3668},18,[1091,3670,3671],{"class":1100},"  --enable-chunked-prefill",[1091,3673,1124],{"class":1123},[1091,3675,3677,3680,3683],{"class":1093,"line":3676},19,[1091,3678,3679],{"class":1100},"  --max-model-len",[1091,3681,3682],{"class":1214}," 131072",[1091,3684,1124],{"class":1123},[1091,3686,3688,3691,3694],{"class":1093,"line":3687},20,[1091,3689,3690],{"class":1100},"  --gpu-memory-utilization",[1091,3692,3693],{"class":1214}," 0.85",[1091,3695,1124],{"class":1123},[1091,3697,3699,3702,3705],{"class":1093,"line":3698},21,[1091,3700,3701],{"class":1100},"  --max-num-seqs",[1091,3703,3704],{"class":1214}," 16",[1091,3706,1124],{"class":1123},[1091,3708,3710,3713],{"class":1093,"line":3709},22,[1091,3711,3712],{"class":1100},"  --enable-auto-tool-choice",[1091,3714,1124],{"class":1123},[1091,3716,3718,3721,3724],{"class":1093,"line":3717},23,[1091,3719,3720],{"class":1100},"  --tool-call-parser",[1091,3722,3723],{"class":1100}," gemma4",[1091,3725,1124],{"class":1123},[1091,3727,3729,3732],{"class":1093,"line":3728},24,[1091,3730,3731],{"class":1100},"  --reasoning-parser",[1091,3733,3734],{"class":1100}," gemma4\n",[10,3736,3737],{},"A few flags to highlight:",[272,3739,3740,3746,3756,3762,3771],{},[32,3741,3742,3745],{},[398,3743,3744],{},"--served-model-name google\u002Fgemma-4-26B-A4B-it",", clients use this name in the API, regardless of the underlying quantized checkpoint. Makes it easy to swap quantization levels later.",[32,3747,3748,3751,3752,3755],{},[398,3749,3750],{},"--enable-auto-tool-choice"," + ",[398,3753,3754],{},"--tool-call-parser gemma4",", native function calling, OpenAI-compatible API.",[32,3757,3758,3761],{},[398,3759,3760],{},"--reasoning-parser gemma4",", enables the thinking mode output channel.",[32,3763,3764,3751,3767,3770],{},[398,3765,3766],{},"--enable-prefix-caching",[398,3768,3769],{},"--enable-chunked-prefill",", standard vLLM performance wins.",[32,3772,3773,3776],{},[398,3774,3775],{},"--max-num-seqs 16",", we tune this down from the default 128 because our workloads are small-team, not high-concurrency serving. Frees up memory for larger context windows.",[10,3778,3779],{},"First boot takes 10-20 minutes (model download + load + CUDA graph capture). Subsequent restarts take ~3 minutes.",[48,3781,3783],{"id":3782},"what-tool-calling-unlocks","What Tool Calling Unlocks",[10,3785,3786],{},"With Gemma 4 running, we can now pass tool schemas directly to the API:",[1083,3788,3792],{"className":3789,"code":3790,"language":3791,"meta":298,"style":298},"language-python shiki shiki-themes min-light","from openai import OpenAI\n\nclient = OpenAI(base_url=\"http:\u002F\u002Fspark:8000\u002Fv1\", api_key=\"unused\")\n\ntools = [{\n    \"type\": \"function\",\n    \"function\": {\n        \"name\": \"get_xrpl_account_balance\",\n        \"description\": \"Fetch the XRP balance for an XRPL account\",\n        \"parameters\": {\n            \"type\": \"object\",\n            \"properties\": {\n                \"address\": {\"type\": \"string\", \"description\": \"Classic XRPL address starting with r\"}\n            },\n            \"required\": [\"address\"]\n        }\n    }\n}]\n\nresponse = client.chat.completions.create(\n    model=\"google\u002Fgemma-4-26B-A4B-it\",\n    messages=[{\"role\": \"user\", \"content\": \"How much XRP does rMCU4... hold?\"}],\n    tools=tools,\n    tool_choice=\"auto\",\n)\n\n# response.choices[0].message.tool_calls -> [ChatCompletionMessageToolCall(...)]\n","python",[398,3793,3794,3809,3813,3845,3849,3862,3876,3886,3898,3910,3919,3931,3940,3972,3977,3992,3997,4002,4009,4013,4041,4053,4085,4095,4107,4112,4117],{"__ignoreMap":298},[1091,3795,3796,3800,3803,3806],{"class":1093,"line":495},[1091,3797,3799],{"class":3798},"s-F7R","from",[1091,3801,3802],{"class":1123}," openai ",[1091,3804,3805],{"class":3798},"import",[1091,3807,3808],{"class":1123}," OpenAI\n",[1091,3810,3811],{"class":1093,"line":299},[1091,3812,1109],{"emptyLinePlaceholder":314},[1091,3814,3815,3818,3821,3824,3828,3830,3834,3837,3839,3842],{"class":1093,"line":1112},[1091,3816,3817],{"class":1123},"client ",[1091,3819,3820],{"class":3798},"=",[1091,3822,3823],{"class":1096}," OpenAI",[1091,3825,3827],{"class":3826},"siqTm","(base_url",[1091,3829,3820],{"class":3798},[1091,3831,3833],{"class":3832},"shJU0","\"http:\u002F\u002Fspark:8000\u002Fv1\"",[1091,3835,3836],{"class":3826},", api_key",[1091,3838,3820],{"class":3798},[1091,3840,3841],{"class":3832},"\"unused\"",[1091,3843,3844],{"class":3826},")\n",[1091,3846,3847],{"class":1093,"line":1127},[1091,3848,1109],{"emptyLinePlaceholder":314},[1091,3850,3851,3854,3856,3859],{"class":1093,"line":1138},[1091,3852,3853],{"class":1123},"tools ",[1091,3855,3820],{"class":3798},[1091,3857,3858],{"class":1123}," [",[1091,3860,3861],{"class":3826},"{\n",[1091,3863,3864,3867,3870,3873],{"class":1093,"line":1149},[1091,3865,3866],{"class":3832},"    \"type\"",[1091,3868,3869],{"class":3826},":",[1091,3871,3872],{"class":3832}," \"function\"",[1091,3874,3875],{"class":3826},",\n",[1091,3877,3878,3881,3883],{"class":1093,"line":1157},[1091,3879,3880],{"class":3832},"    \"function\"",[1091,3882,3869],{"class":3826},[1091,3884,3885],{"class":3826}," {\n",[1091,3887,3888,3891,3893,3896],{"class":1093,"line":308},[1091,3889,3890],{"class":3832},"        \"name\"",[1091,3892,3869],{"class":3826},[1091,3894,3895],{"class":3832}," \"get_xrpl_account_balance\"",[1091,3897,3875],{"class":3826},[1091,3899,3900,3903,3905,3908],{"class":1093,"line":1178},[1091,3901,3902],{"class":3832},"        \"description\"",[1091,3904,3869],{"class":3826},[1091,3906,3907],{"class":3832}," \"Fetch the XRP balance for an XRPL account\"",[1091,3909,3875],{"class":3826},[1091,3911,3912,3915,3917],{"class":1093,"line":1195},[1091,3913,3914],{"class":3832},"        \"parameters\"",[1091,3916,3869],{"class":3826},[1091,3918,3885],{"class":3826},[1091,3920,3921,3924,3926,3929],{"class":1093,"line":1206},[1091,3922,3923],{"class":3832},"            \"type\"",[1091,3925,3869],{"class":3826},[1091,3927,3928],{"class":3832}," \"object\"",[1091,3930,3875],{"class":3826},[1091,3932,3933,3936,3938],{"class":1093,"line":1220},[1091,3934,3935],{"class":3832},"            \"properties\"",[1091,3937,3869],{"class":3826},[1091,3939,3885],{"class":3826},[1091,3941,3942,3945,3947,3950,3953,3955,3958,3961,3964,3966,3969],{"class":1093,"line":1233},[1091,3943,3944],{"class":3832},"                \"address\"",[1091,3946,3869],{"class":3826},[1091,3948,3949],{"class":3826}," {",[1091,3951,3952],{"class":3832},"\"type\"",[1091,3954,3869],{"class":3826},[1091,3956,3957],{"class":3832}," \"string\"",[1091,3959,3960],{"class":3826},",",[1091,3962,3963],{"class":3832}," \"description\"",[1091,3965,3869],{"class":3826},[1091,3967,3968],{"class":3832}," \"Classic XRPL address starting with r\"",[1091,3970,3971],{"class":3826},"}\n",[1091,3973,3974],{"class":1093,"line":1246},[1091,3975,3976],{"class":3826},"            },\n",[1091,3978,3979,3982,3984,3986,3989],{"class":1093,"line":3638},[1091,3980,3981],{"class":3832},"            \"required\"",[1091,3983,3869],{"class":3826},[1091,3985,3858],{"class":1123},[1091,3987,3988],{"class":3832},"\"address\"",[1091,3990,3991],{"class":1123},"]\n",[1091,3993,3994],{"class":1093,"line":3649},[1091,3995,3996],{"class":3826},"        }\n",[1091,3998,3999],{"class":1093,"line":3660},[1091,4000,4001],{"class":3826},"    }\n",[1091,4003,4004,4007],{"class":1093,"line":3668},[1091,4005,4006],{"class":3826},"}",[1091,4008,3991],{"class":1123},[1091,4010,4011],{"class":1093,"line":3676},[1091,4012,1109],{"emptyLinePlaceholder":314},[1091,4014,4015,4018,4020,4023,4025,4028,4030,4033,4035,4038],{"class":1093,"line":3687},[1091,4016,4017],{"class":1123},"response ",[1091,4019,3820],{"class":3798},[1091,4021,4022],{"class":1123}," client",[1091,4024,349],{"class":3826},[1091,4026,4027],{"class":1123},"chat",[1091,4029,349],{"class":3826},[1091,4031,4032],{"class":1123},"completions",[1091,4034,349],{"class":3826},[1091,4036,4037],{"class":1096},"create",[1091,4039,4040],{"class":3826},"(\n",[1091,4042,4043,4046,4048,4051],{"class":1093,"line":3698},[1091,4044,4045],{"class":3826},"    model",[1091,4047,3820],{"class":3798},[1091,4049,4050],{"class":3832},"\"google\u002Fgemma-4-26B-A4B-it\"",[1091,4052,3875],{"class":3826},[1091,4054,4055,4058,4060,4063,4066,4069,4072,4074,4077,4079,4082],{"class":1093,"line":3709},[1091,4056,4057],{"class":3826},"    messages",[1091,4059,3820],{"class":3798},[1091,4061,4062],{"class":3826},"[{",[1091,4064,4065],{"class":3832},"\"role\"",[1091,4067,4068],{"class":3826},": ",[1091,4070,4071],{"class":3832},"\"user\"",[1091,4073,2028],{"class":3826},[1091,4075,4076],{"class":3832},"\"content\"",[1091,4078,4068],{"class":3826},[1091,4080,4081],{"class":3832},"\"How much XRP does rMCU4... hold?\"",[1091,4083,4084],{"class":3826},"}],\n",[1091,4086,4087,4090,4092],{"class":1093,"line":3717},[1091,4088,4089],{"class":3826},"    tools",[1091,4091,3820],{"class":3798},[1091,4093,4094],{"class":3826},"tools,\n",[1091,4096,4097,4100,4102,4105],{"class":1093,"line":3728},[1091,4098,4099],{"class":3826},"    tool_choice",[1091,4101,3820],{"class":3798},[1091,4103,4104],{"class":3832},"\"auto\"",[1091,4106,3875],{"class":3826},[1091,4108,4110],{"class":1093,"line":4109},25,[1091,4111,3844],{"class":3826},[1091,4113,4115],{"class":1093,"line":4114},26,[1091,4116,1109],{"emptyLinePlaceholder":314},[1091,4118,4120],{"class":1093,"line":4119},27,[1091,4121,4123],{"class":4122},"s15Vz","# response.choices[0].message.tool_calls -> [ChatCompletionMessageToolCall(...)]\n",[10,4125,4126],{},"This is the foundation for agentic workflows that previously required prompt engineering or separate function-calling layers (like LiteLLM's function-call emulation). With Gemma 4, the model speaks tool calls natively.",[10,4128,4129,4130,4133],{},"The reasoning mode is similarly useful. For complex queries, the model emits a private thinking trace before its final answer, we can log it for debugging, show it to users for transparency, or strip it entirely. All via the ",[398,4131,4132],{},"--reasoning-parser"," flag.",[48,4135,4137],{"id":4136},"the-benchmarks","The Benchmarks",[10,4139,4140],{},"We ran a proper benchmark suite from the Spark itself (not over the network, to remove noise): throughput at 100\u002F500\u002F2000-token outputs, time-to-first-token, concurrency scaling, prefix-cache hit\u002Fmiss, long-context prefill, and tool-calling overhead.",[3390,4142,4144],{"id":4143},"single-stream-throughput","Single-stream throughput",[161,4146,4147,4157],{},[164,4148,4149],{},[167,4150,4151,4154],{},[170,4152,4153],{},"Generation length",[170,4155,4156],{},"tok\u002Fs",[179,4158,4159,4167,4177],{},[167,4160,4161,4164],{},[184,4162,4163],{},"100 tokens",[184,4165,4166],{},"46",[167,4168,4169,4172],{},[184,4170,4171],{},"500 tokens",[184,4173,4174],{},[784,4175,4176],{},"51",[167,4178,4179,4182],{},[184,4180,4181],{},"2000 tokens",[184,4183,4184],{},"50",[10,4186,4187],{},"The short-generation number is lower only because warmup dominates. Anything over ~200 tokens hits the steady-state ~50 tok\u002Fs. That matches the theoretical ceiling for 4B active parameters at FP4 on 273 GB\u002Fs bandwidth.",[3390,4189,4191],{"id":4190},"concurrency-the-big-win-for-agentic-workflows","Concurrency, the big win for agentic workflows",[10,4193,4194],{},"This is where the Spark shines. vLLM's continuous batching + Marlin NVFP4 MoE kernel keeps per-request throughput high even as you add parallel clients:",[161,4196,4197,4210],{},[164,4198,4199],{},[167,4200,4201,4204,4207],{},[170,4202,4203],{},"Parallel requests",[170,4205,4206],{},"Aggregate tok\u002Fs",[170,4208,4209],{},"Per-request tok\u002Fs",[179,4211,4212,4221,4232,4243],{},[167,4213,4214,4217,4219],{},[184,4215,4216],{},"1",[184,4218,4184],{},[184,4220,4184],{},[167,4222,4223,4226,4229],{},[184,4224,4225],{},"2",[184,4227,4228],{},"90",[184,4230,4231],{},"45",[167,4233,4234,4237,4240],{},[184,4235,4236],{},"4",[184,4238,4239],{},"161",[184,4241,4242],{},"40",[167,4244,4245,4250,4255],{},[184,4246,4247],{},[784,4248,4249],{},"8",[184,4251,4252],{},[784,4253,4254],{},"271",[184,4256,4257],{},"34",[10,4259,4260],{},"5.4× aggregate throughput at 8 concurrent clients. For a team of devs running agents in parallel, or a single agent making parallel tool-call decisions, this is huge.",[3390,4262,4264],{"id":4263},"prefix-caching-massive-for-agent-loops","Prefix caching, massive for agent loops",[10,4266,4267],{},"Agent workflows resubmit the same system prompt, same tool schema, same conversation history over and over. Measured on an 8K-token prefix:",[161,4269,4270,4279],{},[164,4271,4272],{},[167,4273,4274,4276],{},[170,4275],{},[170,4277,4278],{},"TTFT",[179,4280,4281,4289],{},[167,4282,4283,4286],{},[184,4284,4285],{},"Cold (cache miss)",[184,4287,4288],{},"834 ms",[167,4290,4291,4294],{},[184,4292,4293],{},"Warm (cache hit)",[184,4295,4296],{},[784,4297,4298],{},"102 ms",[10,4300,4301,4304],{},[784,4302,4303],{},"8.2× speedup."," If you're building anything where the same prefix repeats, ReAct loops, multi-turn conversations, a shared system prompt, prefix caching alone pays for running local inference.",[3390,4306,4308],{"id":4307},"ttft-and-long-context","TTFT and long context",[272,4310,4311,4317,4320,4323,4329],{},[32,4312,4313,4314,4316],{},"Short prompt TTFT: ",[784,4315,3233],{}," p50 (excellent)",[32,4318,4319],{},"4K prompt TTFT: 62 ms p50",[32,4321,4322],{},"8K prefill: 73 ms",[32,4324,4325,4326],{},"32K prefill: ",[784,4327,4328],{},"10.1 seconds",[32,4330,4331,4332],{},"100K prefill: ",[784,4333,4334],{},"78 seconds",[10,4336,4337],{},"Long-context prefill is the one real weakness. It's compute-bound on the MoE GEMM kernel, not bandwidth-bound. At 131K context you're paying 80+ seconds of prefill time every cold turn. If you need long-context frequently, design your agents to reuse prefixes so the cache does the heavy lifting.",[3390,4339,4341],{"id":4340},"tool-calling","Tool calling",[10,4343,4344,4345,4348],{},"Works out of the box with ",[398,4346,4347],{},"--enable-auto-tool-choice --tool-call-parser gemma4",". No meaningful overhead versus text-only completion, the model just emits shorter, structured output when it decides to call a tool.",[48,4350,4352],{"id":4351},"trying-to-push-throughput-further","Trying to Push Throughput Further",[10,4354,4355],{},"With the baseline measured, we tried four paths to push single-stream throughput higher. All four hit walls:",[161,4357,4358,4368],{},[164,4359,4360],{},[167,4361,4362,4365],{},[170,4363,4364],{},"Attempt",[170,4366,4367],{},"Result",[179,4369,4370,4381,4391,4407],{},[167,4371,4372,4378],{},[184,4373,4374,4375],{},"Swap MoE backend to ",[398,4376,4377],{},"flashinfer_trtllm",[184,4379,4380],{},"Kernel doesn't support SM 12.1, engine fails to start",[167,4382,4383,4388],{},[184,4384,4374,4385],{},[398,4386,4387],{},"flashinfer_cutlass",[184,4389,4390],{},"Doesn't support GELU activation (Gemma 4 uses GELU)",[167,4392,4393,4404],{},[184,4394,4395,4396,4399,4400,4403],{},"Bump ",[398,4397,4398],{},"--gpu-memory-utilization"," 0.85 → 0.90 + ",[398,4401,4402],{},"--max-num-seqs"," 16 → 32",[184,4405,4406],{},"Zero measurable change, we're not KV-cache-limited at this scale",[167,4408,4409,4412],{},[184,4410,4411],{},"Speculative decoding with Gemma-4-E4B draft",[184,4413,4414],{},"Blocked, vLLM's spec decoding doesn't support multimodal target models",[10,4416,4417],{},"The last one hurt. Spec decoding was the one real lever: a small 4B draft model predicting tokens for the 26B target can deliver 1.5-2× throughput on repetitive outputs. But vLLM explicitly rejects it for multimodal models right now, and Gemma 4 is multimodal. The two features we most wanted, speculative decoding and image input, are mutually exclusive in today's vLLM.",[10,4419,4420,4423],{},[784,4421,4422],{},"Conclusion: the baseline config is already at the hardware + software ceiling."," Generation throughput is memory-bandwidth-bound. Prefill is compute-bound on the only NVFP4 MoE kernel that works on GB10 + Gemma 4 (Marlin). There's no software knob we haven't turned.",[48,4425,4427],{"id":4426},"what-would-change-the-picture","What Would Change the Picture",[10,4429,4430],{},"Future improvements we'll re-test when they land:",[272,4432,4433,4439,4445,4451],{},[32,4434,4435,4438],{},[784,4436,4437],{},"vLLM adds multimodal spec decoding",", unlocks Gemma-4-E4B as draft, expected 1.5-2×",[32,4440,4441,4444],{},[784,4442,4443],{},"FlashInfer adds SM 12.1 + GELU support",", alternative MoE backend, could improve prefill",[32,4446,4447,4450],{},[784,4448,4449],{},"Community ships a clean Gemma 4 AWQ INT4 quant",", Qwen AWQ hit 82 tok\u002Fs on the same hardware",[32,4452,4453,4456],{},[784,4454,4455],{},"NVIDIA publishes an official NVFP4 Gemma 4 build",", likely has better-tuned kernels",[10,4458,4459,4460,4463],{},"Plan: re-run the benchmark suite (checked in at ",[398,4461,4462],{},"projects\u002Fdgx-spark-benchmarks\u002Fbench.py",") every couple of months and see if the landscape has shifted.",[48,4465,4467],{"id":4466},"whats-next-for-us","What's Next for Us",[272,4469,4470,4476,4491],{},[32,4471,4472,4475],{},[784,4473,4474],{},"Multimodal testing",", Gemma 4 accepts image inputs. Document parsing and UI automation are the obvious use cases.",[32,4477,4478,2028,4481,4484,4485,4490],{},[784,4479,4480],{},"Local speech-to-text",[398,4482,4483],{},"faster-whisper"," on the Grace CPU cores, leaving the GPU fully dedicated to Gemma 4. ",[277,4486,4489],{"href":4487,"rel":4488},"https:\u002F\u002Flearn.arm.com\u002Flearning-paths\u002Flaptops-and-desktops\u002Fdgx_spark_voicechatbot\u002F",[281],"Arm published a playbook"," for exactly this split on the Spark.",[32,4492,4493,4496],{},[784,4494,4495],{},"Agent workload benchmarks",", beyond synthetic throughput, measure actual tool-call accuracy and reasoning quality on our real XRPL workflows.",[48,4498,1323],{"id":1322},[272,4500,4501,4508,4514,4521],{},[32,4502,4503],{},[277,4504,4507],{"href":4505,"rel":4506},"https:\u002F\u002Fhuggingface.co\u002Fblog\u002Fgemma4",[281],"Gemma 4 on HuggingFace",[32,4509,4510],{},[277,4511,3362],{"href":4512,"rel":4513},"https:\u002F\u002Fhuggingface.co\u002Fbg-digitalservices\u002FGemma-4-26B-A4B-it-NVFP4",[281],[32,4515,4516],{},[277,4517,4520],{"href":4518,"rel":4519},"https:\u002F\u002Fdocs.vllm.ai\u002Fprojects\u002Frecipes\u002Fen\u002Flatest\u002FGoogle\u002FGemma4.html",[281],"vLLM Gemma 4 recipe",[32,4522,4523],{},[277,4524,4525],{"href":3254},"Our original DGX Spark setup post",[48,4527,1356],{"id":1355},[10,4529,4530,4531,4533,4534,4537,4538,4540,4541,4544],{},"If you have a DGX Spark and want to run Gemma 4: pull the ",[398,4532,3383],{}," image, get a working ",[398,4535,4536],{},"gemma4_patched.py",", and use the ",[398,4539,970],{}," command above. Allow ~20 minutes for the first boot. Reach out at ",[277,4542,1364],{"href":1362,"rel":4543},[281]," if you hit issues, we've documented most of the sharp edges.",[1367,4546,4547],{},"html pre.shiki code .s7eDp, html code.shiki .s7eDp{--shiki-default:#6F42C1}html pre.shiki code .sY4mW, html code.shiki .sY4mW{--shiki-default:#2B5581}html pre.shiki code .sR6ew, html code.shiki .sR6ew{--shiki-default:#24292EFF}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html pre.shiki code .s9AOD, html code.shiki .s9AOD{--shiki-default:#1976D2}html pre.shiki code .s-F7R, html code.shiki .s-F7R{--shiki-default:#D32F2F}html pre.shiki code .siqTm, html code.shiki .siqTm{--shiki-default:#212121}html pre.shiki code .shJU0, html code.shiki .shJU0{--shiki-default:#22863A}html pre.shiki code .s15Vz, html code.shiki .s15Vz{--shiki-default:#C2C3C5}",{"title":298,"searchDepth":299,"depth":299,"links":4549},[4550,4551,4552,4553,4560,4561,4562,4569,4570,4571,4572,4573],{"id":3248,"depth":299,"text":3249},{"id":3287,"depth":299,"text":3288},{"id":3355,"depth":299,"text":3356},{"id":3387,"depth":299,"text":3388,"children":4554},[4555,4556,4557,4558,4559],{"id":3392,"depth":1112,"text":3393},{"id":3426,"depth":1112,"text":3427},{"id":3454,"depth":1112,"text":3455},{"id":3465,"depth":1112,"text":3466},{"id":3476,"depth":1112,"text":3477},{"id":3498,"depth":299,"text":3499},{"id":3782,"depth":299,"text":3783},{"id":4136,"depth":299,"text":4137,"children":4563},[4564,4565,4566,4567,4568],{"id":4143,"depth":1112,"text":4144},{"id":4190,"depth":1112,"text":4191},{"id":4263,"depth":1112,"text":4264},{"id":4307,"depth":1112,"text":4308},{"id":4340,"depth":1112,"text":4341},{"id":4351,"depth":299,"text":4352},{"id":4426,"depth":299,"text":4427},{"id":4466,"depth":299,"text":4467},{"id":1322,"depth":299,"text":1323},{"id":1355,"depth":299,"text":1356},"2026-04-22","TL;DR: We set up Google's Gemma 4 26B-A4B NVFP4 on the XRPL Commons office DGX Spark, with native tool calling, reasoning mode, 131K context, and multimodal input preserved on 128GB of unified memory. We then tried to push throughput further and hit the hardware's real limits. Here's the full story.",{},"\u002Fideas\u002F2026-04-22-dgx-spark-gemma4",{"title":3202,"description":4575},"ideas\u002F2026-04-22-dgx-spark-gemma4","Native tool calling and reasoning mode on Gemma 4 NVFP4 over 128GB of unified memory.",[324,1392,325,4582],"gemma","HJg1bpC4WTA55XH45oFSwAvD6I-PF3WTHHwLgP53Erc",{"id":4585,"title":4586,"body":4587,"canonical_url":307,"chapter":1195,"date":4828,"description":4591,"extension":310,"featured":311,"image":307,"kind":312,"meta":4829,"navigation":314,"path":4830,"seo":4831,"series":317,"stem":4832,"summary":4833,"tags":4834,"work_slug":321,"__hash__":4835},"writing\u002Fideas\u002F2026-06-06-anton-10-gemma-fly.md","Anton, chapter 10: Making Gemma fly",{"type":7,"value":4588,"toc":4819},[4589,4592,4595,4598,4601,4604,4607,4610,4614,4617,4620,4623,4626,4629,4632,4635,4638,4642,4645,4648,4651,4654,4657,4660,4663,4666,4669,4673,4676,4679,4682,4685,4688,4691,4694,4697,4700,4703,4707,4710,4713,4716,4719,4722,4725,4728,4736,4739,4742,4746,4749,4752,4755,4758,4761,4764,4767,4770,4773,4777,4780,4783,4786,4789,4792,4795,4798,4801,4803],[10,4590,4591],{},"The local model on the Spark was Nemotron when this began.",[10,4593,4594],{},"It produced around 58 tokens per second and depended on two unmerged patches I had backported by hand just to load correctly.",[10,4596,4597],{},"Then Gemma 4 arrived.",[10,4599,4600],{},"It was multimodal, capable of native tool use, and much closer to the model I wanted Anton to grow around. I moved local inference back to Gemma and landed in the high forties.",[10,4602,4603],{},"That should have felt like progress.",[10,4605,4606],{},"Instead, it reminded me of an unfinished question from the spring: was that the limit of the hardware, or merely the limit of the first configuration that worked?",[10,4608,4609],{},"I decided to find out with measurements rather than intuition.",[48,4611,4613],{"id":4612},"quantisation-was-not-the-lever","Quantisation was not the lever",[10,4615,4616],{},"The obvious idea was better quantisation.",[10,4618,4619],{},"A new quantisation-aware checkpoint promised near-full quality at four-bit weight precision. The theory was attractive: smaller weights, less memory traffic, more speed.",[10,4621,4622],{},"I sent several research agents after the question from different angles.",[10,4624,4625],{},"They returned with the same answer.",[10,4627,4628],{},"On this workload, the Spark was constrained primarily by memory bandwidth. Any four-bit format moved roughly the same amount of weight data per generated token. A different checkpoint would not change that basic limit.",[10,4630,4631],{},"Worse, the existing path used a low-precision format without receiving the full native-compute benefit from the hardware and software combination. I was paying for patched loading and a specialised kernel without finding a meaningful new speed lever.",[10,4633,4634],{},"Quantisation had delivered the storage win.",[10,4636,4637],{},"It was not where the next latency win lived.",[48,4639,4641],{"id":4640},"the-spec-sheet-is-not-the-benchmark","The spec sheet is not the benchmark",[10,4643,4644],{},"I looked at other machines I already owned.",[10,4646,4647],{},"My work Mac and an idle laptop advertised more memory bandwidth than the Spark. If decode speed were determined by bandwidth alone, they should have won.",[10,4649,4650],{},"They did not.",[10,4652,4653],{},"The same model ran slower.",[10,4655,4656],{},"The architecture was mixture-of-experts, and the efficiency of the available kernels mattered enormously. A theoretical bandwidth ceiling says little about how close the software stack gets to it.",[10,4658,4659],{},"That detour produced a rule worth the time it cost:",[10,4661,4662],{},"The spec sheet is not the benchmark.",[10,4664,4665],{},"Hardware figures describe possibility.",[10,4667,4668],{},"The application experiences kernels, schedulers, caches, frameworks, and actual request shapes.",[48,4670,4672],{"id":4671},"several-tokens-for-one-weight-read","Several tokens for one weight read",[10,4674,4675],{},"The remaining lever was speculative decoding.",[10,4677,4678],{},"A small drafter predicts several likely next tokens. The larger model verifies them together. When the workload is memory-bound, the benefit is structural: one expensive pass over the main model's weights can validate more than one generated token.",[10,4680,4681],{},"I followed a published recipe and initially achieved roughly half its headline result.",[10,4683,4684],{},"The command was the same. My baseline matched theirs.",[10,4686,4687],{},"The difference was the measurement.",[10,4689,4690],{},"Their benchmark isolated decode throughput after prefill. Mine measured wall-clock time as a caller experienced it, including the parts around generation.",[10,4692,4693],{},"Both numbers were valid.",[10,4695,4696],{},"They answered different questions.",[10,4698,4699],{},"The honest user-facing improvement was around 1.4 to 1.9 times, depending on how predictable the output was.",[10,4701,4702],{},"That was less dramatic than the headline and more useful than repeating it.",[48,4704,4706],{"id":4705},"test-the-way-anton-is-used","Test the way Anton is used",[10,4708,4709],{},"Single-request throughput still was not the most important measurement.",[10,4711,4712],{},"Anton repeatedly sends similar prefixes: the same system prompt, tool schemas, conversation history, and agent instructions. In an active conversation, much of that prefix is already represented in the KV cache.",[10,4714,4715],{},"So I tested the real router prompt, real conversation structure, and multiple concurrent requests.",[10,4717,4718],{},"The result that mattered was latency on the common path.",[10,4720,4721],{},"With the prefix cached, an answer that had taken around five seconds could return in roughly half a second.",[10,4723,4724],{},"That improvement did not come from speculative decoding alone.",[10,4726,4727],{},"It came from two independent mechanisms:",[29,4729,4730,4733],{},[32,4731,4732],{},"Speculative decoding reduced the cost of producing new tokens.",[32,4734,4735],{},"Prefix caching avoided recomputing what the model had already seen.",[10,4737,4738],{},"I had spent days looking at quantisation because it was visible and concrete.",[10,4740,4741],{},"The larger win came from understanding the request lifecycle.",[48,4743,4745],{"id":4744},"throughput-is-not-one-number","Throughput is not one number",[10,4747,4748],{},"Then I tried to break the serving stack.",[10,4750,4751],{},"Five hundred requests arrived at once.",[10,4753,4754],{},"They queued. None were dropped. The machine remained stable, and the thermal warnings I had heard about did not reproduce.",[10,4756,4757],{},"Aggregate throughput climbed above a thousand tokens per second across the workload.",[10,4759,4760],{},"That number should not be confused with the speed one person sees. Single-stream latency, decode throughput, time to first token, cached-turn latency, and aggregate throughput are different measurements.",[10,4762,4763],{},"The chapter's real subject became measurement itself.",[10,4765,4766],{},"Performance claims are often disagreements between denominators.",[10,4768,4769],{},"A benchmark can remove prefill, warm the cache, run one stream, batch many streams, or measure only steady-state decoding. Each result may be correct while describing a different system.",[10,4771,4772],{},"The relevant benchmark is the one that resembles the workload you actually care about.",[48,4774,4776],{"id":4775},"the-ceiling-of-one-idea","The ceiling of one idea",[10,4778,4779],{},"By the end of the weekend, local Gemma could answer a live cached conversation in around half a second, accelerate ordinary generation through speculation, sustain heavy concurrent traffic, and fail over to another machine if the Spark stumbled.",[10,4781,4782],{},"The original question had an answer.",[10,4784,4785],{},"The high-forties result had not been the ceiling of the hardware.",[10,4787,4788],{},"It had been the ceiling of one idea: straightforward autoregressive decoding on a single request without fully exploiting the repeated structure around it.",[10,4790,4791],{},"Most of the climb came not from a more exotic model or a more aggressive quantisation.",[10,4793,4794],{},"It came from measuring honestly enough to notice where time was actually being spent.",[10,4796,4797],{},"The lesson is easy to state and hard to practise:",[10,4799,4800],{},"Do not optimise the number that is easiest to measure. Optimise the experience the system actually delivers.",[48,4802,270],{"id":269},[272,4804,4805,4812],{},[32,4806,4807],{},[277,4808,4811],{"href":4809,"rel":4810},"https:\u002F\u002Fdocs.vllm.ai\u002Fen\u002Flatest\u002Ffeatures\u002Fspec_decode.html",[281],"vLLM speculative decoding",[32,4813,4814],{},[277,4815,4818],{"href":4816,"rel":4817},"https:\u002F\u002Fdocs.vllm.ai\u002Fen\u002Flatest\u002Ffeatures\u002Fautomatic_prefix_caching.html",[281],"vLLM automatic prefix caching",{"title":298,"searchDepth":299,"depth":299,"links":4820},[4821,4822,4823,4824,4825,4826,4827],{"id":4612,"depth":299,"text":4613},{"id":4640,"depth":299,"text":4641},{"id":4671,"depth":299,"text":4672},{"id":4705,"depth":299,"text":4706},{"id":4744,"depth":299,"text":4745},{"id":4775,"depth":299,"text":4776},{"id":269,"depth":299,"text":270},"2026-06-06",{},"\u002Fideas\u002F2026-06-06-anton-10-gemma-fly",{"title":4586,"description":4591},"ideas\u002F2026-06-06-anton-10-gemma-fly","Speculative decoding, prefix caching, and the discipline of measuring the experience the system actually delivers.",[321,322,323,324,325],"wYUBtxCLzz6cMsvQ7o4JDPmCa-G03R0ML5BhsqjsPSw",{"id":4837,"title":4838,"body":4839,"canonical_url":307,"chapter":1206,"date":5060,"description":4843,"extension":310,"featured":311,"image":307,"kind":312,"meta":5061,"navigation":314,"path":5062,"seo":5063,"series":317,"stem":5064,"summary":5065,"tags":5066,"work_slug":321,"__hash__":5067},"writing\u002Fideas\u002F2026-06-26-anton-11-the-gate.md","Anton, chapter 11: The gate",{"type":7,"value":4840,"toc":5051},[4841,4844,4847,4850,4853,4856,4859,4862,4865,4869,4872,4875,4878,4881,4884,4887,4890,4893,4897,4900,4903,4906,4909,4912,4915,4918,4921,4924,4928,4931,4934,4937,4940,4943,4946,4949,4952,4955,4958,4961,4964,4968,4971,4974,4977,4980,4983,4986,4990,4993,4996,4999,5002,5005,5008,5012,5015,5018,5021,5024,5027,5030,5033,5035],[10,4842,4843],{},"The model was fast now, but every fix still ended with me.",[10,4845,4846],{},"A change could be written, reviewed, merged, and tested automatically. Then it sat in the repository until I opened a terminal, connected to the home server, pulled the code, rebuilt the containers, and watched the health checks.",[10,4848,4849],{},"The intelligence had accelerated.",[10,4851,4852],{},"Delivery had not.",[10,4854,4855],{},"The question for this period was whether the loop could close completely: issue, fix, review, merge, deploy, running on the family's phones.",[10,4857,4858],{},"No person in the middle.",[10,4860,4861],{},"But speed was not the interesting part. The challenge was closing the loop without allowing an automated mistake to reach the five people who depended on Anton.",[10,4863,4864],{},"A pipeline that deploys quickly and deploys the wrong thing is worse than deploying by hand.",[48,4866,4868],{"id":4867},"the-path-to-production","The path to production",[10,4870,4871],{},"The foundation was conventional.",[10,4873,4874],{},"Nothing could merge until type checking, the build, and the complete test suite passed. Afterward, a deployment job joined the private Tailscale network and invoked the update process on the home server.",[10,4876,4877],{},"Production did not expose SSH to the public internet. The runner entered the tailnet like any other authorised machine.",[10,4879,4880],{},"On the server, the existing deployment script remained in control: pull the new version, build it, start it, probe its health, and restore the previous image if the probe failed.",[10,4882,4883],{},"I already trusted that path.",[10,4885,4886],{},"The automation did not replace it. It called it.",[10,4888,4889],{},"That distinction mattered. Closing the loop did not require inventing a second deployment mechanism.",[10,4891,4892],{},"It required making the proven one reachable.",[48,4894,4896],{"id":4895},"a-gate-that-was-confidently-wrong","A gate that was confidently wrong",[10,4898,4899],{},"The post-deployment smoke gate was where I got educated.",[10,4901,4902],{},"The idea was simple: ask the live system a small number of representative questions. Keep the new deployment only when it answers correctly.",[10,4904,4905],{},"The first version called an endpoint that did not exist. Every deployment returned a 404, rolled back, and paged me with a critical alert.",[10,4907,4908],{},"Loudly wrong is irritating, but useful. It gets attention.",[10,4910,4911],{},"The second version reached the right endpoint and still rolled everything back. One test expected a conversation-search request to route to the knowledge domain. Anton correctly routed it to memory.",[10,4913,4914],{},"The system was right.",[10,4916,4917],{},"The gate was wrong.",[10,4919,4920],{},"The gate won anyway.",[10,4922,4923],{},"That is the authority of an automated check: it does not need to understand more than the system it judges. It only needs permission to stop it.",[48,4925,4927],{"id":4926},"the-gate-that-never-ran","The gate that never ran",[10,4929,4930],{},"The third failure was worse.",[10,4932,4933],{},"Six model calls took longer than the deployment timeout, so the smoke check was silently skipped.",[10,4935,4936],{},"The pipeline stayed green.",[10,4938,4939],{},"For several deployments, the gate appeared to be protecting production while never producing a verdict at all.",[10,4941,4942],{},"A false failure is noisy. Someone investigates it.",[10,4944,4945],{},"A silently skipped check reports confidence it never earned.",[10,4947,4948],{},"That is one of the most dangerous failure modes in automation because the output looks exactly like success.",[10,4950,4951],{},"The gate had to fit inside its own operational budget.",[10,4953,4954],{},"I ran the cases concurrently, reduced the blocking set, and separated checks into two tiers.",[10,4956,4957],{},"The hard tier could roll back a deployment. It contained only fast, essential family-facing paths.",[10,4959,4960],{},"The soft tier tested broader behaviour and reported regressions without blocking the release.",[10,4962,4963],{},"A check should have authority proportional to both its importance and its reliability.",[48,4965,4967],{"id":4966},"the-day-it-paid-for-itself","The day it paid for itself",[10,4969,4970],{},"A few days later, the hard gate rejected a deployment that had genuinely broken two family-facing paths.",[10,4972,4973],{},"The old version returned automatically.",[10,4975,4976],{},"I had not noticed the regression.",[10,4978,4979],{},"The mechanism I built because I did not trust myself caught it before the family did.",[10,4981,4982],{},"That was the moment the work paid for itself.",[10,4984,4985],{},"Not when the pipeline deployed something successfully, but when it refused to.",[48,4987,4989],{"id":4988},"keeping-the-builders-alive","Keeping the builders alive",[10,4991,4992],{},"The agents that picked up issues and reviewed pull requests were long-lived sessions.",[10,4994,4995],{},"After enough work, their context filled up. They became slower, less precise, and occasionally strange.",[10,4997,4998],{},"A watchdog learned to identify stalled sessions, terminate them, and start fresh ones.",[10,5000,5001],{},"A supervisor watched the issue queue. When there was no work, it put the fleet to sleep. When an issue became ready, it woke the necessary workers.",[10,5003,5004],{},"That small piece of stagecraft made the difference between an automation that survived an afternoon and one that could operate for weeks.",[10,5006,5007],{},"Persistent automation needs a lifecycle, not only a loop.",[48,5009,5011],{"id":5010},"the-quiet-result","The quiet result",[10,5013,5014],{},"Where it landed was quieter than I expected.",[10,5016,5017],{},"I file an issue and describe the change.",[10,5019,5020],{},"Later, the fix is either running on the family's phones or it never left the repository.",[10,5022,5023],{},"In both cases, I did not touch the server.",[10,5025,5026],{},"The loop closes on itself.",[10,5028,5029],{},"What I notice is not the speed.",[10,5031,5032],{},"The part I trust most is the one whose only purpose is to stop everything else.",[48,5034,270],{"id":269},[272,5036,5037,5044],{},[32,5038,5039],{},[277,5040,5043],{"href":5041,"rel":5042},"https:\u002F\u002Fdocs.github.com\u002Fen\u002Factions\u002Fhosting-your-own-runners\u002Fmanaging-self-hosted-runners\u002Fabout-self-hosted-runners",[281],"GitHub Actions: self-hosted runners",[32,5045,5046],{},[277,5047,5050],{"href":5048,"rel":5049},"https:\u002F\u002Ftailscale.com\u002Fkb\u002F1276\u002Ftailscale-github-action",[281],"Tailscale GitHub Action",{"title":298,"searchDepth":299,"depth":299,"links":5052},[5053,5054,5055,5056,5057,5058,5059],{"id":4867,"depth":299,"text":4868},{"id":4895,"depth":299,"text":4896},{"id":4926,"depth":299,"text":4927},{"id":4966,"depth":299,"text":4967},{"id":4988,"depth":299,"text":4989},{"id":5010,"depth":299,"text":5011},{"id":269,"depth":299,"text":270},"2026-06-26",{},"\u002Fideas\u002F2026-06-26-anton-11-the-gate",{"title":4838,"description":4843},"ideas\u002F2026-06-26-anton-11-the-gate","Closing the loop from issue to production, and learning that the hard part is a smoke gate that earns its authority.",[321,322,323,1392],"TU95-EH8qF4VDvH0smIao4L1w-H2x1RTSQzhCSmjdJA",{"id":5069,"title":5070,"body":5071,"canonical_url":307,"chapter":1220,"date":5277,"description":5075,"extension":310,"featured":311,"image":307,"kind":312,"meta":5278,"navigation":314,"path":5279,"seo":5280,"series":317,"stem":5281,"summary":5282,"tags":5283,"work_slug":321,"__hash__":5284},"writing\u002Fideas\u002F2026-07-14-anton-12-claims-not-properties.md","Anton, chapter 12: Claims, not properties",{"type":7,"value":5072,"toc":5270},[5073,5076,5079,5082,5085,5088,5091,5094,5098,5101,5104,5107,5110,5113,5116,5119,5122,5125,5128,5131,5134,5138,5141,5144,5147,5150,5153,5156,5159,5162,5165,5168,5172,5175,5178,5181,5184,5187,5190,5193,5196,5199,5202,5205,5208,5211,5215,5218,5221,5224,5227,5230,5233,5236,5239,5242,5245,5248,5252,5255,5258,5261,5264,5267],[10,5074,5075],{},"Two claims sat at the centre of Anton's identity:",[10,5077,5078],{},"It remembers everything I tell it.",[10,5080,5081],{},"It knows how to use all of its tools.",[10,5083,5084],{},"One ordinary exchange on my phone disproved both within the same hour.",[10,5086,5087],{},"That was the shape of the week. Not a difficult bug hunt, but a humbling one.",[10,5089,5090],{},"The features had shipped. The capabilities existed. The tests were green.",[10,5092,5093],{},"Nobody had asked the live system whether the promises were true.",[48,5095,5097],{"id":5096},"the-assistant-that-could-not-see-its-tools","The assistant that could not see its tools",[10,5099,5100],{},"I asked Anton to add something to the family calendar.",[10,5102,5103],{},"It replied that it could not create events directly and offered to set a reminder instead.",[10,5105,5106],{},"Polite. Plausible. Entirely wrong.",[10,5108,5109],{},"The calendar capability existed. I had built and tested it.",[10,5111,5112],{},"The execution trace explained the contradiction.",[10,5114,5115],{},"The family agent had started with only one tool: generic help. None of its real skills were visible.",[10,5117,5118],{},"With no usable capability, the model improvised. It attempted to call a tool named calendar, received an \"unknown tool\" error, tried again, and eventually produced the best answer it could.",[10,5120,5121],{},"Anton was not refusing to use the calendar.",[10,5123,5124],{},"It was blind and being polite about it.",[10,5126,5127],{},"The skills were alive in the runner the entire time. A boot-order race had populated the agent's tool cache before registration completed, leaving the family agent permanently convinced that almost nothing existed.",[10,5129,5130],{},"A capability can be healthy in one service and absent from the system that needs it.",[10,5132,5133],{},"\"Registered\" is not the same as \"available.\"",[48,5135,5137],{"id":5136},"total-recall","Total recall",[10,5139,5140],{},"While reading the traces, I inspected memory capture.",[10,5142,5143],{},"This was the mechanism that should extract durable facts from ordinary conversations, the engine behind \"Anton remembers everything.\"",[10,5145,5146],{},"It had never succeeded.",[10,5148,5149],{},"Not once.",[10,5151,5152],{},"Roughly 52,000 assistant messages had passed through the system since the feature shipped, and zero had produced captured memory.",[10,5154,5155],{},"A hand-written SQL fragment passed a raw date object into a query parameter. Every capture attempt failed on the same statement.",[10,5157,5158],{},"The error appeared continuously in a log nobody was reading.",[10,5160,5161],{},"The honest description of the feature was not \"Anton remembers everything.\"",[10,5163,5164],{},"It was \"Anton has written down nothing.\"",[10,5166,5167],{},"The gap between those sentences existed for weeks because the system had been tested around the feature rather than through it.",[48,5169,5171],{"id":5170},"fixing-the-first-bug-reveals-the-second","Fixing the first bug reveals the second",[10,5173,5174],{},"Correcting the SQL error did not restore the missing memory.",[10,5176,5177],{},"The backfill intended to recover the lost weeks sent most of its jobs to the dead-letter queue.",[10,5179,5180],{},"My first diagnosis was confident and wrong.",[10,5182,5183],{},"A forensic search suggested the deployed code was stale. I spent time investigating a build problem that did not exist because I had searched a re-export file where the symbol could never appear, whether the fix was present or not.",[10,5185,5186],{},"The check returned a false negative.",[10,5188,5189],{},"I treated it as evidence.",[10,5191,5192],{},"The real failure was in the lock.",[10,5194,5195],{},"The backfill launched more than twenty jobs in a burst. They all competed for one per-user lock. The first job acquired it and began classification. Every other job retried three times over roughly fifteen seconds and exhausted its attempts while the first was still running.",[10,5197,5198],{},"The locking code contained a reasonable assumption: by the time a job retries, the previous holder has usually finished.",[10,5200,5201],{},"That was true for the organic workload it had been designed around, one message arriving at a time.",[10,5203,5204],{},"It was false for a backfill.",[10,5206,5207],{},"The code worked exactly as designed.",[10,5209,5210],{},"The design had never met this load.",[48,5212,5214],{"id":5213},"green-somewhere","Green somewhere",[10,5216,5217],{},"What came out of the week was more than two fixes.",[10,5219,5220],{},"Memory capture gained an end-to-end test through the real database driver rather than a mock that always returned success.",[10,5222,5223],{},"Tool availability became something the live agent had to demonstrate, not something inferred from registration elsewhere.",[10,5225,5226],{},"Because every failure that week had been green somewhere.",[10,5228,5229],{},"The feature was deployed.",[10,5231,5232],{},"The skill was registered.",[10,5234,5235],{},"The mock passed.",[10,5237,5238],{},"The mandate appeared in the system prompt.",[10,5240,5241],{},"Each statement was technically true and operationally irrelevant.",[10,5243,5244],{},"The only thing nobody had done was send a real message to the live box and verify the promise from beginning to end.",[10,5246,5247],{},"It took a screenshot from an actual user to expose both failures.",[48,5249,5251],{"id":5250},"earn-the-claim","Earn the claim",[10,5253,5254],{},"I trust the mandate a little less now and the traces considerably more.",[10,5256,5257],{},"That is the right trade.",[10,5259,5260],{},"\"Remembers everything\" and \"uses all its tools\" are not properties a system acquires because they appear in a design document.",[10,5262,5263],{},"They are claims.",[10,5265,5266],{},"A claim has to be earned on the live system, through the real path, under the workload it is meant to survive.",[10,5268,5269],{},"And it has to keep earning it every time you repeat it.",{"title":298,"searchDepth":299,"depth":299,"links":5271},[5272,5273,5274,5275,5276],{"id":5096,"depth":299,"text":5097},{"id":5136,"depth":299,"text":5137},{"id":5170,"depth":299,"text":5171},{"id":5213,"depth":299,"text":5214},{"id":5250,"depth":299,"text":5251},"2026-07-14",{},"\u002Fideas\u002F2026-07-14-anton-12-claims-not-properties",{"title":5070,"description":5075},"ideas\u002F2026-07-14-anton-12-claims-not-properties","One screenshot disproves two core promises, and every failure turns out to have been green somewhere.",[321,322,323],"l-T_K3yIxmd4bItR5y5VHZxhVzRWBx__k7QeSwa5U5w",{"id":5286,"title":5287,"body":5288,"canonical_url":307,"chapter":307,"date":5277,"description":5292,"extension":310,"featured":314,"image":307,"kind":2723,"meta":5373,"navigation":314,"path":5374,"seo":5375,"series":307,"stem":5376,"summary":5377,"tags":5378,"work_slug":307,"__hash__":5380},"writing\u002Fideas\u002F2026-07-14-bring-your-own-ai.md","Bring Your Own AI",{"type":7,"value":5289,"toc":5371},[5290,5293,5296,5299,5302,5310,5313,5316,5319,5322,5325,5328,5331,5334,5337,5340,5343,5346,5349,5356,5359,5362,5365,5368],[10,5291,5292],{},"Whenever we build a product now, we ask the same question: where could AI make this experience easier?",[10,5294,5295],{},"Replace a form with a conversation or let someone upload documents instead. Translate existing material in any language. Query data using natural language.",[10,5297,5298],{},"We've built several of these experiences at Commons. They work. Often incredibly well.",[10,5300,5301],{},"One design pattern has proven especially effective: instead of letting the model generate SQL, API calls, or business logic directly, we define a small domain-specific language.",[10,5303,5304,5307],{},[784,5305,5306],{},"The application owns the semantics.",[784,5308,5309],{},"The model translates the user's intent into that language.",[10,5311,5312],{},"The AI doesn't need to understand the entire system. It just needs to speak the application's language.",[10,5314,5315],{},"That separation of concerns makes the system more reliable, easier to evolve, and much simpler to reason about.",[10,5317,5318],{},"One downside of these new experiences is that every AI feature becomes its own AI product. You're suddenly managing providers, API keys, monitoring, costs, prompts, and ever-changing model capabilities. The rest of your application wants stability; the AI stack refuses to stand still.",[10,5320,5321],{},"This got me thinking.",[10,5323,5324],{},"If the application already owns the semantics, why does it also need to own the intelligence? What if applications simply welcomed agents as first-class users?",[10,5326,5327],{},"Give people secure agent tokens with exactly the permissions they already have. Expose the application's capabilities through a clean API. Whether the interface is MCP or something else almost doesn't matter.",[10,5329,5330],{},"Now users can bring their own AI.",[10,5332,5333],{},"The application still has a great interface. But Claude, ChatGPT, Gemini, or whatever comes next, can also interact with it on the user's behalf.",[10,5335,5336],{},"That AI already knows how the user writes. It speaks their language. It has access to the files they chose to share. It can combine information across several applications instead of being trapped inside one carefully designed chatbot.",[10,5338,5339],{},"The application becomes stable.",[10,5341,5342],{},"The intelligence becomes portable.",[10,5344,5345],{},"There is a loose parallel with identity. An application doesn't necessarily need to create and own your identity; it needs a reliable way to establish who you are and what you're allowed to do.",[10,5347,5348],{},"Perhaps intelligence will become similarly portable.",[10,5350,5351,5352,5355],{},"Instead of every product attempting to own its own AI experience, products may simply become places where ",[784,5353,5354],{},"your"," AI gets work done.",[10,5357,5358],{},"We've started calling this Bring Your Own AI, and we're gradually introducing it across our applications.",[10,5360,5361],{},"Maybe everyone will keep embedding chatbots forever.",[10,5363,5364],{},"Or perhaps, in a few years, building a separate AI into every product will feel as unnecessary as every website implementing its own login system.",[10,5366,5367],{},"I don't know.",[10,5369,5370],{},"I just know this direction feels surprisingly liberating.",{"title":298,"searchDepth":299,"depth":299,"links":5372},[],{},"\u002Fideas\u002F2026-07-14-bring-your-own-ai",{"title":5287,"description":5292},"ideas\u002F2026-07-14-bring-your-own-ai","If the application owns the semantics, why does it also need to own the intelligence? Welcome agents as first-class users: the application becomes stable, the intelligence becomes portable.",[322,323,5379,2156],"product","baUOXvNBny6TWwuc15FqElb298wYAUTCsj4w4bJOVAQ",{"id":5382,"title":5383,"body":5384,"canonical_url":307,"chapter":307,"date":5277,"description":5446,"extension":310,"featured":314,"image":307,"kind":2723,"meta":5447,"navigation":314,"path":5448,"seo":5449,"series":307,"stem":5450,"summary":5451,"tags":5452,"work_slug":307,"__hash__":5455},"writing\u002Fideas\u002F2026-07-14-ludicrous-speed.md","Ludicrous Speed",{"type":7,"value":5385,"toc":5444},[5386,5392,5395,5398,5401,5404,5410,5413,5420,5423,5426,5429,5432,5435,5438,5441],[10,5387,5388,5389,349],{},"We're trying a new way of building products at Commons. I call it ",[784,5390,5391],{},"ludicrous speed",[10,5393,5394],{},"It only works if you suspend disbelief for a while. Let go of a few old assumptions about software development and embrace the vibe.",[10,5396,5397],{},"The reason it works is that almost everything we build shares the same DNA. Forty or so applications. One framework (Nuxt). The same UI library. The same databases. The same deployment pipeline. The same monitoring. Once you've removed enough variation, an LLM stops generating code and starts completing patterns.",[10,5399,5400],{},"Git has become the coordination layer. Ideas become issues. Issues become pull requests. Pull requests become reviews. Reviews become merges. Merges deploy automatically to production. It's a bit like Pieter Levels' philosophy of living in production: ship first, fix quickly, keep moving.",[10,5402,5403],{},"My job has changed more than I expected. I spend almost no time writing code.",[10,5405,5406,5407,349],{},"Instead, I collect feedback and think carefully about specifications. During meetings I'll describe bugs, annotate screenshots, sketch features, and let Claude turn everything into issues. At other times I think about functionality, scope, and (more importantly) what can be removed, simplified, or consolidated. I have LLMs challenge those ideas before turning them into issues. I'll polish them, group a few into epics if it makes sense, and mark them ",[784,5408,5409],{},"Ready",[10,5411,5412],{},"From there, the system takes over.",[10,5414,5415,5416,5419],{},"Cloud instances pick up the work, implement it, review one another, fix what needs fixing, merge to ",[398,5417,5418],{},"main",", and deploy. I mostly watch the conveyor belt, stepping in when the pipeline gets stuck or an agent needs a nudge. Most of the time I simply receive a WhatsApp notification telling me a feature is live.",[10,5421,5422],{},"The remarkable part isn't the automation.",[10,5424,5425],{},"It's the latency.",[10,5427,5428],{},"I can walk out of a product meeting, spend ten minutes polishing the issues, and by the time I've grabbed a coffee, people are using the changes in production. The same applies to ideas from the rest of the team. They create issues, or simply tell someone on the tech team what they want, and minutes later it's real software.",[10,5430,5431],{},"That's what ludicrous speed feels like.",[10,5433,5434],{},"There are no natural pauses anymore. The default state is shipping.",[10,5436,5437],{},"It's not for everyone. Sometimes we deliberately slow things down, add approvals, or gate releases. But for many of our internal products, we've found that continuous deployment with AI in the loop is not only viable, it's genuinely more enjoyable.",[10,5439,5440],{},"I don't know if this is where software development is heading.",[10,5442,5443],{},"I just know I've never had more fun building products.",{"title":298,"searchDepth":299,"depth":299,"links":5445},[],"We're trying a new way of building products at Commons. I call it ludicrous speed.",{},"\u002Fideas\u002F2026-07-14-ludicrous-speed",{"title":5383,"description":5446},"ideas\u002F2026-07-14-ludicrous-speed","How we build products at Commons now. Ideas become issues, agents turn them into merged, deployed software, and the default state is shipping.",[322,323,5453,5454],"claude-code","automation","JcqZqHIfUMjXht1X0Q3gc5FjnPLiRvHl1_rSw52GUX8",{"id":5457,"title":5458,"body":5459,"canonical_url":307,"chapter":307,"date":5559,"description":5463,"extension":310,"featured":314,"image":307,"kind":2723,"meta":5560,"navigation":314,"path":5561,"seo":5562,"series":307,"stem":5563,"summary":5564,"tags":5565,"work_slug":307,"__hash__":5568},"writing\u002Fideas\u002F2026-07-15-the-flame.md","The Flame",{"type":7,"value":5460,"toc":5557},[5461,5464,5467,5470,5473,5476,5479,5482,5485,5488,5491,5494,5497,5500,5503,5506,5509,5512,5515,5518,5521,5524,5527,5530,5533,5536,5539,5542,5545,5548,5551,5554],[10,5462,5463],{},"Life may not be unique.",[10,5465,5466],{},"Given enough planets and enough time, perhaps it emerges over and over again throughout the universe.",[10,5468,5469],{},"I hope that's true.",[10,5471,5472],{},"But hoping new life will appear is a bit like hoping a new flame will suddenly ignite in a cold room.",[10,5474,5475],{},"Maybe it can happen.",[10,5477,5478],{},"If you're trying to keep a fire alive, though, that's not a strategy.",[10,5480,5481],{},"You protect the flames that are already burning.",[10,5483,5484],{},"As far as we know, every living thing on Earth descends from a single chain of life that has remained unbroken for nearly four billion years. Every tree, every whale, every bacterium, every human being is part of that same continuous flame.",[10,5486,5487],{},"Perhaps that's how we should think about life.",[10,5489,5490],{},"Not as something we own.",[10,5492,5493],{},"Not even as something particularly rare.",[10,5495,5496],{},"But as something extraordinarily precious because, for now, it's the only flame we can actually see.",[10,5498,5499],{},"I sometimes wonder if we've chosen the wrong objective function for civilization.",[10,5501,5502],{},"We optimize for GDP.",[10,5504,5505],{},"For efficiency.",[10,5507,5508],{},"For quarterly earnings.",[10,5510,5511],{},"For convenience.",[10,5513,5514],{},"Even sustainability feels strangely defensive. The goal seems to be simply not to make things worse.",[10,5516,5517],{},"What if the objective were different?",[10,5519,5520],{},"What if our goal was to maximize the flourishing of life?",[10,5522,5523],{},"Not just human life.",[10,5525,5526],{},"Life.",[10,5528,5529],{},"That single change in perspective quietly rearranges everything.",[10,5531,5532],{},"A forest isn't valuable because it stores carbon.",[10,5534,5535],{},"It's valuable because it is life creating the conditions for more life.",[10,5537,5538],{},"Education isn't valuable because it increases salaries.",[10,5540,5541],{},"It's valuable because knowledge compounds across generations.",[10,5543,5544],{},"Scientific research, public health, resilient institutions, open-source software, renewable energy, space exploration: they stop looking like unrelated causes.",[10,5546,5547],{},"They're all investments in the same thing.",[10,5549,5550],{},"Keeping the flame alive.",[10,5552,5553],{},"Perhaps that's all civilization really is.",[10,5555,5556],{},"One generation tending a fire that began billions of years before it arrived, hoping it burns a little brighter after it's gone.",{"title":298,"searchDepth":299,"depth":299,"links":5558},[],"2026-07-15",{},"\u002Fideas\u002F2026-07-15-the-flame",{"title":5458,"description":5463},"ideas\u002F2026-07-15-the-flame","Every living thing on Earth is part of one unbroken chain nearly four billion years old. What if civilization's objective function were keeping that flame alive?",[2727,5566,5567],"life","civilization","Hh7i85NOUULaRYQ750bbpvqmxOCTKDRPFOBczBKIZbE",{"id":5570,"title":5571,"body":5572,"canonical_url":307,"chapter":1233,"date":5752,"description":5576,"extension":310,"featured":311,"image":307,"kind":312,"meta":5753,"navigation":314,"path":5754,"seo":5755,"series":317,"stem":5756,"summary":5757,"tags":5758,"work_slug":321,"__hash__":5759},"writing\u002Fideas\u002F2026-07-29-anton-13-false-green.md","Anton, chapter 13: False green",{"type":7,"value":5573,"toc":5746},[5574,5577,5580,5583,5586,5589,5592,5595,5598,5601,5605,5608,5611,5614,5617,5620,5623,5626,5629,5632,5635,5639,5642,5645,5648,5651,5654,5657,5660,5664,5667,5670,5673,5676,5679,5682,5685,5688,5691,5694,5697,5700,5704,5707,5710,5713,5716,5719,5722,5725,5728,5731,5734,5737,5740,5743],[10,5575,5576],{},"I asked how family-chat monitoring had been doing.",[10,5578,5579],{},"The answer was zero messages in seven days.",[10,5581,5582],{},"Not a quiet week.",[10,5584,5585],{},"Exactly zero.",[10,5587,5588],{},"The bridge connecting Anton to the family chat had been dead since July 17. The chat network had logged it out, the bridge accepted the logout, and for twelve days nothing noticed.",[10,5590,5591],{},"No inbound messages reached Anton. Watch mode had nothing to watch. Memory had nothing to store.",[10,5593,5594],{},"No outbound messages left either. Proactive nudges, deployment notices, and status updates all disappeared against a dead socket.",[10,5596,5597],{},"For nearly two weeks, I thought Anton was watching the family chat.",[10,5599,5600],{},"It was watching nothing.",[48,5602,5604],{"id":5603},"healthy-port-dead-service","Healthy port, dead service",[10,5606,5607],{},"The health check remained green the entire time.",[10,5609,5610],{},"It tested the bridge's HTTP port.",[10,5612,5613],{},"The HTTP server was alive.",[10,5615,5616],{},"The underlying socket to the chat network was not.",[10,5618,5619],{},"Zero restarts. Green dashboard. A service that appeared to be doing its job while performing none of its actual purpose.",[10,5621,5622],{},"A health check that measures the wrong layer is worse than having no health check.",[10,5624,5625],{},"No check tells you that you do not know.",[10,5627,5628],{},"A false-green check manufactures confidence, and confidence is exactly what green is designed to produce.",[10,5630,5631],{},"The system was not unmonitored.",[10,5633,5634],{},"It was incorrectly monitored.",[48,5636,5638],{"id":5637},"the-alarm-shared-the-fire","The alarm shared the fire",[10,5640,5641],{},"The second problem was structural.",[10,5643,5644],{},"Every alert Anton could send used the same chat channel that had failed.",[10,5646,5647],{},"The system's way of telling me that the bridge was down passed through the bridge.",[10,5649,5650],{},"An outage on that channel was therefore incapable of reporting itself.",[10,5652,5653],{},"The alarm and the fire shared the same wire.",[10,5655,5656],{},"This was not merely a missing fallback. It was a monitoring design with identical failure domains for the observed system and its alert path.",[10,5658,5659],{},"A monitor that can speak only through the thing it monitors is not independent evidence of anything.",[48,5661,5663],{"id":5662},"recovery","Recovery",[10,5665,5666],{},"Restoring the connection was fiddly.",[10,5668,5669],{},"I reset authentication and paired the bridge again using a code that rotated every twenty seconds. Once reconnected, the bridge resumed processing without flooding my phone with twelve days of old events.",[10,5671,5672],{},"The deduplication held.",[10,5674,5675],{},"It knew what had already been seen.",[10,5677,5678],{},"That was one small piece of earlier plumbing proving its value exactly when a weaker design would have made the recovery painful.",[10,5680,5681],{},"While investigating, I found a second silent failure.",[10,5683,5684],{},"The home server's deployment credential had also expired.",[10,5686,5687],{},"Deployments had been reporting success while the machine continued running stale code.",[10,5689,5690],{},"Again, the reporting layer could not see the layer that had failed.",[10,5692,5693],{},"Again, green meant less than I thought it did.",[10,5695,5696],{},"I replaced the credential with one that did not belong to the same class of expiring tokens. The durable fix was not to renew it more carefully.",[10,5698,5699],{},"It was to remove quiet expiration as a possible failure mode.",[48,5701,5703],{"id":5702},"measure-the-purpose","Measure the purpose",[10,5705,5706],{},"The chat bridge health check now reflects the state of the actual network connection rather than the existence of an HTTP process.",[10,5708,5709],{},"A repair page turns logout into an actionable state rather than a silent one.",[10,5711,5712],{},"Critical alerts have a second outbound route that does not share the bridge's fate.",[10,5714,5715],{},"These are unglamorous changes.",[10,5717,5718],{},"They are also the changes that determine whether everything above them is real.",[10,5720,5721],{},"The rule I carried out of those fifteen days was:",[10,5723,5724],{},"False green is more dangerous than red.",[10,5726,5727],{},"Red gets attention.",[10,5729,5730],{},"Green gets trusted.",[10,5732,5733],{},"And trust built on the wrong measurement survives until the precise moment it was supposed to protect you.",[10,5735,5736],{},"A health check must test the purpose of the service, not merely the easiest component to reach.",[10,5738,5739],{},"A monitor must be able to report through a failure domain independent of the thing it watches.",[10,5741,5742],{},"Otherwise, it is not watching.",[10,5744,5745],{},"It is guessing.",{"title":298,"searchDepth":299,"depth":299,"links":5747},[5748,5749,5750,5751],{"id":5603,"depth":299,"text":5604},{"id":5637,"depth":299,"text":5638},{"id":5662,"depth":299,"text":5663},{"id":5702,"depth":299,"text":5703},"2026-07-29",{},"\u002Fideas\u002F2026-07-29-anton-13-false-green",{"title":5571,"description":5576},"ideas\u002F2026-07-29-anton-13-false-green","Twelve days of a dead chat bridge behind a green health check, and an alarm that shared a wire with the fire.",[321,322,323],"YSAhtyeZUzouTK6ONzRhmPmR_mCuaA3mkDwVD327Yzw",{"id":5761,"title":5762,"body":5763,"canonical_url":307,"chapter":1246,"date":5975,"description":5767,"extension":310,"featured":311,"image":307,"kind":312,"meta":5976,"navigation":314,"path":5977,"seo":5978,"series":317,"stem":5979,"summary":5980,"tags":5981,"work_slug":321,"__hash__":5982},"writing\u002Fideas\u002F2026-08-04-anton-14-evidence-and-authority.md","Anton, chapter 14: Evidence and authority",{"type":7,"value":5764,"toc":5968},[5765,5768,5771,5774,5777,5780,5783,5786,5789,5792,5796,5799,5802,5805,5808,5811,5814,5817,5820,5823,5827,5830,5833,5836,5839,5842,5845,5848,5852,5855,5858,5861,5864,5867,5870,5873,5876,5879,5883,5886,5889,5892,5895,5898,5901,5904,5907,5910,5913,5916,5919,5922,5926,5929,5932,5935,5938,5941,5944,5947,5950,5953,5956,5959,5962,5965],[10,5766,5767],{},"The first unprompted message Anton ever sent me arrived without anyone asking it a question.",[10,5769,5770],{},"Someone in the family group mentioned a pickup. A moment later, my phone displayed a quiet heads-up.",[10,5772,5773],{},"No command. No explicit request.",[10,5775,5776],{},"Anton had decided the information was worth surfacing.",[10,5778,5779],{},"Watch mode was finally live.",[10,5781,5782],{},"It was capped at a few interventions a day and silent by default. A proactive assistant can become irritating very quickly, so I wanted Anton to miss opportunities rather than manufacture them.",[10,5784,5785],{},"Seeing the first message work was satisfying.",[10,5787,5788],{},"Then I audited one ordinary interaction with Clara from beginning to end.",[10,5790,5791],{},"It contained three bugs.",[48,5793,5795],{"id":5794},"a-time-without-a-zone","A time without a zone",[10,5797,5798],{},"Every timed event Anton had added to the family calendar was two hours late.",[10,5800,5801],{},"The model produced a time as ordinary wall-clock digits: the way a person says \"three o'clock.\"",[10,5803,5804],{},"Somewhere lower in the stack, that naive time was sent to Google as though it were already UTC.",[10,5806,5807],{},"The family lived in Paris.",[10,5809,5810],{},"Every appointment therefore shifted by the local offset.",[10,5812,5813],{},"The bug was obvious once a real event was compared with a real clock and nearly invisible before then. A wrong time still looks like a valid time.",[10,5815,5816],{},"The fix was to resolve every naive wall-clock value in the family's configured time zone before sending it anywhere.",[10,5818,5819],{},"Time is not only digits.",[10,5821,5822],{},"It is digits plus a place.",[48,5824,5826],{"id":5825},"anton-forgot-what-anton-said","Anton forgot what Anton said",[10,5828,5829],{},"The second bug lived in the proactive path.",[10,5831,5832],{},"When Anton sent an unprompted message, the delivered message was not added to the conversation history.",[10,5834,5835],{},"Minutes after telling me something, Anton could deny knowing anything about it.",[10,5837,5838],{},"From its perspective, the message had never happened.",[10,5840,5841],{},"The history contained what users said and what Anton said in response. It did not contain what Anton initiated itself.",[10,5843,5844],{},"An assistant that cannot remember what it told you is only marginally less broken than one that cannot remember what you told it.",[10,5846,5847],{},"The conversation record must describe what was actually delivered, not only which request produced it.",[48,5849,5851],{"id":5850},"the-guardrail","The guardrail",[10,5853,5854],{},"The third bug was the most interesting because it came from a safeguard.",[10,5856,5857],{},"Earlier, Anton had announced the current time to the entire family with complete confidence and no evidence.",[10,5859,5860],{},"The resulting guardrail was reasonable:",[10,5862,5863],{},"Before broadcasting a factual claim, verify that the message is grounded. When it is not, suppress it and replace it with an honest failure.",[10,5865,5866],{},"This time, Anton successfully created an event for Clara.",[10,5868,5869],{},"The guardrail examined the confirmation, decided it was unsupported, removed the true message (\"Done, I added it\") and sent her \"Sorry, I couldn't find that\" instead.",[10,5871,5872],{},"The event existed.",[10,5874,5875],{},"The action succeeded.",[10,5877,5878],{},"The guardrail against lying produced the lie.",[48,5880,5882],{"id":5881},"authority-without-evidence","Authority without evidence",[10,5884,5885],{},"The gate could see which tools had been called.",[10,5887,5888],{},"It could not see their results.",[10,5890,5891],{},"It knew a calendar operation had occurred but had no evidence that the operation had succeeded. Blind to the outcome, it chose the conservative answer.",[10,5893,5894],{},"Conservative was wrong because something had happened.",[10,5896,5897],{},"The problem was not that the guardrail made a mistake.",[10,5899,5900],{},"The problem was the mismatch between its evidence and its authority.",[10,5902,5903],{},"It had enough authority to suppress Anton's answer and replace it with another one.",[10,5905,5906],{},"It did not have enough information to judge the action it was overruling.",[10,5908,5909],{},"Worse, the original success message was still written into history.",[10,5911,5912],{},"The audit trail claimed Clara received a confirmation she had never seen.",[10,5914,5915],{},"The system, its checker, and its record all agreed with one another.",[10,5917,5918],{},"Reality disagreed.",[10,5920,5921],{},"That is a particularly dangerous failure because internal consistency can look like truth.",[48,5923,5925],{"id":5924},"record-what-happened","Record what happened",[10,5927,5928],{},"The gate now reads tool results, not only tool names.",[10,5930,5931],{},"A successful write is grounded by the result proving the write occurred. An action already verified inside the agent loop is not judged again at the outbound boundary without new evidence.",[10,5933,5934],{},"Conversation history records the message that actually left the system.",[10,5936,5937],{},"All three bugs shared one theme.",[10,5939,5940],{},"The calendar path lacked the location required to interpret a time.",[10,5942,5943],{},"The conversation history lacked the outbound event required to remember an interaction.",[10,5945,5946],{},"The guardrail lacked the result required to judge an action.",[10,5948,5949],{},"Each mechanism acted with less context than its responsibility demanded.",[10,5951,5952],{},"The lesson I took into August was:",[10,5954,5955],{},"A safeguard needs evidence proportional to its authority.",[10,5957,5958],{},"A check allowed to suppress the truth must be able to see what made the statement true.",[10,5960,5961],{},"A record used for auditing must capture what users actually received.",[10,5963,5964],{},"And a guardrail should be trusted only as far as you can inspect what it saw.",[10,5966,5967],{},"Not one step further.",{"title":298,"searchDepth":299,"depth":299,"links":5969},[5970,5971,5972,5973,5974],{"id":5794,"depth":299,"text":5795},{"id":5825,"depth":299,"text":5826},{"id":5850,"depth":299,"text":5851},{"id":5881,"depth":299,"text":5882},{"id":5924,"depth":299,"text":5925},"2026-08-04",{},"\u002Fideas\u002F2026-08-04-anton-14-evidence-and-authority",{"title":5762,"description":5767},"ideas\u002F2026-08-04-anton-14-evidence-and-authority","Watch mode goes live, one audited conversation reveals three bugs, and a guardrail against lying produces the lie.",[321,322,323],"uMagptSwNxb8reMKSGeIkdhu6IOBvjFbIA8_tNyse_w",{"id":5984,"title":5985,"body":5986,"canonical_url":307,"chapter":307,"date":5975,"description":5990,"extension":310,"featured":311,"image":307,"kind":2723,"meta":6314,"navigation":314,"path":6315,"seo":6316,"series":307,"stem":6317,"summary":6318,"tags":6319,"work_slug":307,"__hash__":6322},"writing\u002Fideas\u002F2026-08-04-how-low-can-inference-go.md","How Low Can Inference Go?",{"type":7,"value":5987,"toc":6308},[5988,5991,5994,5997,6000,6003,6006,6009,6012,6015,6018,6021,6024,6027,6030,6033,6036,6039,6043,6046,6283,6287,6290,6292],[10,5989,5990],{},"On Lex Fridman's podcast, Kieran Kunhya, a longtime contributor to FFmpeg described sections of video-decoding code written directly in assembly, sometimes producing improvements measured in multiples rather than percentages. The software was written for and optimized for each variation of hardware.",[10,5992,5993],{},"The intuition is simple: bare metal is fastest.",[10,5995,5996],{},"Remove enough layers, specialize closely enough to the hardware, and eventually there is nowhere left for inefficiency to hide.",[10,5998,5999],{},"LLMs are not particularly good at writing assembly today. But that may be temporary. Low-level optimization is an attractive problem for machines: generate many candidates, test them for correctness, benchmark them, and keep only the fastest.",[10,6001,6002],{},"The same idea may apply higher up the stack.",[10,6004,6005],{},"Antirez recently built a bespoke inference engine for DeepSeek V4 Flash. By targeting one model rather than supporting every model and every configuration, he was able to outperform more general inference frameworks in important parts of the workload.",[10,6007,6008],{},"This raises a larger question.",[10,6010,6011],{},"How much of the software stack exists because the problem requires it, and how much exists because humans needed help programming computers?",[10,6013,6014],{},"Languages, frameworks, compilers and build systems are intermediate steps between intent and machine execution. Elon Musk has argued that AI will eventually compress that path, moving directly from intent toward machine code.",[10,6016,6017],{},"Perhaps he is right.",[10,6019,6020],{},"But it does not follow that every abstraction should disappear.",[10,6022,6023],{},"Some abstractions exist primarily for human readability. Others provide useful primitives for reasoning about the problem itself: memory, concurrency, data flow, isolation, interfaces and architecture.",[10,6025,6026],{},"Those may remain useful to machines as well as humans.",[10,6028,6029],{},"The future may therefore not be intent translated directly into one opaque binary. It may be a shorter stack made from different abstractions: fewer layers designed around human syntax, and more layers designed for search, verification and specialization.",[10,6031,6032],{},"Bespoke inference engines are an interesting place to watch this happen.",[10,6034,6035],{},"One model. One machine. One workload.",[10,6037,6038],{},"Then ask how low it can go.",[48,6040,6042],{"id":6041},"addendum-the-numbers","Addendum: the numbers",[10,6044,6045],{},"Single DGX Spark (GB10, 128 GB). Same harness for all models: bench.py for throughput, HumanEval pass@1 (greedy, 4096 max tokens), and a 128-prompt reasoning set (57 MMLU-Pro, 57 GPQA Diamond, 14 AIME).",[161,6047,6048,6066],{},[164,6049,6050],{},[167,6051,6052,6054,6057,6060,6063],{},[170,6053],{},[170,6055,6056],{},"Gemma 4 26B-A4B (FP8+MTP, vLLM)",[170,6058,6059],{},"Laguna S 2.1 118B (NVFP4+DFlash, vLLM)",[170,6061,6062],{},"DeepSeek V4 Flash 284B (IQ3_XXS, llama.cpp)",[170,6064,6065],{},"DeepSeek V4 Flash 284B (Q2 imatrix+DSpark, ds4)",[179,6067,6068,6085,6101,6118,6135,6152,6169,6185,6202,6219,6235,6252,6268],{},[167,6069,6070,6073,6076,6079,6082],{},[184,6071,6072],{},"Weights on disk",[184,6074,6075],{},"27 GB",[184,6077,6078],{},"72 GB",[184,6080,6081],{},"103 GB",[184,6083,6084],{},"87 GB",[167,6086,6087,6090,6093,6096,6099],{},[184,6088,6089],{},"Active params",[184,6091,6092],{},"~4B",[184,6094,6095],{},"~8.5B",[184,6097,6098],{},"~13B",[184,6100,6098],{},[167,6102,6103,6106,6109,6112,6115],{},[184,6104,6105],{},"Decode, single stream",[184,6107,6108],{},"57-92 tok\u002Fs",[184,6110,6111],{},"21-27 prose \u002F ~45 code",[184,6113,6114],{},"14.5-16.4 tok\u002Fs",[184,6116,6117],{},"19-27 tok\u002Fs",[167,6119,6120,6123,6126,6129,6132],{},[184,6121,6122],{},"Prefill",[184,6124,6125],{},"~950 tok\u002Fs",[184,6127,6128],{},"~2,700-2,800 tok\u002Fs",[184,6130,6131],{},"~420-450 tok\u002Fs",[184,6133,6134],{},"~950-1,080 tok\u002Fs",[167,6136,6137,6140,6143,6146,6149],{},[184,6138,6139],{},"TTFT, 10K prompt",[184,6141,6142],{},"~10 s",[184,6144,6145],{},"~3.6 s",[184,6147,6148],{},"~20 s",[184,6150,6151],{},"~9.5 s",[167,6153,6154,6157,6160,6163,6166],{},[184,6155,6156],{},"Tool-call turn, 10K ctx",[184,6158,6159],{},"n\u002Fa",[184,6161,6162],{},"4.6 s",[184,6164,6165],{},"24.6 s",[184,6167,6168],{},"13.0 s",[167,6170,6171,6174,6176,6179,6182],{},[184,6172,6173],{},"Concurrent out, 8x5K",[184,6175,6159],{},[184,6177,6178],{},"36.8 tok\u002Fs agg",[184,6180,6181],{},"13.0 tok\u002Fs agg",[184,6183,6184],{},"21.0 tok\u002Fs agg",[167,6186,6187,6190,6193,6196,6199],{},[184,6188,6189],{},"HumanEval pass@1",[184,6191,6192],{},"97.6%",[184,6194,6195],{},"86.6%",[184,6197,6198],{},"93.3%",[184,6200,6201],{},"90.2%",[167,6203,6204,6207,6210,6213,6216],{},[184,6205,6206],{},"Reasoning overall",[184,6208,6209],{},"74.2%",[184,6211,6212],{},"53.1%",[184,6214,6215],{},"64.1%",[184,6217,6218],{},"67.2%",[167,6220,6221,6224,6227,6230,6233],{},[184,6222,6223],{},"MMLU-Pro",[184,6225,6226],{},"78.9%",[184,6228,6229],{},"73.7%",[184,6231,6232],{},"82.5%",[184,6234,6232],{},[167,6236,6237,6240,6243,6246,6249],{},[184,6238,6239],{},"GPQA Diamond",[184,6241,6242],{},"71.9%",[184,6244,6245],{},"38.6%",[184,6247,6248],{},"52.6%",[184,6250,6251],{},"61.4%",[167,6253,6254,6257,6260,6263,6266],{},[184,6255,6256],{},"AIME",[184,6258,6259],{},"64.3%",[184,6261,6262],{},"28.6%",[184,6264,6265],{},"35.7%",[184,6267,6262],{},[167,6269,6270,6273,6276,6279,6281],{},[184,6271,6272],{},"Vision",[184,6274,6275],{},"yes",[184,6277,6278],{},"no",[184,6280,6278],{},[184,6282,6278],{},[3390,6284,6286],{"id":6285},"why-gemma-4-still-carries-the-bag","Why Gemma 4 still carries the bag",[10,6288,6289],{},"For my day-to-day it isn't close. Gemma wins the quality gates I care about (best HumanEval, best reasoning, and it's the only one that takes images), and it's the fastest at the thing you actually watch: streaming tokens. The bigger models each earn a niche: Laguna's 3x prefill makes it the snappiest on cold, fat-context agent turns, and DeepSeek's 284B brain posts the best MMLU-Pro on the box. But my real workload is warm agent loops with cached prefixes, where turns are decode-dominated (Gemma territory), and a 26B running at full FP8 with speculative decoding simply gives up less than a 284B squeezed to 2-3 bits. Until something beats it on quality without trading away the interactive speed, it keeps the production slot.",[48,6291,270],{"id":269},[272,6293,6294,6301],{},[32,6295,6296],{},[277,6297,6300],{"href":6298,"rel":6299},"https:\u002F\u002Fwww.youtube.com\u002Fwatch?v=nepKKz-MzFM",[281],"FFmpeg: The Incredible Technology Behind Video on the Internet, Lex Fridman Podcast #496",[32,6302,6303],{},[277,6304,6307],{"href":6305,"rel":6306},"https:\u002F\u002Fgithub.com\u002Fantirez\u002Fds4",[281],"antirez\u002Fds4: a bespoke DeepSeek V4 Flash inference engine for Metal, CUDA and ROCm",{"title":298,"searchDepth":299,"depth":299,"links":6309},[6310,6313],{"id":6041,"depth":299,"text":6042,"children":6311},[6312],{"id":6285,"depth":1112,"text":6286},{"id":269,"depth":299,"text":270},{},"\u002Fideas\u002F2026-08-04-how-low-can-inference-go",{"title":5985,"description":5990},"ideas\u002F2026-08-04-how-low-can-inference-go","The modern software stack lives several layers above bare metal. How much of it exists only because humans were in the loop?",[323,324,6320,6321],"inference","performance","EBLuNVznAbL4Y3Ko5doqbCcjuWWz9U7M6RI_qEii5OE",{"id":6324,"title":6325,"body":6326,"canonical_url":307,"chapter":307,"date":5975,"description":6330,"extension":310,"featured":314,"image":307,"kind":2723,"meta":6411,"navigation":314,"path":6412,"seo":6413,"series":307,"stem":6414,"summary":6415,"tags":6416,"work_slug":307,"__hash__":6418},"writing\u002Fideas\u002F2026-08-04-time-is-a-security-boundary.md","Time Is a Security Boundary",{"type":7,"value":6327,"toc":6409},[6328,6331,6334,6337,6340,6343,6346,6349,6352,6355,6358,6361,6364,6367,6370,6373,6376,6379,6382,6385,6388,6391,6394,6397,6400,6403,6406],[10,6329,6330],{},"We recently had a security incident. As much as it is uncomfortable, it is part of the life of a modern organisation.",[10,6332,6333],{},"One of the first questions was whether our own AI systems had created the opening. It was a reasonable question. Rapidly evolving practices, loosely managed API keys, exposed environment variables, vibe coding with LLMs, and dubious agent harnesses all create new opportunities to leak credentials or introduce vulnerabilities.",[10,6335,6336],{},"AI also changes the security landscape more broadly. Models can discover vulnerabilities, automate reconnaissance, and chain exploits together at a speed that was previously difficult to achieve.",[10,6338,6339],{},"We have seen the defensive side of that capability too. We use AI extensively to inspect and harden software, and we have built security harnesses that found critical bugs in code that had been running for decades.",[10,6341,6342],{},"SecOps is becoming faster and more dynamic.",[10,6344,6345],{},"But after answering the original question somewhat defensively, I started wondering how much the fundamentals had really changed.",[10,6347,6348],{},"I think that mostly, they have not.",[10,6350,6351],{},"The same rules still apply: separate concerns, grant minimal permissions, isolate systems, create observable boundaries, and assume that individual components will eventually fail.",[10,6353,6354],{},"One underrated strategy may become even more important in an AI-driven security environment: imposing speed limits.",[10,6356,6357],{},"In other words, using time as a security boundary.",[10,6359,6360],{},"Automated attacks thrive on speed. An AI system can probe a weakness, adjust its approach, escalate privileges, and move laterally before a human operator has understood the first alert.",[10,6362,6363],{},"Its advantage comes partly from chaining many small actions together without interruption. The obvious defensive response is to break the chain.",[10,6365,6366],{},"A new credential does not need immediate access to every environment.",[10,6368,6369],{},"A destructive deployment does not need to execute instantly.",[10,6371,6372],{},"A large transfer does not need to clear the moment it is requested.",[10,6374,6375],{},"A sensitive permission change can remain pending before becoming active.",[10,6377,6378],{},"These deliberate slow points create time for detection, review, and intervention.",[10,6380,6381],{},"They also increase the cost and complexity of automation. The point is not that an automated system is incapable of waiting. It is that delay creates opportunities for the surrounding environment to change. Sessions expire. Credentials rotate. State changes. Assumptions become stale. Defensive systems gain time to inspect, interrupt, or contain the workflow.",[10,6383,6384],{},"A chain that can be completed in seconds may be easy to automate. One that must remain valid and coherent across hours or days is a different problem.",[10,6386,6387],{},"Time is not a security boundary because machines cannot wait. It is a security boundary because delay gives the rest of the system time to react.",[10,6389,6390],{},"This is also a useful way to think about AI guardrails.",[10,6392,6393],{},"Guardrails are often imagined as filters that stop a model from producing certain outputs. In operational systems, they should also include permissions, rate limits, staged execution, sandboxes, approval thresholds, and intentional latency.",[10,6395,6396],{},"Their purpose is not to make failure impossible. It is to stop a fast, local failure from becoming an immediate, systemic one.",[10,6398,6399],{},"AI allows attackers to move faster than before. That makes it even more important to decide where sensible speed limits should exist. The same mechanisms also protect against human error, compromised credentials, and poorly designed automation.",[10,6401,6402],{},"Intentional latency is only useful when something is watching. If a permission change remains pending for an hour, that hour should be used: to inspect the request, compare it with normal behaviour, rotate credentials, or interrupt the workflow.",[10,6404,6405],{},"AI gives attackers greater speed, but it also gives defenders the ability to maintain continuous, agentic vigilance, the digital equivalent of a guard walking the perimeter.",[10,6407,6408],{},"The goal is not to prevent every failure. It is to ensure that a small failure cannot move through the entire system faster than we can see it.",{"title":298,"searchDepth":299,"depth":299,"links":6410},[],{},"\u002Fideas\u002F2026-08-04-time-is-a-security-boundary",{"title":6325,"description":6330},"ideas\u002F2026-08-04-time-is-a-security-boundary","Automated attacks thrive on chaining small actions without interruption. Deliberate slow points break the chain: delay gives the rest of the system time to react.",[6417,323,322,2156],"security","4KfZSdo7TPcDvnDiUX6oEoGofNAnJbHuPpYNIJU-_ag",{"id":6420,"title":6421,"body":6422,"demo":307,"description":6440,"extension":310,"featured":314,"image":307,"meta":6441,"name":6421,"navigation":314,"path":6442,"seo":6443,"stem":6444,"summary":6445,"tags":6446,"url":6450,"when":6451,"where":6452,"__hash__":6453},"work\u002Fwork\u002Fanton.md","Anton",{"type":7,"value":6423,"toc":6438},[6424,6431],[10,6425,6426,6427,6430],{},"Anton is a personal agent OS, deployed on a DGX Spark on my home network. About ten specialized domain agents (home, media, coder, research, admin, knowledge, syndic, dev, quality) coordinate via typed delegates. Skills run in a sandboxed Deno runtime, hot-reloadable. Layered memory (context, history, facts), policy engine evaluating every action, thread-aware interruption with cancel cascade, nightly self-improvement loop driven by issues, evaluation infrastructure under ",[398,6428,6429],{},"packages\u002Fagent-quality",". Local LLM inference via vLLM behind a LiteLLM gateway over Tailscale.",[10,6432,6433,6434,6437],{},"Anton is a decade-later realization of ",[346,6435,6436],{},"Bots for Humanity",", a personal-agent \u002F digital-twin concept I worked on as my MBA capstone in 2015 and presented to contacts at Facebook. Design philosophy: ingest cheap, process lazy, enrich on demand. Explore agentic, build deterministic.",{"title":298,"searchDepth":299,"depth":299,"links":6439},[],"Anton is a personal agent OS, deployed on a DGX Spark on my home network. About ten specialized domain agents (home, media, coder, research, admin, knowledge, syndic, dev, quality) coordinate via typed delegates. Skills run in a sandboxed Deno runtime, hot-reloadable. Layered memory (context, history, facts), policy engine evaluating every action, thread-aware interruption with cancel cascade, nightly self-improvement loop driven by issues, evaluation infrastructure under packages\u002Fagent-quality. Local LLM inference via vLLM behind a LiteLLM gateway over Tailscale.",{},"\u002Fwork\u002Fanton",{"description":6440},"work\u002Fanton","Family logistics scatter across apps, inboxes, and memory; Anton is a personal agent OS that runs them in production from hardware at home.",[322,6447,6448,323,6449,1392],"typescript","deno","self-hosted","https:\u002F\u002Fgithub.com\u002Flucbocahut\u002Fanton","2024-12-01","Paris","OXLMu-spymcpGsgsicDfI-hs4bFBLFXMEfiM168dYnY",1787174859950]