At the end of August, right before we started rebuilding Fungi in a fresh repository, we went back over five months of prototype code and wrote down the design mistakes we least wanted to make again. That write-up is still in our wiki, filed as history. Its last paragraph is the most useful thing I’ve read about our own code:
Botanical rarely failed because it lacked an abstraction. It failed when an abstraction became a second owner, lived too far from its state, or gave the first-party product a private shortcut.
This note is about that one sentence, because I keep finding it true in places the write-up never even looked.
(The watcher story below gets told from the other side in the restart note.)
The shape of it
Every fact in a system needs one owner: the one place that gets to say what’s true. Is this run alive? Where does this file live? Can this App read a Team’s sessions? Which branch is the real code?
A second owner almost never shows up on purpose. It shows up as a helper. A watcher sitting next to the thing it watches. A cache that’s easier to read than the real store. A special path for our own product, because hey, we trust it. Each one answers the same question as the owner, from a slightly different spot, and for a while the answers match. Then they don’t, and every problem after that is basically the two of them arguing.
Three times we found it
The watcher. Long agent runs need something to keep them going and pick them back up. Our first design put that in a separate Durable Object (a small program with its own database) that watched each run from the outside. It hooked in like a viewer, pinged to keep the run awake, and nudged it when it stalled. It worked, but every edge case meant another patch, and it grew to 1,107 lines. Its write-up nails why: “1,107 lines compensating for being outside the module that owns the run state.” On August 6 we deleted it and moved that job into the agent that actually owns the run. A week later the job list moved into that owner’s own SQLite database. Today our job engine’s README says it “owns each bounded job’s owner-local SQLite row”. The word doing the work there is local.
The workspaces. While we were exploring, the prototype grew four kinds of workspace: a sandbox, a Git repo, a storage workspace and a Computer. Each was a fair experiment, and each had its own idea of where a file lived and how it got saved. On August 16 Levi decided the Computer was the only kind that survives; the other three got deleted, not migrated. That still holds. A Computer today is one durable Workspace that holds the files, and anything that runs code against them plugs into that Workspace as a backend. Later we asked whether Cloudflare’s Sandbox SDK should be one of those backends. Same answer, same reason: a Sandbox keeps its own files, and they disappear when it sleeps. It would be a second owner of the files, with a worse memory.
Our own Apps. Before launch we went hunting for anywhere our own Apps got treatment a stranger’s App couldn’t. We found a handful of little conveniences: a shell that opened one App by its hard-coded id, an icon only our Apps could wear, and file handlers only a built-in default could own. None of them was a big deal on its own. But each one was our product holding a permission outside the grant system, which meant the grant system wasn’t the only owner of “who’s allowed to do what” anymore. Levi’s call on September 26 was blunt: “everything is yours to change / no privileged APIs”. Now a first-party App gets nothing a stranger’s App can’t get through the same front door.
The fix is the same every time
None of these got better with a smarter helper. Every one got better by deleting the second answer and moving the work back next to the state it needs.
You can overdo it the other way too, and the same write-up says so. Splitting a 1,015-line object into a 1,294-line “deep module” didn’t remove an owner. It just moved the complexity somewhere else. One owner doesn’t mean one giant file. It means one place that decides, with as much machinery underneath as the decision actually needs.
Why I keep coming back to it
Most of our architecture rules are this one in a different outfit. A Vishnu world is one simulation, and when it fills up a new world opens, instead of two servers sharing one. Our launch plan only counts something as done when its commit is on the integration branch, because a status doc would otherwise be a second owner of “what shipped”. Even the list of things we kept on Cloudflare is code, in one Worker, so that decision lives in one place.
That write-up was meant to steer the rebuild. It turned out to be the best design review we have. Now when something acts in a way that looks new, I start by asking which two things think they’re in charge.