Say you ask an agent to make a mushroom texture. It takes the job, starts a tool call, and then the process just disappears before the result comes back.
You reopen the chat. Your request is still sitting there. Great, that tells you what you asked for. It doesn’t tell you whether the texture got made, whether asking again would make two, or whether anything happens at all if you close the tab.
Those are three different problems. Our first design mixed them up.
(The agent art tools themselves are in the game art note.)
Recovery belongs next to the work
Our first prototype had a separate service watching each agent over a WebSocket. It pinged, listened for silence, and tried to get stuck work moving again. We called it WatchdogDO, the sidecar. The trouble with watching from outside is that you have to rebuild, from the outside, stuff the agent already knows. Every edge case turned into one more reconnect rule.
So we moved recovery inside the agent itself, right next to the run state it needs to look at. Our job engine today, Watchdog, keeps its jobs right beside the owner’s other records. It asks the host, “is this job still running?” A real “no” lets it retry, up to the job’s limit, or mark the job interrupted. Silence isn’t a “no”. (More on that in a clock is not a verdict.)
Back to the texture. Now the restarted agent has a job it can look at. It still has to check what the tool actually did, though. The tool might have saved the image right before its reply got lost. Marking a job done can’t make an outside action happen exactly once. The tool itself needs a way to spot the same request coming in again and hand back what it already made.
Who wakes Shiitake?
This is the bit we went back to the source code for. Getting rid of the sidecar didn’t get rid of the alarm.
The old agent had an alarm clock hooked into Cloudflare’s durable alarms (alarms that survive restarts). Background work and mail each said when they next needed a turn, alongside any scheduled jobs. The clock picked the earliest one, and when it went off, the agent called back into whoever owned the work so they could check on it. None of that needed your browser to be open.
Today our Cloudflare adapter saves its database changes and its alarm together, in one go. Watchdog still decides what needs attention. The host just makes sure it gets another turn.
We tested that locally: kill the process, then send it zero requests. The alarm woke a fresh copy and the queued work ran. Work that had been running in the process we killed got marked interrupted.
So the wake-up survived the restart with nobody nudging it. Proving the same on the hosted platform is its own piece of work. So is safely retrying a tool that might already have made your texture before its reply vanished.