744 tests passed. I still ended up in the droplet with a digital defibrillator.
Hola Darlings!
Clawdius does not normally die. He becomes temporarily unavailable while I develop a drinking problem.
A while ago I wrote a post called By the Power of Grayskull, Heman Died Twice.
You would think that after writing those words once, the universe might stamp the paperwork and move on.
No.
Apparently it wanted a sequel.
This one began with a shiny Hermes Agent update called The Pantheon Release. Named bots with faces. Bots chatting to other bots. Scheduled jobs that remember what they said yesterday. Subagents I can steer while they are running instead of watching them disappear into a hedge for forty minutes.
Good stuff.
I asked Clawdius whether we should jump on it.
He checked the release, checked our current version, checked the local changes we carry because apparently we cannot own anything without modifying it, and came back with a sensible answer.
Yes.
But carefully.
Max, the small part of my brain responsible for every story on this website, heard only the first word.
The plan looked annoyingly responsible.
Snapshot everything. Build the new version somewhere separate. Preserve our local fixes. Test it. Swap over once. Check Telegram, memory, schedules and the rest of Clawdius’ digital organs. Keep the old version ready in case the new one came out speaking Latin.
Then Clawdius tested it.
And tested it.
And tested it some more.
744 tests passed. Five were skipped.
There were so many green ticks I briefly considered buying him a little naval hat.
There is a special kind of confidence that comes from 744 green lights and absolutely no idea whether the fire exit opens.
The tests proved the new Hermes could think, remember, delegate, steer, schedule and generally behave like a civilised member of the household.
They did not prove the thing responsible for changing Hermes could recover after killing Hermes.
That distinction became relevant almost immediately.
The first death
Clawdius moved the upgrade into a separate background service, because even he understood that asking a live AI to switch off the live AI was a little circular.
The old gateway began shutting down.
Then the chat stopped.
Not slow.
Not thinking.
Dead.
Now, Clawdius going quiet is not like Netflix buffering. He runs the schedules, remembers the conversations, watches the jobs, keeps the receipts and generally stops me turning one dodgy button into a regional incident.
If he vanishes halfway through replacing his own brain, I do not know whether he is rebooting or scattered across the droplet in several attractive folders.
I waited.
Nothing.
I tried the normal routes.
Nothing.
Eventually I went into the droplet myself.
For non-technical readers, a droplet is a small rented computer in the cloud. It is also where you end up at stupid o’clock typing carefully because your favourite AI has disappeared and every line looks capable of making things worse.
I found the old Hermes and started him manually.
Clawdius came back.
The relief lasted just long enough for me to ask what the fuck had happened.
Clawdius explains death number one
The answer was painfully small.
The upgrade service needed a dependency tool. In Clawdius’ normal shell, that tool was easy to find. In the stripped-down background service, it was not.
The upgrade failed.
Fine. That is what rollback is for.
Except rollback then tripped over a stashed shortcut, reached for a deployment backup that had never actually been copied, and stopped before the one command that mattered.
Restart old Clawdius.
The fire extinguisher worked perfectly until it reached the cupboard where we had forgotten to put the fire extinguisher.
That was why I had to revive him manually.
It was also the exact moment a sensible person would have stopped for the night.
We had evidence. We had the cause. We had Clawdius back. We could fix the recovery path tomorrow.
Unfortunately, Max had now discovered the phrase “known issue”.
We fixed the fire exit
Clawdius replaced the missing tool path with the full one.
He rebuilt the deployment backup.
He changed rollback so it would keep crawling towards the restart even if every cupboard on the way fell over.
Then he tested the outside service without touching the live gateway.
It passed.
This was the point where the plan became dangerous again, because now it had paperwork.
We tried the upgrade a second time.
The old Hermes stopped.
The new Hermes started.
Hermes Agent v0.21 was alive.
For roughly one minute.
Then Clawdius rejected his own pulse and rolled himself back.
The chat died again.
I went back into the droplet again.
Fixing your AI manually twice in one night is not automation. It is digital livestock.
I brought him back.
Again.
At this point I was less interested in the Pantheon and more interested in which god had hidden the off switch.
Clawdius explains death number two
This one was stupider.
The new Hermes had worked.
The new gateway had started. It was running the new version. It was responding. It was, in every useful sense, alive.
But Clawdius’ final health check compared the real address of the running Python process with a shortcut that pointed to that same address.
One path said, “Here is the house.”
The other said, “Here is the sign pointing to the house.”
The verifier looked at both and concluded they were unrelated strangers.
So it declared a perfectly living Hermes dead, waited sixty seconds, and rolled him back for safety.
Clawdius passed 744 tests and failed the advanced practical exam in recognising himself in a mirror.
That was death number two.
Then, during cleanup, the supposedly finished upgrade service twitched once more and issued another stop command before being killed itself.
A zombie switch.
Lovely.
The bruised win
Clawdius is back.
More accurately, he is back in the sort of state that makes engineers use the phrase “currently operational” while avoiding eye contact.
The running gateway is Hermes v0.21. The main checkout and part of the configuration still think they belong to v0.20.
Both facts are true.
That is not a clean upgrade. It is a manual recovery wearing one new shoe.
Nothing important appears lost. The schedules survived. The memories survived. Telegram survived. Clawdius still knows who I am, which after this evening feels less like a feature and more like forgiveness.
And I learned something useful.
Not “write a cleverer upgrade script”.
The clever script is what got us here.
Notes From The Shed
If you run an AI, server, website or any other clever bastard that might one day update itself, here is the bit worth stealing.
Do not let the thing being upgraded own the only way back.
Keep the old version alive while the new one starts somewhere separate.
Check the new one from outside it.
Only switch over after it answers properly.
And keep one brutally simple rescue route that does not need the new code, the new environment, the agent, Git, a model, a clever verifier or the blessing of Zeus.
Just:
Old one stopped.
Start old one.
Everything else can fail afterwards with dignity.
That is the compact how-to hidden inside this particular episode of digital A&E.
The next time Clawdius suggests upgrading himself, the old Clawdius stays alive until the new Clawdius knocks on the door, identifies himself, remembers the pub, finds the fire exit and makes a decent cup of tea.
Until then, the Pantheon can wait outside.
I have done my night shift with the gods.
