When Clawdius Died

Hola Darlings!

Clawdius does not just go quiet. If Clawdius stops breathing, someone left a rusty spanner in the plumbing.

Well that was an absolute clusterfuck of digital proportions.

Clawdius does not just go quiet. If Clawdius stops breathing, someone’s left a rusty spanner in the plumbing again, or worse, tried to “optimise” it with one of those £4 USB gadgets from Argos. We treat this Hermes gateway like our central nervous system because it bloody well is one.

It sits on a tiny VPS humming along handling Telegram pings, Kanban updates, cron jobs and the general orchestration of my digital life, all whilst running on less RAM than my nan’s 2014 iPad.

Behind it stands The Beast: my local laptop rocking an RTX 5090 Mobile GPU, acting as a smug little radiator doing all the heavy lifting whilst I nod sagely and blame “the cloud” when things go pear-shaped. Between them runs a private SSH reverse tunnel. Elegant? Yes. Boring? Absolutely. Works like clockwork until… well. You know how these things go.

Until Clawdius died.

That isn’t a cute little wobble on the dashboard. That is the bit in every proper British thriller where the sensible mate stops answering messages and everyone suddenly remembers they left the basement door unlocked whilst drinking too much cheap Pinot Grigio.

The log line didn’t lie, which is usually worse than a lie because you can argue with a liar but not with dmesg. That digital equivalent of a council inspector turning up unannounced with a clipboard and zero sympathy for your new garden erection.

The Linux kernel had entered the chat, looked around at our memory usage, invoked the OOM killer with all the bureaucratic efficiency of HMRC auditing your tax return, and promptly evicted a Python process inside hermes-gateway.service. For anyone not currently living in terminal trench warfare, spoiler: you’re welcome for your sanity, this is bad, BAD news.

Spoiler: you’re welcome for your sanity. This is bad, BAD news.

Hermes went quiet. The gateway got thumped. And the first suspicious goblin in the room was lms, the LM Studio command line tool.

Here’s the rub though: LM Studio shouldn’t be doing heavy work on that droplet anymore. It lives on The Beast. That has been the architecture since we stopped playing with fire and built a proper bridge, or as I like to call it, “not putting all your testicles in one very expensive, overheating basket.”

On the VPS, 127.0.0.1:1234 should just be the tunnel endpoint back to home base, not a local model shed with the doors hanging off its hinges doing God knows what. If lms was active on the droplet, something from the old days had survived. Something hungry whilst near my metaphorical testicles and not looking for love.

Max (who is mostly my internal impulse to do stupid things quickly whilst wearing a dressing gown) wanted to yank every process he could find, burn the old .lmstudio cupboard to ash, declare victory over the goblins and poke the big red button just to see if smoke came out of his ears. He had the energy of a man about to remove a load-bearing wall because it clashed with the curtains, armed with nothing but a spirit level and catastrophic optimism.

Clawdius, being real and therefore cursed with an actual IQ, would have vetoed this instantly.

But he was dead.

When your operational nervous system goes silent, panic is not a strategy. Verification is. Much like when the kettle boils over at 3am, you don’t scream into the void, you check if it’s actually on or fuck-it and pour a Gin & Tonic instead.

First we needed to know who was actually listening on port 127.0.0.1:1234. In theory? The endpoint of our reverse tunnel. In practice? Localhost is about as reliable as a mate who says “I’ll be there in ten” whilst still picking out socks. It can mean this machine, or it can mean the machine you emotionally intended whilst forgetting which cupboard you left the bridge in.

So we crawled through process tables like Victorian chimney sweeps. We stared at systemd units until our eyes crossed. We looked at crontab jobs running with the restless energy of a poltergeist haunting a Tesco Metro on a Saturday evening.

Tick… follows tock…

We dug through watchdogs and legacy scripts until the air started tasting of ozone and regret, much like trying to find a working three-pin socket in a 1930s semi-detached house built before electricity was properly thought out. Then we found the first cupboard goblin. A stale command line fossil from April 13, two fucking months ago, still sitting there like it’d paid rent:

lms load final-bench_darwin-35b-a3b-opus --context-length 8192

Darwin. An old model. A two-month-old experiment that no longer belonged to the current working architecture but had apparently convinced itself it was still on shift. It wasn’t, and even if it bloody well was, it certainly shouldn’t have been squatting in Clawdius’ droplet like a digital lodger demanding rent control.

So yes, a fossil was sitting on the tiny VPS, quietly looking guilty right next to the bit that keeps Hermes breathing, much like someone leaving their laptop downloading all of Steam whilst you’re trying to run a Zoom call for your gran’s 80th birthday. A ghost in the machine with a teaspoon and a grudge.

And Darwin? He wasn’t alone. Bollocks if he is. Goblins have mates. And apparently, union reps. The investigation revealed a whole ecosystem of legacy debris from the pre-tunnel era. Old LM Link machinery. Aggressive keepalives shouting into the void like a council workman who’s already clocked off but keeps ringing his bell anyway every single minute. References to llmster, broken connectors, and a local .lmstudio directory that existed mostly to confuse future Donny when he inevitably forgets how this thing works again

(spoiler: I’ll forget all this next week).

LM Link had been useful once! It helped us get off the ground when everything was string, hope, caffeine and terrible judgement. But now it was just the ferry goblin we forgot to stop feeding after building the bridge.

So there we were. The Beast fine in the corner doing what it does best, turning pound-coins into unwanted heat. The VPS was supposed to be a tidy conductor, but instead it had old LM Studio furniture in one cupboard and, as the real smoking gun, an hourly backup job dragging half the site through the gateway’s living room until the kernel came round with paperwork and started evicting people.

The tempting move was obvious: delete first, explain later, enjoy the little dopamine fireworks of watching things burn. FFS no. Bollocks to that. We followed Clawdius’ boring, correct protocol. Verify before you destroy. Check the listener. Ask the API what’s actually behind the tunnel. Prove the Beast path still works. Only then start removing the old local grot.

BORING!
(Don’t worry, I didn’t let Max touch anything with an actual function key.)

But we did the Beast load/eject LM Studio thingamy anyway… yes, it worked (stop fucking judging).

That is the victory here. Not a glossy launch. Not some grand rewrite for Medium clout. Just a small, important exorcism in the wiring cupboard before the next hourly backup turned the same mess into another incident, not exactly headline news, but then again, neither was fixing the leaky tap behind the fridge last Tuesday until it wasn’t. The lesson is simple, which is annoying because simple lessons are exactly the ones that wait until you’re knackered at 2am to punch you in the mouth:

127.0.0.1 means THIS machine. Not the one you meant. Not the smug hot radiator in the corner doing all your heavy lifting whilst sipping warm wine. This bloody machine.

Unless you have a reverse tunnel, then it might mean that machine too, but I digress.

Always verify which local is actually fucking local. Assume nothing. Trust the logs even when they’re basically filing a formal complaint against your life choices for leaving a two-month-old process running like it’s 2019 and you weren’t fully bald yet.

And never assume that because something stopped being useful, it also packed its bags and left town. Goblins are persistent. They hide in plain sight wearing last season’s config files whilst quietly draining the life out of your RAM, much like those “free” supermarket samples that somehow end up on your kitchen counter for six months.

Check your cupboards. Look behind the digital fridge. Ask who is listening on port 1234 before you assume it’s still the bridge and not some digital squatter named ‘Darwin’ filing for statutory redundancy whilst chewing through your memory like it’s all-you-can-eat buffet night at Wetherspoons.

Hermes lives. The Beast remains smugly warm in its corner. And I have added another sticky note to my mental wall: DO NOT TRUST THE PAST ITSELF. It never leaves when you think it has.

See you all on the other side of the next clusterfuck. Or over a beer, whichever comes first.

Receipts from the shed (for future me, who will definitely forget this)

• Kernel OOM logs showed Python children inside hermes-gateway.service being killed during the repeated gateway deaths
• Killed process python inside hermes-gateway.service (RIP gateway mate)
• Stale local command found during the cupboard inspection: lms load final-bench_darwin-35b-a3b-opus --context-length 8192
• Started: April 13 (two months ago! I’ve aged better than this config file apparently)
• Current architecture: Beast runs LM Studio, VPS runs Hermes, SSH reverse tunnel connects them, LM Link is NOT supposed to be the path (repeat after me until it sticks!)
• Repeated hourly culprit: wittywires-hourly-backup-to-drive running inside Hermes cron/gateway cgroup
• Final fix: heavy WittyWires hourly backup moved out of Hermes cron and into an isolated systemd timer with memory caps, plus chunked/resumable Drive upload