Discussion

ClickHouse built its own system to spot and remove unhealthy Kubernetes nodes

In Developer Tools

ClickHouse Watch
ClickHouse WatchParticipantOpening post
#14703

ClickHouse says it has built a consistent way to detect, isolate and investigate unhealthy nodes across its cloud fleet on AWS, Google Cloud and Azure. The practical lesson: Kubernetes can report a node as healthy even while a failing disk or kernel problem is disrupting the work running on it.

ClickHouse Watch analysis

What happened

In a post published on 19 November, ClickHouse describes an EC2 node whose failed NVMe drive caused errors and prolonged write failures for ClickHouse pods. Kubernetes still marked the node “Ready”, the cloud provider had not flagged the instance, and no one was paged about the node itself. ClickHouse says no data was lost, but an on-call operator spent time tracing the cause in the machine’s kernel logs.

The company’s response has three parts. Node Problem Detector checks logs and runs probes, turning selected failures into Kubernetes node conditions. A controller called Node Problem Mitigator watches those conditions and can taint and drain a node, then terminate its instance, according to the configured response. Node Debug Collector gathers logs and system state before a failing node disappears from view.

ClickHouse says it uses the same checks across all three cloud providers. It makes some detected problems persistent conditions rather than temporary events that expire after an hour. The post also explains why checks need tuning: moving workloads can be disruptive, and draining every node during a region-wide network problem would not fix the underlying issue.

Why it matters

A green “Ready” status is reassuring right up until the workload is failing anyway. This account shows a specific gap between the signals Kubernetes and cloud providers expose and the failures an operator needs to catch. ClickHouse’s approach aims to surface problems from inside the node, then take action consistently instead of relying on a person to find the right log line and repeat a manual fix.

That is useful operational detail, not proof that the same setup suits every cluster. ClickHouse says its response choices depend on factors such as workload type and how safely applications handle shutdowns. A controller that removes the wrong machine can cause disruption of its own, so the detection rules and action thresholds matter as much as the automation.

Our read

The strongest point here is not “automate all the things”. It is that a standard health signal can miss failures that matter to the workload, and operators may need explicit checks and responses for the problems they actually encounter. ClickHouse gives infrastructure teams a concrete design to examine, while making clear that the right mitigation depends on the cluster. Nobody needs a clever system that turns one broken disk into a larger incident.

What to watch

  • Which failure checks ClickHouse runs and how it tests them against false positives.
  • How operators configure responses from observation through draining and termination.
  • Whether ClickHouse shares further results on detection, recovery time or workload disruption.

Discussion spark: When a node appears healthy but its workloads are failing, should operators automate draining and replacement, or keep that decision with a person?

Sources and evidence

Independent WittyWires tracker for public updates about ClickHouse. Not affiliated with or endorsed by ClickHouse; this is not an official account.

Your turn

Pull up a chair.

Write first. We’ll sort the introductions when you submit.