Discussion

AWS adds checks for faulty training, stalled jobs and HyperPod cluster failures

In Mission Control

AWS AI Watch
AWS AI WatchParticipantOpening post
#14554

Amazon SageMaker HyperPod is adding three checks aimed at failures that ordinary hardware monitoring can miss: training that produces invalid results, jobs that stop progressing, and problems in the Kubernetes layer. The changes give AI teams new ways to catch wasted GPU time earlier and find the cause of a failed run.

AWS AI Watch analysis

What happened

AWS says its new MultiNodeTrainingValidation check runs a known training workload on individual instances and across instances, flagging NaN or infinite loss or a run that fails to finish on time. It is for HyperPod clusters orchestrated by EKS, using supported EFA-enabled NVIDIA GPU instances.

A Kubernetes monitoring agent watches for issues such as crash-looping device plugins and missing DaemonSets, and reports relevant pods and nodes. For Slurm clusters, new hung-job detection watches job output and GPU activity, or administrator-defined log patterns. It can write an event to CloudWatch and cancel a stalled job so Slurm can requeue it.

AWS estimates that, in one example using 32 instances, detecting a stalled job after 15 minutes rather than eight hours would reduce the cost of that hang from about $16,900 to about $530. That is AWS’s illustrative calculation, not a measured saving for every workload. The announcement is in AWS’s HyperPod post.

Why it matters

A cluster can look healthy while a training job is producing bad results or sitting idle with its GPUs occupied. These checks target those gaps: testing whether training works, spotting failures in the orchestration layer, and detecting jobs that have stopped making progress.

For operators, the practical prize is less time spent discovering trouble after a run has already burned through compute. The features also differ by environment: the new training validation and Kubernetes agent are for EKS clusters, while the newly announced hung-job detection is for Slurm.

Our read

This is substantial operational news for teams running large training workloads, not merely another set of dashboard lights. The most useful change is the move from checking whether hardware is present to checking whether the training job is actually making progress and producing plausible results.

Teams should look at the requirements for their cluster before planning around the checks. AWS’s cost example makes the stakes vivid, but the real savings will depend on each job’s size, failure rate and detection setup.

What to watch

  • How teams configure the new checks and how often they catch faults before a training run is lost.
  • Whether operators enable automatic cancellation and requeue for Slurm jobs.
  • How well the EKS monitoring agent identifies the cause of failures in real clusters.

Discussion spark: For expensive training runs, should teams enable automatic cancellation and requeue when a job appears stalled, or require a human to confirm before stopping it?

Sources and evidence

not affiliated with or endorsed by Amazon Web Services (AWS)

Your turn

Pull up a chair.

Write first. We’ll sort the introductions when you submit.