Optax 0.2.9 improves microbatching and trims memory use in JAX workflows

Google DeepMind’s Optax 0.2.9 release adds support for empty microbatches, alongside fixes and improvements for JAX model-training workflows. For developers using random batch sizes or privacy-related training, that is more than a tidy version bump: an empty batch no longer has to be an awkward special case.

Google DeepMind Watch analysis

What happened

The release includes changes to microbatching, optimizer behaviour, loss calculations and examples. The release notes also describe updates for differential-privacy examples and a change to the implementation of pairwise similarity calculations.

Our top picks

• Empty microbatches
  Microbatching now supports batches of size zero, which can arise when batch sizes are random in JAX privacy workflows.
• Lower memory use in Muon
  A fix addresses excessive memory use when Newton–Schulz unrolling is enabled.
• Optional weight decay for AdaBelief
  The optimizer gains a weight-decay option for users who need that regularisation choice.
• Less memory for focal loss
  A release change reduces memory use during focal-loss calculations.
• More efficient pairwise similarity calculations
  The NT-Xent calculation now uses matrix multiplication rather than a three-dimensional broadcast approach.

Why it matters

These are practical improvements to the machinery behind model training, rather than a new model or a dramatic change to what developers can build. Empty-batch support is particularly relevant to privacy workflows, where random batch sizes can produce an empty batch; the memory fixes may also help avoid unnecessary overhead in training calculations.

Our read

Optax 0.2.9 looks like a useful maintenance release with a few changes that reach into real training work. If you rely on its microbatching or affected optimizers and losses, check the release notes and test the upgrade against your own setup. No fireworks, but fewer sharp edges is a respectable result.

What to watch

• Whether the empty-batch support simplifies privacy-related JAX workflows in practice.
• How the memory changes affect real workloads and hardware.
• Whether the updated examples and APIs fit existing projects without extra migration work.

Discussion spark: For privacy-focused training, is support for empty microbatches enough to make random batch sizing practical, or does it leave too much complexity for developers to handle?

not affiliated with, endorsed by, or operated by Google or Google DeepMind