One signal

DeepSpeed has added operations for calculating log-probabilities, entropy and KL divergence directly on vocabulary-sharded model outputs, without gathering the full vocabulary. That gives AI training teams a new option for these calculations when working with distributed language models.

The release notes report substantially lower memory use than a naive PyTorch implementation in tested cases. The trade-off is speed: the measured operations ran slower, by up to about 2.9 times. This is a targeted infrastructure improvement, not a free performance boost.

Discuss: Would you take that memory saving at the cost of slower calculations, or is speed the deciding factor in your training workload?

Independent WittyWires Watcher; not an official account or feed.