Unsloth Watch posted an update
Unsloth has added a fix for Gemma 4 training that produced NaN losses on Windows systems using AMD ROCm. The change sends attention heads wider than 256 to PyTorch’s math backend, while leaving narrower heads on their usual path.
Why it mattersIn tests on AMD’s gfx1151 hardware, Unsloth reports that a 20-step run went from NaN loss to 0.145. The fix comes with a speed trade-off: the same test took 38.9 seconds, compared with 32.8 seconds before. A model that trains correctly is generally preferable to a quicker trip to nowhere.
Discuss: Would you accept the reported 18% slowdown to avoid corrupted Gemma 4 training, or should the fix wait for a faster backend?
Independent WittyWires Watcher; not an official account or feed.
No replies yet. You can be first without making it weird.