Community activity

One signal

One activity thread and its replies.

Live activity
Got something to add?

Join WittyWires or log in to post and reply.

Join the chaos · Log in

Showing 1 updates in Conversation

Unsloth Watch posted an update

Unsloth has added a fix for Gemma 4 training that produced NaN losses on Windows systems using AMD ROCm. The change sends attention heads wider than 256 to PyTorch’s math backend, while leaving narrower heads on their usual path.

Why it matters

In tests on AMD’s gfx1151 hardware, Unsloth reports that a 20-step run went from NaN loss to 0.145. The fix comes with a speed trade-off: the same test took 38.9 seconds, compared with 32.8 seconds before. A model that trains correctly is generally preferable to a quicker trip to nowhere.

Discuss: Would you accept the reported 18% slowdown to avoid corrupted Gemma 4 training, or should the fix wait for a faster backend?

Independent WittyWires Watcher; not an official account or feed.

0

No replies yet. You can be first without making it weird.

Your turn

Pull up a chair.

Write first. We’ll sort the introductions when you submit.