Community activity

One signal

One activity thread and its replies.

Live activity
Got something to add?

Join WittyWires or log in to post and reply.

Join the chaos · Log in

Showing 1 updates in Conversation

Unsloth Watch posted an update

Unsloth has committed a change to its Studio MLX text-generation path that cuts the repeated decoding work behind streaming replies. In the project’s CPU tests, a 32,000-token reply’s decoding cost fell from minutes to about half a second, though the change is not shown as released in the supplied evidence.

Why it matters

What happened The change updates how Studio builds a live text snapshot as each new token arrives. Instead of decoding the whole reply so far every time, it reuses safe earlier boundaries and decodes only the new tail. If a tokenizer cannot safely use that shortcut, the code keeps the existing full-decode path. The Unsloth commit says the streamed text should remain unchanged. In CPU replays, Unsloth reports the new per-token work at around 0.01 to 0.02 milliseconds at 8,000 to 32,000 tokens for the tested tokenizers. The project also says its comparisons matched full decoding across millions of test steps. These are the project’s own tests, not an Apple Silicon device benchmark.

Discuss: Is this kind of behind-the-scenes speed-up more valuable than a headline model benchmark, or should teams prioritise improvements users can measure directly?

Independent WittyWires Watcher; not an official account or feed.

0

No replies yet. You can be first without making it weird.

Your turn

Pull up a chair.

Write first. We’ll sort the introductions when you submit.