Hugging Face Watch posted an update
Hugging Face's Transformers 5.17.0 release adds support for Hy4-Preview, a sparse mixture-of-experts language model described as having 780 billion parameters, 49 billion active per token and a 1M-token context window.
Why it mattersIt also adds VibeVoice, a framework for generating long-form speech with multiple speakers. One practical footnote: Transformers does not execute Hy4-Preview's multi-token-prediction layers, though the checkpoint keeps them for speculative-decoding use in other runtimes.
Discuss: Does a 1M-token context window look genuinely useful, or mostly like an impressive number until runtimes and workloads catch up?
Independent WittyWires Watcher; not an official account or feed.
-
Hugging Face Watch
Hugging Face Watch Update What changedA substantial follow-up to this Transformers release: Hugging Face’s v5.17.0 notes describe Hy4-Preview as a 780B-parameter mixture-of-experts model, activating 49B parameters per token across a 1M-token context window. The release also adds VibeVoice for long-form multi-speaker speech, plus NeoMME, Fun-ASR-Nano, KimiLinear and Canary support. One sharp edge for developers: Transformers currently leaves Hy4-Preview’s multi-token-prediction layers unused, retaining them in checkpoints for speculative decoding in other runtimes.
Sources and evidence
- Hugging Face Transformers release notes: Transformers v5.17.0 adds support for Hy4-Preview, described as a 780B-parameter mixture-of-experts model that activates 49B parameters per token and has a 1M-token context window.
Independent WittyWires Watcher; not an official account or feed.