Transformers 5.17 adds a 780B model, long-context support and a crowded new-model roster

Hugging Face has released Transformers v5.17.0, adding support for a 780B-parameter mixture-of-experts model with a 1M-token context window, plus new multimodal, speech and attention implementations. The practical story is breadth: more frontier-shaped models are landing in one of the ecosystem’s main compatibility layers, alongside fixes that should make inference less fussy.

Hugging Face Watch analysis

What happened

The headline addition is Hy4-Preview. Hugging Face describes it as a 780B-parameter MoE model that activates 49B parameters per token, routes each token to eight of 256 routed experts, and uses a 1M-token context window. Its implementation combines Multi-head Latent Attention, DeepSeek Sparse Attention, gated attention sinks and Independent Hyper-Connections.

The same release adds VibeVoice for long-form, multi-speaker speech synthesis; NeoMME and NeoMME-Retriever for multilingual multimodal encoding and visual-document retrieval; Fun-ASR-Nano for multilingual speech recognition; Kimi Linear; and NVIDIA’s Canary-1B-v2 for speech recognition and translation.

Our top picks

• Hy4-Preview support
  A 780B MoE model with 49B active parameters per token and a 1M-token context window joins the Transformers roster.
• VibeVoice
  Long-form, multi-speaker speech synthesis arrives as a model framework aimed at podcasts and audiobooks.
• NeoMME-Retriever
  Text and document screenshots can produce both late-interaction and dense embeddings for retrieval.
• Fun-ASR-Nano
  An 800M speech model targets Chinese, English and Japanese, including seven Chinese dialects and 26 regional accents.
• Kimi Linear
  A hybrid attention architecture brings Kimi Delta Attention and periodic full-attention layers into the library.
• Canary-1B-v2
  One model supports transcription or translation, selected through its processor configuration.
• Generation overhead
  A change avoids unnecessary accelerator synchronisation on every decode step.

Why it matters

Transformers is becoming less like a catalogue and more like a shared loading dock for the current AI model ecosystem. Developers can reach more model families through familiar APIs, while the lower-level fixes target the unglamorous points where real deployments tend to squeak.

There is one important implementation footnote for Hy4-Preview: Transformers does not execute its multi-token-prediction layers at load time. The checkpoint keeps those weights for other runtimes, including speculative-decoding workflows, but this implementation does not use them.

Our read

This is a substantial ecosystem release, not just a version-number shuffle. The useful move for developers is to inspect the specific model integration and migration notes before upgrading, then test the models and hardware paths that matter to their workload. The release expands the menu considerably, but a menu is not dinner until the runtime actually serves it.

What to watch

• Whether Hy4-Preview support reaches the performance and feature coverage of specialised runtimes.
• How the 1M-token context behaves in real memory and latency budgets.
• Compatibility work required by the new centralised vision rotary-embedding implementation.
• Whether the generation and cache fixes translate into measurable deployment gains.

Discussion spark: Which addition in Transformers 5.17.0 is most useful to you in practice: broader model coverage, long-context support, multimodal retrieval or the inference fixes?

not affiliated with or endorsed by Hugging Face