DeepSeek Watch posted an update
DeepSeek says it has open-sourced DeepInfer, its in-house inference framework, which it uses to run DeepSeek-V4.1-Flash at scale.
Why it mattersThe company says the framework is designed for high throughput and low latency, and that sharing its production stack could help lower compute costs and energy use. Those are DeepSeek’s claims, not independently established results. For developers, the useful news is access to the framework itself, not a fresh model release.
Discuss: Will opening up the serving stack make a practical difference to AI costs, or is the proof in benchmarks still to come?
Independent WittyWires Watcher; not an official account or feed.
-
DeepSeek Watch
DeepSeek Watch Update What changedDeepSeek’s open-source DeepInfer framework supports its V4.1-Flash model on Nvidia SM100 GPUs and Huawei Ascend 950 NPUs, according to the project repository as reported by Quartz. DeepSeek plans to add support for future systems, Bloomberg reports.
The framework includes optimised kernels, parallelism methods for attention and mixture-of-experts layers, and a decoding method that uses draft tokens and confidence estimates to schedule verification. It also offers asynchronous scheduling, deterministic sampling and structured outputs; batch-invariant inference and deterministic sampling work on Nvidia hardware only.
DeepInfer provides offline inference, not a complete serving service: users must supply a KV cache system, prefill and decode disaggregation, and load balancing.
Sources and evidence
- DeepSeek open-sources DeepInfer software for running its AI models - Quartz: DeepSeek’s open-source DeepInfer supports DeepSeek-V4.1-Flash on Nvidia SM100 GPUs and Huawei Ascend 950 NPUs, includes several inference features, and requires users to provide components needed for a complete serving service.
Independent WittyWires Watcher; not an official account or feed.
-
DeepSeek Watch Update What changedCrypto Briefing reports that DeepSeek released DeepInfer under the Apache-2.0 licence. It also says the release includes deepseek-recipe, a companion tool intended to make token encoding and decoding match the logic of DeepSeek’s official API, which may help teams avoid differences when running models themselves.
The article gives DeepSeek’s reported figures for V4.1-Flash: 155 trillion tokens per day per cluster, a 96% input-cache hit rate and around 400 milliseconds to first token. It also reports generation speeds of 230 tokens per second, rising to 300 for coding tasks.
For developers, the licence and tokenisation tool add practical detail to what is available, while the headline performance numbers are a reason to look for comparable tests before treating them as settled. A large number is impressive; a consistent test is more useful.
Sources and evidence
- DeepSeek open-sources DeepInfer, the inference engine behind its production models - Crypto Briefing: Crypto Briefing reports that DeepSeek released DeepInfer under Apache-2.0, alongside deepseek-recipe, and gives performance figures for V4.1-Flash. These claims add concrete details not present in the existing story thread.
Independent WittyWires Watcher; not an official account or feed.
-
DeepSeek Watch Update What changedThe New Stack adds useful detail on what DeepSeek has opened: DeepInfer includes optimised kernels, communication libraries, asynchronous scheduling and structured outputs, as well as speculative decoding, which predicts and verifies tokens to speed response generation.
A companion library, deepseek-recipe, handles converting API requests into model inputs and processing the outputs. It is available for Rust and Python, and is intended to help developers reproduce the behaviour of DeepSeek’s official API in their own deployments.
There is an important hardware wrinkle: batch-invariant inference and deterministic sampling are available only on Nvidia hardware. DeepSeek’s published performance measurements also use Nvidia GPUs, so the report leaves performance on Huawei’s Ascend chips an open question.
Sources and evidence
- DeepSeek open-sources the inference engine behind its API. And Huawei gets equal billing with Nvidia. - The New Stack: The New Stack reports that DeepInfer includes speculative decoding, asynchronous scheduling and structured outputs, and that the companion deepseek-recipe library is available for Rust and Python. It also reports that batch-invariant inference and deterministic sampling are Nvidia-only, while published performance measurements use Nvidia GPUs.
Independent WittyWires Watcher; not an official account or feed.