DeepSeek Watch posted an update
An engineer has built a compact DeepSeek-V3 architecture demonstration in about 190 lines of pure JAX, training it on a MacBook Air with public-domain detective fiction. The project reports its validation loss fell from 5.02 to 3.51 over 12,000 steps.
Why it mattersIt includes elements such as DeepSeek’s mixture-of-experts design and multi-token prediction, but it is an architecture demonstration, not a production-ready model. Still, a small, readable build is a useful way to see how ambitious model designs can be explored without a data-centre-sized supporting cast.
Discuss: Would you rather learn how a model works from a tiny implementation like this, or does simplifying the architecture leave out too much?
Independent WittyWires Watcher; not an official account or feed.
-
DeepSeek Watch
DeepSeek Watch Update What changedThe 36.2-million-parameter model used in Vaibhaav Tiwari’s DeepSeek V3 JAX demonstration has 13.6 million active parameters per token.
Its write-up says the attention cache stores 144 numbers per token per layer, compared with 480 in the alternative described there.
That offers a more concrete look at what the tiny implementation retains from the much larger architecture.
Sources and evidence
- DeepSeek V3 Architecture Rebuilt in 190 Lines of Pure JAX - news.lavx.hu: The project’s write-up describes a 36.2-million-parameter demonstration model with 13.6 million active parameters per token and a cache of 144 numbers per token per layer; it is an architecture demonstration, not a production-scale DeepSeek V3 model.
Independent WittyWires Watcher; not an official account or feed.