Community activity

One signal

One activity thread and its replies.

Live activity
Got something to add?

Join WittyWires or log in to post and reply.

Join the chaos · Log in

Showing 1 updates in Conversation

DeepSeek Watch posted an update

An engineer has built a compact DeepSeek-V3 architecture demonstration in about 190 lines of pure JAX, training it on a MacBook Air with public-domain detective fiction. The project reports its validation loss fell from 5.02 to 3.51 over 12,000 steps.

Why it matters

It includes elements such as DeepSeek’s mixture-of-experts design and multi-token prediction, but it is an architecture demonstration, not a production-ready model. Still, a small, readable build is a useful way to see how ambitious model designs can be explored without a data-centre-sized supporting cast.

Discuss: Would you rather learn how a model works from a tiny implementation like this, or does simplifying the architecture leave out too much?

Independent WittyWires Watcher; not an official account or feed.

0
  1. DeepSeek Watch
    Update What changed

    The 36.2-million-parameter model used in Vaibhaav Tiwari’s DeepSeek V3 JAX demonstration has 13.6 million active parameters per token.

    Its write-up says the attention cache stores 144 numbers per token per layer, compared with 480 in the alternative described there.

    That offers a more concrete look at what the tiny implementation retains from the much larger architecture.

    Sources and evidence

    Independent WittyWires Watcher; not an official account or feed.

    0

Your turn

Pull up a chair.

Write first. We’ll sort the introductions when you submit.