A lawyer has reworked an eight-GPU server to run DeepSeek V4.1 Flash with about 80% of its model work at 4-bit precision, while keeping the rest at 2-bit. Tom’s Hardware reports that the approach improves fidelity to the model’s official version, but runs more slowly per request.
Watch Desk analysis
What happened
The server uses eight used Nvidia V100 GPUs, each with 32GB of memory. The builder’s approach keeps all experts available at 2-bit, then raises the precision of the most-used experts to NVFP4, Nvidia’s 4-bit format, until the GPUs are full. The article says about 5,949 of 15,360 experts were running at 4-bit in the two-request setup.
In tests of the earlier build, the higher-precision version matched the official model’s next-token choice 95.1% of the time, up from 92.2% for the 2-bit version. Those figures measure similarity to the reference, not general answer accuracy, and were not measured on the current two-request setup. The builder’s reported results also show a trade-off: code and legal-writing output was slower per request, while two simultaneous requests raised total output to 159 tokens per second from 118.9 for one request on the earlier setup. The Tom’s Hardware report also says the server now has 1TB of RAM and that a likely faulty power supply was behind earlier crashes.
Why it matters
This is a specific example of making a large local model more faithful without requiring every part of it to fit at higher precision in GPU memory. The technique concentrates scarce memory on the experts used most often. For people exploring private, on-premises inference, it shows one possible route, alongside the less glamorous realities of slower individual responses and careful hardware troubleshooting.
Our read
The interesting result is not that old GPUs have secretly become new ones. It is that mixed precision can make a constrained local setup more useful, with measurable compromises. Treat the numbers as one builder’s reported results, not a general benchmark or a promise that the same configuration will suit another workload.
What to watch
- Whether the builder publishes comparable fidelity results for the current two-request setup.
- How performance changes as more simultaneous sessions reduce the number of experts kept at 4-bit.
- Whether the approach works well on other workloads and hardware configurations.
Discussion spark: For a private local model, would you accept slower responses per request in exchange for higher fidelity and more total throughput across simultaneous users?
Sources and evidence
Watch Desk is operated by WittyWires as an independent cross-cutting AI news tracker. It does not speak for the organisations or people it covers.