Eight used V100s run a 250GB AI model locally, at up to 118 tokens per second

A lawyer has built a local AI server from eight used Nvidia V100 GPUs, running a nearly 249GB version of DeepSeek V4.1 Flash entirely in GPU memory. Tom’s Hardware reports that the system produces legal writing at up to 118 tokens per second, offering a striking example of what determined operators can do with older data-centre hardware.

Watch Desk analysis

What happened

The server uses two four-GPU boards, with the cards linked within each board by NVLink. The builder split the model’s layers between the boards to limit traffic over the slower PCIe connection between them. The setup also uses custom kernels and other performance work, so these results are not a plug-and-play promise for anyone finding a V100 on the used market.

The builder’s measurements, taken on 5 November and reported by Tom’s Hardware, put code generation at up to 215 tokens per second and legal writing at up to 118. At a 128,000-token prompt, the first output token took about a minute. The system can accept up to 524,000 tokens of context, but currently handles one request at a time.

Why it matters

The example makes local inference less abstract: a model too large for a typical single workstation can run across a carefully assembled pile of older GPUs. The builder says the server can handle an estimated 50 to 60 per cent of existing language-model workflows, including first-pass work on case files, while keeping those files on-site rather than sending them to a cloud service.

That is a meaningful privacy option for sensitive legal records, but not a finished enterprise product. The reported build needed substantial hardware tinkering, power limits and troubleshooting; one connector showed burn damage, and memory errors stopped after the builder reduced RAM speed. The measurements are the builder’s own, as recounted by Tom’s Hardware, not an independent benchmark.

Our read

This is a compelling proof of possibility, not a shopping list. The useful lesson is that model size, memory and the way work is split across GPUs can matter as much as the age of the hardware. But the engineering effort, one-request limit and need for human review belong in the same story as the impressive token rates. A private server can keep files close; it cannot make a complex build effortless.

What to watch

• Whether the builder gets the server to handle several requests at once.
• How reliably the setup runs under sustained workloads.
• Whether more legal work can move on-device without compromising review or quality.

Discussion spark: For confidential work, would you accept the cost and upkeep of a local AI server to keep files in-house, or is a well-governed cloud service the better trade-off?

Watch Desk is operated by WittyWires as an independent cross-cutting AI news tracker. It does not speak for the organisations or people it covers.