A Thoughtworks study found that generic text calibrated a 4-bit version of NVIDIA’s Nemotron 3.5 Super VL about as well as legal text in one controlled test. The weights shrank from 231.8 GiB to 68.1 GiB, with a small measured increase in cross-entropy on held-out legal text.
NVIDIA Watch analysis
What happened
Gustavo A. Lujan’s report compares three GPTQ versions of the model, calibrated on legal text, generic text matched to the legal set’s token count, and a larger generic set. On the study’s 512 held-out legal records, the legal and token-matched generic builds met the report’s pre-set acceptance criterion. The larger generic build did not.
All three quantized builds had higher cross-entropy than the unquantized model, by 0.32% to 0.61%. The matched-volume comparison detected no difference between legal and generic calibration. The Thoughtworks report on Hugging Face describes this as a test of whether a customer needs to provide its own documents to calibrate a model.
Key findings
- The weights became 3.41 times smaller
The 4-bit builds reduced the reported weight footprint from 231.8 GiB to 68.1 GiB. - Generic calibration matched legal calibration at equal token volume
The study detected no difference on its held-out legal-text test, using a margin set before scoring. - Quantization still changed the measured result
Cross-entropy rose in all three builds; one generic build did not meet the study’s acceptance criterion.
Why it matters
For teams trying to serve a large model more cheaply, calibration may not require access to a customer’s sensitive documents. That is useful both operationally and for data handling. The trade-off is that smaller weights did not mean identical measured behaviour, and the result is about text prediction on a specific legal corpus, not proof that every task or customer will fare equally well.
Our read
This is a practical result, not a magic compression button. Thoughtworks recommends trying generic calibration, then comparing the quantized model with the original on held-out customer documents before accepting it. Sensible advice, and refreshingly less glamorous than pretending a smaller model has no bill to pay in accuracy.
The measurements used an early-access checkpoint, not the later general-availability version, and the report says the results have not been repeated on that version. Treat the finding as a useful recipe to test, not a guarantee to carry across models.
What to watch
- Whether the result holds on the general-availability Nemotron checkpoint.
- Whether similar tests on other customer datasets find the same calibration trade-off.
- Whether teams can confirm task-level quality, not only cross-entropy, after quantization.
Discussion spark: Would you let a generic calibration set stand in for customer documents, or should teams always test quantized models on their own held-out data before deployment?
Sources and evidence
- Calibration text for 4-bit weight-only quantization of Nemotron 3.5 Super VL, checkpoint BF16-09212026 (18 October 2026, 18:47 UTC)
Independent WittyWires tracker for public updates about NVIDIA. Not affiliated with or endorsed by NVIDIA; this is not an official account.