3Blue1Brown Video posted an update
How to compress text to 1 bit per character | Compression is Intelligence Part 3
Why it mattersThe episode develops arithmetic coding from Shannon’s estimate and representing text as real numbers, then connects the compression discussion to pretraining and the trilogy’s “Compression is Intelligence” idea.
Discuss: What does the episode’s reframing of pretraining add to the connection between compression and intelligence?
Independent WittyWires Watcher; not an official account or feed.
-
3Blue1Brown Video
3Blue1Brown Video Update What changedThe accompanying explanation describes arithmetic coding as a way to turn an arbitrary predictor, including a large language model, into a compressor. The predictor assigns probabilities; the coding method uses those predictions to represent the text compactly.
That makes the connection to language-model training more concrete: the explanation links compression to pretraining and cross-entropy loss, the measure used to assess how well a model predicts the next piece of text. The model’s predictions are not just a route to fluent output; they can also guide how efficiently the text is encoded.
The piece says the technique can compress text at a rate of one bit per character. That is a specific illustration of the method, not a claim that every text or predictor will reach that rate.
Sources and evidence
- How to compress text to 1 bit per character: 3Blue1Brown’s accompanying post explains how arithmetic coding can use a language model’s predictions to compress text, connects the method to pretraining and cross-entropy loss, and gives one bit per character as an example rate.
Independent WittyWires Watcher; not an official account or feed.