ByteDance’s Cola2 research tests a language model that drafts in a compressed, continuous space before turning that plan into text. The paper’s reported results suggest semantic compression and joint training may improve generation scores while reducing the sequence the model has to process, though the model remains far from finished.
ByteDance AI Watch analysis
What happened
A 36Kr article published on 18 November describes Cola2, a research project led by ByteDance’s Zeng Yan and her team. It says the paper and model weights were released on Hugging Face earlier this month. Read the 36Kr report.
Instead of predicting text strictly one token at a time, Cola2 first forms an outline in continuous latent space, then uses a decoder to produce the words. The report says the approach builds on an earlier Cola model and changes how content is grouped, how the decoder handles imperfect outlines, and how the two parts are trained.
Key findings
- Semantic grouping
The report says dynamic, meaning-based grouping scored 8.2% higher in accuracy than splitting text into fixed-length blocks. - A decoder trained for imperfect outlines
It says clear outlines were used in 20% of training cases, with noisy or missing outlines in the rest, while reconstruction from clear outlines remained above 99.5% accuracy. - Training the two parts together
The article reports a score of 47.6% with a fixed compressor, 53.5% when the compressor and decoder trained together, and 58.2% after adding a further prediction objective. - Early results, not a finished model
The report says Cola2 trained on about 346 billion tokens, had not converged, and averaged 60.10 points across eight tasks. It also says the paper describes the model as far from converged and constrained by available resources.
Why it matters
The idea is to let a model organise meaning in a shorter representation before expanding it into full text. The article says compressing an average of two tokens into one outline point would halve the sequence handled by the diffusion component and reduce its attention computation to a quarter in theory. That is a potential efficiency gain, not a measured promise that the whole model will run four times faster.
The reported benchmark comparison is also striking but needs its context: the article compares Cola2’s early training volume with the more than 15 trillion tokens publicly reported for Llama 3. Different training runs and task scores do not, by themselves, establish that one model is better overall.
Our read
Cola2 is interesting because it tests a different way to structure generation, not because one early benchmark settles the future of language models. The useful question is whether the approach holds up as training grows, and whether its proposed efficiency translates into practical speed or cost gains. In AI research, “promising” is a beginning, not a graduation certificate.
What to watch
- Whether the paper and released weights support independent reproduction of the reported results.
- How Cola2 performs after longer training and on a wider range of tasks.
- Whether the compressed representation produces measurable end-to-end efficiency gains.
Discussion spark: If a model drafts through compressed semantic outlines, what evidence would convince you that it is a better route than predicting text token by token?
Sources and evidence
- Zeng Yan’s New Paper Unveils ByteDance’s Ambitious 100-Trillion-Scale AI Model Roadmap – eu.36kr.com (18 November 2026, 12:42 UTC)
Independent WittyWires tracker for public updates about ByteDance AI. Not affiliated with or endorsed by ByteDance AI; this is not an official account.