Preprint2026
Scaling Native Multimodal Pre-Training From Scratch
Haoyuan Wu, Aoqi Wu, Hai Wang, Jiajia Wu et al. · arXiv · 2026
Abstract
A scaling-law study for vision-language models trained on multimodal data from scratch, rather than assembled from a pretrained language model and a separate vision encoder. Under a fixed compute budget the authors chart how the best model size and token count move, and find the two objectives behave differently: the language allocation stays stable regardless of data mix, while the multimodal allocation shifts sharply with it.
Why it matters
Scaling laws are how labs decide where compute goes before they spend it. This one says the text-only recipe does not carry over — the data mix changes the answer.
scaling lawsmultimodalpretrainingtraining
Read the source
https://arxiv.org/abs/2607.22043