This is a beautiful analysis of the order in which diffusion models resolve uncertainty about information at different abstraction levels in different representation spaces, with significant implications for latent diffusion.
When does diffusion generate each feature in an arbitrary representation space?
Representation choice changes the generation order of diffusion models. https://arxiv.org/abs/2610.
This paper turns a familiar “vibe” about AI image generation—that pictures appear as rough structure first and crisp detail later—into something measurable.
Diffusion models start from noise and repeatedly denoise it into an image. People have long suspected they learn in a coarse-to-fine order: big shapes and colors emerge before edges, textures, and tiny details. The authors propose feature information dynamics, an information-theoretic clock that estimates *when*, during generation, a particular feature becomes informative.
The key idea is intuitive: if knowing a feature helps the model denoise much more than not knowing it, that feature is probably “coming online” at that moment. Using the I–MMSE identity, they connect the change in a feature’s mutual information to the gap between two denoising losses: the best unconditional denoiser and the best denoiser conditioned on that feature. A bigger gap means the feature is being born; a smaller gap means it is already present or not yet useful.
They also handle hierarchies of features without double-counting. In a chain like class → mask → Canny edges, some information is shared across levels—knowing “dog” already implies a blobby body—while some is genuinely incremental—edges add contour beyond the mask. Their chained decomposition separates “shared” from “new” information so the timing story is cleaner.
Applied to pixel diffusion, the framework quantitatively confirms spectral autoregression: low-frequency structure is decided early and high-frequency detail later. More interestingly, the picture changes across representations. Raw pixels behave differently from latent spaces such as SDVAE, VAVAE, and RAE, so the same conditioning chain can distribute information unevenly across time. That suggests these spaces are not interchangeable backdrops; they shape when semantics, shape, and texture become available—and that deliberately ordered or curriculum-style generation could make diffusion training easier. The code is released to let others audit and extend the measurements. https://github.com/AI4Science-WestlakeU/feature-information-dynamics.
#DiffusionModels #deeplearning #ArtificialIntelligence #computervision
When do features emerge during diffusion? An interactive introduction to feature information density, chained decomposition, spectral autoregression, and representation-dependent generation order.