This is a beautiful analysis of the order in which diffusion models resolve uncertainty about information at different abstraction levels in different representation spaces, with significant implications for latent diffusion.
When does diffusion generate each feature in an arbitrary representation space?
Representation choice changes the generation order of diffusion models. https://arxiv.org/abs/2610.
This paper turns a familiar “vibe” about AI image generation—that pictures appear as rough structure first and crisp detail later—into something measurable.
Diffusion models start from noise and repeatedly denoise it into an image. People have long suspected they learn in a coarse-to-fine order: big shapes and colors emerge before edges, textures, and tiny details. The authors propose feature information dynamics, an information-theoretic clock that estimates *when*, during generation, a particular feature becomes informative.
The key idea is intuitive: if knowing a feature helps the model denoise much more than not knowing it, that feature is probably “coming online” at that moment. Using the I–MMSE identity, they connect the change in a feature’s mutual information to the gap between two denoising losses: the best unconditional denoiser and the best denoiser conditioned on that feature. A bigger gap means the feature is being born; a smaller gap means it is already present or not yet useful.
They also handle hierarchies of features without double-counting. In a chain like class → mask → Canny edges, some information is shared across levels—knowing “dog” already implies a blobby body—while some is genuinely incremental—edges add contour beyond the mask. Their chained decomposition separates “shared” from “new” information so the timing story is cleaner.
