“Humans are ~1,000,000x more sample efficient than LLMs, and there are no good explanations.”
“We’ve found that a big part of the answer is that LLMs are *massively* underparameterized by many orders of magnitude. The brain has 100T synapses learning from ~200M tokens of language. Chinchilla prescribes a 1T-parameter model on 20T tokens. That’s 7 OOMs apart in per token.”
“After a lot of painstaking work, we present new data-constrained scaling laws and learning algorithms to scale to (extrapolated) quadrillion-parameter transformers.”
The current consensus in frontier pre-training is clear: high-quality human text is finite, but compute scaling is not slowing down. The key question for AI leaders is: How do we scale compute when data is constrained?
A breakthrough paper from Q Labs (*Baron et al., Oct 2026• provides a rigorous roadmap: Quadrillion-parameter architectures, aggressive weight decay, and first-class ensembling.
Here is the breakdown of the operational shift every AI research and infrastructure team should track:
1. The Parameter-to-Token Shift.
Traditional Chinchilla scaling prescribes a static ~20:1 token-to-parameter ratio. Humans learn at ~1:500,000 (synapses to tokens). As we enter multi-epoch pre-training on fixed data (U), compute-optimal parameter counts (N) scale with the compute budget rather than remaining capped by dataset size. At 10^{29} FLOPs on 5T tokens, optimal performance demands ~7,000× more per token than Chinchilla—scaling into quadrillion-parameter networks.
2. Dynamic Hyperparameter Transfer Across Epochs.
Holding weight decay fixed across multi-epoch training degrades loss scaling. To prevent overfitting on repeated tokens without choking capacity, optimal weight decay must scale steeply as a power law of both parameter count and epoch count ®. Standard MuP or fixed-regularization transfer rules miss this axis entirely.
3. Ensembling as a Direct Scaling Axis.
Rather than treating training runs as monolithic single checkpoints, ensembling should be modeled as an explicit term in the scaling law:
* Snapshot Ensembling (Zero Compute Overhead):
Implementing cyclic learning rate schedules during hyper-epoch convergence leverages mode connectivity to yield diverse, low-loss snapshot checkpoints from a single run for free.
* Deep Ensembling (Overfitting Mitigation):
Allocating compute toward deep ensembles (M) mitigates the overfitting penalty incurred by larger models, yielding steady compute-to-loss gains where single-model recipes plateau or regress.
Key Infrastructure Takeaways.
* Memory Fabrics Compute Raw TFLOPS: Storing weight states for 10^{15} requires multi-terabyte to petabyte memory topologies, turning silicon photonics, optical circuit switching, and ultra-dense HBM/Host-memory partitioning into primary bottlenecks.
* Storage & Checkpointing I/O: Snapshot cycling requires non-blocking, multi-terabyte-per-second NVMe storage fabrics to continuously capture model states without stalling execution.
#AIResearch #aiscalinglaws #deeplearning #AIInfrastructure #Transformers
Extrapolated to 5T tokens, our law prescribes ~7,000× Chinchilla’s parameter count at 1029 FLOPs, and the count keeps growing with compute.
Bigger models on repeated data need much more weight decay. Our hyperparameter law stays near-optimal at 16× the fitted compute.
With scaled hyperparameters and ensembling, loss keeps falling with compute on fixed data. Prior recipes get worse.