An AI has a limited amount of \u201ccapacity\u201d (brainpower). Early in training, it develops quick, shallow circuits to memorize data because that\u2019s the easiest way to get the right answer. Later, it develops complex circuits for actual reasoning. Because space is limited, these two internal systems are constantly competing for control. Whichever type of data the AI happens to be reading in a specific moment determines which circuit wins the battle.<\/p>\n
People typically assume that LMs stably mature from pattern-matching parrots to generalizable intelligence during pre-training. We build a toy eval suite and show this mental model is wrong: throughout pre-training, LMs frequently and suddenly hop between parrot-like and intelligence-like modes, i.e. distinct algorithms implemented by distinct circuits. We call this mode-hopping<\/em>. Across our suite, LMs can suddenly latch onto memorized or in-context patterns instead of in-context learning, use System 1<\/a> instead of System 2 thinking, pick up what sounds true instead of what is true, fail at multi-hop persona QA, out-of-context reasoning, and emergent misalignment \u2014 then just as suddenly revert and generalize. Mode-hopping is not explained by standard optimization dynamics: it is locally stable and can not be fixed by checkpoint averaging. We instead think of it as a capacity allocation problem: in a capacity-bounded model, generalizable circuits must compete with the shallow ones learned early in training, and the data in each pre-training window decides which circuits win. Our suite provides a cheap set of pre-training monitors and a new lens on generalization. Building upon our insights, we demonstrate three applications: (i) select intermediate pre-training checkpoints that strongly generalize reasoning and alignment, better than the final pre-or mid-training checkpoints, (ii) select pre-training data that controls and stabilizes generalization dynamics, and (iii) test prior generalization predictors, falsifying the monolithic belief that \u201csimpler solutions generalize better\u201d<\/p>\n Building general AI without generalization is doable but meh. We want an intelligence that learns deep, transferable structure, not a parrot that matches shallow patterns. Real generalization would unblock many today\u2019s key open problems: data-efficient (online) learning, short<\/a>cut<\/a> learning<\/a>, transfer capabilities from verifiable domains (math<\/a>, coding<\/a>) to broader non-verifiable yet economically valuable domains<\/a>, and maintain a coherent character that truly<\/a> aligns with human values.<\/p>\n The distinction between parrots and intelligence is computational. Parrots repeat<\/a> in-context patterns<\/a>; intelligence infers in-context functions<\/a>. Parrots encode a persona as bags of disconnected facts and traits; intelligence learns a shared persona representation that connects<\/a> all. Parrots memorize reasoning steps; intelligence forms general reasoning circuits for entity tracking<\/a>, backtracking<\/a>, or even for highly abstract concepts like truth<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":" An AI has a limited amount of \u201ccapacity\u201d (brainpower). Early in training, it develops quick, shallow circuits to memorize data because that\u2019s the easiest way to get the right answer. Later, it develops complex circuits for actual reasoning. Because space is limited, these two internal systems are constantly competing for control. Whichever type of data [\u2026]<\/p>\n","protected":false},"author":709,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[41,2229,6,8],"tags":[],"class_list":["post-237345","post","type-post","status-publish","format-standard","hentry","category-information-science","category-mathematics","category-robotics-ai","category-space"],"_links":{"self":[{"href":"https:\/\/lifeboat.com\/blog\/wp-json\/wp\/v2\/posts\/237345","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/lifeboat.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/lifeboat.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/lifeboat.com\/blog\/wp-json\/wp\/v2\/users\/709"}],"replies":[{"embeddable":true,"href":"https:\/\/lifeboat.com\/blog\/wp-json\/wp\/v2\/comments?post=237345"}],"version-history":[{"count":0,"href":"https:\/\/lifeboat.com\/blog\/wp-json\/wp\/v2\/posts\/237345\/revisions"}],"wp:attachment":[{"href":"https:\/\/lifeboat.com\/blog\/wp-json\/wp\/v2\/media?parent=237345"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/lifeboat.com\/blog\/wp-json\/wp\/v2\/categories?post=237345"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/lifeboat.com\/blog\/wp-json\/wp\/v2\/tags?post=237345"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}