Consider teaching a computer how to read by giving it billions of books. You don’t teach it grammar rules or logic; you simply ask it to play a game: “Look at these words, and guess what word comes next.” To win this game at a world-class level, the computer can’t just memorize phrases. It has to start figuring out how the world works. If it’s reading a mystery novel, it needs to deduce who the killer is to guess the final sentence. If it’s reading a math textbook, it has to understand addition to predict the answer to a problem. This is the core idea explored in a recent scientific paper titled “Algorithmic Compression via Pretrained Neural Networks.”*The researchers look under the hood of today’s Large Language Models (LLMs)—like the AI assistants we use every day—to explain a fascinating mystery: Why does a machine trained merely to predict the next word end up looking like it can think, reason, and solve complex problems? Think about how a ZIP file works on your computer. If you have a massive text file filled with the word “apple” repeated a million times, a compression program won’t save all million words. It will compress it into a short rule: “Repeat ‘apple’ 1,000,000 times.” It turns a massive mountain of data into a tiny, elegant recipe. (learning how to learn). Because the AI is fed a massive, diverse diet of information, it can’t just memorize everything. Instead, it is forced to find the underlying “recipes” or rules behind the data it sees. When you type a prompt into an AI, it doesn’t just look up an answer in a database. It looks at your text, infers the “generative algorithm” (the underlying pattern or logic of what you are asking), and uses that pattern to compress the problem and generate the correct response. In essence, it deduces the hidden rules of the game on the fly. * Discover Complex Logic: When given a sequence of chess moves, the AI doesn’t just guess random moves; it actually reconstructs the abstract rules and evaluations of a chessboard in its digital “mind.” While this framework helps explain why AI is getting so smart, it also opens up big new questions. We know these models are compressing data and finding rules, but we still don’t fully understand the absolute limits of this approach. How close can a practical AI get to that theoretical “perfect” intelligence? What happens when the AI runs out of human data to learn from?
Vitányi was appointed professor of computer science at the University of Amsterdam, and researcher at the National Research Institute for Mathematics and Computer Science in the Netherlands (CWI, initially Mathematical Centre [MC]) where he is currently a CWI Fellow. He was guest professor at the University of Copenhagen in 1978; research associate at the Massachusetts Institute of Technology in 1985/1986; Gaikoku-Jin Kenkyuin (councilor professor) at INCOCSAT at the Tokyo Institute of Technology in 1998; visiting professor at Boston University in 2004, at Monash University in 1996 and at the National ICT of Australia NICTA at University of New South Wales in 2004/2005; visiting professor at and adjunct professor of computer science at the University of Waterloo from 2005.
