The power of human language and thought arises from systematic compositionality—the algebraic ability to understand and produce novel combinations from known components. Fodor and Pylyshyn1 famously argued that artificial neural networks lack this capacity and are therefore not viable models of the mind. Neural networks have advanced considerably in the years since, yet the systematicity challenge persists. Here we successfully address Fodor and Pylyshyn’s challenge by providing evidence that neural networks can achieve human-like systematicity when optimized for their compositional skills. To do so, we introduce the meta-learning for compositionality (MLC) approach for guiding training… More.

---

Over 35 years ago, when Fodor and Pylyshyn raised the issue of systematicity in neural networks¹, today’s models¹⁹ and their language skills were probably unimaginable. As a credit to Fodor and Pylyshyn’s prescience, the systematicity debate has endured. Systematicity continues to challenge models^{11,12,13,14,15,16,17,18} and motivates new frameworks^{34,35,36,37,38,39,40,41}. Preliminary experiments reported in Supplementary Information 3 suggest that systematicity is still a challenge, or at the very least an open question, even for recent large language models such as GPT-4. To resolve the debate, and to understand whether neural networks can capture human-like compositional skills, we must compare humans and machines side-by-side, as in this Article and other recent work^7,42,43. In our experiments, we found that the most common human responses were algebraic and systematic in exactly the ways that Fodor and Pylyshyn¹ discuss. However, people also relied on inductive biases that sometimes support the algebraic solution and sometimes deviate from it; indeed, people are not purely algebraic machines^3,6,7. We showed how MLC enables a standard neural network optimized for its compositional skills to mimic or exceed human systematic generalization in a side-by-side comparison. MLC shows much stronger systematicity than neural networks trained in standard ways, and shows more nuanced behaviour than pristine symbolic models. MLC also allows neural networks to tackle other existing challenges, including making systematic use of isolated primitives^11,16 and using mutual exclusivity to infer meanings⁴⁴.

Our use of MLC for behavioural modelling relates to other approaches for reverse engineering human inductive biases. Bayesian approaches enable a modeller to evaluate different representational forms and parameter settings for capturing human behaviour, as specified through the model’s prior⁴⁵. These priors can also be tuned with behavioural data through hierarchical Bayesian modelling⁴⁶, although the resulting set-up can be restrictive. MLC shows how meta-learning can be used like hierarchical Bayesian models for reverse-engineering inductive biases (see ref. ⁴⁷ for a formal connection), although with the aid of neural networks for greater expressive power. Our research adds to a growing literature, reviewed previously⁴⁸, on using meta-learning for understanding human^49,50,51 or human-like behaviour^52,53,54. In our experiments, only MLC closely reproduced human behaviour with respect to both systematicity and biases, with the MLC (joint) model best navigating the trade-off between these two blueprints of human linguistic behaviour. Furthermore, MLC derives its abilities through meta-learning, where both systematic generalization and the human biases are not inherent properties of the neural network architecture but, instead, are induced from data.

Despite its successes, MLC does not solve every challenge raised in Fodor and Pylyshyn¹. MLC does not automatically handle unpractised forms of generalization or concepts outside the meta-learning distribution, reducing the scope of entirely novel structures it can correctly process (compare the encouraging results on learning novel rules reported in Supplementary Information 1, with its failure on the SCAN and COGS productivity splits). Moreover, MLC is failing to generalize to nuances in inductive biases that it was not optimized for, as we explore further through an additional behavioural and modelling experiment in Supplementary Information 2. In the language of machine learning, we conclude that the meta-learning strategy succeeds when generalization makes a new episode in-distribution with respect to the training episodes, even when the specific test items are out-of-distribution with respect to the study examples in the episode. However, meta-learning alone will not allow a standard network to generalize to episodes that are in turn out-of-distribution with respect to the ones presented during meta-learning. The current architecture also lacks a mechanism for emitting new symbols², although new symbols introduced through the study examples could be emitted through an additional pointer mechanism⁵⁵. Last, MLC is untested on the full complexity of natural language and on other modalities; therefore, whether it can achieve human-like systematicity, in all respects and from realistic training experience, remains to be determined. Nevertheless, our use of standard transformers will aid MLC in tackling a wider range of problems at scale. For example, a large language model could receive specialized meta-training⁵⁶, optimizing its compositional skills by alternating between standard training (next word prediction) and MLC meta-training that continually introduces novel words and explicitly improve systematicity (Fig. 1). For vision problems, an image classifier or generator could similarly receive specialized meta-training (through current prompt-based procedures⁵⁷) to learn how to systematically combine object features or multiple objects with relations.

According to Alphabet’s CEO, Google’s Gemini is just the first of a series of next-generation AI models that Google plans to bring to market in 2024.

With the multimodal Gemini AI model, Google wants to at least catch up with OpenAI’s GPT-4. The model is expected to be released later this year. In the recent quarterly earnings call, Alphabet CEO Sundar Pichai said that Google is “getting the model ready”.

Gemini will be released in different sizes and with different capabilities, and will be used for all internal products immediately, Pichai said. So it is likely that Gemini will replace Google’s current PaLM-2 language model. Developers and cloud customers will get access through Vertex AI.

The AI guys aren’t alright.

---

Anaconda co-founder and CEO Peter Wang predicts that books will soon be replaced by AI-generated, interactive content trips dubbed “thunks.”

Scientists from the University of Science and Technology of China and Tencent’s YouTu Lab have developed a tool to combat “hallucination” by artificial intelligence (AI) models.

Humane hasn’t officially revealed its AI assistant device yet, but the bits of information keep coming.

Humane’s first gadget, the AI Pin, is currently slated to launch on November 9th, but we just got our best look at it yet thanks to a somewhat unexpected source. Before it has even been announced, the AI Pin is one of Time.

The write-up is brief and relatively light on details, but there are a couple of new details, along with the best photo we’ve seen yet of the device. It appears the AI Pin will attach magnetically to your clothing, and… More.

---

Another day, another tiny morsel of info about what Humane is up to.

The Frontier Model Forum, an industry body focused on studying “frontier” AI models along the lines of GPT-4 and ChatGPT, today announced that it’ll pledge $10 million toward a new fund to advance research on tools for “testing and evaluating the most capable AI models.”

The fund, says the Frontier Model Forum — whose members include Anthropic, Google, Microsoft and OpenAI — will support researchers affiliated with academic institutions, research institutions and startups, with initial funding to come from both the Frontier Model Forum and its philanthropic partners, the Patrick J. McGovern Foundation, the David and Lucile Packard Foundation, former Google CEO Eric Schmidt and Estonian billionaire Jaan Tallinn.

The fund will be administered by the Meridian Institute, a nonprofit based in Washington, D.C., which will put out a call for an unspecified number of proposals “within the next few months,” the Frontier Model Forum says. The Institute’s work will be supported by an advisory committee of external experts, experts from AI companies and “individuals with experience in grantmaking,” added the Frontier Model Forum — without specifying who exactly those experts and individuals are or the size of the advisory committee in question.