Feb 6, 2023
Vectors of Cognitive AI: Attention
Posted by Dan Breeden in categories: information science, robotics/AI
Panelists: michael graziano, jonathan cohen, vasudev lal, joscha bach.
The seminal contribution “Attention is all you need” (Vasvani et al. 2017), which introduced the Transformer algorithm, triggered a small revolution in machine learning. Unlike convolutional neural networks, which construct each feature out of a fixed neighborhood of signals, Transformers learn which data a feature on the next layer of a neural network should attend to. However, attention in neural networks is very different from the integrated attention in a human mind. In our minds, attention seems to be part of a top-down mechanism that actively creates a coherent, dynamic model of reality, and plays a crucial role in planning, inference, reflection and creative problem solving. Our consciousness appears to be involved in maintaining the control model of our attention.