Lifeboat Foundation

StreamingLLM is an innovative framework that allows large language models to handle text of infinite length without the need for finetuning. This technique preserves attention sinks to maintain a near-normal attention score distribution. When the sequence of the conversation with the LLM surpasses the model’s context length, retains the KV cache for the attention sink tokens—four initial tokens are sufficient—and discards subsequent tokens to make room for the sliding window tokens. This approach enables the model to extend its context and stabilize its performance without having to recompute the entire KV values.

“The introduction of four initial tokens, as attention sinks, suffices to restore the LLM’s performance,” the researchers write. “In contrast, adding just one or two doesn’t achieve full recovery. We believe this pattern emerges because these models didn’t include a consistent starting token across all input samples during pre-training.”

Under the framework, the KV cache comprises the attention sinks and the rolling KV cache that retains the most recent tokens vital for language modeling. The researchers emphasize the versatility of, stating, design is versatile and can be seamlessly incorporated into any autoregressive language model that employs relative positional encoding.”

Google’s company-defining effort to catch up to ChatGPT creator OpenAI is turning out to be harder than expected.

Google representatives earlier this year told some cloud customers and business partners they would get access to the company’s new conversational AI, a large language model known as Gemini, by November. But the company recently told them not to expect it until the first quarter of next year, according to two people with direct knowledge. The delay comes at a bad time for Google, whose cloud sales growth has slowed while that of its bigger rival, Microsoft, has accelerated. Part of Microsoft’s success has come from selling OpenAI’s technology to its customers.

Fio Omenetto, Silklab, Tufts University.

Transistors were a pivotal innovation in the evolution of electronic technology, and they have played a critical part in the miniaturization and advancement of electronic equipment. However, they could be even better.

German researchers hoping to be the first to successfully measure quantum flickering directly in a completely empty vacuum are setting their sights on 2024.

If successful, the first-of-their-kind experiments are expected to either confirm the existence of quantum energy in the vacuum, a core concept of quantum electrodynamics (QED), or potentially result in the discovery of previously unknown laws of nature.

Quantum Flickering, Ghost Particles, and Energy in the Vacuum.

Physicists from the Eötvös Loránd University (ELTE) have been conducting research on the matter constituting the atomic nucleus utilizing the world’s three most powerful particle accelerators. Their focus has been on mapping the “primordial soup” that filled the universe in the first millionth of a second following its inception.

Intriguingly, their measurements showed that the movement of observed particles bears resemblance to the search for prey of marine predators, the patterns of climate change, and the fluctuations of stock market.

In the immediate aftermath of the Big Bang, temperatures were so extreme that atomic nuclei could not exists, nor could nucleons, their building blocks. Hence, in this first instance the universe was filled with a “primordial soup” of quarks and gluons.

To teach an AI agent a new task, like how to open a kitchen cabinet, researchers often use reinforcement learning—a trial-and-error process where the agent is rewarded for taking actions that get it closer to the goal.

In many instances, a human expert must carefully design a reward function, which is an incentive mechanism that gives the agent motivation to explore. The human expert must iteratively update that reward function as the agent explores and tries different actions. This can be time-consuming, inefficient, and difficult to scale up, especially when the task is complex and involves many steps.

Researchers from MIT, Harvard University, and the University of Washington have developed a new reinforcement learning approach that doesn’t rely on an expertly designed reward function. Instead, it leverages crowdsourced feedback, gathered from many non-expert users, to guide the agent as it learns to reach its goal. The work has been published on the pre-print server arXiv.

Physicists at Martin Luther University Halle-Wittenberg (MLU) and Central South University in China have demonstrated that, combining specific materials, heat in technical devices can be used in computing. Their discovery is based on extensive calculations and simulations. The new approach demonstrates how heat signals can be steered and amplified for use in energy-efficient data processing.

The team’s research findings have been published in the journal Advanced Electronic Materials (“PT-Symmetry Enabled Spintronic Thermal Diodes and Logic Gates.”).

Information signals are encoded as thermal spin waves (red arrows). Logical operations are realized with two magnetic strips (signal conductors) and precisely controlled with current pulses in a spacer (platinum). (Image: Berakdar group)

The ISS just celebrated its 25th anniversary — soon, the station will be hanging up its boots.

A team of AI researchers from EquiLibre Technologies, Sony AI, Amii and Midjourney, working with Google’s DeepMind project, has developed an AI system called Student of Games (SoG) that is capable of both beating humans at a variety of games and learning to play new ones. In their paper published in the journal Science Advances, the group describes the new system and its capabilities.

Over the past half-century, computer scientists and engineers have developed the idea of machine learning and artificial intelligence, in which human-generated data is used to train computer systems. The technology has applications in a variety of scenarios, one of which is playing board and/or parlor games.

Teaching a computer to play a board game and then improving its capabilities to the degree that it can beat humans has become a milestone of sorts, demonstrating how far artificial intelligence has developed. In this new study, the research team has taken another step toward artificial general intelligence—in which a computer can carry out tasks deemed superhuman.