Existing pre-trained models are generally geared towards a particular class of problems. To date, there seems to be still no consensus on what the right architecture and pre-training setup should be. This paper presents a unified framework for pre-training models that are universally effective across datasets and setups. We begin by disentangling architectural archetypes with pre-training objectives — two concepts that are commonly conflated. Next, we present a generalized and unified perspective for self-supervision in NLP and show how different pre-training objectives can be cast as one another and how interpolating between different objectives can be effective. We then propose Mixture-of-Denoisers (MoD), a pre-training objective that combines diverse pre-training paradigms together.

New evidence suggests RNA and peptides may have helped build each other on early Earth.

Richard Feynman — A Curious Character

Today marks the 104th birthday of a legendary scientist, a nobel laureate, an educator, a visionary and an extraordinary human being Dr. Richard Feynman. He is remembered just as well for his scientific brilliance as he is for his playful nature and inquisitive storytelling.

A vast reservoir of ancient water has been found thousands of feet under the ice in western Antarctica, scientists said in a paper published Thursday in the journal Science.

Researchers had long suspected but never before established the existence of such hidden pockets of Antarctic groundwater, which they believe act to lessen friction between ice sheets and underlying bedrock to make the ice more prone to slide from the continent’s interior toward the surrounding ocean.

Should we just be blasting out to the cosmos?

Two new messages will give away our location, one to TRAPPIST-1. But it brings up a few questions which question if METI is so smart after all.

China’s drive for “common prosperity” is called a ‘’profound revolution’’ by state media. It has unnerved investors.