{"id":131319,"date":"2021-11-27T08:23:55","date_gmt":"2021-11-27T16:23:55","guid":{"rendered":"https:\/\/lifeboat.com\/blog\/2021\/11\/yann-lecun-self-supervised-learning-the-dark-matter-of-intelligence-fair-blog-post-explained"},"modified":"2021-11-27T08:23:55","modified_gmt":"2021-11-27T16:23:55","slug":"yann-lecun-self-supervised-learning-the-dark-matter-of-intelligence-fair-blog-post-explained","status":"publish","type":"post","link":"https:\/\/lifeboat.com\/blog\/2021\/11\/yann-lecun-self-supervised-learning-the-dark-matter-of-intelligence-fair-blog-post-explained","title":{"rendered":"Yann LeCun \u2014 Self-Supervised Learning: The Dark Matter of Intelligence (FAIR Blog Post Explained)"},"content":{"rendered":"<p><\/p>\n<p><iframe style=\"display: block; margin: 0 auto; width: 100%; aspect-ratio: 4\/3; object-fit: contain;\" src=\"https:\/\/www.youtube.com\/embed\/Ag1bw8MfHGQ?feature=oembed\" frameborder=\"0\" allow=\"accelerometer; autoplay; encrypted-media; gyroscope;\n   picture-in-picture\" allowfullscreen><\/iframe><\/p>\n<p>Deep Learning systems can achieve remarkable, even super-human performance through supervised learning on large, labeled datasets. However, there are two problems: First, collecting ever more labeled data is expensive in both time and money. Second, these deep neural networks will be high performers on their task, but cannot easily generalize to other, related tasks, or they need large amounts of data to do so. In this blog post, Yann LeCun and Ishan Misra of Facebook AI Research (FAIR) describe the current state of Self-Supervised Learning (SSL) and argue that it is the next step in the development of AI that uses fewer labels and can transfer knowledge faster than current systems. They suggest as a promising direction to build non-contrastive latent-variable predictive models, like VAEs, but ones that also provide high-quality latent representations for downstream tasks.<\/p>\n<p>OUTLINE:<br \/> 0:00 \u2014 Intro &amp; Overview.<br \/> 1:15 \u2014 Supervised Learning, Self-Supervised Learning, and Common Sense.<br \/> 7:35 \u2014 Predicting Hidden Parts from Observed Parts.<br \/> 17:50 \u2014 Self-Supervised Learning for Language vs Vision.<br \/> 26:50 \u2014 Energy-Based Models.<br \/> 30:15 \u2014 Joint-Embedding Models.<br \/> 35:45 \u2014 Contrastive Methods.<br \/> 43:45 \u2014 Latent-Variable Predictive Models and GANs.<br \/> 55:00 \u2014 Summary &amp; Conclusion.<\/p>\n<p>Paper (Blog Post): <a href=\"https:\/\/ai.facebook.com\/blog\/self-supervised-learning-the-dark-matter-of-intelligence\">https:\/\/ai.facebook.com\/blog\/self-supervised-learning-the-da\u2026telligence<\/a>.<br \/> My Video on BYOL: <a href=\"https:\/\/www.youtube.com\/watch?v=YPfUiOMYOEE\">https:\/\/www.youtube.com\/watch?v=YPfUiOMYOEE<\/a><\/p>\n<p>ERRATA:<br \/> - The difference between loss and energy: Energy is for inference, loss is for training.<br \/> - The R(z) term is a regularizer that restricts the capacity of the latent variable. I think I said both of those things, but never together.<br \/> - The way I explain why BERT is contrastive is wrong. I haven\u2019t figured out why just yet, though <span class=\"wp-smiley emoji emoji-smile\" title=\":)\">smile<\/span>  <\/p>\n<p>Video approved by Antonio.<\/p>\n<p>Abstract:<\/p>\n<div class=\"more-link-wrapper\"> <a class=\"more-link\" href=\"https:\/\/lifeboat.com\/blog\/2021\/11\/yann-lecun-self-supervised-learning-the-dark-matter-of-intelligence-fair-blog-post-explained\">Continue reading \u201cYann LeCun \u2014 Self-Supervised Learning: The Dark Matter of Intelligence (FAIR Blog Post Explained)\u201d | &gt;<\/a><\/div>\n","protected":false},"excerpt":{"rendered":"<p>Deep Learning systems can achieve remarkable, even super-human performance through supervised learning on large, labeled datasets. However, there are two problems: First, collecting ever more labeled data is expensive in both time and money. Second, these deep neural networks will be high performers on their task, but cannot easily generalize to other, related tasks, or [\u2026]<\/p>\n","protected":false},"author":661,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[33,6],"tags":[],"class_list":["post-131319","post","type-post","status-publish","format-standard","hentry","category-cosmology","category-robotics-ai"],"_links":{"self":[{"href":"https:\/\/lifeboat.com\/blog\/wp-json\/wp\/v2\/posts\/131319","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/lifeboat.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/lifeboat.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/lifeboat.com\/blog\/wp-json\/wp\/v2\/users\/661"}],"replies":[{"embeddable":true,"href":"https:\/\/lifeboat.com\/blog\/wp-json\/wp\/v2\/comments?post=131319"}],"version-history":[{"count":0,"href":"https:\/\/lifeboat.com\/blog\/wp-json\/wp\/v2\/posts\/131319\/revisions"}],"wp:attachment":[{"href":"https:\/\/lifeboat.com\/blog\/wp-json\/wp\/v2\/media?parent=131319"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/lifeboat.com\/blog\/wp-json\/wp\/v2\/categories?post=131319"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/lifeboat.com\/blog\/wp-json\/wp\/v2\/tags?post=131319"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}