{"id":138374,"date":"2022-04-20T20:22:28","date_gmt":"2022-04-21T01:22:28","guid":{"rendered":"https:\/\/lifeboat.com\/blog\/2022\/04\/0-comments-on-toward-self-improving-neural-networks-schmidhuber-teams-scalable-self-referential-weight-matrix-learns-to-modify-itself"},"modified":"2022-04-20T20:22:28","modified_gmt":"2022-04-21T01:22:28","slug":"0-comments-on-toward-self-improving-neural-networks-schmidhuber-teams-scalable-self-referential-weight-matrix-learns-to-modify-itself","status":"publish","type":"post","link":"https:\/\/lifeboat.com\/blog\/2022\/04\/0-comments-on-toward-self-improving-neural-networks-schmidhuber-teams-scalable-self-referential-weight-matrix-learns-to-modify-itself","title":{"rendered":"0 comments on \u201cToward Self-Improving Neural Networks: Schmidhuber Team\u2019s Scalable Self-Referential Weight Matrix Learns to Modify Itself\u201d"},"content":{"rendered":"<p><a class=\"aligncenter blog-photo\" href=\"https:\/\/lifeboat.com\/blog.images\/0-comments-on-toward-self-improving-neural-networks-schmidhuber-teams-scalable-self-referential-weight-matrix-learns-to-modify-itself2.jpg\"><\/a><\/p>\n<p>Back in 1993, AI pioneer J\u00fcrgen Schmidhuber published the <a href=\"https:\/\/mediatum.ub.tum.de\/doc\/814784\/file.pdf\">paper<\/a><em>A Self-Referential Weight Matrix<\/em>, which he described as a \u201cthought experiment\u2026 intended to make a step towards self-referential machine learning by showing the theoretical possibility of self-referential neural networks whose weight matrices (WMs) can learn to implement and improve their own weight change algorithm.\u201d A lack of subsequent practical studies in this area had however left this potentially impactful meta-learning ability unrealized \u2014 until now.<\/p>\n<p>In the new paper <em>A Modern Self-Referential Weight Matrix That Learns to Modify Itself<\/em>, a research team from The Swiss AI Lab, IDSIA, University of Lugano (USI) &amp; SUPSI, and King Abdullah University of Science and Technology (KAUST) presents a scalable self-referential WM (SRWM) that leverages outer products and the delta update rule to update and improve itself, achieving both practical applicability and impressive performance in game environments.<\/p>\n<p>The proposed model is built upon fast weight programmers (FWPs), a scalable and effective method dating back to the \u201890s that can learn to memorize past data and compute fast weight changes via programming instructions that are additive outer products of self-invented activation patterns, aka keys and values for self-attention. In light of their connection to linear variants of today\u2019s popular transformer architectures, FWPs are now witnessing a revival. Recent studies have advanced conventional FWPs with improved elementary programming instructions or update rules invoked by their slow neural net to reprogram the fast neural net, an approach that has been dubbed the \u201cdelta update rule.\u201d<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Back in 1993, AI pioneer J\u00fcrgen Schmidhuber published the paperA Self-Referential Weight Matrix, which he described as a \u201cthought experiment\u2026 intended to make a step towards self-referential machine learning by showing the theoretical possibility of self-referential neural networks whose weight matrices (WMs) can learn to implement and improve their own weight change algorithm.\u201d A lack [\u2026]<\/p>\n","protected":false},"author":359,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[41,6],"tags":[],"class_list":["post-138374","post","type-post","status-publish","format-standard","hentry","category-information-science","category-robotics-ai"],"_links":{"self":[{"href":"https:\/\/lifeboat.com\/blog\/wp-json\/wp\/v2\/posts\/138374","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/lifeboat.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/lifeboat.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/lifeboat.com\/blog\/wp-json\/wp\/v2\/users\/359"}],"replies":[{"embeddable":true,"href":"https:\/\/lifeboat.com\/blog\/wp-json\/wp\/v2\/comments?post=138374"}],"version-history":[{"count":0,"href":"https:\/\/lifeboat.com\/blog\/wp-json\/wp\/v2\/posts\/138374\/revisions"}],"wp:attachment":[{"href":"https:\/\/lifeboat.com\/blog\/wp-json\/wp\/v2\/media?parent=138374"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/lifeboat.com\/blog\/wp-json\/wp\/v2\/categories?post=138374"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/lifeboat.com\/blog\/wp-json\/wp\/v2\/tags?post=138374"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}