{"id":180909,"date":"2024-01-19T14:23:04","date_gmt":"2024-01-19T20:23:04","guid":{"rendered":"https:\/\/lifeboat.com\/blog\/2024\/01\/calculus-on-computational-graphs-backpropagation"},"modified":"2024-01-19T14:23:04","modified_gmt":"2024-01-19T20:23:04","slug":"calculus-on-computational-graphs-backpropagation","status":"publish","type":"post","link":"https:\/\/lifeboat.com\/blog\/2024\/01\/calculus-on-computational-graphs-backpropagation","title":{"rendered":"Calculus on Computational Graphs: Backpropagation"},"content":{"rendered":"<p><a class=\"aligncenter blog-photo\" href=\"https:\/\/lifeboat.com\/blog.images\/calculus-on-computational-graphs-backpropagation3.jpg\"><\/a><\/p>\n<p>Backpropagation is the key algorithm that makes training deep models computationally tractable. For modern neural networks, it can make training with gradient descent as much as ten million times faster, relative to a naive implementation. That\u2019s the difference between a model taking a week to train and taking 200,000 years.<\/p>\n<p>Beyond its use in deep learning, backpropagation is a powerful computational tool in many other areas, ranging from weather forecasting to analyzing numerical stability \u2013 it just goes by different names. In fact, the algorithm has been reinvented at least dozens of times in different fields (see <a href=\"http:\/\/www.math.uiuc.edu\/documenta\/vol-ismp\/52_griewank-andreas-b.pdf\">Griewank (2010)<\/a>). The general, application independent, name is \u201creverse-mode differentiation.\u201d<\/p>\n<p>Fundamentally, it\u2019s a technique for calculating derivatives quickly. And it\u2019s an essential trick to have in your bag, not only in deep learning, but in a wide variety of numerical computing situations.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Backpropagation is the key algorithm that makes training deep models computationally tractable. For modern neural networks, it can make training with gradient descent as much as ten million times faster, relative to a naive implementation. That\u2019s the difference between a model taking a week to train and taking 200,000 years. Beyond its use in deep [\u2026]<\/p>\n","protected":false},"author":661,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[41,6],"tags":[],"class_list":["post-180909","post","type-post","status-publish","format-standard","hentry","category-information-science","category-robotics-ai"],"_links":{"self":[{"href":"https:\/\/lifeboat.com\/blog\/wp-json\/wp\/v2\/posts\/180909","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/lifeboat.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/lifeboat.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/lifeboat.com\/blog\/wp-json\/wp\/v2\/users\/661"}],"replies":[{"embeddable":true,"href":"https:\/\/lifeboat.com\/blog\/wp-json\/wp\/v2\/comments?post=180909"}],"version-history":[{"count":0,"href":"https:\/\/lifeboat.com\/blog\/wp-json\/wp\/v2\/posts\/180909\/revisions"}],"wp:attachment":[{"href":"https:\/\/lifeboat.com\/blog\/wp-json\/wp\/v2\/media?parent=180909"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/lifeboat.com\/blog\/wp-json\/wp\/v2\/categories?post=180909"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/lifeboat.com\/blog\/wp-json\/wp\/v2\/tags?post=180909"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}