{"id":160505,"date":"2023-03-17T16:24:55","date_gmt":"2023-03-17T21:24:55","guid":{"rendered":"https:\/\/lifeboat.com\/blog\/2023\/03\/the-model-that-changes-everything-alpaca-breakthrough-ft-apples-llm-britgpt-ernie-and-alexatm"},"modified":"2023-03-17T16:24:55","modified_gmt":"2023-03-17T21:24:55","slug":"the-model-that-changes-everything-alpaca-breakthrough-ft-apples-llm-britgpt-ernie-and-alexatm","status":"publish","type":"post","link":"https:\/\/lifeboat.com\/blog\/2023\/03\/the-model-that-changes-everything-alpaca-breakthrough-ft-apples-llm-britgpt-ernie-and-alexatm","title":{"rendered":"The Model That Changes Everything: Alpaca Breakthrough (ft. Apple\u2019s LLM, BritGPT, Ernie and AlexaTM)"},"content":{"rendered":"<p><\/p>\n<p><iframe style=\"display: block; margin: 0 auto; width: 100%; aspect-ratio: 4\/3; object-fit: contain;\" src=\"https:\/\/www.youtube.com\/embed\/xslW5sQOkC8?feature=oembed\" frameborder=\"0\" allow=\"accelerometer; autoplay; encrypted-media; gyroscope;\n   picture-in-picture\" allowfullscreen><\/iframe><\/p>\n<p>8 years of cost reduction in 5 weeks: how Stanford\u2019s Alpaca model changes everything, including the economics of OpenAI and GPT 4. The breakthrough, using self-instruct, has big implications for Apple\u2019s secret large language model, Baidu\u2019s ErnieBot, Amazon\u2019s attempts and even governmental efforts, like the newly announced BritGPT.<\/p>\n<p>I will go through how Stanford put the model together, why it costs so little, and demonstrate in action versus Chatgpt and GPT 4. And what are the implications of short-circuiting human annotation like this? With analysis of a tweet by Eliezer Yudkowsky, I delve into the workings of the model and the questions it rises.<\/p>\n<p>Web Demo: <a href=\"https:\/\/alpaca-ai0.ngrok.io\/\">https:\/\/alpaca-ai0.ngrok.io\/<\/a><\/p>\n<p>Alpaca: <a href=\"https:\/\/crfm.stanford.edu\/2023\/03\/13\/alpaca.html\">https:\/\/crfm.stanford.edu\/2023\/03\/13\/alpaca.html<\/a>.<br \/> Ark Forecast: <a href=\"https:\/\/research.ark-invest.com\/hubfs\/1_Download_Files_ARK-Invest\/Big_Ideas\/ARK%20Invest_013123_Presentation_Big%20Ideas%202023_Final.pdf\">https:\/\/research.ark-invest.com\/hubfs\/1_Download_Files_ARK-I\u2026_Final.pdf<\/a>.<br \/> Eliezer Tweet: <a href=\"https:\/\/twitter.com\/ESYudkowsky\/status\/1635577836525469697\">https:\/\/twitter.com\/ESYudkowsky\/status\/1635577836525469697<\/a><\/p>\n<blockquote class=\"twitter-tweet\" data-width=\"550\" data-dnt=\"true\">\n<p lang=\"en\" dir=\"ltr\">Dear silly Internet saying \u201cbut knowledge distillation is already known\u201d: the key idea here is that the fine-tuning above the base model is comparatively much *easier* and *cheaper* to extract and re-imbue, if you have a new base comparable to the earlier base model.<\/p>\n<p>(Spelling\u2026<\/p>\n<p>\u2014 Eliezer Yudkowsky \u23f9\ufe0f (<a href=\"https:\/\/twitter.com\/ESYudkowsky\">@ESYudkowsky<\/a>) <a href=\"https:\/\/twitter.com\/ESYudkowsky\/status\/1635667349792780288?ref_src=twsrc%5Etfw\">March 14, 2023<\/a><\/p><\/blockquote>\n<p><script async src=\"https:\/\/platform.twitter.com\/widgets.js\" charset=\"utf-8\"><\/script><\/p>\n<p>Self-Instruct: <a href=\"https:\/\/arxiv.org\/pdf\/2212.10560.pdf\">https:\/\/arxiv.org\/pdf\/2212.10560.pdf<\/a>.<br \/> InstructGPT: <a href=\"https:\/\/openai.com\/research\/instruction-following\">https:\/\/openai.com\/research\/instruction-following<\/a>.<br \/> OpenAI Terms: <a href=\"https:\/\/openai.com\/policies\/terms-of-use\">https:\/\/openai.com\/policies\/terms-of-use<\/a>.<br \/> MMLU Test: <a href=\"https:\/\/arxiv.org\/pdf\/2009.03300.pdf\">https:\/\/arxiv.org\/pdf\/2009.03300.pdf<\/a>.<br \/> Apple LLM: <a href=\"https:\/\/www.nytimes.com\/2023\/03\/15\/technology\/siri-alexa-google-assistant-artificial-intelligence.html\">https:\/\/www.nytimes.com\/2023\/03\/15\/technology\/siri-alexa-goo\u2026gence.html<\/a>.<br \/> GPT 4 API: <a href=\"https:\/\/openai.com\/pricing\">https:\/\/openai.com\/pricing<\/a>.<br \/> Llama Models: <a href=\"https:\/\/arxiv.org\/pdf\/2302.13971.pdf\">https:\/\/arxiv.org\/pdf\/2302.13971.pdf<\/a>.<br \/> BritGPT: <a href=\"https:\/\/www.theguardian.com\/technology\/2023\/mar\/15\/uk-to-invest-900m-in-supercomputer-in-bid-to-build-own-britgpt\">https:\/\/www.theguardian.com\/technology\/2023\/mar\/15\/uk-to-inv\u2026wn-britgpt<\/a>.<br \/> Amazon: <a href=\"https:\/\/www.businessinsider.com\/amazons-ceo-andy-jassy-on-chat-cpt-ai-2023-2?r=US&IR=T\">https:\/\/www.businessinsider.com\/amazons-ceo-andy-jassy-on-ch\u2026?r=US&amp;IR=T<\/a><br \/> AlexaTM: <a href=\"https:\/\/arxiv.org\/pdf\/2208.01448.pdf\">https:\/\/arxiv.org\/pdf\/2208.01448.pdf<\/a>.<br \/> Baidu Ernie: <a href=\"https:\/\/www.nytimes.com\/2023\/03\/16\/world\/asia\/china-baidu-chatgpt-ernie.html\">https:\/\/www.nytimes.com\/2023\/03\/16\/world\/asia\/china-baidu-chatgpt-ernie.html<\/a>.<br \/> PaLM API: <a href=\"https:\/\/developers.googleblog.com\/2023\/03\/announcing-palm-api-and-makersuite.html\">https:\/\/developers.googleblog.com\/2023\/03\/announcing-palm-ap\u2026suite.html<\/a>.<\/p>\n<p><a href=\"http:\/\/Patreon.com\/AIExplained\">Patreon.com\/AIExplained<\/a><\/p>\n<div class=\"more-link-wrapper\"> <a class=\"more-link\" href=\"https:\/\/lifeboat.com\/blog\/2023\/03\/the-model-that-changes-everything-alpaca-breakthrough-ft-apples-llm-britgpt-ernie-and-alexatm\">Continue reading \u201cThe Model That Changes Everything: Alpaca Breakthrough (ft. Apple\u2019s LLM, BritGPT, Ernie and AlexaTM)\u201d | &gt;<\/a><\/div>\n","protected":false},"excerpt":{"rendered":"<p>8 years of cost reduction in 5 weeks: how Stanford\u2019s Alpaca model changes everything, including the economics of OpenAI and GPT 4. The breakthrough, using self-instruct, has big implications for Apple\u2019s secret large language model, Baidu\u2019s ErnieBot, Amazon\u2019s attempts and even governmental efforts, like the newly announced BritGPT. I will go through how Stanford put [\u2026]<\/p>\n","protected":false},"author":556,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[39,6],"tags":[],"class_list":["post-160505","post","type-post","status-publish","format-standard","hentry","category-economics","category-robotics-ai"],"_links":{"self":[{"href":"https:\/\/lifeboat.com\/blog\/wp-json\/wp\/v2\/posts\/160505","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/lifeboat.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/lifeboat.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/lifeboat.com\/blog\/wp-json\/wp\/v2\/users\/556"}],"replies":[{"embeddable":true,"href":"https:\/\/lifeboat.com\/blog\/wp-json\/wp\/v2\/comments?post=160505"}],"version-history":[{"count":0,"href":"https:\/\/lifeboat.com\/blog\/wp-json\/wp\/v2\/posts\/160505\/revisions"}],"wp:attachment":[{"href":"https:\/\/lifeboat.com\/blog\/wp-json\/wp\/v2\/media?parent=160505"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/lifeboat.com\/blog\/wp-json\/wp\/v2\/categories?post=160505"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/lifeboat.com\/blog\/wp-json\/wp\/v2\/tags?post=160505"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}