{"id":132971,"date":"2021-12-24T01:22:43","date_gmt":"2021-12-24T09:22:43","guid":{"rendered":"https:\/\/lifeboat.com\/blog\/2021\/12\/azure-ai-milestone-microsoft-kear-surpasses-human-performance-on-commonsenseqa-benchmark"},"modified":"2021-12-24T01:22:43","modified_gmt":"2021-12-24T09:22:43","slug":"azure-ai-milestone-microsoft-kear-surpasses-human-performance-on-commonsenseqa-benchmark","status":"publish","type":"post","link":"https:\/\/lifeboat.com\/blog\/2021\/12\/azure-ai-milestone-microsoft-kear-surpasses-human-performance-on-commonsenseqa-benchmark","title":{"rendered":"Azure AI milestone: Microsoft KEAR surpasses human performance on CommonsenseQA benchmark"},"content":{"rendered":"<p><\/p>\n<p><iframe style=\"display: block; margin: 0 auto; width: 100%; aspect-ratio: 4\/3; object-fit: contain;\" src=\"https:\/\/www.youtube.com\/embed\/j6IFOTdnJMg?feature=oembed\" frameborder=\"0\" allow=\"accelerometer; autoplay; encrypted-media; gyroscope;\n   picture-in-picture\" allowfullscreen><\/iframe><\/p>\n<p><em><a href=\"https:\/\/www.microsoft.com\/en-us\/research\/publication\/human-parity-on-commonsenseqa-augmenting-self-attention-with-external-attention\/\">KEAR (Knowledgeable External Attention for commonsense Reasoning) <\/a>\u2014along with recent milestones in <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/blog\/azure-ai-milestone-new-foundation-model-florence-v1-0-pushing-vision-and-vision-language-state-of-the-art\/\">computer vision<\/a> and <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/blog\/azure-ai-milestone-new-neural-text-to-speech-models-more-closely-mirror-natural-speech\/\">neural text-to-speech <\/a>\u2014is part of a larger <a href=\"https:\/\/azure.microsoft.com\/en-us\/overview\/ai-platform\/\">Azure AI<\/a> mission to provide relevant, meaningful AI solutions and services that work better for people because they better capture how people learn and work\u2014with improved vision, knowledge understanding, and speech capabilities. At the center of these efforts is XYZ-code, a joint representation of three cognitive attributes: monolingual text (X), audio or visual sensory signals (Y), and multilingual (Z). For more information about these efforts, read the <\/em><a href=\"https:\/\/www.microsoft.com\/en-us\/research\/blog\/a-holistic-representation-toward-integrative-ai\/\" target=\"_blank\" rel=\"noreferrer noopener\"><em>XYZ-code blog post<\/em><\/a><em>.<\/em><\/p>\n<p>Last month, our <a href=\"https:\/\/azure.microsoft.com\/en-us\/services\/cognitive-services\/\" target=\"_blank\" rel=\"noreferrer noopener\">Azure Cognitive Services<\/a> team, comprising researchers and engineers with expertise in AI, achieved a groundbreaking milestone by advancing commonsense language understanding. When given a question that requires drawing on prior knowledge and five answer choices, our latest model\u2014 <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/publication\/human-parity-on-commonsenseqa-augmenting-self-attention-with-external-attention\/\" target=\"_blank\" rel=\"noreferrer noopener\">KEAR, Knowledgeable External Attention for commonsense Reasoning <\/a>\u2014performs better than people answering the same question, calculated as the majority vote among five individuals. KEAR reaches an accuracy of 89.4 percent on the <a href=\"https:\/\/www.tau-nlp.org\/commonsenseqa\" target=\"_blank\" rel=\"noreferrer noopener\">CommonsenseQA<\/a> leaderboard compared with 88.9 percent human accuracy. While the CommonsenseQA benchmark is in English, we follow a similar technique for multilingual commonsense reasoning and topped the <a href=\"https:\/\/inklab.usc.edu\/XCSR\/\" target=\"_blank\" rel=\"noreferrer noopener\">X-CSR<\/a> leaderboard.<\/p>\n<p>Although recent large deep learning models trained with big data have made significant breakthroughs in natural language understanding, they still struggle with commonsense knowledge about the world, information that we, as people, have gathered in our day-to-day lives over time. Commonsense knowledge is often absent from task input but is crucial for language understanding. For example, take the question \u201cWhat is a treat that your dog will enjoy?\u201d To select an answer from the choices <em>salad<\/em>, <em>petted<\/em>, <em>affection<\/em>, <em>bone<\/em>, and <em>lots of attention<\/em>, we need to know that dogs generally enjoy food such as bones for a treat. Thus, the best answer would be \u201cbone.\u201d Without this external knowledge, even large-scale models may generate incorrect answers. For example, the <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/publication\/deberta-decoding-enhanced-bert-with-disentangled-attention-2\/\" target=\"_blank\" rel=\"noreferrer noopener\">DeBERTa language model<\/a> selects \u201clots of attention,\u201d which is not as good an answer as \u201cbone.\u201d<\/p>\n","protected":false},"excerpt":{"rendered":"<p>KEAR (Knowledgeable External Attention for commonsense Reasoning) \u2014along with recent milestones in computer vision and neural text-to-speech \u2014is part of a larger Azure AI mission to provide relevant, meaningful AI solutions and services that work better for people because they better capture how people learn and work\u2014with improved vision, knowledge understanding, and speech capabilities. At [\u2026]<\/p>\n","protected":false},"author":359,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1506,41,6],"tags":[],"class_list":["post-132971","post","type-post","status-publish","format-standard","hentry","category-food","category-information-science","category-robotics-ai"],"_links":{"self":[{"href":"https:\/\/lifeboat.com\/blog\/wp-json\/wp\/v2\/posts\/132971","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/lifeboat.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/lifeboat.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/lifeboat.com\/blog\/wp-json\/wp\/v2\/users\/359"}],"replies":[{"embeddable":true,"href":"https:\/\/lifeboat.com\/blog\/wp-json\/wp\/v2\/comments?post=132971"}],"version-history":[{"count":0,"href":"https:\/\/lifeboat.com\/blog\/wp-json\/wp\/v2\/posts\/132971\/revisions"}],"wp:attachment":[{"href":"https:\/\/lifeboat.com\/blog\/wp-json\/wp\/v2\/media?parent=132971"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/lifeboat.com\/blog\/wp-json\/wp\/v2\/categories?post=132971"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/lifeboat.com\/blog\/wp-json\/wp\/v2\/tags?post=132971"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}