{"id":24623,"date":"2024-02-01T03:51:48","date_gmt":"2024-01-31T22:21:48","guid":{"rendered":"https:\/\/farratanews.online\/microsoft-lasers-away-llm-inaccuracies\/"},"modified":"2024-02-01T03:51:48","modified_gmt":"2024-01-31T22:21:48","slug":"microsoft-lasers-away-llm-inaccuracies","status":"publish","type":"post","link":"https:\/\/farratanews.online\/microsoft-lasers-away-llm-inaccuracies\/","title":{"rendered":"Microsoft LASERs away LLM inaccuracies"},"content":{"rendered":"

[ad_1]\n<\/p>\n

\n

During the January Microsoft Research Forum, Dipendra Misra, a senior researcher at Microsoft Research Lab NYC and AI Frontiers, explained how Layer-Selective Rank Reduction (or LASER) can make large language models more accurate.\u00a0<\/p>\n<\/div>\n

\n

With LASER, researchers can \u201cintervene\u201d and replace one weight matrix with an approximate smaller one. Weights are the contextual connections models make. The heavier the weight, the more the model relies on it. So, does replacing something with more correlations and contexts make the model less accurate? Based on their test results, the answer, surprisingly, is no.\u00a0<\/p>\n<\/div>\n

\n

\u201cWe are doing intervention using LASER on the LLM, so one would expect that the model loss should go up as we are doing more approximation, meaning that the model is going to perform bad, right, because we are throwing out information from an LLM, which is trained on large amounts of data,\u201d Misra said. \u201cBut to our surprise, we find that if the right type of LASER intervention is performed, the model loss doesn\u2019t go up but actually goes down.\u201d<\/p>\n<\/div>\n

\n

Misra said his team successfully used LASER on three different open-source models: RoBERTa, Llama 2, and Eleuther\u2019s GPT-J. He said, at times, model improvement increased by 20 to 30 percentage points. For example, the performance of GPT-J for gender prediction based on biographies went from 70.9 percent accuracy to 97.5 percent after a LASER intervention.<\/p>\n<\/div>\n[ad_2]\n","protected":false},"excerpt":{"rendered":"

[ad_1] During the January Microsoft Research Forum, Dipendra Misra, a senior researcher at Microsoft Research Lab NYC and AI Frontiers, explained how Layer-Selective Rank Reduction (or LASER) can make large language models more accurate.\u00a0 With LASER, researchers can \u201cintervene\u201d and replace one weight matrix with an approximate smaller one. Weights are the contextual connections models …<\/p>\n","protected":false},"author":1,"featured_media":24624,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[2],"tags":[],"_links":{"self":[{"href":"https:\/\/farratanews.online\/wp-json\/wp\/v2\/posts\/24623"}],"collection":[{"href":"https:\/\/farratanews.online\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/farratanews.online\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/farratanews.online\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/farratanews.online\/wp-json\/wp\/v2\/comments?post=24623"}],"version-history":[{"count":0,"href":"https:\/\/farratanews.online\/wp-json\/wp\/v2\/posts\/24623\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/farratanews.online\/wp-json\/wp\/v2\/media\/24624"}],"wp:attachment":[{"href":"https:\/\/farratanews.online\/wp-json\/wp\/v2\/media?parent=24623"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/farratanews.online\/wp-json\/wp\/v2\/categories?post=24623"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/farratanews.online\/wp-json\/wp\/v2\/tags?post=24623"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}