ayousanz

<p>LLMで推論を行う際に精度を落とさずにより高速に推論できる技術の調査して、技術選定時の参考にする<br>
プラットフォームは、以下を想定</p>
<ul>
<li>linux(ubuntu)</li>
<li>Windows(OS:11) ,GPUは最大でも4090の一台</li>
<li>Max(M1)</li>
</ul>


<p><span class="embed-block zenn-embedded zenn-embedded-card"><iframe id="zenn-embedded__f5fcbbc852de" src="https://embed.zenn.studio/card#zenn-embedded__f5fcbbc852de" data-content="https%3A%2F%2Fengineering.linecorp.com%2Fja%2Fblog%2Fquantization-lightweighting-llms" frameborder="0" scrolling="no" loading="lazy"></iframe></span><a href="https://engineering.linecorp.com/ja/blog/quantization-lightweighting-llms" style="display:none" target="_blank" rel="nofollow noopener noreferrer">https://engineering.linecorp.com/ja/blog/quantization-lightweighting-llms</a></p>


<blockquote>
<p>評価によると、単一のNVIDIA RTX 4090 GPU上で、様々なLLM（OPT-175Bを含む）において、平均13.20トークン/秒、ピーク29.08トークン/秒のトークン生成レートを達成しました。</p>
</blockquote>
<blockquote>
<p>Google ColobのA100を使用.70Bが 5.64 トークン/秒でVRAMも33.3GBでした。</p>
</blockquote>
<p><span class="embed-block zenn-embedded zenn-embedded-card"><iframe id="zenn-embedded__72716f6aea5ca" src="https://embed.zenn.studio/card#zenn-embedded__72716f6aea5ca" data-content="https%3A%2F%2Fnote.com%2Fnpaka%2Fn%2Fn0f9d16114d6a" frameborder="0" scrolling="no" loading="lazy"></iframe></span><a href="https://note.com/npaka/n/n0f9d16114d6a" style="display:none" target="_blank" rel="nofollow noopener noreferrer">https://note.com/npaka/n/n0f9d16114d6a</a></p>


<blockquote>
<p>ngl=32 : 31トークン/秒</p>
</blockquote>
<p><span class="embed-block zenn-embedded zenn-embedded-card"><iframe id="zenn-embedded__8bf0e9d2028fb" src="https://embed.zenn.studio/card#zenn-embedded__8bf0e9d2028fb" data-content="https%3A%2F%2Fnote.com%2Fnpaka%2Fn%2Fn9eda56d3a463" frameborder="0" scrolling="no" loading="lazy"></iframe></span><a href="https://note.com/npaka/n/n9eda56d3a463" style="display:none" target="_blank" rel="nofollow noopener noreferrer">https://note.com/npaka/n/n9eda56d3a463</a></p>
