<h1 id="issue-to-solve" data-line="0" class="code-line">
<a class="header-anchor-link" href="#issue-to-solve" aria-hidden="true"></a> Issue to Solve</h1>
<h2 id="layer-wise-importance" data-line="1" class="code-line">
<a class="header-anchor-link" href="#layer-wise-importance" aria-hidden="true"></a> Layer wise importance</h2>
<p data-line="2" class="code-line">some layers are not important -&gt; reduce max_KVCache_token<br>
<img src="https://storage.googleapis.com/zenn-user-upload/23c6e83b0193-20250616.png" loading="lazy" class="md-img"></p>
<h2 id="full-cache" data-line="5" class="code-line">
<a class="header-anchor-link" href="#full-cache" aria-hidden="true"></a> Full Cache</h2>
<p data-line="6" class="code-line">waste to keep the same amounts of KVCache for all layers<br>
<img src="https://storage.googleapis.com/zenn-user-upload/90e46836e352-20250616.png" loading="lazy" class="md-img"></p>
<h1 id="key-contributions" data-line="10" class="code-line">
<a class="header-anchor-link" href="#key-contributions" aria-hidden="true"></a> Key Contributions</h1>
<h2 id="h2o-%2B-squeezeattention" data-line="11" class="code-line">
<a class="header-anchor-link" href="#h2o-%2B-squeezeattention" aria-hidden="true"></a> H2O + SqueezeAttention</h2>
<p data-line="12" class="code-line">H20(keep only top-k important tokens) + layerwise k(max_token) based on importance of each layer<br>
<img src="https://storage.googleapis.com/zenn-user-upload/1b16610d35b5-20250616.png" loading="lazy" class="md-img"></p>
<h2 id="sliding-window-%2B-squeezeattention" data-line="15" class="code-line">
<a class="header-anchor-link" href="#sliding-window-%2B-squeezeattention" aria-hidden="true"></a> Sliding Window + SqueezeAttention</h2>
<p data-line="16" class="code-line">Sliding Window(keep only k recent tokens) + layerwise k(max_token) based on importance of each layer<br>
<img src="https://storage.googleapis.com/zenn-user-upload/6cef5be08abe-20250616.png" loading="lazy" class="md-img"></p>
<h1 id="reference" data-line="19" class="code-line">
<a class="header-anchor-link" href="#reference" aria-hidden="true"></a> Reference</h1>
<p data-line="20" class="code-line"><a href="https://arxiv.org/pdf/2404.04793" target="_blank" rel="nofollow noopener noreferrer">SQUEEZEATTENTION: 2D MANAGEMENT OF KVCACHE IN LLM INFERENCE VIA LAYER-WISE OPTIMAL BUDGET</a></p>


[KVCache] SQUEEZEATTENTION

Issue to Solve

Layer wise importance

Full Cache

Key Contributions

H2O + SqueezeAttention

Sliding Window + SqueezeAttention

Reference

Discussion