GithubStar/ktransformers
1
0
Fork 0
mirror of https://github.com/kvcache-ai/ktransformers.git synced 2026-10-22 09:35:57 +08:00
Code Issues Packages Projects Releases Wiki Activity

deploy: 3ebe17eb63

Browse Source
This commit is contained in:
Atream
2025-02-26 14:05:02 +00:00
parent b9680cf5a4
commit df3866caa5
4 changed files with 6 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files
en/DeepseekR1_V3_tutorial.html
+2 -1
View File
@@ -357,7 +357,8 @@ is speed up which is inspiring. So our showcase makes use of this finding</em></
<h2 id="how-to-run"><a class="header" href="#how-to-run">How to Run</a></h2>
<h3 id="v022-longer-context--fp8-kernel"><a class="header" href="#v022-longer-context--fp8-kernel">V0.2.2 longer context &amp; FP8 kernel</a></h3>
<h4 id="longer-context"><a class="header" href="#longer-context">longer context</a></h4>
<p>To use this feature, <a href="https://github.com/flashinfer-ai/flashinfer">install flashinfer</a> first.</p>
<p>To use this feature, <a href="https://github.com/flashinfer-ai/flashinfer">install flashinfer</a> first.
Note: The latest MLA kernel in FlashInfer still has a few minor issues. They are continuously fixing them on the main branch. If you are using FlashInfer, please install it from the main source code.</p>
<p>If you want to use long context(longer than 20K) for prefill, enable the matrix absorption MLA during the prefill phase, which will significantly reduce the size of the kv cache. Modify yaml file like this:</p>
<pre><code>- match:
name: "^model\\.layers\\..*\\.self_attn$"
print.html
+2 -1
View File
@@ -650,7 +650,8 @@ is speed up which is inspiring. So our showcase makes use of this finding</em></
<h2 id="how-to-run"><a class="header" href="#how-to-run">How to Run</a></h2>
<h3 id="v022-longer-context--fp8-kernel"><a class="header" href="#v022-longer-context--fp8-kernel">V0.2.2 longer context &amp; FP8 kernel</a></h3>
<h4 id="longer-context"><a class="header" href="#longer-context">longer context</a></h4>
<p>To use this feature, <a href="https://github.com/flashinfer-ai/flashinfer">install flashinfer</a> first.</p>
<p>To use this feature, <a href="https://github.com/flashinfer-ai/flashinfer">install flashinfer</a> first.
Note: The latest MLA kernel in FlashInfer still has a few minor issues. They are continuously fixing them on the main branch. If you are using FlashInfer, please install it from the main source code.</p>
<p>If you want to use long context(longer than 20K) for prefill, enable the matrix absorption MLA during the prefill phase, which will significantly reduce the size of the kv cache. Modify yaml file like this:</p>
<pre><code>- match:
name: "^model\\.layers\\..*\\.self_attn$"
searchindex.js
+1 -1
View File
File diff suppressed because one or more lines are too long
searchindex.json
+1 -1
View File
File diff suppressed because one or more lines are too long