From 4cd8cded340a5ba9401c23091da11aa672da12f0 Mon Sep 17 00:00:00 2001 From: JimmyPeilinLi Date: Wed, 22 Apr 2026 09:44:01 +0000 Subject: [PATCH] [docs]: align install guides with explicit package flow --- doc/en/SFT/DPO_tutorial.md | 20 +- .../KTransformers-Fine-Tuning_User-Guide.md | 33 +- doc/en/SFT_Installation_Guide_KimiK2.5.md | 25 +- doc/en/SFT_Installation_Guide_KimiK2.md | 287 ++++++++++-------- doc/en/SmallThinker_and_Glm4moe.md | 8 +- ...KTransformers-Fine-Tuning_User-Guide_zh.md | 35 ++- kt-sft/README.md | 35 ++- 7 files changed, 273 insertions(+), 170 deletions(-) diff --git a/doc/en/SFT/DPO_tutorial.md b/doc/en/SFT/DPO_tutorial.md index 8238ce05..5c5b9227 100644 --- a/doc/en/SFT/DPO_tutorial.md +++ b/doc/en/SFT/DPO_tutorial.md @@ -22,24 +22,28 @@ pip install -e ".[torch,metrics]" --no-build-isolation ### Step 3: Install KTransformers -#### Option 1: Install the KTransformers wheel that matches your Torch and Python versions, from https://github.com/kvcache-ai/ktransformers/releases/tag/v0.4.4 -(Note: The CUDA version can differ from that in the wheel filename.) +#### Option 1: Beginner install (recommended) ```Bash -pip install ktransformers-0.4.4+cu128torch28fancy-cp312-cp312-linux_x86_64.whl +pip install ktransformers transformers-kt accelerate-kt ``` -#### Option 2: Install KTransformers from source +`ktransformers` is a lightweight meta-package. This installation path does **not** rely on a runtime patch package. + +#### Option 2: Source install ```Bash git clone --depth 1 https://github.com/kvcache-ai/ktransformers.git -cd ktransformers/kt-sft -export TORCH_CUDA_ARCH_LIST="8.0;8.9;9.0" # set according to your GPU +cd ktransformers +git submodule update --init --recursive -pip install -r "requirements-sft.txt" -KTRANSFORMERS_FORCE_BUILD=TRUE pip install -v . --no-build-isolation +cd kt-kernel +CPUINFER_PARALLEL=32 pip install . +cd .. +pip install -e . +pip install transformers-kt accelerate-kt ``` ### Step 4: Install the Flash-attention wheel that matches your Torch and Python versions, from: https://github.com/Dao-AILab/flash-attention/releases diff --git a/doc/en/SFT/KTransformers-Fine-Tuning_User-Guide.md b/doc/en/SFT/KTransformers-Fine-Tuning_User-Guide.md index 298e2aa3..5bde3dc4 100644 --- a/doc/en/SFT/KTransformers-Fine-Tuning_User-Guide.md +++ b/doc/en/SFT/KTransformers-Fine-Tuning_User-Guide.md @@ -94,10 +94,9 @@ This section shows how to install and use **LLaMA-Factory + KTransformers** for ### Environment Setup -According to the following example, install both the **KTransformers** and **LLaMA-Factory** environments simultaneously. - This time, to simplify the installation process of KTransformers, we have specially packaged a wheel file to avoid local compilation. - The detailed installation steps are as follows: - (Note: Make sure your local **Python version**, **Torch version**, **CUDA version**, and the **KTransformers wheel filename** correspond correctly.) +We support two installation methods for the **KTransformers** + **LLaMA-Factory** stack. + +#### Method 1: Beginner install (recommended) ```shell # 1. Create a conda environment @@ -110,10 +109,11 @@ git clone --depth 1 https://github.com/hiyouga/LLaMA-Factory.git cd LLaMA-Factory pip install -e ".[torch,metrics]" --no-build-isolation -# 3. Install the KTransformers wheel that matches your Torch and Python versions, from https://github.com/kvcache-ai/ktransformers/releases/tag/v0.4.1 (Note: The CUDA version can differ from that in the wheel filename.) -pip install ktransformers-0.4.1+cu128torch27fancy-cp312-cp312-linux_x86_64.whl +# 3. Install the KT release packages explicitly +pip install ktransformers transformers-kt accelerate-kt -# 4. Install flash-attention, download the corresponding file based on your Python and Torch versions from: https://github.com/Dao-AILab/flash-attention/releases +# 4. Install flash-attention, download the matching file from: +# https://github.com/Dao-AILab/flash-attention/releases pip install flash_attn-2.8.3+cu12torch2.7cxx11abiTRUE-cp312-cp312-linux_x86_64.whl # abi=True/False can find from below # import torch @@ -124,6 +124,25 @@ git clone https://github.com/kvcache-ai/custom_flashinfer.git pip install custom_flashinfer/ ``` +`ktransformers` is a lightweight meta-package. The beginner path does **not** rely on a runtime patch package; install `transformers-kt` and `accelerate-kt` explicitly. + +#### Method 2: Source install + +```shell +git clone https://github.com/kvcache-ai/ktransformers.git +cd ktransformers +git submodule update --init --recursive + +cd kt-kernel +CPUINFER_PARALLEL=32 pip install . +cd .. + +pip install -e . +pip install transformers-kt accelerate-kt +``` + +If you need local source changes in the fork packages, install `transformers-kt` and `accelerate-kt` from their source trees instead of the released wheels. + **Usage tip:** In LLaMA-Factory YAML, set `use_kt: true` and pick a `kt_optimize_rule` file to have KTransformers handle the core compute. The features below show typical configs. ### Core Feature 1: Use KTransformers backend to fine-tune ultra-large MoE models diff --git a/doc/en/SFT_Installation_Guide_KimiK2.5.md b/doc/en/SFT_Installation_Guide_KimiK2.5.md index 858bb9ae..a4ea8b1e 100644 --- a/doc/en/SFT_Installation_Guide_KimiK2.5.md +++ b/doc/en/SFT_Installation_Guide_KimiK2.5.md @@ -88,9 +88,30 @@ pip install -e . ```bash conda install -y -c conda-forge libstdcxx-ng gcc_impl_linux-64 conda install -y -c nvidia/label/cuda-11.8.0 cuda-runtime +``` -# Install matching wheels (recommended), from https://github.com/kvcache-ai/ktransformers/releases -pip install ktransformers-.whl +#### Option 1: Beginner install (recommended) + +```bash +pip install ktransformers transformers-kt accelerate-kt +pip install flash_attn-.whl +``` + +`ktransformers` is a lightweight meta-package. This path does **not** rely on a runtime patch package. + +#### Option 2: Source install + +```bash +git clone https://github.com/kvcache-ai/ktransformers.git +cd ktransformers +git submodule update --init --recursive + +cd kt-kernel +CPUINFER_PARALLEL=32 pip install . +cd .. + +pip install -e . +pip install transformers-kt accelerate-kt pip install flash_attn-.whl ``` diff --git a/doc/en/SFT_Installation_Guide_KimiK2.md b/doc/en/SFT_Installation_Guide_KimiK2.md index 451602f7..a0578079 100644 --- a/doc/en/SFT_Installation_Guide_KimiK2.md +++ b/doc/en/SFT_Installation_Guide_KimiK2.md @@ -1,135 +1,152 @@ -## Installation - -### Step 1: Create a conda environment and suit it for KTransformers - -```Bash -conda create -n Kllama python=3.10 # choose from : [3.10, 3.11, 3.12, 3.13] -conda install -y -c conda-forge libstdcxx-ng gcc_impl_linux-64 -conda install -y -c nvidia/label/cuda-11.8.0 cuda-runtime -``` - -### Step 2: Install the LLaMA-Factory environment - -```Bash -git clone --depth 1 https://github.com/hiyouga/LLaMA-Factory.git -cd LLaMA-Factory -pip install -e ".[torch,metrics]" --no-build-isolation -``` - -### Step 3: Install the KTransformers wheel that matches your Torch and Python versions, from https://github.com/kvcache-ai/ktransformers/releases/tag/v0.4.1 - -(Note: The CUDA version can differ from that in the wheel filename.) - -```Bash -pip install ktransformers-0.4.1+cu128torch28fancy-cp310-cp310-linux_x86_64.whl -``` - -### Step 4: Install the Flash-attention wheel that matches your Torch and Python versions, from: https://github.com/Dao-AILab/flash-attention/releases - -```Bash -pip install https://github.com/Dao-AILab/flash-attention/releases/download/v2.8.3/flash_attn-2.8.3+cu12torch2.8cxx11abiTRUE-cp310-cp310-linux_x86_64.whl -# abi=True/False can find from below -# import torch -# print(torch._C._GLIBCXX_USE_CXX11_ABI) -``` - -### Step 5: (Optional) If you want to use flash_infer (otherwise it defaults to triton) - -```Bash -git clone https://github.com/kvcache-ai/custom_flashinfer.git -pip install custom_flashinfer/ -``` - -## Download Model - -Download the official KIMI weights. If the weights are in FP8 format, please refer to [convert_kimi_k2_fp8_to_bf16_cpu.py](https://github.com/kvcache-ai/ktransformers/blob/main/kt-kernel/scripts/convert_kimi_k2_fp8_to_bf16_cpu.py) to convert them to BF16 weights. - -## How to start - -```Python -# For LoRA SFT -USE_KT=1 llamafactory-cli train examples/train_lora/kimik2_lora_sft_kt.yaml -# For Chat with model after LoRA SFT -llamafactory-cli chat examples/inference/kimik2_lora_sft_kt.yaml -# For API with model after LoRA SFT -llamafactory-cli api examples/inference/kimik2_lora_sft_kt.yaml -``` - -**If your** **CPU** **memory is insufficient to exceed 2T to support the Kimi K2, you can use the swap method additionally:** - -```Plain -sudo fallocate -l 200G /data/swapfile -sudo chmod 600 /data/swapfile -sudo mkswap /data/swapfile -sudo swapon /data/swapfile -``` - -For example, we provide the YAML file as follows: (Since the structures of Kimi and DeepSeek are relatively similar, we use deepseek as template in llamafactory) - -（1）examples/train_lora/kimik2_lora_sft_kt.yaml - -```YAML -### model -model_name_or_path: KimiK2-model -trust_remote_code: true - -### method -stage: sft -do_train: true -finetuning_type: lora -lora_rank: 8 -lora_target: all - -### dataset -dataset: identity -template: deepseek -cutoff_len: 2048 -max_samples: 100000 -overwrite_cache: true -preprocessing_num_workers: 16 -dataloader_num_workers: 4 - -### output -output_dir: saves/Kllama_kimik2 -logging_steps: 10 -save_steps: 500 -plot_loss: true -overwrite_output_dir: true -save_only_model: false -report_to: none # choices: [none, wandb, tensorboard, swanlab, mlflow] - -### train -per_device_train_batch_size: 1 -gradient_accumulation_steps: 8 -learning_rate: 1.0e-4 -num_train_epochs: 3.0 -lr_scheduler_type: cosine -warmup_ratio: 0.1 -bf16: true -ddp_timeout: 180000000 -resume_from_checkpoint: null - -### ktransformers -use_kt: true # use KTransformers as LoRA sft backend -kt_optimize_rule: examples/kt_optimize_rules/DeepSeek-V3-Chat-sft-amx-multi-gpu.yaml -cpu_infer: 32 -chunk_size: 8192 -``` - -For more details about --kt_optimize_rule, please refer to https://github.com/kvcache-ai/ktransformers/blob/main/doc/en/KTransformers-Fine-Tuning_User-Guide.md - -（2）examples/inference/kimik2_lora_sft_kt.yaml - -```YAML -model_name_or_path: opensourcerelease/DeepSeek-V3-bf16 -adapter_name_or_path: saves/Kllama_deepseekV3 -template: deepseek -infer_backend: ktransformers # choices: [huggingface, vllm, sglang, ktransformers] -trust_remote_code: true - -use_kt: true # use KTransformers as LoRA sft backend to inference -kt_optimize_rule: examples/kt_optimize_rules/DeepSeek-V3-Chat-sft-amx-multi-gpu.yaml -cpu_infer: 32 -chunk_size: 8192 - -``` +## Installation + +### Step 1: Create a conda environment and suit it for KTransformers + +```Bash +conda create -n Kllama python=3.10 # choose from : [3.10, 3.11, 3.12, 3.13] +conda install -y -c conda-forge libstdcxx-ng gcc_impl_linux-64 +conda install -y -c nvidia/label/cuda-11.8.0 cuda-runtime +``` + +### Step 2: Install the LLaMA-Factory environment + +```Bash +git clone --depth 1 https://github.com/hiyouga/LLaMA-Factory.git +cd LLaMA-Factory +pip install -e ".[torch,metrics]" --no-build-isolation +``` + +### Step 3: Install KTransformers + +#### Option 1: Beginner install (recommended) + +```Bash +pip install ktransformers transformers-kt accelerate-kt +``` + +`ktransformers` is a lightweight meta-package. This installation path does **not** rely on a runtime patch package. + +#### Option 2: Source install + +```Bash +git clone --depth 1 https://github.com/kvcache-ai/ktransformers.git +cd ktransformers +git submodule update --init --recursive + +cd kt-kernel +CPUINFER_PARALLEL=32 pip install . +cd .. + +pip install -e . +pip install transformers-kt accelerate-kt +``` + +### Step 4: Install the Flash-attention wheel that matches your Torch and Python versions, from: https://github.com/Dao-AILab/flash-attention/releases + +```Bash +pip install https://github.com/Dao-AILab/flash-attention/releases/download/v2.8.3/flash_attn-2.8.3+cu12torch2.8cxx11abiTRUE-cp310-cp310-linux_x86_64.whl +# abi=True/False can find from below +# import torch +# print(torch._C._GLIBCXX_USE_CXX11_ABI) +``` + +### Step 5: (Optional) If you want to use flash_infer (otherwise it defaults to triton) + +```Bash +git clone https://github.com/kvcache-ai/custom_flashinfer.git +pip install custom_flashinfer/ +``` + +## Download Model + +Download the official KIMI weights. If the weights are in FP8 format, please refer to [convert_kimi_k2_fp8_to_bf16_cpu.py](https://github.com/kvcache-ai/ktransformers/blob/main/kt-kernel/scripts/convert_kimi_k2_fp8_to_bf16_cpu.py) to convert them to BF16 weights. + +## How to start + +```Python +# For LoRA SFT +USE_KT=1 llamafactory-cli train examples/train_lora/kimik2_lora_sft_kt.yaml +# For Chat with model after LoRA SFT +llamafactory-cli chat examples/inference/kimik2_lora_sft_kt.yaml +# For API with model after LoRA SFT +llamafactory-cli api examples/inference/kimik2_lora_sft_kt.yaml +``` + +**If your** **CPU** **memory is insufficient to exceed 2T to support the Kimi K2, you can use the swap method additionally:** + +```Plain +sudo fallocate -l 200G /data/swapfile +sudo chmod 600 /data/swapfile +sudo mkswap /data/swapfile +sudo swapon /data/swapfile +``` + +For example, we provide the YAML file as follows: (Since the structures of Kimi and DeepSeek are relatively similar, we use deepseek as template in llamafactory) + +（1）examples/train_lora/kimik2_lora_sft_kt.yaml + +```YAML +### model +model_name_or_path: KimiK2-model +trust_remote_code: true + +### method +stage: sft +do_train: true +finetuning_type: lora +lora_rank: 8 +lora_target: all + +### dataset +dataset: identity +template: deepseek +cutoff_len: 2048 +max_samples: 100000 +overwrite_cache: true +preprocessing_num_workers: 16 +dataloader_num_workers: 4 + +### output +output_dir: saves/Kllama_kimik2 +logging_steps: 10 +save_steps: 500 +plot_loss: true +overwrite_output_dir: true +save_only_model: false +report_to: none # choices: [none, wandb, tensorboard, swanlab, mlflow] + +### train +per_device_train_batch_size: 1 +gradient_accumulation_steps: 8 +learning_rate: 1.0e-4 +num_train_epochs: 3.0 +lr_scheduler_type: cosine +warmup_ratio: 0.1 +bf16: true +ddp_timeout: 180000000 +resume_from_checkpoint: null + +### ktransformers +use_kt: true # use KTransformers as LoRA sft backend +kt_optimize_rule: examples/kt_optimize_rules/DeepSeek-V3-Chat-sft-amx-multi-gpu.yaml +cpu_infer: 32 +chunk_size: 8192 +``` + +For more details about --kt_optimize_rule, please refer to https://github.com/kvcache-ai/ktransformers/blob/main/doc/en/KTransformers-Fine-Tuning_User-Guide.md + +（2）examples/inference/kimik2_lora_sft_kt.yaml + +```YAML +model_name_or_path: opensourcerelease/DeepSeek-V3-bf16 +adapter_name_or_path: saves/Kllama_deepseekV3 +template: deepseek +infer_backend: ktransformers # choices: [huggingface, vllm, sglang, ktransformers] +trust_remote_code: true + +use_kt: true # use KTransformers as LoRA sft backend to inference +kt_optimize_rule: examples/kt_optimize_rules/DeepSeek-V3-Chat-sft-amx-multi-gpu.yaml +cpu_infer: 32 +chunk_size: 8192 + +``` diff --git a/doc/en/SmallThinker_and_Glm4moe.md b/doc/en/SmallThinker_and_Glm4moe.md index fb4c9d18..3652bd12 100644 --- a/doc/en/SmallThinker_and_Glm4moe.md +++ b/doc/en/SmallThinker_and_Glm4moe.md @@ -48,10 +48,14 @@ huggingface-cli download --resume-download https://huggingface.co/zai-org/GLM-4. ### 3. Install KTransformers -Follow the official [Installation Guide](https://kvcache-ai.github.io/ktransformers/en/install.html). +For inference, follow the official [Installation Guide](https://kvcache-ai.github.io/ktransformers/en/install.html) and build `kt-kernel` from source. The top-level `ktransformers` PyPI package is only a lightweight meta-package and is not enough by itself for server-side inference. ```bash -pip install ktransformers # or from source if you need bleeding-edge features +git clone https://github.com/kvcache-ai/ktransformers.git +cd ktransformers +git submodule update --init --recursive +cd kt-kernel +pip install . ``` ### 4. Run SmallThinker-21B Inference Server diff --git a/doc/zh/KTransformers-Fine-Tuning_User-Guide_zh.md b/doc/zh/KTransformers-Fine-Tuning_User-Guide_zh.md index f7c301f6..98ccb927 100644 --- a/doc/zh/KTransformers-Fine-Tuning_User-Guide_zh.md +++ b/doc/zh/KTransformers-Fine-Tuning_User-Guide_zh.md @@ -101,34 +101,53 @@ ### 环境安装 -根据下面示例，同时安装KTransformers和LLaMA-Factory环境，这次为了简化KTransformers的安装流程，我们特意封装了wheel包避免本地编译，具体安装步骤如下：（注意对应好本地的python版本、torch版本、cuda版本和不同文件名的KTransformers包） +我们支持两种安装方式，用于搭建 **KTransformers + LLaMA-Factory** 微调环境。 + +#### 方式 1：新手安装（推荐） ```shell -# 1. 安装conda环境 +# 1. 安装 conda 环境 conda create -n Kllama python=3.12 # choose from : [3.11, 3.12, 3.13] conda install -y -c conda-forge libstdcxx-ng gcc_impl_linux-64 conda install -y -c nvidia/label/cuda-11.8.0 cuda-runtime -# 2. 安装llamafactory环境 +# 2. 安装 LLaMA-Factory 环境 git clone --depth 1 https://github.com/hiyouga/LLaMA-Factory.git cd LLaMA-Factory pip install -e ".[torch,metrics]" --no-build-isolation -# 3. 安装对应torch和python版本的KTransformers（CUDA版本可以跟whl命名的不一致），从https://github.com/kvcache-ai/ktransformers/releases/tag/v0.4.1 -pip install ktransformers-0.4.1+cu128torch27fancy-cp312-cp312-linux_x86_64.whl +# 3. 显式安装 KT 发布包 +pip install ktransformers transformers-kt accelerate-kt -# 4. 安装flash-attention，参照python版本和torch版本，从https://github.com/Dao-AILab/flash-attention/releases下载 +# 4. 安装 flash-attention，按 Python 和 Torch 版本从官方 release 下载对应文件 pip install flash_attn-2.8.3+cu12torch2.7cxx11abiTRUE-cp312-cp312-linux_x86_64.whl -# abi=True/False可以用下面代码查看 +# abi=True/False 可以用下面代码查看 # import torch # print(torch._C._GLIBCXX_USE_CXX11_ABI) -# 5. （可选）如果你想使用flash_infer的话（不然默认triton） +# 5. （可选）如果你想使用 flash_infer（否则默认 triton） git clone https://github.com/kvcache-ai/custom_flashinfer.git pip install custom_flashinfer/ ``` +`ktransformers` 现在是轻量级 meta-package。新手安装路径**不再依赖**顶层 runtime patch；请显式安装 `transformers-kt` 和 `accelerate-kt`。 +#### 方式 2：源码安装 + +```shell +git clone https://github.com/kvcache-ai/ktransformers.git +cd ktransformers +git submodule update --init --recursive + +cd kt-kernel +CPUINFER_PARALLEL=32 pip install . +cd .. + +pip install -e . +pip install transformers-kt accelerate-kt +``` + +如果你需要修改 fork 源码，再分别从 `transformers-kt` 和 `accelerate-kt` 的源码目录安装即可。 **使用要点**：在 LLaMA-Factory 的配置 YAML 文件中启用 KTransformers 后端，只需设置 `use_kt: true`，并指定相应的 `kt_optimize_rule` YAML 文件，即可切换到底层由 KTransformers 接管计算。下面我们将通过具体功能来说明如何设置这些配置。 diff --git a/kt-sft/README.md b/kt-sft/README.md index e94e84fa..976c5c15 100644 --- a/kt-sft/README.md +++ b/kt-sft/README.md @@ -104,16 +104,15 @@ This section shows how to install and use **LLaMA-Factory + KTransformers** for ### Environment Setup -According to the following example, install both the **KTransformers** and **LLaMA-Factory** environments simultaneously. - This time, to simplify the installation process of KTransformers, we have specially packaged a wheel file to avoid local compilation. - The detailed installation steps are as follows: - (Note: Make sure your local **Python version**, **Torch version**, **CUDA version**, and the **KTransformers wheel filename** correspond correctly.) +We support two installation methods for the **KTransformers** + **LLaMA-Factory** stack. + +#### Method 1: Beginner install (recommended) ```shell # 1. Create a conda environment conda create -n Kllama python=3.12 # choose from : [3.10, 3.11, 3.12, 3.13] conda install -y -c conda-forge libstdcxx-ng gcc_impl_linux-64 -# ATTENTION: DO NOT skip this step, even if your cuda version is not 11.8! Otherwise, you will get this error: ImportError: libcudart.so.11.0: cannot open shared object file: No such file or directory. +# ATTENTION: DO NOT skip this step, even if your cuda version is not 11.8! conda install -y -c nvidia/label/cuda-11.8.0 cuda-runtime # 2. Install the LLaMA-Factory environment @@ -121,10 +120,11 @@ git clone --depth 1 https://github.com/hiyouga/LLaMA-Factory.git cd LLaMA-Factory pip install -e ".[torch,metrics]" --no-build-isolation -# 3. Install the KTransformers wheel that matches your Torch and Python versions, from https://github.com/kvcache-ai/ktransformers/releases/tag/v0.4.1 (Note: The CUDA version can differ from that in the wheel filename.) -pip install ktransformers-0.4.1+cu128torch27fancy-cp312-cp312-linux_x86_64.whl +# 3. Install the KT release packages explicitly +pip install ktransformers transformers-kt accelerate-kt -# 4. Install flash-attention, download the corresponding file based on your Python and Torch versions from: https://github.com/Dao-AILab/flash-attention/releases +# 4. Install flash-attention, download the matching file from: +# https://github.com/Dao-AILab/flash-attention/releases pip install flash_attn-2.8.3+cu12torch2.7cxx11abiTRUE-cp312-cp312-linux_x86_64.whl # abi=True/False can find from below # import torch @@ -135,6 +135,25 @@ git clone https://github.com/kvcache-ai/custom_flashinfer.git pip install custom_flashinfer/ ``` +`ktransformers` is a lightweight meta-package. The beginner path does **not** rely on a runtime patch package; install `transformers-kt` and `accelerate-kt` explicitly. + +#### Method 2: Source install + +```shell +git clone https://github.com/kvcache-ai/ktransformers.git +cd ktransformers +git submodule update --init --recursive + +cd kt-kernel +CPUINFER_PARALLEL=32 pip install . +cd .. + +pip install -e . +pip install transformers-kt accelerate-kt +``` + +If you need local source changes in the fork packages, install `transformers-kt` and `accelerate-kt` from their source trees instead of the released wheels. + **Usage tip:** In LLaMA-Factory YAML, set `use_kt: true` and pick a `kt_optimize_rule` file to have KTransformers handle the core compute. The features below show typical configs. ### Core Feature 1: Use KTransformers backend to fine-tune ultra-large MoE models