mirror of
https://github.com/kvcache-ai/ktransformers.git
synced 2026-10-17 05:11:19 +08:00
[docs]: align install guides with explicit package flow
This commit is contained in:
JimmyPeilinLi
@@ -22,24 +22,28 @@ pip install -e ".[torch,metrics]" --no-build-isolation
|
||||
|
||||
|
||||
### Step 3: Install KTransformers
|
||||
#### Option 1: Install the KTransformers wheel that matches your Torch and Python versions, from https://github.com/kvcache-ai/ktransformers/releases/tag/v0.4.4
|
||||
|
||||
(Note: The CUDA version can differ from that in the wheel filename.)
|
||||
#### Option 1: Beginner install (recommended)
|
||||
|
||||
```Bash
|
||||
pip install ktransformers-0.4.4+cu128torch28fancy-cp312-cp312-linux_x86_64.whl
|
||||
pip install ktransformers transformers-kt accelerate-kt
|
||||
```
|
||||
|
||||
#### Option 2: Install KTransformers from source
|
||||
`ktransformers` is a lightweight meta-package. This installation path does **not** rely on a runtime patch package.
|
||||
|
||||
#### Option 2: Source install
|
||||
|
||||
```Bash
|
||||
git clone --depth 1 https://github.com/kvcache-ai/ktransformers.git
|
||||
cd ktransformers/kt-sft
|
||||
export TORCH_CUDA_ARCH_LIST="8.0;8.9;9.0" # set according to your GPU
|
||||
cd ktransformers
|
||||
git submodule update --init --recursive
|
||||
|
||||
pip install -r "requirements-sft.txt"
|
||||
KTRANSFORMERS_FORCE_BUILD=TRUE pip install -v . --no-build-isolation
|
||||
cd kt-kernel
|
||||
CPUINFER_PARALLEL=32 pip install .
|
||||
cd ..
|
||||
|
||||
pip install -e .
|
||||
pip install transformers-kt accelerate-kt
|
||||
```
|
||||
|
||||
### Step 4: Install the Flash-attention wheel that matches your Torch and Python versions, from: https://github.com/Dao-AILab/flash-attention/releases
|
||||
|
||||
@@ -94,10 +94,9 @@ This section shows how to install and use **LLaMA-Factory + KTransformers** for
|
||||
|
||||
### Environment Setup
|
||||
|
||||
According to the following example, install both the **KTransformers** and **LLaMA-Factory** environments simultaneously.
|
||||
This time, to simplify the installation process of KTransformers, we have specially packaged a wheel file to avoid local compilation.
|
||||
The detailed installation steps are as follows:
|
||||
(Note: Make sure your local **Python version**, **Torch version**, **CUDA version**, and the **KTransformers wheel filename** correspond correctly.)
|
||||
We support two installation methods for the **KTransformers** + **LLaMA-Factory** stack.
|
||||
|
||||
#### Method 1: Beginner install (recommended)
|
||||
|
||||
```shell
|
||||
# 1. Create a conda environment
|
||||
@@ -110,10 +109,11 @@ git clone --depth 1 https://github.com/hiyouga/LLaMA-Factory.git
|
||||
cd LLaMA-Factory
|
||||
pip install -e ".[torch,metrics]" --no-build-isolation
|
||||
|
||||
# 3. Install the KTransformers wheel that matches your Torch and Python versions, from https://github.com/kvcache-ai/ktransformers/releases/tag/v0.4.1 (Note: The CUDA version can differ from that in the wheel filename.)
|
||||
pip install ktransformers-0.4.1+cu128torch27fancy-cp312-cp312-linux_x86_64.whl
|
||||
# 3. Install the KT release packages explicitly
|
||||
pip install ktransformers transformers-kt accelerate-kt
|
||||
|
||||
# 4. Install flash-attention, download the corresponding file based on your Python and Torch versions from: https://github.com/Dao-AILab/flash-attention/releases
|
||||
# 4. Install flash-attention, download the matching file from:
|
||||
# https://github.com/Dao-AILab/flash-attention/releases
|
||||
pip install flash_attn-2.8.3+cu12torch2.7cxx11abiTRUE-cp312-cp312-linux_x86_64.whl
|
||||
# abi=True/False can find from below
|
||||
# import torch
|
||||
@@ -124,6 +124,25 @@ git clone https://github.com/kvcache-ai/custom_flashinfer.git
|
||||
pip install custom_flashinfer/
|
||||
```
|
||||
|
||||
`ktransformers` is a lightweight meta-package. The beginner path does **not** rely on a runtime patch package; install `transformers-kt` and `accelerate-kt` explicitly.
|
||||
|
||||
#### Method 2: Source install
|
||||
|
||||
```shell
|
||||
git clone https://github.com/kvcache-ai/ktransformers.git
|
||||
cd ktransformers
|
||||
git submodule update --init --recursive
|
||||
|
||||
cd kt-kernel
|
||||
CPUINFER_PARALLEL=32 pip install .
|
||||
cd ..
|
||||
|
||||
pip install -e .
|
||||
pip install transformers-kt accelerate-kt
|
||||
```
|
||||
|
||||
If you need local source changes in the fork packages, install `transformers-kt` and `accelerate-kt` from their source trees instead of the released wheels.
|
||||
|
||||
**Usage tip:** In LLaMA-Factory YAML, set `use_kt: true` and pick a `kt_optimize_rule` file to have KTransformers handle the core compute. The features below show typical configs.
|
||||
|
||||
### Core Feature 1: Use KTransformers backend to fine-tune ultra-large MoE models
|
||||
|
||||
@@ -88,9 +88,30 @@ pip install -e .
|
||||
```bash
|
||||
conda install -y -c conda-forge libstdcxx-ng gcc_impl_linux-64
|
||||
conda install -y -c nvidia/label/cuda-11.8.0 cuda-runtime
|
||||
```
|
||||
|
||||
# Install matching wheels (recommended), from https://github.com/kvcache-ai/ktransformers/releases
|
||||
pip install ktransformers-<matching-version>.whl
|
||||
#### Option 1: Beginner install (recommended)
|
||||
|
||||
```bash
|
||||
pip install ktransformers transformers-kt accelerate-kt
|
||||
pip install flash_attn-<matching-version>.whl
|
||||
```
|
||||
|
||||
`ktransformers` is a lightweight meta-package. This path does **not** rely on a runtime patch package.
|
||||
|
||||
#### Option 2: Source install
|
||||
|
||||
```bash
|
||||
git clone https://github.com/kvcache-ai/ktransformers.git
|
||||
cd ktransformers
|
||||
git submodule update --init --recursive
|
||||
|
||||
cd kt-kernel
|
||||
CPUINFER_PARALLEL=32 pip install .
|
||||
cd ..
|
||||
|
||||
pip install -e .
|
||||
pip install transformers-kt accelerate-kt
|
||||
pip install flash_attn-<matching-version>.whl
|
||||
```
|
||||
|
||||
|
||||
@@ -1,135 +1,152 @@
|
||||
## Installation
|
||||
|
||||
### Step 1: Create a conda environment and suit it for KTransformers
|
||||
|
||||
```Bash
|
||||
conda create -n Kllama python=3.10 # choose from : [3.10, 3.11, 3.12, 3.13]
|
||||
conda install -y -c conda-forge libstdcxx-ng gcc_impl_linux-64
|
||||
conda install -y -c nvidia/label/cuda-11.8.0 cuda-runtime
|
||||
```
|
||||
|
||||
### Step 2: Install the LLaMA-Factory environment
|
||||
|
||||
```Bash
|
||||
git clone --depth 1 https://github.com/hiyouga/LLaMA-Factory.git
|
||||
cd LLaMA-Factory
|
||||
pip install -e ".[torch,metrics]" --no-build-isolation
|
||||
```
|
||||
|
||||
### Step 3: Install the KTransformers wheel that matches your Torch and Python versions, from https://github.com/kvcache-ai/ktransformers/releases/tag/v0.4.1
|
||||
|
||||
(Note: The CUDA version can differ from that in the wheel filename.)
|
||||
|
||||
```Bash
|
||||
pip install ktransformers-0.4.1+cu128torch28fancy-cp310-cp310-linux_x86_64.whl
|
||||
```
|
||||
|
||||
### Step 4: Install the Flash-attention wheel that matches your Torch and Python versions, from: https://github.com/Dao-AILab/flash-attention/releases
|
||||
|
||||
```Bash
|
||||
pip install https://github.com/Dao-AILab/flash-attention/releases/download/v2.8.3/flash_attn-2.8.3+cu12torch2.8cxx11abiTRUE-cp310-cp310-linux_x86_64.whl
|
||||
# abi=True/False can find from below
|
||||
# import torch
|
||||
# print(torch._C._GLIBCXX_USE_CXX11_ABI)
|
||||
```
|
||||
|
||||
### Step 5: (Optional) If you want to use flash_infer (otherwise it defaults to triton)
|
||||
|
||||
```Bash
|
||||
git clone https://github.com/kvcache-ai/custom_flashinfer.git
|
||||
pip install custom_flashinfer/
|
||||
```
|
||||
|
||||
## Download Model
|
||||
|
||||
Download the official KIMI weights. If the weights are in FP8 format, please refer to [convert_kimi_k2_fp8_to_bf16_cpu.py](https://github.com/kvcache-ai/ktransformers/blob/main/kt-kernel/scripts/convert_kimi_k2_fp8_to_bf16_cpu.py) to convert them to BF16 weights.
|
||||
|
||||
## How to start
|
||||
|
||||
```Python
|
||||
# For LoRA SFT
|
||||
USE_KT=1 llamafactory-cli train examples/train_lora/kimik2_lora_sft_kt.yaml
|
||||
# For Chat with model after LoRA SFT
|
||||
llamafactory-cli chat examples/inference/kimik2_lora_sft_kt.yaml
|
||||
# For API with model after LoRA SFT
|
||||
llamafactory-cli api examples/inference/kimik2_lora_sft_kt.yaml
|
||||
```
|
||||
|
||||
**If your** **CPU** **memory is insufficient to exceed 2T to support the Kimi K2, you can use the swap method additionally:**
|
||||
|
||||
```Plain
|
||||
sudo fallocate -l 200G /data/swapfile
|
||||
sudo chmod 600 /data/swapfile
|
||||
sudo mkswap /data/swapfile
|
||||
sudo swapon /data/swapfile
|
||||
```
|
||||
|
||||
For example, we provide the YAML file as follows: (Since the structures of Kimi and DeepSeek are relatively similar, we use deepseek as template in llamafactory)
|
||||
|
||||
（1）examples/train_lora/kimik2_lora_sft_kt.yaml
|
||||
|
||||
```YAML
|
||||
### model
|
||||
model_name_or_path: KimiK2-model
|
||||
trust_remote_code: true
|
||||
|
||||
### method
|
||||
stage: sft
|
||||
do_train: true
|
||||
finetuning_type: lora
|
||||
lora_rank: 8
|
||||
lora_target: all
|
||||
|
||||
### dataset
|
||||
dataset: identity
|
||||
template: deepseek
|
||||
cutoff_len: 2048
|
||||
max_samples: 100000
|
||||
overwrite_cache: true
|
||||
preprocessing_num_workers: 16
|
||||
dataloader_num_workers: 4
|
||||
|
||||
### output
|
||||
output_dir: saves/Kllama_kimik2
|
||||
logging_steps: 10
|
||||
save_steps: 500
|
||||
plot_loss: true
|
||||
overwrite_output_dir: true
|
||||
save_only_model: false
|
||||
report_to: none # choices: [none, wandb, tensorboard, swanlab, mlflow]
|
||||
|
||||
### train
|
||||
per_device_train_batch_size: 1
|
||||
gradient_accumulation_steps: 8
|
||||
learning_rate: 1.0e-4
|
||||
num_train_epochs: 3.0
|
||||
lr_scheduler_type: cosine
|
||||
warmup_ratio: 0.1
|
||||
bf16: true
|
||||
ddp_timeout: 180000000
|
||||
resume_from_checkpoint: null
|
||||
|
||||
### ktransformers
|
||||
use_kt: true # use KTransformers as LoRA sft backend
|
||||
kt_optimize_rule: examples/kt_optimize_rules/DeepSeek-V3-Chat-sft-amx-multi-gpu.yaml
|
||||
cpu_infer: 32
|
||||
chunk_size: 8192
|
||||
```
|
||||
|
||||
For more details about --kt_optimize_rule, please refer to https://github.com/kvcache-ai/ktransformers/blob/main/doc/en/KTransformers-Fine-Tuning_User-Guide.md
|
||||
|
||||
（2）examples/inference/kimik2_lora_sft_kt.yaml
|
||||
|
||||
```YAML
|
||||
model_name_or_path: opensourcerelease/DeepSeek-V3-bf16
|
||||
adapter_name_or_path: saves/Kllama_deepseekV3
|
||||
template: deepseek
|
||||
infer_backend: ktransformers # choices: [huggingface, vllm, sglang, ktransformers]
|
||||
trust_remote_code: true
|
||||
|
||||
use_kt: true # use KTransformers as LoRA sft backend to inference
|
||||
kt_optimize_rule: examples/kt_optimize_rules/DeepSeek-V3-Chat-sft-amx-multi-gpu.yaml
|
||||
cpu_infer: 32
|
||||
chunk_size: 8192
|
||||
|
||||
```
|
||||
## Installation
|
||||
|
||||
### Step 1: Create a conda environment and suit it for KTransformers
|
||||
|
||||
```Bash
|
||||
conda create -n Kllama python=3.10 # choose from : [3.10, 3.11, 3.12, 3.13]
|
||||
conda install -y -c conda-forge libstdcxx-ng gcc_impl_linux-64
|
||||
conda install -y -c nvidia/label/cuda-11.8.0 cuda-runtime
|
||||
```
|
||||
|
||||
### Step 2: Install the LLaMA-Factory environment
|
||||
|
||||
```Bash
|
||||
git clone --depth 1 https://github.com/hiyouga/LLaMA-Factory.git
|
||||
cd LLaMA-Factory
|
||||
pip install -e ".[torch,metrics]" --no-build-isolation
|
||||
```
|
||||
|
||||
### Step 3: Install KTransformers
|
||||
|
||||
#### Option 1: Beginner install (recommended)
|
||||
|
||||
```Bash
|
||||
pip install ktransformers transformers-kt accelerate-kt
|
||||
```
|
||||
|
||||
`ktransformers` is a lightweight meta-package. This installation path does **not** rely on a runtime patch package.
|
||||
|
||||
#### Option 2: Source install
|
||||
|
||||
```Bash
|
||||
git clone --depth 1 https://github.com/kvcache-ai/ktransformers.git
|
||||
cd ktransformers
|
||||
git submodule update --init --recursive
|
||||
|
||||
cd kt-kernel
|
||||
CPUINFER_PARALLEL=32 pip install .
|
||||
cd ..
|
||||
|
||||
pip install -e .
|
||||
pip install transformers-kt accelerate-kt
|
||||
```
|
||||
|
||||
### Step 4: Install the Flash-attention wheel that matches your Torch and Python versions, from: https://github.com/Dao-AILab/flash-attention/releases
|
||||
|
||||
```Bash
|
||||
pip install https://github.com/Dao-AILab/flash-attention/releases/download/v2.8.3/flash_attn-2.8.3+cu12torch2.8cxx11abiTRUE-cp310-cp310-linux_x86_64.whl
|
||||
# abi=True/False can find from below
|
||||
# import torch
|
||||
# print(torch._C._GLIBCXX_USE_CXX11_ABI)
|
||||
```
|
||||
|
||||
### Step 5: (Optional) If you want to use flash_infer (otherwise it defaults to triton)
|
||||
|
||||
```Bash
|
||||
git clone https://github.com/kvcache-ai/custom_flashinfer.git
|
||||
pip install custom_flashinfer/
|
||||
```
|
||||
|
||||
## Download Model
|
||||
|
||||
Download the official KIMI weights. If the weights are in FP8 format, please refer to [convert_kimi_k2_fp8_to_bf16_cpu.py](https://github.com/kvcache-ai/ktransformers/blob/main/kt-kernel/scripts/convert_kimi_k2_fp8_to_bf16_cpu.py) to convert them to BF16 weights.
|
||||
|
||||
## How to start
|
||||
|
||||
```Python
|
||||
# For LoRA SFT
|
||||
USE_KT=1 llamafactory-cli train examples/train_lora/kimik2_lora_sft_kt.yaml
|
||||
# For Chat with model after LoRA SFT
|
||||
llamafactory-cli chat examples/inference/kimik2_lora_sft_kt.yaml
|
||||
# For API with model after LoRA SFT
|
||||
llamafactory-cli api examples/inference/kimik2_lora_sft_kt.yaml
|
||||
```
|
||||
|
||||
**If your** **CPU** **memory is insufficient to exceed 2T to support the Kimi K2, you can use the swap method additionally:**
|
||||
|
||||
```Plain
|
||||
sudo fallocate -l 200G /data/swapfile
|
||||
sudo chmod 600 /data/swapfile
|
||||
sudo mkswap /data/swapfile
|
||||
sudo swapon /data/swapfile
|
||||
```
|
||||
|
||||
For example, we provide the YAML file as follows: (Since the structures of Kimi and DeepSeek are relatively similar, we use deepseek as template in llamafactory)
|
||||
|
||||
（1）examples/train_lora/kimik2_lora_sft_kt.yaml
|
||||
|
||||
```YAML
|
||||
### model
|
||||
model_name_or_path: KimiK2-model
|
||||
trust_remote_code: true
|
||||
|
||||
### method
|
||||
stage: sft
|
||||
do_train: true
|
||||
finetuning_type: lora
|
||||
lora_rank: 8
|
||||
lora_target: all
|
||||
|
||||
### dataset
|
||||
dataset: identity
|
||||
template: deepseek
|
||||
cutoff_len: 2048
|
||||
max_samples: 100000
|
||||
overwrite_cache: true
|
||||
preprocessing_num_workers: 16
|
||||
dataloader_num_workers: 4
|
||||
|
||||
### output
|
||||
output_dir: saves/Kllama_kimik2
|
||||
logging_steps: 10
|
||||
save_steps: 500
|
||||
plot_loss: true
|
||||
overwrite_output_dir: true
|
||||
save_only_model: false
|
||||
report_to: none # choices: [none, wandb, tensorboard, swanlab, mlflow]
|
||||
|
||||
### train
|
||||
per_device_train_batch_size: 1
|
||||
gradient_accumulation_steps: 8
|
||||
learning_rate: 1.0e-4
|
||||
num_train_epochs: 3.0
|
||||
lr_scheduler_type: cosine
|
||||
warmup_ratio: 0.1
|
||||
bf16: true
|
||||
ddp_timeout: 180000000
|
||||
resume_from_checkpoint: null
|
||||
|
||||
### ktransformers
|
||||
use_kt: true # use KTransformers as LoRA sft backend
|
||||
kt_optimize_rule: examples/kt_optimize_rules/DeepSeek-V3-Chat-sft-amx-multi-gpu.yaml
|
||||
cpu_infer: 32
|
||||
chunk_size: 8192
|
||||
```
|
||||
|
||||
For more details about --kt_optimize_rule, please refer to https://github.com/kvcache-ai/ktransformers/blob/main/doc/en/KTransformers-Fine-Tuning_User-Guide.md
|
||||
|
||||
（2）examples/inference/kimik2_lora_sft_kt.yaml
|
||||
|
||||
```YAML
|
||||
model_name_or_path: opensourcerelease/DeepSeek-V3-bf16
|
||||
adapter_name_or_path: saves/Kllama_deepseekV3
|
||||
template: deepseek
|
||||
infer_backend: ktransformers # choices: [huggingface, vllm, sglang, ktransformers]
|
||||
trust_remote_code: true
|
||||
|
||||
use_kt: true # use KTransformers as LoRA sft backend to inference
|
||||
kt_optimize_rule: examples/kt_optimize_rules/DeepSeek-V3-Chat-sft-amx-multi-gpu.yaml
|
||||
cpu_infer: 32
|
||||
chunk_size: 8192
|
||||
|
||||
```
|
||||
|
||||
@@ -48,10 +48,14 @@ huggingface-cli download --resume-download https://huggingface.co/zai-org/GLM-4.
|
||||
|
||||
### 3. Install KTransformers
|
||||
|
||||
Follow the official [Installation Guide](https://kvcache-ai.github.io/ktransformers/en/install.html).
|
||||
For inference, follow the official [Installation Guide](https://kvcache-ai.github.io/ktransformers/en/install.html) and build `kt-kernel` from source. The top-level `ktransformers` PyPI package is only a lightweight meta-package and is not enough by itself for server-side inference.
|
||||
|
||||
```bash
|
||||
pip install ktransformers # or from source if you need bleeding-edge features
|
||||
git clone https://github.com/kvcache-ai/ktransformers.git
|
||||
cd ktransformers
|
||||
git submodule update --init --recursive
|
||||
cd kt-kernel
|
||||
pip install .
|
||||
```
|
||||
|
||||
### 4. Run SmallThinker-21B Inference Server
|
||||
|
||||
@@ -101,34 +101,53 @@
|
||||
|
||||
### 环境安装
|
||||
|
||||
根据下面示例，同时安装KTransformers和LLaMA-Factory环境，这次为了简化KTransformers的安装流程，我们特意封装了wheel包避免本地编译，具体安装步骤如下：（注意对应好本地的python版本、torch版本、cuda版本和不同文件名的KTransformers包）
|
||||
我们支持两种安装方式，用于搭建 **KTransformers + LLaMA-Factory** 微调环境。
|
||||
|
||||
#### 方式 1：新手安装（推荐）
|
||||
|
||||
```shell
|
||||
# 1. 安装conda环境
|
||||
# 1. 安装 conda 环境
|
||||
conda create -n Kllama python=3.12 # choose from : [3.11, 3.12, 3.13]
|
||||
conda install -y -c conda-forge libstdcxx-ng gcc_impl_linux-64
|
||||
conda install -y -c nvidia/label/cuda-11.8.0 cuda-runtime
|
||||
|
||||
# 2. 安装llamafactory环境
|
||||
# 2. 安装 LLaMA-Factory 环境
|
||||
git clone --depth 1 https://github.com/hiyouga/LLaMA-Factory.git
|
||||
cd LLaMA-Factory
|
||||
pip install -e ".[torch,metrics]" --no-build-isolation
|
||||
|
||||
# 3. 安装对应torch和python版本的KTransformers（CUDA版本可以跟whl命名的不一致），从https://github.com/kvcache-ai/ktransformers/releases/tag/v0.4.1
|
||||
pip install ktransformers-0.4.1+cu128torch27fancy-cp312-cp312-linux_x86_64.whl
|
||||
# 3. 显式安装 KT 发布包
|
||||
pip install ktransformers transformers-kt accelerate-kt
|
||||
|
||||
# 4. 安装flash-attention，参照python版本和torch版本，从https://github.com/Dao-AILab/flash-attention/releases下载
|
||||
# 4. 安装 flash-attention，按 Python 和 Torch 版本从官方 release 下载对应文件
|
||||
pip install flash_attn-2.8.3+cu12torch2.7cxx11abiTRUE-cp312-cp312-linux_x86_64.whl
|
||||
# abi=True/False可以用下面代码查看
|
||||
# abi=True/False 可以用下面代码查看
|
||||
# import torch
|
||||
# print(torch._C._GLIBCXX_USE_CXX11_ABI)
|
||||
|
||||
# 5. （可选）如果你想使用flash_infer的话（不然默认triton）
|
||||
# 5. （可选）如果你想使用 flash_infer（否则默认 triton）
|
||||
git clone https://github.com/kvcache-ai/custom_flashinfer.git
|
||||
pip install custom_flashinfer/
|
||||
```
|
||||
|
||||
`ktransformers` 现在是轻量级 meta-package。新手安装路径**不再依赖**顶层 runtime patch；请显式安装 `transformers-kt` 和 `accelerate-kt`。
|
||||
|
||||
#### 方式 2：源码安装
|
||||
|
||||
```shell
|
||||
git clone https://github.com/kvcache-ai/ktransformers.git
|
||||
cd ktransformers
|
||||
git submodule update --init --recursive
|
||||
|
||||
cd kt-kernel
|
||||
CPUINFER_PARALLEL=32 pip install .
|
||||
cd ..
|
||||
|
||||
pip install -e .
|
||||
pip install transformers-kt accelerate-kt
|
||||
```
|
||||
|
||||
如果你需要修改 fork 源码，再分别从 `transformers-kt` 和 `accelerate-kt` 的源码目录安装即可。
|
||||
|
||||
**使用要点**：在 LLaMA-Factory 的配置 YAML 文件中启用 KTransformers 后端，只需设置 `use_kt: true`，并指定相应的 `kt_optimize_rule` YAML 文件，即可切换到底层由 KTransformers 接管计算。下面我们将通过具体功能来说明如何设置这些配置。
|
||||
|
||||
|
||||
+27
-8
@@ -104,16 +104,15 @@ This section shows how to install and use **LLaMA-Factory + KTransformers** for
|
||||
|
||||
### Environment Setup
|
||||
|
||||
According to the following example, install both the **KTransformers** and **LLaMA-Factory** environments simultaneously.
|
||||
This time, to simplify the installation process of KTransformers, we have specially packaged a wheel file to avoid local compilation.
|
||||
The detailed installation steps are as follows:
|
||||
(Note: Make sure your local **Python version**, **Torch version**, **CUDA version**, and the **KTransformers wheel filename** correspond correctly.)
|
||||
We support two installation methods for the **KTransformers** + **LLaMA-Factory** stack.
|
||||
|
||||
#### Method 1: Beginner install (recommended)
|
||||
|
||||
```shell
|
||||
# 1. Create a conda environment
|
||||
conda create -n Kllama python=3.12 # choose from : [3.10, 3.11, 3.12, 3.13]
|
||||
conda install -y -c conda-forge libstdcxx-ng gcc_impl_linux-64
|
||||
# ATTENTION: DO NOT skip this step, even if your cuda version is not 11.8! Otherwise, you will get this error: ImportError: libcudart.so.11.0: cannot open shared object file: No such file or directory.
|
||||
# ATTENTION: DO NOT skip this step, even if your cuda version is not 11.8!
|
||||
conda install -y -c nvidia/label/cuda-11.8.0 cuda-runtime
|
||||
|
||||
# 2. Install the LLaMA-Factory environment
|
||||
@@ -121,10 +120,11 @@ git clone --depth 1 https://github.com/hiyouga/LLaMA-Factory.git
|
||||
cd LLaMA-Factory
|
||||
pip install -e ".[torch,metrics]" --no-build-isolation
|
||||
|
||||
# 3. Install the KTransformers wheel that matches your Torch and Python versions, from https://github.com/kvcache-ai/ktransformers/releases/tag/v0.4.1 (Note: The CUDA version can differ from that in the wheel filename.)
|
||||
pip install ktransformers-0.4.1+cu128torch27fancy-cp312-cp312-linux_x86_64.whl
|
||||
# 3. Install the KT release packages explicitly
|
||||
pip install ktransformers transformers-kt accelerate-kt
|
||||
|
||||
# 4. Install flash-attention, download the corresponding file based on your Python and Torch versions from: https://github.com/Dao-AILab/flash-attention/releases
|
||||
# 4. Install flash-attention, download the matching file from:
|
||||
# https://github.com/Dao-AILab/flash-attention/releases
|
||||
pip install flash_attn-2.8.3+cu12torch2.7cxx11abiTRUE-cp312-cp312-linux_x86_64.whl
|
||||
# abi=True/False can find from below
|
||||
# import torch
|
||||
@@ -135,6 +135,25 @@ git clone https://github.com/kvcache-ai/custom_flashinfer.git
|
||||
pip install custom_flashinfer/
|
||||
```
|
||||
|
||||
`ktransformers` is a lightweight meta-package. The beginner path does **not** rely on a runtime patch package; install `transformers-kt` and `accelerate-kt` explicitly.
|
||||
|
||||
#### Method 2: Source install
|
||||
|
||||
```shell
|
||||
git clone https://github.com/kvcache-ai/ktransformers.git
|
||||
cd ktransformers
|
||||
git submodule update --init --recursive
|
||||
|
||||
cd kt-kernel
|
||||
CPUINFER_PARALLEL=32 pip install .
|
||||
cd ..
|
||||
|
||||
pip install -e .
|
||||
pip install transformers-kt accelerate-kt
|
||||
```
|
||||
|
||||
If you need local source changes in the fork packages, install `transformers-kt` and `accelerate-kt` from their source trees instead of the released wheels.
|
||||
|
||||
**Usage tip:** In LLaMA-Factory YAML, set `use_kt: true` and pick a `kt_optimize_rule` file to have KTransformers handle the core compute. The features below show typical configs.
|
||||
|
||||
### Core Feature 1: Use KTransformers backend to fine-tune ultra-large MoE models
|
||||
|
||||
Reference in New Issue
Block a user