From ab8ad0a110f511ba64e9540282f93e37ec403fb7 Mon Sep 17 00:00:00 2001 From: ZiWei Yuan Date: Sun, 16 Nov 2025 14:40:22 +0800 Subject: [PATCH] [docs]: update web doc (#1625) --- ...s-Fine-Tuning_Developer-Technical-Notes.md | 29 +++++-------------- .../KTransformers-Fine-Tuning_User-Guide.md | 22 ++++---------- doc/en/SFT/injection_tutorial.md | 20 ++++++++----- 3 files changed, 27 insertions(+), 44 deletions(-) diff --git a/doc/en/SFT/KTransformers-Fine-Tuning_Developer-Technical-Notes.md b/doc/en/SFT/KTransformers-Fine-Tuning_Developer-Technical-Notes.md index c1c6d750..7cf5f72a 100644 --- a/doc/en/SFT/KTransformers-Fine-Tuning_Developer-Technical-Notes.md +++ b/doc/en/SFT/KTransformers-Fine-Tuning_Developer-Technical-Notes.md @@ -1,26 +1,13 @@ -- [KTransformers Fine-Tuning × LLaMA-Factory Integration – Developer Technical Notes](#ktransformers-fine-tuning-x-llama-factory-integration-–-developer-technical-notes) - [Introduction](#introduction) - - [Overall View of the KT Fine-Tuning Framework](#overall-view-of-the-kt-fine-tuning-framework) - [Attention (LoRA + KT coexist)](#attention-lora--kt-coexist) - [MoE (operator encapsulation + backward)](#moe-operator-encapsulation--backward) - - [Encapsulation](#encapsulation) - - [Backward (CPU)](#backward-cpu) - [Multi-GPU Loading/Training: Placement strategy instead of DataParallel](#multi-gpu-loadingtraining-placement-strategy-instead-of-dataparallel) - - [KT-LoRA Fine-Tuning Evaluation](#kt-lora-fine-tuning-evaluation) - [Setup](#setup) - [Results](#results) - - [Stylized Dialogue (CatGirl tone)](#stylized-dialogue-catgirl-tone) - - [Translational-Style benchmark (generative)](#translational-style-benchmark-generative) - - [Medical Vertical Benchmark (AfriMed-SAQ/MCQ)](#medical-vertical-benchmark-afrimed-saqmcq) - - [Limitations](#limitations) - -- [Speed Tests](#speed-tests) - - [End-to-End Performance](#end-to-end-performance) - - [MoE Compute (DeepSeek-V3-671B)](#moe-compute-deepseek-v3-671b) + - [Speed Tests](#speed-tests) - [Memory Footprint](#memory-footprint) - - [Conclusion](#conclusion) @@ -36,7 +23,7 @@ This architecture bridges resource gaps, enabling **local fine-tuning of ultra-l Architecturally, LLaMA-Factory orchestrates data/config/training, LoRA insertion, and inference; KTransformers is a pluggable, high-performance operator backend that takes over Attention and MoE under the same training code, enabling **GPU+CPU heterogeneity** to accelerate training and reduce GPU memory. -![image-20251011010558909](../assets/image-20251011010558909.png) +![image-20251011010558909](../../assets/image-20251011010558909.png) We evaluated LoRA fine-tuning with HuggingFace default, Unsloth, and KTransformers backends (same settings and data). **KTransformers** is currently the only solution feasible on **2–4×24GB 4090s** for **671B-scale MoE**, and also shows higher throughput and lower GPU memory for 14B MoEs. @@ -51,7 +38,7 @@ We evaluated LoRA fine-tuning with HuggingFace default, Unsloth, and KTransforme From the table above, it can be seen that for the 14B model, the KTransformers backend achieves approximately 75% higher throughput than the default HuggingFace solution, while using only about one-fifth of the GPU memory. For the 671B model, both HuggingFace and Unsloth fail to run on a single 4090 GPU, whereas KTransformers is able to perform LoRA fine-tuning at 40 tokens/s, keeping the GPU memory usage within 70 GB. -![按照模型划分的对比图_02](../assets/image-compare_model.png) +![按照模型划分的对比图_02](../../assets/image-compare_model.png) @@ -68,11 +55,11 @@ KTransformers provides operator injection (`BaseInjectedModule`), and PEFT provi - **Inheritance:** `KTransformersLinearLora` retains KT’s high-performance paths (`prefill_linear`/`generate_linear`) while accepting LoRA parameters (`lora_A/lora_B`). - **Replacement:** During preparation, we replace original `KTransformersLinear` layers (Q/K/V/O) with `KTransformersLinearLora`, preserving KT optimizations while enabling LoRA trainability. -![image-20251016182810716](../assets/image-20251016182810716.png) +![image-20251016182810716](../../assets/image-20251016182810716.png) After replacement, LoRA is inserted at Q/K/V/O linear transforms (left), and `KTransformersLinearLora` contains both KT fast paths and LoRA matrices (right). -![image-20251016182920722](../assets/image-20251016182920722.png) +![image-20251016182920722](../../assets/image-20251016182920722.png) ### MoE (operator encapsulation + backward) @@ -83,13 +70,13 @@ Given large parameters and sparse compute, we encapsulate the expert computation - **Upstream (PyTorch graph):** we register a custom Autograd Function so the MoE layer appears as **a single node**. In the left figure (red box), only `KSFTExpertsCPU` is visible; on the right, the unencapsulated graph expands routing, dispatch, and FFN experts. Encapsulation makes the MoE layer behave like a standard `nn.Module` with gradients. - **Downstream (backend):** inside the Autograd Function, pybind11 calls C++ extensions for forward/backward. Multiple **pluggable backends** exist (AMX BF16/INT8; **llamafile**). The backend can be switched via YAML (e.g., `"backend": "AMXBF16"` vs. `"llamafile"`). -![image-20250801174623919](../assets/image-20250801174623919.png) +![image-20250801174623919](../../assets/image-20250801174623919.png) #### Backward (CPU) MoE backward frequently needs the transposed weights $W^\top$. To avoid repeated runtime transposes, we **precompute/cache** $W^\top$ at load time (blue box). We also **cache necessary intermediate activations** (e.g., expert projections, red box) to reuse in backward and reduce recomputation. We provide backward implementations for **llamafile** and **AMX (INT8/BF16)**, with NUMA-aware optimizations. -image-20251016182942726 +image-20251016182942726 ### Multi-GPU Loading/Training: Placement strategy instead of DataParallel @@ -117,7 +104,7 @@ LLaMA-Factory orchestration, KTransformers backend, LoRA (rank=8, α=32, dropout Dataset: [NekoQA-10K](https://zhuanlan.zhihu.com/p/1934983798233231689). The fine-tuned model consistently exhibits the target style (red boxes) versus neutral/rational base (blue). This shows **KT-LoRA injects style features** into the generation distribution with low GPU cost. -![image-20251016175848143](../assets/image-20251016175848143.png) +![image-20251016175848143](../../assets/image-20251016175848143.png) #### Translational-Style benchmark (generative) diff --git a/doc/en/SFT/KTransformers-Fine-Tuning_User-Guide.md b/doc/en/SFT/KTransformers-Fine-Tuning_User-Guide.md index 651f7c17..298e2aa3 100644 --- a/doc/en/SFT/KTransformers-Fine-Tuning_User-Guide.md +++ b/doc/en/SFT/KTransformers-Fine-Tuning_User-Guide.md @@ -1,23 +1,13 @@ -- [KTransformers Fine-Tuning × LLaMA-Factory Integration – User Guide](#ktransformers-fine-tuning-x-llama-factory-integration-–-user-guide) - [Introduction](#introduction) - -- [Fine-Tuning Results (Examples)](#fine-tuning-results-examples) - - [Stylized Dialogue (CatGirl tone)](#stylized-dialogue-catgirl-tone) - - [Benchmarks](#benchmarks) - - [Translational-Style dataset](#translational-style-dataset) - - [AfriMed-QA (short answer)](#afrimed-qa-short-answer) - - [AfriMed-QA (multiple choice)](#afrimed-qa-multiple-choice) - + - [Fine-Tuning Results (Examples)](#fine-tuning-results-examples) - [Quick to Start](#quick-to-start) - [Environment Setup](#environment-setup) - [Core Feature 1: Use KTransformers backend to fine-tune ultra-large MoE models](#core-feature-1-use-ktransformers-backend-to-fine-tune-ultra-large-moe-models) - [Core Feature 2: Chat with the fine-tuned model (base + LoRA adapter)](#core-feature-2-chat-with-the-fine-tuned-model-base--lora-adapter) - [Core Feature 3: Batch inference + metrics (base + LoRA adapter)](#core-feature-3-batch-inference--metrics-base--lora-adapter) - - [KT Fine-Tuning Speed (User-Side View)](#kt-fine-tuning-speed-user-side-view) - [End-to-End Performance](#end-to-end-performance) - [GPU/CPU Memory Footprint](#gpucpu-memory-footprint) - - [Conclusion](#conclusion) @@ -33,7 +23,7 @@ Our goal is to give resource-constrained researchers a **local path to explore f As shown below, LLaMA-Factory is the unified orchestration/configuration layer for the whole fine-tuning workflow—handling data, training scheduling, LoRA injection, and inference interfaces. **KTransformers** acts as a pluggable high-performance backend that takes over core operators like Attention/MoE under the same training configs, enabling efficient **GPU+CPU heterogeneous cooperation**. -![image-20251011010558909](../assets/image-20251011010558909.png) +![image-20251011010558909](../../assets/image-20251011010558909.png) Within LLaMA-Factory, we compared LoRA fine-tuning with **HuggingFace**, **Unsloth**, and **KTransformers** backends. KTransformers is the **only workable 4090-class solution** for ultra-large MoE models (e.g., 671B) and also delivers higher throughput and lower GPU memory on smaller MoE models (e.g., DeepSeek-14B). @@ -46,7 +36,7 @@ Within LLaMA-Factory, we compared LoRA fine-tuning with **HuggingFace**, **Unslo † **1400 GB** is a **theoretical** FP16 full-parameter resident footprint (not runnable). **70 GB** is the **measured peak** with KT strategy (Attention on GPU + layered MoE offload). -![按照模型划分的对比图_02](../assets/image-compare_model.png) +![按照模型划分的对比图_02](../../assets/image-compare_model.png) ### Fine-Tuning Results (Examples) @@ -56,7 +46,7 @@ Dataset: [NekoQA-10K](https://zhuanlan.zhihu.com/p/1934983798233231689). Goal: i The figure compares responses from the base vs. fine-tuned models. The fine-tuned model maintains the target tone and address terms more consistently (red boxes), validating the effectiveness of **style-transfer fine-tuning**. -![image-20251016175046882](../assets/image-20251016175046882.png) +![image-20251016175046882](../../assets/image-20251016175046882.png) #### Benchmarks @@ -219,7 +209,7 @@ We recommend **AMX acceleration** where available (`lscpu | grep amx`). AMX supp Outputs go to `output_dir` in safetensors format plus adapter metadata for later loading. -![image-20251016171537997](../assets/image-20251016171537997.png) +![image-20251016171537997](../../assets/image-20251016171537997.png) ### Core Feature 2: Chat with the fine-tuned model (base + LoRA adapter) @@ -244,7 +234,7 @@ We also support **GGUF** adapters: for safetensors, set the **directory**; for G During loading, LLaMA-Factory maps layer names to KT’s naming. You’ll see logs like `Loaded adapter weight: XXX -> XXX`: -![image-20251016171526210](../assets/image-20251016171526210.png) +![image-20251016171526210](../../assets/image-20251016171526210.png) ### Core Feature 3: Batch inference + metrics (base + LoRA adapter) diff --git a/doc/en/SFT/injection_tutorial.md b/doc/en/SFT/injection_tutorial.md index 45188366..c7f72a5e 100644 --- a/doc/en/SFT/injection_tutorial.md +++ b/doc/en/SFT/injection_tutorial.md @@ -4,10 +4,16 @@ ## TL;DR This tutorial will guide you through the process of injecting custom operators into a model using the KTransformers framework. We will use the DeepSeekV2-Chat model as an example to demonstrate how to inject custom operators into the model step by step. The tutorial will cover the following topics: -* [How to write injection rules](#how-to-write-injection-rules) - * [Understanding the structure of the model](#understanding-model-structure) -* [Multi-GPU](#muti-gpu) -* [How to write a new operator and inject it into the model](#how-to-write-a-new-operator-and-inject-into-the-model) +- [TL;DR](#tldr) +- [How to Write Injection Rules](#how-to-write-injection-rules) +- [Understanding Model Structure](#understanding-model-structure) +- [Matrix Absorption-based MLA Injection](#matrix-absorption-based-mla-injection) +- [Injection of Routed Experts](#injection-of-routed-experts) +- [Injection of Linear Layers](#injection-of-linear-layers) +- [Injection of Modules with Pre-calculated Buffers](#injection-of-modules-with-pre-calculated-buffers) +- [Specifying Running Devices for Modules](#specifying-running-devices-for-modules) +- [Muti-GPU](#muti-gpu) +- [How to Write a New Operator and Inject into the Model](#how-to-write-a-new-operator-and-inject-into-the-model) ## How to Write Injection Rules The basic form of the injection rules for the Inject framework is as follows: @@ -38,7 +44,7 @@ Using [deepseek-ai/DeepSeek-V2-Lite-Chat](https://huggingface.co/deepseek-ai/Dee Fortunately, knowing the structure of a model is very simple. Open the file list on the [deepseek-ai/DeepSeek-V2-Lite](https://huggingface.co/deepseek-ai/DeepSeek-V2-Lite-Chat/tree/main) homepage, and you can see the following files:

- Inject-Struction + Inject-Struction

@@ -48,7 +54,7 @@ From the `modeling_deepseek.py` file, we can see the specific implementation of The structure of the DeepSeekV2 model from the `.saftensors` and `modeling_deepseek.py` files is as follows:

- Inject-Struction + Inject-Struction

@@ -171,7 +177,7 @@ DeepseekV2-Chat got 60 layers, if we got 2 GPUs, we can allocate 30 layers to ea

- Inject-Struction + Inject-Struction