docs(release): add verified Kimi RAWINT4 LoRA smoke configs

2026-10-16 21:01:14 +08:00 · 2026-09-10 23:56:36 +08:00
parent 6d7e367c3f
commit ae9cc311c6
4 changed files with 161 additions and 0 deletions
@@ -94,6 +94,10 @@ No exact final package versions are chosen in this draft.

 ## Local verification

+The [Kimi K2.5 reference](examples/kimi-k25/README.md) includes the training and
+FSDP2 YAML from a successful candidate-stack smoke. It is independent evidence,
+not an E2E validation of this workflow's source-built wheels.
+
 ```bash
 python -m pytest -q .github/release/test_four_main.py
 python .github/release/four_main.py --help
@@ -0,0 +1,76 @@
+# Kimi K2.5: verified training reference
+
+Reference for the release hardware gate, not a validated public-PyPI recipe.
+The YAML matches the successful sap4 run on 2026-09-10; only four local paths
+were replaced. No runtime code, dependency pins or release workflow is changed.
+
+## Verified scope
+
+- 8 GPUs, CPU AMX / TP2 / 96 workers, RAWINT4 routed experts (group size 32).
+- 32 Neko examples, B1/GAS1, S512 **upper bound**, two optimizer steps.
+- Packing disabled; CPU and GPU activation recomputation enabled.
+- Loss: **2.26953125 → 2.0419921875**; gradient norm: **0.88868 → 0.84890**.
+- Final save: 610 ordinary LoRA tensors and 360 fused expert LoRA tensors,
+  all finite; all 305 ordinary and 180 expert LoRA B tensors nonzero.
+- Fresh-process SGLang reload on 4 GPUs: all 60 expert LoRA layers loaded;
+  all four TP ranks installed the MLA LoRA correction on 61 layers.
+  Three requests selecting `kimi-k25:release` returned nonempty answers with
+  `finish_reason=stop`, including `17 + 25` → `42`.
+
+This is a train/save/reload smoke, not a throughput or style-convergence
+result. `save_strategy: 'no'` disables intermediate checkpoints; LF still saves
+the final adapter. `save_only_model: true` does not support exact optimizer resume.
+The attention targets in the YAML are supplemented by fused routed-expert LoRA.
+Loading and successful generation do not prove a numerical LoRA effect: an
+enabled/disabled logits comparison was not run for this artifact.
+
+## Environment boundary
+
+The successful run used installed **candidate wheels**, without developer
+`PYTHONPATH` or edits to the original model directory. Third-party dependencies
+were shared with an existing environment. It does not validate the wheels built
+by this PR, a clean installation, or a portable multi-ISA KT wheel.
+
+| Component | Tested source base + candidate changes |
+| --- | --- |
+| KT | `63d06ff6f634bb0be5228d9c3051b23f2fbcd5b8`; previously validated AMX binary |
+| Transformers-KT | `96fa9fd336dfd088df4404e2529bf9be22e71309` + legacy Kimi checkpoint / MoonViT compatibility |
+| Accelerate-KT | Tree identical to main `df853a7da9f28a2b95ad2c8bfcc913e7171b4b10` |
+| LLaMA-Factory | `5226b026949a6f363f121a08a79927d694b37f0e` + `kimi_k25_nothink` template |
+| SGLang-KT (reload) | `3424f35d0e60f03b9e75d9b90941f8f4cb944524` + RAWINT4/composite LoRA, Kimi MLA and static-adapter warmup compatibility |
+
+The candidate changes are prerequisites, not implied by the base SHAs or package
+version numbers. They must land and be pinned before advertising a public-package
+recipe. In particular, this reference does **not** enable CPU retain / GPU
+recompute: the tested RAWINT4 implementation rejects that combination.
+The reload stack also packages KT's adapter converter as `kt-convert-lora`.
+These checks use Python 3.11 / Torch 2.9.1+cu128, independently of the release
+workflow's Python 3.12 build environment.
+
+## Run with the tested stack
+
+Set the model, expert-weight, dataset and new output paths in `train.yaml`.
+Use the original Kimi-K2.5 checkpoint with RAWINT4 routed experts. In `dataset_dir`,
+provide `neko_smoke.json` containing records with `prompt` and `answer`, and this
+`dataset_info.json`:
+
+```json
+{
+  "neko_smoke": {
+    "file_name": "neko_smoke.json",
+    "columns": {"prompt": "prompt", "response": "answer"}
+  }
+}
+```
+
+From this repository root:
+
+```bash
+USE_KT=1 accelerate launch \
+  --config_file .github/release/examples/kimi-k25/fsdp2_8gpu.yaml \
+  --main_process_port 29761 --no_python llamafactory-cli train \
+  .github/release/examples/kimi-k25/train.yaml
+```
+
+The saved directory must include `adapter_model.safetensors`,
+`fused_expert_lora.safetensors` and `kt_adapter_manifest.json`.
@@ -0,0 +1,18 @@
+compute_environment: LOCAL_MACHINE
+distributed_type: FSDP
+fsdp_config:
+  fsdp_auto_wrap_policy: TRANSFORMER_BASED_WRAP
+  fsdp_cpu_ram_efficient_loading: true
+  fsdp_offload_params: true
+  fsdp_reshard_after_forward: true
+  fsdp_state_dict_type: FULL_STATE_DICT
+  fsdp_sync_module_states: true
+  fsdp_transformer_layer_cls_to_wrap: DeepseekV3DecoderLayer
+  fsdp_version: 2
+machine_rank: 0
+mixed_precision: bf16
+num_machines: 1
+num_processes: 8
+rdzv_backend: static
+same_network: true
+use_cpu: false
@@ -0,0 +1,63 @@
+model_name_or_path: /path/to/Kimi-K2.5
+trust_remote_code: true
+use_fast_tokenizer: false
+flash_attn: disabled
+stage: sft
+do_train: true
+finetuning_type: lora
+lora_rank: 8
+lora_alpha: 16
+lora_dropout: 0.0
+lora_target: q_a_proj,q_b_proj,kv_a_proj_with_mqa,kv_b_proj,o_proj
+dataset_dir: /path/to/neko-smoke
+dataset: neko_smoke
+template: kimi_k25_nothink
+enable_thinking: false
+cutoff_len: 512
+packing: false
+neat_packing: false
+train_on_prompt: false
+preprocessing_num_workers: 4
+dataloader_num_workers: 0
+output_dir: /path/to/new-kimi-smoke-output
+logging_steps: 1
+save_steps: 5
+save_total_limit: 1
+save_only_model: true
+report_to: none
+overwrite_output_dir: false
+per_device_train_batch_size: 1
+gradient_accumulation_steps: 1
+learning_rate: 0.0001
+num_train_epochs: 1.0
+lr_scheduler_type: cosine
+warmup_steps: 0
+bf16: true
+max_grad_norm: 1.0
+disable_gradient_checkpointing: false
+use_reentrant_gc: false
+do_eval: false
+eval_strategy: 'no'
+eval_steps: 50
+per_device_eval_batch_size: 1
+prediction_loss_only: true
+seed: 42
+data_seed: 42
+ddp_timeout: 1800
+use_kt: true
+kt_weight_path: /path/to/Kimi-K2.5
+kt_cpu_activation: recompute
+kt_config:
+  kt_backend: RAWINT4
+  kt_expert_weight_format: rawint4
+  kt_weight_lifecycle: persistent
+  kt_num_threads: 96
+  kt_tp_enabled: true
+  kt_threadpool_count: 2
+  kt_num_gpu_experts: 0
+  kt_max_cache_depth: 2
+  kt_share_backward_bb: false
+  kt_force_fused_expert_lora: true
+  kt_model_max_length: 512
+max_steps: 2
+save_strategy: 'no'