GithubStar/ktransformers
1
0
Fork 0
mirror of https://github.com/kvcache-ai/ktransformers.git synced 2026-10-18 14:43:29 +08:00
Code Issues Packages Projects Releases Wiki Activity
1,358 Commits 73 Branches 44 Tags
main
Commit Graph

4 Commits

Author SHA1 Message Date
losewayy a02efe36b5 [feat](kt-kernel): add AVX2 RAWINT4 routed-expert LoRA SFT backend (#2243) 
* [feat](kt-kernel): add AVX2 RAWINT4 routed-expert LoRA SFT backend

* [fix](kt-kernel): fix RAWINT4 SFT wrapper and clamp gradients

Allow the RAWINT4 production wrapper on AVX2 runtimes and reject shared backward pools before construction. Preserve FP32 clamp decisions across BF16 caches, remove unused configuration state, and cover rectangular dimensions, multiblock dX, clamp boundaries, and production wrapper tasks with tighter LoRA gradient checks.

* [fix](kt-kernel): plumb swiglu_limit through the RAWINT4 SFT python path

The production wrapper hard-zeroed swiglu_limit for every format except MXFP4, and both SFT python gates rejected non-zero values for RAWINT4 even though MOESFTConfig carries the field and the AVX2 kernel implements the clamp in forward plus the gradient mask in backward. RAWINT4_SFT now accepts a finite non-negative swiglu_limit end to end, while the AMX K2 part (which computes plain SiLU) fails fast instead of silently dropping the clamp. The production-wrapper contract test is parametrized over swiglu_limit 0 and 2, a model-config plumb test covers plain and nested text_config shapes, the __main__ entry exercises both limits, and bench_rawint4_sft.py gains --swiglu-limit.
 2026-10-12 19:30:24 +08:00
yyj a5d7ad9047 feat(sft) Deepseek v4 flash LoRA (#2216) 
* feat(sft): add native MXFP4 routed-expert LoRA kernels

* test(sft): validate DeepSeek V4 MXFP4 routed experts

* feat(sft): AVX2 tier for MXFP4 routed-expert LoRA SFT

MXFP4_SFT_MOE on non-AVX512 builds, same Python contract as the AMX /
AVX512-BF16 kernel from #2185 (KTMoEWrapper(method="MXFP4_SFT")), for
AVX2+FMA+F16C hosts (Zen 2/3 EPYC, older Xeons).

* operators/avx2/sft_mxfp4-moe.hpp: AVX2_SFT_MXFP4_MOE_TP part on top of
  AVX2_MXFP4_MOE_TP (frozen packed E2M1/UE8M0 base, LoRA gate/up/down,
  V4 swiglu clamp, transpose-free packed-weight dX, authoritative LoRA grads).
* operators/moe-sft-tp.hpp: AVX2 result merge; AVX-512 merge loops and the
  block-FP8 staging guarded so the wrapper builds without AVX-512.
* ext_bindings.cpp: bind MXFP4_SFT_MOE -> TP_MOE_SFT<AVX2_SFT_MXFP4_MOE_TP>
  when USE_AMX_AVX_KERNEL is off; MOESFTBindings no longer AMX-only.
* python/sft/backend.py: runtime reports cpu_variant=avx2, kernel avx2-mxfp4-g32.
* test_mxfp4_sft_numerical.py: inference regression uses AVX2MXFP4_MOE when
  AMXFP4_KGroup_MOE is absent.

Validated on 2x EPYC 7452 (AVX2, no AVX-512): test_mxfp4_sft_numerical.py
TP1/TP2 all stages rel-L2 <= 0.4% (limits 3-5%); validate_dsv4_mxfp4_sft_layer.py
on DeepSeek-V4-Flash-0731 layer 3, TP1 and TP2: passed, forward/grad_input/
LoRA grads rel-L2 0.10-0.29%, base weights unchanged.

Note: the build uses -ffast-math, so exp() past the float range is undefined;
the scalar sigmoid tails clamp the argument like the vector path.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011Mgsduzz2fx1gSvfHbR8RZ

* fix(sft): clamp MXFP4 scalar-tail sigmoid exponent

* feat(sft): integrate native V4 experts with LoRA training and export

Wrap only routed experts while preserving V4 routing and shared branches. Validate native checkpoint provenance and stream a BF16 non-expert cache without dequantizing routed experts.

Add audited static SGLang export and reproducible training, resume and generation acceptance helpers. CPU retain/GPU recompute GAS1/GAS4 smoke and 265 KT regression tests pass; long-run quality and resume acceptance remain in progress.

* fix(sft): retain async inference buffers until CUDA work completes

* test(sft): isolate V4 adapter effects with matched-kernel controls

Allow baseline and non-expert-only exports to retain a zero-B expert adapter so all ablations execute the same native forward kernel. Add paired held-out generation reporting with explicitly diagnostic lexical metrics.

* test(sft): compare PEFT targets semantically during V4 resume audits

* test(sft): cancel queued V4 inference checks after a failed request

* test(sft): preserve raw V4 responses before finite-output validation

* feat: prepare V4 cache atomically and package static adapter export

* build: preserve accepted CUDA carriers across Python ABIs

* test: retain FP8 loading contracts after V4 wrapper integration

* test: exclude historical concatenated-sample smoke from delivery

* test(sft): audit unpacked batches and single-GPU resume state

* docs(sft): align April V4 recipe with approved unpacked delivery

* test: keep frozen-parameter audit indices exact for large tensors

* test: record the loaded native backend in acceptance evidence

* test: timestamp training audit events for startup measurements

* docs: pin V4 serving dependencies and account for reserved token slots

* [refactor](sft): keep V4 follow-up focused on runtime integration

---------

Co-authored-by: 05yuki <ceu.azul.samu@gmail.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
 2026-09-30 21:29:43 +08:00
yyj aa2cd3072f [feat] Add native RAWINT4 routed-expert LoRA SFT for Kimi K2.5/K2.6 (#2186) 
* [feat](sft): add K2 MoE SFT backend

* [feat](sft): enable packed KGroup backward for K2 SFT

* [feat](serving): add KT LoRA to SGLang adapter converter

* [test]: cover KT SFT and composite LoRA serving paths

* [perf](bench): add reusable KT E2E benchmark harness

* [test](sft): add K2 MoE AMX and real Kimi load smoke cases

* [perf](bench): add K2 SFT packed backward profile harness

* [test](sglang): cover Qwen3.5 LoRA shape filters

* [perf](sft): add K2 forward profiling logs

* [docs](sft): clarify K2 packed backward flow

* [docs](sft): number K2 backward comment steps

* chore: mirror sap4 loader baseline

* Add K2 SFT profiling ranges

* Add backward NVTX ranges for K2 SFT profiling

* perf(sft): add rank8 LoRA backward vector paths

* perf(sft): reuse down LoRA bprop cache

* [perf]: add K2 RAWINT4 prefill mat-mat dispatch

* [perf] Refactor K2 RAWINT4 prefill accumulation

* [perf]: add blocked RAWINT4 prefill backend

Cherry-picked from be40f8e with local amx.py conflict resolution.

* Optimize blocked RAWINT4 decode vecmul

* perf(sft): checkpoint qj5090 packed backward baseline

* perf(sft): batch expert LoRA backward as matmul

* perf(sft): align K2 base backward with BF16 matmul

* Optimize K2 SFT backward workspace path

* docs(sft): hand off RAWINT4 KGroup build and Kimi configs

* [feat](kt-kernel): add Kimi RAWINT4 LoRA SFT kernels

* [feat](kt-kernel): integrate Kimi RAWINT4 LoRA SFT

* [fix](sft): preserve input gradients through frozen routers

Build the routing graph when either activations or router parameters require gradients, while respecting the ambient grad mode. Keep frozen router parameters frozen and reject detached routing weights whenever gradients are needed.

Correct the existing frozen-router expectation and cover input-gradient equivalence for linear, DeepSeek/Kimi and GLM routing, checkpoint reuse, grad modes and detached-output failures.

Validated 62 routing/shared-expert/checkpoint tests on PyTorch 2.9.1 and a two-step Kimi K2.5 RAWINT4 four-GPU S512 LoRA smoke with per-rank routing-input gradient audits.

* [perf](kt-kernel): accelerate RAWINT4 prefill and backward

Reuse decoded weight columns across prefill rows and prepack BF16 backward coefficients into bounded per-worker panels. Use AMX BF16 tiles when available, retaining the AVX512-BF16 path and frozen base weights.

Balance expert work, parallelize cache copies and router reductions, and correct the signed INT4 -8 dot-product boundary. Add a CPU replay benchmark and forward/backward panel regressions.

Validated on sap4 with Kimi K2.5, TP1/TP2 replay, AMX and AVX-512 multi-GPU training, gradient windows, and saved LoRA artifacts. User-facing clean builds and AMD hardware remain separate validation work.

* [fix](sft): limit default OpenMP pools to the CPU owner

Keep the affinity-visible physical-core budget on rank 0 and default non-owner distributed ranks to one thread. Preserve explicit generic and KT-specific thread overrides, including intentional single-thread settings.

Cover single-process behavior, non-owner ranks, repeated configuration, and explicit overrides. The 109 configuration, activation-policy, and backend-dispatch tests pass.

---------

Co-authored-by: yyj <yyj@qj5090.local>
Co-authored-by: djh <jd4136@columbia.edu>
 2026-09-10 17:05:16 +08:00
yyj e80614fc2d [feat](kt-sft): Activation reuse & Int8 kernel refactor & native block-FP8 LoRA SFT (#2141) 
* feat(sft): support distributed activation reuse policies

* feat(sft): add frozen-base INT8 LoRA training

* fix(sft): make INT8 expert LoRA rank-zero authoritative

* fix(sft): preserve DeepSeek router LoRA routing

* feat(sft): enable persistent INT8 LoRA training

* perf(sft): accelerate INT8 VNNI with oneDNN BRGEMM

* perf(int8): fuse oneDNN compensation into backward repack

* [feat]: support BF16 expert LoRA training

* [fix]: honor forwarded activation policy in SFT workers

* feat(sft): add native block-FP8 routed expert LoRA

* feat(sft): expose explicit expert placeholder ownership

* fix(sft): publish fused adapter artifacts atomically

* feat(sft): own artifact and adapter lifecycle contracts

* fix(sft): harden artifact and rank-local contracts

* fix(sft): auto-adapt owner before adapter restore

* style(sft): keep lifecycle comments concise

* fix(sft): require fused adapter manifests

* test(sft): use spawn for distributed workers

* fix(sft): preserve runtime checkpoint metadata

* fix(sft): validate wrapped runtime configuration

* fix(sft): preserve expert format provenance

* fix(sft): own routed experts during device dispatch

* test(sft): lock explicit quantization conflict

* fix(cpu): make shared memory buffers lifetime-safe

* release: prepare v0.7.0
 2026-08-17 16:25:14 +08:00