ci(release): gate four-main PyPI promotion on immutable wheels and model E2E

2026-10-16 21:01:14 +08:00 · 2026-09-11 00:26:36 +08:00
parent ae9cc311c6
commit 15d8f4927a
11 changed files with 1409 additions and 121 deletions
@@ -1,109 +1,115 @@
-# Four-main PyPI release preparation (draft)
+# 四仓 main 一键发布（Draft，尚未启用）

-This change prepares the next release while PRs continue landing in the four
-repositories. It does **not** choose tonight's version numbers or freeze tonight's
-main heads in a source file. `release-four-main.yml` resolves them when an operator
-starts a run.
+入口：**KTransformers → Actions → Release four-main stack → Run workflow**。
+选择 `main`，`target=candidate` 只构建和验收；`target=pypi` 才允许上传。
+合并 PR、修改版本号均不会自动上传；此 PR 不改变已经安装的 PyPI 包。

-## Implemented in this draft
+```text
+Run workflow
+  → 两次观测并锁定四仓 main SHA + 工作流 SHA
+  → 检查源码版本、依赖 pin、PyPI 版本是否已占用
+  → 新环境编译一次 → manylinux 修复 → 组装最终五个 wheels
+  → 锁定完整依赖 wheelhouse、两种 extras 的解析结果和 SHA256
+  → 调用 #2195：干净环境安装候选，完成三模型验收
+  → [仅 target=pypi] prod 环境：核验证据、按依赖顺序上传，KT 最后
+  → 调用 #2195：再次新建环境，从正式 PyPI 默认安装并完成三模型验收
+  → 核对版本、文件名和 SHA256，标记 released-and-verified
+```

-1. Read `main` from `kvcache-ai/ktransformers`, `sglang`, `transformers`, and
-   `accelerate`. Retry the complete snapshot if a merge races two consecutive
-   observations. GitHub offers no atomic cross-repository snapshot; a commit
-   arriving after the successful observations belongs to the next candidate.
-2. Record the four full SHAs and the workflow implementation SHA in
-   `source-lock.json`. Downstream jobs fetch only these SHAs, never resolve main
-   again. Full workflow reruns create a new snapshot/artifact; rerunning only a
-   failed build consumes its original snapshot.
-3. Create a fresh CPython 3.12 build environment and source checkouts. Build
-   Accelerate, Transformers, SGLang, KTransformers, KT-Kernel and the SGL CUDA
-   runtime from source. SGLang is independently checked out, rather than read
-   from KT's potentially stale `third_party/sglang` gitlink.
-4. Enable all KT CPU variants and CUDA `80;86;89;90;120`, and SGL's below-SM90
-   code generation plus FA3. These are build settings; architecture coverage is
-   not claimed verified until binary inspection and hardware tests are added.
-5. Record wheel hashes, package versions, dependency metadata, toolchain versions,
-   submodule states and source SHAs. Reject edits to tracked source during the
-   build, missing/duplicate distributions, mismatched wheel metadata, stale
-   intra-stack requirements, or direct conflicting upstream package dependencies
-   in the user `sglang`/`sft` extras. Transitive conflicts still require the
-   isolated dependency-resolution gate below.
+## 验收边界

-Python metadata is checked before native compilation so stale pins fail early.
-The full audit then verifies the actual native wheel versions. Partial build
-outputs and diagnostics are retained even on failure and never become a release.
+复用 #2195，不复制另一套模型测试。qj5090 有任务时排队，不杀其他进程。

-The raw SGL wheel is an intermediate build input, not a new public distribution
-to upload. A successful raw build remains `publishable: false` in the report.
-The workflow uses no PyPI credentials, has no publish job and has no automatic
-push/version trigger. It does not change the current `release-pypi.yml` workflow
-or any existing PyPI version.
+- Qwen3-30B-A3B：LoRA 一个 optimizer step、raw loss 有限、正常退出。
+- DeepSeek-V3.1：同上，每个 rank 都必须有证据。
+- GLM-5.3-Flash：服务 ready、实际 decode、问答语义正确。
+- 同时检查 `pip install "ktransformers[sglang]"` 和
+  `pip install "ktransformers[sglang,sft]"`，分别使用全新 venv。
+- 候选阶段从已核对的本地 wheelhouse 安装，不访问索引。发布后用默认的、
+  **不指定版本**的用户命令从正式 PyPI 安装，核对完整依赖解析结果、下载域名、
+  版本、wheel 文件名和 SHA256，拒绝回退旧版本。
+- 候选阶段不通过、缺一个测试、资源等待超时，均不能进入发布；上传成功但
+  正式 PyPI 复验失败，不能标记“发布验收成功”。

-## Before the first source build
+Kimi 的[实测配置参考](examples/kimi-k25/README.md)保留原样。它不是本流程
+构建 wheels 的验收证据，也未擅自加入当前约定的三模型必测集合。

- The workflow must be registered on the default branch before GitHub can
-  dispatch it. While this PR is a draft, run its CPU tests locally and review
-  the implementation; no GPU workflow has been dispatched by preparing the PR.
- Supply the `self-hosted, linux, x64, gpu, kt-cpu` runner, CUDA 12.8 and the
-  compiler/system prerequisites. `KT_RELEASE_CUDA_HOME` can override the CUDA
-  installation directory. Each run creates a fresh directory under `RUNNER_TEMP`.
- Align version and dependency declarations **in the respective main sources**
-  before calling a candidate consistent. The workflow never rewrites runtime
-  source or silently corrects stale pins. SGLang's existing
-  `SGLANG_KT_VERSION` build interface uses KT's source version.
- The present main sources may still contain incompatible old pins (KT's post1
-  version and Transformers post3 pin versus SGLang's KT post2/Transformers post4
-  requirements). The audit reports these as failures, not a releasable stack.
- Build directories are retained on the runner for diagnostics. The first
-  production iteration must add bounded cleanup/retention after uploads;
-  repeated native builds need substantial free disk space.
+## 源码、版本与产物

-## Required follow-up within this PR before enabling publication
+- 四仓为 `kvcache-ai/{ktransformers,sglang,transformers,accelerate}`，每次自动
+  读取 main，无需手填 SHA。快照后不再读取 main；验收期间的新合并进入下一次
+  候选。KT main 若不同于点击时的工作流 SHA，要求重新点击，避免新源码搭配旧
+  发布逻辑。
+- 所有版本和依赖来自仓库源码。**不是只更新 SHA 就能发版**：发布前须在各仓
+  main 选择未用版本，并对齐交叉依赖。不覆盖运行时代码、依赖或版本，不从旧
+  post2 wheels 补文件。
+- 首版统一构建五个最终发行包，不实现沿用旧 wheel。SGL payload 分布在 KT、
+  KT-Kernel、SGLang 和 Transformers 中，因此即使某仓 Python 逻辑未改，只要
+  载入的新 CUDA 片段变化，其发行版本也必须更新。
+- `carriers.py` 消费本次六个 raw wheels，保留 runtime 文件、许可证和 SM90
+  对象；仅生成 payload 清单/分片及 `WHEEL`、`RECORD`，不改 METADATA 依赖和
+  版本。SGL native wheel 是中间输入，不作为第六个公开发行包上传。
+- CPython 3.12、Linux x86-64、CUDA 12.8、Torch 2.9.1；全部 KT CPU variants，
+  CUDA `80;86;89;90;120`。auditwheel 修复到 manylinux_2_35 后检查 KT CUDA
+  扩展和 SGL common_ops 的 SASS。编译/静态检查不等于每种显卡实测。
+- 每个最终 wheel 必须小于 104 MB。超限、缺架构、ABI 不兼容直接失败；不删除
+  架构、不伪造 manylinux 标签。新版本 native 体积仍需真实构建确认。
+- 构建、诊断、验收、上传证据分别保留；候选 wheelhouse 保留 90 天。
+  GitHub artifact 并非永久存储，成功发布后建议另行归档用于长期追溯。

- **Fresh carrier assembly:** parameterize the already-main SGL carrier tools.
-  They currently hard-code post2 versions/CPython 3.12 and contain one-off
-  metadata replacements/source overlays. Assembly must consume only this run's
-  fresh raw wheels, preserve runtime source, record every generated metadata
-  change, verify CUDA binary architecture coverage, and enforce per-wheel size
-  limits. Never rebuild from the old frozen post2 carriers.
- **Version ownership:** select new, unused versions for changed artifacts. For
-  an unchanged dependency, reuse an existing PyPI wheel only after proving the
-  recorded main source and exact artifact hashes match. Never use
-  `--skip-existing` to conceal a different file with the same version.
- **Isolated wheelhouse:** resolve all dependencies, record their hashes and
-  prove `pip install "ktransformers[sglang]"` and the SFT extra select the expected
-  artifacts, without upstream `transformers`/`accelerate` namespace collisions.
-  Reconstruct installation environments from these artifacts, not source trees.
- **Hardware gates:** wire independently identified SM89 and SM120 runners,
-  isolated caches/venvs, model paths and JSON launch/test profiles. Test GLM
-  inference (including native multimodality), Qwen regression and the new Kimi
-  SFT path as agreed for the release. A run on the wrong architecture, import-only
-  check, missing model, or skipped E2E must not count as a passed hardware gate.
- **Artifact promotion:** depend on all required E2E jobs, download their exact
-  artifacts, recheck SHA256 and upload in dependency order with KT last. Upload
-  retries must reuse the same files. Retain the manifest/evidence long-term and
-  install from production PyPI for the final verification.
- **Cutover:** replace the old release workflow's trigger only after the
-  build-only candidate passes. Keep ordinary main merges separate from the
-  explicit release trigger. Do not merge the frozen-hotfix upload workflow over
-  the source build workflow.
+## 首次启用清单

-The initial runner audit found only `qj5090-runner-1` registered in this repository;
-the independent SM89 runner/profile is therefore an explicit remaining gate.
-No exact final package versions are chosen in this draft.
+**这些条件未完成前，不能宣称已经可以一键发版。**

-## Local verification
+1. 先合入 **#2195（含 release gate 接口）**，再合入 **#2194**。两份 PR 本身
+   不会自动合并或发布。#2194 的 CPU CI 在依赖尚未进入 main 时会明确失败，
+   不能忽略此依赖。
+2. 准备隔离的 `kt-model-e2e` runner、固定 digest 的 CUDA/Python 3.12 镜像、
+   公开 SFT 工具 wheelhouse、三个只读模型快照和可用下载通道。见
+   [模型验收 README](../model-e2e/README.md)。不能直接给带发布密钥的旧 runner
+   添加社区验收标签。
+3. 原生构建 runner 使用 `[self-hosted,linux,x64,gpu,kt-cpu]`，需要 CUDA 12.8、
+   C++ 工具链、足够空间和 manylinux_2_35 兼容的系统库。环境过新导致 ABI 不
+   兼容时应修复环境，不跳过 auditwheel。
+4. 配置 `KT_MODEL_E2E_ENABLED=true`、`KT_FOUR_MAIN_RELEASE_ENABLED=true`；
+   `KT_MODEL_E2E_HOST_CONFIG` / `KT_MODEL_E2E_PYTHON` 指向受管理配置及解释器。
+5. 在 `prod` environment 配置能上传五个项目的 `PYPI_API_TOKEN`，建议设置
+   required reviewers。密钥只进入 GitHub-hosted 的上传 step，不进入编译/GPU
+   容器。如果 prod 有审批，点击 Run workflow 后仍需批准该环境。
+6. 对齐主线版本及依赖后先运行 `target=candidate`，确认真实编译、组装、干净
+   安装和三模型均通过。再由维护者决定运行 `target=pypi`。后者构建自己的候选，
+   仅上传该次验收过的同一批 wheels。

-The [Kimi K2.5 reference](examples/kimi-k25/README.md) includes the training and
-FSDP2 YAML from a successful candidate-stack smoke. It is independent evidence,
-not an E2E validation of this workflow's source-built wheels.
+同时退休旧 `Release to PyPI` 和独立 `Release sglang-kt to PyPI` 入口：取消
+自动 push 触发，手动点击旧入口会提示迁移并失败，旧发布 jobs 不会执行，避免
+绕过新门禁。旧实现暂留文件中供审阅/追溯。
+
+## 失败与重试
+
+- 构建/候选验收失败：没有上传。修改源码后重新运行，得到新快照。
+- 上传部分失败：PyPI 跨项目不是事务，已上传依赖无法自动撤销；KT 最后上传
+  以减小影响。使用 **Re-run failed jobs**，保留原 raw-build 输出和 artifacts，
+  不重新编译；仅允许跳过文件名和 SHA256 完全一致的文件，禁止 `--skip-existing`。
+  内容不同则停止，不覆盖、不自动删包。
+- PyPI 复验失败：保留失败证据，不自动删除/yank 包，由维护者决定如何处理。
+- **Re-run all jobs** 是新候选，不保证得到相同字节；若旧版本已上传，版本预检
+  会拒绝构建，不能拿它代替“重试上传”。
+- 构建完成后只清理本次 mktemp 创建且带 run ID 标记的目录，不删除模型或旧
+  环境。GPU 容器由 #2195 清理，宿主其他任务不受影响。
+
+## 本地检查
+
+依赖 #2195 的文件已在同一源码树中时：

 ```bash
-python -m pytest -q .github/release/test_four_main.py
-python .github/release/four_main.py --help
+PYTHONPATH=.github/model-e2e python -m pytest -q .github/release
+python -m unittest discover -s .github/model-e2e -p 'test_*.py'
 actionlint -config-file .github/release/actionlint.yaml .github/workflows/release-four-main.yml
 ```

-The unit tests need `packaging` and `pytest`; they neither compile GPU kernels nor
-access the network. The source audit and artifact hashes are evidence of what was
-built, not substitutes for the pending installation and E2E tests.
+CPU 测试覆盖清单篡改、错误 run/attempt、缺少模型结果、同版本不同 wheel、
+部分上传、重试和目录清理；不会接触 PyPI 凭据，不能替代真实 CUDA 构建、Actions
+及模型 E2E 验收。
+
+实现参考：[GitHub reusable workflows](https://docs.github.com/en/actions/reference/workflows-and-actions/reusing-workflow-configurations)、
+[Python wheel/RECORD 规范](https://packaging.python.org/en/latest/specifications/binary-distribution-format/)。
@@ -0,0 +1,395 @@
+"""Assemble fresh five-project carriers without version changes or source overlays.
+
+Only WHEEL/RECORD and the generated CUDA payload manifest are rewritten. Runtime
+files (including SM90 and licenses) are retained. Reject oversized wheels instead
+of deleting architectures. Native inputs must first pass auditwheel repair.
+"""
+
+from __future__ import annotations
+
+import argparse
+import base64
+import csv
+import gzip
+import io
+import re
+import shutil
+import stat
+import subprocess
+import tarfile
+import tempfile
+import zipfile
+from pathlib import Path, PurePosixPath
+
+from four_main import inspect_wheel, save_json, sha256
+
+LIMIT = 104_000_000
+PLATFORM = "manylinux_2_35_x86_64"
+MODULES = {
+    "kt-kernel": "sgl_kernel_kt_payload_core",
+    "transformers-kt": "transformers_kt_sgl_kernel_payload",
+    "sglang-kt": "sglang_kt_sgl_kernel_payload",
+    "ktransformers": "ktransformers_sgl_kernel_payload",
+}
+LARGE = ("flash_ops.abi3.so", "sm100/common_ops.abi3.so")
+
+
+def require(condition, message):
+    if not condition:
+        raise ValueError(message)
+
+
+def unpack(wheel, root):
+    root.mkdir()
+    with zipfile.ZipFile(wheel) as archive:
+        names = archive.namelist()
+        require(len(names) == len(set(names)), "Duplicate ZIP entries")
+        records = [name for name in names if name.endswith(".dist-info/RECORD")]
+        require(len(records) == 1, "Require one wheel RECORD")
+        rows = list(csv.reader(io.StringIO(archive.read(records[0]).decode())))
+        record = {row[0]: row[1:] for row in rows}
+        require(len(record) == len(rows), "Duplicate RECORD entries")
+        for info in archive.infolist():
+            path = PurePosixPath(info.filename)
+            require(
+                not path.is_absolute()
+                and ".." not in path.parts
+                and "\\" not in info.filename,
+                "Unsafe wheel path",
+            )
+            require(not stat.S_ISLNK(info.external_attr >> 16), "Symlink in wheel")
+            if info.is_dir():
+                continue
+            require(info.filename in record, "Wheel file not recorded")
+            target = root / info.filename
+            require(
+                path.as_posix() == info.filename and not target.exists(),
+                "Noncanonical or colliding wheel path",
+            )
+            target.parent.mkdir(parents=True, exist_ok=True)
+            with archive.open(info) as source, target.open("wb") as output:
+                shutil.copyfileobj(source, output)
+            if info.filename != records[0]:
+                expected = (
+                    "sha256="
+                    + base64.urlsafe_b64encode(bytes.fromhex(sha256(target)))
+                    .rstrip(b"=")
+                    .decode()
+                )
+                require(
+                    record[info.filename] == [expected, str(target.stat().st_size)],
+                    "Wheel RECORD mismatch",
+                )
+        require(
+            set(record) == {name for name in names if not name.endswith("/")},
+            "Unexpected RECORD entries",
+        )
+
+
+def pack(root, output):
+    dist = next(root.glob("*.dist-info"))
+    record = dist / "RECORD"
+    rows = []
+    for path in sorted(root.rglob("*")):
+        if path.is_file() and path != record:
+            encoded = (
+                base64.urlsafe_b64encode(bytes.fromhex(sha256(path)))
+                .rstrip(b"=")
+                .decode()
+            )
+            rows.append(
+                (
+                    path.relative_to(root).as_posix(),
+                    "sha256=" + encoded,
+                    path.stat().st_size,
+                )
+            )
+    rows.append((record.relative_to(root).as_posix(), "", ""))
+    with record.open("w", newline="") as stream:
+        csv.writer(stream, lineterminator="\n").writerows(rows)
+    with zipfile.ZipFile(
+        output, "w", compression=zipfile.ZIP_DEFLATED, compresslevel=9
+    ) as archive:
+        for path in sorted(root.rglob("*")):
+            if path.is_file():
+                info = zipfile.ZipInfo(
+                    path.relative_to(root).as_posix(), (2020, 1, 1, 0, 0, 0)
+                )
+                info.compress_type = zipfile.ZIP_DEFLATED
+                info.external_attr = 0o100644 << 16
+                with (
+                    path.open("rb") as source,
+                    archive.open(info, "w", force_zip64=True) as target,
+                ):
+                    shutil.copyfileobj(source, target)
+
+
+def retag(root, python, abi):
+    metadata = next(root.glob("*.dist-info/WHEEL"))
+    lines = [
+        line
+        for line in metadata.read_text().splitlines()
+        if line and not line.startswith(("Tag:", "Root-Is-Purelib:"))
+    ]
+    metadata.write_text(
+        "\n".join(lines + ["Root-Is-Purelib: false", f"Tag: {python}-{abi}-{PLATFORM}"])
+        + "\n"
+    )
+
+
+def archive_payload(sgl, output):
+    hashes = {}
+    with (
+        output.open("wb") as raw,
+        gzip.GzipFile(
+            filename="", mode="wb", fileobj=raw, mtime=0, compresslevel=9
+        ) as compressed,
+    ):
+        with tarfile.open(fileobj=compressed, mode="w|") as archive:
+            for name in LARGE:
+                source = sgl / "sgl_kernel" / name
+                hashes[name] = sha256(source)
+                info = archive.gettarinfo(str(source), arcname=name)
+                info.uid = info.gid = info.mtime = 0
+                info.uname = info.gname = ""
+                with source.open("rb") as stream:
+                    archive.addfile(info, stream)
+    return hashes
+
+
+def binary_evidence(roots):
+    evidence = {}
+    for package, root in roots.items():
+        for path in sorted(root.rglob("*.so*")):
+            if not path.is_file():
+                continue
+            sections = subprocess.check_output(
+                ["readelf", "--wide", "--sections", str(path)], text=True
+            )
+            if ".nv_fatbin" not in sections and ".nvFatBinSegment" not in sections:
+                # Most KT CPU variants and repaired system libraries have no CUDA code.
+                continue
+            result = subprocess.run(
+                ["cuobjdump", "--list-elf", str(path)],
+                capture_output=True,
+                text=True,
+                check=True,
+            )
+            arches = sorted(set(re.findall(r"sm_([0-9]+[af]?)", result.stdout)))
+            evidence[package + "/" + path.relative_to(root).as_posix()] = {
+                "sha256": sha256(path),
+                "sass": arches,
+            }
+    common = evidence.get("sgl-kernel-kt/sgl_kernel/sm100/common_ops.abi3.so", {})
+    required = {80, 86, 89, 90, 120}
+    normalize = lambda values: {int(re.match(r"[0-9]+", value)[0]) for value in values}
+    require(
+        required <= normalize(common.get("sass", [])),
+        "SGL common_ops is missing required CUDA SASS architectures",
+    )
+    kt = [entry for name, entry in evidence.items() if name.startswith("kt-kernel/")]
+    require(
+        any(required <= normalize(entry["sass"]) for entry in kt),
+        "KT CUDA extension is missing required SASS architectures",
+    )
+    return evidence
+
+
+def assemble(raw, output, evidence_dir):
+    output.mkdir(exist_ok=False)
+    entries = [
+        inspect_wheel(path) | {"path": path} for path in sorted(raw.glob("*.whl"))
+    ]
+    by_name = {entry["name"]: entry for entry in entries}
+    expected = set(MODULES) | {"accelerate-kt", "sgl-kernel-kt"}
+    require(
+        set(by_name) == expected and len(entries) == len(expected),
+        "Need six fresh inputs",
+    )
+    for package in ("kt-kernel", "sgl-kernel-kt"):
+        require(
+            any(PLATFORM in tag for tag in by_name[package]["tags"]),
+            "Native input must be auditwheel-repaired for " + PLATFORM,
+        )
+    require(
+        not by_name["sgl-kernel-kt"]["requires_dist"],
+        "SGL native dependencies must be explicitly carried by main package metadata",
+    )
+    with tempfile.TemporaryDirectory(
+        prefix="carrier-", dir=evidence_dir.parent
+    ) as temp:
+        temp = Path(temp)
+        roots = {name: temp / name for name in by_name}
+        for name, entry in by_name.items():
+            unpack(entry["path"], roots[name])
+        save_json(evidence_dir / "cuda-binaries.json", binary_evidence(roots))
+        sgl = roots["sgl-kernel-kt"]
+        # Lazy objects are extracted into a cache, not site-packages. An
+        # auditwheel-renamed dependency resolved via $ORIGIN would break there.
+        # Torch/CUDA-runtime libraries are supplied by the pinned torch wheel;
+        # reject any other bundled dependency rather than emitting a broken wheel.
+        bundled = {
+            path.name
+            for path in sgl.rglob("*.so*")
+            if any(part.endswith(".libs") for part in path.parts)
+        }
+        for name in LARGE:
+            dynamic = subprocess.check_output(
+                ["readelf", "--dynamic", str(sgl / "sgl_kernel" / name)], text=True
+            )
+            needed = set(re.findall(r"Shared library: \[([^]]+)\]", dynamic))
+            require(
+                not (needed & bundled),
+                "Lazy payload depends on a relocated auditwheel library; fix the main build linkage",
+            )
+        for name in ("payload_runtime.py", "load_utils.py", "flash_attn.py"):
+            require(
+                (sgl / "sgl_kernel" / name).is_file(),
+                "Locked SGL main lacks the carrier loader: " + name,
+            )
+        archive = temp / "payload.tar.gz"
+        hashes = archive_payload(sgl, archive)
+        # Only the two objects supported by main's lazy loader are externalized.
+        # No SM90 deletion or copied Python implementation from another checkout.
+        for name in LARGE:
+            (sgl / "sgl_kernel" / name).unlink()
+        manifest = sgl / "sgl_kernel/_payload_manifest.py"
+        require(
+            not manifest.exists(), "Raw SGL input already contains a payload manifest"
+        )
+        manifest.write_text(
+            f"VERSION = {by_name['sgl-kernel-kt']['version']!r}\n"
+            f"ARCHIVE_SHA256 = {sha256(archive)!r}\n"
+            f"PAYLOAD_MODULES = {tuple(MODULES.values())!r}\nFILES = {hashes!r}\n"
+        )
+        kt = roots["kt-kernel"]
+        kt_dist = next(kt.glob("*.dist-info"))
+        # Keep Hopper's directly loaded object, but use the lightweight umbrella
+        # carrier's size budget. Pip installs these non-overlapping paths into the
+        # same sgl_kernel directory; no loader changes or namespace .pth hacks.
+        direct_files = []
+        sm90 = sgl / "sgl_kernel/sm90"
+        if sm90.exists():
+            direct_files = [
+                path.relative_to(sgl).as_posix()
+                for path in sm90.rglob("*")
+                if path.is_file()
+            ]
+            destination = roots["ktransformers"] / "sgl_kernel/sm90"
+            require(not destination.exists(), "SM90 carrier path collision")
+            destination.parent.mkdir(parents=True, exist_ok=True)
+            shutil.move(str(sm90), destination)
+        for path in sorted(sgl.rglob("*")):
+            if not path.is_file():
+                continue
+            relative = path.relative_to(sgl)
+            if relative.parts[0].endswith(".dist-info"):
+                # Preserve the raw native distribution's licenses/metadata as
+                # provenance, without creating a conflicting installed package.
+                target = kt_dist / "sgl-native-origin" / Path(*relative.parts[1:])
+            else:
+                require(
+                    not relative.parts[0].endswith(".data"),
+                    "Unsupported SGL wheel .data layout",
+                )
+                target = kt / relative
+            require(not target.exists(), "Carrier path collision: " + str(relative))
+            target.parent.mkdir(parents=True, exist_ok=True)
+            shutil.copyfile(path, target)
+        filenames = {}
+        capacities = {}
+        for name in MODULES:
+            root = roots[name]
+            python, abi = ("cp312", "cp312") if name == "kt-kernel" else ("py3", "none")
+            if name == "kt-kernel":
+                require(
+                    any(
+                        tag.startswith("cp312-cp312-") for tag in by_name[name]["tags"]
+                    ),
+                    "Initial release supports CPython 3.12 only",
+                )
+            retag(root, python, abi)
+            filename = f"{name.replace('-', '_')}-{by_name[name]['version']}-{python}-{abi}-{PLATFORM}.whl"
+            filenames[name] = filename
+            baseline = temp / filename
+            pack(root, baseline)
+            capacities[name] = LIMIT - baseline.stat().st_size - 1_000_000
+            require(
+                capacities[name] > 0, "Base wheel exceeds carrier capacity: " + name
+            )
+        require(
+            sum(capacities.values()) >= archive.stat().st_size,
+            "CUDA archive exceeds five-project size budget; do not drop architectures",
+        )
+        parts = {}
+        remaining = archive.stat().st_size
+        with archive.open("rb") as source:
+            for index, (name, module) in enumerate(MODULES.items()):
+                # Keep a nonempty piece for every loader module, even for tiny fixtures.
+                size = min(capacities[name], remaining - (len(MODULES) - index - 1))
+                require(size > 0, "Invalid payload partition")
+                directory = roots[name] / module
+                directory.mkdir()
+                (directory / "__init__.py").write_text(
+                    "# Generated checksummed CUDA payload carrier.\n"
+                )
+                part = directory / "payload.part"
+                with part.open("wb") as target:
+                    pending = size
+                    while pending:
+                        block = source.read(min(pending, 8 * 1024 * 1024))
+                        require(block, "Truncated payload archive")
+                        target.write(block)
+                        pending -= len(block)
+                remaining -= size
+                parts[name] = {"bytes": size, "sha256": sha256(part)}
+                final = output / filenames[name]
+                pack(roots[name], final)
+                require(
+                    final.stat().st_size < LIMIT,
+                    "Final wheel exceeds PyPI size limit: " + name,
+                )
+                require(
+                    inspect_wheel(final)["requires_dist"]
+                    == by_name[name]["requires_dist"],
+                    "Carrier changed dependency metadata",
+                )
+            require(remaining == 0 and not source.read(1), "Incomplete payload split")
+        accelerate = by_name["accelerate-kt"]["path"]
+        require(
+            accelerate.stat().st_size < LIMIT,
+            "Accelerate wheel exceeds PyPI size limit",
+        )
+        shutil.copyfile(accelerate, output / accelerate.name)
+        save_json(
+            evidence_dir / "assembly.json",
+            {
+                "raw_wheels": [
+                    {k: v for k, v in entry.items() if k != "path"} for entry in entries
+                ],
+                "final_wheels": [
+                    inspect_wheel(path) for path in sorted(output.glob("*.whl"))
+                ],
+                "archive_sha256": sha256(archive),
+                "payload_files": hashes,
+                "parts": parts,
+                "direct_native_carriers": {"ktransformers": direct_files},
+                "runtime_overlays": [],
+                "version_overrides": [],
+                "dependency_overrides": [],
+            },
+        )
+
+
+def main():
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--raw", type=Path, required=True)
+    parser.add_argument("--output", type=Path, required=True)
+    parser.add_argument("--evidence", type=Path, required=True)
+    args = parser.parse_args()
+    args.evidence.mkdir(parents=True, exist_ok=True)
+    assemble(args.raw, args.output, args.evidence)
+
+
+if __name__ == "__main__":
+    main()
@@ -0,0 +1,28 @@
+"""Standard-library-only cleanup, including when build dependency installation fails."""
+
+import argparse
+import os
+import shutil
+from pathlib import Path
+
+
+def cleanup(directory, parent):
+    directory, parent = Path(directory), Path(parent).resolve()
+    if directory.is_symlink():
+        raise ValueError("Refuse symlink cleanup")
+    resolved = directory.resolve(strict=True)
+    if resolved.parent != parent or not resolved.name.startswith("kt-four-main."):
+        raise ValueError("Refuse broad cleanup")
+    if (resolved / ".kt-release-owned").read_text().strip() != os.environ[
+        "GITHUB_RUN_ID"
+    ]:
+        raise ValueError("Build directory is not owned by this run")
+    shutil.rmtree(resolved)
+
+
+if __name__ == "__main__":
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--directory", type=Path, required=True)
+    parser.add_argument("--parent", type=Path, required=True)
+    args = parser.parse_args()
+    cleanup(args.directory, args.parent)
@@ -2,7 +2,7 @@
 """Freeze four public main heads and audit source-built release artifacts.

 This module never uploads packages or changes package versions/dependencies.
-The first workflow iteration deliberately produces raw, non-publishable wheels.
+Raw wheels remain non-publishable; release_stack.py gates final promotion.
 """

 from __future__ import annotations
@@ -21,7 +21,6 @@ from pathlib import Path, PurePosixPath
 from packaging.requirements import Requirement
 from packaging.utils import canonicalize_name, parse_wheel_filename

-
 REPOSITORIES = {
    "ktransformers": "kvcache-ai/ktransformers",
    "sglang": "kvcache-ai/sglang",
@@ -306,7 +305,7 @@ def audit(
        "pending_gates": [
            "Assemble fresh SGL CUDA payload into version-parameterized carrier wheels",
            "Resolve complete dependencies into an isolated, hashed wheelhouse",
-            "Install the final wheels in clean SM89 and SM120 environments and run E2E",
+            "Install the final wheels in the isolated qj5090 runner and run the three-model E2E suite",
            "Promote the exact validated wheel hashes and verify installation from PyPI",
        ],
    }
@@ -0,0 +1,348 @@
+"""Lock dependencies, check E2E evidence, and promote exactly the tested wheels.
+
+The publish command is called only on a GitHub-hosted job with protected PyPI
+credentials, never on the native build or community/model-test runner.
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import subprocess
+import sys
+import time
+import urllib.error
+import urllib.request
+from pathlib import Path
+
+from cleanup import cleanup
+from four_main import inspect_wheel, read_lock, save_json, sha256
+from packaging.version import Version
+
+# #2195 owns the acceptance contract. Merge it before enabling this workflow.
+sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "model-e2e"))
+from contracts import CASES, PACKAGES, require, suite_passed  # noqa: E402
+from release_contracts import verify_install_report, verify_release  # noqa: E402
+
+ORDER = ("accelerate-kt", "transformers-kt", "kt-kernel", "sglang-kt", "ktransformers")
+
+
+def pypi_files(name, version):
+    request = urllib.request.Request(
+        f"https://pypi.org/pypi/{name}/{version}/json",
+        headers={"Cache-Control": "no-cache"},
+    )
+    try:
+        with urllib.request.urlopen(request, timeout=60) as response:
+            return json.load(response)["urls"]
+    except urllib.error.HTTPError as error:
+        if error.code == 404:
+            return []
+        raise
+
+
+def check_unused(wheels, query=pypi_files):
+    for wheel in wheels:
+        version = Version(wheel["version"])
+        require(
+            not version.is_prerelease and not version.local,
+            "This release entry point requires stable source versions",
+        )
+        require(
+            not query(wheel["name"], wheel["version"]),
+            f"{wheel['name']}=={wheel['version']} already exists on PyPI; commit new source versions/pins before rebuilding",
+        )
+
+
+def pip(*args):
+    subprocess.run(
+        [sys.executable, "-m", "pip", "--isolated", *map(str, args)],
+        check=True,
+        timeout=7200,
+    )
+
+
+def make_release(final, root, lock_path, evidence):
+    root.mkdir(exist_ok=False)
+    house = root / "wheelhouse"
+    house.mkdir()
+    wheels = [inspect_wheel(path) for path in sorted(final.glob("*.whl"))]
+    require(
+        {item["name"] for item in wheels} == PACKAGES and len(wheels) == len(PACKAGES),
+        "Need five final wheels",
+    )
+    check_unused(wheels)
+    # Resolve user-facing extras without direct wheel URLs or --no-deps. Fresh
+    # versions prevent PyPI from silently substituting an older same-version file.
+    pip(
+        "download",
+        "--index-url",
+        "https://pypi.org/simple",
+        "--no-cache-dir",
+        "--only-binary=:all:",
+        "--find-links",
+        final,
+        "--dest",
+        house,
+        "ktransformers[sglang,sft]",
+    )
+    house_entries = [inspect_wheel(path) for path in sorted(house.glob("*.whl"))]
+    require(
+        len({entry["name"] for entry in house_entries}) == len(house_entries),
+        "Duplicate dependency version",
+    )
+    require(
+        all(
+            entry["name"]
+            not in {
+                "transformers",
+                "accelerate",
+                "sglang",
+                "sgl-kernel",
+                "sgl-kernel-kt",
+            }
+            for entry in house_entries
+        ),
+        "Conflicting upstream namespace in dependency closure",
+    )
+    for wheel in wheels:
+        require(
+            (house / wheel["filename"]).is_file()
+            and sha256(house / wheel["filename"]) == wheel["sha256"],
+            "Resolver did not select this run's final wheel: " + wheel["name"],
+        )
+    lock = read_lock(lock_path)
+    manifest = {
+        "schema": 1,
+        "run_id": int(os.environ["GITHUB_RUN_ID"]),
+        "run_attempt": int(os.environ["GITHUB_RUN_ATTEMPT"]),
+        "workflow_sha": lock["workflow_sha"],
+        "source_lock": lock,
+        "wheels": {
+            entry["name"]: {
+                "file": entry["filename"],
+                "version": entry["version"],
+                "sha256": entry["sha256"],
+            }
+            for entry in wheels
+        },
+        "wheelhouse": {
+            entry["filename"]: {
+                "name": entry["name"],
+                "version": entry["version"],
+                "sha256": entry["sha256"],
+            }
+            for entry in house_entries
+        },
+        "plans": {},
+    }
+    for extra in ("sglang", "sglang,sft"):
+        report = evidence / ("resolution-" + extra.replace(",", "-") + ".json")
+        pip(
+            "install",
+            "--dry-run",
+            "--ignore-installed",
+            "--no-cache-dir",
+            "--only-binary=:all:",
+            "--no-index",
+            "--find-links",
+            house,
+            "--report",
+            report,
+            f"ktransformers[{extra}]",
+        )
+        selected = {}
+        for item in json.loads(report.read_text())["install"]:
+            name = item["metadata"]["name"].lower().replace("_", "-")
+            matches = [
+                filename
+                for filename, entry in manifest["wheelhouse"].items()
+                if entry["name"] == name
+            ]
+            require(len(matches) == 1, "Unexpected dependency resolution")
+            selected[name] = matches[0]
+        manifest["plans"][extra] = selected
+        verify_install_report(
+            json.loads(report.read_text()), manifest, extra, public=False
+        )
+    save_json(root / "release.json", manifest)
+    digest = sha256(root / "release.json")
+    verify_release(root, digest)
+    with Path(os.environ["GITHUB_OUTPUT"]).open("a") as stream:
+        stream.write(
+            f"manifest_sha256={digest}\nbuild_attempt={manifest['run_attempt']}\n"
+        )
+
+
+def check_evidence(directory, manifest, manifest_digest, stage):
+    directory = Path(directory)
+    request = json.loads((directory / "request.json").read_text())
+    require(request["mode"] == "release-" + stage, "Wrong acceptance stage")
+    require(
+        request["manifest_sha256"] == manifest_digest,
+        "Acceptance tested a different candidate",
+    )
+    require(
+        request["build_run_id"] == manifest["run_id"],
+        "Acceptance belongs to another run",
+    )
+    require(
+        request["build_run_attempt"] == manifest["run_attempt"],
+        "Acceptance belongs to another build attempt",
+    )
+    require(
+        request["build_workflow_sha"]
+        == manifest["workflow_sha"]
+        == request["harness_sha"],
+        "Wrong acceptance harness",
+    )
+    for path in (directory / "host-result.json", directory / "tests/suite.json"):
+        result = json.loads(path.read_text())
+        require(
+            result["status"] == "passed" and suite_passed(result["cases"]),
+            "Incomplete/failed model acceptance",
+        )
+    for extra, label in (("sglang", "serving"), ("sglang,sft", "combined")):
+        report = json.loads(
+            (directory / f"tests/release-{label}-install.json").read_text()
+        )
+        verify_install_report(report, manifest, extra, public=stage == "pypi")
+
+
+def existing_matches(entry, files):
+    if not files:
+        return False
+    require(
+        len(files) == 1,
+        "Published version contains unexpected files; refuse to mix artifacts",
+    )
+    published = files[0]
+    require(not published.get("yanked", False), "Published candidate was yanked")
+    require(
+        published["filename"] == entry["file"]
+        and published["digests"]["sha256"] == entry["sha256"],
+        "PyPI version already contains different bytes; never skip-existing",
+    )
+    return True
+
+
+def promote(root, digest, evidence, output, query=pypi_files, upload=None):
+    manifest = verify_release(root, digest)
+    check_evidence(evidence, manifest, digest, "candidate")
+    require(os.environ.get("GITHUB_REF") == "refs/heads/main", "Publish requires main")
+    require(
+        os.environ.get("GITHUB_EVENT_NAME") == "workflow_dispatch",
+        "Publish requires manual dispatch",
+    )
+    require(
+        os.environ.get("GITHUB_REPOSITORY") == "kvcache-ai/ktransformers",
+        "Publish requires official repository",
+    )
+    require(
+        int(os.environ["GITHUB_RUN_ID"]) == manifest["run_id"],
+        "Publish must use the same workflow run",
+    )
+    require(
+        os.environ["GITHUB_SHA"] == manifest["workflow_sha"], "Publish workflow changed"
+    )
+    if upload is None:
+
+        def upload(path):
+            subprocess.run(
+                [
+                    sys.executable,
+                    "-m",
+                    "twine",
+                    "upload",
+                    "--non-interactive",
+                    "--disable-progress-bar",
+                    "--repository-url",
+                    "https://upload.pypi.org/legacy/",
+                    str(path),
+                ],
+                check=True,
+                timeout=900,
+            )
+
+    # Preflight the ENTIRE batch before any write. On a retry, allow only exact
+    # filename+SHA256 matches from this candidate, never --skip-existing.
+    for name in ORDER:
+        entry = manifest["wheels"][name]
+        existing_matches(entry, query(name, entry["version"]))
+    result = {"manifest_sha256": digest, "status": "uploading", "packages": {}}
+    try:
+        for name in ORDER:
+            entry = manifest["wheels"][name]
+            path = Path(root) / "wheelhouse" / entry["file"]
+            require(sha256(path) == entry["sha256"], "Wheel changed before upload")
+            present = existing_matches(entry, query(name, entry["version"]))
+            if not present:
+                # A failed job may be rerun against the retained candidate. Do
+                # not rebuild or silently continue after an upload error.
+                upload(path)
+            deadline = time.monotonic() + 600
+            while not existing_matches(entry, query(name, entry["version"])):
+                require(
+                    time.monotonic() < deadline,
+                    "PyPI visibility timeout; rerun failed jobs with the same artifact",
+                )
+                time.sleep(15)
+            result["packages"][name] = entry | {"already_present": present}
+            save_json(output, result)
+        result["status"] = "uploaded-awaiting-public-e2e"
+    except BaseException:
+        result["status"] = "partial-or-failed-upload"
+        raise
+    finally:
+        save_json(output, result)
+
+
+def main():
+    parser = argparse.ArgumentParser(description=__doc__)
+    sub = parser.add_subparsers(dest="command", required=True)
+    preflight = sub.add_parser("unused")
+    preflight.add_argument("--wheels", type=Path, required=True)
+    make = sub.add_parser("prepare")
+    make.add_argument("--final", type=Path, required=True)
+    make.add_argument("--root", type=Path, required=True)
+    make.add_argument("--lock", type=Path, required=True)
+    make.add_argument("--evidence", type=Path, required=True)
+    for name in ("publish", "verify"):
+        action = sub.add_parser(name)
+        action.add_argument("--root", type=Path, required=True)
+        action.add_argument("--digest", required=True)
+        action.add_argument("--evidence", type=Path, required=True)
+        action.add_argument("--output", type=Path, required=True)
+    clean = sub.add_parser("cleanup")
+    clean.add_argument("--directory", type=Path, required=True)
+    clean.add_argument("--parent", type=Path, required=True)
+    args = parser.parse_args()
+    if args.command == "unused":
+        wheels = [inspect_wheel(path) for path in args.wheels.glob("*.whl")]
+        if not any(wheel["name"] == "kt-kernel" for wheel in wheels):
+            kt = next(wheel for wheel in wheels if wheel["name"] == "ktransformers")
+            wheels.append({"name": "kt-kernel", "version": kt["version"]})
+        check_unused(wheels)
+    elif args.command == "prepare":
+        make_release(args.final, args.root, args.lock, args.evidence)
+    elif args.command == "publish":
+        promote(args.root, args.digest, args.evidence, args.output)
+    elif args.command == "verify":
+        manifest = verify_release(args.root, args.digest)
+        check_evidence(args.evidence, manifest, args.digest, "pypi")
+        save_json(
+            args.output,
+            {
+                "status": "released-and-verified",
+                "manifest_sha256": args.digest,
+                "run_id": manifest["run_id"],
+                "cases": list(CASES),
+            },
+        )
+    else:
+        cleanup(args.directory, args.parent)
+
+
+if __name__ == "__main__":
+    main()
@@ -0,0 +1,98 @@
+"""Exercise carrier assembly with tiny synthetic wheels, not CUDA execution."""
+
+import sys
+import zipfile
+from pathlib import Path
+
+import pytest
+
+sys.path.insert(0, str(Path(__file__).resolve().parent))
+import carriers
+from four_main import inspect_wheel
+
+
+def raw_wheels(tmp_path):
+    raw = tmp_path / "raw"
+    raw.mkdir()
+    for package in (*carriers.MODULES, "accelerate-kt", "sgl-kernel-kt"):
+        root = tmp_path / package
+        root.mkdir()
+        dist = root / (package.replace("-", "_") + "-1.0.dist-info")
+        dist.mkdir()
+        (dist / "METADATA").write_text(
+            f"Metadata-Version: 2.1\nName: {package}\nVersion: 1.0\n"
+        )
+        tag = (
+            "cp312-cp312-" + carriers.PLATFORM
+            if package in ("kt-kernel", "sgl-kernel-kt")
+            else "py3-none-any"
+        )
+        (dist / "WHEEL").write_text(
+            f"Wheel-Version: 1.0\nRoot-Is-Purelib: false\nTag: {tag}\n"
+        )
+        module = root / package.replace("-", "_")
+        module.mkdir()
+        (module / "__init__.py").write_text("VALUE = 1\n")
+        if package == "sgl-kernel-kt":
+            module = root / "sgl_kernel"
+            module.mkdir()
+            for name in (
+                *carriers.LARGE,
+                "sm90/common_ops.abi3.so",
+                "payload_runtime.py",
+                "flash_attn.py",
+                "load_utils.py",
+            ):
+                path = module / name
+                path.parent.mkdir(parents=True, exist_ok=True)
+                path.write_bytes(b"fixture-data" * 20)
+            (dist / "LICENSE").write_text("SGL license fixture")
+        path = raw / f"{package.replace('-', '_')}-1.0-{tag}.whl"
+        carriers.pack(root, path)
+    return raw
+
+
+def test_fresh_carriers_preserve_runtime_versions_and_sm90(tmp_path, monkeypatch):
+    raw = raw_wheels(tmp_path)
+    monkeypatch.setattr(carriers, "binary_evidence", lambda roots: {"fixture": True})
+    monkeypatch.setattr(carriers.subprocess, "check_output", lambda *args, **kwargs: "")
+    evidence = tmp_path / "evidence"
+    evidence.mkdir()
+    output = tmp_path / "final"
+    carriers.assemble(raw, output, evidence)
+    entries = [inspect_wheel(path) for path in output.iterdir()]
+    assert len(entries) == 5
+    assert {entry["version"] for entry in entries} == {"1.0"}
+    kt = next(output.glob("kt_kernel-*.whl"))
+    with zipfile.ZipFile(kt) as wheel:
+        assert wheel.read("sgl_kernel/payload_runtime.py") == b"fixture-data" * 20
+        assert any(
+            name.endswith("sgl-native-origin/LICENSE") for name in wheel.namelist()
+        )
+        assert "sgl_kernel/flash_ops.abi3.so" not in wheel.namelist()
+        assert "sgl_kernel/_payload_manifest.py" in wheel.namelist()
+    with zipfile.ZipFile(next(output.glob("ktransformers-*.whl"))) as wheel:
+        assert "sgl_kernel/sm90/common_ops.abi3.so" in wheel.namelist()
+    # Every final RECORD is independently checked, including regenerated payloads.
+    for index, path in enumerate(output.iterdir()):
+        carriers.unpack(path, tmp_path / f"verify-{index}")
+
+
+def test_tampered_raw_wheel_is_rejected(tmp_path):
+    raw = raw_wheels(tmp_path)
+    path = next(raw.glob("accelerate*.whl"))
+    with zipfile.ZipFile(path, "a") as wheel:
+        wheel.writestr("unrecorded.py", "bad")
+    with pytest.raises(ValueError, match="not recorded"):
+        carriers.unpack(path, tmp_path / "unpacked")
+
+
+def test_size_limit_fails_without_dropping_architectures(tmp_path, monkeypatch):
+    raw = raw_wheels(tmp_path)
+    monkeypatch.setattr(carriers, "binary_evidence", lambda roots: {})
+    monkeypatch.setattr(carriers.subprocess, "check_output", lambda *args, **kwargs: "")
+    monkeypatch.setattr(carriers, "LIMIT", 100)
+    evidence = tmp_path / "evidence"
+    evidence.mkdir()
+    with pytest.raises(ValueError, match="capacity"):
+        carriers.assemble(raw, tmp_path / "final", evidence)
@@ -0,0 +1,209 @@
+"""Adversarial promotion checks. All remote calls/uploads are mocked."""
+
+import copy
+import json
+import sys
+from pathlib import Path
+
+import pytest
+
+sys.path.insert(0, str(Path(__file__).resolve().parent))
+from contracts import CASES, digest, write_json
+from release_stack import (
+    check_evidence,
+    check_unused,
+    cleanup,
+    existing_matches,
+    promote,
+)
+from test_release_contracts import fixture, report
+
+
+@pytest.fixture
+def candidate(tmp_path, monkeypatch):
+    root = tmp_path / "candidate"
+    root.mkdir()
+    manifest = fixture(root)
+    sha = digest(root / "release.json")
+    evidence = tmp_path / "evidence"
+    (evidence / "tests").mkdir(parents=True)
+    write_json(
+        evidence / "request.json",
+        {
+            "mode": "release-candidate",
+            "manifest_sha256": sha,
+            "build_run_id": 123,
+            "build_run_attempt": 1,
+            "build_workflow_sha": "a" * 40,
+            "harness_sha": "a" * 40,
+        },
+    )
+    suite = {
+        "status": "passed",
+        "cases": {case: {"status": "passed"} for case in CASES},
+    }
+    for name in ("host-result.json", "tests/suite.json"):
+        write_json(evidence / name, suite)
+    for label in ("serving", "combined"):
+        write_json(evidence / f"tests/release-{label}-install.json", report(manifest))
+    monkeypatch.setenv("GITHUB_REF", "refs/heads/main")
+    monkeypatch.setenv("GITHUB_EVENT_NAME", "workflow_dispatch")
+    monkeypatch.setenv("GITHUB_REPOSITORY", "kvcache-ai/ktransformers")
+    monkeypatch.setenv("GITHUB_RUN_ID", "123")
+    monkeypatch.setenv("GITHUB_SHA", "a" * 40)
+    return root, manifest, sha, evidence
+
+
+def files(entry):
+    return [
+        {
+            "filename": entry["file"],
+            "digests": {"sha256": entry["sha256"]},
+            "yanked": False,
+        }
+    ]
+
+
+def test_uploads_tested_bytes_in_dependency_order_with_kt_last(candidate, tmp_path):
+    root, manifest, sha, evidence = candidate
+    uploaded = {}
+
+    def query(name, version):
+        return files(manifest["wheels"][name]) if name in uploaded else []
+
+    def upload(path):
+        name = next(
+            name
+            for name, entry in manifest["wheels"].items()
+            if entry["file"] == path.name
+        )
+        assert digest(path) == manifest["wheels"][name]["sha256"]
+        uploaded[name] = path
+
+    output = tmp_path / "promotion.json"
+    promote(root, sha, evidence, output, query=query, upload=upload)
+    assert list(uploaded) == [
+        "accelerate-kt",
+        "transformers-kt",
+        "kt-kernel",
+        "sglang-kt",
+        "ktransformers",
+    ]
+    assert json.loads(output.read_text())["status"] == "uploaded-awaiting-public-e2e"
+
+
+def test_retry_reuses_only_exact_files_without_reupload(candidate, tmp_path):
+    root, manifest, sha, evidence = candidate
+    promote(
+        root,
+        sha,
+        evidence,
+        tmp_path / "promotion.json",
+        query=lambda name, version: files(manifest["wheels"][name]),
+        upload=lambda path: pytest.fail("Must not re-upload identical published files"),
+    )
+
+
+def test_conflict_in_last_package_fails_before_any_upload(candidate, tmp_path):
+    root, manifest, sha, evidence = candidate
+    conflicting = copy.deepcopy(manifest["wheels"]["ktransformers"])
+    conflicting["sha256"] = "c" * 64
+    with pytest.raises(ValueError, match="different bytes"):
+        promote(
+            root,
+            sha,
+            evidence,
+            tmp_path / "promotion.json",
+            query=lambda name, version: files(conflicting)
+            if name == "ktransformers"
+            else [],
+            upload=lambda path: pytest.fail("No writes before full batch preflight"),
+        )
+
+
+@pytest.mark.parametrize(
+    "field,value",
+    [
+        ("mode", "release-pypi"),
+        ("manifest_sha256", "c" * 64),
+        ("build_run_id", 456),
+        ("build_run_attempt", 2),
+        ("harness_sha", "c" * 40),
+    ],
+)
+def test_wrong_acceptance_identity_fails(candidate, field, value):
+    root, manifest, sha, evidence = candidate
+    path = evidence / "request.json"
+    request = json.loads(path.read_text())
+    write_json(path, request | {field: value})
+    with pytest.raises(ValueError):
+        check_evidence(evidence, manifest, sha, "candidate")
+
+
+@pytest.mark.parametrize("status", ["failed", "resource_unavailable", "not_run"])
+def test_missing_or_failed_model_blocks_promotion(candidate, status):
+    root, manifest, sha, evidence = candidate
+    path = evidence / "tests/suite.json"
+    result = json.loads(path.read_text())
+    result["cases"]["glm53_inference"]["status"] = status
+    write_json(path, result)
+    with pytest.raises(ValueError, match="model acceptance"):
+        check_evidence(evidence, manifest, sha, "candidate")
+
+
+def test_incomplete_suite_never_passes(candidate):
+    root, manifest, sha, evidence = candidate
+    path = evidence / "tests/suite.json"
+    result = json.loads(path.read_text())
+    del result["cases"]["deepseek_v31_lora"]
+    write_json(path, result)
+    with pytest.raises(ValueError):
+        check_evidence(evidence, manifest, sha, "candidate")
+
+
+def test_pypi_check_must_use_public_downloads(candidate):
+    root, manifest, sha, evidence = candidate
+    request = json.loads((evidence / "request.json").read_text())
+    write_json(evidence / "request.json", request | {"mode": "release-pypi"})
+    with pytest.raises(ValueError, match="public PyPI"):
+        check_evidence(evidence, manifest, sha, "pypi")
+
+
+def test_existing_version_cannot_be_rebuilt_under_same_name():
+    with pytest.raises(ValueError, match="already exists"):
+        check_unused(
+            [{"name": "kt-kernel", "version": "1.0"}], query=lambda *args: [{}]
+        )
+
+
+def test_yanked_or_extra_remote_files_are_not_skipped():
+    entry = {"file": "x.whl", "sha256": "a" * 64}
+    for remote in (files(entry) * 2, [files(entry)[0] | {"yanked": True}]):
+        with pytest.raises(ValueError):
+            existing_matches(entry, remote)
+
+
+def test_partial_upload_never_claims_release_success(candidate, tmp_path):
+    root, manifest, sha, evidence = candidate
+
+    def fail(path):
+        raise RuntimeError("network failure")
+
+    output = tmp_path / "promotion.json"
+    with pytest.raises(RuntimeError):
+        promote(root, sha, evidence, output, query=lambda *args: [], upload=fail)
+    assert json.loads(output.read_text())["status"] == "partial-or-failed-upload"
+
+
+def test_cleanup_is_scoped_to_an_owned_run_directory(tmp_path, monkeypatch):
+    monkeypatch.setenv("GITHUB_RUN_ID", "123")
+    with pytest.raises(ValueError):
+        cleanup(tmp_path, tmp_path.parent)
+    owned = tmp_path / "kt-four-main.example"
+    owned.mkdir()
+    (owned / ".kt-release-owned").write_text("456")
+    with pytest.raises(ValueError):
+        cleanup(owned, tmp_path)
+    (owned / ".kt-release-owned").write_text("123")
+    cleanup(owned, tmp_path)
+    assert not owned.exists()
@@ -0,0 +1,27 @@
+name: Release pipeline contracts (CPU only)
+on:
+  pull_request:
+    paths: ['.github/release/**', '.github/model-e2e/**', '.github/workflows/release*.yml', '.github/workflows/model-e2e-release.yml']
+  push:
+    branches: [main]
+    paths: ['.github/release/**', '.github/model-e2e/**', '.github/workflows/release*.yml', '.github/workflows/model-e2e-release.yml']
+permissions:
+  contents: read
+jobs:
+  contracts:
+    runs-on: ubuntu-24.04
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          persist-credentials: false
+      - uses: actions/setup-python@v5
+        with:
+          python-version: '3.12'
+      - run: python -m pip install packaging==25.0 pytest==8.4.2
+      - name: Require the independently reviewed model gate (PR 2195)
+        run: test -f .github/model-e2e/release_contracts.py
+      - name: Test packaging, release identity, failure gates and upload retries
+        env:
+          PYTHONPATH: ${{ github.workspace }}/.github/model-e2e
+        run: python -m pytest -q .github/release
+      - run: python -m unittest discover -s .github/model-e2e -p 'test_*.py'
@@ -1,9 +1,15 @@
-name: Four-main release preparation (build only)
+name: Release four-main stack

-# This draft cannot publish. Do not add a push/version trigger before the final
-# carrier assembly, clean GPU validation and promotion gates are implemented.
+# Manual only. Main/version merges never publish. Requires the #2195 harness.
 on:
  workflow_dispatch:
+    inputs:
+      target:
+        description: 'Build and test only, or publish the same validated wheels to PyPI'
+        type: choice
+        required: true
+        default: candidate
+        options: [candidate, pypi]

 permissions:
  contents: read
@@ -17,7 +23,7 @@ concurrency:
 jobs:
  snapshot:
    name: Freeze the four public main heads
-    if: github.repository == 'kvcache-ai/ktransformers'
+    if: github.repository == 'kvcache-ai/ktransformers' && github.ref == 'refs/heads/main'
    runs-on: ubuntu-24.04
    timeout-minutes: 10
    outputs:
@@ -30,12 +36,26 @@ jobs:
        with:
          python-version: '3.12'
      - run: python -m pip install packaging==25.0
+      - name: Check activation prerequisites before starting a costly build
+        env:
+          RELEASE_ENABLED: ${{ vars.KT_FOUR_MAIN_RELEASE_ENABLED }}
+          E2E_ENABLED: ${{ vars.KT_MODEL_E2E_ENABLED }}
+        run: |
+          test "$RELEASE_ENABLED" = true
+          test "$E2E_ENABLED" = true
+          test -f .github/workflows/model-e2e-release.yml
+          test -f .github/model-e2e/release_contracts.py
      - name: Resolve latest main heads once
        env:
          WORKFLOW_SHA: ${{ github.sha }}
        run: |
          python .github/release/four_main.py snapshot \
            --workflow-sha "$WORKFLOW_SHA" --output source-lock.json
+          python - <<'PY'
+          import json, os
+          lock = json.load(open('source-lock.json'))
+          assert lock['sources']['ktransformers']['sha'] == os.environ['WORKFLOW_SHA'], 'KT main moved after dispatch; start a fresh run'
+          PY
      - name: Name this snapshot
        id: names
        env:
@@ -49,10 +69,14 @@ jobs:
          retention-days: 90

  raw-build:
-    name: Build fresh raw wheels (not a release)
+    name: Build once, assemble carriers, and lock the dependency wheelhouse
    needs: snapshot
    runs-on: [self-hosted, linux, x64, gpu, kt-cpu]
    timeout-minutes: 720
+    outputs:
+      artifact: ${{ steps.candidate.outputs.artifact }}
+      manifest_sha256: ${{ steps.prepare.outputs.manifest_sha256 }}
+      build_attempt: ${{ steps.prepare.outputs.build_attempt }}
    env:
      # Initial scope matches the proven CPython 3.12 release. Expand the
      # Python matrix only after each ABI gets its own installation/E2E gate.
@@ -61,6 +85,14 @@ jobs:
      MAX_JOBS: '4'
      PIP_DISABLE_PIP_VERSION_CHECK: '1'
      PIP_NO_CACHE_DIR: '1'
+      PIP_CONFIG_FILE: /dev/null
+      PIP_INDEX_URL: https://pypi.org/simple
+      PIP_EXTRA_INDEX_URL: ''
+      PYTHONNOUSERSITE: '1'
+      PYTHONPATH: ''
+      PYTHONHOME: ''
+      CONDA_PREFIX: ''
+      CMAKE_PREFIX_PATH: ''
    steps:
      - uses: actions/checkout@v4
        with:
@@ -78,17 +110,20 @@ jobs:
          set -euo pipefail
          release_work="$(mktemp -d "$RUNNER_TEMP/kt-four-main.XXXXXXXX")"
          echo "RELEASE_WORK=$release_work" >> "$GITHUB_ENV"
+          echo "$GITHUB_RUN_ID" > "$release_work/.kt-release-owned"
          python -m venv "$release_work/venv"
          mkdir "$release_work/raw-wheels" "$release_work/evidence"
          cp lock/source-lock.json "$release_work/evidence/source-lock.json"
          echo "$release_work/venv/bin" >> "$GITHUB_PATH"
          echo "$CUDA_HOME/bin" >> "$GITHUB_PATH"
+          echo "VIRTUAL_ENV=$release_work/venv" >> "$GITHUB_ENV"
+          echo "LD_LIBRARY_PATH=$CUDA_HOME/lib64" >> "$GITHUB_ENV"
      - name: Check build prerequisites
        run: |
          set -euo pipefail
          nvidia-smi
          nvcc --version
-          for command in git gcc g++ ninja pkg-config; do
+          for command in git gcc g++ ninja pkg-config readelf cuobjdump; do
            command -v "$command"
          done
          test "$(uname -m)" = x86_64
@@ -99,7 +134,7 @@ jobs:
            pip==25.2 build==1.3.0 packaging==25.0 \
            setuptools==80.9.0 wheel==0.45.1 \
            scikit-build-core==0.11.6 cmake==3.31.6 pybind11==2.13.6 \
-            ninja==1.11.1.4 twine==6.1.0
+            ninja==1.11.1.4 twine==6.1.0 auditwheel==6.4.2 patchelf==0.17.2.4
          python -m pip install torch==2.9.1 --index-url https://download.pytorch.org/whl/cu128
          python -m pip freeze --all > "$RELEASE_WORK/evidence/build-environment.txt"
          nvcc --version > "$RELEASE_WORK/evidence/nvcc.txt"
@@ -130,6 +165,8 @@ jobs:
            --lock lock/source-lock.json --sources "$RELEASE_WORK/sources" \
            --wheels "$RELEASE_WORK/raw-wheels" \
            --output "$RELEASE_WORK/evidence/python-metadata-report.json"
+          python ci-tools/.github/release/release_stack.py unused \
+            --wheels "$RELEASE_WORK/raw-wheels"
      - name: Compile all KT CPU variants and CUDA architectures
        env:
          CPUINFER_BUILD_ALL_VARIANTS: '1'
@@ -160,6 +197,52 @@ jobs:
            --lock lock/source-lock.json --sources "$RELEASE_WORK/sources" \
            --wheels "$RELEASE_WORK/raw-wheels" \
            --output "$RELEASE_WORK/evidence/raw-wheel-report.json"
+      - name: Repair native inputs without changing runtime source
+        run: |
+          set -euo pipefail
+          mkdir "$RELEASE_WORK/repaired"
+          python - <<'PY'
+          import os, pathlib, shutil, subprocess, sys
+          sys.path.insert(0, 'ci-tools/.github/release')
+          from four_main import inspect_wheel
+          work = pathlib.Path(os.environ['RELEASE_WORK'])
+          for path in sorted((work / 'raw-wheels').glob('*.whl')):
+              entry = inspect_wheel(path)
+              if entry['name'] in {'kt-kernel', 'sgl-kernel-kt'}:
+                  subprocess.run([
+                      sys.executable, '-m', 'auditwheel', 'repair', str(path),
+                      '--plat', 'manylinux_2_35_x86_64', '--exclude', 'libcuda.so.1',
+                      '--exclude', 'libtorch.so', '--exclude', 'libtorch_cpu.so',
+                      '--exclude', 'libtorch_cuda.so', '--exclude', 'libtorch_python.so',
+                      '--exclude', 'libc10.so', '--exclude', 'libc10_cuda.so',
+                      '--exclude', 'libcudart.so.12',
+                      '-w', str(work / 'repaired'),
+                  ], check=True)
+              else:
+                  shutil.copyfile(path, work / 'repaired' / path.name)
+          PY
+      - name: Assemble final carriers and inspect CUDA SASS coverage
+        run: |
+          python ci-tools/.github/release/carriers.py \
+            --raw "$RELEASE_WORK/repaired" --output "$RELEASE_WORK/final" \
+            --evidence "$RELEASE_WORK/evidence"
+          python -m twine check "$RELEASE_WORK/final/"*.whl
+      - name: Resolve both user extras and seal the final wheelhouse
+        id: prepare
+        run: |
+          python ci-tools/.github/release/release_stack.py prepare \
+            --final "$RELEASE_WORK/final" --root "$RELEASE_WORK/candidate" \
+            --lock lock/source-lock.json --evidence "$RELEASE_WORK/evidence"
+      - name: Name this immutable candidate
+        id: candidate
+        run: echo "artifact=four-main-candidate-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}" >> "$GITHUB_OUTPUT"
+      - uses: actions/upload-artifact@v4
+        with:
+          name: ${{ steps.candidate.outputs.artifact }}
+          path: ${{ env.RELEASE_WORK }}/candidate/
+          if-no-files-found: error
+          retention-days: 90
+          compression-level: 0
      - name: Retain source and build evidence even when a gate fails
        if: always() && env.RELEASE_WORK != ''
        uses: actions/upload-artifact@v4
@@ -168,7 +251,7 @@ jobs:
          path: ${{ env.RELEASE_WORK }}/evidence/
          if-no-files-found: error
          retention-days: 90
-      - name: Retain raw build outputs for carrier assembly development
+      - name: Retain raw build outputs for diagnosis (never publish these)
        if: always() && env.RELEASE_WORK != ''
        uses: actions/upload-artifact@v4
        with:
@@ -176,11 +259,102 @@ jobs:
          path: ${{ env.RELEASE_WORK }}/raw-wheels/*.whl
          if-no-files-found: ignore
          retention-days: 7
-      - name: State the completed boundary
-        if: always()
+      - name: Clean only this run's build directory after artifact retention
+        if: always() && env.RELEASE_WORK != ''
        run: |
-          echo '### Four-main source build only' >> "$GITHUB_STEP_SUMMARY"
-          echo 'No package was published. Carrier assembly, isolated dependency resolution, clean GPU E2E and promotion remain required.' >> "$GITHUB_STEP_SUMMARY"
+          python ci-tools/.github/release/cleanup.py \
+            --directory "$RELEASE_WORK" --parent "$RUNNER_TEMP"

-  # No publishing job until the documented remaining gates are implemented.
-  # In particular, success() above is NOT sufficient authority to publish.
+  candidate-e2e:
+    needs: raw-build
+    uses: ./.github/workflows/model-e2e-release.yml
+    with:
+      stage: candidate
+      artifact: ${{ needs.raw-build.outputs.artifact }}
+      manifest_sha256: ${{ needs.raw-build.outputs.manifest_sha256 }}
+      build_attempt: ${{ needs.raw-build.outputs.build_attempt }}
+
+  publish:
+    needs: [raw-build, candidate-e2e]
+    if: inputs.target == 'pypi' && needs.candidate-e2e.result == 'success'
+    runs-on: ubuntu-24.04
+    environment: prod
+    timeout-minutes: 90
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          persist-credentials: false
+      - uses: actions/setup-python@v5
+        with:
+          python-version: '3.12'
+      - run: python -m pip install packaging==25.0 twine==6.1.0
+      - uses: actions/download-artifact@v4
+        with:
+          name: ${{ needs.raw-build.outputs.artifact }}
+          path: candidate
+      - uses: actions/download-artifact@v4
+        with:
+          name: ${{ needs.candidate-e2e.outputs.evidence }}
+          path: candidate-evidence
+      - name: Verify all evidence and upload identical wheels, KT last
+        env:
+          MANIFEST_SHA256: ${{ needs.raw-build.outputs.manifest_sha256 }}
+          TWINE_USERNAME: __token__
+          TWINE_PASSWORD: ${{ secrets.PYPI_API_TOKEN }}
+        run: |
+          test -n "$TWINE_PASSWORD"
+          python .github/release/release_stack.py publish \
+            --root candidate --digest "$MANIFEST_SHA256" \
+            --evidence candidate-evidence --output promotion.json
+      - uses: actions/upload-artifact@v4
+        if: always()
+        with:
+          name: four-main-promotion-${{ github.run_id }}-${{ github.run_attempt }}
+          path: promotion.json
+          if-no-files-found: warn
+          retention-days: 90
+
+  public-e2e:
+    needs: [raw-build, publish]
+    uses: ./.github/workflows/model-e2e-release.yml
+    with:
+      stage: pypi
+      artifact: ${{ needs.raw-build.outputs.artifact }}
+      manifest_sha256: ${{ needs.raw-build.outputs.manifest_sha256 }}
+      build_attempt: ${{ needs.raw-build.outputs.build_attempt }}
+
+  released:
+    needs: [raw-build, publish, public-e2e]
+    runs-on: ubuntu-24.04
+    timeout-minutes: 20
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          persist-credentials: false
+      - uses: actions/setup-python@v5
+        with:
+          python-version: '3.12'
+      - run: python -m pip install packaging==25.0
+      - uses: actions/download-artifact@v4
+        with:
+          name: ${{ needs.raw-build.outputs.artifact }}
+          path: candidate
+      - uses: actions/download-artifact@v4
+        with:
+          name: ${{ needs.public-e2e.outputs.evidence }}
+          path: public-evidence
+      - name: Declare release success only after fresh public-PyPI E2E
+        env:
+          MANIFEST_SHA256: ${{ needs.raw-build.outputs.manifest_sha256 }}
+        run: |
+          python .github/release/release_stack.py verify \
+            --root candidate --digest "$MANIFEST_SHA256" \
+            --evidence public-evidence --output release-result.json
+          echo '### Released and verified from public PyPI' >> "$GITHUB_STEP_SUMMARY"
+          echo "Manifest SHA256: $MANIFEST_SHA256" >> "$GITHUB_STEP_SUMMARY"
+      - uses: actions/upload-artifact@v4
+        with:
+          name: four-main-release-result-${{ github.run_id }}-${{ github.run_attempt }}
+          path: release-result.json
+          if-no-files-found: error
+          retention-days: 90
@@ -1,11 +1,6 @@
-name: Release to PyPI
+name: Legacy release (retired - use Release four-main stack)

 on:
-  push:
-    branches:
-      - main
-    paths:
-      - "version.py"
  workflow_dispatch:
    inputs:
      target:
@@ -29,7 +24,14 @@ env:
  SFT_RELEASE_MANIFEST: .github/release/sft-packages.json

 jobs:
+  retired:
+    runs-on: ubuntu-24.04
+    steps:
+      - run: |
+          echo 'This legacy publishing path is retired. Use Actions -> Release four-main stack -> Run workflow.'
+          exit 1
  runner-preflight:
+    needs: retired
    name: Validate release and runner
    if: github.repository == 'kvcache-ai/ktransformers' && github.ref == 'refs/heads/main'
    runs-on: [self-hosted, linux, x64, gpu]
@@ -1,11 +1,6 @@
-name: Release sglang-kt to PyPI
+name: Legacy sglang release (retired - use Release four-main stack)

 on:
-  push:
-    branches:
-      - main
-    paths:
-      - "third_party/sglang"
  workflow_dispatch:
    inputs:
      test_pypi:
@@ -25,7 +20,14 @@ concurrency:
  cancel-in-progress: false

 jobs:
+  retired:
+    runs-on: ubuntu-24.04
+    steps:
+      - run: |
+          echo 'Independent SGLang publishing is retired. Use Actions -> Release four-main stack -> Run workflow.'
+          exit 1
  build-sglang-kt:
+    needs: retired
    name: Build sglang-kt wheel
    runs-on: [self-hosted, linux, x64, gpu]
    outputs: