* riscv: add tiled int8 convolution im2col gemm
Pack weights and input in cache-sized M/N/K tiles with a 1x1 input shortcut.
Use vector-length-agnostic 8/4/2/1-position microkernels with signed
widening accumulation, including an XTheadVector-compatible path.
Reuse bounded per-worker input panels across output-channel tiles and
keep split-K partial sums in the existing int32 output. Wire the GEMM
option into the int8 pipeline while retaining direct convolution and
existing requantization, dequantization and activation handling.
Add focused tests for actual pipeline selection, packing, tails,
split-K accumulation, full-range int8 values and thread-count changes.
* riscv: fix int8 convolution split-K test sizing
A tile balanced for a large artificial K can be smaller than the cache
limit. Choosing that tile size plus one does not guarantee that the
actual test shape splits K, as exposed by the C907 and C908 CI jobs.
Query the tiler for each actual candidate K and grow the candidate until
it spans multiple panels. Keep the transformed-weight K-row assertion
and numerical checks, and report a missing split separately from an
incorrect pipeline selection.
* riscv: write int8 convolution gemm blocks explicitly
* riscv: separate full int8 gemm channel blocks and tails
* riscv: widen int8 convolution gemm to m4 accumulators
* apply code-format changes
* riscv: keep int8 convolution partial sums in packed tiles
* riscv: align int8 convolution gemm helpers with ncnn style
* apply code-format changes
* riscv: unify int8 convolution gemm channel loops
* riscv: reduce int8 im2col indexing and single-k tile overhead
* riscv: use scalar 2x2 int8 gemm and consolidate convolution tests
* mips, loongarch: fix packed int8 im2col K tile offsets
---------
Co-authored-by: nihui <171016+nihui@users.noreply.github.com>
* fix(python): scope custom layer factories to Net lifetime
Replace the process-global ten-slot callback table with stable per-Net userdata. Preserve registrations across clear() and destroy layers before releasing their factories.
Add real-binding regression coverage for repeated and simultaneous Nets, numeric and named registrations, reloads, callback lifetime, subclassing and interpreter shutdown.
Fixes#4560
* fix(python): release superseded custom layer callbacks
* fix(python): guard unencodable named custom layer slots
* fix crash reading past end of memory-mapped model file
DataReaderFromMemory used by mapped model loading was unbounded:
read()/reference() advanced past the mapped region, so a truncated
or corrupt .bin crashed with an access violation inside
ModelBinFromDataReader instead of failing cleanly.
Add a size-aware DataReaderFromMemory ctor that clamps reads and
fails references past the end, use it in the mapped branches of
Net::load_model, and propagate load_model()'s return value which
was previously dropped.
* fix bounded memory reader scans and add loading regressions
* f
* reuse bounded memory reader scan buffer
---------
Co-authored-by: Dr3ladwyn <Dr3ladwyn@users.noreply.github.com>
Co-authored-by: nihui <nihuini@tencent.com>
Keep persistent KV caches on the blob allocator, broadcast single-channel attention masks across heads on CPU and Vulkan paths, and add long multi-token cache regressions.
* fix interp scale factor sampling and pnnx recomputed sizes
Use the original scale factor for linear and cubic interpolation instead
of deriving coordinates from the truncated output size. Preserve explicit
sizes, reference sizes, and size expressions across CPU and Vulkan paths.
Keep same-axis linear behavior, resample non-unit same-size bicubic input,
and clamp interpolation taps for short axes without out-of-bounds reads.
Lower recompute_scale_factor to runtime size expressions, retain the flag
when importing nn.Upsample, and avoid deleting non-unit same-size resizes.
Add independent ncnn reference checks and pnnx static/dynamic round-trip
tests covering fractional up/downscales, recomputed sizes, short axes,
align_corners, and nn.Upsample.
Validated CPU interpolation tests, ASan/UBSan, Vulkan llvmpipe, and pnnx
TorchScript interpolation tests against PyTorch 2.10.0 CPU. The original
bilinear/bicubic reproductions fail on the base and pass with this fix.
Refs: Tencent/ncnn#6836, Tencent/ncnn#6837, Tencent/ncnn#6844
---------
Co-authored-by: Acture <acturea@gmail.com>
Co-authored-by: xy200303 <3483421977@qq.com>
Co-authored-by: nihui <171016+nihui@users.noreply.github.com>
* Reduce memory access in x86 deconvolution Col2im
The SGEMM path fills the output with bias, then reads and updates each
output value. When kernel tiles do not overlap, each value has one source.
Write the source plus bias once, and fill only the output padding.
Use this path for rectangular kernels with unit dilation and stride equal
to the kernel size. Support packs of 1, 4, and 8. Split input rows between
workers when there are fewer output channel groups than threads. Each
worker writes separate output rows. Keep the existing path for other
cases. Add 32 test cases to test_deconvolution.
Test Plan
---------
These commands passed in the local benchmark workspace:
```bash
ctest --test-dir .tools/ncnn-col2im-build -R '^test_deconvolution$' --output-on-failure
.venv-esp/bin/python results/oss/col2im_transfer_20260920/generalization/test_ncnn.py --parallel
.venv-esp/bin/python results/oss/col2im_transfer_20260920/generalization/sanitize_ncnn.py
```
The complete forward test gave equal output bits for 236 cases and
1,433,578 values. The extracted loops passed ASan and UBSan for 300 cases.
The sanitizer test uses a small Mat substitute, not a full sanitizer build.
The local probe scripts are outside this repository.
The final format check and git diff --check passed. The format changes do
not change the tested operations.
This change was prepared with help from an AI assistant.
VkCompute::record_download() skips the staging buffer whenever the blob
allocator is mappable. On an integrated GPU the blob allocator prefers
DEVICE_LOCAL|HOST_VISIBLE "unified" memory, which is mappable but
WRITE-COMBINED rather than HOST_CACHED, so the result is memcpy'd out of
uncached memory.
Uploads are unaffected -- writing to write-combined memory is fast. Only
the device-to-host read collapses, giving a large upload/download
asymmetry on the very same buffers.
Measured on a Radeon 8060S (Strix Halo, RADV GFX1151, 256 GB/s unified
LPDDR5X), 128 MB transfer, median of 3 runs:
host memcpy (reference) 23.0 GB/s
ncnn upload 10.0 GB/s
ncnn download before 0.32 GB/s
ncnn download after 4.19 GB/s 13.1x
The staging allocator is the one that asks for HOST_CACHED, and it was
bypassed in exactly the case where it is needed.
Add VulkanDevice::is_cached() mirroring is_coherent(), track it as a
`cached` flag on VkAllocator alongside `mappable` / `coherent`, and
extend the record_download() test to require both.
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: nihui <nihuini@tencent.com>
* RotaryEmbed: support full-width cos/sin cache for 2D / vision RoPE
The non-interleaved path read only the first embed_dim/2 values of the
cos/sin cache and reused them for both halves of each head vector. That is
correct for standard 1-D RoPE (cos = concat(freqs, freqs), identical halves),
but the pnnx fuse_convert_rotaryembed pass lowers the general expression
x*cos + rotate_half(x)*sin — with a full-width cos — into this op without
checking that the halves are equal. For 2-D / vision RoPE where the halves
differ (cos = concat(cos_h, cos_w), e.g. the Penguin-VL vision encoder) the
second half was rotated with the wrong frequencies, silently producing
wrong output (max|diff| ~3.4 vs PyTorch on the Penguin-VL vision encoder).
Read the second half's cos/sin for the second output half. Backward
compatible via a cache-width guard: when the cache is full width (== embed_dim)
the two halves are independent; when it is embed_dim/2 (the standard-rope form,
as the unit test supplies) the second half reuses the first, so existing
half-width callers stay bit-identical. Applied to every backend
(naive / arm / x86 / loongarch / mips / vulkan) and its bf16/fp16 paths.
Adds a full-width differing-halves test case; the previous test only exercised
half-width caches, which are trivially identical-halves.
After the fix the Penguin-VL vision encoder converts in-graph at
max|diff| = 6.2e-4 vs PyTorch (fp32).
Fixes#6833
* test: add a pnnx test that can actually fail for the full-width cache fix
tests/test_rotaryembed.cpp cannot detect this bug. ncnn's test_layer() compares
the optimized backends against the naive layer, and this PR changes the naive
layer too, so the test stays green whether the semantics are right or wrong —
rebuilding the PR's test against master's src/layer still exits 0.
tools/pnnx/tests/ncnn/ is the only harness in-tree that pins semantics to an
external reference (torch.allclose against PyTorch), so the check belongs there.
Every input is drawn with torch.rand in forward() parameter order, because the
generated test_ncnn_fuse_rotaryembed_ncnn.py takes no arguments: it reseeds with
manual_seed(0) and regenerates each input the same way. An input built any other
way would leave the two sides comparing different tensors, and the test would
fail unconditionally while appearing to test something.
Random caches are also the stronger choice here: the defect is that the layer
reads one cache row for both halves, so it only shows when the halves differ,
which random values guarantee and a hand-built cat(f, f) cache would hide. A
"standard RoPE stays correct" case is deliberately absent — the cache must stay a
pnnx.Input for the fusion pattern to match, and the generator always randomizes
inputs, so that case cannot be expressed in this harness.
Also parameterizes the interleaved path in tests/test_rotaryembed.cpp. On a
CPU-only build that is code-path coverage rather than a correctness check, for
the reason above; its value is on the Vulkan backend.
Not run here: building pnnx needs libtorch. Worth confirming under ctest that the
generated .param really contains RotaryEmbed layers — if the fusion does not
fire, the test passes without exercising anything.
* fix pnnx rotaryembed regression test layout and fusion checks
---------
Co-authored-by: nihui <nihuini@tencent.com>
* pnnx: preserve leading dimensions in spectrogram lowering
* pnnx: fix linear leading dimensions and spectrogram tests
* pnnx: fix padding on native batch axes
---------
Co-authored-by: nihui <nihuini@tencent.com>
Some toolchains accept -march=*xtheadvector and compile the generic
vector intrinsics without actually selecting the XTheadVector target.
This makes CMake enable XTheadVector code paths incorrectly.
Keep the existing intrinsic checks and append test3() referencing the
__riscv_xtheadvector ISA macro. This verifies the same target mode that
the RISC-V layer sources use, while remaining compatible with the
Xuantie C906/C910 toolchain.
* pnnx: store bool MemoryData attributes as raw int8
Keep bool constants as one byte per element and select MemoryData
load_type=3. Implement the reserved raw int8 ModelBin loading mode and
pad exported bool data to four bytes so subsequent weights stay aligned.
This preserves the byte representation used by the proposed logical ops.
Add regression coverage for scalar and padded bool exports in both fp16
modes, multidimensional MemoryData, raw int8 loading through reference
and copy readers, trailing fp32 weights, and missing padding. Document
the format.
Based on the bool attribute serialization fixes proposed in:
https://github.com/Tencent/ncnn/pull/6956https://github.com/Tencent/ncnn/pull/6866Fixes#6855
* tools: preserve byte MemoryData when optimizing models
Save MemoryData load_type so raw int8 constants retain their byte
representation after ModelWriter saves the model. Skip fusion paths that
interpret MemoryData as float32 when its elements are not four bytes.
This change does not add bool arithmetic support or automatic type
promotion. Byte bool storage is intended for future logical, comparison
and selection operators.
Validated ncnnoptimize save and reload for scalar and one-to-five-element
bool constants in fp32 and fp16 storage modes, preserving byte values
and element size in all 12 cases.
---------
Co-authored-by: ktz03 <2484593937@qq.com>
Co-authored-by: Giulio Corradi <53930027+giuliocorradi@users.noreply.github.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
* vulkan: apply atan2 zero workaround to packed shaders
Mesa v3dv has the same broken vector atan(0, 0) behavior as llvmpipe. Apply the existing zero-input workaround for all non-MoltenVK drivers so packed BinaryOp atan2 and ratan2 match the CPU result.
Fixes#6884.
* vulkan: limit packed atan2 zero workaround to llvmpipe and v3dv
---------
Co-authored-by: nihui <nihuini@tencent.com>
dependabot[bot]