GLSL round() leaves the direction of halfway cases to the implementation.
MoltenVK rounds them away from zero, while every cpu implementation rounds
half to even, so 0.5 / 2.5 / -0.5 came out as 1 / 3 / -1 on Apple gpus.
Add exact halfway inputs to test_unaryop for op_type 18.
* riscv: add tiled int8 convolution im2col gemm
Pack weights and input in cache-sized M/N/K tiles with a 1x1 input shortcut.
Use vector-length-agnostic 8/4/2/1-position microkernels with signed
widening accumulation, including an XTheadVector-compatible path.
Reuse bounded per-worker input panels across output-channel tiles and
keep split-K partial sums in the existing int32 output. Wire the GEMM
option into the int8 pipeline while retaining direct convolution and
existing requantization, dequantization and activation handling.
Add focused tests for actual pipeline selection, packing, tails,
split-K accumulation, full-range int8 values and thread-count changes.
* riscv: fix int8 convolution split-K test sizing
A tile balanced for a large artificial K can be smaller than the cache
limit. Choosing that tile size plus one does not guarantee that the
actual test shape splits K, as exposed by the C907 and C908 CI jobs.
Query the tiler for each actual candidate K and grow the candidate until
it spans multiple panels. Keep the transformed-weight K-row assertion
and numerical checks, and report a missing split separately from an
incorrect pipeline selection.
* riscv: write int8 convolution gemm blocks explicitly
* riscv: separate full int8 gemm channel blocks and tails
* riscv: widen int8 convolution gemm to m4 accumulators
* apply code-format changes
* riscv: keep int8 convolution partial sums in packed tiles
* riscv: align int8 convolution gemm helpers with ncnn style
* apply code-format changes
* riscv: unify int8 convolution gemm channel loops
* riscv: reduce int8 im2col indexing and single-k tile overhead
* riscv: use scalar 2x2 int8 gemm and consolidate convolution tests
* mips, loongarch: fix packed int8 im2col K tile offsets
---------
Co-authored-by: nihui <171016+nihui@users.noreply.github.com>
* fix crash reading past end of memory-mapped model file
DataReaderFromMemory used by mapped model loading was unbounded:
read()/reference() advanced past the mapped region, so a truncated
or corrupt .bin crashed with an access violation inside
ModelBinFromDataReader instead of failing cleanly.
Add a size-aware DataReaderFromMemory ctor that clamps reads and
fails references past the end, use it in the mapped branches of
Net::load_model, and propagate load_model()'s return value which
was previously dropped.
* fix bounded memory reader scans and add loading regressions
* f
* reuse bounded memory reader scan buffer
---------
Co-authored-by: Dr3ladwyn <Dr3ladwyn@users.noreply.github.com>
Co-authored-by: nihui <nihuini@tencent.com>
Keep persistent KV caches on the blob allocator, broadcast single-channel attention masks across heads on CPU and Vulkan paths, and add long multi-token cache regressions.
* fix interp scale factor sampling and pnnx recomputed sizes
Use the original scale factor for linear and cubic interpolation instead
of deriving coordinates from the truncated output size. Preserve explicit
sizes, reference sizes, and size expressions across CPU and Vulkan paths.
Keep same-axis linear behavior, resample non-unit same-size bicubic input,
and clamp interpolation taps for short axes without out-of-bounds reads.
Lower recompute_scale_factor to runtime size expressions, retain the flag
when importing nn.Upsample, and avoid deleting non-unit same-size resizes.
Add independent ncnn reference checks and pnnx static/dynamic round-trip
tests covering fractional up/downscales, recomputed sizes, short axes,
align_corners, and nn.Upsample.
Validated CPU interpolation tests, ASan/UBSan, Vulkan llvmpipe, and pnnx
TorchScript interpolation tests against PyTorch 2.10.0 CPU. The original
bilinear/bicubic reproductions fail on the base and pass with this fix.
Refs: Tencent/ncnn#6836, Tencent/ncnn#6837, Tencent/ncnn#6844
---------
Co-authored-by: Acture <acturea@gmail.com>
Co-authored-by: xy200303 <3483421977@qq.com>
Co-authored-by: nihui <171016+nihui@users.noreply.github.com>
* Reduce memory access in x86 deconvolution Col2im
The SGEMM path fills the output with bias, then reads and updates each
output value. When kernel tiles do not overlap, each value has one source.
Write the source plus bias once, and fill only the output padding.
Use this path for rectangular kernels with unit dilation and stride equal
to the kernel size. Support packs of 1, 4, and 8. Split input rows between
workers when there are fewer output channel groups than threads. Each
worker writes separate output rows. Keep the existing path for other
cases. Add 32 test cases to test_deconvolution.
Test Plan
---------
These commands passed in the local benchmark workspace:
```bash
ctest --test-dir .tools/ncnn-col2im-build -R '^test_deconvolution$' --output-on-failure
.venv-esp/bin/python results/oss/col2im_transfer_20260920/generalization/test_ncnn.py --parallel
.venv-esp/bin/python results/oss/col2im_transfer_20260920/generalization/sanitize_ncnn.py
```
The complete forward test gave equal output bits for 236 cases and
1,433,578 values. The extracted loops passed ASan and UBSan for 300 cases.
The sanitizer test uses a small Mat substitute, not a full sanitizer build.
The local probe scripts are outside this repository.
The final format check and git diff --check passed. The format changes do
not change the tested operations.
This change was prepared with help from an AI assistant.
* RotaryEmbed: support full-width cos/sin cache for 2D / vision RoPE
The non-interleaved path read only the first embed_dim/2 values of the
cos/sin cache and reused them for both halves of each head vector. That is
correct for standard 1-D RoPE (cos = concat(freqs, freqs), identical halves),
but the pnnx fuse_convert_rotaryembed pass lowers the general expression
x*cos + rotate_half(x)*sin — with a full-width cos — into this op without
checking that the halves are equal. For 2-D / vision RoPE where the halves
differ (cos = concat(cos_h, cos_w), e.g. the Penguin-VL vision encoder) the
second half was rotated with the wrong frequencies, silently producing
wrong output (max|diff| ~3.4 vs PyTorch on the Penguin-VL vision encoder).
Read the second half's cos/sin for the second output half. Backward
compatible via a cache-width guard: when the cache is full width (== embed_dim)
the two halves are independent; when it is embed_dim/2 (the standard-rope form,
as the unit test supplies) the second half reuses the first, so existing
half-width callers stay bit-identical. Applied to every backend
(naive / arm / x86 / loongarch / mips / vulkan) and its bf16/fp16 paths.
Adds a full-width differing-halves test case; the previous test only exercised
half-width caches, which are trivially identical-halves.
After the fix the Penguin-VL vision encoder converts in-graph at
max|diff| = 6.2e-4 vs PyTorch (fp32).
Fixes#6833
* test: add a pnnx test that can actually fail for the full-width cache fix
tests/test_rotaryembed.cpp cannot detect this bug. ncnn's test_layer() compares
the optimized backends against the naive layer, and this PR changes the naive
layer too, so the test stays green whether the semantics are right or wrong —
rebuilding the PR's test against master's src/layer still exits 0.
tools/pnnx/tests/ncnn/ is the only harness in-tree that pins semantics to an
external reference (torch.allclose against PyTorch), so the check belongs there.
Every input is drawn with torch.rand in forward() parameter order, because the
generated test_ncnn_fuse_rotaryembed_ncnn.py takes no arguments: it reseeds with
manual_seed(0) and regenerates each input the same way. An input built any other
way would leave the two sides comparing different tensors, and the test would
fail unconditionally while appearing to test something.
Random caches are also the stronger choice here: the defect is that the layer
reads one cache row for both halves, so it only shows when the halves differ,
which random values guarantee and a hand-built cat(f, f) cache would hide. A
"standard RoPE stays correct" case is deliberately absent — the cache must stay a
pnnx.Input for the fusion pattern to match, and the generator always randomizes
inputs, so that case cannot be expressed in this harness.
Also parameterizes the interleaved path in tests/test_rotaryembed.cpp. On a
CPU-only build that is code-path coverage rather than a correctness check, for
the reason above; its value is on the Vulkan backend.
Not run here: building pnnx needs libtorch. Worth confirming under ctest that the
generated .param really contains RotaryEmbed layers — if the fusion does not
fire, the test passes without exercising anything.
* fix pnnx rotaryembed regression test layout and fusion checks
---------
Co-authored-by: nihui <nihuini@tencent.com>
* pnnx: store bool MemoryData attributes as raw int8
Keep bool constants as one byte per element and select MemoryData
load_type=3. Implement the reserved raw int8 ModelBin loading mode and
pad exported bool data to four bytes so subsequent weights stay aligned.
This preserves the byte representation used by the proposed logical ops.
Add regression coverage for scalar and padded bool exports in both fp16
modes, multidimensional MemoryData, raw int8 loading through reference
and copy readers, trailing fp32 weights, and missing padding. Document
the format.
Based on the bool attribute serialization fixes proposed in:
https://github.com/Tencent/ncnn/pull/6956https://github.com/Tencent/ncnn/pull/6866Fixes#6855
* tools: preserve byte MemoryData when optimizing models
Save MemoryData load_type so raw int8 constants retain their byte
representation after ModelWriter saves the model. Skip fusion paths that
interpret MemoryData as float32 when its elements are not four bytes.
This change does not add bool arithmetic support or automatic type
promotion. Byte bool storage is intended for future logical, comparison
and selection operators.
Validated ncnnoptimize save and reload for scalar and one-to-five-element
bool constants in fp32 and fp16 storage modes, preserving byte values
and element size in all 12 cases.
---------
Co-authored-by: ktz03 <2484593937@qq.com>
Co-authored-by: Giulio Corradi <53930027+giuliocorradi@users.noreply.github.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
* fix spirv compile failure diagnostics read past shader buffer (#6881)
the shader source slice passed to glslang (comp_datas[3]) is not
null-terminated, since the .comp_data arrays generated by
ncnn_generate_shader_comp_header.cmake are hex-encoded without a
trailing 0x00. the error-printing path in compile_spirv_module()
scanned it with strchr() and checked *p != '\0', causing an
out-of-bounds read and garbled diagnostics whenever a shader fails
to compile.
bound the scan with memchr() using the known source length and print
the trailing partial line with %.*s instead of relying on
null-termination. verified with asan: before, a broken non-null-
terminated shader source reports a stack-buffer-overflow in
compile_spirv_module(); after, the error path prints correctly with
no overrun.
* fix bounded shader version parsing and add regression tests
---------
Signed-off-by: lazypool <lazypool@proton.me>
Co-authored-by: Marcin Sochacki <136440930+marcin-sochacki@users.noreply.github.com>
Co-authored-by: nihui <nihuini@tencent.com>
* Harden Net graph loading and metadata validation
Apply count and blob index validation from Tencent/ncnn PRs #6902,
#6916, #6922 and #6949 to both text and binary graph loading. Bound
header counts to 1,000,000 while preserving repeated blob references
and zero-output layers.
Validate text integer tokens and name lengths, shape hint types,
record lengths and extent arithmetic, feature masks, layer type tags,
and one-blob layer connection counts. Guard missing producers during
extraction and handle null custom layer creators.
Own partially loaded layers before reading their connections, honor
registered destroyers on all failure and CPU replacement paths, and
clear stale graph metadata when reloading or rejecting a model.
Add in-memory Net regressions for malformed and truncated inputs, valid
shape layouts, reload isolation, custom layer lifetime and external
inputs, without requiring filesystem access.
Validated with native tests, SqueezeNet inference, ASan/UBSan and
32-bit big-endian PowerPC tests; compile checked Vulkan and reduced
C++03 configurations.
Sources:
https://github.com/Tencent/ncnn/pull/6902https://github.com/Tencent/ncnn/pull/6916https://github.com/Tencent/ncnn/pull/6922https://github.com/Tencent/ncnn/pull/6949
Co-authored-by: kaltsit-10 <kaltsit-10@users.noreply.github.com>
Co-authored-by: beilzx <2069360659@qq.com>
Co-authored-by: ktz03 <2484593937@qq.com>
The fp32 rational approximation coefficients cannot be represented in
half precision: alpha_9/11/13 underflow to zero and alpha_7 becomes
subnormal, so tanh_ps_f16 saturates at ~1.044 and fast gelu returns
+0.022*|x| for large negative inputs. Transformer models with large
activations (SigLIP2, CLIP ViT) produce input-independent features under
use_fp16_arithmetic.
Refit the rational approximation for fp16 on [0, 4.5], return +/-1
beyond that range, clamp the result to 1, and add large-magnitude test
cases for GELU and TanH.
nihui