132 Commits

Author SHA1 Message Date
風的彷徨 c4393a13b0 pnnx: preserve batch semantics across reshape and flatten (#6934)
Co-authored-by: nihui <nihuini@tencent.com>
2026-10-08 19:08:41 +08:00
nihui 050ba22657 vulkan add shader line coverage with NCNN_COVERAGE (#7016) 2026-09-21 10:14:53 +08:00
nihui 8bc95a1e7f pnnx: store bool MemoryData attributes as raw int8 (#7006)
* pnnx: store bool MemoryData attributes as raw int8

Keep bool constants as one byte per element and select MemoryData
load_type=3. Implement the reserved raw int8 ModelBin loading mode and
pad exported bool data to four bytes so subsequent weights stay aligned.
This preserves the byte representation used by the proposed logical ops.

Add regression coverage for scalar and padded bool exports in both fp16
modes, multidimensional MemoryData, raw int8 loading through reference
and copy readers, trailing fp32 weights, and missing padding. Document
the format.

Based on the bool attribute serialization fixes proposed in:
https://github.com/Tencent/ncnn/pull/6956
https://github.com/Tencent/ncnn/pull/6866

Fixes #6855

* tools: preserve byte MemoryData when optimizing models

Save MemoryData load_type so raw int8 constants retain their byte
representation after ModelWriter saves the model. Skip fusion paths that
interpret MemoryData as float32 when its elements are not four bytes.

This change does not add bool arithmetic support or automatic type
promotion. Byte bool storage is intended for future logical, comparison
and selection operators.

Validated ncnnoptimize save and reload for scalar and one-to-five-element
bool constants in fp32 and fp16 storage modes, preserving byte values
and element size in all 12 cases.

---------

Co-authored-by: ktz03 <2484593937@qq.com>
Co-authored-by: Giulio Corradi <53930027+giuliocorradi@users.noreply.github.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-09-17 10:09:00 +08:00
nihui 5ea98309e3 layer param hardening (#7003)
* harden layer parameter validation

Original patches and author identities:
https://github.com/Tencent/ncnn/pull/6912 (beilzx)
https://github.com/Tencent/ncnn/pull/6919 (FluxMind-O)
https://github.com/Tencent/ncnn/pull/6928 (lxcxjxhx)
https://github.com/Tencent/ncnn/pull/6948 (ktz03)
https://github.com/Tencent/ncnn/pull/6950 (ktz03)
https://github.com/Tencent/ncnn/pull/6971 (LI)
https://github.com/Tencent/ncnn/pull/6974 (LI)
https://github.com/Tencent/ncnn/pull/6977 (LI)

Co-authored-by: beilzx <2069360659@qq.com>
Co-authored-by: FluxMind-O <3283918317@qq.com>
Co-authored-by: lxcxjxhx <lxcxjxhx@users.noreply.github.com>
Co-authored-by: ktz03 <2484593937@qq.com>
Co-authored-by: LI <2484593937@qq.com>
2026-09-15 19:18:46 +08:00
nihui 0cdc3351c8 harden net loading (#6989)
* Harden Net graph loading and metadata validation

Apply count and blob index validation from Tencent/ncnn PRs #6902,
#6916, #6922 and #6949 to both text and binary graph loading. Bound
header counts to 1,000,000 while preserving repeated blob references
and zero-output layers.

Validate text integer tokens and name lengths, shape hint types,
record lengths and extent arithmetic, feature masks, layer type tags,
and one-blob layer connection counts. Guard missing producers during
extraction and handle null custom layer creators.

Own partially loaded layers before reading their connections, honor
registered destroyers on all failure and CPU replacement paths, and
clear stale graph metadata when reloading or rejecting a model.

Add in-memory Net regressions for malformed and truncated inputs, valid
shape layouts, reload isolation, custom layer lifetime and external
inputs, without requiring filesystem access.
Validated with native tests, SqueezeNet inference, ASan/UBSan and
32-bit big-endian PowerPC tests; compile checked Vulkan and reduced
C++03 configurations.

Sources:
https://github.com/Tencent/ncnn/pull/6902
https://github.com/Tencent/ncnn/pull/6916
https://github.com/Tencent/ncnn/pull/6922
https://github.com/Tencent/ncnn/pull/6949

Co-authored-by: kaltsit-10 <kaltsit-10@users.noreply.github.com>
Co-authored-by: beilzx <2069360659@qq.com>
Co-authored-by: ktz03 <2484593937@qq.com>
2026-09-14 14:23:50 +08:00
nihui c0abf4830d harden paramdict parsing and typed access (#6983)
Co-authored-by: Nathan Keys <nathan.keys@pm.me>
Co-authored-by: beilzx <2069360659@qq.com>
2026-09-10 17:23:55 +08:00
nihui 60bfa40044 sdpa flash attention x86 optimization, llm models enable gqa, dispatch for avx2 bfloat2float (#6923)
* enable gqa
* update perf test
2026-09-01 17:35:42 +08:00
nihui 7f162381ab kvcache allocator (#6901)
* decode benchmark
* reject kvcache batch
* enable mha vulkan gemm out_elempack 1
2026-08-17 15:44:30 +08:00
nihui c189d88a8d modelbin bf16 storage type (#6900)
* fix modelwriter on big-endian
2026-08-13 17:19:50 +08:00
nihui a2d60e2b37 fix fp16sa shader compiler error on device without int64, move ncnn glsl macro definitions to ncnn_glsl_ext.comp (#6895)
* workaround vs2017 illegal instruction
* workaround vs2017 abs512 mis-compile
2026-08-10 14:26:43 +08:00
nihui 92b73d6665 x86/arm/mips/loongarch/riscv gemm block quantization optimization (#6831)
* benchncnn_llm int8 models
2026-08-04 14:34:22 +08:00
nihui f74639deeb gemm block quantization infrastructure for llm decoder style (#6814)
* another mha oob read
2026-07-13 10:13:39 +08:00
nihui f6f734f44d vulkan int8 kernels (#6751)
* vulkan: add int8 kernels for core layers
Implement Vulkan int8 paths for convolution, convolutiondepthwise, innerproduct, and gemm with scalar fallback and integer dot product shader branches. Add goal.md and enable existing int8 unit tests for Vulkan coverage.
* enable int8 gpu benchmark
2026-06-17 17:39:49 +08:00
nihui 706c4d1964 support 4d mat in more operators (#6737)
Add 4d Mat handling across generic and optimized backends, extend operator tests, and document supported input and output Mat dimensions.
2026-05-22 15:31:01 +08:00
nihui 8c18666821 feat(vulkan): persist pipeline cache (#6702)
Add single-file Vulkan pipeline cache save/load support with strict device, driver, shader, and payload validation. Cache compiled SPIR-V per shader hash alongside the driver pipeline cache blob, add focused cache tests, and document the current Vulkan pipeline cache architecture and usage.

Limit Vulkan layer test cache growth with a shared test-only pipeline cache window so repeated test_layer() calls can still exercise cache reuse.

Co-authored-by: futz12 <56149058+futz12@users.noreply.github.com>

Co-authored-by: CLV-Iclucia <89170890+CLV-Iclucia@users.noreply.github.com>
2026-05-09 09:37:06 +08:00
nihui 526e5e65ef unroll 4x4 for gemm and sdpa vulkan, vectorize a and b loading, avoid bank conflict (#6524)
* unroll 4x4 for vulkan gemm, vectorize loading a and b
* unroll 4x4 for vulkan sdpa, vectorize loading a and b, avoid bank conflict
* Update glsl-extension.md
* Update glsl-extension.zh.md
* gemm vulkan subgroup optimization

* apply code-format changes

* enable it

---------

Co-authored-by: nihui <171016+nihui@users.noreply.github.com>
2026-01-28 13:55:23 +08:00
nihui fe7c0cd2ec dynamic vulkan loading on apple platforms, honor VK_DRIVER_FILES env (#6499) 2026-01-11 19:54:54 +08:00
Sophon 41187d42b0 simplevk: add qcvkarm64xum.dll from qualcomm windows driver (#6502) 2026-01-11 17:36:28 +08:00
nihui 6499fc0348 vulkan bf16 infrastructure (#6469)
* skip gpu bf16 winograd
* update doc
2026-01-04 19:34:50 +08:00
nihui 559a8e8cb0 rotaryembed layer (#6407)
* add deepseek_v3 attention test
* fuse non interleaved rotary embed
* fuse t5 layernorm without gamma
* test qwen2 attention
* sdpa pattern ++
2025-11-19 17:28:08 +08:00
nihui 11fb99795b sdpa kvcache (#6405) 2025-11-18 11:05:27 +08:00
nihui 641a3e5be8 ncnn sdpa layer (#6397)
* fuse gqa
2025-11-11 11:14:04 +08:00
nihui 63a7a8a590 optional layer support any packing (#6392)
* split gemm tests, enable gemm vulkan any packing
2025-11-07 14:23:53 +08:00
nihui 3ab4b58d1d optional layer support vulkan packing (#6389)
* pyncnn expose layer support vulkan packing
* doc layer support behavior
2025-11-05 16:27:23 +08:00
nihui adee6a0a53 multiheadattention kvcache (#6333)
* skip input layer forward
* unified kvcache layout
* mha kvcache test
2025-10-15 14:19:57 +08:00
nihui a6eda68f23 disable onnx2ncnn build (#6324)
Migrate most of onnx2ncnn functions to pnnx
2025-09-23 19:06:07 +08:00
nihui b5bc05e2e8 drop vulkan shader pack8 (#6270) 2025-08-22 11:05:59 +08:00
nihui 9b91fe5153 implement flip layer and pnnx torch.flip conversion (#6233)
Co-authored-by: 佰阅 <43716063+Baiyuetribe@users.noreply.github.com>
2025-08-06 14:49:16 +08:00
Copilot 4644540ea4 Add Windows XP support merging PRs #6176 and #6177 (#6204)
Co-authored-by: Sugar-Baby <87747602+Sugar-Baby@users.noreply.github.com>
Co-authored-by: AtomAlpaca <66774326+AtomAlpaca@users.noreply.github.com>
2025-07-24 15:45:02 +08:00
GIBEREZ 44982d0d23 About the update to the GLSL documentation after the image functions are deprecated (#6173) 2025-07-12 14:22:30 +08:00
chri321 4c72f52954 docs: update Chinese glsl-extension documentation (#6162)
- synchronize the latest English content to the Chinese documentation
- correct spelling errors in the English version of glsl-extention
- Fix spelling 'enable_validation_layer' in src/gpu.cpp
2025-07-10 14:21:34 +08:00
nihui bd0b111775 vulkan tight fp16p pack1 (#6127) 2025-06-16 23:09:35 +08:00
nihui 80da741307 glsl define ncnn_glsl_version macro (#6003) 2025-04-18 18:49:33 +08:00
nihui ef0b0e631c interp output size expression (#5994) 2025-04-17 14:24:57 +08:00
nihui 39c055d7f2 crop axes starts ends expression (#5976)
* skip dynamic tensor index

* handle clone oom
2025-04-14 19:43:50 +08:00
nihui 84970eed4d vulkan validation layer enables NCNN_LOGE in shader source (#5963)
* NCNN_LOGE in glsl

* Update glsl-extension.md
2025-03-31 16:17:16 +08:00
nihui 6396a732ef reshape shape expression, drop reshape permute, test reshape oom (#5918) 2025-03-10 15:31:34 +08:00
nihui 1e3fcb9dda paramdict value string type, natural array representation (#5915) 2025-02-20 19:04:16 +08:00
nihui 0734b657d9 spectrogram and inverse spectrogram (#5779)
* only supports hann, hamming and all-one window
* inverse spectrogram does not support length parameter
* spectrogram always returns torch.view_as_real(out) as ncnn does not support complex typed mat yet
* inverse spectrogram always accepts torch.view_as_complex(in) as ncnn does not support complex typed mat yet
2024-11-22 16:14:54 +08:00
nihui 66b54cbea2 multiheadattention int8 quantization (#5733)
* x86 vulkan fallback
* comment about bf16s
2024-10-15 16:38:00 +08:00
nihui 1c7af00499 gemm int8 quantization (#5706)
* quantize gemm

* write gemm quantize scales

* update doc

* less openmp args

* x86 riscv fallback

* skip gemm vulkan int8

* fix noint8 test, fix arm bf16 test

* enable vfpv4 on neon build only

* fix gemm vulkan without C

* fp16 pack8 output

* enable elempack=8 only for asimdhp+

* tiled gemm int8 test

* opt arm64 tiles, fix asimdhp dispatch
2024-10-12 19:02:55 +08:00
nihui 5df5413c81 embed int8 quantization and add embed test (#5667) 2024-09-02 18:48:01 +08:00
nihui fdf0df3079 RMSNorm (#5630) 2024-08-15 16:52:06 +08:00
nihui 4c3debae2d multiheadattention scale param (#5526)
* update swiftshader

* skip vs2017 swiftshader
2024-06-23 23:51:02 +08:00
nihui db035d602d update ncnnoptimize layers, lightmode=false keeps original weight (#5414) 2024-04-09 23:53:54 +08:00
luqiang-guo 8ddc85f4dd fix doc dual issue (#5342) 2024-02-22 19:13:31 +08:00
hugo-syn f35eb4b3b8 chore: Fix multiple typos (#5301)
Signed-off-by: hugo-syn <hugo.vincent@synacktiv.com>
2024-01-20 21:54:27 +08:00
nihui 5329d32e74 check vulkan fp16 uniform support and implement lfp conversion without fp16u (#5287) 2024-01-17 15:53:34 +08:00
nihui c222208cc9 feat mask for disable threading, make some extractor setter no-op, update doc (#5270) 2024-01-08 17:42:08 +08:00
張小凡 2ecaf37a3e Fix find GPU driver dll path in windows (#5141) 2023-11-10 19:07:24 +08:00