mirror of
https://github.com/Tencent/ncnn.git
synced 2026-10-17 04:41:47 +08:00
acture
c798db5525
* RotaryEmbed: support full-width cos/sin cache for 2D / vision RoPE The non-interleaved path read only the first embed_dim/2 values of the cos/sin cache and reused them for both halves of each head vector. That is correct for standard 1-D RoPE (cos = concat(freqs, freqs), identical halves), but the pnnx fuse_convert_rotaryembed pass lowers the general expression x*cos + rotate_half(x)*sin — with a full-width cos — into this op without checking that the halves are equal. For 2-D / vision RoPE where the halves differ (cos = concat(cos_h, cos_w), e.g. the Penguin-VL vision encoder) the second half was rotated with the wrong frequencies, silently producing wrong output (max|diff| ~3.4 vs PyTorch on the Penguin-VL vision encoder). Read the second half's cos/sin for the second output half. Backward compatible via a cache-width guard: when the cache is full width (== embed_dim) the two halves are independent; when it is embed_dim/2 (the standard-rope form, as the unit test supplies) the second half reuses the first, so existing half-width callers stay bit-identical. Applied to every backend (naive / arm / x86 / loongarch / mips / vulkan) and its bf16/fp16 paths. Adds a full-width differing-halves test case; the previous test only exercised half-width caches, which are trivially identical-halves. After the fix the Penguin-VL vision encoder converts in-graph at max|diff| = 6.2e-4 vs PyTorch (fp32). Fixes #6833 * test: add a pnnx test that can actually fail for the full-width cache fix tests/test_rotaryembed.cpp cannot detect this bug. ncnn's test_layer() compares the optimized backends against the naive layer, and this PR changes the naive layer too, so the test stays green whether the semantics are right or wrong — rebuilding the PR's test against master's src/layer still exits 0. tools/pnnx/tests/ncnn/ is the only harness in-tree that pins semantics to an external reference (torch.allclose against PyTorch), so the check belongs there. Every input is drawn with torch.rand in forward() parameter order, because the generated test_ncnn_fuse_rotaryembed_ncnn.py takes no arguments: it reseeds with manual_seed(0) and regenerates each input the same way. An input built any other way would leave the two sides comparing different tensors, and the test would fail unconditionally while appearing to test something. Random caches are also the stronger choice here: the defect is that the layer reads one cache row for both halves, so it only shows when the halves differ, which random values guarantee and a hand-built cat(f, f) cache would hide. A "standard RoPE stays correct" case is deliberately absent — the cache must stay a pnnx.Input for the fusion pattern to match, and the generator always randomizes inputs, so that case cannot be expressed in this harness. Also parameterizes the interleaved path in tests/test_rotaryembed.cpp. On a CPU-only build that is code-path coverage rather than a correctness check, for the reason above; its value is on the Vulkan backend. Not run here: building pnnx needs libtorch. Worth confirming under ctest that the generated .param really contains RotaryEmbed layers — if the fusion does not fire, the test passes without exercising anything. * fix pnnx rotaryembed regression test layout and fusion checks --------- Co-authored-by: nihui <nihuini@tencent.com>
102 lines
3.2 KiB
C++
102 lines
3.2 KiB
C++
// Copyright 2025 Tencent
|
|
// SPDX-License-Identifier: BSD-3-Clause
|
|
|
|
#include "testutil.h"
|
|
|
|
static int test_rotaryembed(const ncnn::Mat& a, int interleaved)
|
|
{
|
|
const int embed_dim = a.w;
|
|
const int seqlen = a.h;
|
|
const int num_heads = a.c;
|
|
|
|
ncnn::Mat cos_cache = RandomMat(embed_dim / 2, seqlen);
|
|
ncnn::Mat sin_cache = RandomMat(embed_dim / 2, seqlen);
|
|
|
|
ncnn::ParamDict pd;
|
|
pd.set(0, interleaved);
|
|
|
|
std::vector<ncnn::Mat> weights(0);
|
|
|
|
std::vector<ncnn::Mat> as(3);
|
|
as[0] = a;
|
|
as[1] = cos_cache;
|
|
as[2] = sin_cache;
|
|
|
|
int ret = test_layer("RotaryEmbed", pd, weights, as, 1);
|
|
if (ret != 0)
|
|
{
|
|
fprintf(stderr, "test_rotaryembed failed a=(%d %d %d) interleaved=%d\n", a.w, a.h, a.c, interleaved);
|
|
}
|
|
|
|
return ret;
|
|
}
|
|
|
|
static int test_rotaryembed_0()
|
|
{
|
|
return 0
|
|
|| test_rotaryembed(RandomMat(32, 66, 8), 0)
|
|
|| test_rotaryembed(RandomMat(26, 64, 8), 1)
|
|
|| test_rotaryembed(RandomMat(64, 28, 12), 0)
|
|
|| test_rotaryembed(RandomMat(48, 22, 12), 1)
|
|
|| test_rotaryembed(RandomMat(44, 28, 64), 0)
|
|
|| test_rotaryembed(RandomMat(12, 27, 64), 1)
|
|
|| test_rotaryembed(RandomMat(28, 17, 15), 0)
|
|
|| test_rotaryembed(RandomMat(28, 17, 15), 1);
|
|
}
|
|
|
|
// FULL-WIDTH (embed_dim) cos/sin cache. This is the shape pnnx always emits, because the
|
|
// fused expression multiplies the cache elementwise with the full embed_dim input
|
|
// (tools/pnnx/src/pass_ncnn/fuse_convert_rotaryembed.cpp).
|
|
//
|
|
// non-interleaved: the two halves of the cache carry independent cos/sin (2D / vision rope);
|
|
// a half-width cache is the degenerate case where they are identical.
|
|
// interleaved: only the first embed_dim/2 entries of each row are used, but the row
|
|
// stride is embed_dim -- which is what exercises the cache stride handling.
|
|
static int test_rotaryembed_fullcos(const ncnn::Mat& a, int interleaved)
|
|
{
|
|
const int embed_dim = a.w;
|
|
const int seqlen = a.h;
|
|
|
|
ncnn::Mat cos_cache = RandomMat(embed_dim, seqlen);
|
|
ncnn::Mat sin_cache = RandomMat(embed_dim, seqlen);
|
|
|
|
ncnn::ParamDict pd;
|
|
pd.set(0, interleaved);
|
|
|
|
std::vector<ncnn::Mat> weights(0);
|
|
|
|
std::vector<ncnn::Mat> as(3);
|
|
as[0] = a;
|
|
as[1] = cos_cache;
|
|
as[2] = sin_cache;
|
|
|
|
int ret = test_layer("RotaryEmbed", pd, weights, as, 1);
|
|
if (ret != 0)
|
|
{
|
|
fprintf(stderr, "test_rotaryembed_fullcos failed a=(%d %d %d) interleaved=%d\n", a.w, a.h, a.c, interleaved);
|
|
}
|
|
|
|
return ret;
|
|
}
|
|
|
|
static int test_rotaryembed_1()
|
|
{
|
|
return 0
|
|
|| test_rotaryembed_fullcos(RandomMat(32, 66, 8), 0)
|
|
|| test_rotaryembed_fullcos(RandomMat(64, 28, 12), 0)
|
|
|| test_rotaryembed_fullcos(RandomMat(44, 28, 64), 0)
|
|
|| test_rotaryembed_fullcos(RandomMat(28, 17, 15), 0)
|
|
|| test_rotaryembed_fullcos(RandomMat(32, 66, 8), 1)
|
|
|| test_rotaryembed_fullcos(RandomMat(26, 64, 8), 1)
|
|
|| test_rotaryembed_fullcos(RandomMat(48, 22, 12), 1)
|
|
|| test_rotaryembed_fullcos(RandomMat(12, 27, 64), 1)
|
|
|| test_rotaryembed_fullcos(RandomMat(28, 17, 15), 1);
|
|
}
|
|
|
|
int main()
|
|
{
|
|
SRAND(7767517);
|
|
|
|
return test_rotaryembed_0() || test_rotaryembed_1();
|
|
}
|