mirror of https://github.com/Tencent/ncnn.git synced 2026-10-18 21:54:39 +08:00

Files

T

nihui 8bc95a1e7f pnnx: store bool MemoryData attributes as raw int8 (#7006 )

* pnnx: store bool MemoryData attributes as raw int8

Keep bool constants as one byte per element and select MemoryData
load_type=3. Implement the reserved raw int8 ModelBin loading mode and
pad exported bool data to four bytes so subsequent weights stay aligned.
This preserves the byte representation used by the proposed logical ops.

Add regression coverage for scalar and padded bool exports in both fp16
modes, multidimensional MemoryData, raw int8 loading through reference
and copy readers, trailing fp32 weights, and missing padding. Document
the format.

Based on the bool attribute serialization fixes proposed in:
https://github.com/Tencent/ncnn/pull/6956
https://github.com/Tencent/ncnn/pull/6866

Fixes #6855

* tools: preserve byte MemoryData when optimizing models

Save MemoryData load_type so raw int8 constants retain their byte
representation after ModelWriter saves the model. Skip fusion paths that
interpret MemoryData as float32 when its elements are not four bytes.

This change does not add bool arithmetic support or automatic type
promotion. Byte bool storage is intended for future logical, comparison
and selection operators.

Validated ncnnoptimize save and reload for scalar and one-to-five-element
bool constants in fp32 and fp16 storage modes, preserving byte values
and element size in all 12 cases.

---------

Co-authored-by: ktz03 <2484593937@qq.com>
Co-authored-by: Giulio Corradi <53930027+giuliocorradi@users.noreply.github.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>

2026-09-17 10:09:00 +08:00

90 KiB

Raw Permalink Blame History

AbsVal
ArgMax
BatchNorm
Bias
BinaryOp
BNLL
Cast
CELU
Clip
Concat
Convolution
Convolution1D
Convolution3D
ConvolutionDepthWise
ConvolutionDepthWise1D
ConvolutionDepthWise3D
CopyTo
Crop
CumulativeSum
Deconvolution
Deconvolution1D
Deconvolution3D
DeconvolutionDepthWise
DeconvolutionDepthWise1D
DeconvolutionDepthWise3D
DeepCopy
DeformableConv2D
DetectionOutput
Dequantize
Diag
Dropout
Einsum
Eltwise
ELU
Embed
Erf
Exp
ExpandDims
Flatten
Flip
Fold
GELU
GLU
Gemm
GridSample
GroupNorm
GRU
HardSigmoid
HardSwish
InnerProduct
Input
InstanceNorm
Interp
InverseSpectrogram
LayerNorm
Log
LRN
LSTM
MatMul
MemoryData
Mish
MultiHeadAttention
MVN
Noop
Normalize
Packing
Padding
Permute
PixelShuffle
Pooling
Pooling1D
Pooling3D
Power
PriorBox
PReLU
Proposal
PSROIPooling
Quantize
Reduction
ReLU
Reorg
Requantize
Reshape
RMSNorm
RNN
ROIAlign
ROIPooling
RotaryEmbed
Scale
SDPA
SELU
Shrink
ShuffleChannel
Sigmoid
Slice
Softmax
Softplus
Spectrogram
SPP
Split
StatisticsPooling
Squeeze
Swish
TanH
Threshold
Tile
UnaryOp
Unfold
YoloDetectionOutput
Yolov3DetectionOutput

AbsVal

y = abs(x)

input mat dims: 1d, 2d, 3d, 4d
output mat dims: same as input
one_blob_only
support_inplace

ArgMax

y = argmax(x, out_max_val, topk)

input mat dims: 1d, 2d, 3d, 4d
output mat dims: 1d, or 2d when out_max_val=1
one_blob_only

param id	name	type	default	description
0	out_max_val	int	0
1	topk	int	1

BatchNorm

y = (x - mean) / sqrt(var + eps) * slope + bias

input mat dims: 1d, 2d, 3d, 4d
output mat dims: same as input
one_blob_only
support_inplace

param id	name	type	default	description
0	channels	int	0
1	eps	float	0.f

weight	type	shape
slope_data	float	[channels]
mean_data	float	[channels]
var_data	float	[channels]
bias_data	float	[channels]

Bias

y = x + bias

input mat dims: 1d, 2d, 3d, 4d
output mat dims: same as input
one_blob_only
support_inplace

param id	name	type	default	description
0	bias_data_size	int	0

weight	type	shape
bias_data	float	[channels]

BinaryOp

This operation is used for binary computation, and the calculation rule depends on the broadcasting rule.

C = binaryop(A, B)

input mat dims: bottom0 1d, 2d, 3d, 4d; bottom1 1d, 2d, 3d, 4d or scalar
output mat dims: broadcast result, 1d, 2d, 3d, 4d if with_scalar = 1:

one_blob_only
support_inplace

param id	name	type	default	description
0	op_type	int	0	Operation type as follows
1	with_scalar	int	0	with_scalar=0 B is a matrix, with_scalar=1 B is a scalar
2	b	float	0.f	When B is a scalar, B = b

Operation type:

0 = ADD
1 = SUB
2 = MUL
3 = DIV
4 = MAX
5 = MIN
6 = POW
7 = RSUB
8 = RDIV
9 = RPOW
10 = ATAN2
11 = RATAN2

BNLL

y = log(1 + e^(-x)) , x > 0
y = log(1 + e^x),     x < 0

input mat dims: 1d, 2d, 3d, 4d
output mat dims: same as input
one_blob_only
support_inplace

Cast

y = cast(x)

input mat dims: 1d, 2d, 3d, 4d
output mat dims: same shape as input
one_blob_only
support_packing

param id	name	type	default	description
0	type_from	int	0
1	type_to	int	0

Element type:

0 = auto
1 = float32
2 = float16
3 = int8
4 = bfloat16

CELU

if x < 0    y = (exp(x / alpha) - 1.f) * alpha
else        y = x

input mat dims: 1d, 2d, 3d, 4d
output mat dims: same as input
one_blob_only
support_inplace

param id	name	type	default	description
0	alpha	float	1.f

Clip

y = clamp(x, min, max)

input mat dims: 1d, 2d, 3d, 4d
output mat dims: same as input
one_blob_only
support_inplace

param id	name	type	default	description
0	min	float	-FLT_MAX
1	max	float	FLT_MAX

Concat

y = concat(x0, x1, x2, ...) by axis

input mat dims: multiple inputs with same rank, 1d, 2d, 3d, 4d
output mat dims: same rank as inputs

param id	name	type	default	description
0	axis	int	0

Convolution

x2 = pad(x, pads, pad_value)
x3 = conv(x2, weight, kernel, stride, dilation) + bias
y = activation(x3, act_type, act_params)

input mat dims: 3d
output mat dims: 3d
one_blob_only

param id	name	type	default
0	num_output	int	0
1	kernel_w	int	0
2	dilation_w	int	1
3	stride_w	int	1
4	pad_left	int	0
5	bias_term	int	0
6	weight_data_size	int	0
8	int8_scale_term	int	0
9	activation_type	int	0
10	activation_params	array	[ ]
11	kernel_h	int	kernel_w
12	dilation_h	int	dilation_w
13	stride_h	int	stride_w
14	pad_top	int	pad_left
15	pad_right	int	pad_left
16	pad_bottom	int	pad_top
18	pad_value	float	0.f
19	dynamic_weight	int	0

weight	type	shape
weight_data	float/fp16/int8	[kernel_w, kernel_h, num_input, num_output]
bias_data	float	[num_output]
weight_data_int8_scales	float	[num_output]
bottom_blob_int8_scales	float	[1]
top_blob_int8_scales	float	[1]

Convolution1D

x2 = pad(x, pads, pad_value)
x3 = conv1d(x2, weight, kernel, stride, dilation) + bias
y = activation(x3, act_type, act_params)

input mat dims: 2d
output mat dims: 2d
one_blob_only

param id	name	type	default
0	num_output	int	0
1	kernel_w	int	0
2	dilation_w	int	1
3	stride_w	int	1
4	pad_left	int	0
5	bias_term	int	0
6	weight_data_size	int	0
9	activation_type	int	0
10	activation_params	array	[ ]
15	pad_right	int	pad_left
18	pad_value	float	0.f
19	dynamic_weight	int	0

weight	type	shape
weight_data	float/fp16/int8	[kernel_w, num_input, num_output]
bias_data	float	[num_output]

Convolution3D

x2 = pad(x, pads, pad_value)
x3 = conv3d(x2, weight, kernel, stride, dilation) + bias
y = activation(x3, act_type, act_params)

input mat dims: 4d
output mat dims: 4d
one_blob_only

param id	name	type	default
0	num_output	int	0
1	kernel_w	int	0
2	dilation_w	int	1
3	stride_w	int	1
4	pad_left	int	0
5	bias_term	int	0
6	weight_data_size	int	0
9	activation_type	int	0
10	activation_params	array	[ ]
11	kernel_h	int	kernel_w
12	dilation_h	int	dilation_w
13	stride_h	int	stride_w
14	pad_top	int	pad_left
15	pad_right	int	pad_left
16	pad_bottom	int	pad_top
17	pad_behind	int	pad_front
18	pad_value	float	0.f
21	kernel_d	int	kernel_w
22	dilation_d	int	dilation_w
23	stride_d	int	stride_w
24	pad_front	int	pad_left

weight	type	shape
weight_data	float/fp16/int8	[kernel_w, kernel_h, kernel_d, num_input, num_output]
bias_data	float	[num_output]

ConvolutionDepthWise

x2 = pad(x, pads, pad_value)
x3 = conv(x2, weight, kernel, stride, dilation, group) + bias
y = activation(x3, act_type, act_params)

input mat dims: 3d
output mat dims: 3d
one_blob_only

param id	name	type	default
0	num_output	int	0
1	kernel_w	int	0
2	dilation_w	int	1
3	stride_w	int	1
4	pad_left	int	0
5	bias_term	int	0
6	weight_data_size	int	0
7	group	int	1
8	int8_scale_term	int	0
9	activation_type	int	0
10	activation_params	array	[ ]
11	kernel_h	int	kernel_w
12	dilation_h	int	dilation_w
13	stride_h	int	stride_w
14	pad_top	int	pad_left
15	pad_right	int	pad_left
16	pad_bottom	int	pad_top
18	pad_value	float	0.f
19	dynamic_weight	int	0

weight	type	shape
weight_data	float/fp16/int8	[kernel_w, kernel_h, num_input / group, num_output / group, group]
bias_data	float	[num_output]
weight_data_int8_scales	float	[group]
bottom_blob_int8_scales	float	[1]
top_blob_int8_scales	float	[1]

ConvolutionDepthWise1D

x2 = pad(x, pads, pad_value)
x3 = conv1d(x2, weight, kernel, stride, dilation, group) + bias
y = activation(x3, act_type, act_params)

input mat dims: 2d
output mat dims: 2d
one_blob_only

param id	name	type	default
0	num_output	int	0
1	kernel_w	int	0
2	dilation_w	int	1
3	stride_w	int	1
4	pad_left	int	0
5	bias_term	int	0
6	weight_data_size	int	0
7	group	int	1
9	activation_type	int	0
10	activation_params	array	[ ]
15	pad_right	int	pad_left
18	pad_value	float	0.f
19	dynamic_weight	int	0

weight	type	shape
weight_data	float/fp16/int8	[kernel_w, num_input / group, num_output / group, group]
bias_data	float	[num_output]

ConvolutionDepthWise3D

x2 = pad(x, pads, pad_value)
x3 = conv3d(x2, weight, kernel, stride, dilation, group) + bias
y = activation(x3, act_type, act_params)

input mat dims: 4d
output mat dims: 4d
one_blob_only

param id	name	type	default
0	num_output	int	0
1	kernel_w	int	0
2	dilation_w	int	1
3	stride_w	int	1
4	pad_left	int	0
5	bias_term	int	0
6	weight_data_size	int	0
7	group	int	1
9	activation_type	int	0
10	activation_params	array	[ ]
11	kernel_h	int	kernel_w
12	dilation_h	int	dilation_w
13	stride_h	int	stride_w
14	pad_top	int	pad_left
15	pad_right	int	pad_left
16	pad_bottom	int	pad_top
17	pad_behind	int	pad_front
18	pad_value	float	0.f
21	kernel_d	int	kernel_w
22	dilation_d	int	dilation_w
23	stride_d	int	stride_w
24	pad_front	int	pad_left

weight	type	shape
weight_data	float/fp16/int8	[kernel_w, kernel_h, kernel_d, num_input / group, num_output / group, group]
bias_data	float	[num_output]

CopyTo

self[offset] = src

input mat dims: self 1d, 2d, 3d, 4d; src same rank as self
output mat dims: same shape as self
one_blob_only

param id	name	type	default
0	woffset	int	0
1	hoffset	int	0
13	doffset	int	0
2	coffset	int	0
9	starts	array	[ ]
11	axes	array	[ ]

Crop

y = crop(x)

input mat dims: 1d, 2d, 3d, 4d
output mat dims: 1d, 2d, 3d, 4d depending on crop parameters
one_blob_only

param id	name	type	default
0	woffset	int	0
1	hoffset	int	0
13	doffset	int	0
2	coffset	int	0
3	outw	int	0
4	outh	int	0
14	outd	int	0
5	outc	int	0
6	woffset2	int	0
7	hoffset2	int	0
15	doffset2	int	0
8	coffset2	int	0
9	starts	array	[ ]
10	ends	array	[ ]
11	axes	array	[ ]
19	starts_expr	str	""
20	ends_expr	str	""
21	axes_expr	str	""

CumulativeSum

If axis < 0, we use axis = x.dims + axis

It implements https://pytorch.org/docs/stable/generated/torch.cumsum.html

input mat dims: 1d, 2d, 3d, 4d
output mat dims: same as input
one_blob_only
support_inplace

param id	name	type	default	description
0	axis	int	0

Deconvolution

x2 = deconv(x, weight, kernel, stride, dilation) + bias
x3 = depad(x2, pads, pad_value)
y = activation(x3, act_type, act_params)

input mat dims: 3d
output mat dims: 3d
one_blob_only

param id	name	type	default
0	num_output	int	0
1	kernel_w	int	0
2	dilation_w	int	1
3	stride_w	int	1
4	pad_left	int	0
5	bias_term	int	0
6	weight_data_size	int	0
9	activation_type	int	0
10	activation_params	array	[ ]
11	kernel_h	int	kernel_w
12	dilation_h	int	dilation_w
13	stride_h	int	stride_w
14	pad_top	int	pad_left
15	pad_right	int	pad_left
16	pad_bottom	int	pad_top
18	output_pad_right	int	0
19	output_pad_bottom	int	output_pad_right
20	output_w	int	0
21	output_h	int	output_w
28	dynamic_weight	int	0

weight	type	shape
weight_data	float/fp16	[kernel_w, kernel_h, num_input, num_output]
bias_data	float	[num_output]

Deconvolution1D

x2 = deconv1d(x, weight, kernel, stride, dilation) + bias
x3 = depad(x2, pads, pad_value)
y = activation(x3, act_type, act_params)

input mat dims: 2d
output mat dims: 2d
one_blob_only

param id	name	type	default
0	num_output	int	0
1	kernel_w	int	0
2	dilation_w	int	1
3	stride_w	int	1
4	pad_left	int	0
5	bias_term	int	0
6	weight_data_size	int	0
9	activation_type	int	0
10	activation_params	array	[ ]
15	pad_right	int	pad_left
18	output_pad_right	int	0
20	output_w	int	0
28	dynamic_weight	int	0

weight	type	shape
weight_data	float/fp16	[kernel_w, num_input, num_output]
bias_data	float	[num_output]

Deconvolution3D

x2 = deconv3d(x, weight, kernel, stride, dilation) + bias
x3 = depad(x2, pads, pad_value)
y = activation(x3, act_type, act_params)

input mat dims: 4d
output mat dims: 4d
one_blob_only

param id	name	type	default
0	num_output	int	0
1	kernel_w	int	0
2	dilation_w	int	1
3	stride_w	int	1
4	pad_left	int	0
5	bias_term	int	0
6	weight_data_size	int	0
9	activation_type	int	0
10	activation_params	array	[ ]
11	kernel_h	int	kernel_w
12	dilation_h	int	dilation_w
13	stride_h	int	stride_w
14	pad_top	int	pad_left
15	pad_right	int	pad_left
16	pad_bottom	int	pad_top
17	pad_behind	int	pad_front
18	output_pad_right	int	0
19	output_pad_bottom	int	output_pad_right
20	output_pad_behind	int	output_pad_right
21	kernel_d	int	kernel_w
22	dilation_d	int	dilation_w
23	stride_d	int	stride_w
24	pad_front	int	pad_left
25	output_w	int	0
26	output_h	int	output_w
27	output_d	int	output_w

weight	type	shape
weight_data	float/fp16	[kernel_w, kernel_h, kernel_d, num_input, num_output]
bias_data	float	[num_output]

DeconvolutionDepthWise

x2 = deconv(x, weight, kernel, stride, dilation, group) + bias
x3 = depad(x2, pads, pad_value)
y = activation(x3, act_type, act_params)

input mat dims: 3d
output mat dims: 3d
one_blob_only

param id	name	type	default
0	num_output	int	0
1	kernel_w	int	0
2	dilation_w	int	1
3	stride_w	int	1
4	pad_left	int	0
5	bias_term	int	0
6	weight_data_size	int	0
7	group	int	1
9	activation_type	int	0
10	activation_params	array	[ ]
11	kernel_h	int	kernel_w
12	dilation_h	int	dilation_w
13	stride_h	int	stride_w
14	pad_top	int	pad_left
15	pad_right	int	pad_left
16	pad_bottom	int	pad_top
18	output_pad_right	int	0
19	output_pad_bottom	int	output_pad_right
20	output_w	int	0
21	output_h	int	output_w
28	dynamic_weight	int	0

weight	type	shape
weight_data	float/fp16	[kernel_w, kernel_h, num_input / group, num_output / group, group]
bias_data	float	[num_output]

DeconvolutionDepthWise1D

x2 = deconv1d(x, weight, kernel, stride, dilation, group) + bias
x3 = depad(x2, pads, pad_value)
y = activation(x3, act_type, act_params)

input mat dims: 2d
output mat dims: 2d
one_blob_only

param id	name	type	default
0	num_output	int	0
1	kernel_w	int	0
2	dilation_w	int	1
3	stride_w	int	1
4	pad_left	int	0
5	bias_term	int	0
6	weight_data_size	int	0
7	group	int	1
9	activation_type	int	0
10	activation_params	array	[ ]
15	pad_right	int	pad_left
18	output_pad_right	int	0
20	output_w	int	0
28	dynamic_weight	int	0

weight	type	shape
weight_data	float/fp16	[kernel_w, num_input / group, num_output / group, group]
bias_data	float	[num_output]

DeconvolutionDepthWise3D

x2 = deconv3d(x, weight, kernel, stride, dilation, group) + bias
x3 = depad(x2, pads, pad_value)
y = activation(x3, act_type, act_params)

input mat dims: 4d
output mat dims: 4d
one_blob_only

param id	name	type	default
0	num_output	int	0
1	kernel_w	int	0
2	dilation_w	int	1
3	stride_w	int	1
4	pad_left	int	0
5	bias_term	int	0
6	weight_data_size	int	0
7	group	int	1
9	activation_type	int	0
10	activation_params	array	[ ]
11	kernel_h	int	kernel_w
12	dilation_h	int	dilation_w
13	stride_h	int	stride_w
14	pad_top	int	pad_left
15	pad_right	int	pad_left
16	pad_bottom	int	pad_top
17	pad_behind	int	pad_front
18	output_pad_right	int	0
19	output_pad_bottom	int	output_pad_right
20	output_pad_behind	int	output_pad_right
21	kernel_d	int	kernel_w
22	dilation_d	int	dilation_w
23	stride_d	int	stride_w
24	pad_front	int	pad_left
25	output_w	int	0
26	output_h	int	output_w
27	output_d	int	output_w

weight	type	shape
weight_data	float/fp16	[kernel_w, kernel_h, kernel_d, num_input / group, num_output / group, group]
bias_data	float	[num_output]

DeepCopy

y = copy(x)

input mat dims: 1d, 2d, 3d, 4d
output mat dims: same as input
one_blob_only

DeformableConv2D

x2 = deformableconv2d(x, offset, mask, weight, kernel, stride, dilation) + bias
y = activation(x2, act_type, act_params)

input mat dims: bottom 3d; offset 3d; mask 3d when used
output mat dims: 3d

param id	name	type	default
0	num_output	int	0
1	kernel_w	int	0
2	dilation_w	int	1
3	stride_w	int	1
4	pad_left	int	0
5	bias_term	int	0
6	weight_data_size	int	0
9	activation_type	int	0
10	activation_params	array	[ ]
11	kernel_h	int	kernel_w
12	dilation_h	int	dilation_w
13	stride_h	int	stride_w
14	pad_top	int	pad_left
15	pad_right	int	pad_left
16	pad_bottom	int	pad_top

weight	type	shape
weight_data	float/fp16/int8	[kernel_w, kernel_h, num_input, num_output]
bias_data	float	[num_output]

DetectionOutput

y = decode_ssd(location, confidence, priorbox)

input mat dims: location 1d or 2d; confidence 1d or 2d; priorbox 1d or 2d
output mat dims: 2d when detections are produced

param id	name	type	default
0	num_class	int	0
1	nms_threshold	float	0.05f
2	nms_top_k	int	300
3	keep_top_k	int	100
4	confidence_threshold	float	0.5f

Dequantize

y = x * scale + bias

input mat dims: 1d, 2d, 3d, 4d
output mat dims: same shape as input
one_blob_only
support_inplace

param id	name	type	default	description
0	scale_data_size	int	1
1	bias_data_size	int	0

weight	type	shape
scale_data	float	[scale_data_size]
bias_data	float	[bias_data_size]

Diag

y = diag(x, diagonal)

input mat dims: 1d, 2d
output mat dims: 2d for 1d input, 1d for 2d input
one_blob_only

param id	name	type	default	description
0	diagonal	int	0

Dropout

y = x * scale

input mat dims: 1d, 2d, 3d, 4d
output mat dims: same as input
one_blob_only

param id	name	type	default	description
0	scale	float	1.f

Einsum

y = einsum(equation, x0, x1, ...)

input mat dims: 1d, 2d, 3d, 4d
output mat dims: 1d, 2d, 3d, 4d depending on equation

param id	name	type	default	description
0	equation	array	[ ]	equation string encoded as int array

Eltwise

y = elementwise_op(x0, x1, ...)

input mat dims: multiple inputs with same shape, 1d, 2d, 3d, 4d
output mat dims: same as input

param id	name	type	default	description
0	op_type	int	0
1	coeffs	array	[ ]

Operation type:

0 = PROD
1 = SUM
2 = MAX

ELU

if x < 0    y = (exp(x) - 1) * alpha
else        y = x

input mat dims: 1d, 2d, 3d, 4d
output mat dims: same as input
one_blob_only
support_inplace

param id	name	type	default	description
0	alpha	float	0.1f

Embed

y = embedding(x)

input mat dims: 1d token index vector
output mat dims: 2d

param id	name	type
0	num_output	int
1	input_dim	int
2	bias_term	int
3	weight_data_size	int
18	int8_scale_term	int

weight	type	shape
weight_data	float	[weight_data_size]
bias_term	float	[num_output]
weight_data_int8_scales	float	[1]

Erf

y = erf(x)

input mat dims: 1d, 2d, 3d, 4d
output mat dims: same as input
one_blob_only
support_inplace

Exp

if base == -1   y = exp(shift + x * scale)
else            y = pow(base, (shift + x * scale))

input mat dims: 1d, 2d, 3d, 4d
output mat dims: same as input
one_blob_only
support_inplace

param id	name	type	default
0	base	float	-1.f
1	scale	float	1.f
2	shift	float	0.f

ExpandDims

input mat dims: 1d, 2d, 3d
output mat dims: 2d, 3d, 4d depending on axes
one_blob_only

param id	name	type	default	description
3	axes	array	[ ]

Flatten

Reshape blob to 1 dimension

input mat dims: 1d, 2d, 3d, 4d
output mat dims: 1d
one_blob_only

Flip

input mat dims: 1d, 2d, 3d, 4d
output mat dims: same as input
one_blob_only

param id	name	type	default	description
0	axes	array	[ ]

Fold

y = fold(x)

input mat dims: 2d
output mat dims: 3d
one_blob_only

param id	name	type	default
0	num_output	int	0
1	kernel_w	int	0
2	dilation_w	int	1
3	stride_w	int	1
4	pad_left	int	0
11	kernel_h	int	kernel_w
12	dilation_h	int	dilation_w
13	stride_h	int	stride_w
14	pad_top	int	pad_left
15	pad_right	int	pad_left
16	pad_bottom	int	pad_top
20	output_w	int	0
21	output_h	int	output_w

GELU

if fast_gelu == 1   y = 0.5 * x * (1 + tanh(0.79788452 * (x + 0.044715 * x * x * x)));
else                y = 0.5 * x * erfc(-0.70710678 * x)

input mat dims: 1d, 2d, 3d, 4d
output mat dims: same as input
one_blob_only
support_inplace

param id	name	type	default	description
0	fast_gelu	int	0	use approximation

GLU

If axis < 0, we use axis = x.dims + axis

GLU(a,b)=a⊗σ(b)

where a is the first half of the input matrix and b is the second half.

axis specifies the dimension to split the input

input mat dims: 1d, 2d, 3d, 4d
output mat dims: same rank as input, with selected axis halved
one_blob_only

param id	name	type	default	description
0	axis	int	0

Gemm

a = transA ? transpose(x0) : x0
b = transb ? transpose(x1) : x1
c = x2
y = (gemm(a, b) + c * beta) * alpha

input mat dims: A 2d or 3d; B 2d or 3d; C optional scalar, 1d, 2d, 3d
output mat dims: 2d or 3d

param id	name	type	default	description
0	alpha	float	1.f
1	beta	float	1.f
2	transA	int	0
3	transb	int	0
4	constantA	int	0
5	constantB	int	0
6	constantC	int	0
7	constantM	int	0
8	constantN	int	0
9	constantK	int	0
10	constant_broadcast_type_C	int	0
11	output_N1M	int	0
12	output_elempack	int	0
13	output_elemtype	int	0
14	output_transpose	int	0
18	quantize_term	int	0	0=no quant, nonzero below 400=legacy int8, 4xx/6xx=weight-only block quant, 8xx=dynamic W8A8 per-block
20	constant_TILE_M	int	0
21	constant_TILE_N	int	0
22	constant_TILE_K	int	0

weight	type	shape
A_data	float/fp16/int8	[M, K] or [K, M]
B_data	float/fp16/int8	[N, K] or [K, N]
C_data	float	[1], [M] or [N] or [1, M] or [N,1] or [N, M]
A_data_int8_scales	float	[M]
B_data_int8_scales	float	[1]
B_data_quantize_scales	float	[ceil(K / block_size), N] for block quantized constant B
B_data_input_scales	float	[K] for block quantized constant B with input scale

For block quantized Gemm:

constantA=0, constantB=1, transA=0, transB=1
output is fp32 pack1 with output_N1M=0, output_elempack=0, output_transpose=0
B_data is tagged int8 bytes with shape [ceil(K * weight_bits / 8), N]
quantize_term = bits * 100 + input_scale * 10 + block_code
block_code: 0=32, 1=64, 2=128
4xx and 6xx keep A in fp32 and dequantize int4/int6 weights while accumulating
8xx dynamically quantizes A independently for every row and block, computes signed int8 dot products with the constant W8 weights, accumulates in int32, applies the per-block activation and weight descales, and writes fp32 output; there is no W8A32 compatibility path

GridSample

Given an input and a flow-field grid, computes the output using input values and pixel locations from grid.

For each output location output[:, h2, w2], the size-2 vector grid[h2, w2, 2] specifies input pixel[:, h1, w1] locations x and y, 
which are used to interpolate the output value output[:, h2, w2]

This function is often used in conjunction with affine_grid() to build Spatial Transformer Networks .

input mat dims: bottom 3d or 4d; grid 3d or 4d
output mat dims: 3d or 4d

param id	name	type	default	description
0	sample_type	int	1
1	padding_mode	int	1
2	align_corner	int	0
3	permute_fusion	int	0	fuse with permute

Sample type:

1 = Nearest
2 = Bilinear
3 = Bicubic

Padding mode:

1 = zeros
2 = border
3 = reflection

GroupNorm

split x along channel axis into group x0, x1 ...
l2 normalize for each group x0, x1 ...
y = x * gamma + beta

input mat dims: 1d, 2d, 3d, 4d
output mat dims: same as input
one_blob_only
support_inplace

param id	name	type	default	description
0	group	int	1
1	channels	int	0
2	eps	float	0.001f	x = x / sqrt(var + eps)
3	affine	int	1

weight	type	shape
gamma_data	float	[channels]
beta_data	float	[channels]

GRU

Apply a single-layer GRU to a feature sequence of T timesteps. The input blob shape is [w=input_size, h=T] and the output blob shape is [w=num_output, h=T].

y = gru(x)
y0, hidden y1 = gru(x0, hidden x1)

input mat dims: 2d sequence
output mat dims: 2d sequence
one_blob_only if bidirectional

param id	name	type	description
0	num_output	int	hidden size of output
1	weight_data_size	int	total size of weight matrix
2	direction	int	0=forward, 1=reverse, 2=bidirectional

weight	type	shape
weight_xc_data	float/fp16/int8	[input_size, num_output * 3, num_directions]
bias_c_data	float/fp16/int8	[num_output, 4, num_directions]
weight_hc_data	float/fp16/int8	[num_output, num_output * 3, num_directions]

Direction flag:

0 = forward only
1 = reverse only
2 = bidirectional

HardSigmoid

y = clamp(x * alpha + beta, 0, 1)

input mat dims: 1d, 2d, 3d, 4d
output mat dims: same as input
one_blob_only
support_inplace

param id	name	type	default	description
0	alpha	float	0.2f
1	beta	float	0.5f

HardSwish

y = x * clamp(x * alpha + beta, 0, 1)

input mat dims: 1d, 2d, 3d, 4d
output mat dims: same as input
one_blob_only
support_inplace

param id	name	type	default	description
0	alpha	float	0.2f
1	beta	float	0.5f

InnerProduct

x2 = innerproduct(x, weight) + bias
y = activation(x2, act_type, act_params)

input mat dims: 1d, 2d, 3d, 4d
output mat dims: 1d, or 2d for batched 2d input
one_blob_only

param id	name	type	default
0	num_output	int	0
1	bias_term	int	0
2	weight_data_size	int	0
8	int8_scale_term	int	0
9	activation_type	int	0
10	activation_params	array	[ ]

weight	type	shape
weight_data	float/fp16/int8	[num_input, num_output]
bias_data	float	[num_output]
weight_data_int8_scales	float	[num_output]
bottom_blob_int8_scales	float	[1]

Input

y = input

input mat dims: none
output mat dims: 1d, 2d, 3d, 4d depending on shape parameters
support_inplace

param id	name	type
0	w	int
1	h	int
11	d	int
2	c	int

InstanceNorm

split x along channel axis into instance x0, x1 ...
l2 normalize for each channel instance x0, x1 ...
y = x * gamma + beta

input mat dims: 3d, 4d
output mat dims: same as input
one_blob_only
support_inplace

param id	name	type	default	description
0	channels	int	0
1	eps	float	0.001f	x = x / sqrt(var + eps)
2	affine	int	1

weight	type	shape
gamma_data	float	[channels]
beta_data	float	[channels]

Interp

if dynamic_target_size == 0     y = resize(x) by fixed size or scale
else                            y = resize(x0, size(x1))

input mat dims: 3d
output mat dims: 3d
one_blob_only if dynamic_target_size == 0

param id	name	type	default
0	resize_type	int	0
1	height_scale	float	1.f
2	width_scale	float	1.f
3	output_height	int	0
4	output_width	int	0
5	dynamic_target_size	int	0
6	align_corner	int	0
9	size_expr	str	""

Resize type:

1 = Nearest
2 = Bilinear
3 = Bicubic

InverseSpectrogram

x1 = x as complex
x1 = x1 * sqrt(norm) if normalized
y = istft(x1)
y1 = unpad(y) if center

if returns == 0 return y1 as complex
if returns == 1 return y1 real
if returns == 2 return y1 imag

input mat dims: 2d, 3d
output mat dims: 1d
one_blob_only

param id	name	type	default	description
0	n_fft	int	0
1	returns	int	1
2	hoplen	int	n_fft / 4
3	winlen	int	n_fft
4	window_type	int	0	0=ones 1=hann 2=hamming
5	center	int	1
7	normalized	int	0	0=no 1=n_fft 2=window-l2-energy

LayerNorm

split x along outmost axis into part x0, x1 ...
l2 normalize for each part x0, x1 ...
y = x * gamma + beta by elementwise

input mat dims: 1d, 2d, 3d, 4d
output mat dims: same as input
one_blob_only
support_inplace

param id	name	type	default	description
0	affine_size	int	0
1	eps	float	0.001f	x = x / sqrt(var + eps)
2	affine	int	1

weight	type	shape
gamma_data	float	[affine_size]
beta_data	float	[affine_size]

Log

if base == -1   y = log(shift + x * scale)
else            y = log(shift + x * scale) / log(base)

input mat dims: 1d, 2d, 3d, 4d
output mat dims: same as input
one_blob_only
support_inplace

param id	name	type	default
0	base	float	-1.f
1	scale	float	1.f
2	shift	float	0.f

LRN

if region_type == ACROSS_CHANNELS   square_sum = sum of channel window of local_size
if region_type == WITHIN_CHANNEL    square_sum = sum of spatial window of local_size
y = x * pow(bias + alpha * square_sum / (local_size * local_size), -beta)

input mat dims: 3d
output mat dims: 3d
one_blob_only
support_inplace

param id	name	type	default
0	region_type	int	0
1	local_size	int	5
2	alpha	float	1.f
3	beta	float	0.75f
4	bias	float	1.f

Region type:

0 = ACROSS_CHANNELS
1 = WITHIN_CHANNEL

LSTM

Apply a single-layer LSTM to a feature sequence of T timesteps. The input blob shape is [w=input_size, h=T] and the output blob shape is [w=num_output, h=T].

y = lstm(x)
y0, hidden y1, cell y2 = lstm(x0, hidden x1, cell x2)

input mat dims: 2d sequence
output mat dims: 2d sequence
one_blob_only if bidirectional

param id	name	type	default	description
0	num_output	int	0	output size of output
1	weight_data_size	int	0	total size of IFOG weight matrix
2	direction	int	0	0=forward, 1=reverse, 2=bidirectional
3	hidden_size	int	num_output	hidden size

weight	type	shape
weight_xc_data	float/fp16/int8	[input_size, hidden_size * 4, num_directions]
bias_c_data	float/fp16/int8	[hidden_size, 4, num_directions]
weight_hc_data	float/fp16/int8	[num_output, hidden_size * 4, num_directions]
weight_hr_data	float/fp16/int8	[hidden_size, num_output, num_directions]

Direction flag:

0 = forward only
1 = reverse only
2 = bidirectional

MatMul

y = matmul(A, B)

input mat dims: A 1d, 2d, 3d, 4d; B 1d, 2d, 3d, 4d
output mat dims: 1d, 2d, 3d, 4d depending on input ranks and batch broadcast

param id	name	type	default	description
0	transB	int	0

MemoryData

y = data

input mat dims: none
output mat dims: 1d, 2d, 3d, 4d depending on shape parameters

param id	name	type	default	description
0	w	int	0
1	h	int	0
11	d	int	0
2	c	int	0
21	load_type	int	1	1=fp32, 3=int8

weight	type	shape
data	float/int8	[w, h, d, c]

For load_type=3, data consists of raw int8 bytes without a type tag, padded with zeros to a multiple of 4 bytes.

Mish

y = x * tanh(log(exp(x) + 1))

input mat dims: 1d, 2d, 3d, 4d
output mat dims: same as input
one_blob_only
support_inplace

MultiHeadAttention

q_affine = affine(q) / (embed_dim / num_head)
k_affine = affine(k) or reuse kv_cache part
v_affine = affine(v) or reuse kv_cache part
split q k v into num_head part q0, k0, v0, q1, k1, v1 ...
for each num_head part
    qk = q * k
    qk = qk + attn_mask if attn_mask exists
    softmax(qk)
    qkv = qk * v
    merge qkv to out
y = affine(out)

input mat dims: q 2d or 3d; k 2d or 3d; v 2d or 3d
output mat dims: 2d or 3d

param id	name	type	default	description
0	embed_dim	int	0
1	num_heads	int	1
2	weight_data_size	int	0	qdim = weight_data_size / embed_dim
3	kdim	int	embed_dim
4	vdim	int	embed_dim
5	attn_mask	int	0
6	scale	float	1.f / sqrt(embed_dim / num_heads)
7	kv_cache	int	0
18	quantize_term	int	0	0=no quant, nonzero below 400=legacy int8, 4xx/6xx=weight-only block quant, 8xx=dynamic W8A8 per-block

weight	type	shape
q_weight_data	float/fp16/int8	[embed_dim * qdim]
q_bias_data	float	[embed_dim]
k_weight_data	float/fp16/int8	[embed_dim * kdim]
k_bias_data	float	[embed_dim]
v_weight_data	float/fp16/int8	[embed_dim * vdim]
v_bias_data	float	[embed_dim]
out_weight_data	float/fp16/int8	[qdim * embed_dim]
out_bias_data	float	[qdim]
q_weight_data_int8_scales	float	[embed_dim]
k_weight_data_int8_scales	float	[embed_dim]
v_weight_data_int8_scales	float	[embed_dim]
out_weight_data_int8_scales	float	[1]
q_weight_data_quantize_scales	float	[ceil(qdim / block_size), embed_dim] for block quantized weight
k_weight_data_quantize_scales	float	[ceil(kdim / block_size), embed_dim] for block quantized weight
v_weight_data_quantize_scales	float	[ceil(vdim / block_size), embed_dim] for block quantized weight
out_weight_data_quantize_scales	float	[ceil(embed_dim / block_size), qdim] for block quantized weight
q_weight_data_input_scales	float	[qdim] for block quantized weight with input scale
k_weight_data_input_scales	float	[kdim] for block quantized weight with input scale
v_weight_data_input_scales	float	[vdim] for block quantized weight with input scale
out_weight_data_input_scales	float	[embed_dim] for block quantized weight with input scale

Block quantized MultiHeadAttention stores q/k/v/out weights as tagged int8 bytes. The quantize_term rule is the same as Gemm: 4xx and 6xx are weight-only, while 8xx dynamically quantizes the projection activations per row and block and runs W8A8 projection Gemm. The portable base layer computes the complete MHA directly without creating other layers; architecture-derived CPU layers route only the q/k/v/out projections through their persistent optimized Gemm pipelines. QK, softmax, PV, KV-cache storage, and the fp32 output contract are unchanged.

MVN

if normalize_variance == 1 && across_channels == 1      y = (x - mean) / (sqrt(var) + eps) of whole blob
if normalize_variance == 1 && across_channels == 0      y = (x - mean) / (sqrt(var) + eps) of each channel
if normalize_variance == 0 && across_channels == 1      y = x - mean of whole blob
if normalize_variance == 0 && across_channels == 0      y = x - mean of each channel

input mat dims: 3d, 4d
output mat dims: same as input
one_blob_only

param id	name	type	default	description
0	normalize_variance	int	0
1	across_channels	int	0
2	eps	float	0.0001f	x = x / (sqrt(var) + eps)

Noop

y = x

input mat dims: 1d, 2d, 3d, 4d
output mat dims: same as input

Normalize

if across_spatial == 1 && across_channel == 1      x2 = normalize(x) of whole blob
if across_spatial == 1 && across_channel == 0      x2 = normalize(x) of each channel
if across_spatial == 0 && across_channel == 1      x2 = normalize(x) of each position
y = x2 * scale

input mat dims: 3d, 4d
output mat dims: same as input
one_blob_only
support_inplace

param id	name	type	default	description
0	across_spatial	int	0
1	channel_shared	int	0
2	eps	float	0.0001f	see eps mode
3	scale_data_size	int	0
4	across_channel	int	0
9	eps_mode	int	0

weight	type	shape
scale_data	float	[scale_data_size]

Eps Mode:

0 = caffe/mxnet x = x / sqrt(var + eps)
1 = pytorch x = x / max(sqrt(var), eps)
2 = tensorflow x = x / sqrt(max(var, eps))

Packing

y = wrap_packing(x)

input mat dims: 1d, 2d, 3d, 4d
output mat dims: same shape as input
one_blob_only

param id	name	type	default
0	out_elempack	int	1
1	use_padding	int	0
2	cast_type_from	int	0
3	cast_type_to	int	0
4	storage_type_from	int	0
5	storage_type_to	int	0

Padding

y = pad(x, pads)

input mat dims: 1d, 2d, 3d, 4d
output mat dims: same rank as input

param id	name	type	default
0	top	int	0
1	bottom	int	0
2	left	int	0
3	right	int	0
4	type	int	0
5	value	float	0
6	per_channel_pad_data_size	int	0
7	front	int	stride_w
8	behind	int	pad_left

weight	type	shape
per_channel_pad_data	float	[per_channel_pad_data_size]

Padding type:

0 = CONSTANT
1 = REPLICATE
2 = REFLECT

Permute

y = reorder(x)

input mat dims: 1d, 2d, 3d, 4d
output mat dims: 1d, 2d, 3d, 4d depending on order

param id	name	type	default	description
0	order_type	int	0

Order Type:

0 = WH WHC WHDC
1 = HW HWC HWDC
2 = WCH WDHC
3 = CWH DWHC
4 = HCW HDWC
5 = CHW DHWC
6 = WHCD
7 = HWCD
8 = WCHD
9 = CWHD
10 = HCWD
11 = CHWD
12 = WDCH
13 = DWCH
14 = WCDH
15 = CWDH
16 = DCWH
17 = CDWH
18 = HDCW
19 = DHCW
20 = HCDW
21 = CHDW
22 = DCHW
23 = CDHW

PixelShuffle

if mode == 0    y = depth_to_space(x) where x channel order is sw-sh-outc
if mode == 1    y = depth_to_space(x) where x channel order is outc-sw-sh

input mat dims: 3d
output mat dims: 3d
one_blob_only

param id	name	type	default	description
0	upscale_factor	int	1
1	mode	int	0

Pooling

x2 = pad(x, pads)
x3 = pooling(x2, kernel, stride)

input mat dims: 3d
output mat dims: 3d

param id	name	type	default
0	pooling_type	int	0
1	kernel_w	int	0
2	stride_w	int	1
3	pad_left	int	0
4	global_pooling	int	0
5	pad_mode	int	0
6	avgpool_count_include_pad	int	0
7	adaptive_pooling	int	0
8	out_w	int	0
11	kernel_h	int	kernel_w
12	stride_h	int	stride_w
13	pad_top	int	pad_left
14	pad_right	int	pad_left
15	pad_bottom	int	pad_top
18	out_h	int	out_w

Pooling type:

0 = MAX
1 = AVG

Pad mode:

0 = full padding
1 = valid padding
2 = tensorflow padding=SAME or onnx padding=SAME_UPPER
3 = onnx padding=SAME_LOWER

Pooling1D

x2 = pad(x, pads)
x3 = pooling1d(x2, kernel, stride)

input mat dims: 2d
output mat dims: 2d

param id	name	type	default
0	pooling_type	int	0
1	kernel_w	int	0
2	stride_w	int	1
3	pad_left	int	0
4	global_pooling	int	0
5	pad_mode	int	0
6	avgpool_count_include_pad	int	0
7	adaptive_pooling	int	0
8	out_w	int	0
14	pad_right	int	pad_left

Pooling type:

0 = MAX
1 = AVG

Pad mode:

0 = full padding
1 = valid padding
2 = tensorflow padding=SAME or onnx padding=SAME_UPPER
3 = onnx padding=SAME_LOWER

Pooling3D

x2 = pad(x, pads)
x3 = pooling3d(x2, kernel, stride)

input mat dims: 4d
output mat dims: 4d

param id	name	type	default
0	pooling_type	int	0
1	kernel_w	int	0
2	stride_w	int	1
3	pad_left	int	0
4	global_pooling	int	0
5	pad_mode	int	0
6	avgpool_count_include_pad	int	0
7	adaptive_pooling	int	0
8	out_w	int	0
11	kernel_h	int	kernel_w
12	stride_h	int	stride_w
13	pad_top	int	pad_left
14	pad_right	int	pad_left
15	pad_bottom	int	pad_top
16	pad_behind	int	pad_front
18	out_h	int	out_w
21	kernel_d	int	kernel_w
22	stride_d	int	stride_w
23	pad_front	int	pad_left
28	out_d	int	out_w

Pooling type:

0 = MAX
1 = AVG

Pad mode:

0 = full padding
1 = valid padding
2 = tensorflow padding=SAME or onnx padding=SAME_UPPER
3 = onnx padding=SAME_LOWER

Power

y = pow((shift + x * scale), power)

input mat dims: 1d, 2d, 3d, 4d
output mat dims: same as input
one_blob_only
support_inplace

param id	name	type	default
0	power	float	1.f
1	scale	float	1.f
2	shift	float	0.f

PriorBox

y = generate_priorbox(feature, image)

input mat dims: feature 3d; image 3d when used
output mat dims: 1d for mxnet multibox prior, otherwise 2d

PReLU

if x < 0    y = x * slope
else        y = x

input mat dims: 1d, 2d, 3d, 4d
output mat dims: same as input
one_blob_only
support_inplace

param id	name	type	default	description
0	num_slope	int	0

weight	type	shape
slope_data	float	[num_slope]

Proposal

y = generate_proposals(score, bbox, im_info)

input mat dims: score 3d; bbox 3d; im_info 1d
output mat dims: roi 3d; roi_score 3d when used

PSROIPooling

y = psroipooling(x, roi)

input mat dims: bottom 3d; roi 1d
output mat dims: 3d

param id	name	type	default
0	pooled_width	int	7
1	pooled_height	int	7
2	spatial_scale	float	0.0625f
3	output_dim	int	0

Quantize

y = float2int8(x * scale)

input mat dims: 1d, 2d, 3d, 4d
output mat dims: same shape as input
one_blob_only

param id	name	type	default	description
0	scale_data_size	int	1

weight	type	shape
scale_data	float	[scale_data_size]

Reduction

y = reduce_op(x * coeff)

input mat dims: 1d, 2d, 3d, 4d
output mat dims: 1d, 2d, 3d, 4d depending on axes and keepdims
one_blob_only

param id	name	type	default	description
0	operation	int	0
1	reduce_all	int	1
2	coeff	float	1.f
3	axes	array	[ ]
4	keepdims	int	0
5	fixbug0	int	0	hack for bug fix, should be 1

Operation type:

0 = SUM
1 = ASUM
2 = SUMSQ
3 = MEAN
4 = MAX
5 = MIN
6 = PROD
7 = L1
8 = L2
9 = LogSum
10 = LogSumExp

ReLU

if x < 0    y = x * slope
else        y = x

input mat dims: 1d, 2d, 3d, 4d
output mat dims: same as input
one_blob_only
support_inplace

param id	name	type	default	description
0	slope	float	0.f

Reorg

if mode == 0    y = space_to_depth(x) where x channel order is sw-sh-outc
if mode == 1    y = space_to_depth(x) where x channel order is outc-sw-sh

input mat dims: 3d
output mat dims: 3d
one_blob_only

param id	name	type	default	description
0	stride	int	1
1	mode	int	0

Requantize

x2 = x * scale_in + bias
x3 = activation(x2)
y = float2int8(x3 * scale_out)

input mat dims: 1d, 2d, 3d, 4d
output mat dims: same shape as input
one_blob_only

param id	name	type	default
0	scale_in_data_size	int	1
1	scale_out_data_size	int	1
2	bias_data_size	int	0
3	activation_type	int	0
4	activation_params	int	[ ]

weight	type	shape
scale_in_data	float	[scale_in_data_size]
scale_out_data	float	[scale_out_data_size]
bias_data	float	[bias_data_size]

Reshape

y = reshape(x)

input mat dims: 1d, 2d, 3d, 4d
output mat dims: 1d, 2d, 3d, 4d depending on output shape parameters
one_blob_only

param id	name	type	default
0	w	int	-233
1	h	int	-233
11	d	int	-233
2	c	int	-233
6	shape_expr	str	""

Reshape flag:

0 = copy from bottom
-1 = remaining
-233 = drop this dim(default)

RMSNorm

split x along outmost axis into part x0, x1 ...
root mean square normalize for each part x0, x1 ...
y = x * gamma by elementwise

input mat dims: 1d, 2d, 3d, 4d
output mat dims: same as input
one_blob_only
support_inplace

param id	name	type	default	description
0	affine_size	int	0
1	eps	float	0.001f	x = x / sqrt(var + eps)
2	affine	int	1

weight	type	shape
gamma_data	float	[affine_size]

RNN

Apply a single-layer RNN to a feature sequence of T timesteps. The input blob shape is [w=input_size, h=T] and the output blob shape is [w=num_output, h=T].

y = rnn(x)
y0, hidden y1 = rnn(x0, hidden x1)

input mat dims: 2d sequence
output mat dims: 2d sequence
one_blob_only if bidirectional

param id	name	type	description
0	num_output	int	hidden size of output
1	weight_data_size	int	total size of weight matrix
2	direction	int	0=forward, 1=reverse, 2=bidirectional

weight	type	shape
weight_xc_data	float/fp16/int8	[input_size, num_output, num_directions]
bias_c_data	float/fp16/int8	[num_output, 1, num_directions]
weight_hc_data	float/fp16/int8	[num_output, num_output, num_directions]

Direction flag:

0 = forward only
1 = reverse only
2 = bidirectional

ROIAlign

y = roialign(x, roi)

input mat dims: bottom 3d; roi 1d
output mat dims: 3d

param id	name	type	default
0	pooled_width	int	0
1	pooled_height	int	0
2	spatial_scale	float	1.f
3	sampling_ratio	int	0
4	aligned	int	0
5	version	int	0

ROIPooling

y = roipooling(x, roi)

input mat dims: bottom 3d; roi 1d
output mat dims: 3d

param id	name	type	default
0	pooled_width	int	0
1	pooled_height	int	0
2	spatial_scale	float	1.f

RotaryEmbed

Apply rotary positional embeddings with cos and sin cache

y1 = x1 * cos - x2 * sin
y2 = x1 * sin + x2 * cos

input mat dims: bottom 3d; cos cache 2d; sin cache 2d
output mat dims: 3d

param id	name	type	default	description
0	interleaved	int	0

Scale

if scale_data_size == -233  y = x0 * x1
else                        y = x * scale + bias

input mat dims: 1d, 2d, 3d, 4d
output mat dims: same as input
one_blob_only if scale_data_size != -233
support_inplace

param id	name	type	default	description
0	scale_data_size	int	0
1	bias_term	int	0

weight	type	shape
scale_data	float	[scale_data_size]
bias_data	float	[scale_data_size]

SDPA

scaled dot product attention
for each num_head part
    qk = q * k
    qk = qk + attn_mask if attn_mask exists
    softmax(qk)
    qkv = qk * v

input mat dims: q 3d; k 3d; v 3d; attn_mask optional 2d
attn_mask values must be finite; use a large negative value for masked positions and keep at least one key unmasked for each query
output mat dims: 3d

param id	name	type	default	description
5	attn_mask	int	0
6	scale	float	0.f	auto = 1.f / sqrt(embed_dim)
7	kv_cache	int	0
18	int8_scale_term	int	0

SELU

if x < 0    y = (exp(x) - 1.f) * alpha * lambda
else        y = x * lambda

input mat dims: 1d, 2d, 3d, 4d
output mat dims: same as input
one_blob_only
support_inplace

param id	name	type	default	description
0	alpha	float	1.67326324f
1	lambda	float	1.050700987f

Shrink

if x < -lambd y = x + bias
if x >  lambd y = x - bias
else          y = x

input mat dims: 1d, 2d, 3d, 4d
output mat dims: same as input
one_blob_only
support_inplace

param id	name	type	default	description
0	bias	float	0.0f
1	lambd	float	0.5f

ShuffleChannel

if reverse == 0     y = shufflechannel(x) by group
if reverse == 1     y = shufflechannel(x) by channel / group

input mat dims: 3d, 4d
output mat dims: same as input
one_blob_only

param id	name	type	default	description
0	group	int	1
1	reverse	int	0

Sigmoid

y = 1 / (1 + exp(-x))

input mat dims: 1d, 2d, 3d, 4d
output mat dims: same as input
one_blob_only
support_inplace

Slice

split x along axis into slices, each part slice size is based on slices array

input mat dims: 1d, 2d, 3d, 4d
output mat dims: multiple outputs with same rank as input

param id	name	type	default
0	slices	array	[ ]
1	axis	int	0
2	indices	array	[ ]

Softmax

softmax(x, axis)

input mat dims: 1d, 2d, 3d, 4d
output mat dims: same as input
one_blob_only
support_inplace

param id	name	type	default	description
0	axis	int	0
1	fixbug0	int	0	hack for bug fix, should be 1

Softplus

y = log(exp(x) + 1)

input mat dims: 1d, 2d, 3d, 4d
output mat dims: same as input
one_blob_only
support_inplace

Spectrogram

x1 = pad(x) if center
y = stft(x1)
y = y / sqrt(norm) if normalized

if power == 0 return y as real
if power == 1 return magnitude
if power == 2 return square of magnitude

input mat dims: 1d
output mat dims: 2d, or 3d when power=0 returns real and imaginary components
one_blob_only

param id	name	type	default	description
0	n_fft	int	0
1	power	int	0
2	hoplen	int	n_fft / 4
3	winlen	int	n_fft
4	window_type	int	0	0=ones 1=hann 2=hamming
5	center	int	1
6	pad_type	int	2	0=CONSTANT 1=REPLICATE 2=REFLECT
7	normalized	int	0	0=no 1=n_fft 2=window-l2-energy
8	onesided	int	1

SPP

y = spatial_pyramid_pooling(x)

input mat dims: 3d
output mat dims: 3d
one_blob_only

param id	name	type	default	description
0	pooling_type	int	0
1	pyramid_height	int	1

Split

y0, y1 ... = x

input mat dims: 1d, 2d, 3d, 4d
output mat dims: multiple outputs same as input

StatisticsPooling

y = mean(x) or concat(mean(x), stddev(x))

input mat dims: 3d, 4d
output mat dims: 1d
one_blob_only

param id	name	type	default	description
0	include_stddev	int	0

Squeeze

input mat dims: 1d, 2d, 3d, 4d
output mat dims: 1d, 2d, 3d, 4d depending on axes
one_blob_only

param id	name	type	default
0	squeeze_w	int	0
1	squeeze_h	int	0
11	squeeze_d	int	0
2	squeeze_c	int	0
3	axes	array	[ ]

Swish

y = x / (1 + exp(-x))

input mat dims: 1d, 2d, 3d, 4d
output mat dims: same as input
one_blob_only
support_inplace

TanH

y = tanh(x)

input mat dims: 1d, 2d, 3d, 4d
output mat dims: same as input
one_blob_only
support_inplace

Threshold

if x > threshold    y = 1
else                y = 0

input mat dims: 1d, 2d, 3d, 4d
output mat dims: same as input
one_blob_only
support_inplace

param id	name	type	default	description
0	threshold	float	0.f

Tile

y = repeat tiles along axis for x

input mat dims: 1d, 2d, 3d, 4d
output mat dims: 1d, 2d, 3d, 4d depending on repeat parameters
one_blob_only

param id	name	type	default
0	axis	int	0
1	tiles	int	1
2	repeats	array	[ ]

UnaryOp

y = unaryop(x)

input mat dims: 1d, 2d, 3d, 4d
output mat dims: same as input

one_blob_only
support_inplace

param id	name	type	default	description
0	op_type	int	0	Operation type as follows

Operation type:

0 = ABS
1 = NEG
2 = FLOOR
3 = CEIL
4 = SQUARE
5 = SQRT
6 = RSQ
7 = EXP
8 = LOG
9 = SIN
10 = COS
11 = TAN
12 = ASIN
13 = ACOS
14 = ATAN
15 = RECIPROCAL
16 = TANH
17 = LOG10
18 = ROUND
19 = TRUNC

Unfold

y = unfold(x)

input mat dims: 3d
output mat dims: 2d
one_blob_only

param id	name	type	default
0	num_output	int	0
1	kernel_w	int	0
2	dilation_w	int	1
3	stride_w	int	1
4	pad_left	int	0
11	kernel_h	int	kernel_w
12	dilation_h	int	dilation_w
13	stride_h	int	stride_w
14	pad_top	int	pad_left
15	pad_right	int	pad_left
16	pad_bottom	int	pad_top

YoloDetectionOutput

y = yolo_detection_output(x0, x1, ...)

input mat dims: multiple 3d feature maps
output mat dims: 2d when detections are produced

param id	name	type	default
0	num_class	int	20
1	num_box	int	5
2	confidence_threshold	float	0.01f
3	nms_threshold	float	0.45f
4	biases	array	[ ]

Yolov3DetectionOutput

y = yolov3_detection_output(x0, x1, ...)

input mat dims: multiple 3d feature maps
output mat dims: 2d when detections are produced

param id	name	type	default
0	num_class	int	20
1	num_box	int	5
2	confidence_threshold	float	0.01f
3	nms_threshold	float	0.45f
4	biases	array	[ ]
5	mask	array	[ ]
6	anchors_scale	array	[ ]

90 KiB Raw Permalink Blame History

AbsVal

ArgMax

BatchNorm

Bias

BinaryOp

BNLL

Cast

CELU

Clip

Concat

Convolution

Convolution1D

Convolution3D

ConvolutionDepthWise

ConvolutionDepthWise1D

ConvolutionDepthWise3D

CopyTo

Crop

CumulativeSum

Deconvolution

Deconvolution1D

Deconvolution3D

DeconvolutionDepthWise

DeconvolutionDepthWise1D

DeconvolutionDepthWise3D

DeepCopy

DeformableConv2D

DetectionOutput

Dequantize

Diag

Dropout

Einsum

Eltwise

ELU

Embed

Erf

Exp

ExpandDims

Flatten

Flip

Fold

GELU

GLU

Gemm

GridSample

GroupNorm

GRU

HardSigmoid

HardSwish

InnerProduct

Input

InstanceNorm

Interp

InverseSpectrogram

LayerNorm

Log

LRN

LSTM

MatMul

MemoryData

Mish

MultiHeadAttention

MVN

Noop

Normalize

Packing

Padding

Permute

PixelShuffle

Pooling

Pooling1D

Pooling3D

Power

PriorBox

PReLU

Proposal

PSROIPooling

Quantize

Reduction

90 KiB

Raw Permalink Blame History