mirror of
https://github.com/ggml-org/llama.cpp.git
synced 2026-09-24 13:37:01 +02:00
b11055
1043
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 chiheb ben cheikh
|
eb1e1f495f | json-schema : accept escaped hyphen in regex patterns (#29127) | ||
 Georgi Gerganov
|
5b59b83f4e |
metal : add MoE and SSM_CONV fusion optimizations (#28948)
* metal : add top-k MoE fusion Adds a Metal fusion for SOFT_MAX + ARGSORT + GET_ROWS with optional routing-weight normalization and scale, matching the top-k MoE fusion available in the CUDA and Vulkan backends. The fused kernel writes the selected expert ids and routing weights directly, eliding the separate softmax, argsort, get-rows, sum-rows, clamp, div and scale kernels. Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp * metal : add MoE weighted reduction fusion Fuses MUL(experts, weights) plus the expert VIEW/ADD chain into one kernel that computes the weighted sum directly. The graph_optimize hook keeps the expert and weight buffers alive until the fused output so the allocator cannot reuse them while the kernel is still reading them. Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp * tests : expose MoE weighted reduction in fusion baseline Use 2 experts per token in the generated MoE test models so the Metal MoE weighted reduction fusion (MUL + ADD) is exercised by test-fusion. Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp * metal : fuse RMS_NORM + SCALE Adds NORM/RMS_NORM + SCALE fusion to the Metal backend by reusing the norm+mul kernel with a scalar scale flag. Adds test coverage for both NORM+SCALE and RMS_NORM+SCALE and regenerates the fusion baseline. Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp * metal : use function constant for RMS_NORM + SCALE Replaces the runtime use_scale karg with a Metal function constant. The norm+mul kernel is compiled with FC_norm_use_scale=false for MUL fusion and FC_norm_use_scale=true for SCALE fusion, so the fused kernel has no runtime branch. Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp * metal : use function constant for top-k MoE with_norm Replaces the runtime with_norm karg with a Metal function constant. The top-k MoE kernel is compiled separately for the normalized and non-normalized routing variants, removing the runtime branch. Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp * metal : rename moe_weighted_reduction suffix to moe_reduce Shortens the MoE weighted-reduction fusion identifiers, kernel, pipeline, matcher, args struct, and test op name from moe_weighted_reduction to moe_reduce. Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp * metal : add MUL_MAT + UNARY and MUL_MAT + ADD + UNARY fusion Adds dense mat-vec activation fusion for sigmoid/silu and bias+softplus. The mat-vec kernels apply the activation/bias epilogue via function constants, avoiding the separate unary/add passes. Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp * metal : revert MUL_MAT + UNARY and MUL_MAT + ADD + UNARY fusion The mat-vec activation fusion regressed decode throughput on Qwen3.6-35B-A3B by ~8% (tg32 81.5 vs 88.5 t/s). The regression is caused by loss of concurrency: the standalone unary kernels previously overlapped with other mat-vec work, while fusing the activation into the mat-vec kernel serializes it on the critical path. Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp * metal : add SSM_CONV + UNARY (silu) fusion The SSM_CONV kernels apply silu directly via a function constant, eliding the separate unary pass. Regenerates the fusion baseline. Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp * metal : address fusion review comments - Fix declaration/table alignment - Rename top-k MoE kargs fields to val_clamp / val_scale - Move moe-reduce alloc-deps handling into a general fusion helper - Remove the public moe-reduce matcher API Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp * metal : fix unused parameter in top-k MoE fusion check Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp * metal : guard SSM_CONV fusion lookup behind use_fusion Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp * metal : track all fused outputs in graph reorder Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp * metal : keep top-k MoE logits alive until fused output Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp * metal : refactor alloc deps to pattern-driven approach Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp * metal : check fused kernel destination in concurrency tracking Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp * meta : forward graph_optimize to underlying backends Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp * metal : use vector for fusion table Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp * meta : keep graph_optimize unimplemented Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp * parallel : fix non-deterministic prompt selection Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp * parallel : support dummy models and add global logits run hash Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp * metal : sync cross-device copies with destination completion event Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp * metal : avoid const_cast in fusion alloc deps Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp * metal : skip fusions with aliased sources Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp * metal : hide fusion pattern definition Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp * metal : use vector fusion op sequences Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp * metal : drop redundant struct keywords Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp * metal : add alloc deps comment separator Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp * metal : generalize fusion output memory ranges Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp * metal : rename fusion out_offsets to outs Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp * metal : avoid dst vector in memory range check Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp * metal : optimize fusion matching and multi-output handling - use pointer arithmetic for fusion info count lookup - avoid heap allocations in top-k MoE and MoE reduce pattern matchers - use fusion outs for multi-output subgraph checks Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp * Revert "parallel : support dummy models and add global logits run hash" This reverts commit 57c7caf941c1b43c270fd5009c9f175063522e96. * fusion : update MTL.csv * metal : unroll constant loops in top-k MoE kernel Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp * metal : use function constants for top-k MoE n_expert and top_k Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp * metal : rename fusion kargs to scale and clamp Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp * metal : use function constants for moe_reduce and ssm_conv Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp * fusion : update MTL.csv |
||
|
Georgi Gerganov
|
efa28e950e |
test-llama-archs : generate dummy test vocab (#29084)
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp |
||
 Todor Boinovski
|
18a04f09c2 |
hexagon: HMX flash-attention head_dim padding (support DK=DV=72) (#26539)
Allow HMX flash-attention to run with head_dim not a multiple of 64 (e.g. SigLIP head_dim=72), by operating on DK/DV rounded up to 64 with zero-filled tail lanes. |
||
 Masashi Yoshimura
|
44be98f057 |
ggml-webgpu: fix supports_op condition for GET_ROWS (#28978)
* fix get_rows vec4 handling * Add src strides checking to vec4_aligned of get_rows and the new test case. |
||
 Nikita Gordeev
|
8ed1a55efc |
cmake : fix build when GGML_CPU=OFF and GGML_CUDA=ON (#29026)
* fix: build fails when GGML_CPU=OFF and GGML_CUDA=ON * fix: eol in examples/convert-llama2c-to-ggml/CMakeLists.txt file |
||
 drluoto
|
5c53396b89 |
vulkan: raise the hoisted row-id limit for mul_mat_id from 256 to 512 experts (#28501)
* vulkan: raise the hoisted row-id limit for mul_mat_id to 512 experts The expert-count shader (count_experts.comp) sizes its shared arrays with BLOCK_SIZE, which is 256. Because of that, row-id hoisting is switched off for any model with more than 256 experts, and every mul_mat_id workgroup has to rescan the whole ids tensor on its own. Qwen3.8-Flash-Next has 512 experts and was quietly running on that slow path. This change sizes the arrays with a separate MAX_EXPERTS constant (512), clears them in a loop instead of one entry per thread, and raises the matching limit on the host side. On Strix Halo at batch 2048 the expert matmuls drop from 12.5 to 9.5 ms (iq3_s) and from 14.0 to 7.5 ms (iq4_nl) per op, and prompt processing gets about 19 % faster at 8k tokens. test-backend-ops MUL_MAT_ID passes (891/891) with new 512-expert test cases. Assisted-by: Claude Fable 5.1 * vulkan: raise the hoisted row-id limit for mul_mat_id to 1024 experts Follow-up to review feedback: 1024 matches LLAMA_MAX_EXPERTS instead of stopping at 512. The three shared arrays in count_experts.comp grow to 3 * 1024 * 4 = 12 KiB, which fits the 16 KiB that Vulkan guarantees for maxComputeSharedMemorySize. Adds mul_mat_id test cases at 1024 experts alongside the existing 512 ones. test-backend-ops MUL_MAT_ID passes on Vulkan (RADV, Strix Halo, Radeon 8060S): 889/889. |
||
  Yuri KhrustalevandJohannes Gäßler
|
81aeaeb74b |
gguf : align the data section relative to the GGUF start, not the file (#28993)
* gguf : align the data section relative to the GGUF start, not the file gguf_init_from_file_ptr reads a GGUF from the current file position, but padded the data section from file offset 0, so a GGUF embedded at an offset that is not a multiple of the alignment loaded without error and returned wrong tensor data. Also adds llama_adapter_lora_init_from_file_ptr, and disables mmap with a warning when an embedded data section is not aligned, instead of asserting in ggml. Assisted-by: Claude Opus 5 * llama : load lora from path through the FILE* variant The test now checks that mmap is disabled only for an unaligned offset. Assisted-by: Claude Fable 5.1 * Update ggml/src/gguf.cpp Co-authored-by: Johannes Gäßler <[email protected]> * Update include/llama.h Co-authored-by: Johannes Gäßler <[email protected]> * llama : error on unaligned mmap of an embedded GGUF, drop test-load-file-ptr --------- Co-authored-by: Johannes Gäßler <[email protected]> |
||
  GeorgeandSigbjørn Skjæret
|
7d6f5d02bb |
model : add support for HrmTextForCausalLM (DFM Mimir 1B) (#27625)
* model : add support for HrmTextForCausalLM (DFM Mimir 1B) HRM-Text runs two transformer stacks (low, high) in an alternating cycle over the same token stream. The low-cycle state z_l starts from a learned [n_embd] tensor and is broadcast over positions. - conversion: new writer for the fused gqkv projection (order gate,q,k,v) remapped to llama.cpp q/k/v plus a separate sigmoid gate tensor - loader: block_count = lps * h_cycles * (l_cycles + 1) cache slots aliasing 2*lps physical blocks via struct copies - graph: looped build with sigmoid-gated attention, SwiGLU FFN and parameterless RMS norms; learned embedding_scale applied in build_inp_embd - saver: pointer-deduplicated layer loop (looped archs alias tensors) - tests: hrm_text fixture (lps 1, h 2, l 3) in test-llama-archs Limitations: causal attention only - the upstream prefix-LM mode is not implemented (the prefix_lm GGUF key round-trips unused). The KV cache holds one entry per pass: 128 layers for Mimir 1B, i.e. 4x a same-width 32-layer model - about 3072 MiB at ctx 4096 in F16 (halves with q8_0 KV + FA). Every token runs all 128 block passes, so decode cost is roughly 4x a dense model of equal width (2.65 t/s BF16, 8-thread desktop CPU). Verified against the HF reference: identical argmax at 334/334 positions across 20 prompts (BF16 GGUF vs FP32 golden). q8_0 requant: 95.8% top-1, all remaining misses inside the HF top-5 (accumulated error over 128 sequential blocks). AI usage disclosure: YES Used GLM-5.3 for the majority of code AI-generated under my direction, all gates verified locally. All in all I could say that I have written less than 20% of the code and most of the heavy lifting has been done by the model. As such, this should be considered experimental. * Update conversion/hrm_text.py Co-authored-by: Sigbjørn Skjæret <[email protected]> * Update src/llama-arch.cpp Co-authored-by: Sigbjørn Skjæret <[email protected]> * convert : add gguf_writer methods for hrm_text metadata replace raw add_uint32/add_bool calls with dedicated GGUFWriter methods, following the add_embedding_scale pattern Assisted-by: GLM-5.3 * convert : map regular hrm_text tensors via tensor_mapping delegate unfused checkpoints to the base tensor mapping; training-style attn. names are renamed to self_attn. so the patterns match Assisted-by: GLM-5.3 * model : format hrm-text build_* calls as in other models one argument group per line, matching sibling model files Assisted-by: GLM-5.3 * llama : move hrm z_l_init table entries out of the nemotron group place the name and tensor-info entries with the other global input tensors Assisted-by: GLM-5.3 * convert : slim down hrm_text comments Assisted-by: GLM-5.3 * convert : build hrm_text block tensor names from the {bid} template The tensor map holds concrete per-block names, so format the template with the computed layer index before handing it to super(). * llama : name hrm metadata keys in their own hrm. namespace The four keys are arch-independent, unlike the arch-substituted Keys.LLM entries, so group them under Keys.HRM (like Keys.Split) and rename the llm_kv entries to LLM_KV_HRM_*. Only our own GGUFs carry the old hrm_text.* keys; they are regenerated. * Update src/llama-model-saver.cpp Co-authored-by: Sigbjørn Skjæret <[email protected]> * llama : keep hrm metadata keys arch-substituted Per review: the GGUF keys stay "{arch}.h_cycles" style, so the Python members drop the LLM_KV_HRM_ prefix and keep arch templates; C++ keeps the LLM_KV_HRM_* enums. GGUF output is unchanged - existing files and HF uploads stay valid. * Update gguf-py/gguf/constants.py Co-authored-by: Sigbjørn Skjæret <[email protected]> * Update src/llama-arch.cpp Co-authored-by: Sigbjørn Skjæret <[email protected]> * Update src/llama-arch.cpp Co-authored-by: Sigbjørn Skjæret <[email protected]> * convert : rename hrm writer methods to add_hrm_* Generic names like add_h_cycles/add_prefix_lm are too broad on the shared GGUFWriter; prefix them with hrm_ like the metadata keys. * model : fix meta-split lookup for archs with aliased cache slots Cache tensors of archs that alias physical blocks across looped slots (hrm_text, nanbeige with num_loops > 1) can reference block indices without weight tensor names. Take the output projection from the layer array instead of asserting; all other lookups are unchanged. * model : replicate hrm_text tensors on meta devices instead of splitting The aliased cache slots rotate split states differently from their physical weights, so the meta-split execution invariants (set_rows requires the cache state to match the token indices) cannot hold for any device count. Replicate all hrm_text tensors on every meta device instead; single-device and non-meta paths are unchanged. Assisted-by: Claude Sonnet --------- Co-authored-by: Sigbjørn Skjæret <[email protected]> |
||
 Aman Gupta
|
37b53fd454 | qwen4exp: add hc ops (#28901) | ||
 
|
0a8b29a607 |
metal: fix NaN in mul_mm_id when activations exceed f16 range (#26223)
* test-backend-ops: reproduce MUL_MAT_ID NaN for activations beyond f16
The Metal mul_mm_id path narrows src1 to `half` for the simdgroup MMA
(`S1 = half` in every instantiation; ggml-metal.metal:10582 and :10595,
mirrored at :10643/:10654 in the tensor-ops path). f16 saturates at
65504, so a model whose activations exceed that produces inf, and
`simdgroup_multiply_accumulate` then turns the whole 8x8 accumulator
tile into NaN. The mul_mv_id path used below `ne21_mm_id_min` (32)
carries the same values in f32 and is correct, as is every CPU path.
This was untestable before: `init_mul_mat_id_tensors` initializes
uniform [-1, 1], so no existing case can drive an operand out of f16
range. `test_mul_mat_id` gains an `amax` parameter (default 1.0f,
preserving the historical init exactly) that scales only the f32
activations, leaving the quantized weights in their normal range.
Six cases: n=16 sits below the mul_mv_id -> mul_mm_id switch and is the
control that must stay green; n=32 and n=64 are above it and fail on
Metal today. Two shapes, because this is not model- or size-specific —
q4_K at 128 experts / 4 active / 4096x2048 mirrors a real model, and
q8_0 at 8 experts / 2 active / 512x256 shows the same failure at
minimal size.
Observed on Apple M2 Max, macOS, llama.cpp b10156:
MUL_MAT_ID(type_a=q8_0,...,n=32,k=256,amax=100000.000000):
[MUL_MAT_ID] NaN at index 0 (MTL0=nan CPU=583442.375000) FAIL
The real model behind this is Mistral Small 4 (arch mistral4, 128
experts / 4 active), one of whose layers reaches ~1e5 activations: on
Metal every prefill of >=32 tokens returns an entirely NaN vocabulary,
while <32 tokens is correct.
Note kernel_mul_mm (dense) has the identical conversion at :10273 and
:10286 and is expected to fail the same way; it is not covered here.
Found and written by Claude Opus 5 (via Claude Code).
* metal: fix NaN in mul_mm_id when activations exceed f16 range
kernel_mul_mm_id narrows src1 to `half` for the simdgroup MMA operands
(`S1 = half` in every instantiation). f16 saturates at 65504, so a model
whose activations exceed that produces inf on load, and
simdgroup_multiply_accumulate then propagates NaN across the whole 8x8
accumulator tile. The result is an entirely NaN output — not a precision
loss, a total loss. The mul_mv_id path taken below ne21_mm_id_min (32)
keeps the same values in f32 and is correct, as is every CPU path, so
the same model produces correct logits for short inputs and NaN for
long ones.
Fix: rescale src1 by a power of two so it fits, and undo the scale on
the f32 accumulator at the store. A two-stage reduction computes
max(|src1|) and writes the pair (1/scale, scale) into scratch chained
off the destination buffer, in the same style as the existing tpe/ids
id-mapping scratch. The matmul multiplies on load and on store.
This is exact, not approximate, for two reasons: the dot product is
linear, so one tensor-wide factor commutes through the accumulation;
and the factor is a power of two, so both multiplications are exact in
binary floating point. When max(|src1|) already fits — every model that
works today — the factor is exactly 1.0 and the output is bit-identical
to before. Accumulation was already f32 and is unchanged; only the
operand narrowing was ever the problem.
The reduction is two-stage (256 threadgroups into partials, then one
threadgroup folding them) specifically so it stays bandwidth-bound. A
single-threadgroup version was measured first and cost up to +451%
median on prefill — the scan serialized against an otherwise idle GPU.
It is also dispatched only on the mm path, so decode never pays for it.
Measured on Apple M2 Max, `test-backend-ops perf -o MUL_MAT_ID -b MTL0`,
99 cases, versus the same build without this change:
n=1/4/8 (mul_mv_id, decode) : -0.8% / -0.8% / -0.4% median (noise)
n=32 (mul_mm_id, prefill) : +1.73% median
n=64 : +1.30% median
n=128 : +1.80% median
n=256 : +3.98% median
n=512 : +3.74% median, +7.20% worst
overall : +1.14% median
Correctness, same machine:
- the six new test-backend-ops cases go from 4 FAIL / 2 OK to all OK,
with the n=16 controls (mul_mv_id path) unchanged;
- `test-backend-ops -b MTL0` full run: 0 failures, no regression;
- Mistral-Small-4-119B (arch mistral4, 128 experts / 4 active) now
generates correctly at the default n_ubatch of 512, in both
UD-IQ3_S and UD-Q4_K_XL quantizations. Before this, every prefill of
>= 32 tokens returned an all-NaN vocabulary and only n_ubatch <= 31
(forcing the mul_mv_id path) worked.
Likely fixes #25722 (mistral4 empty output on Metal above ~300 tokens,
FA on and off, generation degenerating to a single control token — the
signature of argmax over an all-NaN distribution). #20668 may be the
same defect attributed to a bad GGUF.
Note kernel_mul_mm (dense) has the identical narrowing at the
corresponding load sites and is expected to fail the same way; it is
left alone here to keep this change reviewable. Also possible, and left
for later: scaling per output column rather than per tensor, which
would preserve more precision when a single token is the hot one.
Found, diagnosed and fixed by Claude Opus 5 (via Claude Code).
* metal : make requested edits
- remove verbose comments
- explain rationale as requested
Generative AI disclosure: Claude made the edits as requested.
* metal : stack mul_mm_id map0 with amax_part
Implement @ggerganov suggestion to stack amax_part + map0. Mean 2.6% faster (worst -0.7%, best -4.1%). Win grows with batch size. Benchmarked on a hot M2 Max after reboot.
Generative AI disclosure:
Co-Authored-By: Claude Fable 5 <[email protected]>
* cont : fix var scope
* cont : comment out tests temporarily
Comment out tess to not break CI temporarily
Assisted-by: Claude Fable 5.1
---------
Co-authored-by: Claude Fable 5 <[email protected]>
Co-authored-by: Georgi Gerganov <[email protected]>
|
||
|
Sigbjørn SkjæretandGeorgi Gerganov
|
583926e3ac |
ci : add self-hosted webgpu to hf-jobs (#28712)
* add self-hosted vulkan and webgpu to hf-jobs * try t4-medium * cont : adjust cpu backend threads * try t4-small again * restore cm jobs --------- Co-authored-by: Georgi Gerganov <[email protected]> |
||
 Mohamed Elashri
|
5431581326 |
cuda: support row-contiguous SUM_ROWS (#26308)
* cuda: support row-contiguous SUM_ROWS * organize the code and add GGML_OP_MEAN to support row-contiguous tensors using the same shared kernel, and add a test to MEAN permute/slice * Keep original comments and add if/else branch |
||
 Ruben Ortlam
|
fc82583e65 |
vulkan: support sparse Flash Attention (#28105)
* vulkan: add sparse Flash Attention support for DSV4/GLM * tune implementation * add tests * avoid nondeterministic atomicAdd * add cm2 decode vector support * simplify logic and make variable names more consistent * add cm2 f16vec4 binding for decode vector |
||
 Yanzhao Wang
|
1e7bcf3da4 |
metal : add FA kernels for HSK=96, HSV=64 (MiniCPM3) (#28599)
* metal : add FA kernels for HSK=96, HSV=64 (MiniCPM3) MiniCPM3 sets attention.key_length to 96 and does not set attention.value_length, which defaults to n_embd / n_head = 64. Metal had no (96, 64) instantiation, so -fa auto aborted on the missing kernel_flash_attn_ext_vec_f16_dk96_dv64. Instantiate the tile kernel at (96, 64) for every K/V type that already has (96, 96), and the vec kernel for the NE=4 configurations. Of the NE values the vec dispatch considers, only NE=4 works here, because NL = 32/NE has to divide both DK/4 = 24 and DV/4 = 16. * tests : avoid redundant FA vec slice coverage |
||
 Daniel Bevenius
|
69eb250670 |
cmake : use PROJECT_SOURCE_DIR instead of CMAKE_SOURCE_DIR (#28771)
This commit updates cmake to use PROJECT_SOURCE_DIR instead of CMAKE_SOURCE_DIR for paths in function calls. The motivation for this is that when using add_subdirectory, CMAKE_SOURCE_DIR is fixed to the top-level projects source directory, that is the caller of add_subdirectory and not the llama.cpp root which means that common/common.h header will not be resolved. Refs: https://github.com/ggml-org/llama.cpp/pull/28091#issuecomment-5636106377 |
||
|
Daniel Bevenius
|
f3a184b153 |
cmake : remove precompiled headers (#28892)
This commit removes the precompiled headers that I added in Commit
|
||
|
Georgi Gerganov
|
bbdd9f246e |
tests : add fusion baseline README and broaden fusion CI triggers (#28893)
* tests : add README for updating the per-backend fusion baselines Assisted-by: pi:llama.cpp/Qwen3.8-27B * ci : trigger fusion on changes to test-llama-archs.cpp and src/models the dummy models and their architectures drive the fusion baselines, so a change to either can alter the per-fusion counters and should re-run the fusion job. Assisted-by: pi:llama.cpp/Qwen3.8-27B * tests : merge the fusion build commands in the README assisted-by: pi:llama.cpp/Qwen3.8-27B * pi : require explicit permission before posting PR/issue comments assisted-by: pi:llama.cpp/Qwen3.8-27B |
||
 Alex
|
3d10bcd197 |
llama: add Maple 20B-A1B ternary MoE architecture (CPU) (#27000)
* gguf-py: add Maple tensor constants
Add MODEL_ARCH.MAPLE, its "maple" name, and the tensor list for the
Maple 20B-A1B ternary MoE architecture: token embeddings, output,
attention with Q/K RMS norms, and per-expert FFN tensors.
* convert: add Maple HF->GGUF converter
Register MapleForCausalLM in the HF architecture map and add the
converter for the Maple 20B-A1B ternary MoE model: 24 layers, 256
experts with 8 active, sliding-window attention (SWA-512) interleaved
with global attention at a 3:1 ratio, partial rotary factor 0.5, and
per-expert weight stacking into merged 3D tensors.
* llama: add Maple architecture (20B-A1B ternary MoE)
Add the Maple 20B-A1B ternary MoE architecture: 24 layers, 256
experts with 8 active, sliding-window attention (SWA-512) interleaved
with global attention at a 3:1 ratio, and ternary TQ1_0/TQ2_0
quantization support.
- register LLM_ARCH_MAPLE between MAMBA2 and JAMBA
- implement llama_model_maple: Q/K RMS norms after projection (GEMMA4
style), rope applied only on SWA layers (nope_on_global_attention),
ISWA KV cache, and MoE FFN with swiglu gate clamp at +7 (DEEPSEEK4
style)
- mark MAPLE as unsupported by the model saver (roundtrip skipped)
* tests: mark Maple as MoE-mandatory
Maple is always-MoE: the model throws when n_expert == 0, so the
test harness must only run the MoE config for LLM_ARCH_MAPLE.
* maple: apply review feedback (n_ff_exp_arr, get_arr, rope params)
- load_arch_hparams: use n_ff_exp_arr + n_ff_exp() accessor (upstream
changed these from a scalar member during the rebase)
- sliding_window_pattern: get_arr, the pattern is mandatory for this arch
- partial_rotary_factor: read only from rope_parameters (base.py mirrors
the top-level key automatically)
- document why TOKEN_EMBD/OUTPUT are forced to F16 (they are the two
dense tensors in Maple, and the reference GGUFs ship them as F16)
- add @ModelBase.example("deepgrove/maple-preview")
* tests: add Maple to the SWA pattern array list
get_arr for maple.attention.sliding_window_pattern requires an array, but
the harness only emitted a per-layer array for the arches in its list, so
test-llama-archs -a maple failed to load the model.
Assisted-by: DeepSeek Harness
* maple: move swiglu_clamp_exp to the converter
The loader prefilled 7.0 and read the key optionally. The converter now
writes it and the loader reads it as required, because llama-graph.cpp
skips the clamp when the limit is 0 and an optional read would silently
run unclamped. The test harness provides the key for the same reason.
Also drops tensor_force_quant: base.py already forces FFN_GATE_INP to F32
and TOKEN_EMBD/OUTPUT to F16 for ternary file types.
Assisted-by: DeepSeek Harness
* convert: fix the LazyBase func signature in the Maple converter
ty flagged the stack() closure: it takes no argument, while LazyBase is
annotated with func: Callable[[Any], Any]. Pass the tensor list through
args instead of closing over it, the same way kimi_k3 does, so the
callable shape matches.
Assisted-by: DeepSeek Harness
|
||
  cwriterandcwriter
|
21f6b0d22c |
sycl: rfc: Use radix select for top_k (#28670)
* sycl: GPU-resident TOP_K for large k, parallelised over the device
The SYCL backend refused GGML_OP_TOP_K above k = 32 and let it fall back to
the CPU, a backend round-trip per call. The limit was not conservatism: the
scan-merge kernels keep (split_block + 1) * k candidate (value, index) pairs
in SLM, so at k = 128 a work-group already needs 132 KB and cannot launch.
qwen4exp's sparse-attention indexer asks for k = 2048 in 12 layers on every
token, so this fired at every context length.
Add a radix select for large k. The k-th largest is found by four
most-significant-first passes over an order-preserving unsigned key: histogram
the digit over the candidate set, walk the buckets from the top, and recurse
into the one where the running count reaches what is still needed. SLM holds
the histogram rather than candidates, so the footprint is independent of k.
A final pass emits every column beating the pivot plus exactly as many
pivot-equal columns as are still missing, so duplicate keys still yield
exactly k distinct indices. Output order is not required and is not paid for:
ggml-cpu/ops.cpp swaps its first two outputs to say so.
The key folds -0.0 onto +0.0 so its equivalence classes match the reference
comparator, under which the two tie. NaN has no defined order in the reference
(its comparator is not a strict weak order there); here +NaN keys above +inf
and -NaN below -inf, which at least makes the result deterministic.
One work-group per row leaves the device idle whenever a graph has fewer rows
than it has cores, which at batch size 1 means one work-group full stop:
qwen4exp tops-k a tensor of shape [n_kv, n_tokens/n_stream, n_stream], so
token generation gives nrows == 1, and the backend sampler reshapes logits to
a single row as well. Measured, ne=[200000,1] and ne=[200000,16] cost 358.0 us
and 363.4 us -- sixteen rows for 1.5% more wall-clock.
So also spread a row over several groups when there are too few rows to cover
the device. Per-pass state moves to global memory and each digit pass becomes
its own launch, since a work-group barrier can no longer span the row. Groups
accumulate in SLM and contribute 256 global atomics each, keeping global
traffic per-group rather than per-element, and the last group of a row -- the
one whose fetch_add returns G-1 -- performs that pass's scan, holding the
launch count at one per digit plus one emit. The group count comes from the
device and is floor-divided by nrows, so a row count that already covers the
device is left whole and pays nothing. Below 64K columns the single-group
kernel finishes inside the cost of the extra launches and stays in charge.
Reading the row's prefix/mask/need through a device-scope atomic_ref costs
more than the sweep it guards: those loads are uncached, so passes 2-4 ran at
49 us against 12 us for pass 1. One lane reads them into SLM and the group
takes them from there -- 208 us -> 44.6 us at ne=[131072,1], k=2048.
The block size now takes the device's max_work_group_size instead of a cap of
512. The cap was never a floor, so a device reporting 512 is unaffected; one
allowing 1024 was being given half its width.
Finally, put the scan-merge gate where the two paths actually cross. That
kernel's cost climbs with k while the radix select's does not; measured over
widths from 2 to 200K columns and row counts from 1 to 8192, radix is ahead
everywhere from k = 8 up and behind at k <= 2, where scan-merge's smaller
fixed cost wins. The short-row corner (ncols=2, nrows=65536, as in bailingmoe2
group selection) is exactly where radix loses at low k, and the gate keeps it
on scan-merge.
Op-level against the CPU-fallback path this replaces, and against the
single-group radix select for the split: 4.98x at ne=[131072,1] k=2048,
6.65x at ne=[151936,1] k=40, 13.35x at k=20, 118x at ne=[65000,16] k=32.
No measured shape regressed. End to end on 3x Arc Pro B60 with
Qwen3.8-Flash-Next UD-IQ4_XS, llama-bench tg64, the parallelisation is worth
5.91 -> 6.05 t/s at d=131072 and a wash at shallower depths. Perplexity over
wikitext-2 is unchanged within noise at both 512 and 81920 context.
test-backend-ops: 525/525 TOP_K (previously every k > 32 case was refused),
880/880 MUL_MAT_ID. Perf coverage added for k > 32 at large widths and for the
short-row corner, neither of which was exercised before.
* move topk-select to topk-radix.{cpp|hpp}
---------
Co-authored-by: cwriter <cwriter@localhost>
|
||
  fairydreamingandStanisław Szymczyk
|
5f436dddb4 |
tests : exclude HY_V4 from WebGPU test-llama-archs tests (#28855)
Co-authored-by: Stanisław Szymczyk <[email protected]> |
||
 Michael Taylor
|
243a3082d4 |
tests : fix typo in test-quant-type-selection for nemotron 3 nano (#28835)
Corrects a typo in `tests/test-quant-type-selection` for the Nvidia Nemotron 3 Nano 30B A3B model, which was referred to as *nvidia-nemotron-nano-3-30b-a3b*. The error made the test skip that test case, rather than failing the test. [no release] |
||
|
Georgi Gerganov
|
4a89937354 | tests : reduce FA test sizes (#28842) | ||
 Aldehir Rojas
|
790cf51aab |
chat : improve parsing of complex types in qwen3-coder (#28742)
* chat : improve schema support in qwen3 parser * cont : clean up grammar a bit |
||
|
Aldehir Rojas
|
acecd56032 |
common : implement common_schema internal representation for JSON schemas (#28736)
* common : implement common_schema types
* common : implement a json schema optimizer
* common : reduce optimizations
* common : refactor json-schema-to-grammar to use common_schema
* common : use common_trie
* common/schema : implement type/kind resolution
* cont : cleanup
* cont : remove common_chat_tool_parameters
* cont : simplify schema resolution
* cont : pass common_schema through the json-schema-to-grammar builder
* cont : cleanup
* cont : move enums under common_schema and add type enum
* cont : reduce test cases
* cont : clean up
* cont : clean up
* refactor : rename common_schema_parse to common_schema_from_json
* tests : fix gcc dangling-reference warning in test-json-schema
* tests : take the schema label as const char * to satisfy gcc dangling-reference
* refactor : rename common_schema_builder parse_* methods to build_*
* cont : fix may_be_string
* cont : properly handle empty tool parameters
* cont : add tests for empty $ref
* cont : remove dead code
* cont : update docs
* cont : make "{}" mean any object for json_object as well
* cont : restore (min|max)Length to imply string type
* cont : rename common_schema to common_chat_schema
|
||
|
Sigbjørn Skjæret
|
ae9afff8d2 | jinja : support dot property integer literals (#28817) | ||
|
Georgi Gerganov
|
b78a39a2f9 |
ci : run test-backend-ops as a dedicated ci/run.sh test (#28740)
* ci : run test-backend-ops as a dedicated gg test Run test-backend-ops as a separate gg test in ci/run.sh so it is executed outside ctest. With GG_BUILD_HIGH_PERF it keeps the existing CPU-only invocation (-b CPU); otherwise it runs all available backends without a backend filter. Remove the dedicated backend-ops workflow and keep test-backend-ops as a built target that is not registered with ctest to avoid duplicate runs. Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp * ci : run test-backend-ops earlier and enable high-perf on kleidiai Move the test-backend-ops gg test before test-llama-archs. Enable GG_BUILD_HIGH_PERF and LLAMA_ARG_THREADS on the Graviton4 KleidiAI job and use the standard self-hosted results/mnt paths. Add TODO markers for decoupling tests from libllama. Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp * ci : run test-backend-ops in parallel Pass -j $(nproc) to test-backend-ops in both high-perf and all-backend modes. Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp * ci : disable parallel tests for ROCm * cont : disable parallel tests with MoltenVK |
||
 Rohanjames1997
|
982937a333 |
tests: extend test-quantize-fns to test nrc=2 (i8mm) kernels (#16234)
* Test for nrc=2 as well | i8mm kernels * Trigger only on supported HW * Remove trailing whitespace * Address review comment * test: properly prepare nrc=2 inputs with independent data per row * tests : make nrc=2 dot product inputs distinct Assisted-by: Kiro * tests : use non-trivial strides in nrc=2 dot product test * tests : fail nrc=2 dot product test on non-finite errors |
||
 Foad Abo Dahood
|
5bda51bfbc |
metal : skip the empty half of the mul_mm_id token tile (#28301)
kernel_mul_mm_id splits its NR1 = 32 token tile into two 16-row halves and skips the upper half when the expert did not fill it, on both the tensor and simdgroup paths. The tB extents are corrected to (NK, NR1H) for the [NR1][NK] row-major tile. The B tile is staged unconditionally, as on master: rows past nr1 restage a clamped duplicate of a valid row, lie in the output-row dimension so they never contribute to a valid row, and are dropped by the final store loop. test-backend-ops: re-draw the expert ids between perf iterations of test_mul_mat_id so MoE perf numbers are not warm-cache, and add token-tile boundary coverage using n_used == n_mats, which routes every token to every expert so each expert receives exactly n rows; n = 32, 33, 47, 48, 49 reach mul_mm_id and leave a last tile of 32, 1, 15, 16 and 17 rows. |
||
|
Daniel Bevenius
|
3bcfeb700f |
cmake : add PCH and unity build to improve build times (#28091)
* scripts : add initial profiling script (wip)
* src : add precompile headers (PCH) for models.h
* common : add common.h as PCH
* ggml : add PCH for ggml-impl.h
* mtmd : use PCH for models.h
* scripts : add script to build with Server/Tools/Tests
* server : add PCH for common.h
* docs: add profiling progress notes (wip)
* ggml : add exclude for GCC + SVE on ARM
Refs: https://github.com/ggml-org/llama.cpp/actions/runs/33393906061/job/99493756214?pr=28091
* ggml : attempt to fix use of std::hardware_destructive_inference_size
Refs: https://github.com/ggml-org/llama.cpp/actions/runs/33396221677/job/99501265689?pr=28091
* squash! ggml : attempt to fix use of std::hardware_destructive_inference_size
Add a version check for GCC 12 to conditionally apply the `-Winterference-size`
pragma.
* editorconfig : exclude profiling reports dir
This directory will not be included in the merge later and this commit
can be ignore at that point. Just fixing to keep CI happy.
* ggml : skip PCH for gcc on non-x86 architectures
* tests : add PCH for peg-parser/tests.h
There are 7 peg-parser tests that can share one PCH instead of then each
parsing the full tests.h.
* common : add PCH for chat.h
* docs : update linux build profiling full results
Just updating after a number of PCH additions. These are not exact
figures and will vary a bit from run to run, but they give a general idea
of the performance impact of PCH.
* cmake : introduce unity build for models
This commit introduces a unity build for the models to improve
compilation time.
The improvements were roughly the following:
```console
+------------------------+-----+------------+------------+------------+
| Build | TUs | Frontend | Backend | Total |
+------------------------+-----+------------+------------+------------+
| Full, master | 396 | 811.0 s | 692.2 s | 1,503.2 s |
| Full, with PCH | 405 | 380.0 s | 664.7 s | 1,044.7 s |
| Full, with PCH + UB | 264 | 357.7 s | 635.7 s | 993.4 s |
+------------------------+-----+------------+------------+------------+
TU = Translation Unit.
Full = includes Server, Tools, and Tests.
PCH = precompiled headers.
UB = unity build for models.
```
* docs : update linux profiling table with unitiy build results
* docs : update mac profiling results to include unity build [no ci]
* docs: remove profiling reports
* scripts : merge build profile scripts into one script
I was lazy before and just copied the first script to enable Tests,
Server, and Tools. This now merges them into a single script.
* Revert "editorconfig : exclude profiling reports dir" [no ci]
This reverts commit
|
||
|
Georgi Gerganov
|
a2878d30df |
metal : single-source fusion table + fusion debug rework (#28164)
* metal : rework fusion patterns into a single table All fusable op patterns for the Metal backend are now declared once in a fusion table (ggml-metal-fuse.cpp) and consumed by both the graph optimizer (ggml_metal_fuse_max, packing) and the op encoders (ggml_metal_fuse_next, compute). The two phases share the same pattern table plus ggml_can_fuse_subgraph_ext for the structural checks, and differ only in the mode used for the pattern check (STRUCTURAL at optimize time, since tensors are not allocated yet, and FULL at compute time, including Metal buffer placement). This also protects the snake activation (MUL + SIN + SQR + MUL + ADD) from being reordered during graph optimization, which was previously unprotected. Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 * metal : fix absolute output indices in fusion patterns ggml_can_fuse_subgraph_ext expects the outputs array to contain absolute graph node indices (it indexes cgraph->nodes[outputs[i]]), but the fusion table query was passing a relative index (n_ops - 1). As a result the last node of every pattern was not recognized as an output and was subjected to the elidable use-count check, which failed for essentially all fusions. This silently disabled the norm/MUL fusion and caused a ~5% token-generation regression. Pass the absolute graph index of the last node instead. Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 * metal : fuse gated_delta_net with cache cpy Add GGML_METAL_FUSE_GDN_CACHE to the fusion table: when the gated_delta_net kernel is followed by a cpy that scatters its recurrent state snapshots into the KV cache, the kernel writes the snapshots straight into the cache buffer and the trailing cpy is elided. The gdn output has other consumers (the attn scores view), so unlike the elision-chain patterns this is not a simple chain: a 'raw' flag on the fusion pattern skips the generic chain/shape and ggml_can_fuse_subgraph_ext checks, making the pattern-specific check callback the sole validator. Packing (ggml_metal_fuse_max) now matches on the same view-transparent node sequence that the compute phase uses, so the gdn + cache cpy group is packed along with any intermediate views and stays adjacent through the reorder. The fused cpy is a view consumer of the gdn (it writes the cache directly), so its mem-range is skipped in the encoder; the skip is restricted to CPY nodes consuming the previous fused node through a view so other fusions are unaffected. Add test_gated_delta_net_cache_fusion and register 5 cases. Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 * metal : drop is_view_consumer mem-range skip The is_view_consumer skip was carried over from the upstream gated_delta_net cache-fusion draft, but it is not needed: keeping the elided cpy's mem-range in the concurrency tracker only ever adds a (conservative) memory barrier at the fusion point. It can never remove a barrier, so it cannot introduce a race. The worst case is one spurious barrier per gdn+cache-cpy fusion, which is within run-to-run noise on Qwen3.5-0.8B Q8_0. Dropping the check keeps the mem-range loop uniform for all fused groups. Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 * metal : rename gated_delta_net fused state output args Rename the fused cache-write kernel argument to match the rest of the kargs: state_out_stride -> nb_out (and widen it to uint64_t), and the local buffer id bid_state_out -> bid_out. Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 * metal : rename raw fusion flag to unsafe raw did not convey that the flag opts a fusion pattern out of the generic elision-chain safety net (ggml_can_fuse_subgraph_ext + chain/shape checks). rename it to 'unsafe' to make explicit that the pattern's check callback is the sole validator and must re-establish the safety guarantees itself. Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 * metal : tidy fusion pattern checks and table - const-correct ggml_metal_fuse_outputs buffer - annotate unused check-callback parameters - drop a redundant size_t cast - align the ops/table initializers and add blank-line separation Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 * metal : add generic fusion stats via ad-hoc proc-address API Add a device-owned fusion context that lets a test tool count how many times each fusion pattern fires and toggle fusion. It is exposed through the ad-hoc ggml_backend_reg_get_proc_address mechanism with generic names so the testing tool is backend-agnostic: - ggml_backend_fusion_stats_init: start collecting fusion stats; when a context is created afterwards it registers the labels/counters and encodes single-threaded (n_cb == 0) so the counters are race-free - ggml_backend_fusion_stats_reset / _get_stats / _set_enabled The context lives on the metal device (not on the last backend context), so counters accumulate across contexts and reads are always consistent. The enable/disable toggle is initialized from GGML_METAL_FUSION_DISABLE and can be overridden by the test through set_enabled. Labels are synthesized from the fuse table via ggml_metal_fuse_label (e.g. "GATED_DELTA_NET+CPY"). Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 * tests : add fusion count regression test with per-backend baseline test-fusion runs every dummy model generated by test-llama-archs on a single backend (single-threaded encoding, n_cb == 0) with fusion enabled and disabled, and for each mode (prefill / decode) reports the per-fusion counters and the NMSE between the fused and unfused logits, plus the NMSE against a CPU reference. A fusion pattern that silently stops matching (or fires when it should not) is caught as a regression by comparing the counters against a committed per-backend TSV baseline: - --record writes the golden baseline, --check (default) validates it - the unfused run doubles as a control: its counters must be all-zero - NMSE is skipped when it is NaN or the arch is already broken on the device (e.g. plamo2 on Metal), so the count check is the hard gate - baseline counts depend only on graph structure, not weights (verified stable across weight seeds) - the fusion stats API is resolved through the ad-hoc get_proc_address mechanism with generic names; a backend that does not export it makes the test fail with an error The committed MTL0.tsv baseline covers 110 dummy archs (298 rows). Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 * tests : rename fusion api helpers to match stats_init signature Align the test with the ad-hoc fusion stats API: fusion_stats_init no longer takes an enable bool (stats are turned on by calling it), so the proc-address wrappers and typedefs are renamed to the api_* convention. Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 * tests : rename backend to device in fusion test CLI The fusion test operates on a compute device (e.g. MTL0), not a backend, so rename the --backend argument to --device and the backend_name variable to device_name. Keep "backend" where it refers to the ggml backend interface (the ad-hoc proc-address mechanism). Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 * tests : add --model and --help to fusion test --model FILE runs the fusion regression test over a single model file instead of enumerating a --models DIR. --models and --model are mutually exclusive. Also add a --help/-h option that prints the usage. Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 * tests : use backend base name for fusion baseline output The fusion test is invoked with a specific device name (e.g. MTL0), but its output - the recorded baseline and the header it writes - should be named after the backend base name (e.g. MTL, via ggml_backend_reg_name), since the counters depend on the backend, not on the specific device index. Rename the committed baseline to MTL.tsv. Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 * tests : run fusion test from ci instead of ctest The fusion test needs Metal and generates a lot of dummy models, so it does not belong in the generic ctest suite. Move it to ci/run.sh as gg_run_test_fusion, gated on GG_BUILD_METAL like gg_run_test_llama_archs_tensor_split: it generates the dummy models with test-llama-archs -o and then validates the fusion counts against the committed baseline. test-fusion.cpp is still built (llama_build) but no longer registered as a ctest. Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 * tests : align fusion baseline TSV columns Pad the TSV fields to fixed widths so the columns line up regardless of the variable arch and fusion-label lengths, and trim each field on parse so the padded file is still accepted. Regenerate the committed MTL.tsv baseline in the padded format (data unchanged, verified identical modulo padding). Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 * tests : widen label column and align fusion TSV header Give the label column more room (28 chars) and fix the column header widths so they match the data rows (moe/mode/label), keeping the header aligned with the values. Regenerate the MTL.tsv baseline in the new format (data unchanged). Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 * tests : switch fusion baseline from TSV to CSV Use comma-separated values like the rest of the project, keeping the padded, aligned columns. Split on ',' and trim on parse. Rename the committed baseline to MTL.csv (data unchanged, verified identical modulo padding/separator). Update the ci/run.sh check path accordingly. Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 * cont : rebase + update MTL stats * tests : avoid graph reallocations for some archs * metal : tidy fusion debugging context and op init - simplify the shared fusion debugging context comments - shorten the ggml_metal_fusion struct comment - align the ggml_metal_fuse struct fields and comments - move the fusion parameter of ggml_metal_op_init right after dev Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 * tests : dedup fusion baseline into any mode prefill and decode always produce the same per-graph fusion count, so store a single row per label with mode = "any" and the per-graph count instead of two rows. this halves the baseline size and keeps the check stable. Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 * ci : move fusion model generation to a separate step the dummy models generated by test-llama-archs are reused by other tests, so generate them once in their own step instead of inside test_fusion. Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 * tests : bump nmse thold * models : fix plamo2 graph * tests : remove "skip" logic from test-fusion * tests : set qwen3tts dummy vocab to codec head size the dummy qwen3tts model used a vocab of 4096 while the codec head is 3072, so the graph padded the output with -inf which made the NMSE in test-fusion produce NaN. use the exact codec head size instead so the padding is not generated at all. Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 * tests : regen fusion baseline reflect the plamo2 graph fix, which changed its fusion pattern split (RMS_NORM+MUL 11->10, RMS_NORM+MUL+ADD 3->4; same total). Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 * ci : skip dummy model generation on OpenVINO test-llama-archs does not build on the OpenVINO platform, so do not try to generate the dummy models there. Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 * cont : minor * tests : enable test-llama-archs on windows * cont : disable on windows + workaround * metal : naming nits * test-fusion : add instructions to update baseline * context : fix Kimi-K3 graph reserve * fusion : update MTL * cont : fix naming * metal : rework fusion info storage Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp * metal : align fusion info API Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp * metal : use opaque fusion handle in ad-hoc API Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp * ci : move fusion test to dedicated workflow Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp * cont : run only on ggml changes * cont : simplify * fusion : remove multi-output stuff for now * ci : fix typo |
||
 Piotr Wilkin (ilintar)andJohannes Gäßler
|
16378d93f9 |
CUDA/HIP: Flash Attention tuning (gfx1201) (#28102)
* HIP: enable mma FA for head size 256 on RDNA4, tune configs Assisted-by: Claude Assisted-by: Codex * HIP: prefer whole-tile FA grids over stream-k on AMD WMMA Assisted-by: Claude Assisted-by: Codex * revise stream_k logic * revise kernel selection logic --------- Co-authored-by: Johannes Gäßler <[email protected]> |
||
 Jeff Bolz
|
6788edb4f3 |
vulkan: small M matrix optimizations for qwen (#28457)
* vulkan: optimize m=1 mul_mat by swapping A/B * vulkan: Improve small M perf Allow split_k with small M. Make small vs med tile selection (for coopmat2) depend on M, not just N. |
||
 Gaurav Garg
|
c32d1dabe8 | tests : increase tolerance for Add fusion tests (#28691) | ||
|
Georgi Gerganov
|
e5a8d439ce | tests : drop SYCL special-casing in test-backend-ops.cpp (#28688) | ||
 Todor BoinovskiandMax Krasnyansky
|
6d9c82ea2b |
hexagon: rope updates (#28628)
* hexagon: vectorize RoPE theta cache on v75 * hexagon: vectorize MROPE/IMROPE theta pick * hexagon: tighten NEOX RoPE rotate and aligned tail copy * hex-rope: use inplace rope for all scenarios * hex-rope: remove ctx->spad usage and legacy timers * hex-rope: add kernel params and enforce vtcm reqs at the host * hex-rope: cleanup unused params and tighten the mode checks * hex-rope: add missing ops header --------- Co-authored-by: Max Krasnyansky <[email protected]> |
||
 Pascal
|
b31b71f3a0 |
jinja: treat a null left operand of in as a plain lookup (#28620)
Templates that default an optional variable to none and then test its membership in a map hit an error, while the same expression is a normal lookup returning false in Jinja. The undefined counterpart of this case was already handled just above. |
||
|
Georgi Gerganov
|
30b6a755e2 |
tests : use less threads for data initialization (#28325)
* tests : use 1 thread for data initialization * cont : scale threads with number of elements * cont : adjust |
||
 Pepper Gray
|
ca86fb222e |
llama : add missing headers (#28566)
* fix compile-error: add missing header Bug: #28557 Signed-off-by: Pepper Gray <[email protected]> * fix compile-error: add missing header Bug: #28559 Signed-off-by: Pepper Gray <[email protected]> * fix compile-error: add missing header Bug: #28560 Signed-off-by: Pepper Gray <[email protected]> * fix compile-error: add missing header Bug: #28561 Signed-off-by: Pepper Gray <[email protected]> * fix compile-error: add missing header Bug: #28562 Signed-off-by: Pepper Gray <[email protected]> * fix compile-error: add missing header Bug: #28564 Signed-off-by: Pepper Gray <[email protected]> --------- Signed-off-by: Pepper Gray <[email protected]> |
||
 Ankit Khandelwal
|
64e9bceb2c |
vulkan : fuse UNARY(GELU|SIGMOID|SILU|SOFTPLUS) + MUL (#27220)
* vulkan : fuse UNARY(SIGMOID|SILU|SOFTPLUS) + MUL
* vulkan : fuse UNARY(SIGMOID|SILU|SOFTPLUS) + MUL
- implement fusion in unary.comp behind UNARY_MUL_FUSION ifdef,
specialized pipelines per op instead of runtime branching
- fuse adjacent nodes only, ordering handled by graph_optimize
- drop runtime consumer scan and pending_unary_mul deferral
* vulkan : fuse UNARY(GELU|SIGMOID|SILU|SOFTPLUS) + MUL
1. GELU: gelu_mul_f32/f16 pipelines registered, CREATE_UNARY_MUL(gelu), GELU in dispatch + fuse gate + perf fusion name
2. Renamed/moved: gate is now ggml_vk_can_fuse_unary_mul(cgraph, unary_idx, mul_idx), placed with the other can-fuse helpers
3. norepeat both variants: each op gets plain (spec {0}) + _norepeat (spec {1}) pipelines from the same SPIR-V, selected via ggml_are_same_shape(src0, src1); the shape gate now allows broadcast (other dims equal-or-1)
4. graph_optimize: lambda deleted; standard "// UNARY + MUL: pull the consuming MUL forward" block added alongside the SSM_CONV/ROPE/MUL_MAT reorderings, with the same "other src must be weights or already processed" readiness check
* vulkan : align unary_mul fusion with binary kernel layout, relax gelu test tolerance
- schedule the fused kernel like mul.comp (256 threads x 2 unrolled
iterations), recovering a 10-18% prompt-processing regression
- allow 5e-7 f32 error for gelu_mul: the shader evaluates gelu with an
exp-based tanh identity while the CPU reference uses tanhf (~1 ulp)
* vulkan : use ggml_can_repeat in UNARY+MUL fusion shape check
The fused kernel indexes src1 via per-dim fastmod (generic_binary_head.glsl),
which is exact whenever the other operand tiles into the unary result -- not
just when its dims are equal or 1. Replace the hand-rolled loop with
ggml_can_repeat(other, unary) so the check matches the kernel's actual
capability and reuses the standard helper. Argument order matters: reversed,
it would wrongly admit graphs where the unary result is mul->src[1] and the
other operand is larger, producing truncated output.
Also add a rep_ne0 layout to the fused unary+mul backend tests covering a
non-1 repeat factor along dim 0.
* vulkan : fuse UNARY+MUL pairs separated by zero-compute nodes
gemma4's per-layer embedding gating builds gelu -> view_2d_slice -> mul,
where the intervening view is a zero-compute node aliasing an input that
was computed much earlier. Strict adjacency requirements meant neither
CUDA nor the vulkan unary+mul fusion handled this pattern.
Extend ggml_vk_graph_optimize to detect a UNARY whose consuming MUL is
separated only by unscheduled zero-compute nodes (GGML_OP_NONE, VIEW,
RESHAPE, TRANSPOSE, PERMUTE) and schedule those nodes ahead of the pair,
making it adjacent so the existing fusion applies. The reorder is guarded
by ggml_vk_can_fuse_unary_mul, a source-availability check for every
interleaved node, and the protected fusion patterns (topk_moe*, snake);
if fusion is later rejected the reordered graph still executes correctly,
just unfused.
Add a view_mid layout to the fused unary+mul backend tests replicating
the gemma4 pattern.
* vulkan : support OP-on-B in UNARY+MUL fusion
Some models apply the unary activation to the smaller MUL operand, e.g.
qwen3next/qwen35moe shared-expert gating builds ffn_shexp * sigmoid(gate)
with a [1,n_tokens] gate tensor. This shape was correctly rejected before:
the fused kernel derives its iteration extent from the unary tensor and
would leave most of the destination unwritten, and the generic same-shape
requirement in ggml_can_fuse blocked the pair outright.
Add UNARY_MUL_B_FUSION shader variants computing dst = src0 * OP(src1):
the OP operand rides the existing per-dim fastmod indexing, while the
iteration extent now comes from mul. Route {UNARY, MUL} pairs through a
local can-fuse variant that drops the generic same-shape rule and instead
requires the unary result to tile into mul->src[0] (ggml_can_repeat);
pairs with the unary as src0 keep the previous direction check, and
equal-shape pairs keep using the original pipelines.
Add a "gate" layout to the fused unary+mul backend tests covering the
shared-expert gate shape for gelu/sigmoid/silu/softplus in f32 and f16.
* vulkan : fold unary+mul view-hoisting into graph_optimize dep checks
Replace the dedicated UNARY + EMPTY* + MUL scanning block with two small
extensions to the existing scheduling logic:
- a consuming MUL may now join its in-set UNARY across a gap of unused
zero-compute nodes (NONE/VIEW/RESHAPE/TRANSPOSE/PERMUTE), instead of
requiring strict adjacency
- while doing so, such zero-compute blockers are ignored for this pair
Fusion validity is still decided later by ggml_vk_can_fuse at dispatch
time, so a rejected pair simply executes adjacent-but-unfused. Note the
relaxation must stay scoped to this pattern: exempting zero-compute
blockers globally reproduces silent output corruption on gemma3n.
* vulkan : select unary_mul OP-on-B via specialization constant
Replace the UNARY_MUL_B_FUSION compile-time shader variants with an
op_on_b specialization constant on the existing unary_mul SPIR-V,
mirroring how the norepeat flag is handled. The four {op}_mul_b_{f32,f16}
shader artifacts are gone - the OP-on-B pipelines reuse the base SPIR-V
with two-entry {norepeat, op_on_b} spec lists - and the duplicated store
expression is collapsed into a single runtime branch that the driver
prunes per specialization.
The constant is declared only under UNARY_MUL_FUSION so every other
binary pipeline keeps its single-entry specialization list.
* vulkan : replace unary_mul pipeline switches with a lookup table
Collapse the four nested selection switches in ggml_vk_unary_mul into a
single indexed lookup against a pipeline_unary_mul[4][2][2][2] table
([unary op][f16][norepeat][op_on_b]), whose trailing dims mirror the
{norepeat, op_on_b} spec constant list. The op axis uses a small shared
index helper that also replaces the switch in ggml_vk_can_fuse_unary_mul,
making it the only place that maps ops to the table.
Pipeline names are unchanged. Adding another supported op now requires
one macro invocation line and one helper case instead of edits in four
separate switches.
* vulkan : use ggml_can_fuse_subgraph for unary_mul pairs
Replace the hand-rolled pair validation in ggml_vk_can_fuse_unary_mul_pair
(bounds, op match, compute flags, single-use elision) with the shared
ggml_can_fuse_subgraph helper; backend-specific shape/type rules remain in
ggml_vk_can_fuse_unary_mul. Unlike ggml_can_fuse, the subgraph helper has
no same-shape requirement, so it covers both operand slots including
OP-on-B gates, and additionally rejects intermediates flagged as graph
outputs and validates view-source confinement.
The outputs parameter takes absolute node indices into the cgraph.
* Fix Whitespace
* vulkan : drop redundant unary_mul gap check in graph_optimize
The zero-compute nodes separating a UNARY from its consuming MUL are
already scheduled ahead of the pair by pass 2 of an earlier
optimization window, so the scoped gap tolerance added for this pattern
is unreachable in practice - disabling it leaves gemma-3n dispatch
counts unchanged (841 GELU_MUL per pass). Remove the flag, the empty
blocker exemption, and the now-unused gap helper, restoring the strict
adjacency requirement of the UNARY -> MUL pull-forward.
Keep the relaxation scoped out entirely: generalizing "zero-compute
nodes never block" beyond this pattern previously reproduced silent
output corruption on gemma3n.
* vulkan: fix whitespace (tab in indent)
* vulkan: fix whitespace (extra blank line)
* vulkan : move op_on_b spec constant to unary.comp
op_on_b is only used by the fused unary*mul path. Keep
generic_binary_head.glsl generic by defining it in unary.comp
instead. Same constant_id=1 and guard, no functional change.
* vulkan : make RMS_NORM/UNARY fusion gap-tolerant for views
Strict j==c+1 blocked RMS_NORM->MUL and UNARY->MUL when a
VIEW sits between (e.g. rms_norm -> view -> mul). Allow
c==back() with an empty-or-scheduled gap, matching the
review suggestion to check src linkage instead of adjacency.
Scoped to the two blessed pairs; safe because gaps can only
contain zero-compute nodes.
* vulkan : trim comments in UNARY+MUL fusion
Assisted-by: Muse Spark
|
||
|
Georgi Gerganov
|
5a6caa05fc |
ggml : update ggml_prec specification (#26675)
* ggml : update ggml_prec specification [no ci] * cont : add GGML_PREC_BF16 * cont : rework API * cont : use new API * cont : swap arg order * cont : support for MUL_MAT_ID * cont : fix accidental remove of "break;" * cont : return bools, add doc TAG_GGML_PREC, clean-up * cont : add search tag * cont : ws |
||
 Frank Dai
|
9dcf84e5ae | model : support Kimi-K3 recurrent-state rollback (#28466) | ||
|
Todor Boinovski
|
050dde50c9 |
hexagon: add RELU and LEAKY_RELU ops (#28585)
* hexagon: add RELU op * hexagon: add LEAKY_RELU op too |
||
|
Pascal
|
f114f91f9e |
tests : initialize the L2_NORM batch array (#28553)
* tests: bind the L2_NORM batch count to a local GCC cannot prove the loop fills norms up to the index read after it while the bound is a class member, so it reports a maybe uninitialized use. Reading the count once into a local restores the tracking. * tests: initialize the L2_NORM batch array The read after the fill loop is only provably defined once the array carries an initializer, which GCC 12 requires on the aarch64 Release build where warnings are fatal. |
||
 Titaniumtown
|
dbeb37548e |
sycl: add a batched L2_NORM kernel (#28222)
* sycl: add a batched L2_NORM kernel
* sycl: batch consecutive L2_NORM siblings in the graph dispatch
Measured on Intel Arc Pro B70 (Battlemage), Qwen3.6-27B Q4_K_M, f16 KV,
npp=128 ntg=128 npl=2, GGML_SYCL profiler:
L2_NORM dispatches 12480 -> 6240
L2_NORM device time 68.77 -> 39.14 ms (-43%)
total device time 6782 -> 6748 ms (-0.5%)
wall decode t/s flat
* tests: add L2_NORM_BATCH coverage
|
||
  Kevin HopperandKevin Hopper
|
7a333e7240 |
vulkan: add DeepSeek-V4 hyper-connection fused ops (DSV4_HC_COMB/PRE/POST) (#26578)
* vulkan: add DeepSeek-V4 hyper-connection fused ops (DSV4_HC_COMB/PRE/POST) CUDA has these ops from the DeepSeek-V4 merge and Metal gained them in PR 26459. Vulkan was the last major backend running the unfused primitive chain. On DeepSeek-V4-Flash the unfused Sinkhorn comb chain alone takes about 32% of decode op time on gfx1151 (Strix Halo), spread over roughly 16k dispatches per token. dsv4_hc_comb runs the full 20-iteration Sinkhorn in registers. A token's 4x4 comb matrix lives in 16 consecutive subgroup lanes, with idst in bits 0-1 and isrc in bits 2-3 to match the CPU reference layout, so subgroupShuffleXor by 1|2 reduces rows and by 4|8 reduces columns. One dispatch replaces about 137 strictly ordered node executions per site. The shuffle masks never cross a 16-lane boundary, so a subgroup of size 64 packs 4 independent tokens. dsv4_hc_pre and dsv4_hc_post handle the elementwise stream collapse and fan-out, with per-token coefficients staged in shared memory. GGML_VK_DISABLE_DSV4_HC disables all three ops. The _COMB, _PRE and _POST variants gate each op independently so a single kernel can be bisected against the unfused graph. Adds eval cases at the production n_iter=20 across batch sizes that cross subgroup and workgroup boundaries. * vulkan: dsv4 hc review fixes Drop the per-op env-var disables and device flags, the stride divisibility check (ggml guarantees it) and the workgroup-count fallback in supports_op. Trim the comb shader comments to the lane layout. --------- Co-authored-by: Kevin Hopper <[email protected]> |
||
 Jeff BolzandAiChiTuDouPian
|
0cae43063c |
vulkan: support type-aligned GET_ROWS (#28253)
* vulkan: fall back to CPU for GET_ROWS with misaligned offsets
The Vulkan GET_ROWS shader asserts when a tensor's backing-buffer offset
plus view_offs is misaligned w.r.t. minStorageBufferOffsetAlignment
(see init_pushconst_tensor_offsets). Previously this caused a hard crash
on models using ggml_view + ggml_get_rows (e.g. Qwen3-TTS, Qwen3-VL).
Return false from supports_op() in the misaligned case so the scheduler
falls back to CPU, matching the existing pattern for PAD_REFLECT_1D and
other unsupported op/shape combinations.
Repro: llama-tts -m Qwen3-TTS-*.gguf -mm mmproj-*.gguf -ngl 99
Crash: GGML_ASSERT(dst->op != GGML_OP_GET_ROWS || (a_offset == 0 && ...)) failed
* vulkan: trim comment for GET_ROWS misalign fallback
* vulkan: fix file corruption in gated_linear_attn struct
* vulkan: properly handle misaligned offsets in GET_ROWS quantized path
- get_rows_quant.comp was missing get_aoffset()/get_boffset()/get_doffset()
calls that are already present in get_rows.comp, causing GGML_ASSERT crashes
when GET_ROWS operates on views with non-zero view_offs, as produced by
KV cache slices in Qwen3-TTS and Qwen3-VL.
- Remove the defensive misalignment GGML_ASSERT in init_pushconst_tensor_offsets
for the binary push-constants specialization, since both get_rows.comp and
get_rows_quant.comp now correctly apply per-tensor base offsets.
- Remove the workaround CPU fallback in supports_op() for GET_ROWS, since the
Vulkan backend now handles misaligned offsets natively (no more bailout).
- Add backend test coverage with view_src0=true (ggml_view_4d into a padded
tensor) for F32, F16, Q4_0, Q4_K, Q8_0, and I32 types, exercising both the
non-quantized (get_rows.comp) and quantized (get_rows_quant.comp) paths
with non-zero view_offs that reproduce the original Qwen3-TTS crash.
* tests: trim redundant comments in test_get_rows vs0 region
* tests: trim redundant comments in test_get_rows vs0 region (follow-up)
* vulkan: bind tensor base for binary ops, pass full view_offs via push constants
For ops using vk_op_binary_push_constants (GET_ROWS, ADD, SUB, MUL, etc.),
bind the view_src base and pass the full view_offs divided by type_size via
push constant misalign_offsets. This avoids truncation when misalign_bytes is
not a multiple of quantized block size.
ggml_vk_tensor_subbuffer gains a use_view_offs parameter. When false, the
binding points to vk_tensor_offset (base) and size includes view_offs.
init_pushconst_tensor_offsets<binary> computes a/b/d_offset directly from
tensor->view_offs, which is always row-aligned and therefore exact.
Added non-zero view offset (offset_rows=3) backend tests for GET_ROWS across
all_types with be1={1,7}, v={false,true}, skipping gradient setup for view
tensors (GGML_OP_VIEW fails ggml_set_param).
All 223 GET_ROWS tests pass on Vulkan (NVIDIA RTX 5060 Ti).
* vulkan: bind aligned offset for binary ops, pass adjusted misalign via push constants
For ops using vk_op_binary_push_constants (GET_ROWS, ADD, SUB, etc.), bind
the buffer to an aligned position near the view offset (not the tensor base)
and pass the adjusted misalignment via push constants.
ggml_vk_get_adjusted_misalign finds the smallest misalign that is both a
multiple of minStorageBufferOffsetAlignment and type_size, ensuring
misalign/type_size is exact (no truncation for quantized block types).
ggml_vk_tensor_subbuffer gains use_view_offs parameter. When false, binds
to (target - adjusted_misalign) instead of the view_src base, keeping the
offset small enough for 16-bit/8-bit push constant fields.
Added non-zero view offset (offset_rows=3) backend tests for GET_ROWS across
all_types with be1={1,7}, v={false,true}, skipping gradient setup for view
tensors (GGML_OP_VIEW fails ggml_set_param).
All 223 GET_ROWS tests pass on Vulkan (NVIDIA RTX 5060 Ti).
* vulkan: bind aligned offset for binary ops, fix UMA offset mismatch
For ops using vk_op_binary_push_constants (GET_ROWS, ADD, SUB, etc.), bind
the buffer to an aligned position near the view offset (not the tensor base)
and pass the adjusted misalignment via push constants.
Added ggml_vk_tensor_physical_offset to unify physical offset lookup across
UMA and non-UMA devices. On UMA, resolves via ggml_vk_host_get(tensor->data);
otherwise uses vk_tensor_offset(t) + t->view_offs. Both get_misalign_bytes and
the new ggml_vk_get_adjusted_misalign helper build on top of this function,
so buffer bindings and push constant offsets are always consistent regardless
of device memory model.
ggml_vk_get_adjusted_misalign finds the smallest misalign that is both a
multiple of minStorageBufferOffsetAlignment and type_size, ensuring
misalign/type_size is exact (no truncation for quantized block types) while
remaining small enough for 16-bit/8-bit push constant fields
(adjusted_misalign < lcm(align, type_size)).
ggml_vk_tensor_subbuffer gains use_view_offs parameter. When false, binds
to (physical_offset - adjusted_misalign) on both UMA and discrete GPUs,
fixing a bug where the UMA host_get path previously skipped the adjusted
misalign binding and returned the target offset directly.
Added non-zero view offset (offset_rows=3) backend tests for GET_ROWS across
all_types with be1={1,7}, v={false,true}, skipping gradient setup for view
tensors (GGML_OP_VIEW fails ggml_set_param).
All 223 GET_ROWS tests pass on Vulkan (NVIDIA GeForce RTX 5060 Ti).
* finish misalignment fix
* supports_op changes for openvino/webgpu
---------
Co-authored-by: AiChiTuDouPian <[email protected]>
|
||
|
Jeff Bolz
|
9ac8c408a3 |
vulkan: rms_norm fusion opportunities (#28024)
Support RMS_NORM + MUL + ADD (+ MUL) and RMS_NORM + VIEW + SET_ROWS. Extend ROPE + VIEW + SET_ROWS to support IMROPE. Worth around 4% in gemma4 on my system. |
||
 Anjielon
|
8fe90e1fbf |
vulkan: add TQ1_0 support (mm, mat-vec, mat-vec-id, dequant, get_rows) (#27765)
* vulkan: add TQ1_0 support (mm, mat-vec, dequant, get_rows) * vulkan: pack TQ1_0 powers of 3 into a 32-bit constant Replaces the constant array with a packed 32-bit value (7 bits per entry, max 81 < 128) extracted with shift/mask, as suggested in review — avoids a constant array that may not be kept in registers. test-backend-ops on gfx1151: tq1_0 MUL_MAT 11/11, MUL_MAT_ID 6/6, GET_ROWS 4/4, unchanged. * vulkan: address review - shared TQ1_0 decode helpers, fix standalone dequant shader Review feedback from jeffbolznv, all points: - Move the packed-pow3 decode into shared helpers in types.glsl (tq1_0_byte_of / tq1_0_digit_of / tq1_0_trit) and use them from dequant_funcs.glsl, mul_mm_funcs.glsl, dequant_funcs_cm2.glsl and dequant_tq1_0.comp instead of repeating the logic. The cm2 path also drops its constant array for the packed-constant extraction. - Translate all remaining comments to English. - dequant_tq1_0.comp: use dequant_head.glsl. The shader previously declared its own single-field push constant while the pipeline is created with the 5-field layout, so p.ne read the wrong field - confirmed broken, as suspected in review. - Fix wg_denoms for the standalone dequant pipeline: one invocation decodes 4 elements with local_size 256, so a workgroup covers 256*4 elements, not 256*16. With the old value the dispatcher launched a quarter of the required workgroups. Verified by temporarily forcing the dequant + f16 matmul path for TQ1_0 (hack not committed): test-backend-ops MUL_MAT passes through the rewritten standalone shader, and the standard MUL_MAT / MUL_MAT_ID / GET_ROWS tq1_0 cases still pass on Vulkan (AMD gfx1151). * vulkan: address review — English comments, shared tq1_0_trit, trim TQ1_0 test cases - mul_mat_vec_tq1_0.comp: drop leftover non-English comment and the local POW3_PACKED constant; all decode sites now call tq1_0_trit() from types.glsl - types.glsl / dequant_funcs_cm2.glsl: ASCII-only, drop stale reviewer note - test-backend-ops: remove the oversized MUL_MAT_ID case (432 MiB A tensor, ~172 GFLOP reference); move the two remaining ones next to the other backend-specific mul_mat_id one-offs and document why they are needed * metal: decline TQ1_0 for GET_ROWS and mat-mul in supports_op The new TQ1_0 cases in test-backend-ops exposed that the Metal backend claimed support for GET_ROWS/MUL_MAT/MUL_MAT_ID with TQ1_0 sources while having no such kernels (ggml_metal_library_compile_pipeline aborted on the missing kernel_get_rows_tq1_0). Decline the type so the ops fall back to the CPU, matching the existing NVFP4 handling on the same lines. Assisted-by: Claude Fable 5 * vulkan: trim the TQ1_0 comments Addresses @0cc4m's review: keep only what the code does not already say. Removed the block-format recaps (the layout is right there in the struct) and the step-by-step decode walkthrough. Kept the two facts a reader cannot infer: the 8-bit truncation is part of the format, not an optimisation, and the powers of 3 are packed into one uint so they do not end up in a constant array that may miss the registers. No functional change. * vulkan: address review — trim comments, fold Metal check, drop unused _v Per @0cc4m's review: - dequant_funcs.glsl, dequant_funcs_cm2.glsl: drop the "see types.glsl" pointers — they apply to every quant and say nothing specific. - dequant_tq1_0.comp: drop the wg_denoms note. It is a precondition, not information. - mul_mm_funcs.glsl: same pointer removed. - types.glsl: the comment on tq1_0_trit is down to the one fact the code cannot show — the 8-bit truncation is part of the format, matching the C reference, not an optimisation. - dequant_funcs_cm2.glsl: removed dequantFuncTQ1_0_v and its define. You were right that it is optional: it wrapped four scalar decodes and vectorised nothing, and mul_mm_cm2.comp already guards the path with `#if defined(dequantFuncA_v)` (DATA_A_F32 omits it the same way). - ggml-metal-device.m: folded TQ1_0 into the existing NVFP4 check instead of a separate block, and dropped both comments. - test-backend-ops.cpp: the two mul_mat_id cases stay — they cover the block-stride loop and the per-expert base offset that k == 256 alone never reaches — but the comment is now one line instead of five. Kept: the one-line labels on the three block regions in mul_mat_vec_tq1_0.comp and on tq1_0_byte_of(). Those state the 5-trits-per-byte packing, which the loop bounds do not show. Happy to remove them too if you prefer. Re-verified on AMD gfx1151 (Vulkan), test-backend-ops, 2/2 backends passed: MUL_MAT 9 TQ1_0 cases, MUL_MAT_ID 5, GET_ROWS 4 — all OK, no failures. The coopmat2 path is unchanged apart from the removed _v define. |
||
 
|
3ad1ba7336 |
[Model] Support for Spark2_5ForCausalLM implementation (#27868)
* Add Spark3 Model * rename spark3 -> spark2_5 Co-authored-by: Sigbjørn Skjæret <[email protected]> Co-authored-by: dongjiang <[email protected]> |