* metal : fix more leaks due to missing autoreleasepools
* metal : rename variable
* metal : fix another missing pool warning
Co-authored-by: YiChen Lv <[email protected]>
---------
Co-authored-by: YiChen Lv <[email protected]>
* qwen4exp: follow up fixes
* -kvu NaN collapse fix
Assisted-by: Claude
* indexer cache ext.x/ext.y restore fix
Assisted-by: Claude
* kv-cells: rename seq_set to seq_get_all
seq_get is already taken by the single-id getter, so the suggested name
cannot be overloaded on return type alone.
Assisted-by: Claude
* memory-hybrid-idx: implement set_input_qsa on the memory class
The context held the whole implementation, where the pattern elsewhere is a
thin context forwarding to the memory class, as llama_kv_cache_context does
for set_input_kq_mask. The body reads no context state, so it moves unchanged
and the context keeps a forwarder.
Also shortens the seq_get_all comment as suggested.
* tests: check that a sequence state survives a save/restore round-trip
Saves seq 0, erases it, restores the blob and saves again, requiring the two
blobs to match. Compares blobs rather than generated text, which cannot see a
field dropped on the way back in.
Note this passes on master for qwen4exp, so it does not demonstrate the
ext.x/ext.y drop this PR fixes; reaching that needs 2D mrope content.
* tests: give the synthetic qwen4exp a PLE so the state test bites
has_cell_ext() is n_pos_per_embd() > 1 || ple_n_heads > 0, and the indexer
cache sets rope_type = NONE, so without a PLE it serializes no cell ext at
all and the round-trip test cannot see a dropped ext.x/ext.y. With one,
removing the ext_set restore in state_read_meta fails the test: 198 of
335692 bytes differ, first at offset 282092.
Loading such a model needed two fixes:
- the row count of per_layer_token_embd came from require_weight(), which a
model synthesised from metadata alone has no file to answer. Derive it
from the head ranges and prefer the file's padded count where there is one.
- the PLE conv history is a row of the recurrent cache, so a PLE on a full
attention layer dereferenced a null p_l. Reject it at load time instead.
The meta mirror is skipped for qwen4exp. It returned NaN logits before this
fixture carried a PLE, which the nmse check passes since a NaN comparison is
false, and aborts with one. -sm tensor on real devices works.
Assisted-by: Claude
* llama: disable -sm tensor for qwen4exp
test-llama-archs skipped the tensor split for this arch from inside the
test, so the arch still advertised support it does not have. Declare it in
llm_arch_supports_sm_tensor instead and drop the test-side exception; the
existing llm_arch_supports_sm_tensor branch then does the skipping.
Assisted-by: Claude
* metal : request Metal 4.0 language version for the tensor API
* metal : load the tensor API kernels from a separate metallib
* tests : add external-metallib tensor API regression test
* metal : fix metallib build order for the tensor API kernels
MTP speculative decoding needs the target state to move back by the
number of rejected draft tokens. Without rollback support the context
is classified as SEQ_RM_TYPE_FULL and the server serializes the whole
recurrent state to host memory on every round, which costs more than
the drafting saves.
The recurrent cache already holds n_rs_seq + 1 snapshot planes and the
delta net writes its SSM state into them, but build_conv_state_at wrote
a single plane, so a rollback restored a convolution history that was
never captured. It now writes one snapshot per slot, each ending one
token earlier, for the delta net QKV convolution and for the PLE
convolution alike.
Measured on Qwen3.8-Flash-Next UD-Q4_K_XL with the standalone MTP
draft, n-max 3 and a single slot: decoding reaches 183 tok/s on code
and 144 tok/s on prose. The same branch before this change, where the
server falls back to checkpointing the state to host memory, reaches
123 and 83 tok/s, for 108 tok/s without a draft.
* qwen4exp: sum the indexer heads by slices
The head reduction went through a transpose and a sum_rows over ne[1],
which left sum_rows with ne0 = 4, one block per row for a four element
reduction, and the transpose copied the whole block by token surface
twice on the way in.
The heads are adjacent on ne[1], so each one is a strided view and the
sum is a short chain of adds.
RTX PRO 6000, Qwen3.8-Flash-Next UD-Q4_K_XL, fa on, 55k context, warm
runs on top of #28011:
prompt processing 2170 -> 2366 t/s
Generation is unaffected. The removed work scales with n_blocks by
n_tokens, so the gain grows with context and with ubatch size.
* qwen4exp: drop the redundant cont on the indexer query
rope returns a freshly allocated, contiguous tensor, so the reshape that
feeds the matmul does not need a copy. ggml_reshape_3d asserts
contiguity, so a layout that would need the cont cannot slip through
silently.
Greedy output is unchanged token for token.
Address review from @ggerganov
* Batched gemm for grid IQ quants
Style updates and a bit more performance
Clean up comments
Move code around
Vectorize IQ panel decode, lower threshold for speedup
IQ panel: single-source gather layout, gate bias, vectorize interleave
Add ggml_gemm_iqp_8x8_q8_K_p4 kernel, remove gather buffer
Move IQ panel code out of repack into iqp.cpp, clean up comments
Another comment sweep
* Add myself as iqp.* codeownder
* Remove ggml_cpu_iqp_scratch_offset and ggml_cpu_iqp_src1_conv_size
* Renaming and moving
* The other half of renaming and moving
* Move macros and ggml_cpu_iqp_mul_mat_id_min_batch definition
* Update ggml/src/ggml-cpu/iqp.h
Co-authored-by: Georgi Gerganov <[email protected]>
* Add iqp_rows work buffer
* Revert "Add iqp_rows work buffer"
This reverts commit 425542991e.
* Add NUMA fallback
* Add 10 row batch tests for IQP coverage on all grid IQ types
* Swap assert for return false in support check
* Move IQP mul_mat_id test
---------
Co-authored-by: Georgi Gerganov <[email protected]>
* kv cache : batch state restore scatter reads per contiguous run
When restoring state into non-contiguous destination cells (e.g. a
prompt-cache snapshot into a fragmented ring), state_read_data issued
one small copy per KV cell - ~1.4M copies of a few KiB each for a
40k+ token restore, taking 25-63 s on the CUDA backend.
The snapshot stores cell rows in cell order, so a maximal run of
consecutive destination indices maps to one contiguous block and can
be restored with a single copy. Precompute the runs once and use them
in all three scatter loops (K, V, transposed V). Byte-identical.
The on-device reader copies with a byte cursor when the read and
write chunking differs, so the batched reads are safe for it as well.
Batching makes equal tensor counts with a different split reachable
(save ranges [2,1] vs restore runs [1,2]); the next commit teaches the
reader's 1:1 path to fall back to the byte cursor in that case.
Verified in a production setup: 1,363,616 copies / 25-63 s -> 224
copies / 221-424 ms for the same restores (42,603 cells, 4 runs).
Assisted-by: Claude Code (unsloth/qwen3.8-27b)
* context : fall back to the byte cursor when read and write chunking differ
the on-device reader copies saved state back with a 1:1 copy by tensor
index whenever the write and read sides recorded the same number of
tensors, guarded by a per-tensor size assert.
equal tensor counts do not imply equal chunking: a state restore may
batch its reads per contiguous run of destination cells while the save
used per-range reads, so both sides can record two tensors that split
the same data differently, and the assert aborts in all builds.
compare the per-tensor sizes and only take the 1:1 path when the
chunking actually matches, otherwise fall through to the existing
byte-cursor copy. both sides enumerate the same logical data in the
same order, so the cursor copy is well-defined across tensor
boundaries.
Assisted-by: Claude Code (unsloth/qwen3.8-27b)
* tests : cover state restore scatter reads on host and on-device paths
decode the same prefix on two sequences, interleaving the seq 0 cells
between the seq 1 cells, so the seq 1 cells are isolated from each
other in the kv cache (three cells, two saved ranges). save the seq 1
state, free the interleaved seq 0 cells, and restore: the destination
is then non-contiguous (two runs), and the restore-side chunking has
the same tensor count as the save-side with a different split, so the
scatter path is batched per contiguous run and the on-device reader's
byte-cursor fallback is exercised.
the restored state is saved again on the host and compared byte for
byte with the first save: the blob is serialized in sequence cell
order, so the two saves are identical if and only if the scatter
restore wrote exactly the same KV content. this documents the
byte-identical guarantee of the run-batched scatter reads.
one test per io backend: the host (CPU) path and the on-device path.
Assisted-by: Claude Code (unsloth/qwen3.8-27b)
* ui: copy the displayed text of grouped agentic responses
Agentic sessions render as a single entry anchored on the first
assistant turn, whose content is typically just the first tool call,
so the copy button wrote an empty string to the clipboard. Derive the
text sections of the whole session and copy them joined, matching the
visible response. Plain messages keep the previous behavior.
* const
* dflash : fuse the encoder into the KV injection decode
The encoder is a single fc + norm, but running it as a separate
llama_encode forced a device-to-host round trip of its output before the
injection decode could re-upload it, plus a second graph build per
round. Fold the encoder into the decoder's embd branch and feed the
target features directly to one llama_decode.
Assisted-by: Claude Fable
* nit
* Apply batched suggestions from code review
Co-authored-by: Ruixiang Wang <[email protected]>
* Fix missing references from renaming
---------
Co-authored-by: Sigbjørn Skjæret <[email protected]>
Co-authored-by: Ruixiang Wang <[email protected]>
* vulkan: RDNA3 static mat-vec rows above four columns
On RDNA3 above four columns a static 4 rows for all types benches faster than
the default.
* vulkan: RDNA3 static mat-vec-id rows
mul_mat_vec_id has no column dimension to switch on. On my Strix Halo machine,
a static 4 is faster here than the defaults across types and batch sizes.
* rpc: avoid serializing buffers from other servers
Only include remote buffer pointers when the buffer belongs to the RPC dispatcher receiving the graph. Add a two-server regression test for cross-server tensor serialization.
Assisted-by: Codex
* cont : add ref
---------
Co-authored-by: Georgi Gerganov <[email protected]>
for_each_token_in tested all LLAMA_MAX_SEQ sequences for every used cell,
while a cell almost always belongs to one. The scan now stops once the
cell's own sequences have been seen. Same visit order, same callback
arguments, so behaviour is unchanged.
get_prev_tokens is the only caller, so this affects the n-gram path.
RTX PRO 6000, Qwen3.8-Flash-Next UD-Q4_K_XL, fa on, warm runs:
55k context generation 56.3 -> 74.3 t/s
132k context generation 33.6 -> 50.9 t/s
Prompt processing is unchanged, the scan is amortised over the ubatch
there. The gain follows the number of used cells, so it grows with
context and is invisible on short prompts.
The optimized path grouped warp lanes by token and required
warp_size % n_expert_used == 0, with a single hardcoded exception
padding 6 up to 8. Every other count fell back to the generic path,
which walks the tokens one at a time with a warp reduction per token,
for each of the n_expert blocks.
The lane group only has to divide the warp, and the loop body already
guards the padded lanes with iex < n_expert_used, so the padding
generalizes to the next power of two. The 6 -> 8 case and every count
already dispatched keep the exact same padding as before.
n_expert_used = 10 now reaches the fast path. Measured on
Qwen3.8-Flash-Next (512 experts, 10 used) at 55k context on an
RTX PRO 6000, warm runs with the first one discarded:
prompt processing 2334 -> 2600 t/s
Token generation is unaffected, since a single token leaves nothing to
walk. Other expert counts reach the fast path by adding their case to
the dispatch.
- DFlash2 NVFP4 draft models produced almost no accepted speculative
tokens because the Q, K, V, and output projection scales were not
passed to the corresponding graph operations.
Rename the --tensor-read-lazy CLI argument to --lazy-mode, to match the
internal lazy_mode parameter, and add a -lzm shorthand. Sync the READMEs.
Assisted-by: pi:llama.cpp/Qwen3.8-27B
some backends (Metal, SYCL, WebGPU) require additional memory for
fleeting data for certain ops, which is reflected in their
get_alloc_size implementations.
add ggml_backend_op_alloc_size_may_expand() to the backend utils,
listing these ops, and assert in ggml_backend_buft_get_alloc_size
that a backend expanding the alloc size of a compute op only does so
for ops listed in the helper.
use the helper in the RPC backend to decide whether to query the
remote server for the actual alloc size, instead of a hardcoded list.
Assisted-by: pi:llama.cpp/Qwen3.8-27B
Lukasz Stolcman
Jhen-Jie Hong
Sigbjørn Skjæret
Niklas WenzelandYiChen Lv
Georgi Gerganov
Neo Zhang
Daniel Han
Xuan-Son Nguyen
James Francis
Ludovic Henry
Jonathan Clohessy
Pascal
Buğra Özgürsoy
ynankani
BartowskiandGeorgi Gerganov
itsnotoger
Hongqiang Wang
Jaden_Mach
Simon Teixidor
Ruben Ortlam
Shenghan Yang
codemonkey
hmirinandGeorgi Gerganov
Aman Gupta
itterative
LunalFresh
JamePeng
Ryan C
Nils Gladitz
Daya Adianto
Alessandro de Oliveira Faria (A.K.A.CABELO)