improve the --fit algorithm to take into account the actual peak
required VRAM for a given context size on a SYCL backend.
This includes both properly accounting for how much VRAM is required
when the allocated context is fully used (which makes the reported
context drop below what it did before, but stop it OOMing) as well
as preventing some overly-conservative calculations which meant too much
VRAM was being reserved.
Tested on a Arc b70 with unsloth's qwen3.8 (Q4_K_XL), able to get 262144 context,
fully usable, with q8_0 KV and MTP and 4k ubatch size using --fit-target 1
The N padding is needed for mul_mat, but not mul_mat_id. For mul_mat_id,
we indirect the row index through a shared memory lookup table which avoids
any OOB row coordinate. But that callback doesn't bounds check K, so we
actually need K padding instead.
* port setup-build.ps1 to setup_sdk.py, to facilitate installation of Hexagon and OpenCL SDKs on Windows
* rename setup_sdk.py -> setup-sdk.py
* flake8 fix: print() -> logger.info()
---------
Co-authored-by: Kristopher Urquhart <[email protected]>
* vulkan: fix missing view-alias dependencies in ggml_vk_graph_optimize
is_src_of doesn't treat two views of one tensor as dependent, so the optimizer reorders nodes across aliased reads and writes.
Result: silently wrong tokens under greedy decoding, different output on every server start, and invalid speculative-decoding acceptance, with nothing logged.
Hits Qwen3.8's recurrent state (and any model with view-aliased state) on AMD and NVIDIA Vulkan. CUDA is clean.
Compare view_src bases on both sides.
Fixes#27805
* vulkan: don't treat view/no-op nodes as aliasing dependencies
Nodes whose op is NONE, RESHAPE, TRANSPOSE, VIEW or PERMUTE execute nothing, so aliasing through them is not a real dependency. The previous base comparison matched them anyway, which only costs the optimizer reordering freedom.
Co-authored-by: Jeff Bolz <[email protected]>
* vulkan: make the lambda parameter const and capture is_empty in is_src_of
Code will not compile without these changes.
is_src_of has an empty capture list, so is_empty was not visible inside it, and is_empty took a non-const pointer, while is_src_of receives const ones. Other call sites pass non-const pointers, which still convert as usual.
---------
Co-authored-by: Jeff Bolz <[email protected]>
* ggml : fix conv_transpose_2d for multiple batches
ggml_compute_forward_conv_transpose_2d_impl only computed the first
batch (ne[3] of the destination); every batch after the first was left
as zero. Both the src1 permutation and the main compute loop now iterate
over the batch dimension, and the work buffer size in ggml_graph_plan is
scaled by the src1 batch count so the extra permuted batches fit. A
multi-batch test case is added to test-backend-ops.
Fixesggml-org/ggml#1448
* metal : fix conv_transpose_2d for multiple batches
The kernel only computed batch 0 of the input (src1->ne[3]); every
output batch after the first was left as zero, so multi-batch
conv_transpose_2d results diverged from the CPU reference.
The grid now covers all batches (OW x OH x OC x N), the kernel decodes
the batch from the grid z coordinate and offsets both the input and
destination indices accordingly. nb3 is passed in the kernel args.
Assisted-by: pi:llama.cpp/Qwen3.8-27B
---------
Co-authored-by: Georgi Gerganov <[email protected]>
* vulkan: add hoisting support for row IDs and expert count in shaders
* use hoisted row ids in coopmat2
* vulkan: address review feedback on count_experts
- use vk_op_count_experts_push_constants instead of a raw uint vector
- apply the fastdiv trick to the ne00 div/mod in count_experts
- compute the per-expert offsets with subgroupExclusiveAdd when the
device supports it, keeping the serial path as fallback
- document the data_d layout and the hoisted_row_id_words bound
- drop a leftover debug print in ggml_vk_matmul_id
* vulkan: use init_pushconst_fastdiv for count_experts push constants
* vulkan: refine comments for row ID hoisting and data layout in count_experts shader
* Whitespace
---------
Co-authored-by: Jeff Bolz <[email protected]>
This adds fa_vec_tuned_table records for Apple M4 to ggml-metal-tuning.cpp.
Includes F16, Q4_0, Q4_1, Q5_0, Q5_1, and Q8_0. (M4, 10 GPU Cores)
Co-authored-by: Strongtut <[email protected]>
* OpenVINO Backend: Fuse IM2COL + MatMul convolution into OpenVINO convolution
* ci:ggml-ov: Skip recurrent state rollback tests
* ci:ggml-ov: Skip recurrent state rollback tests
* Update OPENVINO.md
* ggml-openvino : add env-var gated op support debugging
* Fix ggml_rope_set_offset case
* OpenVINO backend: Support Whisper.cpp
* Fix code style
* openvino : enable qwen35 on NPU
Static shapes:
- get_graph_input_shape() left the s_copy / s_copy-leaf inputs dynamic
([1,1,1,-1]) even in static mode, which propagated a dynamic slot dim through
GET_ROWS into the conv/GDN state, the state reshapes and the GDN output.
- With -np 1 the s_copy defrag remainder gathers zero rows; short-circuit that
CPY to the untouched cache instead of emitting a degenerate Slice/Concat, and
skip binding its zero-byte ggml tensor as an output (the dynamic path already
did the latter, the static path wrote the full cache over a 0-byte buffer).
Token-count independence:
- In static mode the compiled model's token count is the prefill chunk size or
1, not the captured cgraph's. Offsets derived from the captured count were
therefore wrong. Anchor the GDN state slice at the end of the packed
[attn | state] output and drop the rs_src_begin runtime inputs, and make
VIEWs over the GDN output / conv_input pass through so the consumer does the
slicing.
- CONT could not identify its token axis when the graph was captured with a
single token (every trailing dim has the same stride and size 1) and baked
the captured shape into the prefill model.
Chunked prefill:
- The last chunk is padded with fabricated tokens. Attention masks them, but
the recurrent path folded them into cache_r/cache_s permanently. Add a
chunk_valid_len runtime input, use it to zero g and beta for padded steps
(making the recurrence an exact identity) and to end the conv snapshot window
at the last valid token, and disable the recurrent-cache reset after the
first chunk so earlier chunks are not wiped.
- get_is_prefill() and the chunk loop bound read inp_pos->ne[0] directly, but
IMROPE stacks 4 position planes, so every decode step was run through the
padded prefill model and the loop ran extra out-of-bounds chunks.
cache_rs_reset_idx/len now stay runtime Parameters in static mode, since
can_reuse_statically() does not invalidate the cached model on ComputeParams
changes. Add GGML_OPENVINO_FORCE_STATIC to exercise the static path on CPU.
* Update to OpenVINO 2026.3.1
* ggml-openvino: forward NPU compilation mode parameters
Add GGML_OPENVINO_NPU_COMPILE_CONFIG to the backend's cached environment so callers can configure the NPU compiler without using the generic property escape hatch.
When the value is non-empty, pass it to OpenVINO as NPU_COMPILATION_MODE_PARAMS. This enables settings such as optimization-level=3 for NPU compilation while preserving the existing behavior when the variable is unset and leaving CPU and GPU configuration unchanged.
Document the variable, its NPU-only scope, and the optimization-level=3 example in the OpenVINO backend runtime configuration table.
* ggml-openvino : support RELU, POOL_2D, QUICK_GEGLU, and ROLL ops
* reorder op table
* exclude GPU/NPU failing POOL_2D case
* move op type detection to compute_op_case
* Relax rope supported cases
* Fix pool case
* Update openvino doc, gpu driver in ov docker
* openvino: remove unused static remote context branch
* openvino: parallelize static model build
* Apply editorconfig
---------
Co-authored-by: Mostafa Faheem <[email protected]>
Co-authored-by: Ravi Panchumarthy <[email protected]>
Co-authored-by: zhaixuejun1993 <[email protected]>
Measured at a live KV length of 34816 (32768 depth plus one 2048 ubatch),
on Qwen3.8 27B Q4_K_S:
per tensor 4 * 34816 * 256 * 2 B = 71.3 MB
staged per call K and V, so 2x = 142.6 MB
traffic per call read once, write once = 285.2 MB
traffic per ubatch 285.2 MB * 16 calls = 4.56 GB
One ubatch is one ggml_cgraph submission (llama_context::process_ubatch ->
graph_compute), so that 4.56 GB is the cost of a single 2048-token prefill
chunk, and it scales with the live KV length: the first ubatch of the same run,
at seq = 2048, moves 0.27 GB.
Reproduce the two measured inputs with:
GGML_SCHED_DEBUG=2 llama-bench -m MODEL -p 8 -n 0 -r 1 -ngl 0 \
-fa on -ctk f16 -ctv f16 -v > nd.txt 2>&1
grep -E 'n_layer|n_head_kv|n_embd_head_k' nd.txt
awk '/node # 0 /{g++} g==1 && /\(FLASH_ATTN\)/{n++} END{print n+0}' nd.txt
Ruben Ortlam
Nick Farrell
Jeff Bolz
Jhen-Jie Hong
kurquharandKristopher Urquhart
Xuan-Son Nguyen
Eric A StaleeandJeff Bolz
Tekin ErtekinandGeorgi Gerganov
ravel7524andJeff Bolz
StrongtutandStrongtut
Ozymandias_EBON
Titaniumtown