improve the --fit algorithm to take into account the actual peak
required VRAM for a given context size on a SYCL backend.
This includes both properly accounting for how much VRAM is required
when the allocated context is fully used (which makes the reported
context drop below what it did before, but stop it OOMing) as well
as preventing some overly-conservative calculations which meant too much
VRAM was being reserved.
Tested on a Arc b70 with unsloth's qwen3.8 (Q4_K_XL), able to get 262144 context,
fully usable, with q8_0 KV and MTP and 4k ubatch size using --fit-target 1
The N padding is needed for mul_mat, but not mul_mat_id. For mul_mat_id,
we indirect the row index through a shared memory lookup table which avoids
any OOB row coordinate. But that callback doesn't bounds check K, so we
actually need K padding instead.
* port setup-build.ps1 to setup_sdk.py, to facilitate installation of Hexagon and OpenCL SDKs on Windows
* rename setup_sdk.py -> setup-sdk.py
* flake8 fix: print() -> logger.info()
---------
Co-authored-by: Kristopher Urquhart <[email protected]>
* vulkan: fix missing view-alias dependencies in ggml_vk_graph_optimize
is_src_of doesn't treat two views of one tensor as dependent, so the optimizer reorders nodes across aliased reads and writes.
Result: silently wrong tokens under greedy decoding, different output on every server start, and invalid speculative-decoding acceptance, with nothing logged.
Hits Qwen3.8's recurrent state (and any model with view-aliased state) on AMD and NVIDIA Vulkan. CUDA is clean.
Compare view_src bases on both sides.
Fixes#27805
* vulkan: don't treat view/no-op nodes as aliasing dependencies
Nodes whose op is NONE, RESHAPE, TRANSPOSE, VIEW or PERMUTE execute nothing, so aliasing through them is not a real dependency. The previous base comparison matched them anyway, which only costs the optimizer reordering freedom.
Co-authored-by: Jeff Bolz <[email protected]>
* vulkan: make the lambda parameter const and capture is_empty in is_src_of
Code will not compile without these changes.
is_src_of has an empty capture list, so is_empty was not visible inside it, and is_empty took a non-const pointer, while is_src_of receives const ones. Other call sites pass non-const pointers, which still convert as usual.
---------
Co-authored-by: Jeff Bolz <[email protected]>
* ggml : fix conv_transpose_2d for multiple batches
ggml_compute_forward_conv_transpose_2d_impl only computed the first
batch (ne[3] of the destination); every batch after the first was left
as zero. Both the src1 permutation and the main compute loop now iterate
over the batch dimension, and the work buffer size in ggml_graph_plan is
scaled by the src1 batch count so the extra permuted batches fit. A
multi-batch test case is added to test-backend-ops.
Fixesggml-org/ggml#1448
* metal : fix conv_transpose_2d for multiple batches
The kernel only computed batch 0 of the input (src1->ne[3]); every
output batch after the first was left as zero, so multi-batch
conv_transpose_2d results diverged from the CPU reference.
The grid now covers all batches (OW x OH x OC x N), the kernel decodes
the batch from the grid z coordinate and offsets both the input and
destination indices accordingly. nb3 is passed in the kernel args.
Assisted-by: pi:llama.cpp/Qwen3.8-27B
---------
Co-authored-by: Georgi Gerganov <[email protected]>
Ruben Ortlam
Nick Farrell
Jeff Bolz
Jhen-Jie Hong
kurquharandKristopher Urquhart
Xuan-Son Nguyen
Eric A StaleeandJeff Bolz
Tekin ErtekinandGeorgi Gerganov