Superminaren/llama.cpp
1
0
Fork 0
mirror of https://github.com/ggml-org/llama.cpp.git synced 2026-09-04 02:37:27 +02:00
Code Issues Packages Projects Releases Wiki Activity

CUDA: Allow concurrent streams per split for multi-GPU (#28198)

Browse Source 
* CUDA: Allow CUDA optimization per split for multi-GPU.

Previous guard caused multi-GPU to skip the graph optimization.  The
graph is already split per device and the optimization doesnt run
over the whole model but once per split, and thus should be allowed.
However, the CUDA event ggml_cuda_concurrent_event belongs to
whichever GPU was "current" when created. If the pass ran while
GPU 0 was current, it would stick and during event creation for the
second GPU it would land on GPU 0.

The fix: set the device explicitly ggml_cuda_set_device(cuda_ctx->device);
Default behaviour remains unchanged, only active for GGML_CUDA_GRAPH_OPT=1.
Explicit device setting pattern re-used from ggml_backend_cuda_graph_compute.

* Update ggml/src/ggml-cuda/ggml-cuda.cu

Co-authored-by: Aman Gupta <[email protected]>

---------

Co-authored-by: tannerbruhn <[email protected]>
Co-authored-by: Aman Gupta <[email protected]>
This commit is contained in:
Tanner Bruhn
2026-09-03 18:03:03 +08:00
committed by GitHub
co-authored by Aman Gupta tannerbruhn
parent c7bda030e7
commit 0ba6499c3b
1 changed files with 3 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
ggml/src/ggml-cuda/ggml-cuda.cu
+3 -1
View File
@@ -4542,10 +4542,12 @@ static void ggml_backend_cuda_graph_optimize(ggml_backend_t backend, ggml_cgraph
ggml_cuda_stream_context & stream_context = cuda_ctx->stream_context();
stream_context.reset();
if (!use_cuda_graph || ggml_backend_cuda_get_device_count() != 1) {
if (!use_cuda_graph) {
return;
}
ggml_cuda_set_device(cuda_ctx->device);
// number of out-degrees for a particular node
std::unordered_map<const ggml_tensor *, int> fan_out;
// reverse mapping of node to index in the cgraph