Three related changes to the Vulkan -sm tensor (tensor-parallel) all-reduce,
developed and measured on 2x Radeon AI PRO R9700 (RDNA4/GFX1201, Mesa 26.1.2):
1. Shrink the comm staging buffers on the prefill->decode transition.
ensure() only ever grew the host/tmp buffers, to the peak prefill micro-batch
(~10 MB at n_ubatch=512), then reused them for the tiny (~20 KB) decode
all-reduces. On RADV the imported external host memory is made visible across
devices on every timeline-semaphore signal, at a cost proportional to the
resident buffer size, so an oversized leftover cap stalled every decode step
(cross-device wait ~770 us vs ~30 us), collapsing decode from ~30 to ~2.4 t/s
and staying stuck for the whole session (the AMD "multi-turn crawl"). ensure()
now also shrinks when the request is much smaller than cap, with a one-time
semaphore wait so the realloc is safe against the previous async all-reduce.
Decode after a large prefill: 2.4 -> ~31 t/s, flat across prefill sizes.
2. Decide proxy vs native cross-device sync before creating the progress
timelines. They were created as exportable up front, which aborted init on
devices that cannot export timeline semaphores (e.g. llvmpipe, RADV on older
Mesa) instead of falling back to the portable CPU proxy. Now create exportable
timelines only for the native path and plain ones for the proxy path.
3. Add ggml_backend_vk_comm_allreduce_tree: an opt-in recursive halving/doubling
all-reduce for power-of-two device counts (2*log2(n) cross-device steps vs the
ring's 2*(n-1); bandwidth-optimal). Enabled via GGML_VK_COMM_TREE; the ring
stays the default and handles non-power-of-two counts. The step schedule is
built as in a reference simulation verified for n=2..8; validated at n=2
(native and forced proxy) to match the ring's greedy output byte-for-byte.
Assisted-by: Claude Opus 4.8 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01SazLJJfgpjt9Kq7JXKKnuw
With the ring as the default there is no reason to keep the slower paths:
- Remove ggml_backend_vk_comm_allreduce_pipeline (the O(n^2) all-to-all) and
GGML_VK_COMM_PIPELINE. The ring is now the unconditional large-tensor path;
the comm->ring flag and pipe_round are gone, and pipeline_ok (the "has two
queues" gate the ring needs) is renamed ring_ok.
- Remove fp32 staging and GGML_VK_COMM_FP32. The ring always stages F16 (fp32
accumulator preserved); its fp32 branch and use_f16 are removed.
Net ~-260 lines. Verified byte-identical greedy output (ring / proxy) and clean
build on 4x A16. The decode single-shot and the meta-backend butterfly fallback
are untouched.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01ApKCQ32VLqUW4Kus6tUvBL
- Remove the GGML_VK_COMM_D2D peer-buffer path entirely (helpers,
VK_KHR_external_memory_fd / VK_EXT_external_memory_dma_buf detection+enable,
comm fields, init validation, ensure() branch). On AMD it regresses badly:
the dmabuf import lands in GTT (PCIe P2P not established on stock kernels),
so peer reads stall the GPU (pp/tg down 40-90%, power collapses). Not viable
without box access to validate true VRAM P2P; revisit later.
- Make the O(n) ring the default large-tensor AllReduce. The old all-to-all
pipeline is now opt-in via GGML_VK_COMM_PIPELINE (was: ring opt-in via
GGML_VK_COMM_RING).
- Remove the GGML_VK_COMM_OFF toggle (forced the meta-backend butterfly on
Vulkan); the custom comm is always better. The generic butterfly fallback in
the meta backend stays -- it is the shared fallback for CUDA/SYCL and for
Vulkan configs without a usable custom comm (e.g. MoltenVK).
Verified byte-identical greedy across ring / pipeline / proxy / fp32 on 4x A16.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01ApKCQ32VLqUW4Kus6tUvBL
Remove all comment lines and trailing comments from the tensor-parallel comm
implementation and its device-extension hooks. No behaviour change (verified
byte-identical greedy output across ring/proxy/D2D/butterfly after the strip).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01ApKCQ32VLqUW4Kus6tUvBL
The ring transported fp32 chunks; the pipeline already halves host/peer traffic
by staging F16. Bring the ring to parity: keep tensors[i] as the fp32
accumulator (matching the pipeline's precision) but cast each chunk to F16 for
transport. The cast is folded into the recv step (the just-reduced chunk is the
next step's send), so it costs only one extra pre-cast prog value rather than a
doubled scheme; per-step up16 slots avoid a send-buffer WAR. GGML_VK_COMM_FP32
still forces fp32.
Verified byte-identical greedy output vs fp32 ring / pipeline / butterfly on
4x A16, and no regression (ring-f16 ~= ring-fp32 ~= pipeline on A16 and 4090).
The bandwidth win only shows when comm is exposed (comm-bound hosts); on this
NVIDIA box the ring's transfer/compute overlap hides the comm, so F16 is neutral
here -- same comm-hidden reason the other comm micro-opts are neutral on NVIDIA.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01ApKCQ32VLqUW4Kus6tUvBL
Opt-in path that replaces host-memory staging in the tensor-parallel AllReduce
with direct peer reads of another GPU's VRAM over PCIe P2P: each device's
partial lives in an exportable device buffer (DMA_BUF), peers import it, and the
existing comm reads host_buf[k][i] -- now peer VRAM -- unchanged. Targets the
O(n^2)-host-bandwidth scaling collapse; pairs with the O(n) ring. Ordering still
uses native/CPU-proxy semaphores (proxy auto-selects on RADV), so this is the
"D2D data + proxy semaphores" combo.
Enables VK_KHR_external_memory_fd + VK_EXT_external_memory_dma_buf. DMA_BUF is
the cross-device handle type (OPAQUE_FD memory is spec-locked to one physical
device); this matches the amdgpu PCIe-P2P dma-buf mechanism RADV/ROCm use.
Status: the fast path is AMD-targeted and UNVALIDATED -- NVIDIA's Vulkan driver
rejects cross-device fd import (vkGetMemoryFdPropertiesKHR -> memoryTypeBits=0;
confirmed against the Vulkan spec's same-deviceUUID rule and NVIDIA's own
statements), and NVIDIA has no fd-import or device-group P2P for unlinked GPUs.
So on NVIDIA it logs once and gracefully falls back to host staging -- verified
byte-identical and at host speed (pp2048 692 vs 695) on 4x A16, not the slow
butterfly. An AMD multi-GPU rig is needed to validate the actual P2P fast path
(test recipe accompanies this work, incl. how to prove real VRAM P2P vs a silent
amdgpu GTT fallback).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01ApKCQ32VLqUW4Kus6tUvBL
The ring AllReduce was gated to the native peer-import path (!comm->proxy),
so on RADV/cross-vendor setups (which use the CPU-proxy bridge because
OPAQUE_FD timeline export is unsupported) GGML_VK_COMM_RING had no effect.
Mirror the pipeline's proxy pattern in the ring: the per-step recv wait on
the previous neighbour's transfer timeline, and the cross-round WAR wait on
the next neighbour's compute timeline, are routed through the device's pxy
semaphore and a bridge enqueued for the helper thread. Reserve nsteps+1 pxy
values per round (nsteps recv bridges + 1 WAR bridge).
Validated byte-identical: native ring == proxy ring == butterfly
(660b3d04a269) on 4x A16 forced-proxy. fp32 staging only; F16 is a follow-up.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01ApKCQ32VLqUW4Kus6tUvBL
The GPU-side cross-device ordering imports each peer's OPAQUE_FD timeline
semaphore, but OPAQUE_FD payloads are driver-private, so the import only works
when all devices share a driver (e.g. two NVIDIA GPUs). On mixed drivers or
vendors it is out of spec.
Add a portable fallback: a helper thread polls each peer's progress/upload
timeline and host-signals a local timeline that the consumer's download is
parked on (core timeline semaphores plus host signal/wait, no imported handle).
Both the chunked pipeline (prefill) and the single-shot (decode) paths are
bridged, so proxy mode no longer drops decode to the meta-backend butterfly.
A capability gate (vkGetPhysicalDeviceExternalSemaphoreProperties plus a
driverUUID match) selects the proxy deterministically on unsupported configs;
GGML_VK_COMM_PROXY forces it, and the import try/catch stays as a safety net.
Measured within ~4% of the native-import path on decode and on par for
prefill, with byte-identical output.
Assisted-by: Claude Opus 4.8
Implements the backend-agnostic comm hook (ggml_backend_comm_init /
_allreduce_tensor / _free, discovered by the meta backend via
get_proc_address) for the Vulkan backend, so tensor-parallel inference no
longer falls back to the meta backend's CPU-barriered butterfly AllReduce.
Consumer GPUs have no P2P here, so the reduce stages through host memory, but
everything is ordered on the GPU via exported timeline semaphores (no CPU
barriers between layers). Each slice is split into chunks: the dedicated
transfer queue streams this device's slice out to shared host memory while the
compute queue pulls each peer chunk back as soon as it lands, so the two PCIe
directions overlap (full-duplex). Partials are cast to F16 before the host
transfer to halve the bytes on the bandwidth-bound link and added straight into
the fp32 result via the mixed-type add pipeline. Large prefill activations use
this pipeline; small (decode) tensors take a single-shot path where the fixed
per-call overhead dominates.
Roughly 2.5-3x the butterfly fallback; at long context it overtakes -sm layer
and is competitive with CUDA/NCCL on prefill. GGML_VK_COMM_OFF disables the
custom comm (falls back to butterfly); GGML_VK_COMM_FP32 forces fp32 staging.
Assisted-by: Claude Opus 4.8
When a per-device allocation exceeds the backend's max buffer size (e.g. a large
KV cache), ggml-alloc returns a multi_buffer wrapping several real buffers.
Compute-graph views inherited that multi_buffer as their backend buffer, so a
backend that casts tensor->buffer->context to its own buffer-context type (the
Vulkan backend does, e.g. in ggml_vk_tensors_overlap) dereferenced garbage and
crashed with -sm tensor (issue #22197).
A view aliases its source's storage, so it must reference the source's real
sub-buffer: set t_ij->buffer = t_ij->view_src->buffer. This is the correct ggml
invariant and a no-op in the single-buffer case.
Assisted-by: Claude Opus 4.8
* CUDA: Improve performance via less synchronizations between token (#17795)
* Adds CPU-to-CUDA copy capability to
ggml_backend_cuda_cpy_tensor_async()
* Adds function to relax sync requirements between input copies on
supported backends (CUDA for now)
* Exchanges synchronous copy with async copy function.
* Adds macro guards to allow compilation in non-CUDA builds
* Reworked backend detection in ggml-backend.cpp to avoid linking
conflicts
* Relax requirement of checks in async CUDA copies from backend and buffer type to just buffer type, to avoid linking issues
* Minor cleanup
* Makes opt-in to relax use of explicit syncs more general. Backends like
vulkan which require a synchronization between HtoD copies and graph
execution could also adopt this change now.
* Reintroduces stricter check for CPU->CUDA backend async copy via
GGML_DEVICE_TYPE_CPU.
* Corrects initialization of ggml_backend_sync_mode in
ggml_backend_sched_split initialization
* Simplifies synchronizations to adhere to `saaasg` pattern.
* Apply suggestion from @ggerganov (src->buffer to buf_src)
Co-authored-by: Georgi Gerganov <[email protected]>
* Apply suggestion from @ggerganov (src->buffer to buf_src) v2
Co-authored-by: Georgi Gerganov <[email protected]>
---------
Co-authored-by: Georgi Gerganov <[email protected]>
* Apply suggestions from @johannesgaessler code review
Co-authored-by: Johannes Gäßler <[email protected]>
* Adds single-GPU synchronizations to multi-GPU settings to fix hip backend pipeline parallel bugs.
* Scheduler Hardening: Exclude hip/MUSA from copy_from_host CPU split ->
GPU split optimization
* Scheduler Hardening: Re-adding original additional synchronizations for
non-async backends
* Adds disclaimer to hip/musa exclusion of copy_from_host. Highlights that it is out of
precaution, but that no perf-impact is visible, and that it can be
revisited separately anytime.
---------
Co-authored-by: Georgi Gerganov <[email protected]>
Co-authored-by: Johannes Gäßler <[email protected]>
* vulkan: add INTEL_PRE_XE2 arch enum and enable coopmat1 on Intel Xe-LPG Plus (1/3, Xe1-ARLH)
Co-authored-by: Xia, Jie <[email protected]>
Co-authored-by: Liu, Russell <[email protected]>
* Address comments of bf16 and trailing whitespace
* Rename INTEL_PRE_XE2 to INTEL_XE1 and remove driver workaround
* Add Windows driver check
---------
Co-authored-by: Xia, Jie <[email protected]>
Co-authored-by: Liu, Russell <[email protected]>
* ggml-cpu: fix SVE leftover path in ggml_vec_dot_f32
2D convolutions with kernel size 9 produced different results on SVE
enabled ARM devices. After debugging it turned out that ggml_vec_dot_f32
was using data from inactive lanes.
Use svmla_f32_m(pg, sum1, ax1, ay1) so inactive lanes retain sum1.
* cont : clean-up
---------
Co-authored-by: Georgi Gerganov <[email protected]>
* server: SSE replay buffer, survives client disconnect
Opt in on POST /v1/chat/completions when the client sends
X-Stream-Resume: 1 and a non empty X-Conversation-Id. The conv id is
the session identity end to end, no extra opaque token. The drain
runs detached server side and buffers SSE bytes, the generation
survives HTTP disconnect, F5, or lets users switch from iOS Safari
to another app without losing the actively generated response.
Routes:
GET /v1/stream/<conv_id>?from=N replay
GET /v1/streams[?conversation_id=X] list, drives sidebar spinners
DELETE /v1/stream/<conv_id> Stop, idempotent
Router parent fans out to children for list and delete, probes on GET
to route to the owner, fans out DELETE on POST so "one session per
conv" holds across model swaps.
WebUI: the layout snapshots /v1/streams at mount and on
visibilitychange, the sidebar reflects live inferences across all
convs. The chat page reattaches on mount, append vs fresh is detected
from existing content so continue mid stream keeps its prefix.
update_slots: on llama_memory_seq_rm refusal at a deep position, full
clear of the seq and reprefill from zero instead of GGML_ABORT.
OAI strict path unchanged when the opt in headers are absent.
* server: create stream session only after post_tasks succeeds
* server, ui: drop X-Stream-Resume, X-Conversation-Id alone enables the replay buffer
* server: drop magic 17, derive the X-Conversation-Id header length from sizeof at build time
* refactor: address review feedback from ngxson
* server-context: cleaning
* server-stream: fix use-after-free on rd
Guard stop_producer with a shared alive flag, flipped by on_stream_end
before rd dies. Prevents a late cancel (session eviction by a later
POST on the same conv_id, or a DELETE arriving after the producer
ended) from touching a destroyed rd.
* ui: fix cross-conversation contamination
Scope streaming flags per conv so one finishing does not unflag the
others, guard discoverActiveStream against concurrent runs to avoid
duplicate attaches, and stop racing syncRemoteRunningStreams for the
sidebar set.
* server-http: keep request alive in detached SSE drain
The response next() lambda may reach into *request via &req long
after on_complete reset the request shared_ptr. Capture request in
the detached thread so it outlives the drain.
* ui: address review feedback from coder543
Forward Authorization to /v1/stream and /v1/streams fetches, the resumable routes
must obey --api-key like the rest of the API.
Wrap reader.read() in a try/catch, the underlying connection drop rejects with
TypeError instead of resolving done=true, treat it as a premature end of stream
so the existing resume loop kicks in.
Freeze the model at session start in chatStreamingStates.model and thread it
through cancel and resume, the dropdown selection may have changed since the
POST and the server side identity is fixed at that time.
* format
* ui: remove unused selectedModelName
* server-stream: poll session->is_cancelled() in stream_aware_should_stop
Address review feedback from coder543. The cancel propagation through
rd.stop() relies on the slot eventually processing the cancel task and
posting a result that notifies the recv condvar, remove_waiting_task_ids
does not notify directly. Add a defensive poll on session->is_cancelled()
so the producer-side next() loop exits on its next iteration after
cancel() without waiting for the cancel task to round trip through a slot.
* server-stream, ui: replace GET /v1/streams with POST /v1/streams/lookup
Address review feedback from coder543. Listing live sessions leaks the
conversation_id of every concurrent user, which defeats the random UUID
unguessability. The new route takes {conversation_ids: [...]} in the
body and returns matches only for the ids the caller already owns, so
foreign UUIDs stay private. The router fans out the same POST to every
child and aggregates, the WebUI passes the convs visible in its sidebar.
* ui: read conv ids from IndexedDB in syncRemoteRunningStreams
The conversations store is not hydrated yet at +layout onMount, so the
sidebar spinners stayed off for background convs until the user clicked
on them. Read straight from the DB to dodge the init race.
* server-models: deduplicate stream lookup timeouts behind one constant
* ui: extract visibility kick grace into a stream constant, bump to 1000 ms
* make it safer & more simple
* server-stream: survive client disconnect via stream_pipe::finish_producer
After the RAII rewrite the generation stopped the moment the client
disconnected. httplib bails its content provider on the is_peer_alive
check at the top of write_content_chunked, so returning true from the
provider never keeps it producing: the response resets, rd is destroyed
and its task gets cancelled.
Reinstate the disconnect survival inside the pipe. stream_pipe gains
finish_producer, which pumps the response next() into the ring buffer
until the generation ends, and mark_producer_done for the clean wire
end. server-http only triggers them: mark before sink.done on a clean
close, finish in on_complete when the peer left early. No detach, no
stream logic in server-http beyond the trigger, and the strict OAI path
is untouched when no pipe is attached.
Known limitation: finish_producer pumps synchronously on the http
worker, so a disconnected stream keeps its worker busy until the
generation ends. A follow-up will move the drain off the http worker so
no worker is held.
* server-stream: drain disconnected streams on a manager owned thread
The previous commit pumped the post disconnect drain synchronously in
on_complete, on the http worker, so a disconnected stream kept its
worker busy until the generation ended. Under a wave of reloads or tab
closes that pins workers from the pool.
Move the drain off the http worker. on_complete now hands the response
to stream_session_manager::adopt_orphan, which pumps it to completion on
a manager owned thread and releases the worker at once. One thread per
disconnected stream still generating, stored in a list, joined and
reaped on the next adopt, by the GC, and at shutdown. No detach, the
thread lifecycle is fully owned by the manager. needs_drain gates the
handoff so a cleanly finished stream never spawns a thread, and the
strict OAI path stays untouched when no pipe is attached.
stop_gc now cancels sessions before finalizing them, so an in flight
drain sees is_cancelled and exits instead of blocking the shutdown join
until the generation ends naturally.
* ui: add missing JSDoc
* server-stream: drain on the http worker, drop the manager thread
Address @ngxson review: httplib runs a large dynamic pool and a worker
blocked in next() sits on a condvar instead of burning cpu, so draining
the rest of the generation on that worker is fine and much simpler than
a dedicated thread.
on_complete calls finish_producer directly again. Removes adopt_orphan,
the orphan thread list and its reaping, the stop_gc session cancel that
only existed to unblock those threads, and the now dead drain_shutdown
flag.
* server-stream: split stream_pipe into producer and consumer classes
Address @ngxson review: one class covering both ends was messy. stream_pipe
is now a base holding the session and is_cancelled, with stream_pipe_producer
(write, mark_producer_done, finish_producer, cleanup, finalizes on destruct)
and stream_pipe_consumer (read only, no finalize) deriving from it.
Drops the is_producer_ discriminator and its runtime guards, the type now
encodes the role. res.spipe is retyped to shared_ptr<stream_pipe_producer>
since it is only ever a producer. No behavior change.
* server-stream: rename producer methods to unix pipe semantics
Address @ngxson review: mark_producer_done becomes done(), finish_producer
becomes close(), matching a unix pipe write end. The producer_done_ member
follows as done_. write() is unchanged. No behavior change.
* server, ui: route resumable streams via a conv map, persist resume identity
Address ngxson review: drop the polling probe, proxy_post records a conv_id ->
model map and the stream routes resolve the owning child with one lookup. The
map is the single source of truth, the ::model suffix stays for child session
uniqueness but the router never parses it.
UI: the server keys a session by the POST time identity (conv::model), but reload
probed with the bare conv id and missed model tagged sessions, so F5 stopped the
stream and sidebar spinners stayed off. Persist the model and rebuild the exact
identity on resume, single conv and bulk sidebar both send it.
Add unit coverage for the identity round trip.
* ui: resolve continue target by id to stop cross-conversation flash on switch
* ui: skip stream resume when the abort is intentional
* server: move the conv id to model map into a self contained tracker
Address review from ngxson: server_models held two mutexes side by side, the
global one and a bare conv_model_mu guarding a loose map, which made the locking
hard to follow. Wrap the map and its lock in a small conv_model_tracker struct
that owns its mutex, one mutex per struct. The remember, lookup and forget
methods move inline into the tracker, server_models exposes a single conv_models
member and the routes call models.conv_models.lookup and friends. No behavior
change, the map stays the single source of truth for routing resumable streams
to a child.
* ui: replace stream magic values with enums and shared constants
Address review from allozaur: lift the inline literals around the resumable
stream code into named symbols so the intent is explicit and reusable.
* ui: fold the stream resume and discovery helpers into ChatService
Address review from allozaur: drop the two standalone stream-*.service files.
They were used only by the chat service and store, carried no shared state, and
did not follow the static class pattern the other services use, so a separate
abstraction was not warranted. Move the helpers onto ChatService as static
methods. No behavior change, tests now exercise them through ChatService.
* docs: document the SSE replay buffer in server README-dev
Add the resumable streaming section, list stream_session_manager in the
backend component inventory, and link PR 23226 in the related PRs.
* ui: align attachServerStream call with onCompletionId param in handleStreamResponse
* server-http: rename del_ to del to match get and post
* ui: address review feedback from allozaur
* ui: drop duplicate SSE constants, keep sse.ts canonical
* ui: use svelte:document for the visibilitychange listener
address review from allozaur: replace the manual document.addEventListener
in onMount with a declarative <svelte:document onvisibilitychange>. svelte
handles attach, detach and SSR, so the typeof document guard and the onMount
cleanup go away. onMount keeps only the first load snapshot.
* server: trim redundant stream drain comments
Address review from ngxson
* server: balance and clean up stream comments
remove redundant comments and tighten the verbose ones across the resumable
stream code, keeping the concurrency and lifetime rationale that is not obvious
from the code. also fix two stale comments in server.cpp and server-models.h
that still described the old ::model suffix probe and fan out routing, now
replaced by the conv_id -> model map
Address review from ngxson
* ui: balance and clean up stream comments
dedup repeated rationale (frozen conv::model identity, the lookup privacy note,
the abort patterns) down to one canonical spot, tighten the verbose blocks, and
keep the concurrency and resume-offset reasoning. fix stale comments in
stream-identity.ts and chat.service.ts that still described the old loopback
probe and fan out routing, now the conv_id -> model map.
---------
Co-authored-by: Xuan Son Nguyen <[email protected]>
Adds an opt-in LLAMA_BUILD_MTMD CMake option so build-xcframework.sh
can link libmtmd.a into the framework binary without pulling in the
rest of tools/ (which doesn't cross-build cleanly to iOS/tvOS/visionOS).
- CMakeLists.txt: new option, default OFF. When on with
LLAMA_BUILD_TOOLS=OFF, only the tools/mtmd subdir is added. Useful
for any binding that wants just libmtmd (Apple XCFramework, WASM).
- tools/mtmd/CMakeLists.txt: gate the CLI exe targets on
LLAMA_BUILD_TOOLS. Gating on LLAMA_BUILD_COMMON is not enough — it
defaults ON in standalone builds and visionOS xcodebuild then fails
with "install TARGETS given no BUNDLE DESTINATION for MACOSX_BUNDLE
executable target 'llama-mtmd-cli'".
- build-xcframework.sh: turn the option on, pass -DLLAMA_BUILD_MTMD,
add libmtmd.a to combine_static_libraries, and copy mtmd.h and
mtmd-helper.h into the framework Headers dir. The umbrella module
map then exposes them, so Swift / Obj-C consumers can import the
mtmd C API directly.
After this, nm on ios-arm64/llama.framework/llama shows 52 _mtmd_
symbols. Verified end-to-end: a Swift target links the produced
framework and calls mtmd_default_marker, mtmd_bitmap_init, etc.
without a shim on macos / iphoneos / iphonesimulator / xros slices.
Co-authored-by: Abraham Gonzalez <[email protected]>
* Sycl tp stage1 (#1)
* SYCL: tensor parallelism (--split-mode tensor) for dual-GPU
Adds the comm_init/comm_free/comm_allreduce_tensor trio that the
meta-backend queries via get_proc_address to enable backend-specific
all-reduce, mirroring the pattern used by ggml-cuda.cu.
For N=2 (the common dual-GPU case) implements a degenerate ring
all-reduce with two size-branched paths:
* Small (nelem < 32768): FP32 direct memcpy + per-device ADD kernel
chained via depends_on(memcpy_event). 4 SYCL submissions/call.
* Large (nelem >= 32768): BF16-compressed. Each device compresses
FP32 -> BF16 in a local outbox, cross-device memcpys to the peer's
inbox (HALF the PCIe bytes), then decompresses + adds into the
local FP32 partial. 6 SYCL submissions/call but PCIe bytes halved
-- wins for any tensor where PCIe dominates kernel time.
Threshold and BF16 path pattern mirror the CUDA NCCL allreduce.
Storage: ONE persistent uint8_t buffer per device, 4 * nelem bytes
(matches both path layouts: FP32 nelem floats; BF16 outbox+inbox =
2 * nelem uint16_t each). Single alloc+free per device keeps the
SYCL pool's strict-LIFO invariant trivial.
Initial impl handles N=2 FP32 contiguous tensors. Other cases return
false, causing the meta-backend to use its generic butterfly fallback.
Per-call sync is intentionally omitted. SYCL in-order queue semantics
ensure that the meta-backend's next compute on the same per-device
queue waits for our final ADD, and the next allreduce's first op on
the same persistent buffer waits via the same queue. Only comm_free
does an explicit final wait.
OneCCL is NOT used: OneCCL 2021.17 hardcodes single-device-per-process
in communicator_impl.hpp:47 (condition devices.size() == 1), which is
incompatible with llama.cpp's single-process multi-GPU model.
Measured on dual Intel Arc Pro B70 (NEO 26.05.x, oneAPI 2025.3 +
DPC++ nightly):
Llama-3.3-70B Q4_K_M, -sm tensor -fa 1 -ctk f16 -ctv f16:
pp512 = 377.08 t/s (vs 313.65 layer mode = +20.2%)
tg128 = 17.40 t/s (vs 9.74 layer mode = +78.6%)
Qwen3-Coder-Next-80B-A3B Q3_K_M (MoE):
pp512 = 216.56 t/s (vs 156.58 meta-backend butterfly = +38.3%)
tg128 = 17.60 t/s (vs 14.31 meta-backend butterfly = +23.0%)
Qwen3-4B Q4_K_M:
pp64 = 984.51 t/s, tg16 = 49.29 t/s
Llama-3.3-70B in SYCL TP now comfortably beats production layer mode
on both prefill and decode. Coder-Next-80B-A3B (MoE) also wins on
both — the BF16 path is what unlocks the many-medium-allreduces
prefill pattern.
Build/CMake: no changes. No new dependencies. ~210 lines added across
ggml-sycl.h and ggml-sycl.cpp.
* Fix comments
* documentation update to address PR feedback
* Bring over my device-to-device memcpy chagnes
* move the dev2dev_memcpy calls to the upstream 7-parameter variety
* Fix a typo and remove a trailing whitespace
Piotr Wilkin
Piotr Wilkin
Adrien Gallouët
Xuan-Son Nguyen
nullname
Sanjay Ahari
Jeff Bolz
leonardHONG
Tarek DakhranandGeorgi Gerganov
PascalandXuan Son Nguyen
Jassieluo
shaofeiqi
Oliver Simons
Kashif Rasul
fairydreamingandStanisław Szymczyk
Christopher Albert
Abraham GonzalezandAbraham Gonzalez
David Spruill
Neo Zhang
lhez