* Revert "ci : disable ubuntu-rocm (#26969)"
This reverts commit 9558fa44c9.
* ci: set ccache compiler_check=content for ROCm build
The ROCm toolchain is pip-installed fresh on every run, so the clang binary's
mtime changes each time. With ccache's default compiler_check=mtime that
invalidates the whole cache and warm builds only reached ~70% hits. Hash the
compiler contents instead so the cache survives toolchain reinstalls.
* Update ccache size to 1GB
We're waivering with so many architectures built, we need a bigger
ccache limit.
* merge fix
---------
Co-authored-by: Jim Wu <[email protected]>
Co-authored-by: Sigbjørn Skjæret <[email protected]>
* ci : add older, min and dry-run options to ccache-clear
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* pi : add note about not wrapping lines in PR descriptions
[no ci]
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* opencl: fold the gpt-oss MoE bias adds into swiglu_oai
Default on, opt out with GGML_OPENCL_FUSE_MOE_BIAS_GLU=0.
* opencl: fold the MoE down-projection bias into the combine
Default on, opt out with GGML_OPENCL_FUSE_MOE_BIAS_COMBINE=0.
The 'Create release' step had no id, so steps.create_release.outputs.id
resolved to an empty string in the 'Upload nightly-tag.txt' step. The
uploadReleaseAsset call then hit /releases//assets and failed with HTTP
404 (Unhandled error: HttpError), e.g. run 32513839499.
Add id: create_release to the step; the action already exposes the id
output.
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* ci : run ccache-clear as the last step of release jobs
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* update disabled job too to force rebase
---------
Co-authored-by: Sigbjørn Skjæret <[email protected]>
Similar to ggml/scripts/release.sh: validates repo state, creates a
release candidate branch (llama-rc-vX.Y.Z), bumps LLAMA_VERSION_* in
CMakeLists.txt and commits the version bump.
Usage: ./scripts/release.sh [major|minor|patch] [--dry-run]
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* ui : rework the settings registry into ordered raw-data sections
SETTINGS_REGISTRY becomes an ordered SettingsSectionEntry[] array; the
array order is the sidebar display order. Section titles, color mode
options and title radio options are declared inline in their section or
entry. Entries gain showInUi; MCP servers, the system-message toggle and
the title LLM flag become hidden entries of their own section.
Derived values (config defaults, help info, chat sections, numeric field
lists, syncable parameters) are still derived here; they move to their
actual consumers in follow-up commits.
* ui : extract settings localStorage persistence into SettingsService
Stateless load/save of the settings config and user-override keys, plus the
legacy theme key migration. Business logic (default merging, mobile
sendOnEnter default, applying the migrated theme) stays in the store.
* ui : move the settings exit route into ROUTES
SETTINGS_FALLBACK_EXIT_ROUTE is just a route, so it lives with the other
routes as ROUTES.SETTINGS_EXIT.
* ui : derive the syncable parameter list in the parameter sync service
The syncable parameter mapping is only consumed by the sync service, so
derive it there from the registry instead of exporting it from the
constants file.
* ui : restore isPrivate for API key masking
* ui : clean up settings registry and router fetch guard
Drop the per-entry section field (duplicates the parent slug and is
never read) and guard the router model fetch on fields?.length so the
Tools/Import-Export pages with empty fields are excluded again.
Assisted-by: pi
* ui : merge sampling and penalties settings into one section
Assisted-by: pi
As agreed in ggml discussion #1579, the official semver releases now
include a nightly-tag.txt asset containing the tag of the corresponding
nightly release (e.g. b10485). The Web UI assets are published to the
HF bucket under the nightly tag, so this makes them discoverable for
each official release.
- make-release-desc.sh: expose the resolved nightly tag as a
nightly_tag output
- make-release.yml: create nightly-tag.txt from that tag, upload it
as a release asset (skipped on dry-run), mention it in the release
body and in the dry-run summary
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* Add DMMV Q4_K and Q6_K ESIMD kernels
Configure cmake build with -DGGML_SYCL_ESIMD=ON to enable.
Signed-off-by: Todd Malsbary <[email protected]>
* Refactor ESIMD kernels to share common code
Signed-off-by: Todd Malsbary <[email protected]>
* Move control of ESIMD from compile to runtime
Signed-off-by: Todd Malsbary <[email protected]>
* Use ESIMD by default when available
Signed-off-by: Todd Malsbary <[email protected]>
* Fix possible error when using ESIMD by default
While not an issue in the current version, this will become an
issue when additional QK ESIMD kernels are added (such as Q2_K).
Signed-off-by: Todd Malsbary <[email protected]>
* Add explicit unroll to ESIMD kernels
Signed-off-by: Todd Malsbary <[email protected]>
* Tidy up ESIMD kernels a bit
Signed-off-by: Todd Malsbary <[email protected]>
* Add a reordered Q2_K MMVQ kernel
Signed-off-by: Todd Malsbary <[email protected]>
* Add DMMV Q2_K ESIMD kernel
Signed-off-by: Todd Malsbary <[email protected]>
---------
Signed-off-by: Todd Malsbary <[email protected]>
Resolve the TODO in test_flash_attn_ext: the branch that creates V as a
sub-view of K (MLA-based models) was hardcoded for the 576/512 head shapes.
Add a v_is_view_of_k test case parameter (default false) and select the
sub-view branch on it; the existing 576/512 (DeepSeek MLA) cases now pass it
explicitly, so the test coverage is unchanged.
Also add more V-is-sub-view-of-K cases: the 320/256 (Mistral4 MLA) and
192/128 head shapes, and full views with equal head sizes (128/128 F16,
64/64 q8_0).
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* Add DMMV Q4_K and Q6_K ESIMD kernels
Configure cmake build with -DGGML_SYCL_ESIMD=ON to enable.
Signed-off-by: Todd Malsbary <[email protected]>
* Refactor ESIMD kernels to share common code
Signed-off-by: Todd Malsbary <[email protected]>
* Move control of ESIMD from compile to runtime
Signed-off-by: Todd Malsbary <[email protected]>
* Use ESIMD by default when available
Signed-off-by: Todd Malsbary <[email protected]>
* Fix possible error when using ESIMD by default
While not an issue in the current version, this will become an
issue when additional QK ESIMD kernels are added (such as Q2_K).
Signed-off-by: Todd Malsbary <[email protected]>
* Add explicit unroll to ESIMD kernels
Signed-off-by: Todd Malsbary <[email protected]>
* Tidy up ESIMD kernels a bit
Signed-off-by: Todd Malsbary <[email protected]>
* Add DMMV Q5_K ESIMD kernel
Signed-off-by: Todd Malsbary <[email protected]>
* Remove redundant copyright notice
Signed-off-by: Todd Malsbary <[email protected]>
---------
Signed-off-by: Todd Malsbary <[email protected]>
* opencl: keep the vocab-scale K-quant lm_head on the CPU on the Adreno A7X
* opencl: revise comments
---------
Co-authored-by: Li He <[email protected]>
The Tensor API mat-mat path of kernel_mul_mm (GGML_METAL_HAS_TENSOR) fed a
static K=32 tile to the matmul2d op on every iteration. On the last, partial
K tile (ne00 % 32 != 0) the src1 slice extends past the K extent of the
tensor, and the op reads those out-of-bounds elements (undefined behavior per
the MSL specification, section 2.22.2). Depending on stale memory contents,
this corrupted the result or produced NaN.
Make the matmul2d op use dynamic_extent for K, and clamp the K extent of both
operand tensor views to the remaining valid K range (min(32, K - loop_k)) per
iteration, so the op reads exactly the valid K range on every iteration
(mirroring the tail handling of the MPP matmul2d examples). On K-aligned
inputs the clamp degenerates to the full 32-wide tile: the only difference
from the static-K op is that the dynamic-K op derives K from the operand
extents and edge-checks the tile against the tensor extents (a handful of
integer ops per iteration).
Add test-backend-ops MUL_MAT cases with K not a multiple of 32 to exercise
the unaligned K path.
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* opencl: decline KV-convert flash_attn variants on Adreno A7X (compiler SIGSEGV)
The Adreno 740 (A7X) compiler E031.41 crashes inside clBuildProgram when
building the flash_attn programs whose KV path is mixed-type or dequantized:
flash_attn_f32_f16, flash_attn_f32_q8_0, flash_attn_f32_q4_0. It is a driver
crash rather than a compile-error return, so build_program_from_source_ex()
cannot catch it. The uniform f32 and f16 programs build correctly.
Decline the three KV-convert variants on the A7X in supports_op so they never
lazy-compile; those attention layers run on the CPU backend instead. Same
idiom as the existing Intel DK=512 and X1E carve-outs.
test-backend-ops FLASH_ATTN_EXT on the 740: 226 OK / 0 FAIL, previously exit
139. Other parts are unaffected - the gate is dead code there.
* opencl: fix q6_K flat mul_mat on older Adreno E031 compilers, gated
kernel_mul_mv_q6_K_f32_flat produces ~10x-wrong output on the older Adreno
E031 compilers while q4_K and q5_K are correct. Four codegen defects, each
confirmed on-device against the CPU reference:
1. 64-bit ulong arithmetic is miscompiled, so every weight and scale read
hit the wrong address - the primary cause, and why q5_K (int offsets)
was unaffected. The block index is computed in int and widened only
inside the pointer expression.
2. The vectorized dequant (int4/float4 bit-ops, convert_*4, dot()) is
miscompiled; the 6-bit weights are reconstructed and the dot done
scalar.
3. vload4 of the f32 activations is miscompiled; replaced by a
scalar-indexed load.
4. The accumulation is miscompiled unless a side effect forces the partial
sums to materialize. A printf under a guard the compiler cannot prove
false acts as a zero-cost optimizer barrier; its placement is
load-bearing.
The defect tracks the compiler, not the GPU generation: it reproduces on
E031.38 (Adreno 642L) and E031.41 (Adreno 740) and is fixed by E031.45
(Adreno 619), so the workarounds are gated on the compiler version. Where
they are not needed they cost real throughput - 42.4 -> 35.1 GFLOPS on an
Adreno 840 q6_K GEMV. The explicit compiler-type check is required, not
redundant: newer_than_or_same() is false for every non-E031 compiler, so
negating it alone would enable the workarounds on E17 and DX.
test-backend-ops MUL_MAT is 919/919 on the Adreno 740, 642L, 619, 840 and
850; the 740 and 642L were 909/919 before. The 642L additionally needs the
A6X per-kernel-program support to reach these tests at all.
* ui: Extract server stream lifecycle from chatStore into ChatStreamManager
Discovery, attach/replay, resume retry and the remote-running snapshot
formed a cohesive cluster inside chatStore. It now lives in
chat-streams.svelte.ts as ChatStreamManager, owned by chatStore, which
keeps the public entry points as delegates so components are
unchanged. chatStore: 2877 -> 2418 lines.
* ui: Extract user interaction gates from agenticStore into AgenticGates
Tool permission requests, turn-limit continue prompts and queued
steering messages are the state the loop waits on between turns. They
had no coupling to session state, so they now live in
agentic-gates.svelte.ts; agenticStore keeps delegates so components
are unchanged. agenticStore: 1196 -> 1073 lines.
* ui: Compose MCP resources under mcpStore.resources
Resource state was a second import scope next to mcpStore. Consumers
now go through mcpStore.resources, so the MCP surface is one store;
mcp-resources.svelte.ts stays a separate file owned by mcpStore.
* ui: Reorganize stores into domain namespaces
* fix: Update stale doc comments
* ui: Consolidate conv running-state into a chat activity ledger
Running-state was split across chatStore.chatLoadingStates (local
pipes), ChatStreamManager.remoteRunningConvs (backend sessions) and
attachingConvs (attach lifecycle), unioned by hand in
getAllLoadingChats and cross-cleaned by setChatLoading calling
streams.clearRemoteRunning - the 'spinner ghosts until tab toggle'
workaround.
chatActivityStore now owns both sets with one transition per event:
markLocal / localEnded (local pipe end also drops the stale remote
hint, no cross-owner call) / applyRemoteSnapshot (diffed). The
sidebar reads chatStore.activity.loadingConvs through the unchanged
getAllLoadingChats entry point.
Consequences:
- isStreamingActive and its five manual writers are gone; isStreaming()
now reports whether the active conversation has a live streaming
pipe, which is what all four consumers (assistant row, stop action,
context gauge, chat screen) actually check
- isLoading/isReasoning become derived from the per-conv maps plus
the active conversation, dropping the manual resync in
syncLoadingStateForChat and clearUIState
- attachingConvs and the last-attach coordination disappear from
ChatStreamManager
- getAllStreamingChats (no consumers) is removed
* ui: Give store collaborators narrow host interfaces
Collaborators took 'host: typeof <store>', i.e. the store's entire
public surface, which is how chatStore's streamChatCompletion,
createAssistantMessage, getApiOptions and setStreamingActive got
widened to public. Replace with per-collaborator interfaces carrying
only the members each one drives:
- ChatStreamHost (chat/streams) - activity, processing, streaming
states, abort controller, loading/streaming setters
- ChatFlowsHost (chat/flows) - streaming core, message creation,
per-conv state setters
- McpHealthHost (mcp/health) - connection registry + reconnection
- ModelPropsHost / ModelStatusHost (models) - model rows, feed
updates; the managers write modalities/status back onto the host's
rows, so those members stay writable
- ConversationsPreferencesHost (conversations) - the active row and
the conversation list
The store classes now declare 'implements <Host>' so the contract is
visible at the class level, and the 'import type { <store> }' back
references in the collaborators disappear entirely - the host
contract is local to each collaborator file, and collaborators can
no longer reach around their slice. Members stay public (structural
typing), but the collaborator side is now compiler-enforced.
* test: Chat Activity store test
* refactor: Cleanup
* chore: Remove legacy architecture docs
* ui: Memoize findMessageIndex for the streaming hot path
Streaming looks up the same message index on every chunk, a linear
scan of activeMessages each time. Cache the last lookup and reuse it
after validating the id still sits at the same position (O(1)); any
structural change to the array fails validation and falls back to a
full scan.
* ui: Throttle per-chunk stream state writes to localStorage
saveStreamState ran JSON.stringify + a synchronous localStorage.setItem
on every decoded chunk of the stream. The read loop now goes through a
new saveStreamStateThrottled (one write per conversation per 500ms,
latest value held pending); the public saveStreamState keeps its
immediate-write contract for stream start and pre-fetch, and also
resets the throttle window.
A pending offset is force-flushed at resume boundaries (resumeStream
reads the offset back from localStorage), on visibilitychange->hidden
and on pagehide, so a reload always finds a usable offset. The resume
offset only needs to be roughly current since the server retransmits
from a line boundary and the client discards its partial line.
Adds unit tests for the throttled/flush/clear interplay.
* ui: Compute context gauge timing stats in one pass
currentRead/Fresh/Cache/Output were separate deriveds, each running a
full reverse scan of activeMessages for the last assistant timings,
and cumulative ran its own forward scan plus an agentic filter - 4-5
O(n) passes per chunk while streaming. Replace with a single
summarizeAssistantTimings() pass (last assistant timings, last
agentic llm totals and the cumulative sums) feeding a shared derived
snapshot. Semantics unchanged, including the live-stats overrides and
the agentic llm-totals branch.
* agentic : clear session state when a conversation is deleted
Every conversation that ran an agentic flow left an AgenticSession in the
store forever; clearSession was never called. conversationsStore now
notifies deletion listeners and agenticStore drops the matching sessions,
avoiding a circular import back into conversationsStore.
* chat : extract ChatService.normalizeMessagesForApi
The DB->API message normalization (convert + drop empty system messages)
was duplicated in sendMessage, preEncode and the agentic flow. Extract it
into one shared method and call it from all three.
* sse : share record splitting and data extraction
splitSseRecords and extractSseDataPayload centralize the record-boundary
splitting and data: line extraction used by parseSseJsonStream and the
models status feed. chat.service keeps its own line-based parser for
resume support.
* api : delegate apiFetchWithParams to apiFetch
apiFetchWithParams duplicated apiFetch's headers/fetch/error handling
body-for-body; it only differs in URL construction. Build the URL and
delegate.
* chat flows : dedupe title, timings and cleanup handling
- conversationsStore.applyTitleFromContent centralizes the title-from-first-
message logic duplicated in 5 places
- ChatProcessingStore.applyStreamTimings centralizes the onTimings handler
shared by the chat and continue flows
- host.cleanupStreaming centralizes the loading/streaming/processing reset
repeated across the continue flow's exit paths
* conversations : centralize conversation update mirroring
rename, pin, mcp override, reasoning effort and cwd all repeated the same
write-DB-then-mirror-into-list-and-active dance. A single
applyConversationUpdate(id, updates) on the host collapses all five and
removes the forgot-to-mirror-one-field bug class. Drops the redundant
array reassignment in setCwd (deep field assignment is reactive).
* mcp : dedupe tool execution, server parsing and tool indexing
- executeTool delegates to executeToolByName (only diff was argument parsing)
- drop the private #parseServerSettings copy; use parseMcpServerSettings
- cache getServers() keyed on the raw config value (hot path)
- indexServerTools() unifies the three identical toolsIndex rebuild loops
Assisted-by: Claude
* mcp : share cursor pagination and tool indexing
- MCPService.paginate() collapses the identical do-while loops in
listAllResources and listAllResourceTemplates
- promoteHealthCheckToConnection now uses indexServerTools like the other
connect paths
Assisted-by: Claude
* database : share message parent-child bookkeeping
- addChildToParent() dedups the append-to-children update in createMessageBranch
and createSystemMessage
- removeChildFromParent() dedups the remove-from-children cleanup in deleteMessage
and deleteMessageCascading
- bulkAdd the cloned messages when forking a conversation instead of one add
per message
Assisted-by: Claude
* chore: Lint/format
* fix: `pagehide` event from `window`
* refactor: Api Fetch util
* docs : rewrite architecture sections in README
Update the high-level diagram, routes, hooks, stores, services and data
flow tables to match the current UI structure (mcp/settings/search
routes, agentic/tools/mcp stores, MCPService/ToolsService/SandboxService,
/tools API). Fix stale architectural patterns for per-conversation state
and modality validation.
* chore : add ESLint rule for blank lines between accessors
Enforce a blank line between consecutive class accessors. The core
padding-line-between-statements rule does not cover class members, so a
local rule is needed.
* refactor : reorder store members and unify naming
Order store class members as public fields, private fields, constructor,
getters, public methods, then private methods. Normalize private naming
to the `private` keyword (drop `#` and the `_` prefix where there is no
matching public getter). Rename conversationsStore.init() to
initialize() to match the other stores.
* refactor : prefix lookup methods with get in agentic and chat stores
Unify bare-name lookup methods with the get* prefix used across the
other stores (mcp, models, tools, settings). Renames currentTurn,
totalToolCalls, lastError, streamingToolCall, executingToolCallId,
pendingPermissionRequest, pendingContinueRequest,
pendingSteeringMessageContent, pendingSteeringMessageExtras in the
agentic store and pendingMessageContent, pendingMessageExtras in the
chat store. Updates the two consuming components and a doc comment.
* refactor: Clean up comments in stores' and services' code
* chore : add ESLint rule for class member ordering
Enforce structural order (public fields -> private fields -> constructor ->
getters -> setters -> public methods -> private methods) with alphabetical
sorting within each group via perfectionist/sort-classes. Dependency
detection keeps Svelte $derived fields in a valid dependency order instead
of alphabetizing them, since Svelte rejects forward references.
Assisted-by: Claude
* refactor : reorder class members to match new ESLint rule
Apply the sort-classes rule across stores, services, hooks and utils.
Pure reordering - verified no logic changes by comparing sorted line
multisets before/after. All tests and svelte-check pass.
* feat: add --mmproj-device arg & backwards compatible MTMD_BACKEND_DEVICE env var
* feat: load mmproj device backend immediately, add -mmdev shortflag
* fix: its a pointer now get the name
* clean up
* gen docs
* nits
---------
Co-authored-by: Xuan Son Nguyen <[email protected]>
* CI: Use LLVM's OpenMP over MSFT_DEBUG_non_redist on Windows
Currently, we ship the non-redist debug version of microsoft's libomp.
This PR changes this to official LLVM's release, also packaging
the license as needed.
* Remove LLVM SHA from job name to increase legibility
* Add temp validations to CI
* Revert "Add temp validations to CI"
This reverts commit eef97c88b5bac280803ebb3c3b7bb09f89b0fd88.
* Build OpenMP in CI
* Make OpenMP fetch self-contained in cmake and cache in CI
* Robustify Licens-packaging
1. Ship OpenMP license, not LLVM's.
2. Invalidate cache also on checksum of the license
* Remove stale reference in docs/build.md
* No longer package base license in release
This was scope-creep
* Add explanatory comment to OpenMP license
* Remove arm64 smoke
Forgot this during conflict resolution during rebase of
c54c0e9cf6030a5a54ce8bdd81b3e146d9787d42
* Remove GGML_OPENMP_FETCH_CACHE_DIR as requested by @CISC
* whitespace changes
* CUDA: runtime GGML_CUDA_MMVQ_MAX to tune the mvq->MMQ decode crossover
Add a runtime override of the mul_mat_vec_q -> MMQ batch crossover
(default MMVQ_MAX_BATCH_SIZE). Lowering it routes batches above the
threshold from the CUDA-core vector kernel to the int8 MMQ tensor-core
path, which is faster once quantized decode becomes compute-bound at
B>1 (measured +23-41% at B=8 on RTX 5090 for Q4_K dense, no low-batch loss).
The value is parsed once and clamped to [1, MMVQ_MAX_BATCH_SIZE], since
mul_mat_vec_q asserts ncols_dst <= that; invalid input warns and falls
back to the default. The override is applied consistently in both the
mul_mat_vec_q and MUL_MAT_ID dispatch paths. Default behavior unchanged.
* Added Blackwell specific switch point, to reduce dependence on runtime env var.
* Add per-HW switch point values for DGX Spark and removing runtime env var
* Adding switch points for Ada, tested on RTX 4090
* Modifying DGX Spark numbers based on latest run and adding some comments and small functional changes relating to MoE
* Reverting an unnecessary conditional
* Update ggml/src/ggml-cuda/mmvq.cu
---------
Co-authored-by: praneshgo <[email protected]>
Co-authored-by: Oliver Simons <[email protected]>
Tiwei Bie
Xuan-Son NguyenandSigbjørn Skjæret
Georgi Gerganov
Kartik SirohiandSigbjørn Skjæret
Shahir BIn ZulfikerandSigbjørn Skjæret
Hongqiang Wang
Niklas Wenzel
Aleksander Grygier
Charles Xu
Todd Malsbary
HumerousGorgonandNeo Zhang
Ian Faust
Chris Danis
vk
Eve
John-Henry LimandXuan Son Nguyen
Tarek Dakhran
Jeff Bolz
Oliver Simons
Aritro Bandyopadhyay
Aldehir Rojas