The device_info loop iterates over the discovered devices and gets
the available and total memory counts. With the CUDA backend (and
possibly others too) this requires creating a GPU context, which,
in case of CUDA, results in a 550 MB VRAM allocation.
For this information to be used in any way, the log verbosity must
be set to LOG_LEVEL_TRACE. If it's not, including in the default
configuration, the contexts get created, memory sizes get queried,
then the log function quietly discards the data.
In certain cases the user may not want to use any GPU resources.
The device_loop iteration is the only place touching the GPU that
cannot be skipped.
Fix by checking the verbosity level and skipping the loop if there
would be no output.
* DeepseekV4: fix rollback with multi-seq
* fix model loading
* make pending rollback single use
* only clear cache for seq_id for full load
* add assert for compress ratio
* make graph topology static
* pass true instead of flags in clear_compressed
* cont : clean-up + TODOs
---------
Co-authored-by: Georgi Gerganov <[email protected]>
* ui : add browser-style conversation tabs store
Track open conversation tabs in order, persisted to localStorage and
pruned against the loaded conversation list on init. The chat layout
syncs the route's tab on every navigation, so any way of reaching a
conversation opens a tab for it.
* ui : add temporary new-chat tabs
New-chat tabs are unsaved conversations carrying a temporary id used
directly as the route (#/chat/<id>). They live in memory and are only
persisted to the database - keeping the same id so the route and tab
stay stable - when the first message is sent. Deleting one drops it
without confirmation, and deleting conversations now closes their tabs.
* ui : render conversation tab bar in chat layout
Desktop-only tab bar above the chat screen, one tab per open
conversation or new-chat tab. The active tab follows the route id;
clicking navigates, middle-click or the close button closes (switching
to the left neighbor), and a trailing + starts a new chat. Tabs appear
only on chat-id routes; the bare #/ new-chat view has none. The bare
route stays put unless a prompt/model deep-link routes it to a new-chat
tab.
* ui : route new-chat entry points through tabs
The sidebar New chat item, Cmd+Shift+O, the search page and the
arrow-key fallback now open a new-chat tab instead of navigating to the
?new_chat URL, which is removed. New chat is no longer a special route
but a tab like any other conversation.
* ui : track sidebar expanded state in a shared ui store
Move the desktop sidebar expanded/collapsed state out of deviceStore into a
dedicated uiStore so the chat tab bar can react to it.
Assisted-by: pi
* chat : add opt-in conversation tabs setting
Add a Display setting that turns browser-style conversation tabs on or off,
enabled by default.
Assisted-by: pi
* chat : add browser-style conversation tabs with a new-chat screen
Track open conversations as tabs above the chat, one per open chat, plus a
single New chat tab for the bare `#/` route. New chat is just the `#/`
screen - no temporary conversations - and its tab is dropped when navigating
away. Sending the first message creates a real conversation and opens a tab
for it.
Assisted-by: pi
* chat : turn tab bar into a horizontally scrollable carousel
Make the tab bar a horizontally scrollable carousel with edge scroll buttons
and active-tab centering, and align its styling with the sidebar.
Assisted-by: pi
* chat : restyle the scroll-to-bottom button to match tab styling
Assisted-by: pi
* chat : add close-tab keyboard shortcut
Assisted-by: pi
* chat : soften tab bar fade and dim inactive tabs
Assisted-by: pi
* feat: Add stop button to tabs
* refactor: Componentize
* ui : fix carousel scrollability detection
Observe the content wrapper as well as the container, since adding overflowing items does not change the container's own box size. Also expose an onScrollableChange callback.
Assisted-by: pi
* ui : add unified ScrollCarousel component
Single carousel component with top/center variants, gap and scroll options, and hover-revealed chevrons. Rename the HorizontalScrollCarousel accessibility story accordingly.
Assisted-by: pi
* ui : migrate carousels to ScrollCarousel
Switch the settings mobile header, attachments list, thumbnail strip, and MCP resources to the unified component, and drop HorizontalScrollCarousel.
Assisted-by: pi
* ui : improve chat tabs carousel UX
Scroll newly added tabs into view, fade overflowing tabs at the edges, and hide the New chat button while a new-chat tab is open.
Assisted-by: pi
* refactor: Naming
* chat : add keyboard shortcut to jump between conversation tabs
Shift+Cmd/Ctrl+Left/Right cycles the open tabs, mirroring the existing
Shift+Cmd/Ctrl+Up/Down conversation navigation.
Assisted-by: pi
* chat : make the whole tab item act as a link
The full tab is now a link instead of only the inner label button, while
the stop and close buttons stay interactive by swallowing their clicks.
Assisted-by: pi
* chat : adjust tab bar width and use a shared offset variable
Widen the tab bar for the expanded sidebar and rename the tab bar height
variable to --chat-tabs-offset with a smaller value so the chat screen
min-height accounts for the overlay without overshooting.
Assisted-by: pi
* chat : account for the tab bar offset in the assistant min-height
Subtract the tab bar offset when it is shown so the last assistant message
does not overflow the available viewport space.
Assisted-by: pi
* refactor: Post-review fixes
* ui : restore deep links on the chat start page
- handle ?model selection, with ?load=true eager router loading
- ?q now creates a conversation, sends the prompt, and clears the params
- show the not-available-model dialog for unknown models
- never block mount on the conversation list
Assisted-by: pi
* ui : fix tab item link nesting and centralize tab constants
- the tab anchor covers the whole item while stop/close stay siblings,
so interactive elements are never nested inside the anchor
- cmd/ctrl/middle clicks are left to the browser (new window)
- extract the tab labels, the active-tab data attribute, and the
sidebar-offset max widths into constants
Assisted-by: pi
* ui : tidy scroll carousel hook and keep mobile header arrows on
- drop the dead scrollLeft/scrollRight helpers and the unused
onScrollableChange/scrollBy props
- init the carousel once instead of inside a derived
- restore items-start on the center variant
- always show the settings header arrows on touch
Assisted-by: pi
* ui : keep the new-chat tab across reloads and fall back on close
- the new-chat sentinel is no longer pruned on init, so reloading on
the bare new-chat route keeps the tab the user is on
- closing the active conversation falls back to the new-chat screen
when Conversation tabs are off
Assisted-by: pi
* ui : don't block startup on the conversation list
- prune persisted tabs after the list loads in the background instead
of awaiting it during init
- openNewChat now returns void; its return value was never read
Assisted-by: pi
* ui: fix routing nits
* chore: Update doc comments
* refactor: Mark fire-and-forget openNewChat calls as `void`
* chat: fix the deep-linked prompt, the tab width and the tab shortcuts
The chat start page creates the conversation and hands the prompt over
to the chat route, which still sees it in the query string. Sending it
on both sides queues the second copy as a pending message, which shows
up as a stray user bubble once the answer lands and vanishes on reload
since it never reaches the database.
The tab bar takes the max width of the collapsed sidebar while it is
expanded, and the other way round.
The tab list is pruned against a snapshot of the loaded conversations,
so a conversation created while that list is still loading loses its
tab even though the route just opened it. The active tab then falls out
of the list and the cycling shortcut jumps to an edge on every keypress
instead of moving one tab over. Tabs synced from the route are kept as
they are, only the persisted ones are pruned.
The rich chat input claims ctrl or alt with shift and an arrow for its
badge-aware word jump, which now belongs to the tab cycling shortcut.
Holding shift hands the key combination over, the plain word jump is
unchanged.
The close-tab shortcut consumes the event before checking whether the
setting is on, and the logo background loses its importance flag.
---------
Co-authored-by: Pascal <[email protected]>
* fit: also take into account n_streams
* server: make the draft context follow the target context
With a non-unified KV cache the target context now holds n_ctx_train
tokens per sequence, while the draft context was still created with
n_ctx = 0 and fell back to n_ctx_train / n_streams per sequence. A slot
filled beyond that point makes the draft batch fail to decode, and the
server answers 500 on the request.
The draft context now takes its size from the target context, so both
hold the same number of tokens per sequence. Contexts that share their
cells with the target no longer need the kv_size override.
The memory reserved for the draft model before fitting is measured at
the largest context the target can take, since the draft context grows
with the target and a fixed byte margin cannot express that.
* fit: take an optional second model into account
Illustrates the alternative discussed on the draft context fix. The
memory of a draft or MTP context is currently handed to the fit as a
fixed byte margin, which cannot express a memory that grows with the
context the fit is still deciding on.
common_fit_params now takes an optional second model that shares the
devices of the main one. Its context follows the main context and its
memory is measured again whenever that context changes, so the reduce
path stays exact instead of conservative. A model that cannot be
measured on its own, such as a shared cell MTP context, is skipped with
a warning and the main model is fitted alone.
This drops the reservation block in the server, which no longer has to
probe the trained context size of the target to guess an upper bound.
---------
Co-authored-by: Pascal <[email protected]>
* Revert "ci : disable ubuntu-rocm (#26969)"
This reverts commit 9558fa44c9.
* ci: set ccache compiler_check=content for ROCm build
The ROCm toolchain is pip-installed fresh on every run, so the clang binary's
mtime changes each time. With ccache's default compiler_check=mtime that
invalidates the whole cache and warm builds only reached ~70% hits. Hash the
compiler contents instead so the cache survives toolchain reinstalls.
* Update ccache size to 1GB
We're waivering with so many architectures built, we need a bigger
ccache limit.
* merge fix
---------
Co-authored-by: Jim Wu <[email protected]>
Co-authored-by: Sigbjørn Skjæret <[email protected]>
* ci : add older, min and dry-run options to ccache-clear
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* pi : add note about not wrapping lines in PR descriptions
[no ci]
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* opencl: fold the gpt-oss MoE bias adds into swiglu_oai
Default on, opt out with GGML_OPENCL_FUSE_MOE_BIAS_GLU=0.
* opencl: fold the MoE down-projection bias into the combine
Default on, opt out with GGML_OPENCL_FUSE_MOE_BIAS_COMBINE=0.
The 'Create release' step had no id, so steps.create_release.outputs.id
resolved to an empty string in the 'Upload nightly-tag.txt' step. The
uploadReleaseAsset call then hit /releases//assets and failed with HTTP
404 (Unhandled error: HttpError), e.g. run 32513839499.
Add id: create_release to the step; the action already exposes the id
output.
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* ci : run ccache-clear as the last step of release jobs
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* update disabled job too to force rebase
---------
Co-authored-by: Sigbjørn Skjæret <[email protected]>
Similar to ggml/scripts/release.sh: validates repo state, creates a
release candidate branch (llama-rc-vX.Y.Z), bumps LLAMA_VERSION_* in
CMakeLists.txt and commits the version bump.
Usage: ./scripts/release.sh [major|minor|patch] [--dry-run]
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* ui : rework the settings registry into ordered raw-data sections
SETTINGS_REGISTRY becomes an ordered SettingsSectionEntry[] array; the
array order is the sidebar display order. Section titles, color mode
options and title radio options are declared inline in their section or
entry. Entries gain showInUi; MCP servers, the system-message toggle and
the title LLM flag become hidden entries of their own section.
Derived values (config defaults, help info, chat sections, numeric field
lists, syncable parameters) are still derived here; they move to their
actual consumers in follow-up commits.
* ui : extract settings localStorage persistence into SettingsService
Stateless load/save of the settings config and user-override keys, plus the
legacy theme key migration. Business logic (default merging, mobile
sendOnEnter default, applying the migrated theme) stays in the store.
* ui : move the settings exit route into ROUTES
SETTINGS_FALLBACK_EXIT_ROUTE is just a route, so it lives with the other
routes as ROUTES.SETTINGS_EXIT.
* ui : derive the syncable parameter list in the parameter sync service
The syncable parameter mapping is only consumed by the sync service, so
derive it there from the registry instead of exporting it from the
constants file.
* ui : restore isPrivate for API key masking
* ui : clean up settings registry and router fetch guard
Drop the per-entry section field (duplicates the parent slug and is
never read) and guard the router model fetch on fields?.length so the
Tools/Import-Export pages with empty fields are excluded again.
Assisted-by: pi
* ui : merge sampling and penalties settings into one section
Assisted-by: pi
As agreed in ggml discussion #1579, the official semver releases now
include a nightly-tag.txt asset containing the tag of the corresponding
nightly release (e.g. b10485). The Web UI assets are published to the
HF bucket under the nightly tag, so this makes them discoverable for
each official release.
- make-release-desc.sh: expose the resolved nightly tag as a
nightly_tag output
- make-release.yml: create nightly-tag.txt from that tag, upload it
as a release asset (skipped on dry-run), mention it in the release
body and in the dry-run summary
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* Add DMMV Q4_K and Q6_K ESIMD kernels
Configure cmake build with -DGGML_SYCL_ESIMD=ON to enable.
Signed-off-by: Todd Malsbary <[email protected]>
* Refactor ESIMD kernels to share common code
Signed-off-by: Todd Malsbary <[email protected]>
* Move control of ESIMD from compile to runtime
Signed-off-by: Todd Malsbary <[email protected]>
* Use ESIMD by default when available
Signed-off-by: Todd Malsbary <[email protected]>
* Fix possible error when using ESIMD by default
While not an issue in the current version, this will become an
issue when additional QK ESIMD kernels are added (such as Q2_K).
Signed-off-by: Todd Malsbary <[email protected]>
* Add explicit unroll to ESIMD kernels
Signed-off-by: Todd Malsbary <[email protected]>
* Tidy up ESIMD kernels a bit
Signed-off-by: Todd Malsbary <[email protected]>
* Add a reordered Q2_K MMVQ kernel
Signed-off-by: Todd Malsbary <[email protected]>
* Add DMMV Q2_K ESIMD kernel
Signed-off-by: Todd Malsbary <[email protected]>
---------
Signed-off-by: Todd Malsbary <[email protected]>
Resolve the TODO in test_flash_attn_ext: the branch that creates V as a
sub-view of K (MLA-based models) was hardcoded for the 576/512 head shapes.
Add a v_is_view_of_k test case parameter (default false) and select the
sub-view branch on it; the existing 576/512 (DeepSeek MLA) cases now pass it
explicitly, so the test coverage is unchanged.
Also add more V-is-sub-view-of-K cases: the 320/256 (Mistral4 MLA) and
192/128 head shapes, and full views with equal head sizes (128/128 F16,
64/64 q8_0).
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* Add DMMV Q4_K and Q6_K ESIMD kernels
Configure cmake build with -DGGML_SYCL_ESIMD=ON to enable.
Signed-off-by: Todd Malsbary <[email protected]>
* Refactor ESIMD kernels to share common code
Signed-off-by: Todd Malsbary <[email protected]>
* Move control of ESIMD from compile to runtime
Signed-off-by: Todd Malsbary <[email protected]>
* Use ESIMD by default when available
Signed-off-by: Todd Malsbary <[email protected]>
* Fix possible error when using ESIMD by default
While not an issue in the current version, this will become an
issue when additional QK ESIMD kernels are added (such as Q2_K).
Signed-off-by: Todd Malsbary <[email protected]>
* Add explicit unroll to ESIMD kernels
Signed-off-by: Todd Malsbary <[email protected]>
* Tidy up ESIMD kernels a bit
Signed-off-by: Todd Malsbary <[email protected]>
* Add DMMV Q5_K ESIMD kernel
Signed-off-by: Todd Malsbary <[email protected]>
* Remove redundant copyright notice
Signed-off-by: Todd Malsbary <[email protected]>
---------
Signed-off-by: Todd Malsbary <[email protected]>
* opencl: keep the vocab-scale K-quant lm_head on the CPU on the Adreno A7X
* opencl: revise comments
---------
Co-authored-by: Li He <[email protected]>
The Tensor API mat-mat path of kernel_mul_mm (GGML_METAL_HAS_TENSOR) fed a
static K=32 tile to the matmul2d op on every iteration. On the last, partial
K tile (ne00 % 32 != 0) the src1 slice extends past the K extent of the
tensor, and the op reads those out-of-bounds elements (undefined behavior per
the MSL specification, section 2.22.2). Depending on stale memory contents,
this corrupted the result or produced NaN.
Make the matmul2d op use dynamic_extent for K, and clamp the K extent of both
operand tensor views to the remaining valid K range (min(32, K - loop_k)) per
iteration, so the op reads exactly the valid K range on every iteration
(mirroring the tail handling of the MPP matmul2d examples). On K-aligned
inputs the clamp degenerates to the full 32-wide tile: the only difference
from the static-K op is that the dynamic-K op derives K from the operand
extents and edge-checks the tile against the tensor extents (a handful of
integer ops per iteration).
Add test-backend-ops MUL_MAT cases with K not a multiple of 32 to exercise
the unaligned K path.
Assisted-by: pi:llama.cpp/Qwen3.8-27B
Bartosz Taudul
Aman GuptaandGeorgi Gerganov
Gaurav Garg
Aleksander GrygierandPascal
Alessandro de Oliveira Faria (A.K.A.CABELO)
Aman Karki
Xuan-Son Nguyen
Safi UllahandJeff Bolz
Tiwei Bie
Kartik SirohiandSigbjørn Skjæret
Shahir BIn ZulfikerandSigbjørn Skjæret
Hongqiang Wang
Niklas Wenzel
Charles Xu
Todd Malsbary
HumerousGorgonandNeo Zhang
Ian Faust
Chris Danis
vk
Eve