* model: add NextN/MTP speculative decoding support for GLM_DSA (GLM-5.2)
Adds GLM-5.2 NextN/MTP as a --spec-type draft-mtp target: nextn tensor
loading via the qwen35moe/step35-style presence probe, a graph_mtp
builder (enorm/hnorm/eh_proj + dense MLA + sigmoid-gated MoE with
shared expert + shared head with fallbacks, _s scale tensors passed
for NVFP4), t_h_nextn extraction in the trunk graph, and MTP-context
KV setup: the draft head runs dense MLA, so the MTP context uses a
plain attention KV cache holding only the nextn layer(s) (same
pattern as the hybrid Qwen3.5 MTP context) while the main context
keeps the DSA cache, now filtered to trunk layers only.
Co-Authored-By: Claude Fable 5 <[email protected]>
* convert : support --mtp/--no-mtp export for GlmMoeDsaForCausalLM (GLM-5.2)
Opt GLM-5.2 into the supports_mtp_export contract (post-#25641 shape,
mirroring HYV3Model/Step35Model): --no-mtp drops the appended NextN
block (blk.78) and its nextn_predict_layers KV; --mtp keeps only the
NextN block plus shared embeddings/norm/lm_head. Default (bundled)
output is unchanged.
Co-Authored-By: Claude Fable 5 <[email protected]>
---------
Co-authored-by: Claude Fable 5 <[email protected]>
* mtmd : add Nemotron 3 Nano Omni support (parakeet)
This commit adds support for the subsampling and encoder part of
Nemotron Nemo 3 omni model.
The Parakeet subsampling/encoder were taken from parakeet.cpp which
is currently a pull request against whisper.cpp. I've tried to copy the
code a close as possible to hopefully enable easy patching between the
these two project later.
Refs: https://github.com/ggml-org/whisper.cpp/pull/3735
* mtmd : generate rel pos tensor in graph instead of in conversion [no ci]
This commit removes the generation of the relative positional tensor in
the model conversion script and instead computes it in the encoder
graph. This is only done for the window of positions required for the
current audio sample.
* mtmd : add clip_get_model to clip API [no ci]
This commit adds a function to get access to the clip_model. It also
removes the two functions clip_get_mel_filter_tensor, and
clip_get_window_tensor(const struct clip_ctx * ctx) which can now use
clip_get_model to access the model tensors that it needs.
* mtmd : read mel_filters and window into hparams
* mtmd : use set_input_f32 lambda [no ci]
* mtmd : add better asserts for mel_filters and hann window [no ci]
* mtmd : add missing size_t cast
* mtmd : change type of pad to size_t
* mtmd : zero initialize samples_padded
* mtmd : remove unsued ctx member from parakeet preprocessor
* mtmd : make log_mel_spectrogram_parakeet_worker_thread private static
* mtmd : sync/update parakeeet impl with latest whisper.cpp
This commit updates the parakeet code in mtmd to reflect the latest
updates to parakeet.cpp in whisper.cpp.
A follow up commit will address the currently hardcoded dw_pad and see
if we can add n_conv_kernel as a model metadata field.
* mtmd : add audio_conv_kernel_size to model conversion
This commit updates the model conversion to read the conv_kernel_size
field from the sound_config section of the models config.json file.
It then uses this field instead of the hardcoded values in parakeet.cpp.
* mtmd : cleanup [no ci]
* conversion : call super().filter_tensors [no ci]
* do not discard result of super filter_tensors
* mtmd : use build_mm instead of ggml_mul_mat
* mtmd : use build_ffn
* mtmd : move and reuse get_vector lambda
* mtmd : use build_inp_raw for parakeet
* mtmd : throw exception in get_scalar instead of assert
* mtmd : fix std::min call
* mtmt : use .c_str in throw clause in get_vector
* mtmd : check for F32 type and non-empty tensor in get_vector
The get_vector lambda is used by get_scalar but also standalone to read
in the mel_filters and the window data. Therefor we are not checking
for 1D tensors but allowing multiple dimensions. We do have a check in
get_scalar to verify the size of the vector.
* mtmd : replace hardcoded 1101 for n_tokens_real
* mtmd : assert subsampling_factor is 8
This commit adds an assert of the parakeet subsampling factor to check
that it is 8.
The motivation for this is that this model currently has three
convolutions with a stride of 2. If the underlying model updates the
subsampling factor these convolution operations will need to be updated
and this will produce and error if this occurs.
* mtmd : remove unused ggml_tensors attn_pos_w and mm_norm_w
* mtmd : remove single thread path
This commit removes the single thread path which was a left over from
the original parakeet.cpp where n_threads is configurable.
* fix some security issues
---------
Co-authored-by: Sigbjørn Skjæret <[email protected]>
Co-authored-by: Xuan Son Nguyen <[email protected]>
* spec: add DSpark speculative decoding
DSpark (DeepSpec, 2026) on top of the merged DFlash drafter. It reuses the
DFlash encoder/decoder graph, target feature extraction and KV-cache injection,
and the verify/accept path unchanged; the draft model is a new "dspark" arch
adding a low-rank Markov head (markov_w1/w2) and an optional (unused here)
confidence head. No new public APIs.
The proposal is the only change: the block is anchor-first (position 0 already
predicts the first draft) and the decoder graph applies a semi-autoregressive,
previous-token conditioned logit bias in-graph, chained per block position:
logits'(i) = logits(i) + markov_w2 . markov_w1[prev(i)]
prev(0) = the block's anchor token, prev(i>0) = argmax(logits'(i-1))
vectorized across all blocks in the batch; the anchors are fed through a
dedicated graph input (token 0 of every block). Greedy stays lossless
(verify unchanged, same as DFlash).
- new arch "dspark" (llama_model_dspark : llama_model_dflash, reuses the graph,
loads the markov/confidence tensors; shares the target's embed/lm_head).
- Qwen3DSparkModel converter.
- new spec type "draft-dspark" (common_speculative_impl_draft_dspark :
common_speculative_impl_draft_dflash, overrides draft() only: submits whole
anchor-first blocks and greedily reads back the biased logits).
* spec: read draft block size in the dflash impl
* docs: add DSpark section to speculative.md
* spec: keep dspark block size read in the dspark impl
* dspark : add TODOs for incomplete parts
- confidence head is loaded but not used yet
- confidence-scheduled prefix pruning is not implemented
- the in-graph Markov chain is greedy-only
- only Qwen3 backbones are supported for now (also noted in docs)
* spec: fold DSpark into the DFlash arch
Address review: drop LLM_ARCH_DSPARK and the dspark.block_size /
markov_rank GGUF keys. A DSpark draft now converts to a DFlash GGUF;
the Markov head tensors are detected by presence (like eagle3 d2t),
block_size is read from the existing dflash.block_size key, and the
block anchors are taken as a strided view of the decoder's token
input instead of a separate graph input.
* spec: add confidence-based draft pruning for DSpark
The DSpark confidence head predicts per-position acceptance of the
drafted block. --spec-draft-conf-min truncates the block at the first
position below the threshold (default 0 = disabled).
* fold the dspark impl into dflash, selected by spec type
* address review comments
* dspark: clean up and improve naming
* update readme
* remove trailing whitespace
* dflash: draft full n_max blocks, defer dp.n_max to the central truncation
The DSpark markov head views the draft batch as a uniform [n_seqs x block]
grid, but the per-seq dp.n_max clamp could produce blocks of different
sizes, silently corrupting the strided views and the resulting logits.
Drop the clamp and always draft the full n_max block for every sequence:
dp.n_max is already enforced by the central truncation in
common_speculative_draft(), the same way eagle3 handles it.
Co-authored-by: Zaire404 <[email protected]>
* dflash: assert the markov head block-uniformity invariant, require the conf head
With the draft batch always submitting equal-size n_max blocks, a
non-divisible token count can only mean the batch was split across
ubatches or a caller broke the layout - fail loudly instead of silently
dropping the markov bias. The block_drafts > block_size early return
stays: worst-case graph reserve passes legitimately build with
n_seq_tokens > block_size.
Also make conf_proj required when the markov head is present: the
confidence head is part of the DSpark checkpoint format, and a missing
head would otherwise leave --spec-draft-conf-min silently reading stale
embeddings instead of confidences.
Co-authored-by: Zaire404 <[email protected]>
* dspark: fold conf_min into p_min
p_min and conf_min express the same thing - the minimum predicted
survival probability for a drafted position - differing only in how the
estimate is obtained: token probability for regular drafters, the
trained confidence head for DSpark. The DSpark readback never used
p_min, so reuse it for the confidence threshold and drop the separate
--spec-draft-conf-min flag. Both defaulted to 0 (disabled), so behavior
is unchanged.
Co-authored-by: Zaire404 <[email protected]>
* dflash: note the confidence broadcast workaround
Requested in review: the ggml_repeat only adapts the [1, n_tok]
confidences to the n_embd-wide embd_nextn transport so that
llama_get_embeddings_nextn can be reused - not a placeholder.
Co-authored-by: Zaire404 <[email protected]>
* cont : clarify
[no ci]
---------
Co-authored-by: Ruixiang Wang <[email protected]>
Co-authored-by: Zaire404 <[email protected]>
Co-authored-by: Georgi Gerganov <[email protected]>
* Add preliminary MiniMax-M3 support
Text-only port that re-uses existing components: MiniMax-M2 style GQA with
per-head QK-norm and partial rotary, DeepSeek-V3 style leading-dense and
routed/shared experts, and swigluoai activation. Sparse attention is not
yet supported (dense fallback); vision tower and MTP heads are dropped.
* MiniMax-M3 vision tower (mmproj + clip graph)
* Delete m3_vision_ref.py
* Update clip.cpp
* MSA
* Update constants.py
* Update minimax.py
* Cache creation. Working withotu flash attention
* Added flash attention for sparse layers
* Decomposed slow cpu OP into GPU + CPU ops. Massive speedup over long ctx
* Rewrote indexer op to be cuda native. Modified flash attention to match per group block picking
* Implement sparse attention calc out of stock ops.
* Fix a cache allocation and cont issue
* Fixed -fa auto crash, flagged debug spots
* Delete vocab.json
* Delete model.safetensors.index.json
* Delete generation_config.json
* Delete Minimax directory
* Handled multi stream case to fall back on Dense Attention
* Development scaffolding cleanup. No functional change to the decode or
4-way paths. Full debug harness remains at <8136a9c68ed7a5eb009aa67bba3fda8062f4648f> for reproducing the
selection-parity validation.
* Remove redundant comment from minimax-m3.cpp
* Changed 3 Gelu Ops for vision into Gelu_erf ops
* Assert that n_kv is multiple of 128
* Rename MSA index tensors to indexer convention
Note: All GGUFs generated before this change will need to be regenerated.
* Fix incorrect Assert
* Review driven changes (#3)
* Remove comment from conversion minimax.py
Co-authored-by: Sigbjørn Skjæret <[email protected]>
* Remove whitespaces from constants.py
Co-authored-by: Sigbjørn Skjæret <[email protected]>
* Tighten comment in minimax.py
Co-authored-by: Sigbjørn Skjæret <[email protected]>
* inherit MiniMax-M3 from MiniMax-M2
* drop dead text_config fallbacks
* Add indexer writer methods
* Reuse LLM_FFN_SWIGLU_OAI_MOE
* Remove duplicate indexer setters, add only block_size/local_blocks, follow value naming convention
* Fix conversion error /gguf_writer.py
Co-authored-by: Sigbjørn Skjæret <[email protected]>
* Update gguf-py/gguf/gguf_writer.py
Co-authored-by: Sigbjørn Skjæret <[email protected]>
* Update gguf-py/gguf/tensor_mapping.py
Co-authored-by: Sigbjørn Skjæret <[email protected]>
* Update conversion/minimax.py
Co-authored-by: Sigbjørn Skjæret <[email protected]>
* Update conversion/minimax.py
Co-authored-by: Sigbjørn Skjæret <[email protected]>
* Remove whitespace in src/llama-kv-cache.cpp
Co-authored-by: Sigbjørn Skjæret <[email protected]>
* Remove Whitespace in Update src/llama-model.h
Co-authored-by: Sigbjørn Skjæret <[email protected]>
* Remove whitespace in src/llama-hparams.h
Co-authored-by: Sigbjørn Skjæret <[email protected]>
* Update minimax_m3.cpp
Rewrite code comment based on feedback and to better reflect the actual architecture, and reuse existing build_vit
* Rename minimax_m3.cpp to minimax-m3.cpp
* Update CMakeLists.txt
* Remove debug code from clip.cpp
* Update clip.cpp
* Update comments in tools/mtmd/models/minimax-m3.cpp
* Permute Q/K at conversion, drop precomputed sin/cos
* Log cache size on launch, block ctx shift, support prompt caching
Log indexer cache size on launch
Disallow ctx shift
Support prompt caching
* Update minimax-m3.cpp
* Optimize implementation, add multi stream support.
Fully rewrote minimax-m3.cpp for speed and buffer size gains:
Unified the 4-way + decode, 1 FA call per layer instead of 4, with the groups mapped onto ne[3]
Custom CPU op now emits block-level mask, expanded on GPU, which causes CPU to GPU transfer to shrinks at prefill
Decode: ~25 nodes/layer vs ~50, no per-group concats/conts
Unified selection semantics, so both regimes rank bs + local bias (position-anchored local force), which means prefill/decode can no longer disagree on selection
can_reuse on the MSA bias input. Graph reuse at decode restored (was rebuilding the full graph every token)
In-place mask adds, shrinking compute buffer ~6.8 to ~4.2 GiB at ub2048/62k
Multi-stream: MSA now runs with -np N when kv_unified=false. Decode stays batched across streams (still 1 FA call), prefill loops per stream. dense fallback only for --kv-unified + multi-seq
Measured effect on expert offload bound setup: decode 6.2(4WAY)–7.15(MSA_decode) -> 7.7~7.8 t/s, flat from 5k to 60k+. prefill around 10% faster. buffer about 20% smaller, multi-user support.
* set default cache type to F32
* Fix potential DSA double indexer cache allocation bug, only allocate in-cache k_idx for archs that opt in
* remove F16 downcasts in MSA attention, force F32 indexer score accum
* Add Minimax eos to llama vocab
* Guard edge case where idx cache can become stale after a tail trim
* Update llama-kv-cache.h
* Update llama-kv-cache.cpp
* Update llama-kv-cache.cpp
* Update llama-kv-cache.h
* Change resize Pad to none, resize alg to Bicubic Pillow
* Review driven changes
* Update llama-kv-cache.cpp
* rm unrotated pos_t
* fused rope w + pad
* rename merge --> merger for consistency
* add review skill for mtmd
* graph should use hparams n_merge
* fix lint
---------
Co-authored-by: Daniel Han <[email protected]>
Co-authored-by: Sigbjørn Skjæret <[email protected]>
Co-authored-by: Xuan Son Nguyen <[email protected]>
* Add preliminary MiniMax-M3 support
Text-only port that re-uses existing components: MiniMax-M2 style GQA with
per-head QK-norm and partial rotary, DeepSeek-V3 style leading-dense and
routed/shared experts, and swigluoai activation. Sparse attention is not
yet supported (dense fallback); vision tower and MTP heads are dropped.
* MiniMax-M3 vision tower (mmproj + clip graph)
* Delete m3_vision_ref.py
* Update clip.cpp
* MSA
* Update constants.py
* Update minimax.py
* Cache creation. Working withotu flash attention
* Added flash attention for sparse layers
* Decomposed slow cpu OP into GPU + CPU ops. Massive speedup over long ctx
* Rewrote indexer op to be cuda native. Modified flash attention to match per group block picking
* Implement sparse attention calc out of stock ops.
* Fix a cache allocation and cont issue
* Fixed -fa auto crash, flagged debug spots
* Delete vocab.json
* Delete model.safetensors.index.json
* Delete generation_config.json
* Delete Minimax directory
* Handled multi stream case to fall back on Dense Attention
* Development scaffolding cleanup. No functional change to the decode or
4-way paths. Full debug harness remains at <8136a9c68ed7a5eb009aa67bba3fda8062f4648f> for reproducing the
selection-parity validation.
* Remove redundant comment from minimax-m3.cpp
* Changed 3 Gelu Ops for vision into Gelu_erf ops
* Assert that n_kv is multiple of 128
* Rename MSA index tensors to indexer convention
Note: All GGUFs generated before this change will need to be regenerated.
* Fix incorrect Assert
* Review driven changes (#3)
* Remove comment from conversion minimax.py
Co-authored-by: Sigbjørn Skjæret <[email protected]>
* Remove whitespaces from constants.py
Co-authored-by: Sigbjørn Skjæret <[email protected]>
* Tighten comment in minimax.py
Co-authored-by: Sigbjørn Skjæret <[email protected]>
* inherit MiniMax-M3 from MiniMax-M2
* drop dead text_config fallbacks
* Add indexer writer methods
* Reuse LLM_FFN_SWIGLU_OAI_MOE
* Remove duplicate indexer setters, add only block_size/local_blocks, follow value naming convention
* Fix conversion error /gguf_writer.py
Co-authored-by: Sigbjørn Skjæret <[email protected]>
* Update gguf-py/gguf/gguf_writer.py
Co-authored-by: Sigbjørn Skjæret <[email protected]>
* Update gguf-py/gguf/tensor_mapping.py
Co-authored-by: Sigbjørn Skjæret <[email protected]>
* Update conversion/minimax.py
Co-authored-by: Sigbjørn Skjæret <[email protected]>
* Update conversion/minimax.py
Co-authored-by: Sigbjørn Skjæret <[email protected]>
* Remove whitespace in src/llama-kv-cache.cpp
Co-authored-by: Sigbjørn Skjæret <[email protected]>
* Remove Whitespace in Update src/llama-model.h
Co-authored-by: Sigbjørn Skjæret <[email protected]>
* Remove whitespace in src/llama-hparams.h
Co-authored-by: Sigbjørn Skjæret <[email protected]>
* remove multimodal code upon maintainer request. Will be made as a separate PR
* Whitespace clean in tensor_mapping.py
* Log cache size on launch, block ctx shift, support prompt caching
Log indexer cache size on launch
Disallow ctx shift
Support prompt caching
* Update minimax-m3.cpp
* Optimize implementation, add multi stream support.
Fully rewrote minimax-m3.cpp for speed and buffer size gains:
Unified the 4-way + decode, 1 FA call per layer instead of 4, with the groups mapped onto ne[3]
Custom CPU op now emits block-level mask, expanded on GPU, which causes CPU to GPU transfer to shrinks at prefill
Decode: ~25 nodes/layer vs ~50, no per-group concats/conts
Unified selection semantics, so both regimes rank bs + local bias (position-anchored local force), which means prefill/decode can no longer disagree on selection
can_reuse on the MSA bias input. Graph reuse at decode restored (was rebuilding the full graph every token)
In-place mask adds, shrinking compute buffer ~6.8 to ~4.2 GiB at ub2048/62k
Multi-stream: MSA now runs with -np N when kv_unified=false. Decode stays batched across streams (still 1 FA call), prefill loops per stream. dense fallback only for --kv-unified + multi-seq
Measured effect on expert offload bound setup: decode 6.2(4WAY)–7.15(MSA_decode) -> 7.7~7.8 t/s, flat from 5k to 60k+. prefill around 10% faster. buffer about 20% smaller, multi-user support.
* set default cache type to F32
* Fix potential DSA double indexer cache allocation bug, only allocate in-cache k_idx for archs that opt in
* remove F16 downcasts in MSA attention, force F32 indexer score accum
* Add Minimax eos to llama vocab
* Guard edge case where idx cache can become stale after a tail trim
* Update llama-kv-cache.h
* Update llama-kv-cache.cpp
* Update llama-kv-cache.cpp
* Update llama-kv-cache.h
* Update llama-kv-cache.cpp
* Review driven changes
* style fix
* indexer hparams are required
* fix tests
* fix lint
---------
Co-authored-by: Daniel Han <[email protected]>
Co-authored-by: Sigbjørn Skjæret <[email protected]>
Co-authored-by: Xuan Son Nguyen <[email protected]>
* Start building graph - reuse deepseek32
* Enable kv cache and rotation for glm_dsa architecture
Just follow Deepseek 3.2 for now.
* Reuse prev_top_k for "shared" indexer layers
* GLM 5.2 uses LLAMA_ROPE_TYPE_NORM for the indexer.
This is transformers' `apply_rotary_pos_emb_interleave`
* Default indexer types to GLM pattern
Previous converted GGUFs like https://huggingface.co/unsloth/GLM-5.2-GGUF write indexer weights to _all_ layers, even if they are only required for "full" types. This PR relies on a new key "%s.attention.indexer.types"; if absent, it will use the default GLM 5.2 schedule as defined in https://huggingface.co/zai-org/GLM-5.2/blob/main/config.json#L26.
Note that conversion is not saving this key yet.
* Save indexer types to gguf, restore on load
* Use ggml_lightning_indexer when cparams.fused_lid
Co-authored-by: fairydreaming <[email protected]>
* GLM 5 and 5.1 use full indexers
Co-authored-by: fairydreaming <[email protected]>
* Fix indentation
* Ensure array is zero-filled
* Prefer explicit std::fill
* Assert prev_top_k exists for shared indexer
---------
Co-authored-by: fairydreaming <[email protected]>
Microsoft BitNet Hugging Face configs use BitNetForCausalLM while the
converter only registered BitnetForCausalLM, causing conversion to fail
with "Model BitNetForCausalLM is not supported".
Register both spellings in TEXT_MODEL_MAP and the Bitnet model class.
Fixesggml-org/llama.cpp#25629
- Add a supports_mtp_export capability to ModelBase so architectures can opt
into --mtp and --no-mtp without extending a central class allowlist.
- Enable the capability for the existing Qwen3.5/3.6 and Step3.5/3.7
implementations, and for HY V3, whose converter already supports
filtering the appended MTP layers.
* model: add Hy3 (hy_v3) architecture support
Adds Tencent Hunyuan 3 (HF architecture HYV3ForCausalLM, GGUF arch
hy_v3): a MoE decoder stack with per-head Q/K RMSNorm, a sigmoid
router with expert selection bias, an always-active ungated shared
expert, and leading dense block(s) (first_k_dense_replace).
The base implementation is ported from charlie12345's fork
(https://github.com/charlie12345/ROCmFPX, src/models/hyv3.cpp),
adapted to current mainline APIs (hparams.n_layer(), build_qkv,
build_moe_ffn with fused gate_up + scale tensors, output_s).
Note: blk.N.exp_probs_b is stored without a .bias suffix for
compatibility with existing hy_v3 GGUFs produced by that fork.
Co-Authored-By: charlie12345 <[email protected]>
Co-authored-by: Piotr Wilkin <[email protected]>
Assisted-by: Claude Fable 5
* convert: add dsv4 conversion
* add basic setup
* add llm_graph_input_dsv4
* add save-load state
* add sinkhorn eps - correction by @fairydreaming
* add rope fix
* cleanup dead code
* fix bugs
* support pro model: added by @fairydreaming
* remove redundant V cache
* Chat template
* remove debugging leftovers
* Add mechanism for inlining templates based on architecture
* s/deepseek-v4-flash/deepseek4/g
* s/deepseek-v4-flash/deepseek4/g continued
* enable graph reuse
* enable FA
* fix test llama archs
* rename
* compatibility with antirez ds4 GGUFs
* simplified set_gguf_parameters() by calling super class method, replaced moe.score_func with expert_gating_func.
* reserve worst-case kv-cache
* revert max split inputs
* address review comments
* add padding to enable FA
* pad only the final value of plan.n_kv to 256
* remove built-in cpp chat template
* cont: remove cpp built-in template
* rm outdated test
* replace ggml_view_3d() with ggml_reshape_3d()
Co-authored-by: Georgi Gerganov <[email protected]>
* only support n_seq=1 for now
* remove unused var
* cont: remove unused var
* use scale bias
* use correct ptr for can_reuse
* remove gen-chat-inline-templates.py
* simplify graph reuse
* cont: cleanup
* remove unused inputs
* enable partial checkpointing
* add correct shape for kq_mask + set llama_model_n_swa to 0 for dsv4
* precompute source_idx + add comment about dummy write
* support multi-seq
* remove restored_trim_pos
* use split_equal when possible
* fix indent
* address review comments
* use LLM_KV
* fix ci
---------
Co-authored-by: Piotr Wilkin <[email protected]>
Co-authored-by: Stanisław Szymczyk <[email protected]>
Co-authored-by: Xuan Son Nguyen <[email protected]>
Co-authored-by: fairydreaming <[email protected]>
Co-authored-by: Georgi Gerganov <[email protected]>
Mistral explicitly sets `moe` and `llama_4_scaling` to `null` in
params.json, breaking `key in dict` checks during conversion. Replace
with `dict.get(key) is not None` where this matters.
Fixes `convert-hf-to-gguf.py --mistral-format Mistral-Medium-3.5-128B`