 Georgi Gerganov
|
14e733e36f
|
spec : refactor params (#22397)
* spec : refactor params
* cont : fix
* cont : rename "sparam" to "sampling"
* cont : add spec params category
* cont : add info about removed arguments
* cont : skip param length check for spec params
* cont : adapt server tests
|
2026-04-28 09:07:33 +03:00 |
|
|
Georgi Gerganov
|
de71b5f81c
|
server : refactor "use checkpoint" logic (#22114)
|
2026-04-20 08:42:37 +03:00 |
|
 Sascha RogmannandGeorgi Gerganov
|
455d8e4be8
|
server : speculative checkpointing (#19493)
* server : speculative decoding using checkpoints
* server : fix draft check with checkpoints
* server : rename spec vars
* server : log levels
* server : refactored spec logic to speculative.cpp
* server : renamed spec checkpoints option
* server : fix spec checkpoints, logging
* speculative : checkpoints with draft model, logging
* server : n_tokens_cur and create_checkpoint in draft
* server : fix server_speculative_callback (slot.id)
* spec : fix ngram-map/begin idx_last_check
* spec : init ckpt (begin() wasn't called)
* chore: update webui build output
* server : restore sampler in spec checkpoint and clear mem
* cont : avoid --spec-use-checkpoints argument
* cont : remove server_prompt_checkpoint_with_size
* spec : rename (leave_draft_state)
* cont : clean-up
* cont : do not ignore partial drafts even if the are short
* cont : spec callback owned by session
* cont : simplify
* cont : avoid empty speculative session
* cont : simplify
* cont : simplify
* cont : enable mtmd speculative decoding
* cont : keep the spec sampler alive
* cont : simplify
* cont : fix nullptr deref + draft checkpoints
* cont : remove common_speculative_accept_response
* cont : remove callback
* cont : simplify
* cont : minor
* cont : simplify
* cont : fix accepted number
---------
Co-authored-by: Georgi Gerganov <[email protected]>
|
2026-04-19 10:24:06 +03:00 |
|
|
Georgi Gerganov
|
dfde5993ea
|
common : add common_speculative_is_compat() (#19270)
* llama : add llama_memory_can_rm_suffix()
* Revert "llama : add llama_memory_can_rm_suffix()"
This reverts commit d30e59b62a.
* spec : check if the target context is compatible for spec decoding
|
2026-02-06 16:47:22 +02:00 |
|
|
Georgi Gerganov
|
dabaa2e77a
|
spec : add ngram-mod (#19164)
* spec : add ngram-mod
* cont : simplify + keep track of occupancy
* cont : cleanup
* cont : move initialization to common/speculative
* cont : cleanup
* cont : cleanup
* cont : fix
|
2026-01-30 18:21:48 +02:00 |
|
|
72d3b1898a
|
spec : add self‑speculative decoding (no draft model required) + refactor (#18471)
* server: introduce self-speculative decoding
* server: moved self-call into speculative.cpp
* can_speculate() includes self-speculation
Co-authored-by: Georgi Gerganov <[email protected]>
* server: can_speculate() tests self-spec
* server: replace can_speculate() with slot.can_speculate()
Co-authored-by: Sigbjørn Skjæret <[email protected]>
* common: use %zu format specifier for size_t in logging
Co-authored-by: Sigbjørn Skjæret <[email protected]>
* server: can_speculate() requires a task instance
* common: ngram map, config self-speculative decoding
* common: add enum common_speculative_type
* common: add vector of speculative states
* common: add option --spec-draftless
* server: cleanup (remove slot.batch_spec, rename)
* common: moved self-spec impl to ngram-map
* common: cleanup (use common_speculative_state_draft)
* spec : refactor
* cont : naming
* spec: remove --spec-config
* doc: (draftless) speculative decoding
* common: print performance in spec decoding
* minor : cleanup
* common : better names
* minor : cleanup + fix build
* minor: comments
* CODEOWNERS: add common/ngram-map.* (#18471)
* common : rename speculative.draftless_type -> speculative.type
* ngram-map : fix uninitialized values
* ngram-map : take into account the input can become shorter
* ngram-map : revert len check for now
* arg : change `--spec-draftless` -> `--spec-type`
* spec : add common_speculative_state::accept()
* spec : refactor + add common_speculative_begin()
* spec : fix begin() call with mtmd
* spec : additional refactor + remove common_speculative_params
---------
Co-authored-by: Georgi Gerganov <[email protected]>
Co-authored-by: Sigbjørn Skjæret <[email protected]>
|
2026-01-28 19:42:42 +02:00 |
|
 g2mtandGeorgi Gerganov
|
94933c8c2e
|
server : implement universal assisted decoding (#12635)
* llama-server : implement universal assisted decoding
* Erase prompt tail for kv-cache
* set vocab_dft_compatible in common_speculative
* rename ctx_main to ctx_tgt
* move vocab_dft_compatible to spec struct
* clear mem_dft, remove mem
* detokenize id_last for incompatible models
* update comment
* add --spec-replace flag
* accept special tokens when translating between draft/main models
* Escape spec-replace
* clamp draft result to size to params.n_draft
* fix comment
* clean up code
* restore old example
* log common_speculative_are_compatible in speculative example
* fix
* Update common/speculative.cpp
Co-authored-by: Georgi Gerganov <[email protected]>
* Update common/speculative.cpp
Co-authored-by: Georgi Gerganov <[email protected]>
* Update common/speculative.cpp
Co-authored-by: Georgi Gerganov <[email protected]>
---------
Co-authored-by: Georgi Gerganov <[email protected]>
|
2025-07-31 14:25:23 +02:00 |
|
|
Georgi Gerganov
|
abd4d0bc4f
|
speculative : update default params (#11954)
* speculative : update default params
* speculative : do not discard the last drafted token
|
2025-02-19 13:29:42 +02:00 |
|
 Maxim Evtush
|
7b891bdc86
|
fix: typos in documentation files (#11791)
* Update ggml.c
* Update arg.cpp
* Update speculative.h
|
2025-02-10 23:21:31 +01:00 |
|
|
Georgi Gerganov
|
d9d54e498d
|
speculative : refactor and add a simpler example (#10362)
* speculative : refactor and add a simpler example
ggml-ci
* speculative : clean-up and add comments and TODOs [no ci]
* speculative : manage context in common_speculative
ggml-ci
* speculative : simplify
ggml-ci
* speculative : simplify (cont)
ggml-ci
* speculative : add --draft-min CLI arg
* speculative : minor fixup
* make : build fixes
* speculative : do not redraft previous drafts
ggml-ci
* speculative : fix the draft sampling
ggml-ci
* speculative : fix compile warning
* common : refactor args
ggml-ci
* common : change defaults [no ci]
* common : final touches
ggml-ci
|
2024-11-25 09:58:41 +02:00 |
|