Xuan-Son Nguyen
22b8e310b9
server: re-design yield_to_queue thread model ( #27133 )
...
* run common_speculative_process in worker
* swap worker <--> main thread design
2026-08-15 16:48:40 +02:00
Xuan-Son Nguyen
77918caf30
server: allow accessing /metrics and /slots during llama_decode() ( #27041 )
...
* server_queue::worker
* call llama_decode inside yield_to_queue
* also handle process_mtmd_chunk
* clean up
* nits
* rm test
2026-08-14 13:23:10 +02:00
Xuan-Son Nguyen
4b4d13ae72
server: (router) add model management API ( #23976 )
...
* wip
* server: (router) add SSE realtime updates API
* nits
* wip
* add download API
* add download api
* update docs
* add delete endpoint
* fix std::terminate
* fix crash
* fix 2
* add tests
* nits
2026-06-17 18:04:58 +02:00
Xuan-Son Nguyen
60130d18f9
server: add SSE ping interval ( #24013 )
2026-06-02 14:14:55 +02:00
Xuan-Son Nguyen
cb47092b00
server: bump timeout to 3600s ( #23842 )
...
* server: bump timeout to 3600s
* nits: change wording
2026-05-29 10:23:17 +02:00
Xuan-Son Nguyen
49bfddeca1
server: allow router to report child instances sleep status ( #20849 )
...
* server: allow router to report child instances sleep status
* refactor
* move sleeping to state
* nits
2026-03-22 18:33:52 +01:00
Xuan-Son Nguyen
a04c2b06a3
server: improve slots scheduling for n_cmpl ( #18789 )
...
* server : make sure children tasks are scheduled to launch with parent
* fix
* add comment pointing to this PR
* fix
* clean up
* more debug messages
* add pop_deferred_task with specific ID version
* improve the logic
* simple approach
* no double move
* correct return type of launch_slots_with_parent_task
2026-01-15 17:10:28 +01:00
Xuan-Son Nguyen
6ce863c803
server: prevent data race from HTTP threads ( #18263 )
...
* server: prevent data race from HTTP threads
* fix params
* fix default_generation_settings
* nits: make handle_completions_impl looks less strange
* stricter const
* fix GGML_ASSERT(idx < states.size())
* move index to be managed by server_response_reader
* http: make sure req & res lifecycle are tied together
* fix compile
* fix index handling buggy
* fix data race for lora endpoint
* nits: fix shadow variable
* nits: revert redundant changes
* nits: correct naming for json_webui_settings
2025-12-22 14:23:34 +01:00
Xuan-Son Nguyen
ddcb75dd8a
server: add auto-sleep after N seconds of idle ( #18228 )
...
* implement sleeping at queue level
* implement server-context suspend
* add test
* add docs
* optimization: add fast path
* make sure to free llama_init
* nits
* fix use-after-free
* allow /models to be accessed during sleeping, fix use-after-free
* don't allow accessing /models during sleep, it is not thread-safe
* fix data race on accessing props and model_meta
* small clean up
* trailing whitespace
* rm outdated comments
2025-12-21 02:24:42 +01:00
Xuan-Son Nguyen and bandoti
6c2131773c
cli: new CLI experience ( #17824 )
...
* wip
* wip
* fix logging, add display info
* handle commands
* add args
* wip
* move old cli to llama-completion
* rm deprecation notice
* move server to a shared library
* move ci to llama-completion
* add loading animation
* add --show-timings arg
* add /read command, improve LOG_ERR
* add args for speculative decoding, enable show timings by default
* add arg --image and --audio
* fix windows build
* support reasoning_content
* fix llama2c workflow
* color default is auto
* fix merge conflicts
* properly fix color problem
Co-authored-by: bandoti <[email protected] [email protected]
2025-12-10 15:28:59 +01:00
Xuan-Son Nguyen
951520ddb0
server: delegate result_state creation to server_task ( #17835 )
...
* server: delegate result_state creation to server_task
* remove unued states
* add more docs
2025-12-08 17:04:38 +01:00
Xuan-Son Nguyen and Georgi Gerganov
c4c10bfb86
server: move msg diffs tracking to HTTP thread ( #17740 )
...
* server: move msg diffs tracking to HTTP thread
* wip
* tool call tests ok
* minor : style
* cont : fix
* move states to server_response_reader
* add safe-guard
* fix
* fix 2
---------
Co-authored-by: Georgi Gerganov <[email protected]
2025-12-04 15:46:08 +01:00
Xuan-Son Nguyen and Georgi Gerganov
ab49f094d2
server: move server-context to its own cpp|h ( #17595 )
...
* git mv
* add server-context.h
* add server-context.h
* clean up headers
* cont : cleanup
* also expose server_response_reader (to be used by CLI)
* fix windows build
* decouple server_routes and server_http
---------
Co-authored-by: Georgi Gerganov <[email protected]
2025-11-29 22:04:44 +01:00
Xuan-Son Nguyen and Georgi Gerganov
b8372eecd9
server: split server.cpp code into server/common/task/queue ( #17362 )
...
* add server-task, server-common
* add server-queue
* rm redundant includes
* move enum stop_type to server-task
* server : headers cleanup
---------
Co-authored-by: Georgi Gerganov <[email protected]
2025-11-24 14:41:53 +01:00
Xuan-Son Nguyen
