sampling : stop short if backend sampler sampled a token

This commit modifies the graph building logic to immediately continue
when a token has already been sampled by the backend sampler.

It also updates the test for backend temporary sampling to include
top-k and distribution samplers in the chain to verify that they are not
producing any logits (they are not run).
This commit is contained in:
Daniel Bevenius
2025-12-04 08:13:49 +01:00
parent cce3b2a8ad
commit 87b2719eca
2 changed files with 3 additions and 0 deletions
+1
View File
@@ -2100,6 +2100,7 @@ void llm_graph_context::build_sampling() const {
if (data.sampled != nullptr) {
res->t_sampled[seq_id] = data.sampled;
ggml_build_forward_expand(gf, data.sampled);
continue;
}
if (data.probs != nullptr) {