From 25aa77a4e2ad491b511e223f2de5950b7e1946ee Mon Sep 17 00:00:00 2001 From: danielhanchen Date: Wed, 26 Aug 2026 00:31:26 +0000 Subject: [PATCH] quantize: size the output buffer exactly instead of nelements * 4 The per-tensor output buffer was sized `nelements * 4`, described as an upper bound. It is a very loose one: the output is at most 2 bytes per element (f16/bf16) and usually well under 1.1 (q8_0 and below), so between 2x and 4x of it is never touched. The exact size is already known here, since it is what the quantization loop writes, what new_size sums to, and what the GGUF metadata is asserted against a few lines later. On a model whose largest tensor is a few GB none of this matters. On Qwen3.8-Flash-Next it does: per_layer_token_embd is 51.2 G elements, so the buffer was 205 GB where 54 GB is needed at q8_0 and 32 GB at q4_1. Measured on that model, VmHWM of a live llama-quantize was 485 GB per process. Three of them fit in 2 TB and five did not, which is what an OOM-killed quant ladder looks like. This removes about 150 GB of that. Byte-identical output, verified against the same binary built at the parent commit: q4_K, q8_0, q5_K, q6_K and IQ4_XS, over BF16 and F32 sources, with and without a PLE table present. Six cases, six matching md5s. --- src/llama-quant.cpp | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/src/llama-quant.cpp b/src/llama-quant.cpp index eb94c8a212..a9611bdaf2 100644 --- a/src/llama-quant.cpp +++ b/src/llama-quant.cpp @@ -1266,8 +1266,15 @@ static void llama_model_quantize_impl(const std::string & fname_inp, const std:: LLAMA_LOG_INFO("converting to %s .. ", ggml_type_name(new_type)); fflush(stdout); - if (work.size() < (size_t)nelements * 4) { - work.resize(nelements * 4); // upper bound on size + // Exact output size: ggml_row_size(new_type, ne0) per row, ne1 rows, ne2 slices -- + // what the loop writes, what new_size sums to, and what the GGUF metadata is + // asserted against. The previous `nelements * 4` was a loose upper bound, invisible + // on a normal model but 205 GB of dead address space on Qwen3.8-Flash-Next's 51.2 G + // element PLE table -- about half of what OOMed a 2 TB machine at five-wide. + const size_t out_size = + ggml_row_size(new_type, tensor->ne[0]) * tensor->ne[1] * tensor->ne[2]; + if (work.size() < out_size) { + work.resize(out_size); } new_data = work.data();