quantize: size the output buffer exactly instead of nelements * 4

The per-tensor output buffer was sized `nelements * 4`, described as an upper
bound. It is a very loose one: the output is at most 2 bytes per element
(f16/bf16) and usually well under 1.1 (q8_0 and below), so between 2x and 4x of
it is never touched. The exact size is already known here, since it is what the
quantization loop writes, what new_size sums to, and what the GGUF metadata is
asserted against a few lines later.

On a model whose largest tensor is a few GB none of this matters. On
Qwen3.8-Flash-Next it does: per_layer_token_embd is 51.2 G elements, so the
buffer was 205 GB where 54 GB is needed at q8_0 and 32 GB at q4_1.

Measured on that model, VmHWM of a live llama-quantize was 485 GB per process.
Three of them fit in 2 TB and five did not, which is what an OOM-killed quant
ladder looks like. This removes about 150 GB of that.

Byte-identical output, verified against the same binary built at the parent
commit: q4_K, q8_0, q5_K, q6_K and IQ4_XS, over BF16 and F32 sources, with and
without a PLE table present. Six cases, six matching md5s.
This commit is contained in:
danielhanchen
2026-08-26 13:41:50 +00:00
committed by Daniel Han
parent 4597a86f12
commit 25aa77a4e2
+9 -2
View File
@@ -1266,8 +1266,15 @@ static void llama_model_quantize_impl(const std::string & fname_inp, const std::
LLAMA_LOG_INFO("converting to %s .. ", ggml_type_name(new_type));
fflush(stdout);
if (work.size() < (size_t)nelements * 4) {
work.resize(nelements * 4); // upper bound on size
// Exact output size: ggml_row_size(new_type, ne0) per row, ne1 rows, ne2 slices --
// what the loop writes, what new_size sums to, and what the GGUF metadata is
// asserted against. The previous `nelements * 4` was a loose upper bound, invisible
// on a normal model but 205 GB of dead address space on Qwen3.8-Flash-Next's 51.2 G
// element PLE table -- about half of what OOMed a 2 TB machine at five-wide.
const size_t out_size =
ggml_row_size(new_type, tensor->ne[0]) * tensor->ne[1] * tensor->ne[2];
if (work.size() < out_size) {
work.resize(out_size);
}
new_data = work.data();