f32_conv_buf held the whole dequantized tensor, which is 204.8 GB for
per_layer_token_embd alone and dies with std::bad_alloc long before the
work buffer is reached. Dequantize and quantize in bands of whole rows
instead, capping the f32 staging at 1 GiB per band.
Rows are independent and the imatrix is indexed by column, so band
boundaries cannot change any output byte. Bands nest inside the existing
per-expert loop so each expert slice keeps its own imatrix, and a band is
kept to at least one quantization chunk per worker thread so the existing
multithreading still has work. F32 sources still stage nothing and are
banded by pointer arithmetic into the tensor.
llama_tensor_dequantize_impl now takes a first element offset; the single
caller is updated.
(cherry picked from commit 658c22549613555dbce57a772be4de8509eba3ee)
danielhanchen
ef9fa1ba1f