* Batched gemm for grid IQ quants
Style updates and a bit more performance
Clean up comments
Move code around
Vectorize IQ panel decode, lower threshold for speedup
IQ panel: single-source gather layout, gate bias, vectorize interleave
Add ggml_gemm_iqp_8x8_q8_K_p4 kernel, remove gather buffer
Move IQ panel code out of repack into iqp.cpp, clean up comments
Another comment sweep
* Add myself as iqp.* codeownder
* Remove ggml_cpu_iqp_scratch_offset and ggml_cpu_iqp_src1_conv_size
* Renaming and moving
* The other half of renaming and moving
* Move macros and ggml_cpu_iqp_mul_mat_id_min_batch definition
* Update ggml/src/ggml-cpu/iqp.h
Co-authored-by: Georgi Gerganov <[email protected]>
* Add iqp_rows work buffer
* Revert "Add iqp_rows work buffer"
This reverts commit 425542991e.
* Add NUMA fallback
* Add 10 row batch tests for IQP coverage on all grid IQ types
* Swap assert for return false in support check
* Move IQP mul_mat_id test
---------
Co-authored-by: Georgi Gerganov <[email protected]>
* Add ctx-per-slot argument for unifid KV cache
* Swap out ctx fractions for ctx pool slots
* Formatting cleanup
* Remove ctx-pool-slots, make ctx-per-slot an int
* refactor it
---------
Co-authored-by: Xuan Son Nguyen <[email protected]>
* Add unit test coverage for llama_tensor_get_type
* Fix merge conflicts, add more schemas
* clang formatter changes
* Trailing whitespace
* Update name
* Start rebase
* Updating files with upstream changes prior to rebase
* Changes needed from rebase
* Update attn_qkv schema, change throw behaviour
* Fix merge conflicts
* White space
* Update with latest changes to state counters
* Revert accidental personal CLAUDE.md changes
* Change quotation mark
* Reuse metadata.name since we have it
* Move test-only stuff out of llama-quant.cpp
* Hide the regex functionality back in llama-quant.cpp, use a unique pointer to a new struct 'compiled_tensor_type_patterns' which contains the patterns
* cont : inital deslop guidelines
* Cleanup based on review comments
* Continue cleanup
* Small cleanup
* Manually set proper ordering of tensors, mostly applies to gemma
* Formatting
* Update tests/test-quant-type-selection.cpp
Co-authored-by: Sigbjørn Skjæret <[email protected]>
* Fix merge conflicts
---------
Co-authored-by: Georgi Gerganov <[email protected]>
Co-authored-by: Sigbjørn Skjæret <[email protected]>
* Add model metadata loading from huggingface for use with other tests
* Add incremental chunking instead of full redownload, fix caching issue and add warning when it fails
* Add support for split models, load metadata from each individual split file, also avoid mmproj
* Code cleanup, revert incremental downloading
* Only compile when cpp-httplib has SSL support
* Fix formatting
* Make a few GLM tensors not required
layer.nextn.shared_head_head and layer.nextn.embed_tokens are both excluded from GLM 4.6 resulting in the model not loading after conversion/quantization, this marks those tensors as not required which makes it work
* Update llama-model.cpp
layer.nextn.shared_head_norm also not required in case of future models
* Add Arcee AFM support
* Add draft update code
* Fix linter and update URL, may still not be final
* Update src/llama-model.cpp
Co-authored-by: Xuan-Son Nguyen <[email protected]>
* Remote accidental blank line
---------
Co-authored-by: Xuan-Son Nguyen <[email protected]>
BartowskiandGeorgi Gerganov
