Superminaren/llama.cpp
1
0
Fork 0
mirror of https://github.com/ggml-org/llama.cpp.git synced 2026-09-11 04:56:56 +02:00
Code Issues Packages Projects Releases Wiki Activity
4,876 Commits 780 Branches 7,393 Tags
7f02ee562efae35fa0abcd8f4ae3bbfe3728be27
Commit Graph
4876 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Georgi Gerganov 7f02ee562e context : decouple inputs, llama_graph_i become const (WIP) 
ggml-ci
 2025-02-28 16:30:41 +02:00
Georgi Gerganov 38db8a5861 llama : introduce concept of llama_memory 
ggml-ci
 2025-02-28 10:51:17 +02:00
Georgi Gerganov 828effd9d7 kv-cache : basic abstraction 
ggml-ci
 2025-02-27 16:00:29 +02:00
Georgi Gerganov 82675a0180 Merge branch 'master' into gg/llama-kv-cache 
ggml-ci
 2025-02-27 15:10:18 +02:00
Georgi Gerganov 952feedfca context : disable encoder embd tensor for now 
ggml-ci
 2025-02-27 15:07:10 +02:00
Vladimir Vuksanovic b95c8af37c cmake: Fix ggml backend dependencies and installation (#11818) 
* Fix dependencies between ggml and backends

ggml backends link only to ggml-base and ggml links to all backends.

* Fix installation of ggml backends

Set up GNUInstallDirs before setting the installation directory of ggml backends
b4784 		2025-02-27 09:42:48 +02:00
Ting LouandXuan-Son Nguyen a800ae46da llava : add struct for FFI bindgen (#12079) 
* add struct for FFI bindgen

* Apply suggestions from code review

---------

Co-authored-by: Xuan-Son Nguyen <[email protected]>
b4783 		2025-02-26 15:26:52 +01:00
Sigbjørn Skjæret 69050a11be Refactor gguf scripts to improve metadata handling (#11909) 
* Refactor gguf scripts to improve metadata handling

Added contents method to ReaderField class
Added endianess property to GGUFReader class

* update scripts

* fix import

* remove unused import

* attempt to work around flake and pyright errors

* second attempt

* give up, ignore type

* bump version

* apply newbyteorder fixes
gguf-v0.16.0 		2025-02-26 08:04:48 -05:00
Aleksei Nikiforov 3567ee3a94 gguf-py: enable reading non-native endian files (#12081) 
Currently self.byte_order is never used.
Actually use it to byteswap read data to
allow reading big endian files on little endian systems
and vice versa.

Now it's possible to convert little-endian model
into a big-endian model and back
on a little-endian system.
 2025-02-26 11:39:27 +00:00
Kante Yin 53e4db1012 readme : update infra list (#9096) 
Signed-off-by: kerthcet <[email protected]>
 2025-02-26 09:49:36 +02:00
Olivier Chafik d7cfe1ffe0 docs: add docs/function-calling.md to lighten server/README.md's plight (#12069) 2025-02-25 18:52:56 +00:00
Jeff Bolz a82c9e7c23 vulkan: fix assertion when qy_needs_dequant (#12068) 
Looks like a copy/paste bug from qx_needs_dequant.
b4778 		2025-02-25 16:30:21 +01:00
Georgi Gerganov 4efe989886 context : pass embeddings tensor from encoder to decoder 
ggml-ci
 2025-02-25 16:11:17 +02:00
rhjdvsgsgks 401af80b54 server: handle echo=false on /v1/completions (#12060) b4777 2025-02-25 12:52:52 +01:00
JuddandJudd c132239bfb add OP sigmoid (#12056) 
Co-authored-by: Judd <[email protected]>
b4776 		2025-02-25 12:32:20 +01:00
Molly Sophia 393fca629e ggml-cpu: Fix build with sve (#12059) 
* ggml-cpu: Fix build with sve

Signed-off-by: Molly Sophia <[email protected]>

* ggml-cpu: Remove unused variable in sve q3_k vec dot

Signed-off-by: Molly Sophia <[email protected]>

---------

Signed-off-by: Molly Sophia <[email protected]>
b4775 		2025-02-25 19:28:22 +08:00
Rémy O 61d4f39dfe vulkan: implement more backpropagation operators (#11914) 
* vulkan: implement GGML_OP_ROPE_BACK

* vulkan: implement GGML_OP_RMS_NORM_BACK

* vulkan: implement GGML_OP_SILU_BACK

* vulkan: implement GGML_OP_SOFTMAX_BACK
b4774 		2025-02-25 12:04:45 +01:00
Olivier Chafik 0b52745649 server: support add_generation_prompt query param (#12062) b4773 2025-02-25 10:40:22 +00:00
Georgi Gerganov e2b3294f2c context : fix enc-dec state save/load 
ggml-ci
 2025-02-25 12:14:34 +02:00
Georgi Gerganov e5bc5f8e02 context : enc-dec is now working 
ggml-ci
 2025-02-25 12:10:34 +02:00
Alex Brooks 4d1051a40f Add Doc for Converting Granite Vision -> GGUF (#12006) 
* Add example docs for granite vision

Signed-off-by: Alex-Brooks <[email protected]>
 2025-02-25 10:46:05 +01:00
Vitali Lovich 3e9a2860e9 llama : expose llama_model_n_head_kv in the API (#11997) 
It's useful to be able to have this from the library layer as it's a key
parameter of the model (e.g. to figure out how much KV cache memory is
needed).
b4771 		2025-02-25 11:29:33 +02:00
Gian-Carlo PascuttoandGeorgi Gerganov 58d07a8043 metal : copy kernels for quant to F32/F16 conversions (#12017) 
metal: use dequantize_q templates

---------

Co-authored-by: Georgi Gerganov <[email protected]>
b4770 		2025-02-25 11:27:58 +02:00
lhezShawn GuSkyler Szot
 		34a846b584 opencl: fix for small models (#11950) 
* opencl: fix small shape gemv, remove unused extensions

* opencl: fix `transpose_16`, `dump_tensor`, enforce subgroup size

* opencl: fix for token length < 4

* opencl: use wave size of 64 for all Adreno GPUs

---------

Co-authored-by: Shawn Gu <[email protected]>
Co-authored-by: Skyler Szot <[email protected]>
b4769 		2025-02-24 14:47:07 -07:00
Georgi Gerganov be58e30017 enc-dec : compose wip 
ggml-ci
 2025-02-24 18:12:24 +02:00
Alex BrooksandXuan-Son Nguyen 7a2c913e66 llava : Add Granite Vision Support (#11794) 
* Add super wip scripts for multimodal granite gguf

Signed-off-by: Alex-Brooks <[email protected]>

* Add example for converting mmgranite to gguf

Signed-off-by: Alex-Brooks <[email protected]>

* remove hardcoded path

Signed-off-by: Alex-Brooks <[email protected]>

* Add vision feature layer to gguf params

Signed-off-by: Alex-Brooks <[email protected]>

* Clean up llava surgery and remove name substitution hacks

Signed-off-by: Alex-Brooks <[email protected]>

* Add transformers llava next tensor name mapping

Signed-off-by: Alex-Brooks <[email protected]>

* Make siglip / openclip mutuall exclusive

Signed-off-by: Alex-Brooks <[email protected]>

* Fix projector linear substitution

Signed-off-by: Alex-Brooks <[email protected]>

* Fix linear 2 substitution index

Signed-off-by: Alex-Brooks <[email protected]>

* Increase max flattened gridpoints to 64

Signed-off-by: Alex-Brooks <[email protected]>

* Fix hardcoded concat for multiple feature layers

Signed-off-by: Alex-Brooks <[email protected]>

* Pull vision feature layers out of gguf keys

Signed-off-by: Alex-Brooks <[email protected]>

* fix num gridpoints and use all layers

Signed-off-by: Alex-Brooks <[email protected]>

* Avoid dropping last image encoder layer in llava models

Signed-off-by: Alex-Brooks <[email protected]>

* Use 10 for max number of patches

Signed-off-by: Alex-Brooks <[email protected]>

* Standardize vision feature layers

Signed-off-by: Alex-Brooks <[email protected]>

* Cleanup logs

Signed-off-by: Alex-Brooks <[email protected]>

* Update comment for vision feature layer init

Signed-off-by: Alex-Brooks <[email protected]>

* Update notes for alternative to legacy llm conversion script

Signed-off-by: Alex-Brooks <[email protected]>

* Fix notes rendering

Signed-off-by: Alex-Brooks <[email protected]>

* Add v prefix to vision feature layer log

Signed-off-by: Alex-Brooks <[email protected]>

* Use current defaults for feature layer

Signed-off-by: Alex-Brooks <[email protected]>

* Use constant for max gridpoints / feat layers, style fixes

Signed-off-by: Alex-Brooks <[email protected]>

* clarify non-negative feature layers

Signed-off-by: Alex-Brooks <[email protected]>

* Remove CLIP_API from func signature

Signed-off-by: Alex-Brooks <[email protected]>

* USE MAX_IMAGE_FEATURE_LAYERS const in layer calc

Signed-off-by: Alex-Brooks <[email protected]>

* Clarify feature layers are non negative ints and not uint

Signed-off-by: Alex-Brooks <[email protected]>

* Fix condition for reading feature layers

Signed-off-by: Alex-Brooks <[email protected]>

* pop last llava layer when feature layers are unset

Signed-off-by: Alex-Brooks <[email protected]>

* Fix unset vision layer 0

Signed-off-by: Alex-Brooks <[email protected]>

* Update examples/llava/clip.cpp

Co-authored-by: Xuan-Son Nguyen <[email protected]>

* Reenable assertion for out of bounds get_rows

Signed-off-by: Alex-Brooks <[email protected]>

* Use std vector for gridpoints and feature layers

Signed-off-by: Alex-Brooks <[email protected]>

* Caculate max feature layer at load time

Signed-off-by: Alex-Brooks <[email protected]>

* Include base patch for granite vision allocation

Signed-off-by: Alex-Brooks <[email protected]>

* Fix trailing whitespace

Signed-off-by: Alex-Brooks <[email protected]>

* Add max num patches = 10 back for minicpmv

Signed-off-by: Alex-Brooks <[email protected]>

* Use unordered set to store feature layers

Co-authored-by: Xuan-Son Nguyen <[email protected]>
Signed-off-by: Alex-Brooks <[email protected]>

* Use max feature layer for postnorm

Signed-off-by: Alex-Brooks <[email protected]>

* Apply suggestions from code review

---------

Signed-off-by: Alex-Brooks <[email protected]>
Co-authored-by: Xuan-Son Nguyen <[email protected]>
b4768 		2025-02-24 17:09:51 +01:00
Neo Zhang Jianyuandarthw 08d5986290 [SYCL] Optimize mul_mat for Q4_0 on Intel GPU (#12035) 
* opt performance by reorder for Intel GPU

* detect hw type and save opt feature, and print opt feature

* correct name

* support optimize graph once when compute graph, record the opt status in tensor->extra, make CI passed

* add env variable GGML_SYCL_DISABLE_OPT for debug

* use syclex::architecture replace the custom hw define, update the guide for GGML_SYCL_DISABLE_OPT

* add performance data

* mv getrows functions to separeted files

* fix global variables

---------

Co-authored-by: arthw <[email protected]>
b4767 		2025-02-24 22:33:23 +08:00
Georgi Gerganov 9cd78f11a1 context : explicit llama_context_i abstract interface 
ggml-ci
 2025-02-24 13:38:11 +02:00
Aleksei Nikiforov 651adf4b66 gguf_convert_endian.py: implement byteswapping for q4_k and q6_k (#11349) 2025-02-24 11:27:01 +00:00
Akarshan Biswas 8303e8b0fb SYCL: Fix GGML_SYCL_DEBUG macro (#11995) b4765 2025-02-24 10:18:25 +00:00
Georgi Gerganov 4a1054b552 context : reuse built_attn_mha 
ggml-ci
 2025-02-24 11:29:52 +02:00
Georgi Gerganov a5a85a3bc0 context : fix recurrent reserve 
ggml-ci
 2025-02-24 08:59:12 +02:00
Georgi Gerganov 0699a44c83 context : remove redundant virtual, protected -> private 
ggml-ci
 2025-02-23 20:02:11 +02:00
Georgi Gerganov 6378112cb5 graph : remove the build_kv_... API from llama_graph_i 
ggml-ci
 2025-02-23 19:39:22 +02:00
Florent BENOIT 7ad0779f5d run: allow to customize prompt by env var LLAMA_PROMPT_PREFIX (#12041) 
Signed-off-by: Florent Benoit <[email protected]>
b4764 		2025-02-23 17:15:51 +00:00
Eric Curtin f777a73e18 Some llama-run cleanups (#11973) 
Use consolidated open function call from File class. Change
read_all to to_string(). Remove exclusive locking, the intent for
that lock is to avoid multiple processes writing to the same file,
it's not an issue for readers, although we may want to consider
adding a shared lock. Remove passing nullptr as reference,
references are never supposed to be null. clang-format the code
for consistent styling.

Signed-off-by: Eric Curtin <[email protected]>
b4763 		2025-02-23 13:14:32 +00:00
Georgi Gerganov 372fa3a894 cont : enc should work now, next is dec 
ggml-ci
 2025-02-23 12:20:23 +02:00
Aaron TeoJinyang Hejunchao-zhao
 		af7747c95a ggml-cpu: Support s390x SIMD Instruction Set (#12019) 
* ggml: add s390x ARCH_FLAGS for compilation

Signed-off-by: Aaron Teo <[email protected]>

* ggml: add SIMD for s390x using vector intrinsics

SIMD is activated for:
* ggml_vec_dot_f32
* ggml_vec_dot_f16
* ggml_vec_mad_f32
* ggml_vec_mad_f16
* ggml_vec_mad_f32_unroll
* ggml_vec_scale_f32
* ggml_vec_scale_f16

SIMD is NOT activated for:
* ggml_vec_dot_f16_unroll (pending bugfix)

Signed-off-by: Aaron Teo <[email protected]>

* ggml: fix missing escape character in GGML_F32x4_REDUCE

Signed-off-by: Aaron Teo <[email protected]>

* ggml: add temporary patch for GGML_F32_ARR and GGML_F16_ARR

Signed-off-by: Aaron Teo <[email protected]>

* ggml: fix s390x GGML_F32x4_REDUCE

Signed-off-by: Aaron Teo <[email protected]>

* ggml: full SIMD activation for F32,F16 s390x

Signed-off-by: Aaron Teo <[email protected]>

* ggml: add option to disable s390x VXE/VXE2

Signed-off-by: Aaron Teo <[email protected]>

* ggml: change vecintrin.h include to ggml-cpu-impl

* add __VXE__ and __VXE2__ macros

Signed-off-by: Aaron Teo <[email protected]>

* cmake: add s390x target detection for VX/VXE/VXE2

Signed-off-by: Aaron Teo <[email protected]>

* ggml: move s390x vector intrinsics to ggml-cpu-impl.h

Signed-off-by: Aaron Teo <[email protected]>

* ggml: s390x Q8_0 SIMD

Signed-off-by: Aaron Teo <[email protected]>

* ggml: correct documentation for Q8_0

Signed-off-by: Aaron Teo <[email protected]>

* ggml: s390x reduce code complexity Q8_0

Signed-off-by: Aaron Teo <[email protected]>

* ggml: s390x bugfix typo Q8_0

Signed-off-by: Aaron Teo <[email protected]>

* ggml: s390x SIMD activated for Q4_1

Signed-off-by: Aaron Teo <[email protected]>

* ggml: s390x inline vec_reve

Signed-off-by: Aaron Teo <[email protected]>

* ggml: s390x SIMD activation for Q4_0

Signed-off-by: Aaron Teo <[email protected]>

* ggml: add VXE backend feature

Signed-off-by: Aaron Teo <[email protected]>

* ggml: remove test.py

Signed-off-by: Aaron Teo <[email protected]>

* ggml: s390x SIMD activation for quantize_row_q8_0

Signed-off-by: Aaron Teo <[email protected]>

* ggml: s390x SIMD activation for quantize_row_q8_1

Signed-off-by: Aaron Teo <[email protected]>

* ggml: s390x SIMD activation for iq4_xs

Signed-off-by: Aaron Teo <[email protected]>

* ggml: bugfix iq4_xs

Signed-off-by: Aaron Teo <[email protected]>

* ggml: s390x SIMD activation for iq4_nl

Signed-off-by: Aaron Teo <[email protected]>

* ggml: add float, double, and long vector data type

Signed-off-by: Aaron Teo <[email protected]>

* ggml: clean up iq4_xs SIMD

Signed-off-by: Aaron Teo <[email protected]>

* ggml: fix improper use of restrict keyword

Signed-off-by: Aaron Teo <[email protected]>

* ggml: update warning message for ggml_vec_tbl

Signed-off-by: Aaron Teo <[email protected]>

* ggml: untested implementation of ggml_vec_dot_iq2_xxs_q8_K

Signed-off-by: Aaron Teo <[email protected]>

* ggml: update ggml_vec_dot_q4_1_q8_1 to use typedefs

Signed-off-by: Aaron Teo <[email protected]>

* ggml: switch to restrict for iq4_nl

Signed-off-by: Aaron Teo <[email protected]>

* ggml: slight dot product speed improvement for q4_1_q8_1

Signed-off-by: Aaron Teo <[email protected]>

* ggml: s390x SIMD activation for q6_K

Signed-off-by: Aaron Teo <[email protected]>

* ggml: add missing `_t` to ggml_int8x16x4_t

Signed-off-by: Aaron Teo <[email protected]>

* ggml: fix missing `_t` for ggml_vec_xl_s8x4

Signed-off-by: Aaron Teo <[email protected]>

* ggml: fix more missing `_t`

Signed-off-by: Aaron Teo <[email protected]>

* ggml: add unroll and prefetch to Q8_0

increase of 3.86% for prompt processing and 32.22% for token generation

Signed-off-by: Aaron Teo <[email protected]>

* ggml: patch Q8_0 to use proper vector sizes

Signed-off-by: Aaron Teo <[email protected]>

* ggml: optimise Q8_0 dot prod compute kernel further

Signed-off-by: Aaron Teo <[email protected]>

* ggml: add unroll and prefetch to Q4_1

Signed-off-by: Aaron Teo <[email protected]>

* ggml: refactor Q6_K variable naming for readability

Signed-off-by: Aaron Teo <[email protected]>

* ggml: fix Q6_K typos

Signed-off-by: Aaron Teo <[email protected]>

* ggml: s390x SIMD activation for Q5_K

Signed-off-by: Aaron Teo <[email protected]>

* ggml: fix wrong char*x16_t naming

Signed-off-by: Aaron Teo <[email protected]>

* ggml: Q5_K y0 wrong signness

Signed-off-by: Aaron Teo <[email protected]>

* ggml: fix Q5_K invalid uchar type

Signed-off-by: Aaron Teo <[email protected]>

* ggml: fix Q5_K invalid uchar type

Signed-off-by: Aaron Teo <[email protected]>

* ggml: s390x SIMD activation for Q4_K

Signed-off-by: Aaron Teo <[email protected]>

* ggml: fix Q4_K invalid vector intrinsics

Signed-off-by: Aaron Teo <[email protected]>

* ggml: simplify ggml_padd_s16 compute kernel

Signed-off-by: Aaron Teo <[email protected]>

* ggml: correct ggml-cpu vxe wording

Signed-off-by: Aaron Teo <[email protected]>

* ggml: change ggml_aligned_malloc alignment to 256

256 is the cache line size for s390x platforms

Signed-off-by: Aaron Teo <[email protected]>

* ggml: resolve pr merge via cherry-pick 225bbbf

Signed-off-by: Aaron Teo <[email protected]>

* ggml : fix LoongArch compile error with 128-bit SIMD (#11701)

* ggml: resolve pr merge via cherry-pick 4571953

Signed-off-by: Aaron Teo <[email protected]>

* ggml: cmake remove fork when determining s390x machine type

thank you @ericcurtin

Signed-off-by: Aaron Teo <[email protected]>

---------

Signed-off-by: Aaron Teo <[email protected]>
Co-authored-by: Jinyang He <[email protected]>
Co-authored-by: junchao-zhao <[email protected]>
b4762 		2025-02-22 21:39:24 +00:00
Johannes Gäßler a28e0d5eb1 CUDA: app option to compile without FlashAttention (#12025) b4761 2025-02-22 20:44:34 +01:00
Ting LouandXuan-Son Nguyen 36c258ee92 llava: build clip image from pixels (#11999) 
* llava: export function `clip_build_img_from_pixels` to build image from pixels decoded by other libraries instead of stb_image.h for better performance

* Apply suggestions from code review

---------

Co-authored-by: Xuan-Son Nguyen <[email protected]>
b4760 		2025-02-22 15:28:28 +01:00
Georgi Gerganov f3e64859ed ci : fix arm upload artifacts (#12024) 
* ci : fix arm upload artifacts

* cont : fix archive name to use matrix
b4759 		2025-02-22 15:03:00 +02:00
Johannes Gäßler 5fa07c2f93 CUDA: optimize FA for GQA + large batches (#12014) 2025-02-22 12:20:17 +01:00
Rohanjames1997 335eb04a91 ci : Build on Github-hosted arm64 runners (#12009) 2025-02-22 11:48:57 +01:00
Georgi Gerganov cf756d6e0a server : disable Nagle's algorithm (#12020) b4756 2025-02-22 11:46:31 +01:00
Gian-Carlo Pascutto d70908421f cuda: Add Q5_1, Q5_0, Q4_1 and Q4_0 to F32 conversion support. (#12000) b4755 2025-02-22 09:43:24 +01:00
Daniel Bevenius de8b5a3624 llama.swiftui : add "Done" dismiss button to help view (#11998) 
The commit updates the help view in the llama.swiftui example to use a
NavigationView and a Done button to dismiss the help view.

The motivation for this is that without this change there is now way to
dimiss the help view.
b4754 		2025-02-22 06:33:29 +01:00
Georgi Gerganov f5e80208c5 wip enc-dec 2025-02-21 19:17:47 +02:00
Georgi Gerganov c4c0a4d13c Merge branch 'master' into gg/llama-kv-cache 2025-02-21 19:14:07 +02:00
Georgi Gerganov 51f311e057 llama : skip loading unused tensors (#12004) 
* llama : assign unknown/unused tensors to host buffer type

ggml-ci

* llama : skip unused tensors

ggml-ci
b4753 		2025-02-21 18:33:18 +02:00
Georgi Gerganov 3753b30d65 context : fix n_outputs init 
ggml-ci
 2025-02-21 15:53:26 +02:00