Francis Couture-Harpin
ec50944bf6
ggml-quants : fix build failure on Windows
2024-06-28 20:41:13 -04:00
Francis Couture-Harpin
bfd2f21fb4
bitnet : replace 1.58b with b1.58, as in the paper
2024-06-28 20:38:12 -04:00
Francis Couture-Harpin
0996149911
convert-hf : allow converting the weird BitNet 1.3B
...
Its FFN size is 5460 which is not convenient.
The offending tensors are kept in F16,
which makes the final model 5.01 bpw.
2024-06-27 02:06:28 -04:00
Francis Couture-Harpin
961e293833
convert-hf : simplify BitNet pre-quantization
...
This still results in the exact same tensor weights and scales,
but it reveals some weirdness in the current algorithm.
2024-06-27 02:06:28 -04:00
Francis Couture-Harpin
89dc3b254c
ggml-quants : use ceiling division when quantizing q1_3
2024-06-27 02:06:28 -04:00
Francis Couture-Harpin
9465ec6e12
ggml-quants : ARM NEON vec_dot for q2_2 and q1_3
2024-06-27 02:06:28 -04:00
Francis Couture-Harpin
638ad52f87
ggml-quants : cleanup Q1_3 code formatting
2024-06-27 02:06:28 -04:00
Francis Couture-Harpin
ef1e345c85
ggml-quants : Q2_2 now faster than Q4_K on with AVX2
2024-06-27 02:06:28 -04:00
Francis Couture-Harpin
48b73b8498
ggml-quants : substract 1 when back in epi8
...
This makes the 1.625 bpw type go faster than q4_0. Still not the fastest.
2024-06-27 02:06:28 -04:00
Francis Couture-Harpin
7ef4254a92
ggml-quants : faster 1.625 bpw AVX2 vec_dot
...
Not using a lookup table anymore makes it match q4_0 speed.
* gguf-py : fix formatting
* llama : remove spaces on empty line
2024-06-27 02:06:28 -04:00
Francis Couture-Harpin
bd807499f7
ggml-quants : 1.625 bpw ternary packing for BitNet 1.58b
2024-06-27 02:06:22 -04:00
Raj Hammeer Singh Hada
ac146628e4
Fix llama-android.cpp for error - "common/common.h not found" ( #8145 )
...
- Path seems to be wrong for the common.h header file in llama-android.cpp file. Fixing the path so the Android Build doesn't fail with the error "There is no file common/common.h"
2024-06-27 03:57:57 +02:00
Daniel Bevenius
9b31a40c6d
clip : suppress unused variable warnings ( #8105 )
...
* clip : suppress unused variable warnings
This commit suppresses unused variable warnings for the variables e in
the catch blocks.
The motivation for this change is to suppress the warnings that are
generated on Windows when using the MSVC compiler. The warnings are
not displayed when using GCC because GCC will mark all catch parameters
as used.
Signed-off-by: Daniel Bevenius <[email protected] [email protected]
2024-06-27 01:50:09 +02:00
Georgi Gerganov
c70d117c37
scripts : fix filename sync
2024-06-26 23:25:22 +03:00
slaren
ae5d0f4b89
ci : publish new docker images only when the files change ( #8142 )
2024-06-26 21:59:28 +02:00
slaren
31ec3993f6
ggml : add GGML_CUDA_USE_GRAPHS option, restore GGML_CUDA_FORCE_CUBLAS (cmake) ( #8140 )
2024-06-26 21:34:14 +02:00
slaren
c7ab7b612c
make : fix missing -O3 ( #8143 )
2024-06-26 21:20:22 +03:00
Georgi Gerganov
f2d48fffde
sync : ggml
2024-06-26 19:39:19 +03:00
Georgi Gerganov
4713bf3093
authors : regen
2024-06-26 19:36:44 +03:00
Georgi Gerganov
0e814dfc42
devops : remove clblast + LLAMA_CUDA -> GGML_CUDA ( #8139 )
...
ggml-ci
2024-06-26 19:32:07 +03:00
Georgi Gerganov
a95631ee97
readme : update API notes
2024-06-26 19:26:13 +03:00
Georgi Gerganov and slaren
f3f65429c4
llama : reorganize source code + improve CMake ( #8006 )
...
* scripts : update sync [no ci]
* files : relocate [no ci]
* ci : disable kompute build [no ci]
* cmake : fixes [no ci]
* server : fix mingw build
ggml-ci
* cmake : minor [no ci]
* cmake : link math library [no ci]
* cmake : build normal ggml library (not object library) [no ci]
* cmake : fix kompute build
ggml-ci
* make,cmake : fix LLAMA_CUDA + replace GGML_CDEF_PRIVATE
ggml-ci
* move public backend headers to the public include directory (#8122 )
* move public backend headers to the public include directory
* nix test
* spm : fix metal header
---------
Co-authored-by: Georgi Gerganov <[email protected] [email protected]
2024-06-26 18:33:02 +03:00
Isaac McFadyen and Johannes Gäßler
8854044561
Clarify default MMQ for CUDA and LLAMA_CUDA_FORCE_MMQ flag ( #8115 )
...
* Add message about int8 support
* Add suggestions from review
Co-authored-by: Johannes Gäßler <[email protected] [email protected]
2024-06-26 08:29:28 +02:00
Johannes Gäßler
c8771ab5f8
CUDA: fix misaligned shared memory read ( #8123 )
2024-06-26 08:28:02 +02:00
Eddie-Wang
494165f3b6
llama : extend llm_build_ffn() to support _scale tensors ( #8103 )
2024-06-26 09:27:46 +03:00
Olivier Chafik
9b2f16f805
json: better support for "type" unions (e.g. nullable arrays w/ typed items) (#7863 )
...
* json: better suport for "type" arrays (e.g. `{"type": ["array", "null"], "items": {"type": "string"}}`)
* json: add test for type: [array, null] fix
* update tests
2024-06-26 01:46:35 +01:00
Olivier Chafik
6777c544bd
json: fix additionalProperties, allow space after enum/const (#7840 )
...
* json: default additionalProperty to true
* json: don't force additional props after normal properties!
* json: allow space after enum/const
* json: update pydantic example to set additionalProperties: false
* json: prevent additional props to redefine a typed prop
* port not_strings to python, add trailing space
* fix not_strings & port to js+py
* Update json-schema-to-grammar.cpp
* fix _not_strings for substring overlaps
* json: fix additionalProperties default, uncomment tests
* json: add integ. test case for additionalProperties
* json: nit: simplify condition
* reformat grammar integ tests w/ R"""()""" strings where there's escapes
* update # tokens in server test: consts can now have trailing space
2024-06-26 01:45:58 +01:00
jukofyork
163d50adaf
fixes #7999 (adds control vectors to all build_XXX() functions in llama.cpp [needs testing] ( #8060 )
...
* fixes #7999
The `build_command_r` forgot to add the control vector.
* Fixes qwen2 too
* Fixed all models' control vectors
* Removed double calls to `cb(cur, "l_out", il)`
* Moved control vector logic to llama_control_vector:apply_to()
2024-06-25 22:47:40 +02:00
fairydreaming and Stanisław Szymczyk
6fcbf68235
llama : implement Unigram tokenizer needed by T5 and FLAN-T5 model families ( #5763 )
...
* llama : add T5 model architecture, tensors and model header parameters
* llama : add implementation of Unigram tokenizer with SentencePiece-like text normalization using precompiled charsmap
---------
Co-authored-by: Stanisław Szymczyk <[email protected]
2024-06-25 21:14:35 +02:00
Daniel Bevenius and Clint Herron
e6bf007744
llama : return nullptr from llama_grammar_init ( #8093 )
...
* llama : return nullptr from llama_grammar_init
This commit updates llama_grammar_init to return nullptr instead of
throwing an exception.
The motivation for this is that this function is declared inside an
extern "C" block and is intended/may be used from C code which will not
be able to handle exceptions thrown, and results in undefined behavior.
On Windows and using MSVC the following warning is currently generated:
```console
C:\llama.cpp\llama.cpp(13998,1): warning C4297: 'llama_grammar_init':
function assumed not to throw an exception but does
C:\llama.cpp\llama.cpp(13998,1): message :
__declspec(nothrow), throw(), noexcept(true), or noexcept was specified
on the function
```
Signed-off-by: Daniel Bevenius <[email protected] [email protected] [email protected] [email protected]
2024-06-25 15:07:28 -04:00
Olivier Chafik
84631fe150
json: support integer minimum, maximum, exclusiveMinimum, exclusiveMaximum (#7797 )
...
* json: support minimum for positive integer values
* json: fix min 0
* json: min + max integer constraints
* json: handle negative min / max integer bounds
* json: fix missing paren min/max bug
* json: proper paren fix
* json: integration test for schemas
* json: fix bounds tests
* Update json-schema-to-grammar.cpp
* json: fix negative max
* json: fix negative min (w/ more than 1 digit)
* Update test-grammar-integration.cpp
* json: nit: move string rules together
* json: port min/max integer support to Python & JS
* nit: move + rename _build_min_max_int
* fix min in [1, 9]
* Update test-grammar-integration.cpp
* add C++11-compatible replacement for std::string_view
* add min/max constrained int field to pydantic json schema example
* fix merge
* json: add integration tests for min/max bounds
* reshuffle/merge min/max integ test cases
* nits / cleanups
* defensive code against string out of bounds (apparently different behaviour of libstdc++ vs. clang's libc++, can't read final NULL char w/ former)
2024-06-25 20:06:20 +01:00
slaren
dd047b476c
disable docker CI on pull requests ( #8110 )
2024-06-25 19:20:06 +02:00
joecryptotoo
925c30956d
Add healthchecks to llama-server containers ( #8081 )
...
* added healthcheck
* added healthcheck
* added healthcheck
* added healthcheck
* added healthcheck
* moved curl to base
* moved curl to base
2024-06-25 17:13:27 +02:00
Brian
c8ad35955a
Gguf dump start data offset via --data-offset and some extra refactor ( #8054 )
...
* gguf-dump: add --data-offset
* gguf-dump: add tensor data offset table
* gguf-dump: refactor GGUFReader for clarity
* gguf-dump: add --data-alignment
* gguf-dump.py: Rename variables and adjust comments
start_data_offset --> data_offset
_build_tensors_info_fields --> _build_tensor_info
2024-06-25 22:03:25 +10:00
Xuan Son Nguyen
49c03c79cd
cvector: better prompt handling, add "mean vector" method ( #8069 )
...
* remove completions file
* fix inverted vector
* add mean method
* code style
* remove inverted pca hotfix
2024-06-25 13:59:54 +02:00
Xuan Son Nguyen and Georgi Gerganov
48e6b92cc3
Add chat template support for llama-cli ( #8068 )
...
* add chat template support for llama-cli
* add help message
* server: simplify format_chat
* more consistent naming
* improve
* add llama_chat_format_example
* fix server
* code style
* code style
* Update examples/main/main.cpp
Co-authored-by: Georgi Gerganov <[email protected] [email protected]
2024-06-25 21:56:49 +10:00
HanishKVC
3791ad2193
SimpleChat v3.1: Boolean chat request options in Settings UI, cache_prompt ( #7950 )
...
* SimpleChat: Allow for chat req bool options to be user controlled
* SimpleChat: Allow user to control cache_prompt flag in request
* SimpleChat: Add sample GUI images to readme file
Show the chat screen and the settings screen
* SimpleChat:Readme: Add quickstart block, title to image, cleanup
* SimpleChat: RePosition contents of the Info and Settings UI
Make it more logically structured and flow through.
* SimpleChat: Rename to apiRequestOptions from chatRequestOptions
So that it is not wrongly assumed that these request options are
used only for chat/completions endpoint. Rather these are used
for both the end points, so rename to match semantic better.
* SimpleChat: Update image included with readme wrt settings ui
* SimpleChat:ReadMe: Switch to webp screen image to reduce size
2024-06-25 21:27:35 +10:00
HatsuneMikuUwU33
f702a90e24
Update control vector help ( #8104 )
2024-06-25 10:44:48 +02:00
Meng, Hengyu
083bacce14
[SYCL] Re-enabled mul_mat_batched_sycl ( #8095 )
2024-06-25 10:19:20 +08:00
Johannes Gäßler
2df373ac40
CUDA: fix matrix multiplication algorithm choice ( #8102 )
2024-06-25 01:22:33 +02:00
Johannes Gäßler
3b099bcd9c
CUDA: fix MMQ writeback for int8 tensor cores ( #8100 )
2024-06-24 22:15:33 +02:00
Johannes Gäßler
a818f3028d
CUDA: use MMQ instead of cuBLAS by default ( #8075 )
2024-06-24 17:43:42 +02:00
d62e4aaa02
gguf-py : fix tensor groups for encoder-decoder models in gguf-dump.py ( #8090 )
...
Co-authored-by: Stanisław Szymczyk <[email protected] [email protected]
2024-06-24 14:13:39 +02:00
Johannes Gäßler
9a590c8226
CUDA: optimize MMQ int8 tensor core performance ( #8062 )
...
* CUDA: optimize MMQ int8 tensor core performance
* only a single get_mma_tile_x_k function
* simplify code, make functions constexpr
2024-06-24 12:41:23 +02:00
52fc8705a0
Option to split during conversion ( #6942 )
...
* support splits in convert.py
* Support split by size and dry run to write estimated shards/filesizes
* Move split functionality to new GGUFManager class
* fix improper function signature
* tentative push of convert-hf-to-gguf support
* resolve merge + SplitArguments for easier parsing
* Fix eager tensor memory leak and remove convert.py changes
Removed a memory leak caused by unexpected reference retention to eager tensors.
Also removed GGUFManager functionality in convert.py in favor of specializing for convert-hf-to-gguf.py.
* refactor SplitStrategy to be a deque
Instead of having SplitStrategy have a `data` field that is a deque, just have SplitStrategy be a subclass of deque itself.
* fix Q8 quantization
* remove unnecessary imports in gguf_manager
* fix final? merge issue
* fix gguf_writer placement and remove comments
* oops, actually fix gguf_writer placement
* reduce duplicated code from gguf_writer
* further simplify GGUFManager
* simplify even further and standardize with GGUFWriter
* reduce diffs with master
* form shards while adding tensors, SHA256 sums agree with master
* re-add type hint
Co-authored-by: compilade <[email protected] [email protected] [email protected] [email protected] [email protected] #7827
* kv/ti data are still wrong
* try to refactor kv data (still fails)
* fix ti data messiness
* tidy up
* fix linting
* actually make the linter happy
* cleanup round 1
* remove SplitStrategy, SplitArguments
* appease linter
* fix typing and clean up
* fix linting
* Update gguf-py/gguf/gguf_writer.py
Co-authored-by: compilade <[email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected]
2024-06-24 19:42:03 +10:00
slaren
8cb508d0d5
disable publishing the full-rocm docker image ( #8083 )
2024-06-24 08:36:11 +03:00
Yann Follet and Brian
646ef4a9cf
embedding : more cli arguments ( #7458 )
...
* add parameters for embeddings
--embd-normalize
--embd-output-format
--embd-separator
description in the README.md
* Update README.md
fix tipo
* Trailing whitespace
* fix json generation, use " not '
* fix merge master
* fix code formating
group of parameters // embedding
print usage for embedding parameters
---------
Co-authored-by: Brian <[email protected]
2024-06-24 08:30:24 +03:00
fairydreaming and Stanisław Szymczyk
de0d6a68ac
gguf-py, convert-hf : model conversion support for T5 and FLAN-T5 model variants ( #5763 )
...
* gguf-py : add T5 model architecture
* gguf-py : add separate tensors for encoder and decoder
* gguf-py : add new model header parameters: decoder_start_token_id, attention.relative_buckets_count, tokenizer.ggml.remove_extra_whitespaces, tokenizer.ggml.precompiled_charsmap
* convert-hf : add model conversion support for T5ForConditionalGeneration and T5WithLMHeadModel
---------
Co-authored-by: Stanisław Szymczyk <[email protected]
2024-06-24 07:06:05 +02:00
slaren
95f57bb5d5
ggml : remove ggml_task_type and GGML_PERF ( #8017 )
...
* ggml : remove ggml_task_type and GGML_PERF
* check abort_callback on main thread only
* vulkan : remove usage of ggml_compute_params
* remove LLAMA_PERF
2024-06-24 03:07:59 +02:00
Eddie-Wang and root
e112b610a1
llama : add support for BitnetForCausalLM ( #7931 )
...
* hf bitnet v1
* hf bitnet e2e v2
* finish bitnet e2e
* finish f16 hf bitnet e2e
* remove unsed
* finish bitnet i2 e2e
* move i2s to quantize v1
* move i2 to quantize
* clean code
* clean code 2
* fix codestyle
* fix code
* fix
* fix code
* fix merge
* remove unused
* change table name
* fix whitespace
* delete redundant
* i2_s to absmax
* finish i2_s/i8_s vec_dot x86 simd
* i2s->q22
* fix code
* remove block scale
* add dequantize
* fix seq
* update avx2
* remove q2_2
* remove q22_grid
* fix whitespace
* reuse llm_build_kv
* fix bo
---------
Co-authored-by: root <root@wangjinheng>
2024-06-23 21:27:57 +03:00
Francis Couture-Harpin
Raj Hammeer Singh Hada
Daniel Bevenius
Georgi Gerganov
slaren
Isaac McFadyenandJohannes Gäßler
Eddie-Wang
Olivier Chafik
jukofyork
fairydreamingandStanisław Szymczyk
joecryptotoo
Brian
Xuan Son Nguyen
HanishKVC
HatsuneMikuUwU33
Meng, Hengyu
Yann FolletandBrian
