Georgi Gerganov e093dd2382 tests : re-enable tokenizer tests (#8611) ...

* models : remove duplicated gpt-2 vocab

* models : remove old stablelm vocab

* tests : re-enable MPT tokenizer tests

* tests : re-enable DeepSeek tokenizer tests

* cmake : sort

ggml-ci

2024-07-22 13:32:49 +03:00

..

.editorconfig

…

ggml-vocab-aquila.gguf

…

ggml-vocab-baichuan.gguf

…

ggml-vocab-bert-bge.gguf

…

ggml-vocab-bert-bge.gguf.inp

Inference support for T5 and FLAN-T5 model families (#5763)

2024-07-04 15:46:11 +02:00

ggml-vocab-bert-bge.gguf.out

Inference support for T5 and FLAN-T5 model families (#5763)

2024-07-04 15:46:11 +02:00

ggml-vocab-command-r.gguf

…

ggml-vocab-command-r.gguf.inp

Inference support for T5 and FLAN-T5 model families (#5763)

2024-07-04 15:46:11 +02:00

ggml-vocab-command-r.gguf.out

Inference support for T5 and FLAN-T5 model families (#5763)

2024-07-04 15:46:11 +02:00

ggml-vocab-deepseek-coder.gguf

…

ggml-vocab-deepseek-coder.gguf.inp

Inference support for T5 and FLAN-T5 model families (#5763)

2024-07-04 15:46:11 +02:00

ggml-vocab-deepseek-coder.gguf.out

Inference support for T5 and FLAN-T5 model families (#5763)

2024-07-04 15:46:11 +02:00

ggml-vocab-deepseek-llm.gguf

…

ggml-vocab-deepseek-llm.gguf.inp

Inference support for T5 and FLAN-T5 model families (#5763)

2024-07-04 15:46:11 +02:00

ggml-vocab-deepseek-llm.gguf.out

Inference support for T5 and FLAN-T5 model families (#5763)

2024-07-04 15:46:11 +02:00

ggml-vocab-falcon.gguf

…

ggml-vocab-falcon.gguf.inp

Inference support for T5 and FLAN-T5 model families (#5763)

2024-07-04 15:46:11 +02:00

ggml-vocab-falcon.gguf.out

Inference support for T5 and FLAN-T5 model families (#5763)

2024-07-04 15:46:11 +02:00

ggml-vocab-gpt-2.gguf

…

ggml-vocab-gpt-2.gguf.inp

Inference support for T5 and FLAN-T5 model families (#5763)

2024-07-04 15:46:11 +02:00

ggml-vocab-gpt-2.gguf.out

Inference support for T5 and FLAN-T5 model families (#5763)

2024-07-04 15:46:11 +02:00

ggml-vocab-gpt-neox.gguf

…

ggml-vocab-llama-bpe.gguf

…

ggml-vocab-llama-bpe.gguf.inp

Inference support for T5 and FLAN-T5 model families (#5763)

2024-07-04 15:46:11 +02:00

ggml-vocab-llama-bpe.gguf.out

Inference support for T5 and FLAN-T5 model families (#5763)

2024-07-04 15:46:11 +02:00

ggml-vocab-llama-spm.gguf

…

ggml-vocab-llama-spm.gguf.inp

Inference support for T5 and FLAN-T5 model families (#5763)

2024-07-04 15:46:11 +02:00

ggml-vocab-llama-spm.gguf.out

Inference support for T5 and FLAN-T5 model families (#5763)

2024-07-04 15:46:11 +02:00

ggml-vocab-mpt.gguf

…

ggml-vocab-mpt.gguf.inp

Inference support for T5 and FLAN-T5 model families (#5763)

2024-07-04 15:46:11 +02:00

ggml-vocab-mpt.gguf.out

Inference support for T5 and FLAN-T5 model families (#5763)

2024-07-04 15:46:11 +02:00

ggml-vocab-phi-3.gguf

…

ggml-vocab-phi-3.gguf.inp

Inference support for T5 and FLAN-T5 model families (#5763)

2024-07-04 15:46:11 +02:00

ggml-vocab-phi-3.gguf.out

Inference support for T5 and FLAN-T5 model families (#5763)

2024-07-04 15:46:11 +02:00

ggml-vocab-qwen2.gguf

…

ggml-vocab-qwen2.gguf.inp

Inference support for T5 and FLAN-T5 model families (#5763)

2024-07-04 15:46:11 +02:00

ggml-vocab-qwen2.gguf.out

Inference support for T5 and FLAN-T5 model families (#5763)

2024-07-04 15:46:11 +02:00

ggml-vocab-refact.gguf

…

ggml-vocab-refact.gguf.inp

Inference support for T5 and FLAN-T5 model families (#5763)

2024-07-04 15:46:11 +02:00

ggml-vocab-refact.gguf.out

Inference support for T5 and FLAN-T5 model families (#5763)

2024-07-04 15:46:11 +02:00

ggml-vocab-starcoder.gguf

…

ggml-vocab-starcoder.gguf.inp

Inference support for T5 and FLAN-T5 model families (#5763)

2024-07-04 15:46:11 +02:00

ggml-vocab-starcoder.gguf.out

Inference support for T5 and FLAN-T5 model families (#5763)

2024-07-04 15:46:11 +02:00