Kashif RasulandSigbjørn Skjæret afcda09d15 vocab : fix HybridDNA tokenizer (#23466) ...

* vocab : mark hybriddna k-mers to avoid BPE token collisions

* improved loop

---------

Co-authored-by: Sigbjørn Skjæret <[email protected]>

2026-05-22 11:17:31 +02:00

..

__init__.py

Refactor: convert_hf_to_gguf.py (#17114)

2026-05-15 15:18:12 +02:00

afmoe.py

Refactor: convert_hf_to_gguf.py (#17114)

2026-05-15 15:18:12 +02:00

arctic.py

Refactor: convert_hf_to_gguf.py (#17114)

2026-05-15 15:18:12 +02:00

baichuan.py

Refactor: convert_hf_to_gguf.py (#17114)

2026-05-15 15:18:12 +02:00

bailingmoe.py

Refactor: convert_hf_to_gguf.py (#17114)

2026-05-15 15:18:12 +02:00

base.py

vocab : fix HybridDNA tokenizer (#23466)

2026-05-22 11:17:31 +02:00

bert.py

Refactor: convert_hf_to_gguf.py (#17114)

2026-05-15 15:18:12 +02:00

bitnet.py

Refactor: convert_hf_to_gguf.py (#17114)

2026-05-15 15:18:12 +02:00

bloom.py

Refactor: convert_hf_to_gguf.py (#17114)

2026-05-15 15:18:12 +02:00

chameleon.py

Refactor: convert_hf_to_gguf.py (#17114)

2026-05-15 15:18:12 +02:00

chatglm.py

Refactor: convert_hf_to_gguf.py (#17114)

2026-05-15 15:18:12 +02:00

codeshell.py

Refactor: convert_hf_to_gguf.py (#17114)

2026-05-15 15:18:12 +02:00

cogvlm.py

Refactor: convert_hf_to_gguf.py (#17114)

2026-05-15 15:18:12 +02:00

command_r.py

Refactor: convert_hf_to_gguf.py (#17114)

2026-05-15 15:18:12 +02:00

dbrx.py

Refactor: convert_hf_to_gguf.py (#17114)

2026-05-15 15:18:12 +02:00

deci.py

Refactor: convert_hf_to_gguf.py (#17114)

2026-05-15 15:18:12 +02:00

deepseek.py

Refactor: convert_hf_to_gguf.py (#17114)

2026-05-15 15:18:12 +02:00

dots1.py

Refactor: convert_hf_to_gguf.py (#17114)

2026-05-15 15:18:12 +02:00

dotsocr.py

Refactor: convert_hf_to_gguf.py (#17114)

2026-05-15 15:18:12 +02:00

dream.py

Refactor: convert_hf_to_gguf.py (#17114)

2026-05-15 15:18:12 +02:00

ernie.py

Refactor: convert_hf_to_gguf.py (#17114)

2026-05-15 15:18:12 +02:00

exaone.py

Refactor: convert_hf_to_gguf.py (#17114)

2026-05-15 15:18:12 +02:00

falcon_h1.py

Refactor: convert_hf_to_gguf.py (#17114)

2026-05-15 15:18:12 +02:00

falcon.py

Refactor: convert_hf_to_gguf.py (#17114)

2026-05-15 15:18:12 +02:00

gemma.py

Refactor: convert_hf_to_gguf.py (#17114)

2026-05-15 15:18:12 +02:00

glm.py

Refactor: convert_hf_to_gguf.py (#17114)

2026-05-15 15:18:12 +02:00

gpt2.py

Refactor: convert_hf_to_gguf.py (#17114)

2026-05-15 15:18:12 +02:00

gpt_oss.py

Refactor: convert_hf_to_gguf.py (#17114)

2026-05-15 15:18:12 +02:00

gptneox.py

Refactor: convert_hf_to_gguf.py (#17114)

2026-05-15 15:18:12 +02:00

granite.py

Refactor: convert_hf_to_gguf.py (#17114)

2026-05-15 15:18:12 +02:00

grok.py

Refactor: convert_hf_to_gguf.py (#17114)

2026-05-15 15:18:12 +02:00

grovemoe.py

Refactor: convert_hf_to_gguf.py (#17114)

2026-05-15 15:18:12 +02:00

hunyuan.py

mtmd, model : merge HunyuanOCR into HunyuanVL and fix OCR vision precision (#23329)

2026-05-21 00:35:37 +02:00

internlm.py

Refactor: convert_hf_to_gguf.py (#17114)

2026-05-15 15:18:12 +02:00

internvl.py

Refactor: convert_hf_to_gguf.py (#17114)

2026-05-15 15:18:12 +02:00

jais.py

Refactor: convert_hf_to_gguf.py (#17114)

2026-05-15 15:18:12 +02:00

jamba.py

Refactor: convert_hf_to_gguf.py (#17114)

2026-05-15 15:18:12 +02:00

januspro.py

Refactor: convert_hf_to_gguf.py (#17114)

2026-05-15 15:18:12 +02:00

kimi_linear.py

Refactor: convert_hf_to_gguf.py (#17114)

2026-05-15 15:18:12 +02:00

kimivl.py

Refactor: convert_hf_to_gguf.py (#17114)

2026-05-15 15:18:12 +02:00

lfm2.py

Refactor: convert_hf_to_gguf.py (#17114)

2026-05-15 15:18:12 +02:00

lighton_ocr.py

Refactor: convert_hf_to_gguf.py (#17114)

2026-05-15 15:18:12 +02:00

llada.py

Refactor: convert_hf_to_gguf.py (#17114)

2026-05-15 15:18:12 +02:00

llama4.py

Refactor: convert_hf_to_gguf.py (#17114)

2026-05-15 15:18:12 +02:00

llama.py

vocab : add Carbon-3B (HybridDNATokenizer) support (#23410)

2026-05-21 08:34:32 +02:00

llava.py

Refactor: convert_hf_to_gguf.py (#17114)

2026-05-15 15:18:12 +02:00

maincoder.py

Refactor: convert_hf_to_gguf.py (#17114)

2026-05-15 15:18:12 +02:00

mamba.py

Refactor: convert_hf_to_gguf.py (#17114)

2026-05-15 15:18:12 +02:00

mimo.py

Refactor: convert_hf_to_gguf.py (#17114)

2026-05-15 15:18:12 +02:00

minicpm.py

Refactor: convert_hf_to_gguf.py (#17114)

2026-05-15 15:18:12 +02:00

minimax.py

Refactor: convert_hf_to_gguf.py (#17114)

2026-05-15 15:18:12 +02:00

mistral3.py

Refactor: convert_hf_to_gguf.py (#17114)

2026-05-15 15:18:12 +02:00

mistral.py

Refactor: convert_hf_to_gguf.py (#17114)

2026-05-15 15:18:12 +02:00

mpt.py

Refactor: convert_hf_to_gguf.py (#17114)

2026-05-15 15:18:12 +02:00

nemotron.py

Refactor: convert_hf_to_gguf.py (#17114)

2026-05-15 15:18:12 +02:00

olmo.py

Refactor: convert_hf_to_gguf.py (#17114)

2026-05-15 15:18:12 +02:00

openelm.py

Refactor: convert_hf_to_gguf.py (#17114)

2026-05-15 15:18:12 +02:00

orion.py

Refactor: convert_hf_to_gguf.py (#17114)

2026-05-15 15:18:12 +02:00

pangu.py

Refactor: convert_hf_to_gguf.py (#17114)

2026-05-15 15:18:12 +02:00

phi.py

Refactor: convert_hf_to_gguf.py (#17114)

2026-05-15 15:18:12 +02:00

pixtral.py

Refactor: convert_hf_to_gguf.py (#17114)

2026-05-15 15:18:12 +02:00

plamo.py

Refactor: convert_hf_to_gguf.py (#17114)

2026-05-15 15:18:12 +02:00

plm.py

Refactor: convert_hf_to_gguf.py (#17114)

2026-05-15 15:18:12 +02:00

qwen3vl.py

convert : fix Qwen3 ASR conversion (#23081)

2026-05-15 18:38:39 +02:00

qwen.py

update bid to match each layers MTP source (#23237)

2026-05-18 12:37:12 +08:00

qwenvl.py

Refactor: convert_hf_to_gguf.py (#17114)

2026-05-15 15:18:12 +02:00

refact.py

Refactor: convert_hf_to_gguf.py (#17114)

2026-05-15 15:18:12 +02:00

rwkv.py

Refactor: convert_hf_to_gguf.py (#17114)

2026-05-15 15:18:12 +02:00

sarashina2.py

Refactor: convert_hf_to_gguf.py (#17114)

2026-05-15 15:18:12 +02:00

smallthinker.py

Refactor: convert_hf_to_gguf.py (#17114)

2026-05-15 15:18:12 +02:00

smolvlm.py

Refactor: convert_hf_to_gguf.py (#17114)

2026-05-15 15:18:12 +02:00

stablelm.py

Refactor: convert_hf_to_gguf.py (#17114)

2026-05-15 15:18:12 +02:00

starcoder.py

Refactor: convert_hf_to_gguf.py (#17114)

2026-05-15 15:18:12 +02:00

step3.py

Refactor: convert_hf_to_gguf.py (#17114)

2026-05-15 15:18:12 +02:00

t5.py

Refactor: convert_hf_to_gguf.py (#17114)

2026-05-15 15:18:12 +02:00

ultravox.py

Refactor: convert_hf_to_gguf.py (#17114)

2026-05-15 15:18:12 +02:00

wavtokenizer.py

Refactor: convert_hf_to_gguf.py (#17114)

2026-05-15 15:18:12 +02:00

xverse.py

Refactor: convert_hf_to_gguf.py (#17114)

2026-05-15 15:18:12 +02:00

youtuvl.py

Refactor: convert_hf_to_gguf.py (#17114)

2026-05-15 15:18:12 +02:00