mirror of
https://github.com/ggml-org/llama.cpp.git
synced 2026-09-04 02:37:27 +02:00
Aleksander Grygier
5b6ddc9675
* ui : rework the settings registry into ordered raw-data sections SETTINGS_REGISTRY becomes an ordered SettingsSectionEntry[] array; the array order is the sidebar display order. Section titles, color mode options and title radio options are declared inline in their section or entry. Entries gain showInUi; MCP servers, the system-message toggle and the title LLM flag become hidden entries of their own section. Derived values (config defaults, help info, chat sections, numeric field lists, syncable parameters) are still derived here; they move to their actual consumers in follow-up commits. * ui : extract settings localStorage persistence into SettingsService Stateless load/save of the settings config and user-override keys, plus the legacy theme key migration. Business logic (default merging, mobile sendOnEnter default, applying the migrated theme) stays in the store. * ui : move the settings exit route into ROUTES SETTINGS_FALLBACK_EXIT_ROUTE is just a route, so it lives with the other routes as ROUTES.SETTINGS_EXIT. * ui : derive the syncable parameter list in the parameter sync service The syncable parameter mapping is only consumed by the sync service, so derive it there from the registry instead of exporting it from the constants file. * ui : restore isPrivate for API key masking * ui : clean up settings registry and router fetch guard Drop the per-entry section field (duplicates the parent slug and is never read) and guard the router model fetch on fields?.length so the Tools/Import-Export pages with empty fields are excluded again. Assisted-by: pi * ui : merge sampling and penalties settings into one section Assisted-by: pi
llama.cpp
LLM inference in C/C++
manifesto / ggml / ops / maintainer PRs / compile times / lib llama API / llama-server REST API
Quick start
A few options to get llama.cpp installed on your machine:
- Visit https://llama.app and follow the instructions
- Run with Docker - see our Docker documentation
- Download pre-built binaries from the releases page
- Build from source by cloning this repository - check out our build guide
Once installed:
# Download and run a model directly from Hugging Face
llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF
# Launch OpenAI-compatible API server
llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF
|
|
|
Description
The main goal of llama.cpp is to enable LLM (and VLM) inference with minimal setup and state-of-the-art performance on
a wide range of hardware - locally and in the cloud.
- Plain C/C++ implementation without any dependencies
- Apple silicon is a first-class citizen - optimized via ARM NEON, Accelerate and Metal frameworks
- AVX, AVX2, AVX512 and AMX support for x86 architectures
- RVV, ZVFH, ZFH, ZICBOP and ZIHINTPAUSE support for RISC-V architectures
- 1.5-bit, 2-bit, 3-bit, 4-bit, 5-bit, 6-bit, and 8-bit integer quantization for faster inference and reduced memory use
- Custom CUDA kernels for running LLMs on NVIDIA GPUs (support for AMD GPUs via HIP and Moore Threads GPUs via MUSA)
- Vulkan and SYCL backend support
- CPU+GPU hybrid inference to partially accelerate models larger than the total VRAM capacity
The llama.cpp project is build on top of the ggml library.
Supported backends
| Backend | Target devices |
|---|---|
| BLAS | All |
| BLIS | All |
| CANN | Ascend NPU |
| CUDA | Nvidia GPU |
| HIP | AMD GPU |
| Hexagon [In Progress] | Snapdragon |
| IBM zDNN | IBM Z & LinuxONE |
| MUSA | Moore Threads GPU |
| Metal | Apple Silicon |
| OpenCL | Adreno GPU |
| OpenVINO [In Progress] | Intel CPUs, GPUs, and NPUs |
| RPC | All |
| SYCL | Intel GPU |
| VirtGPU | VirtGPU APIR |
| Vulkan | GPU |
| WebGPU | All |
| ZenDNN | AMD CPU |
Documentation
Tools
Development
- How to build
- Running on Docker
- Build on Android
- Multi-GPU usage
- Performance troubleshooting
- GGML tips & tricks
- XCFramework
- Completions
- Models
- Release process
Contributing
- Contributors can open PRs
- Collaborators will be invited based on contributions
- Maintainers can push to branches in the
llama.cpprepo and merge PRs into themasterbranch - Any help with managing issues, PRs and projects is very appreciated!
- Read the CONTRIBUTING.md for more information
Acknowledgements
- yhirose/cpp-httplib - Single-header HTTP server, used by
llama-server- MIT license - nothings/stb - Single-header image format decoder, used by multimodal subsystem - Public domain
- nlohmann/json - Single-header JSON library, used by various tools/examples - MIT License
- mackron/miniaudio - Single-header audio format decoder, used by multimodal subsystem - Public domain
- sheredom/subprocess.h - Single-header process launching solution for C and C++ - Public domain