GitHub - ggml-org/llama.cpp: LLM inference in C/C++

GitHub

Quick start

A few options to get llama.cpp installed on your machine:

Visit

https://llama.app

and follow the instructions

Run with Docker - see our

Docker documentation

Download pre-built binaries from the

releases page

Build from source by cloning this repository - check out

our build guide

Once installed:

# Download and run a model directly from Hugging Face llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF # Launch OpenAI-compatible API server llama serve -hf ggml-org/Qwen3.5-0.8B-GGUFDescription

The main goal of llama.cpp is to enable LLM (and VLM) inference with minimal setup and state-of-the-art performance on a wide range of hardware - locally and in the cloud.

Plain C/C++ implementation without any dependencies

Apple silicon is a first-class citizen - optimized via ARM NEON, Accelerate and Metal frameworks

AVX, AVX2, AVX512 and AMX support for x86 architectures

RVV, ZVFH, ZFH, ZICBOP and ZIHINTPAUSE support for RISC-V architectures

1.5-bit, 2-bit, 3-bit, 4-bit, 5-bit, 6-bit, and 8-bit integer quantization for faster inference and reduced memory use

Custom CUDA kernels for running LLMs on NVIDIA GPUs (support for AMD GPUs via HIP and Moore Threads GPUs via MUSA)

Vulkan and SYCL backend support

CPU+GPU hybrid inference to partially accelerate models larger than the total VRAM capacity

The llama.cpp project is build on top of the

ggml

library.

Supported backends