GitHub - vllm-project/llm-compressor: Transformers-compatible library for applying various compression algorithms to LLMs for optimized deployment with vLLM
llmcompressor is the fast, efficient, and easy-to-use library for optimizing models for deployment with vLLM, including:
Comprehensive set of quantization algorithms and transforms for weight, activation, KV cache, and attention quantization
Seamless integration with Hugging Face models and repositories
Models saved in the compressed-tensors format, compatible with vLLM
DDP and disk offloading support for compressing very large models with hardware efficiency
✨ Read the announcement blog
! ✨
———
📊 Help us improve by taking our
💬 Join us on the
and share your questions, thoughts, or ideas in:
#sig-quantization
#llm-compressor
———
🚀 What's New!
Big updates have landed in LLM Compressor! To get a more in-depth look, check out the
.
Some of the exciting new features include:
NVFP4 Quantized GLM 5.3-Flash: NVFP4 quantized checkpoint for GLM-5.3-Flash. Expert layers have been quantized to NVFP4 and MTP layers have been quantized to FP8 on a per-block basis
Qwen3.8 NVFP4, FP8, and INT4 Quantized Checkpoints: NVFP4 and FP8 quantized checkpoints for Qwen3.8-2.4T-A95B, along with an INT4 checkpoint for Qwen3.8-27B, have been created by the Red Hat AI team. Of particular note, Qwen3.8-2.4T-A95B-NVFP4-REAP-25 combines REAP expert pruning with NVFP4 quantization — 25% of the least-salient experts are pruned prior to quantization, further reducing VRAM requirements while maintaining accuracy recovery. Models:
RedHatAI/Qwen3.8-2.4T-A95B-NVFP4-REAP-25
RedHatAI/Qwen3.8-2.4T-A95B-NVFP4-FP8
RedHatAI/Qwen3.8-2.4T-A95B-NVFP4
RedHatAI/Qwen3.8-2.4T-A95B-FP8
Examples:
Qwen3.8-2.4T-A95B NVFP4+FP8 Example
Qwen3.8-2.4T-A95B REAP + NVFP4 Example
Nemotron 3.5 Lightning FP8 Quantized Checkpoint: An FP8 quantized checkpoint for
has been created by the Red Hat AI team using GPTQ-based FP8 quantization.
RedHatAI/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-FP8
Nemotron 3.5 Lightning FP8 Example
Muse-Glimmer-30B FP8, NVFP4, and INT4 Quantized Checkpoints: FP8, NVFP4, and INT4 checkpoints for
have been created by the Red Hat AI team, enabling single-GPU deployment of this multimodal model.
RedHatAI/Muse-Glimmer-30B-FP8-block
RedHatAI/Muse-Glimmer-30B-NVFP4
RedHatAI/Muse-Glimmer-30B-W4A16
Muse-Glimmer FP8_Block Example
Kimi-K3 NVFP4 and FP8 Quantized Checkpoints: NVFP4 and FP8 quantized checkpoints for Kimi-K3 have been created by the Red Hat AI team.
Hy3 NVFP4+FP8 Quantized Checkpoint: A quantized checkpoint for
has been created by the Red Hat AI team, combining NVFP4 quantization of MoE layers with FP8 quantization of attention layers to significantly reduce VRAM requirements while maintaining accuracy recovery.
GLM-5.2 NVFP4+FP8 Example and Checkpoints: Quantized checkpoints for
have been created by the Red Hat AI team using DDP + disk offloading in under 2 hours. The full precision model requires 1.6T of VRAM, but NVFP4 quantization of MoE layers and FP8 quantization of attention layers reduces the model size by >70% while maintaining state-of-the-art accuracy recovery on GPQA.
REAP Expert Pruning Modifier:
reduces the VRAM requirements to run Mixture-of-Experts models by structurally removing less-relevant experts in each layer. With relevancy proxied by a saliency metric calculated from calibration forward pass data, REAP achieves a desired expert sparsity (set by the user) while aiming to minimize the impact of the pruned experts. The modifier implementation is in
and can be used as a template for implementing other expert pruning algorithms. Examples and additional documentation can be found below:
REAP Prune Qwen/Qwen3-30B-A3B-Instruct-2507 to 25% Sparsity
REAP Prune moonshotai/Moonlight-16B-A3B-Instruct to 25% Sparsity
Supported Precisions and Types
Activation Quantization: W8A8 (int8 and fp8), W4AFP8, Microscale (NVFP4, MXFP4, MXFP8)
Mixed Precision: W4A16, W8A16, MXFP8A16, MXFP4A16, NVFP4A16
Attention and KV Cache Quantization: FP8, NVFP4
Low/Arbitrary-bit Quantization: WNA4, WNA8, WNA16
Supported Algorithms
Simple PTQ
GPTQ
AWQ
SmoothQuant
AutoRound
Rotation-based (SpinQuant, QuIP)
REAP expert pruning
Quantizing your model, step-by-step
Please refer to our
step-by-step compression guide
for detailed information about selecting quantization schemes, algorithms, and their use cases.
Additional information about LLM Compressor functionality is also available in our
and
.
Installation
pip install llmcompressorGet Started
End-to-End Examples
Applying quantization with llmcompressor:
Weight and Activation Quantization
Activation quantization to int8
Activation quantization to fp8
Activation quantization to MXFP8
Activation quantization to fp4 (NVFP4)
Activation quantization to fp4 (MXFP4)
Activation quantization to fp4 using AutoRound
Activation quantization to fp8 and weight quantization to int4
Weight Only Quantization
Weight only quantization to fp4 (NVFP4 format)
Weight only quantization to fp4 (MXFP4 format)
Weight only quantization to int4 using GPTQ
Weight only quantization to int4 using AWQ
Weight only quantization with AutoRound (wNa16)
Attention and KV Cache Quantization
KV Cache quantization to fp8 using per-head
Attention quantization to NVFP4 with SpinQuant (experimental)
Architecture-Specific Quantization
Quantizing Vision-Language Models
Quantizing Audio-Language Models
Non-Uniform Quantization
Quantizing Models Non-uniformly
Big Model Quantization Support
Quantizing large models with sequential onloading
Quantizing large models with disk offloading
Model-Free Definition Quantization
Quantizing models without a Hugging Face model definition
DDP Quantization
Distributed data parallel quantization with GPTQ
Quick Tour
Let's quantize Qwen3-30B-A3B with FP8 weights and activations using the Round-to-Nearest algorithm.
Note that the model can be swapped for a local or remote HF-compatible checkpoint and the recipe may be changed to target different quantization algorithms or formats.
Apply Quantization
Quantization is applied by selecting an algorithm and calling the oneshot API.
fromcompressed_tensors.offloadimportdispatch_modelfromtransformersimportAutoModelForCausalLM, AutoTokenizerfromllmcompressorimportoneshotfromllmcompressor.modifiers.quantizationimportQuantizationModifierMODEL_ID="Qwen/Qwen3-30B-A3B"# Load model.model=AutoModelForCausalLM.from_pretrained(MODEL_ID) tokenizer=AutoTokenizer.from_pretrained(MODEL_ID) # Configure the quantization algorithm and scheme.# In this case, we:# * quantize the weights to FP8 using RTN with block_size 128# * quantize the activations dynamically to FP8 during inferencerecipe=QuantizationModifier( targets="Linear", scheme="FP8_BLOCK", ignore=["lm_head", "re:.*mlp.gate$"], ) # Apply quantization.oneshot(model=model, recipe=recipe) # Confirm generations of the quantized model look sane.print("========== SAMPLE GENERATION ==============") dispatch_model(model) input_ids=tokenizer("Hello my name is", return_tensors="pt").input_ids.to( model.device ) output=model.generate(input_ids, max_new_tokens=20) print(tokenizer.decode(output[0])) print("==========================================") # Save to disk in compressed-tensors format.SAVE_DIR=MODEL_ID.split("/")[1] +"-FP8-BLOCK"model.save_pretrained(SAVE_DIR) tokenizer.save_pretrained(SAVE_DIR)Inference with vLLM
The checkpoints created by llmcompressor can be loaded and run in vllm:
Install:
pip install vllmRun:
fromvllmimportLLMmodel=LLM("Qwen/Qwen3-30B-A3B-FP8-BLOCK") output=model.generate("My name is")Questions / Contribution
If you have any questions or requests open an
and we will add an example or documentation.
We appreciate contributions to the code, examples, integrations, and documentation as well as bug reports and feature requests!
.
Citation
If you find LLM Compressor useful in your research or projects, please consider citing it:
@software{llmcompressor2024, title={{LLM Compressor}}, author={Red Hat AI and vLLM Project}, year={2024}, month={8}, url={https://github.com/vllm-project/llm-compressor}, }