GitHub - ModelCloud/GPTQModel: LLM model quantization (compression) toolkit with HW acceleration support for Nvidia, AMD, Intel GPU and Intel/AMD/Apple CPU via HF, vLLM, and SGLang.
GPT-QModel ⚡
LLM model quantization (compression) toolkit with hw acceleration support for NVIDIA CUDA, AMD ROCm, Huawei Ascend NPU, Intel XPU, and Intel/AMD/Apple CPUs via HF, vLLM, and SGLang. ✨
Latest News 🗞️🚀
09/09/2026 7.5.0-dev main: 🛠️ Fixed quantized-linear empty-input handling, AWQ higher-rank inputs, and AWQ/ParoQuant CUDA validation and device/stream dispatch (
). Removed full-model scans during per-module finalization (
), fixed GPTQ calibration mask indexing across GPUs (
), and corrected the startup banner to report the package checkout's Git commit (
).
09/08/2026 7.5.0-dev main: ✨ Added k2_horizon quantization support for K2-Horizon dense and MoVA/MoE models (
).
09/07/2026
: 🎉 Added resumable quantization checkpoints, shared-input Hessian deduplication, lm_head and embedding requantization, and updated native GGUF support. Added GLM-5.3-Flash, Apertus 1.5, and XHToken ouro / spark2_5 quantization support, plus quantization, JIT cache, and Triton compatibility fixes.
08/31/2026
: ✨ Added HunyuanOCR, NVIDIA LocateAnything-3B, and Qwen3.8-Flash-Next quantization support. Added tile-misaligned GPTQ/AWQ Marlin support, reduced QQQ packing memory, and improved JIT extension cache reuse.
08/25/2026
: ✨ Added lm_head and embedding quantization lifecycle support; Unlimited-OCR, DeepSeek V3.2, Mage-VL, Muse Glimmer, OLMo 3, and SmolLM3 model support.
08/19/2026
: 🚀🔥⚡ Added the Swordfish Blackwell GPTQ/AWQ kernel, planar GPTQ checkpoint formats, native MPS quantization, and Cohere Compass / A.X-K2 model support, with quantization performance and reliability improvements.
07/25/2026
: ✨ Added quantized embedding inference and Solar Open, Solar Open 2, Intern S2 Preview, Inkling, and Poolside Laguna S 2.1 model support.
07/20/2026
: ✨ Added DeepSeek VL/VL2/OCR2 and Nemotron H Puzzle model support, Windows ExllamaV2 compatibility, and ModelOpt NVFP4 dequantization.
07/03/2026
: ✨ Added LFM2/LFM2-VL, MiniMax M3 VL, Cohere2 MoE, Gemma4 Unified, and text-only multimodal model definitions.
06/08/2026
: ✨ Added Laguna, ERNIE 4.5 VL MoE, Ling 2.6 Flash, Nemotron 3 Nano Omni, GLM4V MoE, Zamba/Zamba2, MiniCPM-V 4.6, DeepSeek V4, MiMo V2, Ovis 2.5/2.6, Intern S1, Nemotron Labs Diffusion, and Hunyuan V1 model support.
04/28/2026
: 🚀 Added Huawei Ascend NPU support through native torch kernels for GPTQ, AWQ, ParoQuant, GGUF, QQQ, and EXL3. CUDA kernels are now JIT-compiled, reducing wheel size and building only the kernels in use; Marlin supports NVIDIA Turing+ GPUs, with new GLM 5/5.1, InternVL Chat, Gemma3n, GLM-OCR, GLM-ASR, and Falcon Mamba model support.
04/02/2026
: 🎉 New quantization methods: ParoQuant, GGUF, FP8, EXL3, and FOEM: First-Order Error Matters. Added PrismML/Bonsai 1bit model quantization (inference only), faster ParoQuant/AWQ kernels, ParoQuant optimization scope control: module (Paro Lite) or layer (Paro reference), plus Gemma4, MiniCPM-O, MiniCPM-V, and GLM4 MoE Lite model support.
03/19/2026
: ✨HF Transformers 5.3.0 support with auto-defusing of fused models via pypi pkg:
. Qwen 3.5 family support added. New fast HF cpu kernels for GPTQ/AWQ added. Experimental INT8 cpu kernel added for GPTQ.
Archived News * 02/10/2026 [5.7.0](
https://github.com/ModelCloud/GPTQModel/releases/tag/v5.7.0
): ✨New `MoE.Routing` config with `Bypass` and `Override` options to allow multiple brute-force MoE routing controls for higher quality quantization of MoE experts. Combined with `FailSafeStrategy`, GPT-QModel now has three separate control settings for efficient MoE expert quantization. `AWQ` `qcfg.zero_point` property has been merged with a unified `sym` symmetry property; `zero_point=True` is now `sym=False`. Fixed `AWQ` `sym=True` packing/inference and quantization compatibility with some Qwen3 models. Exaone 4.0 support. 12/31/2025 5.7.0-dev: ✨New FailSafe config and FailSafeStrategy, auto-enabled by default, to address uneven routing of MoE experts resulting in quantization issues for some MoE modules. Smooth operations are introduced to FailSafeStrategy to reduce the impact of outliers in FailSafe quantization using RTN by default. Different FailSafeStrategy and Smoothers can be selected. Threshold to activate FailSafe can also be customized. New Voxtral and Glm-4v model support, plus audio dataset calibration for Qwen2-Omni. AWQ compatibility fix for GLM 4.5-Air.
12/17/2025
: Fixed uv compatibility. Both uv and pip installs will now show UI progress for external wheel/dependency downloads. Fixed macOS and AWQMarlin kernel loading import regressions. Resolved most multi-arch compile issues on Ubuntu, Arch, RedHat and other distros. Fixed multi-arch build issues and Tritonv2 kernel launch bug on multi-GPUs. Fixed 3-bit Triton GPTQ kernel dequant/inference and license property compatibility issue with latest pip/setuptools.
12/9/2025
: ✨New HF Kernel for CPU optimized for AMX, AVX2 and AVX512. Auto module tree for auto-model support. Added AfMoE and Dots1 model support. Fixed pre-layer pass quantization speed regression. Improved HF Transformers, PEFT and Optimum support for both GPTQ and AWQ. Fixed many AWQ compatibility bugs and regressions.
11/9/2025
: ✨New Intel CPU and XPU hardware-optimized AWQ TorchFusedAWQ kernel. Torch Fused kernels now compatible with torch.compile. Fixed AWQ MoE model compatibility and reduced VRAM usage.
11/3/2025
: ✨MiniMax M2 support with
. New VramStrategy.Balanced quantization property for reduced memory usage for large MoE on multi-3090 (24GB) devices. ✨Marin model. New AWQ Torch reference kernel. Fixed AWQ Marlin kernel for bf16. Fixed GLM 4.5/4.6 MoE missing mtp layers on model save (HF bug). Modular refactor. 🎉AWQ support out of beta with full feature support including multi-GPU quant and MoE VRAM saving. ✨Brumby (attention free) model support. ✨IBM Granite Nano support. New calibration_concat_separator config option.
10/24/2025
: 🎉 Data-parallel quant support for MoE models on multi-GPU using nogil Python. offload_to_disk support enabled by default to massively reduce CPU RAM usage. New Intel and AMD CPU hardware-accelerated TorchFused kernel. Packing stage is now 4x faster and now inlined with quantization. VRAM pressure for large models reduced during quantization. act_group_aware is 16k+ times faster and now the default when desc_act=False for higher quality recovery without inference penalty of desc_act=True. New beta quality AWQ support with full gemm, gemm_fast, marlin kernel support. LFM, Ling, Qwen3 Omni model support. Bitblas kernel updated to support Bitblas 0.1.0.post1 release. Quantization is now faster with reduced VRAM usage. Enhanced logging support with LogBar.
09/16/2025
: hyb_act renamed to act_group_aware. Removed finicky torch import within setup.py. Packing bug fix and prebuilt PyTorch 2.8 wheels.
09/12/2025
: ✨ New Models Support: Qwen3-Next, Apertus, Kimi K2, Klear, FastLLM, Nemotron H. New fail_safeboolean toggle to .quantize() to patch-fix non-activated MoE modules due to highly uneven MoE model training. Fixed LavaQwen2 compatibility. Patch-fixed GIL=0 CUDA error for multi-GPU. Fixed compatibility with autoround + new transformers.
09/04/2025
: ✨ Meituan LongCat Flash Chat, Llama 4, GPT-OSS (BF16), and GLM-4.5-Air support. New experimental mock_quantization config to skip complex computational code paths during quantization to accelerate model quant testing.
08/21/2025
: 🎉 New Group Aware Reordering (GAR) support. New models support: Bytedance Seed-OSS, Baidu Ernie, Huawei PanGu, Gemma3, Xiaomi Mimo, Qwen 3/MoE, Falcon H1, GPT-Neo. Memory leak and multiple model compatibility fixes related to Transformers >= 4.54. Python >= 3.13t free-threading support added with near N x GPU linear scaling for quantization of MoE models and also linear N x CPU core scaling of packing stage. Early access PyTorch 2.8 fused-ops on Intel XPU for up to 50% speedup.
10/17/2025 5.0.0-dev main: 👀: EoRA now multi-GPU compatible. Fixed both quality stability in multi-GPU quantization and VRAM usage. New LFM and Ling models support.
09/30/2025 5.0.0-dev main: 👀: New Data Parallel + Multi-GPU + Python 3.13T (PYTHON_GIL=0) equals 80%+ overall quant time reduction of large MoE models vs v4.2.5.
09/29/2025 5.0.0-dev main: 🎉 New Qwen3 Omni model support. AWQ Marlin kernel integrated + many disk offload, threading, and memory usage fixes.
09/24/2025 5.0.0-dev main: 🎉 Up to 90% CPU memory saving for large MoE models with faster/inline packing! 26% quant time reduction for Qwen3 MoE! AWQ Marlin kernel added. AWQ Gemm loading bug fixes. act_group_aware now faster and auto enabled for GPTQ when desc_act is False for higher quality recovery.
09/19/2025 5.0.0-dev main: 👀 CPU memory saving of ~73.5% during quantization stage with new offload_to_disk quantization config property defaults to True.
09/18/2025 5.0.0-dev main: 🎉 AWQ quantization support! Complete refactor and simplification of model definitions in preparation for future quantization formats.
08/19/2025 4.0.0-dev main: Fixed quantization memory usage due to some models' incorrect application of config.use_cache during inference. Fixed Transformers >= 4.54.0 compatibility which changed layer forward return signature for some models.
08/18/2025 4.0.0-dev main: GPT-Neo model support. Memory leak fix in error capture (stack trace) and fixed lm_head quantization compatibility for many models.
07/31/2025 4.0.0-dev main: New Group Aware Reordering (GAR) support and preliminary PyTorch 2.8 fused-ops for Intel XPU for up to 50% speedup.
07/03/2025 4.0.0-dev main: New Baidu Ernie and Huawei PanGu model support.
07/02/2025 4.0.0-dev main: Gemma3 4B model compatibility fix.
05/29/2025 4.0.0-dev main: Falcon H1 model support. Fixed Transformers 4.52+ compatibility with Qwen 2.5 VL models.
05/19/2025 4.0.0-dev main: Qwen 2.5 Omni model support.
05/05/2025 4.0.0-dev main: Python 3.13t free-threading support added with near N x GPU linear scaling for quantization of MoE models and also linear N x CPU core scaling of packing stage.
04/29/2025 3.1.0-dev (Now 4.) main: Xiaomi Mimo model support. Qwen 3 and 3 MoE model support. New arg for quantize(..., calibration_dataset_min_length=10) to filter out bad calibration data that exists in public dataset (wikitext).
04/13/2025
: 🎉 New experimental v2 quantization option for improved model quantization accuracy validated by GSM8K_PLATINUM
vs original gptq. New Phi4-MultiModal model support. New Nvidia Nemotron-Ultra model support. New Dream model support. New experimental multi-GPU quantization support. Reduced VRAM usage. Faster quantization.
04/2/2025
: New Qwen 2.5 VL model support. New samples log column during quantization to track module activation in MoE models. Loss log column now color-coded to highlight modules that are friendly/resistant to quantization. Progress (per-step) stats during quantization now streamed to log file. Auto bfloat16 dtype loading for models based on model config. Fixed kernel compile for PyTorch/ROCm. Slightly faster quantization and auto-resolve some low-level OOM issues for smaller VRAM GPUs.
03/12/2025
: ✨ New QQQ quantization method and inference support! New Google Gemma 3 zero-day model support. New Alibaba Ovis 2 VL model support. New AMD Instella zero-day model support. New GSM8K Platinum and MMLU-Pro benchmarking support. Peft Lora training with GPT-QModel is now 30%+ faster on all GPU and IPEX devices. Auto-detect MoE modules not activated during quantization due to insufficient calibration data. ROCmsetup.py compatibility fixes. Optimum and Peft compatibility fixes. Fixed Peftbfloat16 training.
03/03/2025
: 🎉 GPTQ quantization internals are now broken into multiple stages (processes) for feature expansion. Synced Marlin kernel inference quality fix from upstream. Added reduced-precision Marlin accumulation mode via environment control (GPTQMODEL_MARLIN_USE_FP32=0 disables it, default is enabled). ModelScope support added. Logging and CLI progress bar output has been revamped with sticky bottom progress. Fixed generation_config.json save and load. Fixed Transformers v4.49.0 compatibility. Fixed compatibility of models without bos. Fixed group_size=-1 and bits=3 packing regression. Fixed Qwen 2.5 MoE regressions. Added CI tests to track regression in kernel inference quality and sweep all bits/group_sizes. Delegate logging/progress bar to
package. Fixed ROCm version auto-detection in setup install.
02/12/2025
: ⚡ Offload tokenizer fixes to
package. Optimized lm_head quant time and VRAM usage. Optimized DeepSeek v3/R1 model quant VRAM usage. Fixed Optimum compatibility regression in v1.8.1. 3x speed-up for Torch kernel when using PyTorch >= 2.5.0 with model.optimize(). New calibration_dataset_concat_size option to enable calibration data concat mode to mimic original GPTQ data packing strategy which may improve quant speed and accuracy for datasets like wikitext2.
02/08/2025
: ⚡ DeepSeek v3/R1 model support. New flexible weight packing: allow quantized weights to be packed to [int32, int16, int8] dtypes. Triton and Torch kernels support full range of new QuantizeConfig.pack_dtype. New auto_gc: bool control in quantize() which can reduce quantization time for small model with no chance of OOM. New buffered_fwd: bool control in model.quantize(). Over 50% quantization speed-up for visual (vl) models.
Fixed bits=3 packing and group_size=-1 regression in v1.7.4.
01/26/2025
: New compile() API for ~4-8% inference TPS improvement. Faster pack() for post-quantization model save. Triton kernel validated for Intel/XPU when Intel Triton packages are installed. Fixed Transformers (bug) downcasting tokenizer class on save.
01/20/2025
: New Telechat2 (China Telecom) and PhiMoE model support. Fixed lm_head weights duplicated in post-quantize save() for models with tied-embedding.
01/19/2025
: Effective BPW (bits per weight) will now be logged during load(). Reduce loading time on Intel Arc A770/B580 XPU by 3.3x. Reduce memory usage in MLX conversion and fix Marlin kernel auto-select not checking CUDA compute version.
01/17/2025
: 👀 ✨ backend.MLX added for runtime-conversion and execution of GPTQ models on Apple's MLX framework on Apple Silicon (M1+). ✨ lm_head quantization now fully supported by GPT-QModel without external pkg dependency.
01/07/2025
: 🎉 New OpenAI API compatible endpoint via model.serve(host, port). Auto-enable flash-attention2 for inference. Fixed sym=False loading regression.
01/06/2025
: ⚡25% faster quantization. 35% reduction in VRAM usage vs v1.5. 👀 AMD ROCm (6.2+) support added and validated for 7900XT+ GPU. Auto-tokenizer loader via load() API. For most models you no longer need to manually init a tokenizer for both inference and quantization.
01/01/2025
: 🎉 2025! Added QuantizeConfig.device to clearly define which device is used for quantization: default = auto. Non-quantized models are always loaded on CPU by-default and each layer is moved to QuantizeConfig.device during quantization to minimize VRAM usage. Compatibility fixes for attn_implementation_autoset in latest transformers.
12/23/2024
: Multi-modal (image-to-text) optimized quantization support has been added for Qwen 2-VL and Ovis 1.6-VL. Previous image-to-text model quantizations did not use image calibration data, resulting in less than optimal post-quantization results. Version 1.5.0 is the first release to provide a stable path for multi-modal quantization: only text layers are quantized.
12/19/2024
: Windows 11 support added/validated. Ovis VL model support with image dataset calibration. Fixed dynamic loading. Reduced quantization VRAM usage.
12/15/2024
: macOS GPU (Metal) and CPU (M+) support added/validated for inference and quantization. Cohere 2 model support added.
12/13/2024
: Added Qwen2-VL model support. mse quantization control exposed in QuantizeConfig. Monkey patch patch_vllm() and patch_hf() API added to allow Transformers/Optimum/PEFT and vLLM to correctly load GPT-QModel quantized models while upstream PRs are in pending status.
12/10/2024
EvalPlus harness integration merged upstream. We now support both the legacy evaluation harness and EvalPlus. Added pure torch Torch kernel. Refactored Cuda kernel to be DynamicCuda kernel. Triton kernel now auto-padded for max model support. Dynamic quantization now supports both positive +::default, and -: negative matching which allows matched modules to be skipped entirely for quantization. Fixed auto-Marlin kernel selection. Added auto-kernel fallback for unsupported kernel/module pairs. Lots of internal refactor and cleanup in preparation for transformers/optimum/peft upstream PR merge. Deprecated the saving of Marlin weight format since Marlin supports auto conversion of gptq format to Marlin during runtime.
11/29/2024
Olmo2 model support. Intel XPU acceleration via IPEX. Model sharding Transformer compatibility fix due to API deprecation in HF. Removed triton dependency. Triton kernel now optionally dependent on triton package.
11/26/2024
Zero-Day Hymba model support. Removed tqdm and rogue dependency.
11/24/2024
HF GLM model support. ClearML logging integration. Use device-smi and replace gputil + psutil dependencies. Fixed model unit tests.
11/11/2024 🚀
Meta MobileLLM model support added. legacy evaluation integration merged upstream. Intel/IPEX CPU inference merged replacing QBits (deprecated). Auto-fix/patch ChatGLM-3/GLM-4 compatibility with latest transformers. New .load() and .save() API.
10/29/2024 🚀
IBM Granite model support. Full auto-buildless wheel install from PyPI. Reduce max CPU memory usage by >20% during quantization. 100% CI model/feature coverage.
10/12/2024 ✨
Move AutoRound to optional and fix pip install regression in v1.0.8.
10/11/2024 ✨
Add wheel for Python 3.12 and CUDA 11.8.
10/08/2024 ✨
Fixed Marlin (faster) kernel was not auto-selected for some models.
09/26/2024 ✨
Fixed Llama 3.2 vision quantized loader.
09/26/2024 ✨
Partial Llama 3.2 Vision model support (mllama): only text-layer quantization layers are supported for now.
09/26/2024 ✨
Integrated Liger Kernel support for ~1/2 memory reduction on some models during quantization. Added control toggle to disable parallel packing.
09/18/2024 ✨
Added Microsoft GRIN-MoE and MiniCPM3 support.
08/16/2024 ✨
Support Intel/AutoRound v0.3, prebuilt whl packages, and PyPI release.
08/14/2024 ✨
40% faster packing, fixed Python 3.9 compatibility, added evaluation API.
08/10/2024 🚀
Added LG EXAONE 3.0 model support. New dynamic per layer/module flexible quantization where each layer/module may have different bits/params. Added proper sharding support to backend.BITBLAS. Auto-heal quantization errors due to small damp values.
07/31/2024 🚀
Ported vllm/nm gptq_marlin inference kernel with expanded bits (8bits), group_size (64,32), and desc_act support for all GPTQ models with FORMAT.GPTQ. Auto-calculate auto-round nsamples/seglen parameters based on calibration dataset. Fixed save_quantized() called on pre-quantized models with non-supported backends. HF transformers dependency updated to ensure Llama 3.1 fixes are correctly applied to both quant and inference.
07/25/2024 🚀
: Added Llama-3.1 support, Gemma2 27B quant inference support via vLLM, auto pad_token normalization, fixed auto-round quant compatibility for vLLM/SGLang, and more.
07/13/2024 🚀
: Run quantized models directly using GPT-QModel with fast vLLM or SGLang backend! Both vLLM and SGLang are optimized for dynamic batching inference for maximum TPS (check usage under examples). Marlin backend also got full end-to-end in/out features padding to enhance current/future model compatibility.
07/08/2024 🚀
: InternLM 2.5 model support added.
07/08/2024 🚀
:
QUANT_METHOD support added for a potentially higher quality quantization with lm_head module quantization support for even more VRAM reduction: format export to FORMAT.GPTQ for max inference compatibility.
07/05/2024 🚀
: CUDA kernels have been fully deprecated in favor of Exllama(v1/v2)/Marlin/Triton.
07/03/2024 🚀
: HF Transformers integration added and bug fixed Gemma 2 support.
07/02/2024 🚀
: Added Gemma 2 support, faster quality/benchmark calculations on GPU, and more code/arg refactor.
06/30/2024 🚀
: Added auto-padding of model in/out-features for exllama and exllama v2. Fixed quantization of OPT and DeepSeek V2-Lite models. Fixed inference for DeepSeek V2-Lite.
06/29/2024 🚀
: With 3 new models (DeepSeek-V2, DeepSeek-V2-Lite, DBRX Converted), BITBLAS new format/kernel, proper batching of calibration dataset resulting > 50% quantization speedup, security hash check of loaded model weights, tons of refactor/usability improvements, bug fixes, and much more.
06/20/2924 ✨
: Thanks for all the work from ModelCloud team and the open-source ML community for their contributions!
Special Notes 📝
PrismAI/Bonsai inference sample script. GPT-QModel loads Prism/Bonsai GGUF checkpoints through its native GGUF loading path and internal GGUF runtime shim. No external gguf PyPI package is required.
fromgptqmodelimportGPTQModelmodel=GPTQModel.load("prism-ml/Bonsai-1.7B-gguf") # or: model = GPTQModel.load("prism-ml/Bonsai-1.7B-gguf", profile="low_memory")tokens=model.generate( "Who wrote Romeo and Juliet?", max_new_tokens=128, )[0] print(model.tokenizer.decode(tokens, skip_special_tokens=True))What is GPT-QModel? 🤔
GPT-QModel is a production-ready LLM model compression/quantization toolkit with hw-accelerated inference support for both CPU/GPU via HF Transformers, vLLM, and SGLang.
GPT-QModel currently supports GPTQ, AWQ, ParoQuant, QQQ, GGUF, FP8, EXL3, GPTAQ, EoRA, GAR and FOEM, with more quantization methods and enhancements planned.
GPT-QModel is a modular design supporting multiple quantization methods and feature extensions.
FeatureGPT-QModelTransformersvLLMSGLangLora TrainingGPTQ✅✅✅✅✅AWQ✅✅✅✅✅ParoQuant✅xxx✅GGUF✅xxxxFP8✅xxxxExllama V3 / EXL3✅xxxxEoRA✅✅✅✅xGroup Aware Act Reordering✅✅✅✅✅QQQ✅xxxxRotation✅xxxxGPTAQ✅✅✅✅✅FOEM✅✅✅✅✅GGUF, FP8, EXL3, and ParoQuant are currently native GPT-QModel quantization/runtime paths. vLLM and SGLang integration currently targets GPTQ and AWQ.
Quant Method / Format / Backend Matrix 📋
Canonical backend names are shown below. Method-specific aliases are only accepted where explicitly implemented by that quant method.
Quant MethodFormatsBackends / KernelsMETHOD.GPTQFORMAT.GPTQ, FORMAT.GPTQ_V2, FORMAT.MARLIN, FORMAT.BITBLASFORMAT.GPTQ: BACKEND.GPTQ_TORCH_ATEN, BACKEND.GPTQ_MACHETE, BACKEND.GPTQ_MARLIN, BACKEND.GPTQ_EXLLAMA_V2, BACKEND.GPTQ_TORCH_FUSED, BACKEND.GPTQ_TRITON, BACKEND.GPTQ_BITBLAS, BACKEND.GPTQ_TORCH, BACKEND.GPTQ_TORCH_INT8
FORMAT.GPTQ_V2: BACKEND.GPTQ_TORCH_ATEN, BACKEND.GPTQ_EXLLAMA_V2, BACKEND.GPTQ_TORCH_FUSED, BACKEND.GPTQ_TRITON, BACKEND.GPTQ_BITBLAS, BACKEND.GPTQ_TORCH, BACKEND.GPTQ_TORCH_INT8
FORMAT.MARLIN: BACKEND.GPTQ_MARLIN
FORMAT.BITBLAS: BACKEND.GPTQ_BITBLASMETHOD.AWQFORMAT.GEMM, FORMAT.GEMV, FORMAT.GEMV_FAST, FORMAT.LLM_AWQ, FORMAT.MARLIN, FORMAT.BITBLASFORMAT.GEMM: BACKEND.AWQ_TORCH_ATEN, BACKEND.AWQ_MACHETE, BACKEND.AWQ_MARLIN, BACKEND.AWQ_EXLLAMA_V2, BACKEND.AWQ_GEMM, BACKEND.AWQ_GEMM_TRITON, BACKEND.AWQ_TORCH_FUSED, BACKEND.AWQ_TORCH, BACKEND.AWQ_TORCH_INT8, BACKEND.AWQ_BITBLAS
FORMAT.GEMV: BACKEND.AWQ_GEMV
FORMAT.GEMV_FAST: BACKEND.AWQ_GEMV_FAST
FORMAT.LLM_AWQ: BACKEND.AWQ_GEMV_FAST
FORMAT.MARLIN: BACKEND.AWQ_MACHETE, BACKEND.AWQ_MARLIN
FORMAT.BITBLAS: BACKEND.AWQ_BITBLASMETHOD.PAROFORMAT.PAROQUANTBACKEND.PAROQUANT_CUDA, BACKEND.PAROQUANT_TRITONMETHOD.QQQFORMAT.QQQBACKEND.QQQ, BACKEND.QQQ_TORCHMETHOD.GGUFFORMAT.GGUFBACKEND.GGUF_TRITON, BACKEND.GGUF_CPP_CUDA, BACKEND.GGUF_CPP_CPU, BACKEND.GGUF_TORCHMETHOD.FP8FORMAT.FP8BACKEND.FP8_TORCHMETHOD.BITSANDBYTESFORMAT.BITSANDBYTESBACKEND.BITSANDBYTESMETHOD.EXL3FORMAT.EXL3BACKEND.EXL3_EXLLAMA_V3, BACKEND.EXL3_TORCHBACKEND.VLLM, BACKEND.SGLANG, and BACKEND.MLX are external runtime backends and are not part of the native kernel matrix above.
Marlin uses GPTQMODEL_MARLIN_USE_FP32 (default: enabled) to control fp32 accumulation.
Features ✨
✨ Native integration with HF
,
, and
🚀
and
inference integration for quantized models with format = FORMAT.[GPTQ/AWQ]
✨ GPTQ, AWQ, ParoQuant, QQQ, GGUF, FP8, EXL3, GPTAQ, and FOEM quantization support.
✨ Current GGUF tensor assignments are supported, with native quantization and dequantization for Q1_0, Q2_0, TQ1_0, TQ2_0, and MXFP4, plus native NVFP4 dequantization. Prism Bonsai Q1_0_g128 remains accepted as a compatibility alias for the official 128-element Q1_0 layout.
🚀 Quantize MoE models with ease even with extreme routing activation bias via Moe.Routing and/or FailSafe.
🚀 Data Parallelism for 80%+ quantization speed reduction with Multi-GPU.
🚀 Optimized for Python >= 3.13t (free threading) with lock-free threading.
✨ Linux, macOS, Windows platform support for CUDA (NVIDIA), NPU (Huawei Ascend), XPU (Intel), ROCm (AMD), MPS (Apple Silicon), CPU (Intel/AMD/Apple Silicon).
✨ Dynamic per-module mixed quantization control: each layer/module can have a unique quantization config or be excluded from quantization.
🚀 Intel Torch 2.8 fused kernel support for XPU [Arc + Datacenter Max] and CPU [avx, amx].
🚀 Python 3.13.3t (free-threading, GIL disabled) support for multi-GPU accelerated quantization for MoE models and multi-core CPU boost for packing.
✨ Asymmetric Sym=False support.
✨ lm_head module quant inference support for further VRAM reduction.
🚀
optimized tile based inference.
💯 CI unit-test coverage for all supported models and kernels including post-quantization quality regression.
Who's Using GPT-QModel? 🌐
Selected public references where teams or companies explicitly mention GPT-QModel in documentation, integration notes, or quantized model usage. This is not an exhaustive customer list.
Hugging Face
Intel
NVIDIA
Alibaba Cloud
Quality: GPTQ 4bit can match native BF16 🏆
🤗
ModelCloud quantized Vortex models on HF
Model Support 🤖
The table mirrors every explicit registration in gptqmodel.models.auto.MODEL_MAP, including text-model and backward-compatibility aliases. Qwen 3.5 registrations require Transformers 5.2.0 or newer.
Model familyRegistered Transformers model_type valuesA.X-K2axk2AfMoE / TrinityafmoeApertus 1 / 1.5apertus, apertus1p5, apertus1p5_textBaichuanbaichuanBailing MoE / Hybrid (LING / RING)bailing_moe, bailing_hybridBloombloomBrumbybrumbyChatGLMchatglmCodeGencodegenCohere 1 / 2 / 2 MoE / Compass (North Mini / Micro Vision)cohere, cohere2, cohere2_moe, cohere_compassDBRX / DBRX Converteddbrx, dbrx_convertedDeciLMdeciDeepSeek V2 / V3 / V3.2 / V4 / VL / VL2 / OCR2deepseek_v2, deepseek_v3, deepseek_v32, deepseek_v4, deepseek_vl, deepseek_vl_v2, deepseek_ocr2Dots1dots1DreamdreamERNIE 4.5 / MoE / VL MoEernie4_5, ernie4_5_moe, ernie4_5_moe_vl, ernie4_5_vl_moeEXAONE 3 / 4exaone, exaone4Falcon / Falcon H1 / Falcon Mambafalcon, falcon_h1, falcon_mambaGemma 1-4 / 3n / Unifiedgemma, gemma2, gemma3, gemma3_text, gemma3n, gemma3n_text, gemma4, gemma4_text, gemma4_unified, gemma4_unified_textGLM / GLM4 / GLM4V / GLM5 / OCR / ASRglm, glm4, glm4_moe, glm4_moe_lite, glm4v, glm4v_moe, glm4v_moe_text, glm5_next, glm_moe_dsa, glm_ocr, glmasrGPT-2gpt2GPT BigCodegpt_bigcodeGPT-Neo / GPT-NeoXgpt_neo, gpt_neoxGPT-OSSgpt_ossGPT-JgptjGranite / Granite MoE Hybridgranite, granitemoehybridGRIN-MoEgrinmoeHRMhrm_textHunyuan V1 / VL / OCRhunyuan_v1_dense, hunyuan_v1_moe, hunyuan_vlHY-V3hy_v3HymbahymbaInklinginkling_mm_modelInstellainstellaIntern S1 / S2 Previewinterns1, intern_s2_previewInternLM 1 / 2 / 2.5internlm, internlm2InternVL Chatinternvl_chatK2-Horizon (Dense / MoVA)Kimi K2 / K2.5kimi_k2, kimi_k25KlearklearLagunalagunaLFM2 / LFM2 MoE / LFM2-VLlfm2, lfm2_moe, lfm2_vlLLaDA2 MoEllada2_moeLlama 1-4 / TinyLlama / Nemotron Ultrallama, llama4, llama4_textLlama 3.2 VLmllama, mllama_text_modelFastVLM / LLaVA-Qwen2llava_qwen2LocateAnythinglocateanythingLongCat Flashlongcat_flashLongLLaMAlongllamaMage-VLmage_vlMarinmarinMiMo / MiMo V2mimo, mimo_v2MiniCPM / MiniCPM3 / MiniCPM-O / MiniCPM-Vminicpm, minicpm3, minicpmo, minicpmv, minicpmv4_6MiniMax M2 / M3-VLminimax, minimax_m2, minimax_m3_vlMistral / Mistral3 / Ministral3mistral, mistral3, ministral3MixtralmixtralMobileLLMmobilellmMOSSmossMPTmptMuse Glimmermuse_glimmerNemotron NAS / H / H Puzzle / Omni / Labs Diffusionnemotron-nas, nemotron_h, nemotron_h_puzzle, nemotronh_nano_omni_reasoning_v3, nemotron_labs_diffusionOLMo 2 / 3olmo2, olmo3OPToptOuroouroOvis 1.6 / 2 / 2.5 / 2.6 MoE / 2.6 Nextovis, ovis2, ovis2_5, ovis2_6_moe, ovis2_6_nextPanGu-αgpt_panguPhi 1-4 / Phi MoEphi, phi3, phi4mm, phimoeQwen 1-4 / 3.5 / 3.6 / 3.8 / MoE / Nextqwen, qwen2, qwen2_moe, qwen3, qwen3_moe, qwen3_next, qwen3_5, qwen3_5_text, qwen3_5_moe, qwen3_5_moe_text, qwen4_expQwen 2 / 2.5 / 3 VLqwen2_vl, qwen2_vl_text, qwen2_5_vl, qwen2_5_vl_text, qwen3_vlQwen 2.5 / 3 Omniqwen2_5_omni, qwen3_omni_moeRefinedWebrefinedWeb, refinedWebModelSeed-OSSseed_ossSmolLM3smollm3Solar Open / Open 2solar_open, solar_open2Spark 2.5spark2_5StableLMstablelm, stablelm_epochStarCoder2starcoder2TeleChat2telechatUnlimited-OCRunlimited-ocrVoxtralvoxtralXVERSExverseYiyiZamba / Zamba2zamba, zamba2Prism Bonsai GGUF checkpoints are supported for inference only through GPT-QModel's native GGUF path and internal GGUF runtime. Bonsai checkpoints load through the normal model path or repo argument and do not require the external gguf package. Prism model quantization is not included.
Platform and HW Support 🖥️
GPT-QModel is validated on Linux, macOS, and Windows 11:
PlatformDeviceOptimized ArchKernels🐧 LinuxNVIDIA GPU✅Turing+ (sm_75+)Machete, Marlin, Exllama V3 / EXL3, Exllama V2, AWQ GEMM/GEMV, ParoQuant CUDA/Triton, GGUF CUDA/Triton, QQQ, BitBLAS, Triton, BitsAndBytes, Torch🐧 LinuxAMD GPU✅7900XT+, ROCm 6.2+Exllama V2, AWQ GEMM/GEMV, QQQ, FP8 Torch, Torch🐧 LinuxHuawei Ascend NPU✅Ascend 910B, torch-npu / CANNNative Torch kernels for GPTQ, AWQ, ParoQuant, GGUF, QQQ, and EXL3🐧 LinuxIntel XPU✅Arc, Datacenter MaxTorchFused, TorchFusedAWQ, FP8 Torch, Torch🐧 LinuxIntel/AMD CPU✅avx, amxTorchFused, TorchFusedAWQ, TorchAten int4, TorchInt8, GGUF C++, BitsAndBytes, Torch🍎 macOSGPU (Metal) / CPU✅Apple Silicon, M1+Torch, FP8 Torch, MLX via conversion🪟 WindowsGPU (NVIDIA) / CPU✅NVIDIATorchMarlin and JIT CUDA kernels now support NVIDIA Turing+ (sm_75+) GPUs. Huawei Ascend NPU support uses native Torch kernels through torch-npu / CANN.
Install 💾
PIP/UV 💿
# You can install optional modules like autoround, ipex, vllm, sglang, bitblas.# Example: pip install -v gptqmodel[vllm,sglang,bitblas] pip install -v gptqmodel uv pip install -v gptqmodelThe package depends on ninja for first-use JIT kernel compilation.
Install from source 🛠️
# clone repo git clone https://github.com/ModelCloud/GPTQModel.git &&cd GPTQModel # python3-dev is required for some source installs apt install python3-dev # pip: install from source# You can install optional modules like vllm, sglang, bitblas.# Example: pip install -v .[vllm,sglang,bitblas] pip install -v .Inference 🔮
Three-line API to use GPT-QModel for GPTQ model inference:
fromgptqmodelimportGPTQModelmodel=GPTQModel.load("ModelCloud/Llama-3.2-1B-Instruct-gptqmodel-4bit-vortex-v2.5") result=model.generate("Uncovering deep insights begins with")[0] # tokensprint(model.tokenizer.decode(result)) # string outputTo use models from
instead of HuggingFace Hub, set an environment variable:
export GPTQMODEL_USE_MODELSCOPE=TrueFP32 accumulation toggle 🔢
Some AWQ and ParoQuant CUDA/Triton kernels support an fp32 accumulation mode to reduce numerical drift during fused quantized matmul. This setting defaults to True because accuracy is prioritized over speed.
# default behavior: higher accuracy, slightly lower speed on some kernelsexport GPTQMODEL_FP32_ACCUM=1 # optional speed-first mode for some kernelsexport GPTQMODEL_FP32_ACCUM=0JIT kernel cache and multi-process quantization ⚙️
JIT-compiled kernels are cached at ~/.cache/gptqmodel/torch_extensions by default. Multiple processes on one host may safely share the cache: builds are serialized with a cross-process file lock that the OS releases automatically if a process dies.
Machete's pinned CUTLASS checkout and generated CUDA sources are kept in the versioned user cache (~/.cache/gptqmodel, or GPTQMODEL_CACHE_DIR; when set, XDG_CACHE_HOME/gptqmodel is used). Set GPTQMODEL_CUTLASS_DIR to use an already-installed, read-only CUTLASS 4.7.1 checkout. GPTQMODEL_OFFLINE=1 disables downloads and requires that checkout or a verified cache hit already exist. For environments where compilation is not allowed, point GPTQMODEL_MACHETE_PRECOMPILED_LIBRARY at a compatible Machete shared library; an invalid or missing explicit library is reported as an error and does not fall back to JIT. The generated source cache can be populated ahead of time by prewarming the extension:
python -c "from gptqmodel import extension; extension.load('machete')"# optional: relocate the kernel cache (e.g. one cache per process)export GPTQMODEL_TORCH_EXTENSIONS_DIR=/path/to/cache # optional: max seconds to wait for another process's in-flight build before# falling back to non-JIT paths. Default: 600 or 5x the kernel's compile# baseline, whichever is larger.export GPTQMODEL_TORCH_OPS_LOCK_TIMEOUT=600Notes:
This is a runtime toggle. It does not change model weights or saved checkpoints.
It mainly affects some fused AWQ and ParoQuant CUDA/Triton kernels. Dense/dequantize fallback paths are mostly unaffected.
1 is recommended for regression testing and quality-sensitive evaluation. 0 may be useful when chasing a small latency win and the quality tradeoff is acceptable.
OpenAI API compatible endpoint 🌐
# load model using above inference guide firstmodel.serve(host="0.0.0.0",port="12345")Quantization 🔧
Basic example of using GPT-QModel to quantize an LLM model:
fromdatasetsimportload_datasetfromgptqmodelimportGPTQConfig, GPTQModelmodel_id="meta-llama/Llama-3.2-1B-Instruct"quant_path="Llama-3.2-1B-Instruct-gptqmodel-4bit"calibration_dataset=load_dataset( "allenai/c4", data_files="en/c4-train.00001-of-01024.json.gz", split="train" ).select(range(1024))["text"] quant_config=GPTQConfig(bits=4, group_size=128) model=GPTQModel.load(model_id, quant_config) # increase `batch_size` to match GPU/VRAM specs to speed up quantizationmodel.quantize(calibration_dataset, batch_size=1) model.save(quant_path)Checkpoint and Resume Long Quantization Jobs 💾
Checkpointing commits completed transformer layers so an interrupted quantization job can reload the original model and continue without repeating those layers. Use an explicit checkpoint path that survives process restarts:
fromgptqmodelimportCheckpointConfig, CheckpointStopped, GPTQConfig, GPTQModelquant_config=GPTQConfig( bits=4, group_size=128, offload_to_disk=True, ) model=GPTQModel.load(model_id, quant_config) try: model.quantize( calibration_dataset, batch_size=1, checkpoint=CheckpointConfig( path="checkpoints/Llama-3.2-1B-Instruct-gptqmodel-4bit", resume="auto", interval="layer:1", keep_last=2, ), ) exceptCheckpointStopped: # Run the script again with the same source, calibration, config, and path.raiseSystemExit(75) model.save(quant_path)Ctrl+C and SIGTERM finish the current layer, publish a safe checkpoint, and raise CheckpointStopped. A hard kill resumes from the last published layer; the unfinished layer is repeated. Resume validation requires the same source weights, calibration data, quantization settings, software versions, and device topology. Current checkpoint support is limited to Llama and Qwen3 MoE model types, requires offload_to_disk=True, and does not yet support dynamic exclusions, rotation, or embedding/lm_head quantization.
See
Quantization checkpointing and resume
for resume policies, supported methods, safe-stop behavior, storage requirements, recovery procedures, and operational gotchas.
Other Quantization Formats 📦
QuantizeConfig remains the broad factory. The concrete config classes are now GPTQConfig, AWQConfig, ParoConfig, QQQConfig, RTNConfig, GGUFConfig, FP8Config, BitsAndBytesConfig, and EXL3Config.
GPTQ, AWQ, ParoQuant, and EXL3 are calibration-based. GGUF and FP8 are weight-only and should be quantized with calibration=None.
Preprocessors 🧹preprocessors=[...] adds optional module-weight preparation steps before quantization or repacking. They are available on GPTQConfig, AWQConfig, ParoConfig, RTNConfig, GGUFConfig, FP8Config, and BitsAndBytesConfig.
SmootherConfig: apply weight smoothing before quantization.
AutoModuleDecoderConfig: decode FP8/FP4 source modules to a dense target_dtype before downstream quantization or repacking.
TensorParallelPadderConfig: opt-in tensor-parallel padding metadata for TP-aligned packing.
importtorchfromgptqmodelimportGGUFConfig, GPTQConfigfromgptqmodel.quantizationimport ( AutoModuleDecoderConfig, SmoothMAD, SmootherConfig, TensorParallelPadderConfig, ) gptq_cfg=GPTQConfig( bits=4, group_size=128, preprocessors=[ SmootherConfig(smooth=SmoothMAD(k=2.0)), AutoModuleDecoderConfig(target_dtype=torch.bfloat16), TensorParallelPadderConfig(), ], ) gguf_cfg=GGUFConfig( bits=4, format="q_k_m", preprocessors=[ AutoModuleDecoderConfig(target_dtype=torch.bfloat16), TensorParallelPadderConfig(), ], )GGUF Example: Llama 3.2 1B InstructfromgptqmodelimportBACKEND, GGUFConfig, GPTQModelmodel_id="meta-llama/Llama-3.2-1B-Instruct"quant_path="Llama-3.2-1B-Instruct-GGUF-Q4_K_M"qcfg=GGUFConfig( bits=4, format="q_k_m", ) model=GPTQModel.load(model_id, qcfg) model.quantize(calibration=None, backend=BACKEND.GGUF_TORCH) model.save(quant_path)FP8 Example: Llama 3.2 1B InstructfromgptqmodelimportBACKEND, GPTQModelfromgptqmodel.quantizationimportFP8Configmodel_id="meta-llama/Llama-3.2-1B-Instruct"quant_path="Llama-3.2-1B-Instruct-FP8-E4M3"qcfg=FP8Config( format="float8_e4m3fn", # or "float8_e5m2"bits=8, weight_scale_method="row", ) model=GPTQModel.load(model_id, qcfg) model.quantize(calibration=None, backend=BACKEND.GPTQ_TORCH) model.save(quant_path)Exllama V3 / EXL3 Example: Llama 3.2 1B Instructfromdatasetsimportload_datasetfromgptqmodelimportBACKEND, GPTQModelfromgptqmodel.quantizationimportEXL3Configmodel_id="meta-llama/Llama-3.2-1B-Instruct"quant_path="Llama-3.2-1B-Instruct-EXL3"calibration_dataset=load_dataset( "allenai/c4", data_files="en/c4-train.00001-of-01024.json.gz", split="train", ).select(range(1024))["text"] qcfg=EXL3Config( bits=4.0, # target average bits-per-weighthead_bits=6.0, # optional higher bitrate for attention heads / sensitive tensorscodebook="mcg", # one of: mcg, mul1, 3inst ) model=GPTQModel.load(model_id, qcfg) model.quantize(calibration_dataset, batch_size=1, backend=BACKEND.EXL3_EXLLAMA_V3) model.save(quant_path)MoE Quantization 🧩
Some MoE (mixture of experts) models have extremely uneven/biased routing (distribution of tokens) to the experts causing some expert modules to receive close-to-zero activated tokens, thus failing to complete calibration-based quantization (GPTQ/AWQ). To better quantize these heavily biased MoE routed modules, GPT-QModel exposes 3 controls:
Moe.Routing = ExpertsRoutingOverride: Manually override the num_experts_per_tok used for model routing math, i.e., if a model only routes 4 experts per token out of 48 total experts, you can set this equal to 24 for 50% routing or 48 for 100% routing. ExpertsRoutingOverride requires the model exposes num_experts_per_tok or equivalent configuration control.
Moe.Routing = ExpertsRoutingBypass: Brute-force and bypass all routing math so all experts receive all activated tokens. This is akin to ExpertsRoutingOverride.num_experts_per_tok set to total number of experts. ExpertsRoutingBypass is enabled/tested for some models and, due to the lifecycle complexity, it needs to be validated for every model.
FailSafe: This is enabled by default and is a naive weight-only quantization technique using simple (naive) quantization methods such as nearest with optional smoothing. There are various FailSafeStrategy options, along with SmoothMethod options, to complement this feature. FailSafe does not require activations but has higher quantization error loss than normally activated GPTQ/AWQ. It is fast and applicable for all MoE models.
FailSafe can be combined with ExpertsRoutingOverride. There is no single best way to quantize MoE, and we recommend users to test all three methods.
Quantized Inference 🔍
# test post-quant inferencemodel=GPTQModel.load(quant_path) result=model.generate("Uncovering deep insights begins with")[0] # tokensprint(model.tokenizer.decode(result)) # string outputEoRA Accuracy Recovery: Enhanced Post-Quant Error Recovery via Lora 🎯
GPT-QModel supports EoRA, a LoRA method developed by Nvidia that can further improve the accuracy of the quantized model.
# EoRa is currently only validated for GPTQ# higher rank improves accuracy at the cost of VRAM usage# suggestion: test rank 64 and 32 before 128 or 256 as latter may overfit while increasing memory usageeora=Lora( # for eora generation, path is adapter save path; for load, it is loading pathpath=f"{quant_path}/eora_rank32", rank=32, ) # provide a previously GPTQ-quantized model pathGPTQModel.adapter.generate( adapter=eora, model_id_or_path=model_id, quantized_model_id_or_path=quant_path, calibration_dataset=calibration_dataset, calibration_dataset_concat_size=0, ) # post-eora inferencemodel=GPTQModel.load( model_id_or_path=quant_path, adapter=eora ) tokens=model.generate("Capital of France is")[0] result=model.tokenizer.decode(tokens) print(f"Result: {result}") # For more details on EoRA, please see docs/eora/# Please use the benchmark tools in later part of this README to evaluate EoRA effectivenessHow to Add Support for a New Model 🛠️
Read the
code which explains in detail via comments how the model support is defined. Use it as a guide for PRs to add new models. Most models follow the same pattern.
Shared-input metadata (:in=<tag>) 🔗
Modules that consume the same activation tensor (e.g. q_proj/k_proj/v_proj after input_layernorm) produce identical GPTQ Hessians (H = XᵀX), so the Hessian only needs to be collected once per group. BaseQModel.shared_input_plan(model_config, quantize_config) derives these groups from module_tree:
Default: every quantizable leaf is its own singleton group. Subset digits (:0) describe execution/quantization order, not tensor identity, so they are never used to infer sharing.
Opt in with :in=<tag>: sibling leaves (same parent) with the same tag share an input, e.g. "q_proj:0:in=x", "k_proj:0:in=x", "v_proj:0:in=x" or "gate_proj:0:in=x", "up_proj:0:in=x". Tags are scoped per parent. Different tags never share (MLA: "q_b_proj:1:in=q_a", "kv_b_proj:1:in=kv_a" read different latents).
A leaf repeated across module_tree variants must carry identical flags; conflicts raise at plan time.
:! / :? leaves and :in= tags never change the emitted subset blocks or quantization order.
Runtime dedup is per subset block: the looper captures one block at a time and elects the first group member in that block as leader; the other members in the same block skip Hessian capture and adopt a private copy of the leader's H. A tag whose members sit in different blocks (e.g. in_proj_qkv:0 / in_proj_z:1) is still validated by the probe but deduplicates nothing (SharedInputGroup.dedup_followers, SharedInputPlan.dedup_count reflect this).
Tags are inert until the definition lists the model_type in its own shared_input_verified_model_types (not inherited). Unlisted model types (including Llama-clone subclasses that inherit module_tree) get singleton plans and never skip capture; tests/module_tree/test_shared_input_cpu_forward.py enforces that every listed type has a real-forward case.
Only add :in= tags after verifying them against a real (tiny, CPU) model with gptqmodel.models.shared_input.probe_shared_inputs(layer, plan, forward); it hooks every planned module, runs forward, and reports groups whose inputs differ (mismatches), identical inputs that were not declared (undeclared), planned modules that do not exist (missing_modules) and groups that never ran (unverified, e.g. un-routed experts). report.ok is strict (fully_verified); use has_errors when un-routed experts are expected. See tests/module_tree/test_shared_input*.py for the covered definitions.
Pair with Evaluation for post-quantization LLM Benchmarks 📊
GPT-QModel evaluation is integrated into
, a modern benchmarking toolkit with 153 of the world's most widely used benchmark suites. We highly recommend using Evalution to measure post-quant accuracy recovery after quantization instead of relying on narrow regression-only language-model metrics.
# install Evalution pip install Evalution Below is a short example running gsm8k_platinum through Evalution's native GPT-QModel engine.
importevalutionasevalrun= ( eval.GPTQModel( backend="marlin", device="cuda:0", ) .model(eval.Model(path="ModelCloud/Llama-3.2-1B-Instruct-gptqmodel-4bit-vortex-v1")) .run(eval.benchmarks.gsm8k_platinum(apply_chat_template=True, batch_size=16)) ) print(run.to_dict()["tests"][0]["metrics"])Dynamic Quantization (Per Module QuantizeConfig Override) ⚙️
QuantizeConfig.dynamic is a dynamic control that allows specific matching modules to be skipped for quantization (negative matching) or have a unique [bits, group_size, sym, desc_act, mse, pack_dtype] property override per matching module vs base QuantizeConfig (positive match with override).
Sample QuantizeConfig.dynamic usage:
dynamic= { # `.*\.` matches the layers_node prefix # layer index starts at 0 # positive match: layer 19, gate module r"+:.*\.18\..*gate.*": {"bits": 4, "group_size": 32}, # positive match: layer 20, gate module (prefix defaults to positive if missing)r".*\.19\..*gate.*": {"bits": 8, "group_size": 64}, # negative match: skip layer 21, gate moduler"-:.*\.20\..*gate.*": {}, # negative match: skip all down modules for all layersr"-:.*down.*": {}, } Group Aware Reordering (GAR) 🔄
Group Aware Reordering (GAR) is an enhanced activation reordering scheme developed by Intel to improve the accuracy of quantized models without incurring additional inference overhead. Unlike traditional activation reordering, GAR restricts permutations to within individual groups or rearrangements of entire groups. This ensures each group's associated scales and zero-points remain efficiently accessible during inference, thereby avoiding any inference-time overhead.
How to enable GAR:
Set the act_group_aware parameter to True and disable the default activation reordering by setting desc_act to False in your QuantizeConfig. For example:
quant_config=QuantizeConfig(bits=4, group_size=128, act_group_aware=True)Experimental Features 🧪
Using GPTAQ (Experimental, not MoE compatible, and results may not be better than original) ⚗️
Enable GPTAQ quantization by setting gptaq = GPTAQConfig(...).
# Note GPTAQ is currently experimental, not MoE compatible, and requires 2-4x more VRAM to execute# We have many reports of GPTAQ not working better or exceeding GPTQ so please use for testing only# If OOM on 1 GPU, please set CUDA_VISIBLE_DEVICES=0,1 to 2 GPUs and gptqmodel will auto use second GPUquant_config=QuantizeConfig(bits=4, group_size=128, gptaq=GPTAQConfig(alpha=0.25, device="auto"))Using FOEM 🧮
FOEM (First-order error matters) adds first-order error compensation for GPTQ-style quantization. Enable FOEM by setting foem = FOEMConfig(...).
# FOEM default hyperparameters are alpha=0.0 and beta=0.2quant_config=QuantizeConfig(bits=4, group_size=128, foem=FOEMConfig(alpha=0.0, beta=0.2, device="auto"))Migrating from AutoGPTQ and AutoAWQ 🔄
GPT-QModel has fully supplanted AutoGPTQ and AutoAWQ for HF Transformers/Optimum/Peft integration. Model inference has drop-in support with zero changes.
For model quantization, there are some config changes for AutoAWQ:
AutoAWQ: version property is now format. zero_point is now sym (Symmetric Quantization): sym = True is equivalent to zero_point = False
Models quantized by GPT-QModel are inference compatible with HF Transformers (minus dynamic), vLLM, and SGLang.
Attributions 📚
GPTQ: IST-DASLab, main-author: Elias Frantar, arXiv:2210.17323
AWQ: main-authors: Lin, Ji and Tang, Jiaming and Tang, Haotian and Yang, Shang and Dang, Xingyu and Han, Song
ParoQuant: Z-Lab, main-authors: Yesheng Liang, Haisheng Chen, Song Han, and Zhijian Liu.
,
EoRA: Nvidia, main-author: Shih-Yang Liu, arXiv preprint arXiv:2410.21271.
GAR: Intel, main-author: T Gafni, A Karnieli, Y Hanani,
GPTAQ: Yale Intelligent Computing Lab, main-author: Yuhang Li, arXiv:2504.02692.
Fast Hadamard Transform:
Dao-AILab/fast-hadamard-transform
, by Tri Dao, vendored into gptqmodel_ext/hadamard/ under the BSD-3-Clause license. A copy of the license is included in gptqmodel_ext/hadamard/LICENSE.
Swordfish Kernel: Blackwell (>= sm100) GPTQ/AWQ kernel from
.
QQQ: Meituan, main-author Ying Zhang, arXiv:2406.09904
FOEM: Zheng, Xingyu and Qin, Haotong and Li, Yuye and Chu, Haoran and Wang, Jiakai and Guo, Jinyang and Magno, Michele and Liu, Xianglong
Citations 📖
# GPT-QModel@misc{qubitium2024gptqmodel, author = {ModelCloud.ai and [email protected]}, title = {GPT-QModel}, publisher = {GitHub}, journal = {GitHub repository}, howpublished = {\url{https://github.com/modelcloud/gptqmodel}}, note = {Contact: [email protected]}, year = {2024}, } # GPTQ@article{frantar-gptq, title={{GPTQ}: Accurate Post-training Compression for Generative Pretrained Transformers}, author={Elias Frantar and Saleh Ashkboos and Torsten Hoefler and Dan Alistarh}, journal={arXiv preprint arXiv:2210.17323}, year={2022} } # AWQ@article{lin2023awq, title={AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration}, author={Lin, Ji and Tang, Jiaming and Tang, Haotian and Yang, Shang and Dang, Xingyu and Han, Song}, journal={arXiv}, year={2023} } # ParoQuant@inproceedings{liang2026paroquant, title = {{ParoQuant: Pairwise Rotation Quantization for Efficient Reasoning LLM Inference}}, author = {Liang, Yesheng and Chen, Haisheng and Han, Song and Liu, Zhijian}, booktitle = {International Conference on Learning Representations (ICLR)}, year = {2026} } # GGUF / llama.cpp@misc{ggerganov2023gguf, author = {Georgi Gerganov and ggml-org contributors}, title = {llama.cpp and the GGUF model format}, publisher = {GitHub}, journal = {GitHub repository}, howpublished = {\url{https://github.com/ggml-org/llama.cpp}}, note = {Canonical GGUF implementation and format reference; see also \url{https://github.com/ggml-org/llama.cpp/wiki/dev-notes}}, year = {2023} } # EoRA@article{liu2024eora, title={EoRA: Training-free Compensation for Compressed LLM with Eigenspace Low-Rank Approximation}, author={Liu, Shih-Yang and Yang, Huck and Wang, Chien-Yi and Fung, Nai Chit and Yin, Hongxu and Sakr, Charbel and Muralidharan, Saurav and Cheng, Kwang-Ting and Kautz, Jan and Wang, Yu-Chiang Frank and others}, journal={arXiv preprint arXiv:2410.21271}, year={2024} } # GPTAQ@article{li2025gptaq, title={GPTAQ: Efficient Finetuning-Free Quantization for Asymmetric Calibration}, author={Yuhang Li and Ruokai Yin and Donghyun Lee and Shiting Xiao and Priyadarshini Panda}, journal={arXiv preprint arXiv:2504.02692}, year={2025} } # FOEM@inproceedings{zheng2026first, title={First-order error matters: Accurate compensation for quantized large language models}, author={Zheng, Xingyu and Qin, Haotong and Li, Yuye and Chu, Haoran and Wang, Jiakai and Guo, Jinyang and Magno, Michele and Liu, Xianglong}, booktitle={Proceedings of the AAAI Conference on Artificial Intelligence}, volume={40}, number={34}, pages={28883--28891}, year={2026} } # QQQ @article{zhang2024qqq, title={QQQ: Quality Quattuor-Bit Quantization for Large Language Models}, author={Ying Zhang and Peng Zhang and Mincong Huang and Jingyang Xiang and Yujie Wang and Chao Wang and Yineng Zhang and Lei Yu and Chuan Liu and Wei Lin}, journal={arXiv preprint arXiv:2406.09904}, year={2024} } # ExLlama V3 / EXL3@misc{turboderp2026exllamav3, author = {turboderp and exllamav3 contributors}, title = {ExLlamaV3 and the EXL3 quantization format}, publisher = {GitHub}, journal = {GitHub repository}, howpublished = {\url{https://github.com/turboderp-org/exllamav3}}, note = {Project repository and EXL3 format documentation: \url{https://github.com/turboderp-org/exllamav3/blob/master/doc/exl3.md}}, year = {2026} } # Group Aware Reordering (GAR)@article{gar, title={Dual Precision Quantization for Efficient and Accurate Deep Neural Networks Inference, CVPRW 2025.}, author={T. Gafni, A. Karnieli, Y. Hanani}, journal={arXiv preprint arXiv:2505.14638}, year={2025} } # Marlin Kernel@article{frantar2024marlin, title={MARLIN: Mixed-Precision Auto-Regressive Parallel Inference on Large Language Models}, author={Frantar, Elias and Castro, Roberto L and Chen, Jiale and Hoefler, Torsten and Alistarh, Dan}, journal={arXiv preprint arXiv:2408.11743}, year={2024} } # Swordfish Kernel@misc{alpindale2026swordfish, author = {AlpinDale}, title = {Swordfish: A Weight-Quantized {GEMM} Family for {NVIDIA} Blackwell}, howpublished = {\url{https://blog.alpindale.net/posts/swordfish/}}, year = {2026} } Quick Notes 🗒️
Limit log level 🔇
GPT-QModel uses a shared LogBar logger. Set the level once near process startup:
fromlogbarimportLogBarLogBar.shared().setLevel("WARNING") # DEBUG, INFO, WARNING, ERROR, CRITICALApply Triton nogil patch early in multi-package scripts 🩹
If your script imports multiple Triton users (for example gptqmodel, vllm, and sglang), apply the patch at the very top before other Triton-related imports:
fromgptqmodelimportTritonPatch# Fix Triton crashing under nogil/free-threading Python 3.13+ where the kernel cache storage in Triton is not thread-safeTritonPatch.apply()License 📜
GPT-QModel is licensed under the Apache-2.0 license. The optional Swordfish kernel sources vendored under gptqmodel_ext/swordfish/ are licensed under the AGPL-3.0-or-later license and are only compiled/linked at runtime via JIT. A copy of the Swordfish license is included in gptqmodel_ext/swordfish/licenses/LICENSE and licenses/SWORDFISH.