Paper page - GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers

Elias Frantar ,

Published on Oct 31, 2022

Authors:

Abstract

GPTQ, a novel one-shot weight quantization method using approximate second-order information, significantly reduces the computational and storage requirements of large GPT models while maintaining accuracy and enabling execution on a single GPU.

Generative Pre-trained Transformer

models, known as

GPT

OPT

, set themselves apart through breakthrough performance across complex language modelling tasks, but also by their extremely high computational and storage costs. Specifically, due to their massive size, even

inference

for large, highly-accurate

GPT

models may require multiple performant

GPU

s, which limits the usability of such models. While there is emerging work on relieving this pressure via

model compression

, the applicability and performance of existing compression techniques is limited by the scale and complexity of

GPT

models. In this paper, we address this challenge, and propose

GPT

Q, a new one-shot weight quantization method based on

approximate second-order information

, that is both highly-accurate and highly-efficient. Specifically,

GPT

Q can quantize

GPT

models with 175 billion parameters in approximately four

GPU

hours, reducing the bitwidth down to 3 or 4 bits per weight, with negligible accuracy degradation relative to the uncompressed baseline. Our method more than doubles the compression gains relative to previously-proposed

one-shot quantization

methods, preserving accuracy, allowing us for the first time to execute an 175 billion-parameter model inside a single

GPU

for generative

inference

. Moreover, we also show that our method can still provide reasonable accuracy in the extreme quantization regime, in which weights are quantized to 2-bit or even ternary quantization levels. We show experimentally that these improvements can be leveraged for

end-to-end inference speedups

over

FP16

, of around 3.25x when using high-end

GPU

s (NVIDIA A100) and 4.5x when using more cost-effective ones (NVIDIA A6000). The implementation is available at https://github.com/IST-DASLab/

Get this paper in your agent:

hf papers read 2210.17323