GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers

https://arxiv.org/abs/2210.17323

One-shot 3-4 bit weight quantization with approximate second-order information; 175B model quantized in about four GPU hours with minimal accuracy loss; about 3.25x (A100) and 4.5x (A6000) faster generation than FP16.

On-device small language models for characters