AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration

https://arxiv.org/abs/2306.00978

4-bit weight-only quantization protecting about 1% salient weights chosen from activation statistics; TinyChat runtime over 3x faster than Hugging Face FP16 on desktop and mobile GPUs.

On-device small language models for characters