LLM Pruning and Distillation in Practice: The Minitron Approach

https://arxiv.org/abs/2408.11796

Depth and width pruning plus distillation: Llama 3.1 8B to 4B and Mistral NeMo 12B to Mistral-NeMo-Minitron-8B; slight teacher fine-tuning on the distillation set helps when original data is unavailable; weights released.

On-device small language models for characters