on-device-language-models technique sourced
On-device small language models for characters
Pruned, distilled, and 4-bit quantized language, speech, and embedding models that run on the player's GPU beside rendering to drive character decisions, dialogue, and voice without per-token server cost.
mechanism
A 12B or 8B model is pruned in depth and width and distilled into an 8B, 4B, or 2B student (Minitron), then weights are quantized to 3-4 bits with second-order (GPTQ) or activation-aware (AWQ) methods, shrinking memory so a 2B model fits in about 1.5 GB of VRAM. An in-game inference SDK runs the models in-process and schedules them alongside graphics on the same GPU. A character loop chains on-device speech recognition, the small model (perceive, decide, act, reflect), text-to-speech, and facial animation, with an in-process embedding model for memory retrieval; slow strategic planning can be offloaded to a larger cloud model while fast decisions stay local. Fine-tuning a small model on a narrow game task can beat much larger general models on that task.
limitations
- Hardware floor: the inference SDK needs an RTX 30-series class GPU with 8 GB of VRAM minimum, which excludes many players and consoles, and the model competes with rendering for VRAM and GPU time.
- Small models are less capable at multi-step reasoning, so vendors offload strategic planning to cloud models, reintroducing cost and latency.
- Quality and safety filters must also run locally; local weights can be extracted or modified by players.
- Player reception risk: inZOI drew negative user reviews in May 2025 after Krafton announced its generative AI use.
- Exact model sizes and latency in shipped titles are mostly undisclosed.
relations
6 stored edgesAdvisors
Total War: PHARAOH's ACE advisor answers questions about the game's systems from a small on-device model connected to game data, instead of firing authored lines when indicators cross thresholds.
Memories and thoughts
Characters generate text thoughts about events and reflect on past decisions, giving each agent a running inner monologue rather than a list of authored mood modifiers.
Natural-language interaction
Free-text and voice input to characters runs on the player's GPU, removing per-token server cost and network round trips, so open conversation can be offered to every player of a premium game.
Unit commands
A co-playable teammate such as PUBG Ally takes spoken orders about movement, looting, and strategy mid-match and acts on them, a command channel that needs no menus or ping wheel.
Utility action selection
Smart Zois pick actions with a small model that reads needs, surroundings, and inner thoughts rather than only scored need curves, while the game's action set stays the same.
backlinks
computed from other recordsclaims
6 with sourcesMinitron pruning and distillation compressed Mistral NeMo 12B to an 8B model and Llama 3.1 8B to 4B, and NVIDIA's ACE character stack lists Minitron 8B, 4B, and 2B instruct models, the 2B fitting in about 1.5 GB of VRAM.
GPTQ quantizes weights to 3-4 bits with minimal accuracy loss and generates 3.25-4.5x faster than FP16; AWQ's 4-bit runtime is over 3x faster than FP16 on desktop and mobile GPUs.
inZOI (early access March 2025) ships an on-device SLM through which typed prompts influence Zois' actions and thoughts; an August 2025 update made Smart Zois observe surroundings and react to events better and align actions with inner thoughts.
KRAFTON ran a public PUBG Ally beta (Ally Duo, 17 June to 1 July 2026) in which an AI teammate takes voice instructions via an on-device small language model built on NVIDIA ACE with speech-to-text and text-to-speech.
NVIDIA's in-game inference SDK requires an RTX 30-series class GPU or newer with at least 8 GB of VRAM (12 GB recommended), and some plugins need RTX 40-series.
Hidden Door reports that a 1.7B model tuned on real play beat a 120B general model on one of its core narrative tasks.