XGrammar: Flexible and Efficient Structured Generation Engine for Large Language Models

https://arxiv.org/abs/2411.15100

CFG-constrained decoding; vocabulary split into context-independent tokens prechecked and context-dependent tokens checked at runtime with a persistent stack; grammar work overlapped with GPU execution; up to 100x faster than prior solutions, near-zero end-to-end overhead.

Grammar-constrained decoding for valid game actions