While training and deploying our latest document parsing model, OCRia-1, we noticed a recurring bottleneck: inference was not fast enough. And this lined up with our experience customers had as well, latency is always one of the top asks for any document parsing workflow.

So, we set out to work tinkering, and came back with a nearly 10x improvement in token generation, going from ~285 tokens/sec to ~2800 tokens/sec on a single B200 GPU.

This means we can iterate on benchmarks faster and deliver results faster to users. We accomplished this by combining a ton of smaller improvements: reducing the bytes moved during inference, producing more tokens per target forward pass, and reducing the overall GPU and CPU overhead of each step. Some of these were easy wins with established practices (quantization, speculative decoding), while others required us to dig very deep into implementation details.

Why decoding dominates latency

OCRia is largely driven by image inputs with a text prompt, and outputs a structured textual representation of the document. Pages that are dense end up requiring longer outputs to represent them properly. Our primary focus for optimization then became increasing our generation speed across concurrent requests on a single GPU (B200 in this case).