LlamaIndex reports nearly 10x faster token generation for OCRia-1 on one B200
LlamaIndex reports that its optimizations raised OCRia-1 token generation from about 285 to 2,800 tokens per second on a single NVIDIA B200 GPU, a vendor-reported result not independently verified here. The stack combines workload-aware mixed-precision quantization, DFlash2 speculative decoding, ReplaySSM, custom kernels and 17 additional optimizations. LlamaIndex says OCRia-1 is available through LlamaParse across all tiers and plans to contribute generally applicable improvements to vLLM.
LlamaIndex says the faster stack enables quicker benchmark iteration and result delivery for document-parsing workflows, where latency is a frequent customer request.
While training and deploying our latest document parsing model, OCRia-1, we noticed a recurring bottleneck: inference was not fast enough. And this lined up with our experience customers had as well, latency is always one of the top asks for any document parsing workflow.
So, we set out to work tinkering, and came back with a nearly 10x improvement in token generation, going from ~285 tokens/sec to ~2800 tokens/sec on a single B200 GPU.
This means we can iterate on benchmarks faster and deliver results faster to users. We accomplished this by combining a ton of smaller improvements: reducing the bytes moved during inference, producing more tokens per target forward pass, and reducing the overall GPU and CPU overhead of each step. Some of these were easy wins with established practices (quantization, speculative decoding), while others required us to dig very deep into implementation details.
Why decoding dominates latency
OCRia is largely driven by image inputs with a text prompt, and outputs a structured textual representation of the document. Pages that are dense end up requiring longer outputs to represent them properly. Our primary focus for optimization then became increasing our generation speed across concurrent requests on a single GPU (B200 in this case).
#LlamaIndex#OCRia-1#LlamaParse#DFlash2#ReplaySSM#vLLM#NVIDIA B200#FlashInfer#FlashAttention#Triton#cuDNN