LlamaIndex releases ExtractBench Pro for document extraction's long tail
LlamaIndex has released ExtractBench Pro, an extension of the ExtractBench benchmark it launched two months earlier, built to cover long-tail customer documents and schemas the original 370-document set never covered. All 100 documents are synthesized by deterministic pipelines from seeds and configs, and are split into six challenges: document conventions, house vocabulary, house counting rules, drawn relations, charts inside documents and long chart sheets. LlamaIndex says it tested twelve systems — four extraction APIs, five coding agents and its three LlamaExtract tiers — and added a metric for the proportion of documents with perfect 100% Value F1; the results are vendor-run with no independent replication supplied.
LlamaIndex says extraction remains unsolved even as scores on the original ExtractBench approach saturation, and that rigid pipelines struggle on ExtractBench Pro while agentic systems with the right tools can adapt.
Two months ago, we released ExtractBench to measure and improve schema-based document extraction. What we learned from it has accelerated our research: in the LlamaExtract v2.5 release, even our lowest-cost tier scores above 93% value F1.
But the benchmark also had blind spots. Some real customers brought complex documents and schemas that the original 370 ExtractBench documents never covered. This is the long tail of document extraction. It’s why extraction remains unsolved, even as scores on ExtractBench climb toward saturation.
Today we’re releasing ExtractBench Pro, an extension built from those real long-tail documents. It covers six new challenges, each drawn from ways that real customers broke our assumptions. In the samples below, you’ll find values that appear only in intricate charts, rules on one page that change the meaning of another, dense domain jargon, chaotic layouts, and more. Rigid pipelines struggle on ExtractBench Pro, but agentic systems with the right tools can adapt.
