OCR Accuracy Benchmarks and LLM OCR Leaderboards

Last updated August 2026.

The two OCR benchmarks that leaderboards are built on, OmniDocBench and olmOCR-Bench, contain no receipts and no invoices. OmniDocBench is 1,651 PDF pages across ten document types: academic literature, presentations, black and white books, colorful textbooks, exam papers, handwritten notes, magazines, research reports, financial reports and newspapers. olmOCR-Bench is 1,403 PDFs and 7,010 unit tests, of which 3,385 test whether a model reads mathematical formulas correctly. So a model that tops either leaderboard has demonstrated it can parse a physics paper. It has demonstrated nothing about a faded thermal receipt from a gas station, and the ranking should not be used as though it had.

That is not an attack on the benchmarks. Both are careful, well documented academic work, and they measure exactly what they claim to. The problem is downstream, in how the scores get repeated. A composite number designed to compare document parsing research gets quoted in vendor comparisons as though it predicted expense extraction quality, and buyers make procurement decisions on it.

What is an OCR accuracy benchmark?

An OCR accuracy benchmark is a fixed set of documents with hand verified ground truth, plus a scoring method, used to rank engines on identical input. The three parts that decide whether a score means anything to you are the document mix, the metric, and how the metric is aggregated. Change any one and the ranking can invert.

The document mix is the part almost nobody checks. A benchmark built from arXiv preprints measures column detection, equation layout and reading order. A benchmark built from receipts would measure tolerance for thermal fade, skew, crumple and merchant layouts that follow no standard. An engine can be excellent at one and mediocre at the other, because the underlying problems barely overlap.

What does OmniDocBench measure?

OmniDocBench is the document parsing benchmark from OpenDataLab, published at CVPR 2025. It covers 1,651 PDF pages across ten document types, with block level annotations in 28 categories and page attribute tags for language, column layout, watermarks, scan quality and background.

Look closely at that overall score. It is the average of three numbers, and one of the three is formula recognition. A receipt contains no mathematical formulas, so a third of the headline composite is measuring a capability that is entirely irrelevant to expense processing. Two models separated by four points on the composite could be identical on the only axis you care about, or reversed.

The financial reports category is the closest thing to a business document in the set, and it is still not close. A financial report is typeset, digitally generated, flat and high resolution. It shares almost no failure modes with a receipt that spent three weeks in a coat pocket.

What does olmOCR-Bench measure?

olmOCR-Bench, from the Allen Institute for AI, takes a different and in some ways more rigorous approach. Instead of scoring similarity to a reference transcription, it runs 7,010 discrete unit tests against 1,403 PDFs, each test asserting something checkable about the output.

Broken down by what is being asserted, the 7,010 tests split into math formula accuracy (3,385), natural reading order (1,061), table accuracy (1,020), text absence such as correctly dropping headers and footers (823), and text presence (721).

Math is 48 percent of the benchmark. If you are choosing an engine to read expense receipts, roughly half of the score you are comparing is measuring whether the model can transcribe an integral sign. The table accuracy tests are genuinely relevant, since receipt line items are a table, but they are 15 percent of the total and they are tables from academic papers rather than from point of sale printers.

The unit test design is worth stealing regardless. Asserting that a specific value must appear, that a header must not appear, and that two spans must appear in a given order is a far better evaluation pattern than cosine similarity against a reference blob. It is also exactly how you should build your own test set.

Why is there no receipt OCR benchmark?

Because receipts are private financial documents. Academic benchmarks need publishable, redistributable corpora, and arXiv papers, out of copyright books and newspapers can be shared freely. A representative receipt corpus would be thousands of real purchase records belonging to real people, which nobody can lawfully publish at scale. The small academic receipt datasets that do exist are old, mostly non US, and far too clean to be representative.

So the gap is structural, not an oversight, and it is not going to close. Which means the only receipt OCR benchmark that will ever describe your workload is one built from your own documents. There is no version of this where a public leaderboard does that job for you.

What do OCR vendors publish instead?

Almost nothing, once you check the primary documentation rather than the marketing pages. Google states plainly that Document AI does not provide a metric for Accuracy, reporting precision, recall and F1 against a test set you supply. AWS Textract returns a per item confidence percentage and no accuracy figure at all, noting only that detections with a low confidence might need visual confirmation by a human. Azure publishes an estimated accuracy score for custom template models alone, calculated against your own labeled training data, and recommends targeting 80 percent or higher, or close to 100 percent for financial and medical records.

We collected all of that in one place, along with what each metric hides, on the OCR accuracy page. The short version is that when a comparison article quotes a tool as 99.5 percent accurate, the number did not originate with the company that built the engine.

What is a leaderboard score actually good for?

It is a useful shortlist filter and a poor decision rule. A model sitting near the bottom of OmniDocBench is unlikely to surprise you on receipts, so leaderboards are reasonable for eliminating candidates. They are not reasonable for choosing between the top handful, where the gaps are small, the document mix is wrong for you, and the ordering changes with every release.

Rankings also go stale unusually fast in this field. New model releases reshuffle the top of both leaderboards every few months, which means a comparison table published six months ago is describing a race that has already been rerun. If you find yourself citing a benchmark position in a procurement document, put the date next to it.

How to build a benchmark that predicts your results

Borrow the olmOCR-Bench method and apply it to your own documents. It takes an afternoon and produces a number that survives scrutiny.

Start by pulling 50 to 100 real documents from the last three months, weighted the way your actual inbox is weighted. If a third of what you process is thermal paper photographed on a phone, a third of the test set should be too. Include five documents you fully expect to fail, because knowing where the ceiling sits matters as much as knowing the average.

Then write assertions rather than transcriptions. For each document, record the merchant, date, subtotal, sales tax and total you know to be correct, and decide in advance what counts as a match. Is 03/04/2026 an acceptable rendering of March 4th? Is a merchant name without its legal suffix correct? Settling that before you look at any output is what stops an evaluation drifting into a justification.

Grade per field and per document, and record which fields fail rather than only how many. Line items and sales tax fail far more often than merchant name and total, and they are the fields a vendor demo never stresses. Finally, multiply the residual error rate by your monthly volume and by the minutes it takes to find and fix one bad entry. That figure, set against the published per page cost on the OCR API pricing page, is the actual comparison.

Accuracy does not stay where you measured it

A benchmark is a photograph, and production is a film. Merchants redesign their receipts, a supplier switches to a new billing system, a model provider ships an update, and the extraction quality you measured in March quietly stops describing July. The teams that get burned are not the ones who skipped the evaluation, they are the ones who ran it once.

The fix is to keep a small labeled sample running against live output on a schedule and alert on movement, which is the same discipline any team applies to a data pipeline whose freshness and schema need watching after the initial build. Extraction quality is a data quality problem once it is in production, and it responds to the same monitoring.

Cheap in house signals help too, and they need no labels. Track the share of documents where line items plus sales tax fail to reconcile against the stated total. Track the share of fields returned below your confidence threshold. Neither is an accuracy measurement, but both move before your ledger does, which is the property that matters.

How accurate is LLM OCR compared to traditional OCR?

On messy, varied and handwritten documents a vision language model reads substantially better than a rule based engine, which is why the leaderboards are now dominated by them. On clean, uniform, high volume text the gap narrows to the point where Tesseract remains hard to beat on cost, since it is free. We worked through that comparison in detail in LLM OCR vs Tesseract.

The more important difference is not accuracy but failure behavior, and no benchmark score captures it. Tesseract returns visibly garbled characters when it cannot read something. A language model completes the pattern and returns a clean, confident, wrong number. Two engines with identical benchmark scores can impose completely different review costs depending on whether their errors announce themselves. The trade offs are laid out on the LLM OCR page, and what the numbers mean for receipts specifically is in how accurate receipt OCR really is.

The benchmark that decides anything is yours

Public OCR benchmarks are good science aimed at a different question than the one a finance team is asking. They rank document parsing research on academic PDFs and mathematical notation, and they do it transparently. They were never built to tell you whether a gas station receipt will import into your books with the right sales tax, and reading them as though they were is how tools get chosen badly.

Build the small, ugly, representative test set instead. Grade fields rather than characters, keep the arithmetic check running after you deploy, and treat every published percentage as a claim to verify rather than a fact to compare. If you want a starting point, run a stack of your worst receipts through the bulk receipt scanner and grade the output yourself, or wire the same extraction into your own evaluation harness through the receipt OCR API. Either way the number you end up with will be worth more than any leaderboard position.