Why AI agents get tables wrong — and what to do instead

What actually goes wrong

Give an agent a table and ask for the line items. It returns JSON. The JSON parses, the keys are right, the numbers look like numbers. The pipeline continues. Nothing raises an error, because nothing went wrong in a way software can notice.

What went wrong is in the content. Two rows with a shared cell became one row. A row whose text sat slightly low against a rule got dropped. A column with a blank cell shifted every value in that row one position left, so the amount is now sitting under the tax column. Each of those produces output that is structurally perfect and factually wrong.

This is the specific danger of table work for agents, as opposed to prose. A model that misreads a sentence usually produces something visibly odd. A model that misreads a table produces a table. The failure has no signature, so it propagates — into a reconciliation, a total, a decision — and surfaces much later as a number nobody can explain.

What rasterising throws away

A PDF stores glyphs with coordinates. For every word on the page the file knows its position, its size and its font. Columns are not something a reader infers from a PDF — they are a consequence of x-coordinates that the document states outright.

Rasterising renders those instructions into a grid of pixels and discards the instructions. The column boundary that was an exact number becomes something to be inferred from whitespace. The row that was a y-coordinate becomes an alignment judgement. Every question the file could answer precisely is converted into a question that must be estimated.

And that is the standard pipeline: PDF, to image, to vision model, to JSON. Each step is reasonable on its own and the sequence throws away the answer before asking the question. Improving the model at the end does not recover the coordinates deleted at the start — it just produces a more confident estimate of something that was known exactly two steps earlier.

Reading the geometry instead

The table extraction reads the PDF’s word geometry directly. Column boundaries come from the x-coordinates the file already records; rows come from y-coordinates; a merged cell that spans two columns is read as spanning two columns because the geometry says so, not because a model decided it looked that way.

Two properties follow, and both matter more than accuracy claims. It is DETERMINISTIC: the same file produces the same table on every run, which means a discrepancy in your pipeline is a real change in the document rather than sampling noise. And it is FREE of a model call for the table pass, which is why it can run on every page rather than only where somebody is willing to pay for a vision call.

Tables come back as JSON rows with a header, and optionally as markdown or HTML, with cell geometry available. The geometry is the part worth keeping: it lets a reviewer jump from an extracted number straight to the region of the page it came from, which turns a disputed figure into a thirty-second check instead of a re-read.

When there is no geometry

Somebody photographs a payslip on their desk, or a document arrives as a flatbed scan. There are no glyph coordinates in a picture of a page; there is only the picture. The deterministic path is not available, and pretending otherwise would be the worst possible answer.

So the fallback is explicit. A vision model reads the table and returns rows. There is no cell geometry, because there is nothing to derive it from. The response records which path ran, so a caller can distinguish "exact, from the file" from "read from an image" rather than treating both as the same fact.

That distinction is the whole point. A pipeline that silently degrades from measurement to estimation, and reports both identically, has removed the one signal that would let a reviewer know when to look harder.

The capability contract

An agent should not have to discover a tool’s limits by trial and error, and it should not have to trust prose. So the capability list is an endpoint rather than a page: free, no key, and machine-readable.

Ten options are available and seven more are planned. Asking for a planned one returns an error naming it — never a silent no-op, which is the failure mode that quietly produces incomplete output for months before anybody notices. An agent can read the contract, see that an option is not available yet, and plan around it.

Two things in that contract are unusual enough to point at. Every option carries a `not_detected` list — the things it deliberately does not do. For tables that reads: vertical merges are reported as rowspan 1 rather than guessed, and a table split across pages with no repeated header is not stitched. For layout: heading levels come from a font-size heuristic, and more than two columns is not modelled.

And every option carries a `measured` flag. Six of the ten are marked measured. Four are marked not — including tables, the flagship. That is uncomfortable to publish and it is the correct thing to publish: an unmeasured capability that claims measurement is how a tool ends up trusted for something nobody ever tested.

What it will not do

VERTICAL MERGES ARE NOT RECONSTRUCTED. A cell spanning three rows is reported as rowspan 1. Guessing the span is possible and would be wrong often enough to poison the rows around it, so the honest answer is the flat one, declared in the contract rather than discovered in production.

CROSS-PAGE TABLES ARE NOT STITCHED unless the header repeats. Without a repeated header there is no reliable way to know that the rows on page four continue the table on page three, as opposed to starting a new one.

THE PAGE LIMIT IS TEN, with fifty fields per call and up to twenty tables of two hundred rows each. These are bounds, not targets — a request past them is rejected rather than silently truncated, because a truncated extraction that looks complete is the same class of failure as a misread column.

AND EXTRACTION IS NOT VERIFICATION. The values returned are what the document shows. Whether the document is genuine is a different question, answered by a different set of checks — a fabricated invoice extracts perfectly.

How to test any extraction tool

ONE: give it the same document twice, once as the native PDF and once as a photograph of the same page. If the two results are identical in quality, the tool is rasterising the PDF and discarding the geometry. If the PDF result is better AND the response says why, it is reading the file.

TWO: ask what it does not detect. A tool that cannot answer has not looked closely at its own edges, and a tool that answers in general terms is answering about the category rather than about itself. Merged cells and cross-page tables are the two that separate careful implementations from confident ones.

THREE: ask which features have been measured, and against what. "Measured" should mean a corpus, a method and a number that somebody could disagree with. Anything else is a claim about intent.

We have not run anyone else’s extraction tool, so this page contains no comparison table. Publishing accuracy numbers for products we have not tested would be exactly the kind of claim the three questions above are designed to catch.