A confidence score is not a check

Every line off your vendor's invoice ends up somewhere you will act on. So how do you verify invoice data extraction is correct?

The usual answer is a confidence score. Each field comes back with a number between zero and one. You review anything under a threshold you picked.

That number came from the model that read the document. It is the model grading its own homework.

What a confidence score actually tells you

A confidence score is a self-report. It says how sure the reader was, not whether the reading is right.

A model can be confident and wrong. It can be unsure and correct. Neither case shows up in the number.

This is not useless, and it is worth having. But it is evidence about the reader, not about the document.

Arithmetic does not have an opinion

Your vendor's invoice carries its own proof. Quantities, unit prices, line amounts, freight, tax, and a printed total.

Those numbers have to agree with each other. Whether they agree can be settled without asking the model anything.

Not everything on the page can be checked this way, and that is worth saying. A part number or SKU has nothing to test it against. An amount does, which makes amounts the safer half of any vendor document.

How to verify invoice data extraction is correct, check by check

Here is what runs on every document before a single row reaches you. There are eight of them. Each is a plain function of the document, with no model call inside it.

• The lines add up to the printed total. Line amounts, plus tax and freight, against the total your vendor printed. Vendors disagree about whether freight is a line or a header field, so every reading is tried. The gap has to be two cents or less.
• Every item line has a quantity and a price. A line missing either one is named, by line number.
• The document has a number on it. An invoice or PO number, printed by your vendor. Without one there is nothing to check next month's arrival against.
• The line numbers run in order. The same line number twice means a page was read twice. A long gap means a continuation page is missing.
• The currency is one we can hold exactly. Amounts are integer hundredths, never floating point. A currency written to three decimal places will not fit, so it stops rather than dropping a digit.
• Each line multiplies out. Quantity times unit price, against the amount printed on that line.
• This document has not come from this vendor before. The vendor name and the document number together, against every document already read.
• And one that does start with the model. The fields it marked as hard to read, counted against the length of the document.

The one we take from the model is the weakest one here

Only that last check starts with the model. It gathers the fields the model marked as hard to read. It adds the fields that were printed but could not be parsed at all.

Then it counts flagged lines against the length of the document. Three bad rows out of forty reads differently from three out of four.

It is the weakest check of the set, and that should be said plainly. It rests on a self-report, which is the thing this piece opened by distrusting.

It stays because it can only ever ask for a second look. It cannot clear a document the arithmetic rejected, and it releases nothing on its own.

What happens when a check fails

A failure goes to a person, not your spreadsheet.

The document stops. Every line is shown, the failing rows are marked, and each reason is written in plain words. Nothing goes to a CSV or a sheet until someone looks.

Some checks fail the document outright. Others mark a single row and name the likely reason. Either way the document waits for a person.

The two checks that are about your money

Most of that list is about our reading being right. Two of them are about something that costs you.

The first is the same invoice number arriving twice from the same vendor. A fingerprint of the file catches only the identical file, and a re-scan defeats it. Matching the vendor and the document number together survives that — here is why the pairing matters.

On a new account it has nothing to compare against yet. It says so, rather than reporting a pass it did not earn. It gets better every month you use it.

The second is a line that does not multiply out to the amount printed on it. Twelve at 4.15 should read 49.80. When it does not, someone should look at that line.

Line discounts, price breaks and per-thousand pricing all produce a gap that is entirely legitimate. So that check names the likely reason rather than just refusing the row.

The duplicate check needs no such hedging. Paying the same bill twice is worth a false alarm.

Check it against a document of your own

Take a vendor PO or invoice off your desk and drop it in here. Every line comes back, with every check named and its result shown. Nothing is stored and there is no sign-up.