Reading documents automatically: from recognition to a verified posting

On Monday, the shared mailbox holds seventeen PDF invoices, three phone photos of delivery notes and a scanned stack of forms. The PDFs all look different, and two of them are themselves just scans. Someone in the office opens them one after another, reads off the supplier, date, invoice number, amounts and tax rate and types them into accounting. This costs hours and is not the part where skill is needed.

This starting picture is widespread in small businesses, practices and trades. It hardly stands out because each individual invoice takes only a few minutes. Added up, however, it ties up time that is missing elsewhere, and it creates errors in inconspicuous places: swapped digits, a forgotten early-payment discount, an invoice booked twice because it arrived once by post and once by email.

On top of that, knowledge depends on one person. Whoever has been entering incoming invoices for years knows that a certain supplier hides the date in the page footer and another puts the customer number in large type in front of the invoice number. This experience is written down nowhere. If the person is out, troubleshooting starts from zero, and any automation must first make this knowledge visible before it can replicate it.

The idea of leaving this to software is obvious. In this scenario, we assume that the business is ready for it but does not want to trust blindly. That is exactly where the professional question starts: not whether a program can recognize text, but how to ensure that only verified data ends up in accounting.

Recognition systems are often advertised with a single hit rate. In practice, this figure says little, because it leaves open what it refers to: characters, fields or complete documents. A document with twelve relevant fields is only captured correctly when all twelve are right. Even with high accuracy per field, a noticeable share of documents with at least one error remains.

More dangerous than the error is the silent error. If a 1 is read as a 7 or a decimal comma shifts, the result looks plausible. Nobody notices until the payment has been triggered or the tax advisor asks. A human also makes typing mistakes, but often notices from the total that something does not fit. An automated system does that only if you explicitly teach it this check.

Then there are documents that do not fit the pattern at all: credit notes with negative amounts, collective invoices covering several delivery notes, invoices in foreign currency, handwritten additions, stamps over numbers. And new suppliers keep appearing whose layout the system has never seen. A serious solution needs a way to handle all of this that is not called guessing.

Classic OCR converts an image into text, nothing more. It delivers characters and their position, but no understanding of which part is the invoice number and which is the customer number. To obtain fields, you need rules or templates: search to the right of the word “Invoice no.”, take the number in the box at the bottom right. This works reliably as long as a supplier keeps its layout and breaks at the first redesign.

AI extraction takes a different approach. A model considers text and layout together and assigns fields by meaning, even for layouts it has never seen. This saves maintaining templates and copes with variants. The price is a different kind of error: the model can deliver plausible but invented or swapped values, and its decisions are harder to trace than a rule.

The third way is often forgotten: not reading at all. Since the introduction of e-invoicing, many invoices exist as structured files, for example in the XRechnung or ZUGFeRD format, whose fields can be read directly. Recognition is unnecessary here, and reading becomes a schema check. A good intake route therefore first asks whether structured data is present and only then turns to OCR or AI.

We would not pit the methods against each other but tier them. Structured data first, rules or templates for known suppliers with a stable layout, AI extraction for everything else. Each tier has a different error profile, and the validation layer that follows must be able to handle all of them.

The analysis leads to a basic decision: reading must never decide a posting on its own. The extraction delivers suggestions, a separate layer checks them, and only what passes both continues automatically. This is slower to build than a script that writes fields into accounting, but it is the only way to say later why something went through.

The alternative, having a human proofread every receipt in addition, seems safe but saves hardly anything. People check poorly when almost everything is correct: attention drops, and the rare error slips through. It is better to direct attention to where the system itself has doubts or a rule has been violated. A small, well-justified review queue is more effective than a large stack that gets waved through.

Another trade-off concerns the provider. Ready-made receipt recognition services are quick to set up and learn from many documents, but they send the receipts to third parties and adapt poorly to special cases. A custom solution based on open components takes more effort but gives control over data flow, rules and costs. For sensitive records, such as those from medical practices, the data flow weighs more heavily. Then the question of data processing agreements and storage location belongs at the beginning, not the end.

Our recommendation would be to start small: with one document type, usually incoming invoices, and with the five to ten most frequent suppliers. Anyone who builds the validation chain cleanly first can connect further document types later without reinventing the safeguards.

A receipt arrives through a fixed entry point: a mailbox, an upload, a scan folder. Right here it is given a file checksum and a first duplicate check. Recognition follows, choosing the appropriate route depending on the source. The result is not a posting record but a data record with fields, the associated text passages and a confidence score per field.

Field validation first checks the format: is the date a date, is the IBAN formally valid, does the VAT ID have the expected pattern? Then comes the plausibility check. Does net plus tax equal gross? Does the tax amount match the rate, in Germany typically 19 or 7 percent? Do the line items add up to the total? Do we know the supplier, and does their bank account match the one we had before?

Cross-checking against what is already in the house is especially valuable. If there is a purchase order or delivery note for the invoice, quantities and amounts should match. If this supplier's invoice number has appeared before, it is a suspected duplicate. If a bank account differs, that is a classic fraud pattern and never belongs in the automatic path, even with high recognition confidence.

At the end of the chain is a decision with three possible outcomes: process automatically, send to the review queue, or reject with a reason. Every decision is logged, together with the rule that triggered it. The log is unglamorous, but it is what makes the process assessable in the first place.

Technically, the solution consists of a few building blocks. An intake service accepts files and stores them immutably. A recognition service delivers the fields, either from the structure of an e-invoice, through rules for known layouts, or through a model that evaluates text and position together. A validation layer checks the results, and a workflow distributes them. All parts can be swapped independently, which matters given how quickly models are developing.

Confidence is the trickiest concept. Many models output a value between zero and one for each field, but it is only partly calibrated: a value of 0.9 does not mean that nine out of ten answers are correct. We would therefore not set thresholds by gut feeling but calibrate them on real documents. You run a sample through, compare it with manual entry, and set the threshold where the errors begin.

Mandatory fields such as amount, date, invoice number and supplier get stricter thresholds than secondary fields such as payment term or reference. In addition, for amounts we check independently of confidence whether the invoice adds up arithmetically. An invoice with high confidence but an inconsistent total goes to the queue. One with medium confidence but a consistent total, a known supplier and a matching purchase order can pass through.

The review queue itself is a simple interface: the document on the left, the recognized fields on the right with highlighted passages of text, and below it the reason why the document ended up here. The person corrects, confirms and can suggest a rule, for example to create a new supplier. Every correction is saved, because it is the most valuable data source for later improvement.

Credit notes and cancellations carry negative amounts and refer to earlier invoices. A system that only knows standard invoices books them as receipts. A rule must therefore check document type and sign together and link a credit note to the original document. Equally tricky are combined invoices with several tax rates: here it is not enough to check one rate, every line item must match its own rate, and the subtotals must add up to the total.

Phone photos are a chapter of their own. Skewed shots, shadows and creases degrade any recognition. Preprocessing that straightens pages, crops them and assesses quality helps. If quality is too poor, the system should not guess but request a new image. That is less convenient than a result, but more honest than a wrong number.

With forms, such as patient questionnaires or registrations, handwriting comes into play. Handwriting recognition is considerably less reliable than recognition of printed text, and digits in form fields are particularly error-prone. We would generally keep a four-eyes principle for sensitive fields there and limit automation to pre-filling. The same applies to documents containing health data: they require careful clarification of storage location and processing before any service sees them.

Finally, language: invoices from abroad use different date formats, decimal separators and tax terms. An amount like “1.234,50” means something different from “1,234.50”. The rules must know the country of origin and the currency, otherwise errors by a factor of a thousand arise that even look consistent in the totals check.

The benefit of such a solution is plausible but cannot be quantified across the board. It is reasonable to expect that routine documents from known suppliers pass through with little effort and that the time per document falls. How much depends on the document mix, the quality of the templates and the number of doubtful cases. A promise like “80 percent less work” would be invented without the company's actual documents.

A lot can be measured, and before the start. You take a sample of several hundred real documents, capture them manually once as a reference, and have the system read the same ones. Then you count the errors per field and per document, separated into harmless and consequential. What matters is the distinction between the rate of fully correct documents and the rate of errors that went through unnoticed.

In operation, ongoing metrics are added: share of automatically processed documents, share going to the review queue, time to approval, share of fields corrected in the queue, and the random sample from the automatic path. The last metric is the most honest, because it measures what the system considered correct and was not. Drawing twenty or thirty automatically processed documents each month is enough for a first picture.

Error-freedom cannot be promised. Even a good chain occasionally lets a wrong document through, and responsibility for the bookkeeping stays with the company. Nor can one say in advance how quickly the investment pays off. A pilot phase with a clear stop criterion makes sense: if the error rate in the automatic path stays above the agreed limit after calibration, the threshold is lowered and more human work is planned in.

With recognition, a document is not done; it must also be findable and stay unchanged for years. The original belongs linked with the extracted data and the decision log, not just the posting entry. Anyone who later has to deal with a tax audit or an inquiry wants to see the document and trace how it was booked. Renaming by clear rules and write-protected storage help more than a complicated archive product.

For orientation: in Germany, statutory retention periods of several years apply to accounting records and business letters, often in the range of six to ten years depending on the document type, and there are requirements for immutability and traceability. The periods have changed in the past. This is not legal advice. Clarify the specific arrangement, including whether the paper original may be discarded after scanning, with your tax advisor.

Three lessons go the furthest. First, the quality of the input matters more than the choice of model: a fixed mailbox and scanning route, clean PDFs and asking suppliers for e-invoices improve everything downstream. Second, the validation layer is the real value, because models change while rules stay. Third, corrections from the queue are not a nuisance but training material for rules, supplier lists and thresholds.

It is also part of the picture what should not be automated. Payment approvals, changes to master data such as bank details, and decisions on unclear tax questions stay with people. The automation prepares, documents and warns, but it does not bear responsibility. Anyone who draws this line explicitly builds a system the team trusts because it knows its limits.

Reading documents automatically pays off when many similar documents arrive regularly and capturing them ties up time that would be better spent elsewhere. What matters is not the recognition technology but the chain after it: validation rules, confidence, a queue for doubtful cases and a log. With this chain, an error is a visible event; without it, a silent loss.

The next step is small and costs nothing but a little time. Collect a hundred random documents from the last months, sort them roughly by supplier and format, and note how many of them already exist in structured form. This count shows where automation pays off at all and which edge cases to expect. On that basis, a pilot with a clear measurement plan can be sketched.

Plan time for the reference. Manually capturing the sample looks like a detour, but it is the foundation of every later statement: without known correct values, you can neither set a threshold nor claim an error rate. Anyone who saves here ends up with a system whose reliability nobody can prove, not even the vendor.

We are happy to review this sample with you and tell you clearly whether a pilot is worthwhile or whether a clean intake route and e-invoices are enough as a first step. That too can be the best result: less technology, fewer errors.