How we measured 99% accuracy, and how you can check it.

1. What the 99% number means

“Accuracy” can describe characters, fields, line items, or whole documents. Our benchmark uses field-value accuracy: for each value covered by the answer key, did FigureIQ return the correct value in the correct field?

We compared the final values returned by the FigureIQ pipeline with independently published annotations. There is no fuzzy matching or similarity threshold. A value is either correct under a written rule or it is not.

We lead with notation-adjusted accuracy because it treats $ and USD, or us and usa, as the same meaning. We also publish strict exact-match accuracy so readers can see the result without those two allowances.

2. Data and sample

We used FATURA, a public invoice dataset with 10,000 documents, 50 distinct layouts, and published field annotations. Each layout has 200 invoices.

The test set contains 2,000 invoices: exactly 40 from every layout. This balanced design prevents a common layout from dominating the result. The exact document-ID list is part of the reproduction materials; use it rather than drawing a new random sample.

3. What we tested

FigureIQ combines two model calls with deterministic processing. The benchmark measures the finished output that a customer receives, after all five stages:

This final output is the only result an external evaluator can observe and reproduce. We therefore do not use private intermediate stages to support the headline claim.

4. Ground truth and scope

Public annotations and product schemas rarely line up perfectly, so the mapping between them matters. We wrote the mapping rules before inspecting FigureIQ’s outputs and kept a change log. That reduces the risk of changing the answer key to favour the system.

For example, a FATURA tax annotation such as VAT (3.85%): 10.26$ maps to separate tax name, rate, amount, and currency fields. The source currency remains $ in the strict answer key; the scoring layer decides whether it is equivalent to USD.

Three answer-key states

Across the sample, the answer key speaks to 30,120 values: 28,000 annotated-present values used in the accuracy denominator and 2,120 annotated-absent values used for the hallucination check. Unannotated fields are never treated as correct nulls.

What is in - and out - of scope

FigureIQ’s schema has 72 fields. Thirty-nine FATURA annotations map to 26 comparison targets because some annotations form one address block or a repeated tax structure. Thirty-three fields are outside the benchmark; 21 of those are line-item cells, for which FATURA does not provide cell-level answers.

We also excluded 400 candidate values on 120 invoices where the two schemas cannot represent the same fact cleanly: 240 single-tax slots on multi-rate invoices, which are measured through repeated tax rows instead, and 160 buyer/bill-to values where FATURA distinguishes two parties but FigureIQ has one customer slot. These exclusions are fixed by the mapping contract, not by whether FigureIQ was right or wrong.

5. Scoring rules

We apply the same preparation to the answer-key value and the FigureIQ value before comparing them. The rules are deliberately simple and symmetric:

There is no edit-distance threshold, semantic similarity score, or numerical tolerance beyond two-decimal representation. A one-cent difference is wrong.

The two notation allowances

The adjusted score adds only the equivalences below. The list is closed: if a pair differs in any other way, it remains wrong.

Correctly empty fields do not inflate the accuracy numerator. Hallucinations use a separate denominator because they ask a different question: does the system invent a value where the answer key says none exists?

6. Results

The following results cover the final shipped output for all 2,000 invoices. The 95% intervals resample layouts rather than individual invoices, preserving the dependence among documents with the same template.

Where the remaining errors occur

The practical pattern is clear: core financial fields were stronger than contact details. The system also emitted 175 values across the 2,120 annotated-absent checks. We report those separately rather than hiding them inside the accuracy figure.

7. Reproduce the benchmark

A reader should be able to rebuild the final-output figures without access to FigureIQ’s internal logs. The shortest reliable path is:

1. Download FATURA and its annotations. Start from the public dataset described in the FATURA paper.
2. Select the published document IDs. Use the fixed 2,000-document list - 40 invoices from each of 50 layouts. IDs follow FATURA’s convention, such as Template10_Instance110.
3. Run every invoice through FigureIQ. Retain the final returned document after normalisation and checks, not an intermediate extraction.
4. Build the answer key with the mapping contract. Split combined tax annotations; store discounts as positive magnitudes; parse dates from their printed value and calendar; retain the printed currency; compare addresses as whole blocks; and mark a field annotated-absent only when its label exists on that layout.
5. Apply the comparison rules in section 5. Produce the exact and notation-adjusted results, plus the separate annotated-absent check. Confirm the checkpoints below before analysing individual fields.

8. What this benchmark does not cover

• Line-item extraction. FATURA does not provide the cell-level answers needed to score it rigorously, so line items are not part of the 99.379% result.
• US sales tax. The sample covers VAT and GST, including multi-rate cases, but does not test US sales-tax conventions.
• Reliable per-field confidence. FigureIQ does not yet expose a calibrated confidence value for every field. Document-level flags were too broad in this test - 72% of documents were flagged to catch 84% of documents with an error - so human spot-checking still matters.

Those boundaries are part of the result, not footnotes to it. The claim is strong within the measured field set and should not be extended to tasks the benchmark did not test.