Validator comparison: same defect, different answers

Why this exists

Your software says the invoice is fine. Your customer’s system says it is not. Both cite the same rule set, and nobody can settle it, because there is no public dataset either side can be measured against.

This is that dataset. Every document differs from the control in exactly one documented way, generated by a script rather than written by hand. When two validators disagree about a document, the disagreement is about that one change and nothing else.

Method

One base invoice, one mutation per case, applied by corpus/build.py. The results file keeps what a document was built to trip separate from what a validator actually reported — a dataset that asserts its own results proves nothing.

What the first run shows

Four documents never reach the business rules

Remove a mandatory field — the invoice number, the issue date, every line — and the document is already invalid against the UBL schema. What comes back is cvc-complex-type.2.4.a, and BR-02 never gets to fire. A comparison that scores "did it report BR-02?" will call this a miss when it is nothing of the sort.

Three documents are rejected with no reason given

With no specification identifier — in UBL and in CII alike — or a Peppol BIS one where an XRechnung one belongs, no scenario matches. So no schema is applied and no rule is evaluated. The answer is "not acceptable" with an empty findings list: refused, without anything being named. That is exactly what a supplier whose software emits Peppol BIS to a German recipient runs into. The table says "nothing" a fourth time, against the document that is not well-formed XML — there for the obvious reason that the file cannot be read at all.

Mustang names the rule where KoSIT reports only the schema

With the invoice number gone, KoSIT reports cvc-complex-type.2.4.a and stops. Mustang reports BR-02 — same invoice, same norm, but a sentence the sender can act on. It runs the other way too: the out-of-order element is caught by KoSIT and missed by Mustang entirely. Neither is uniformly better; they are strict in different places.

The same missing line, two entirely different answers

An invoice with no lines reports cvc-complex-type.2.4.b in UBL — a schema error nobody can act on. The same document in CII reports BR-16, BR-CO-10 and BR-S-08: three named rules in plain language. Same defect, same norm, very different usefulness.

KoSIT against Mustang

Two independent implementations of the same rule set, the same 29 documents. They agree on 22 and disagree on 7 — and the differences cut both ways.

7/29 disagree · Only the cases where they differ.

KoSIT v2026-08-31 · Mustang 2.17.0

UBL against CII: same defect, two syntaxes

Nine cases exist twice, once in UBL and once in CII, carrying the same change. A norm that demands the same of both syntaxes should report the same thing nine times.

1/9 differs

Every document

Validated with NormAPI · v2026-08-31

Download the data

CC BY 4.0. Attribution: NormAPI, normapi.de.

How to cite this measurement

NormAPI (2026). Validator comparison: same defect, different answers. https://normapi.com/en/study/validator-comparison

Measured against the rule-set release named above. KoSIT updates the rule set roughly twice a year; we repeat the run with each release and carry the date here forward. If the version above is older than the one currently live, this run has not been redone yet.

• Document list (CSV)
• Results (CSV) — KoSIT
• Results (CSV) — Mustang