SIGNAL reads 43 sources six times a day, groups the same event across outlets, and publishes twelve stories a day. The sources are in English. For months the output was in Japanese.
That sounds like one job. It is two: summarise, and translate, in the same pass, with no intermediate anyone reads. Everything I later found wrong happened in that seam.
The checker that reported clean
There was already a verifier. Every proper noun and every number in a summary had to appear in the article it came from, or the summary got rewritten. It ran on every story. It almost never complained.
Then I switched the output to English and re-ran it on the same material. 480 words compared, 172 flagged as "not in the source" — 36% of the summary.
Twelve of those 172 were capitalised. The other 160 were words like *goes*, *beats*, *new*, *hotline*. The verifier had no test for what counted as a name. Any run of three or more Latin letters was checked against the article.
Why nobody noticed for months
In Japanese output, almost every Latin-script word *is* a proper noun — GPT-6, Qwen, Cloudflare. The rule "check every Latin word" and the rule "check every proper noun" produce the same answer, right up until the language changes. The checker was never correct. It was correct-shaped for one input.
€403M
The fix was narrow: only compare words containing a digit or an internal capital — GPT-6, Qwen-Image-2.1 — or words capitalised mid-sentence. Strip possessives. Split on hyphens. 480 comparisons became 53, with two false positives, both hyphen artefacts. On real model output afterwards: 76 names compared, zero false positives.
With the noise gone, a real error was sitting underneath it. Ireland fined Google €403M. The Japanese said 403 *oku* yen — wrong currency, and wrong by roughly three orders of magnitude.
The checker had passed it, and it was right to by its own rule. *403* appeared in the source. The digits matched. Nothing in the comparison looked at the symbol in front of them or the unit behind them.
480 → 53
words compared, after the filter
172 → 0
false positives, on real output
1
genuine mistranslation the noise was hiding
What it checks now
- Currency, by symbol (€ $ ¥ £) and by code (EUR USD JPY GBP CNY), in both directions — a currency that appears in the translation but not the original, and one that disappears from the original
- Magnitude, by normalising M, B, 億 and 兆 to actual values before comparing, so $10B against 10億ドル fails and $10B against 100億ドル passes
And when the Japanese still disagrees with the English after one rewrite, the story publishes in English alone rather than not at all. A missing translation is a gap. A confident wrong number is a lie with a citation attached.
A verifier is only valid for the shape of input it was built against. Change the language, the format, the source — and it keeps returning the same reassuring answer for a reason that no longer holds.