Contents9 sections
Correctness metrics are saturating, and the residual is not noise
Evaluation of generated language, whatever the generator, has been organised around one construct: is the output correct? Benchmarks measure it, reference-based metrics estimate it, and where neither fits, an LLM judge is asked to grade it. These instruments were designed for a regime in which systems regularly failed at the task, and in that regime they ranked systems well.
That regime is ending across generative AI. Machine translation has the field's longest-running correctness measures (BLEU and chrF for surface overlap, then COMET and its successors for learned semantic adequacy), and on professional in-domain content, strong general-purpose systems and domain-adapted models now routinely land within noise of each other on all of them. This is the saturation problem: as systems converge near the top of the quality range, metric deltas shrink toward measurement error and ranking power collapses. The standard response has been to build better correctness estimators.
We believe that response misses what the residual actually is. When two systems tie on correctness and a professional reviser still confidently prefers one of them, the preference is usually not about correctness at all. It is about conformance: this organisation's mandated terms, its prohibited variants, its headings, its number and punctuation conventions, its formatting invariants. Every enterprise deploying generative AI has this layer, whether the output is a translation, a summary, a support response, or product documentation. In professional translation the layer is contractual, it dominates revision effort precisely when correctness is already high, and no widely used metric measures it directly. That combination, high stakes, mature correctness baselines, and decades of curated approved output to learn from, is what makes translation the right laboratory for a problem the whole field has.
Observations that forced the question
Some independent results, months apart, convinced us the missing measure is structural rather than incidental.
A rank inversion. In a financial-translation comparison, the system with the highest BLEU was systematically wrong on the domain's most salient convention, producing a fluent, plausible, non-house rendering of "cash flow statement" where the client mandates a specific form. The interesting part is the mechanism: the fine-tuning data contained the mandated form far more often than the variant, and the variant still won, because the base model's pre-training prior outweighed the supervised signal. Surface-overlap metrics rewarded the output; the client's acceptance criteria rejected it. When a metric and a paying reviser disagree in sign, the metric is measuring the wrong construct.
A reproducible dissociation. Benchmarking a strong general-purpose model against domain-adapted models across several client domains, adequacy metrics returned near-ties in pairing after pairing, while hand-coded convention checks (required headings, mandated terms, number formats) separated the systems widely, in every domain, in the same direction. We had effectively built an ad hoc, unformalised version of the same instrument twice. Both times it captured a signal adequacy metrics could not see, and both times the signal agreed with human revision judgements.
The pattern is a dissociation between two constructs the field tends to conflate: task correctness and conventional conformance. The first is what our metrics measure. The second is what professional acceptance turns on once correctness is high. Nothing about the dissociation is specific to translation; translation is merely where both constructs are old enough to be measured against decades of professionally revised ground truth.
What we're building
Our answer, in active development in a project we call arbitr-style-alignment, is a style compliance score (SCS): a deterministic conformance measure, defined per convention, reported alongside adequacy and never in place of it. The scoring rule fits in four lines:
domain_pass = every required string is present (may be vacuously true) no_wrong_variant = no banned variant appears (may be vacuously true) full_pass = domain_pass AND no_wrong_variant score = 100 x full_passes / scored_items (reported per convention)
Three details of that rule are load-bearing. Either side of a rule may be empty and is then satisfied by default: a rule that only lists required terms is first-class, because real client termbases rarely enumerate what must not appear. How strictly two strings are compared is decided per convention, never globally; why that matters is the subject of the construct-validity section below. And segments a system never generated are excluded from that system's denominator and reported as missing coverage, not silently absorbed into the score.
A convention is a data artifact, not code. One synthetic example, matching the demo below:
{
"dimension": "term:air mass",
"rule": "term",
"src": "An air mass will cross the region on Saturday morning.",
"expected_terms": ["masse d'air"],
"wrong_variants": ["masse atmosphérique"],
"match_mode": "caseless"
}
Three properties matter for measurement validity. It is deterministic: no model in the loop, no prompt, no sampling; identical inputs give identical scores, and every verdict is auditable down to the character. It is hypothesis-only: scoring reads the system output and, where a rule requires it, the source; it never reads the human reference at scoring time, which keeps reference register from confounding a conformance verdict. It is dimension-structured: the reporting unit is the named convention, not the aggregate, so the output is a diagnosis (which convention fails, on which systems) rather than a single number that buries it.
The rule inventory is deliberately not ours. Each of the seven rule types (required terminology, forbidden variants, patterned conventions such as number formats, source-copy spans, casing, source-target numeric agreement, placeholder integrity) maps to a named error category in MQM, the field's existing quality-assessment framework. SCS is best understood as the deterministically checkable subset of MQM, instrumented.
Scope is equally explicit. SCS does not measure naturalness, register, or semantic correctness; those remain the province of adequacy metrics and human judgement. A high SCS beside a low adequacy score is a meaningful configuration (mandated vocabulary inside a wrong sentence), which is exactly why the two axes must be reported together and never merged.
A worked measurement
The figure below is a live instance of the scoring rule: one source segment, two synthetic system outputs that are adequacy-equivalent (they mean the same thing), and three conventions from a hypothetical weather-domain rule set. It also embeds, deliberately, the central measurement hazard of the whole approach, made explicit in the construct-validity section further down. Run the checks, then enable the fold option and run them again.
System A · adequacy 0.87
System B · adequacy 0.87
That demo is three rules. Production suites compile hundreds of conventions across dozens of named dimensions from a customer's own assets, and report per dimension, so the output isn't a grade, it's a to-do list: which conventions are settled across every system, and which word to fix.
Relation to prior work
Terminology-aware evaluation is not a new idea. The WMT community has run terminology shared tasks, and terminology-constrained MT has its own literature. Against that line, this work differs in three specific commitments:
Negative constraints are first-class. Existing terminology evaluation predominantly checks presence of required terms. Professional conformance is equally about prohibition: the plausible, adequate, banned variant that must not appear. SCS scores a two-sided verdict per convention, with vacuous truth on whichever side a rule does not specify, because real client termbases are usually positive-only and must not be penalised for it.
The construct is broader than terminology. Headings, casing policy, number and punctuation formats, full-width versus half-width characters, do-not-translate spans, placeholder integrity: all enumerable, all deterministically checkable, all invisible to adequacy metrics, and all present in the MQM taxonomy already. Restricting the construct to terminology understates both the problem and the available standard.
Rule acquisition is treated as part of the research problem, not an input assumption. Shared tasks provide the term list. Production does not. That gap is the subject of the next section but one.
On LLM-as-judge approaches: we use judges where judgement is genuinely required, and this layer is not that. A convention check is a string predicate. Interposing a language model adds cost, latency, and a stochastic error source to a question that has a deterministic answer, and it surrenders the property this metric exists to provide: verdicts auditable down to the character.
The construct-validity problem in a "trivial" metric
A string predicate looks like the one component of an evaluation stack that cannot be wrong. It has exactly one free parameter, and that parameter turns out to be a genuine measurement decision: the equivalence relation over strings. Unicode offers many codepoint sequences that render near-identically (typographic versus typewriter apostrophes, full-width versus half-width brackets and digits), and the checker must either quotient them out or distinguish them. Both choices are construct-validity threats, in opposite directions, and we hit both on real data.
Normalising globally destroys conventions whose content is the codepoint. One of our Japanese financial rules tests for full-width brackets, （）, a genuine formatting requirement of the domain. A standard Unicode normalisation pass, applied suite-wide, folded （） to () in the hypotheses before checking, making the rule unsatisfiable: it failed every output, including perfect ones, and the suite lost 14 points with no code defect anywhere. The convention under test had been normalised out of existence.
Refusing to normalise manufactures failures out of typography. On a real client dictionary, one system produced a mandated French term correctly but with the typographic apostrophe that word processors insert by default, where the dictionary held the typewriter form. Nearly half of that system's apparent terminology violations on the suite were this single codepoint. The aggregate was plausible; the verdict was false.
The resolution is that the equivalence relation is per-dimension, not global. A rule whose content is character width must see raw codepoints; a vocabulary rule should quotient typography away. This is a measurement decision that must be made convention by convention, and it is a standing threat, so it carries a standing procedural control: on any newly compiled rule set, the failure list is read by hand before its number is quoted. Both incidents above were caught that way, and neither was visible in the aggregate. That audit is cheap, and it is only possible at all because every verdict is a readable line; there is no equivalent operation on a neural metric's score.
Rule acquisition as weak supervision
The scientifically interesting half of the project is not the checker. It is where the rules come from. Client conventions rarely exist as written rules; they exist implicitly in the translation memory, years of approved, revised output in which the mandated form always appears and the banned variant never does. Our hypothesis: those conventions are latent in the TM and can be extracted by mining, with no per-client hand-authoring, which would turn per-client conformance evaluation from a consulting engagement into a batch process.
Because a miner tuned on the clients it is scored against "works" by construction, we pre-registered the evaluation before the optimisation loop ran: gold rule sets frozen in advance (hand-authored earlier, independently of the miner), mining code prohibited from referencing any gold term, an untouched held-out domain as a generalisation probe, and pass thresholds fixed together with the downgrade we would publish on failure (recall failure demotes the claim from zero-curation mining to assisted authoring; generalisation failure narrows it to same-domain scaling). Under that protocol the miner passed its recall gates and recovered four of five hand-verified conventions on the held-out domain with zero domain-specific tuning. The remaining pre-registered gate, precision under linguist review, has not yet run; every mined rule set is therefore labelled indicative, including below.
The most instructive result is negative. Synonym-class wrong variants, the highest-value convention type, are structurally unmineable from a curated TM: curation is precisely the process that removed those errors before storage, so the negative class is censored from the data by the same quality process that makes the data trustworthy. No amount of miner tuning fixes a censored distribution. The planned remedy is model-contrast mining: generate over the same sources with a generic system and treat systematic divergences from the approved references as candidate negatives, drawing the wrong-variant class from errors systems actually make rather than from a corpus that by construction contains none.
Empirical behaviour
Two measurements from development, spanning the two rule-acquisition paths. System identities withheld pending publication.
The claim
The quality of generated language, as bought and sold professionally, is at least two constructs: task correctness, which the field measures well and which is saturating, and conventional conformance, which determines acceptance once correctness is high and which the field does not measure at all. The second construct is enumerable, deterministically checkable, groundable in an existing quality framework, and, we now have early evidence, largely extractable from an organisation's own historical approved output.
We are validating this in translation because translation is the one generative domain with the data to do it properly: professional revisers as ground truth, a mature error taxonomy, and years of curated output per client. But the two-construct structure travels. A summary, a support reply, and a product description all have an organisational voice, mandated vocabulary, and formatting invariants that no correctness benchmark or LLM judge scores, and the same instrument applies: enumerate the conventions, mine them from the archive of approved output, check them deterministically, report them beside correctness. Cheaply, reproducibly, auditably, and without adding another model to grade the first one.
That is the gap this work addresses. The method, the validation results, and the code follow when the protocol closes.
This is currently still in research phase, we're not ready to release this, or our paper yet... but stay tuned.
Originally published on labs.straker.ai.
