Contents7 sections
NVIDIA released Riva-Translate-4B-Instruct-v2 recently: a compact, openly licensed translation model covering 36 languages. It sits in exactly the weight class our Tiri models live in, a few billion parameters, designed to be served cheaply. When a well-resourced lab / company ships a strong model in your class, you have two options. You can hope your customers don't notice, or you can run the comparison yourself before anyone asks.
We ran the comparison: Riva against Tiri, three client domains, across metrics we and, the industry, trust. This post is about what we found, and why the result says something more interesting than a scoreboard.
The setup
We benchmarked Riva against our Tiri model family on: financial content in Japanese, weather content in French, and health content across four language pairs.
As we know, discipline matters more than the tooling, so briefly: frozen held-out sets that no model saw during training, the same public evaluation harness for every system, and multiple metric families reported side by side rather than a single flattering score. Surface metrics tell you about overlap with the reference. Neural adequacy metrics tell you whether the meaning survived. Conformance measures tell you whether the output reads the way the client requires. All three, every arm, no exceptions, and the test sets were frozen before Riva ever ran.
One more thing, because it's the first question any fair reader should ask of a vendor benchmark: Riva was served on its own stock chat template, with the language pair pinned exactly where the model expects to find it. We evaluated Riva the way NVIDIA intends Riva to be run. If a competitor benchmarks our models one day, that's the courtesy we'd ask of them too!
That last metric family deserves a sentence, because it decided this benchmark. Adequacy asks whether the translation says the right thing. Conformance asks whether it says it the way this specific user / business / organization says things: their terminology, their register, their headings, their formatting conventions. We've written before about why we think that second construct is systematically under-measured (see "Designing new metrics to define model success" on this blog).
Financial content in Japanese
Japanese financial translation is one of the harder things we do, and it's where a model should struggle: rigid terminology, number conventions, and register expectations that barely exist in general web text. There is very little, good, publicly-available data to train within this domain.
Like for like, our Tiri model beat Riva by 15.8 chrF points and led on every metric we ran, including adequacy. Reading the outputs tells you why the gap is that wide. Riva produced fluent Japanese with Chinese-influenced calques for core financial terms, and rendered large numbers literally where the domain convention writes them differently. A professional reader spots these in seconds. An adequacy metric mostly doesn't, because the meaning survives, and it is overall correct.
Weather content in French
This followed a fixed operational register: bulletins use canonical headings and phrasing that readers and downstream systems rely on. We added Riva as a fourth system to an existing comparison alongside two big frontier models we'd already evaluated.
On adequacy, Riva landed close to the big frontier engines, which is excellent. On the register, the picture changed. The human reference produces the canonical heading form at a 41% rate, our model at 42%, and Riva at 22%. Riva also repeatedly substituted a plausible synonym heading the reference never uses. Every one of those substitutions is correct French, and wrong output.
Health content across four language pairs
Four language directions, a thousand segments per direction, seed-fixed sampling. This is the domain where the two models looked most alike, which makes it the most instructive test.
On adequacy, the result was a match: differences sat within noise across the board. Stop measuring there and you'd call the models interchangeable. On overlap with the organisation's own approved phrasing, our Tiri model won every direction, all four. Same meaning, different words, and specifically not the words this client signs off.
Then we scored it with SCS, and the tie broke
Adequacy metrics had just told us the two systems were mostly interchangeable. So we brought out the instrument we built precisely for this situation: the Style Compliance Score we introduced in "Designing new metrics to define model success". Deterministic checks, no model grading another model, every rule mined from a customer's brand and style, every check grounded in MQM.
On the two suites built for this arm, our Tiri model scored 83 and 93. Riva scored 54 and 68.
A roughly thirty-point conformance gap between two systems that adequacy metrics call a tie. That is not a rounding difference or a judgement call. It is a deterministic, auditable measurement of the thing the client actually reviews for: their terminology, their conventions, their voice. Adequacy told us both models can handle the task. SCS told us which one can speak as this client, and by how much.
This is the pattern we built the metric to expose, and it's why the widget below has a third button. Flip through the lenses in order and you watch the argument assemble itself.
What the pattern means
Across three unrelated domains the same signature kept appearing: adequacy close, register not close. That's what the top of the quality curve looks like now. Good models of any stripe get the meaning right, exactly as you'd expect as the whole field improves. What separates them is whether the output conforms to the conventions of the user / organisation that has to publish it, and conformance isn't converging, because conformance isn't in the training data unless you put it there.
The difference this time is that we didn't have to argue the gap qualitatively. SCS put a number on it, deterministically and reproducibly, and the number was around thirty points in Tiri's favour. Benchmarks like this used to end in a judgement call. Now they end in a measurement, and that measurement is fast becoming the most commercially important number we produce for a client: it is the gap between output you publish and output you pay someone to fix.
Two things said plainly, because a benchmark post that only flatters its author's models isn't worth reading. First, Riva is a good model. At its size, in the locales it targets, it earns its reputation, and for a use case like basic NMT it's an okay choice. Second, our win is exactly as narrow as it sounds: The Tiri model family, working with a client's own brand and voice, wins. That's the product working as designed, not evidence our models are better at translation in general.
But that narrow win is the whole business. Enterprises don't buy translation in general. They buy their translation: their terms, their formats, their voice, at scale, with a post-editing bill that shrinks every time the model gets the register right the first time.
The habit behind the post
There's a second point here, less about the result and more about the practice. We benchmark competitor releases as a matter of routine, with the same rigour as our own promotion gates: frozen data, fixed method, every metric family reported. Not because we expect to win every arm forever, but because the alternative is finding out from a customer.
When you build models for a living, a frontier release in your weight class isn't a threat to ignore. It's free calibration. Either it confirms your models are earning their keep, or it tells you precisely where the next training run needs to go. Both outcomes are worth more than comfort.
This time, the Tiri models held, all three of them. A general-purpose model gets the meaning. Our models get the meaning in the client's voice, and the distance between those two doesn't show up on an adequacy leaderboard. It's where the value lives.
Originally published on labs.straker.ai.
