Contents4 sections
Most decoder-only translation models hit a ceiling with standard supervised fine-tuning (SFT). They produce accurate results, but they often fail to capture the specific style, tone, or terminology preferences that make it a truly a linguist. It's a problem we've seen repeatedly: the model is technically "correct," but it doesn't "speak the language" of the business.
We recently ran a series of experiments to explore what it would look like to close this "style gap" using Direct Preference Optimization (DPO) on one of our new style of Tiri decoder models.
The Known Problem
Even when a decoder model gets close to our foundational encoder-decoder models on machine quality metrics like BLEU or COMET, human judges (and automated LLM panels) often prefer the our encoder-decoder architecture.
It isn't about accuracy for the translation - it's about domain-specific "vibe."
The decoder models were correct, but they didn't feel natural in specialized technical contexts. They might use a general term where a client has a very specific preference, or they might miss the nuances of compound noun patterns in languages like German.
While this is not a problem for our main Tiri model family, there are cases where we need to use Decoder-only models to support specific environments.
So, if decoder-only models are going to be successful in the world of translation, and up to our high standards, we need a way to steer the model's style without breaking its underlying performance.
A Data-Driven Refinement
Rather than trying to fix this through more SFT (which tends to plateau), we have pointed our research toward Preference Learning. The idea behind it is simple: show the model thousands of examples of "this version is better than that version" and let it learn the signal.
The process we worked on followed a few key steps:
1. High-Confidence Labeling
We generated ~10k translation segments and used a panel of high-performing LLMs as judges to independently rank each pair. Ideally you would use humans here, but in interest of costs in this research, that was out of scope.
To keep the signal clean, we only trained on unanimous judgements. If all judges didn't agree that the reference style was closest to reference, we threw the data out. This gave us a high-confidence dataset of nearly 4,000 preference pairs. Quality of signal beats quantity every time.
2. Iterative Testing
We ran ~70 experiments across different loss functions, learning rates, and LoRA configurations. We weren't just looking for a one-off win - we wanted a repeatable recipe for style alignment, and to fundamentally understand how the decoder architecture processes these RL methods.
3. Measuring the "Vibe" shift
We saw a confirmed uplift of +0.20 on MetricX and +0.34 in our weighted internal scores. While those numbers might look small on paper, they are well above the measurement noise floor, especially in a short burst research sprint like this. In short; the process is working.
What we're learning
A few things became clear during this research:
Clean signal is the only signal. Using majority-vote data (where judges disagreed) actually performed worse than using a smaller, unanimous dataset. In preference learning, noise is expensive.
Stylistic shift is real. Over 37% of the segments were modified during the DPO process. The majority of these changes moved the output closer to the desired domain style - better handling of specific terminology rather than just correcting meaning. It's the difference between a generic translation and a professional one.
DPO is a general-purpose recipe. This wasn't a one-off fix for one language pair. Next stages of this research is to extend PL and RL to other methodologies and begin drawing comparisons between them.
Wrapping up
The goal for our Tiri models isn't just to be a "good" translator. It's to be an expert one, for your business.
While our foundational models do not need this method, exploring methods like DPO or GPRO allows us a way to bridge that final gap on Decoder-Only models in situations where our foundational models are not supported.
Sometimes the best way to improve a model isn't to give it more data, but to give it better feedback on what "good" generations actually look like.
Originally published on labs.straker.ai.
