See the quantization for yourself: multiple model precisions, live

A few weeks ago we [moved all serving to int8-quantized models](/blog/moving-to-compressed-models): ~3.8× smaller, several times faster, for a quality cost our measurements put below the noise floor of how machine translation is even scored. We stand by that. But there's a gap between "our metrics show no meaningful degradation" and "you can see for yourself that nothing got worse" — and on a language-preservation project, that gap matters more than the metric.

So we closed it. The translator can now serve the same model at more than one precision at once. When a backend machine has them, the translate page shows a "Model version" selector — int8 · compressed next to fp32 · original — and you choose which one does the work. Same sentence, same screen, two precisions, your eyes.

Why bother, if the metrics already said it's fine?

Because quality drift from quantization is subtle by nature, and subtle is exactly what aggregate metrics hide.

• A corpus-wide score like BLEU or chrF can move a tenth of a point while a specific word — a kinship term, a rare verb form, a place name — quietly degrades. Averages forgive the cases a speaker would notice first.
• Our models are narrow-domain and low-resource. The literature on quantization is mostly written about big, high-resource models. We'd rather not assume our languages behave like the average in someone else's paper.
• These languages belong to their communities. "Trust our benchmark" is the wrong posture for work that is supposed to be auditable by the people who actually speak Ewondo, Ghomálá', or Fe'efe'e.

We migrated to int8 because the evidence was good. We're exposing the comparison because being able to be wrong in public is the whole point of doing this openly.

How it works

Each backend worker advertises the models it can serve. Previously that was one list of language pairs; now each entry can carry a precision tag — francais-ewondo@int8, francais-ewondo@fp32 — and the site groups them into the version selector. The pivot still works the same way (X → French → Y); the chosen precision just rides along both hops.

Two properties we cared about:

• It's robust by construction. A precision only appears if the model file for it genuinely exists on that machine. If a server expects an fp32 copy that was never converted, the line simply doesn't show up — no broken option, no error, it's just absent. Missing means invisible.
• It's flexible per machine. Different backends can carry different sets. A free CPU node might serve int8 only; a beefier GPU box that comes online can add fp16 or the full fp32 originals. Whatever a server actually has, that's what you can pick while it's the one answering — so swapping in a new backend is as simple as asking it "what do you have?" and showing the answer.

What we'll be watching

This is also a small experiment in listening. With more than one option in front of people, we expect more — and more specific — feedback ("the fp32 version got this proverb right and int8 didn't"). Every translation already has a leave a comment link, and it now records which version produced the output. We'll be reading those reports closely to learn whether the multi-option world surfaces anything our metrics missed.

If it turns out int8 really is indistinguishable in practice — as we expect — then the selector is reassurance, freely given. If it turns out there's a class of cases where it isn't, we'd much rather hear it from a speaker than never know. Either way, the honest move was the same: stop asking you to take our word for it, and hand you the switch.

The fp32 originals were always archived. Now, where a machine has the resources, they're not just archived — they're one dropdown away.