Live Voice Translation in Minecraft

My server brings together people who speak Spanish, English, Portuguese, and Japanese. Voice chat worked, but half the conversations were lost because of the language barrier. So I built a system that listens to what you say, transcribes it, and translates it into each player’s language, displaying it in a speech bubble above your head.

Everything runs on my own machines, with no paid APIs, and the piece doing most of the heavy lifting is a VPS with a processor from 2010.

Summary

• What it does: transcribes Plasmo Voice proximity chat and translates it into each player’s Minecraft language. The text appears in a speech bubble above the speaker’s head.
• How: the Minecraft server sends a copy of the audio to a separate Python service (Vosk for recognition, OPUS-MT on CTranslate2 for translation) and receives the text back through a new protocol packet.
• Real-world latency: provisional text appears while you speak; the final sentence arrives ~0.7 s after you stop, and the translation ~0.2 s later.
• Cost: €0 in API fees. An 8-vCPU VPS can handle 8–10 people speaking at once with the good model.
• What didn’t work: translating into Japanese with small open models is unusable, and without a jargon glossary the translator turns “creeper” into things that shouldn’t be published.

How a sentence travels

You speak ──▶ Minecraft server ──▶ Voice service (another machine)
                    │                       │  Opus → 16 kHz → Vosk
                    │                       │  final text → translation
                    ◀────── text ───────────┘
                    │
                    └──▶ Speech bubble above your head, in each listener's language

Three decisions shaped the design:

1. Audio is copied on the server, not on the clients. The Minecraft server already receives everyone’s voice and has the key to decrypt it, so it opens a single connection to the voice service and sends the Opus audio as-is, without decoding it. That’s about 30 kbps per speaking player, instead of the 256 kbps it would take to send uncompressed audio.

2. Recognition runs outside the JVM. It eats several CPU cores, and I don’t want it stealing CPU from Minecraft’s main thread, which is what causes in-game stuttering. I can also restart the service or switch engines without touching the server.

3. If nobody is going to read it, don’t transcribe it. Audio is only sent if someone in range has subtitles enabled. This is the biggest saving of all: most of the time, the system consumes no resources.

Speech recognition: measure before choosing

The VPS is a 2010 Xeon X5675: 8 vCPUs, no AVX, no GPU. That rules out Whisper right away, so I went with Vosk (Kaldi), which is real streaming and provides provisional text while you speak.

Before writing a single line of the mod, I built a benchmark that simulates N players speaking at the same time in real time and measures processing delay. Results with the large Spanish model:

Simultaneous streams	Cores used	p95 delay	Verdict
2	0.55	71 ms	OK
8	2.62	99 ms	OK
10	3.91	129 ms	OK
12	6.90	794 ms	Falls behind

I learned two things here:

• The cost per stream increases with load. 12 streams should have required 3.6 cores, but they required 6.9. It wasn’t the provider stealing CPU (vmstat showed steal at 0), but rather the slow memory of a fifteen-year-old processor.
• A single process with threads performs better than multiple processes, because they all share the same model copy in memory: 2.3 GB instead of 14 GB.

I also tried tuning the large model. Removing its final neural re-scoring (rnnlm) reduced the cost by 27% with almost no loss in quality. Changing the decoder’s search parameters made no difference: the cost is in the acoustic neural network, and without AVX, no tuning can fix that.

Translation: the cheap option works

For translation, I use OPUS-MT (the Helsinki-NLP models), converted to CTranslate2 with int8 quantization. They’re 79 MB per language pair, and CTranslate2 only requires SSE4.1, so they run on the old Xeon.

Metric (on the VPS)	es→en	en→es
Time per sentence (p50 / p95)	197 / 245 ms	191 / 222 ms
Sentences per second, 2 threads	9.8	10.9
Sentences per second, 4 threads	17.5	18.4

This was the project’s pleasant surprise: the old processor translates almost as fast as my modern PC because the model is small.

Three design details matter more than the model itself:

• Translate once per language, not once per listener. If ten people are listening and they speak three different languages, that’s three translations, not ten.
• Only translate final sentences. Translating provisional text that gets corrected word by word would multiply the cost and make the subtitles flicker.
• Cache recent sentences. In a game, “let’s go”, “wait”, and “I’m coming” get repeated a lot.

Languages that don’t fit the grid

Covering several languages with pairwise models has an obvious problem: with N languages, you need N×(N−1) models. With 4 languages, that’s 12; with 20, it would be 380.

The classic solution is English as a pivot: if there’s no direct model, I translate into English and then from English into the target language. My router tries the direct model first and only chains translations when necessary. It takes twice as long (~1.1 s instead of ~0.5 s) and loses some fidelity, but avoids maintaining hundreds of models.

Two interesting cases came up.

Japanese: recognition yes, translation no

From Japanese into other languages, it works well: “危ないクリーパーが後ろにいる” came out as “Watch out! There’s a creeper behind me.”

Going the other way was a disaster. Helsinki’s only open English→Japanese model is trained on biblical texts, and “watch out there is a creeper behind you” turned into something about vultures gathering. Unusable.

Decision: a Japanese player can speak and everyone else can read the translation, but they read other people’s sentences in the original language. If no translation route exists, the system simply sends the original text; nothing needs to be configured.

Portuguese: two dialects, one model

For Portuguese, there is a modern model (opus-mt-tc-big-en-pt) that accepts a dialect tag: >>pob<< for Brazilian and >>por<< for European Portuguese. I integrated it by automatically detecting, from the model’s vocabulary, whether it supports those tags.

The honest result: the tag barely changes the text. I requested European Portuguese and kept getting “ônibus” and “trem” (Brazilian) instead of “autocarro” and “comboio”. The mechanism is in place in case better models appear, but for now it doesn’t promise what it can’t deliver.

The client preserves the region only for languages where it matters: es_mx and es_es are both “es”, but pt_br and pt_pt are sent in full. These two Portuguese dialects aren’t translated between; they’re read as-is.

And slang breaks everything

The worst translation in the project was Portuguese→Spanish, through English:

“olha que tens um creeper atrás de ti” → “Look, you have a prostitute behind you”

“Spawn” also became “offspring”, and “nether” became “lower part”. General-purpose machine translation doesn’t know the game’s vocabulary, and pivoting through English makes the damage worse. The solution is a glossary of protected terms that are never translated. That’s next on my list, and a perfect example of how these systems fail precisely on the words used most often in their context.

Language, without menus

One detail I’m happy with: there’s no language selector. The client sends the language set in Minecraft, and the server uses it to determine which language you’re speaking and which one to translate into. If you play in Spanish and someone else plays in English, each of you reads the other person’s messages in your own language without touching a single setting.

The only options I added are “show speech bubbles” and “allow my voice to be transcribed”, because transcribing people’s voices means processing personal data, and that should be asked about, not assumed. By default, nothing is saved to disk.

What I’d do with more budget

The pairwise language design has a clear ceiling. Beyond around six languages, a GPU with a single multilingual model (Whisper or similar) makes more sense than one model per language: you go from 20 models and 30 GB of RAM to two models and better quality across all languages at once. The code already has a separate engine interface, so this would mean adding a class, not rebuilding everything.

For now, the system does what it promised: people understand each other.

What I learned

1. Measure before building. The benchmark I made before touching the mod determined the model, the number of simultaneous users, and even which part of the model to disable.
2. Zero latency doesn’t exist, and it’s worth saying so. Nothing can transcribe a word before you’ve finished saying it. The realistic goal was “make it feel instant”, and provisional text while you speak achieves that.
3. Degrade gracefully, don’t break. If the voice service goes down, voice chat keeps working; only subtitles stop appearing. If there’s no CPU available, the sentence is skipped. If there’s no translation, the original is shown.
4. The interesting failures happen at the edges: a model trained on the Bible, a dialect tag that does nothing, and “creeper” turned into an insult.