Private AI document translation for sensitive documents
For a merger agreement, a patient record or financial statements headed for a foreign regulator, use an open-weight model you run yourself as the first-pass translator, not Google Translate or DeepL. Those tools are fine for a menu or an email, but the moment you paste a sensitive document in, it goes to a third party whose terms you probably haven't read closely enough to bet a client relationship on.
The legal and medical translation industry has known this for years, which is why certified translators still exist and why hospitals sign business associate agreements with translation vendors instead of using whatever's free. A managed translation vendor is slow and expensive for routine internal work; a consumer tool is fast but leaks. A self-hosted model sits between them: usable translation quality, and no translation service or model provider ever sees the document.
Who actually needs this
- Law firms handling cross-border matters, where the document is privileged and a third-party processor can itself be a breach of that privilege, independent of what the vendor's contract says.
- Hospitals and clinics translating patient records or intake forms, where the content is protected health information and routing it through a consumer tool is very likely a violation of whatever health-data rules apply.
- Finance and accounting teams translating statements, filings, or due-diligence material containing material non-public information, where a leak isn't just embarrassing, it's a regulatory event.
- In-house teams at multinationals translating contracts and HR documents at volume, where the aggregate cost of a consumer tool's per-document limits, or a paid API's per-token billing, adds up faster than a flat hourly rate for one machine.
What open-weight models can actually do here
Translation quality from general-purpose open models has closed most of the gap with dedicated translation engines for the language pairs that matter to most businesses (the major European languages, Chinese, Japanese, Arabic). Models like Qwen2.5 and Llama 3.3 handle document-length translation with reasonable fidelity to register and terminology, particularly if you give them domain context in the prompt. For specialized text (dense legal or medical terminology, idiom-heavy source material) a human reviewer should still check the output before it goes anywhere official. Treat the model as a first-pass translator that removes most of the manual work, not a certified translation service.
If your volume justifies it, a dedicated translation model like NLLB-200 (Meta's No Language Left Behind, open-weight, covering 200 languages) is worth running alongside a general model, since it's purpose-built for translation rather than adapted to it. For most business use, though, a general instruction-tuned model with a well-written prompt does the job and saves you running two model servers.
Setting it up
The simplest version is a local model served through an OpenAI-compatible API, called from a small script that reads a document, translates it in chunks, and writes the result back out. For PDFs and Word documents, extract the text first, translate, then re-insert it, since asking a model to also preserve exact document formatting tends to produce worse translations than treating format handling separately.
# serve a general-purpose model with strong multilingual coverage vllm serve Qwen/Qwen2.5-72B-Instruct --host 0.0.0.0 --port 8000 # translate.py - chunk-and-translate a text file, called per document import sys, requests def translate(text, target_lang, domain_hint): prompt = ( f"Translate the following {domain_hint} text into {target_lang}. " "Preserve legal/technical terminology precisely. Do not summarize or omit anything. " "Output only the translation.\n\n" + text ) r = requests.post("http://spark.internal:8000/v1/chat/completions", json={ "model": "Qwen2.5-72B-Instruct", "messages": [{"role": "user", "content": prompt}], "temperature": 0.2, }) return r.json()["choices"][0]["message"]["content"] if __name__ == "__main__": src, target, domain = sys.argv[1], sys.argv[2], sys.argv[3] text = open(src, encoding="utf-8").read() print(translate(text, target, domain))
Run as python translate.py contract.txt German legal. The domain_hint argument matters more than it looks: telling the model it's translating a legal contract versus a medical chart changes how it handles ambiguous terms and register, and it costs nothing to specify.
For long documents, chunk by section rather than by a fixed character count, so the model always sees complete sentences and paragraph context. A contract clause split mid-sentence across two API calls translates worse than the same clause sent whole.
Where the hardware fits
A 72B-class model at Q4 quantization needs roughly 45 GB of weights, which fits inside a single Spark's 128 GB of unified memory with room for long documents in context. If you're translating at volume across a legal or medical team, that's one machine handling the whole department's throughput, at a flat $0.79/hour rather than a per-page or per-token bill that scales with exactly the volume you're trying to serve.
The setup cost is small: stand up the model server, write the chunking script, and have someone who speaks both languages review the first batch of output against the source. After that it's a tool anyone on the team can point a document at, with no translation service in the path.