Blog/Self-hosted speech and voice AI
Private AI

Self-hosted speech and voice AI

By Samuel Seidel · September 9, 2026 · 8 min read

Transcription and voice assistant pipelines can run on one self-hosted machine, so raw call audio never goes to a third-party speech API. That matters more for voice than for text. A call recording carries a person's actual voice, which some jurisdictions treat as biometric data, plus whatever they said, and people say things on a call they wouldn't write down. A speech API receives the raw recording, not a summary of it.

The two workloads: transcription and voice assistants

Transcription is the simpler case: audio in, text out. Speech-to-text has become one of the more mature open-weight AI categories, and the resulting text can feed anything text-based: a meeting summarizer, a support ticket, a searchable call archive, or the retrieval setup covered in self-hosted semantic search.

A voice assistant is a pipeline built on top of that: speech-to-text turns what someone says into text, an LLM decides how to respond, and text-to-speech turns the reply back into audio. Each stage is a separate model, and each can run on the same machine, so the whole round trip (someone's voice, the words they said, the reply generated for them) stays on that machine.

What fits on a single Spark

GPUwerk hasn't published benchmarked throughput for speech models the way it has for text-generation benchmarks, so treat this as a description of what fits, not a measured performance claim. A speech-to-text model, an LLM for the assistant's reasoning and reply text, and a text-to-speech model can all coexist within a single Spark's 128 GB of unified memory, the same way an embeddings model, vector database and LLM coexist in a document search setup. Unified memory is what makes that practical on one machine: three separate model types, each with its own memory footprint, share the same pool rather than needing three separate GPUs sized for the largest one.

What that means in practice depends on your setup: model sizes, audio quality, expected latency and concurrent call volume all move the answer. If throughput or latency at a given call volume is a hard requirement, test it against your own workload instead of assuming a number. I'd rather say that plainly than publish a figure nobody has measured.

Why self-hosting matters more for voice than text

Support calls, sales calls and internal meeting recordings usually include people who never agreed to have their voice sent to a specific AI vendor, even if they were told the call "may be recorded for quality purposes." That's a lower bar than informed consent to a specific third party handling the audio. Running speech-to-text and the downstream LLM yourself keeps the audio closer to the boundary that sentence implies, instead of extending it to whichever API vendor sits behind the transcription feature.

The same logic applies to internal voice assistants that employees talk to for scheduling, IT requests or quick lookups. Their voice input often carries incidental context nobody meant to share as data: a name mentioned in passing, background conversation picked up before someone realized the mic was live. With the whole pipeline self-hosted, that audio doesn't end up in a speech vendor's logs.

Getting started without overcommitting

Start with transcription, not a full voice assistant. It's a single model with a well-understood interface (audio file in, text out), and it slots into an existing workflow without a whole conversational pipeline designed up front. Run meeting recordings or support call audio through a self-hosted transcription model, feed the output into the text tooling you already have, and check whether transcript quality and turnaround meet the bar before investing in the full speech-to-text-to-LLM-to-speech loop.

Audio quality going in affects output quality more than most people expect. A clean single-speaker recording transcribes far more reliably than a multi-speaker call with crosstalk and background noise, and no amount of downstream LLM cleanup fully compensates for a bad source recording. If accuracy on real call audio matters, test with your own recordings rather than a demo clip before building a workflow around the result.

Where the hardware fits

A single Spark at $0.79/hour in EU-Central holds a speech-to-text model, an instruct model and a text-to-speech model together, which covers the shape of most internal use cases: transcribing meetings or calls, or a voice assistant for a small-to-mid-size team. Call-center volume, with many simultaneous audio streams needing low-latency transcription, needs more memory and throughput than one machine has; a linked pair of Sparks at $1.79/hour is the next step, and it's worth measuring against your actual call volume rather than extrapolating from a single Spark.

Related pages

First top-up: pay $20, get $30 in credit

Keep voice data off third-party speech APIs.

A dedicated DGX Spark in EU-Central, $0.79/hour, running transcription and voice assistant models on a dedicated machine assigned only to you.

Create account Private LLM hosting