What is multimodal AI?
Multimodal AI is a model that processes more than one modality, meaning more than one type of input or output, within a single model. The common case today is text and images together: a vision-language model that takes a photo and a text question and answers in text, or a model that accepts text and audio. This is different from stitching together separate specialized models, one for vision and one for language, and passing text between them. A genuinely multimodal model is trained so the different modalities share representations inside the same network, letting it reason about an image and the accompanying text jointly rather than treating the image as a caption fed in as a second step.
How images become tokens a language model can use
A vision-language model typically has a vision encoder that turns an image into a sequence of embeddings, then a projection step that maps those embeddings into the same space the language model's token embeddings live in. From there, the language model processes them alongside text tokens using the same attention mechanism it already has, treating image embeddings as more entries in the sequence rather than something fundamentally different. This is why a single image can add a substantial number of tokens to a request's context, sometimes several hundred to a few thousand depending on resolution and the encoder, which is worth accounting for when estimating context budget for image-heavy workloads.
What this adds at inference time
The language model portion of a multimodal model runs the same way any text model does: weights get read from memory once per token during decode, the same bandwidth-bound arithmetic our benchmarks page walks through for text-only models applies. What multimodal adds is upfront cost during the vision-encoding and prefill stage, since an image's tokens have to be processed before generation starts, and a larger KV cache footprint per request because of how many tokens an image can contribute. None of that changes the underlying memory-bandwidth ceiling on generation speed; it changes how much context budget and prefill time a given request consumes before generation even begins.
Where it's genuinely useful versus where it's not needed
Multimodal input matters when the task actually involves non-text data: reading a scanned document, describing a photo, reviewing a screenshot, transcribing and reasoning over audio. For a workload that's purely text in, text out, a multimodal model brings no benefit and adds unnecessary vision-encoder overhead and cost. Our models page lists which catalog models support image or audio input, and our self-hosted speech and voice AI post covers the audio-input case specifically if that's the modality in question.