DGX Spark/Guides/Running embeddings and vector search on a DGX Spark
RAG

Running embeddings and vector search on a DGX Spark

By Samuel Seidel · Published September 9, 2026

This guide shows how to run an embedding model and a vector database on one Spark next to your generation model, with a memory budget that holds up as the corpus grows. If you are still deciding whether RAG is the right tool, start with the next section; for the RAG-versus-agents framing, see the RAG vs. agents post.

What RAG does, and what it doesn't fix

Retrieval-augmented generation answers a question by first retrieving relevant text and putting it into the prompt. Documents are split into chunks, each chunk is turned into a vector by an embedding model, and the vectors go into a database. A query is embedded the same way, the closest chunks are found, and the language model answers from them. A model only knows its training data; RAG supplies current or private documents at query time, and when a fact changes you update the index, not the model.

It only works if retrieval surfaces the right text. If the search returns irrelevant chunks, or the answer needs several documents that were not retrieved together, the model can still answer wrongly and confidently. Chunking, embedding quality and the number of chunks per query decide how reliable the system is, which is why most of this guide is about those three.

Running all of it on one dedicated machine means documents are chunked, embedded and answered from on that machine, not sent to a third-party embedding or chat API. GPUwerk still operates the hardware under its DPA and keeps standard administrator access, so clear contracts or HR records with your own compliance function first.

What runs where

A RAG pipeline on one Spark typically has three components sharing the box's 128 GB of unified memory: your generation model, an embedding model, and a vector database. The embedding model only runs during ingestion and at query time to embed the user's question. It isn't in the hot path for every generated token the way the main model is, so it doesn't need to stay loaded continuously if memory is tight. The vector database, by contrast, needs its index resident (or memory-mapped) whenever a query comes in, and its footprint scales with corpus size rather than model size.

Choosing and serving the embedding model

Embedding models are small next to generation models: most useful open ones run from 100M to 8B parameters, so even a strong one is a modest addition to the memory budget, typically well under 5 GB for a larger multilingual model at a reasonable quantization. Serve it through the same engine you're already running: vLLM supports embedding models via its /v1/embeddings endpoint (launch with --task embed for a model that supports it), and llama.cpp's server exposes a similar embeddings endpoint for GGUF embedding models. Running it as a separate small process alongside your main generation model is usually simpler than trying to multiplex both onto one engine instance.

Model choice affects retrieval quality more than almost anything else in the pipeline. A default or small embedding model is often tuned for size over accuracy. For anything beyond a quick prototype, benchmark a couple of candidates against your own documents rather than assuming the first one you try is good enough. Retrieval quality varies a lot by domain and language, and no single model is best across all corpora. Which open embedding and reranker models score best on MTEB in November 2026, with licenses and sizes, is in best open embedding models.

Choosing a vector store

For a corpus in the low millions of chunks or fewer, an embedded vector database that runs in-process or as a lightweight local service (Chroma, Qdrant in embedded mode, or a Postgres instance with pgvector if you already run Postgres) avoids adding another network hop and another service to operate. These handle approximate nearest-neighbor search well within that range without needing a dedicated cluster. Move to a standalone vector database server, still self-hosted on the same box or a neighboring one, only once you need features an embedded store doesn't offer: multi-tenant isolation, hybrid search combining dense and sparse retrieval, or a corpus large enough that index build time becomes a real operational concern.

Memory budgeting

Three things draw from the same 128 GB pool, and you have to account for all of them:

Most RAG deployments on a single Spark are generation-model-bound, not vector-index-bound: the index rarely competes seriously for memory unless the corpus is very large or the embedding dimension is unusually high. Confirm this for your own numbers before assuming it.

A basic ingestion and query flow

The ingestion side: chunk your documents, embed each chunk through the embedding model's endpoint, and write the resulting vectors plus metadata into the vector store's index. The query side: embed the incoming question with the same model (embedding a query with a different model than the one that indexed the corpus produces meaningless similarity scores), retrieve the top-k nearest chunks from the vector store, and pass them as context into the generation model's prompt. If you also use prompt caching, put the retrieved context near the end of the prompt: it changes per query and shouldn't sit in the static, cacheable portion.

Practical checklist

See why self-host semantic search and RAG vs. agents for the conceptual background, or the memory management guide for budgeting alongside other models on the same box.

First top-up: pay $20, get $30 in credit

Build your RAG pipeline on one dedicated machine.

Rent a dedicated Spark from $0.79/h, billed by the minute, and run your embedding model, vector store and generation model in one place.

Create account Read the vLLM guide