Running embeddings and vector search on a DGX Spark
This guide shows how to run an embedding model and a vector database on one Spark next to your generation model, with a memory budget that holds up as the corpus grows. If you are still deciding whether RAG is the right tool, start with the next section; for the RAG-versus-agents framing, see the RAG vs. agents post.
What RAG does, and what it doesn't fix
Retrieval-augmented generation answers a question by first retrieving relevant text and putting it into the prompt. Documents are split into chunks, each chunk is turned into a vector by an embedding model, and the vectors go into a database. A query is embedded the same way, the closest chunks are found, and the language model answers from them. A model only knows its training data; RAG supplies current or private documents at query time, and when a fact changes you update the index, not the model.
It only works if retrieval surfaces the right text. If the search returns irrelevant chunks, or the answer needs several documents that were not retrieved together, the model can still answer wrongly and confidently. Chunking, embedding quality and the number of chunks per query decide how reliable the system is, which is why most of this guide is about those three.
Running all of it on one dedicated machine means documents are chunked, embedded and answered from on that machine, not sent to a third-party embedding or chat API. GPUwerk still operates the hardware under its DPA and keeps standard administrator access, so clear contracts or HR records with your own compliance function first.
What runs where
A RAG pipeline on one Spark typically has three components sharing the box's 128 GB of unified memory: your generation model, an embedding model, and a vector database. The embedding model only runs during ingestion and at query time to embed the user's question. It isn't in the hot path for every generated token the way the main model is, so it doesn't need to stay loaded continuously if memory is tight. The vector database, by contrast, needs its index resident (or memory-mapped) whenever a query comes in, and its footprint scales with corpus size rather than model size.
Choosing and serving the embedding model
Embedding models are small next to generation models: most useful open ones run from 100M to 8B parameters, so even a strong one is a modest addition to the memory budget, typically well under 5 GB for a larger multilingual model at a reasonable quantization. Serve it through the same engine you're already running: vLLM supports embedding models via its /v1/embeddings endpoint (launch with --task embed for a model that supports it), and llama.cpp's server exposes a similar embeddings endpoint for GGUF embedding models. Running it as a separate small process alongside your main generation model is usually simpler than trying to multiplex both onto one engine instance.
Model choice affects retrieval quality more than almost anything else in the pipeline. A default or small embedding model is often tuned for size over accuracy. For anything beyond a quick prototype, benchmark a couple of candidates against your own documents rather than assuming the first one you try is good enough. Retrieval quality varies a lot by domain and language, and no single model is best across all corpora. Which open embedding and reranker models score best on MTEB in November 2026, with licenses and sizes, is in best open embedding models.
Choosing a vector store
For a corpus in the low millions of chunks or fewer, an embedded vector database that runs in-process or as a lightweight local service (Chroma, Qdrant in embedded mode, or a Postgres instance with pgvector if you already run Postgres) avoids adding another network hop and another service to operate. These handle approximate nearest-neighbor search well within that range without needing a dedicated cluster. Move to a standalone vector database server, still self-hosted on the same box or a neighboring one, only once you need features an embedded store doesn't offer: multi-tenant isolation, hybrid search combining dense and sparse retrieval, or a corpus large enough that index build time becomes a real operational concern.
Memory budgeting
Three things draw from the same 128 GB pool, and you have to account for all of them:
- The generation model's weights and KV cache, sized per the memory management guide if you're running it alongside anything else.
- The embedding model, small on its own, but still a fixed cost while it's loaded.
- The vector index, which scales with corpus size and embedding dimension. A rough estimate: number of chunks × embedding dimension × 4 bytes (for float32 vectors) plus index overhead, which varies by index type but commonly adds 20 to 50% on top of the raw vector data for graph-based indexes like HNSW. A million chunks at 1024 dimensions in float32 is roughly 4 GB of raw vectors before index overhead: small next to a 70B model's weights, but worth checking against your actual corpus size rather than assuming it's negligible.
Most RAG deployments on a single Spark are generation-model-bound, not vector-index-bound: the index rarely competes seriously for memory unless the corpus is very large or the embedding dimension is unusually high. Confirm this for your own numbers before assuming it.
A basic ingestion and query flow
The ingestion side: chunk your documents, embed each chunk through the embedding model's endpoint, and write the resulting vectors plus metadata into the vector store's index. The query side: embed the incoming question with the same model (embedding a query with a different model than the one that indexed the corpus produces meaningless similarity scores), retrieve the top-k nearest chunks from the vector store, and pass them as context into the generation model's prompt. If you also use prompt caching, put the retrieved context near the end of the prompt: it changes per query and shouldn't sit in the static, cacheable portion.
Practical checklist
- Serve the embedding model through the same engine (vLLM or llama.cpp) you're already running, as a separate small process rather than multiplexed onto the generation model.
- Benchmark embedding model choice against your own documents; don't assume a default is good enough for anything beyond a prototype.
- Start with an embedded vector store for corpora in the low millions of chunks; move to a standalone server only when you need features it doesn't have.
- Budget vector index memory against corpus size and embedding dimension, not against a rule of thumb, and check it against the generation model's footprint rather than assuming it's negligible.
- Always embed queries with the same model that indexed the corpus; a mismatch silently degrades retrieval without an obvious error.
See why self-host semantic search and RAG vs. agents for the conceptual background, or the memory management guide for budgeting alongside other models on the same box.