LangChain against your own DGX Spark endpoint
This page points an existing LangChain chain or agent at a model on your own DGX Spark. You change two constructor arguments on ChatOpenAI, base_url and api_key, and nothing else in your code. That works because LangChain's OpenAI integration wraps the client from the openai Python package.
You need:
- A running GPUwerk instance with an OpenAI-compatible endpoint: a console model preset, or your own engine on a base image.
- For a console preset, an API key with the
inferencescope, created in the console. - Python with
langchain-openaiinstalled (pip install langchain-openai). - No SSH key. GPUwerk configures the machine, and the console's browser terminal works without one. Add your own key only if you want your own SSH client, scp or port forwarding.
Start with a running endpoint
- Pick the endpoint. With a console model preset, copy the HTTPS endpoint from the instance page (it looks like
https://app-<instance-id>.gpuwerk.com) and add/v1; pass your inference-scope API key asapi_key. With your own vLLM server,<spark-host>below is your instance's hostname,<name>.gpuwerk.com. Start it with--api-keyor reach it through an SSH tunnel, because the instance's public IPv4 address makes an open port reachable from the internet. Put LiteLLM in front if several applications will call the model. - Confirm it answers before you write any Python:
curl http://<spark-host>:8000/v1/models
For a preset, add-H "Authorization: Bearer $GPUWERK_API_KEY"and use its HTTPS URL. You should see JSON listing the model ID, for exampleqwen3-coder-30b.
ChatOpenAI against your Spark
LangChain's ChatOpenAI class (in langchain-openai) takes base_url and api_key arguments that map directly onto the underlying OpenAI client's constructor:
from langchain_openai import ChatOpenAI
llm = ChatOpenAI(
model="Qwen/Qwen3.8-27B-FP8",
base_url="http://<spark-host>:8000/v1",
api_key="not-needed-unless-you-put-a-gateway-in-front",
)
response = llm.invoke("Summarize this ticket in two sentences.")
print(response.content)
Run it and you should see a two-sentence summary printed. Set model to the ID from /v1/models, or the --served-model-name you gave vLLM. For a preset, base_url is the HTTPS endpoint plus /v1 and api_key is your inference-scope key. Against your own engine without a key, api_key still has to be a non-empty string, because the client refuses to start without one.
Streaming works as it does against a public API, because LangChain reads Server-Sent Events from /v1/chat/completions. Tokens should print one after another:
for chunk in llm.stream("Explain KV cache in two sentences."):
print(chunk.content, end="", flush=True)
Embeddings, same pattern
If you serve an embedding model next to your chat model, OpenAIEmbeddings takes the same arguments. That gives you a retrieval chain that runs entirely on your instance:
from langchain_openai import OpenAIEmbeddings
embeddings = OpenAIEmbeddings(
model="bge-large-en",
base_url="http://<spark-host>:8000/v1",
api_key="not-needed-unless-you-put-a-gateway-in-front",
)
This only works if your engine serves an embedding model. The console presets serve chat models only, so you run the embedding model yourself. vLLM serves embeddings through the same OpenAI-compatible API when the loaded model supports it; check /v1/models and the engine's docs before you wire this in.
Agents and tool calling
LangChain's tool-calling agents (create_tool_calling_agent or LangGraph's prebuilt agents) need the model to emit function calls in the OpenAI format. Not every open model does this reliably. When it fails, you usually get a plausible answer that never called the tool, not an error. Test a single-tool agent against your model before you build anything larger on it. On GPUwerk's console presets, the llama.cpp presets start with --jinja, which enables tool calls; the Qwen3 Coder 30B vLLM preset does not start with vLLM's --enable-auto-tool-choice, so tool-calling agents need a different preset or your own vLLM server with that flag.
Adding a gateway in front
With LiteLLM between LangChain and the engine, change base_url to the gateway's address, api_key to a virtual key and model to the alias from LiteLLM's config. You get per-application keys, budgets, and fallback to a second Spark or a cloud model, and the rest of the chain stays the same:
llm = ChatOpenAI(
model="spark-qwen",
base_url="http://localhost:4000/v1",
api_key="sk-your-virtual-key",
)
What actually breaks
- A 404 that reads like a connection failure. Cause: the
modelstring does not match what the server has loaded. Fix: copy the ID from/v1/models. - 401 or 403 from a preset endpoint. Cause: missing key, or a key without the
inferencescope. Fix: create an inference-scope key in the console and pass it asapi_key. - The agent answers but never calls the tool. Cause: the model does not emit OpenAI-style function calls reliably. Fix: test tool calling on that model first, or pick a model that supports it.
- Errors once history or retrieved context grows. Cause: the prompt passed the server's context limit, which is set when the model is served, not in LangChain. Fix: raise
--max-model-lenon your own engine, or trim history and retrieved chunks. - Streamed responses arrive all at once. Cause: a reverse proxy or load balancer buffers Server-Sent Events. Fix: turn off response buffering on it.
ChatOpenAIimport fails or behaves differently. Cause: older tutorials import it fromlangchain.chat_models; it now lives in the separatelangchain-openaipackage. Fix: import fromlangchain_openaiand pin that package's version.
LangGraph and other LangChain-adjacent tools
LangGraph uses the same chat model objects as LangChain, so the ChatOpenAI instance above works as a node in a LangGraph graph with no extra wiring. The same goes for any library that accepts a LangChain BaseChatModel instead of shipping its own API client: point that one instance at your Spark and everything built on it follows.