Docs/SDKs
SDKs

LangChain against your own DGX Spark endpoint

By Samuel Seidel · Updated September 9, 2026 · 6 min read

This page points an existing LangChain chain or agent at a model on your own DGX Spark. You change two constructor arguments on ChatOpenAI, base_url and api_key, and nothing else in your code. That works because LangChain's OpenAI integration wraps the client from the openai Python package.

You need:

On this page
  1. Start with a running endpoint
  2. ChatOpenAI against your Spark
  3. Embeddings, same pattern
  4. Agents and tool calling
  5. Adding a gateway in front
  6. What actually breaks
  7. LangGraph and other LangChain-adjacent tools

Start with a running endpoint

  1. Pick the endpoint. With a console model preset, copy the HTTPS endpoint from the instance page (it looks like https://app-<instance-id>.gpuwerk.com) and add /v1; pass your inference-scope API key as api_key. With your own vLLM server, <spark-host> below is your instance's hostname, <name>.gpuwerk.com. Start it with --api-key or reach it through an SSH tunnel, because the instance's public IPv4 address makes an open port reachable from the internet. Put LiteLLM in front if several applications will call the model.
  2. Confirm it answers before you write any Python:
    curl http://<spark-host>:8000/v1/models
    For a preset, add -H "Authorization: Bearer $GPUWERK_API_KEY" and use its HTTPS URL. You should see JSON listing the model ID, for example qwen3-coder-30b.

ChatOpenAI against your Spark

LangChain's ChatOpenAI class (in langchain-openai) takes base_url and api_key arguments that map directly onto the underlying OpenAI client's constructor:

from langchain_openai import ChatOpenAI

llm = ChatOpenAI(
    model="Qwen/Qwen3.8-27B-FP8",
    base_url="http://<spark-host>:8000/v1",
    api_key="not-needed-unless-you-put-a-gateway-in-front",
)

response = llm.invoke("Summarize this ticket in two sentences.")
print(response.content)

Run it and you should see a two-sentence summary printed. Set model to the ID from /v1/models, or the --served-model-name you gave vLLM. For a preset, base_url is the HTTPS endpoint plus /v1 and api_key is your inference-scope key. Against your own engine without a key, api_key still has to be a non-empty string, because the client refuses to start without one.

Streaming works as it does against a public API, because LangChain reads Server-Sent Events from /v1/chat/completions. Tokens should print one after another:

for chunk in llm.stream("Explain KV cache in two sentences."):
    print(chunk.content, end="", flush=True)

Embeddings, same pattern

If you serve an embedding model next to your chat model, OpenAIEmbeddings takes the same arguments. That gives you a retrieval chain that runs entirely on your instance:

from langchain_openai import OpenAIEmbeddings

embeddings = OpenAIEmbeddings(
    model="bge-large-en",
    base_url="http://<spark-host>:8000/v1",
    api_key="not-needed-unless-you-put-a-gateway-in-front",
)

This only works if your engine serves an embedding model. The console presets serve chat models only, so you run the embedding model yourself. vLLM serves embeddings through the same OpenAI-compatible API when the loaded model supports it; check /v1/models and the engine's docs before you wire this in.

Agents and tool calling

LangChain's tool-calling agents (create_tool_calling_agent or LangGraph's prebuilt agents) need the model to emit function calls in the OpenAI format. Not every open model does this reliably. When it fails, you usually get a plausible answer that never called the tool, not an error. Test a single-tool agent against your model before you build anything larger on it. On GPUwerk's console presets, the llama.cpp presets start with --jinja, which enables tool calls; the Qwen3 Coder 30B vLLM preset does not start with vLLM's --enable-auto-tool-choice, so tool-calling agents need a different preset or your own vLLM server with that flag.

Adding a gateway in front

With LiteLLM between LangChain and the engine, change base_url to the gateway's address, api_key to a virtual key and model to the alias from LiteLLM's config. You get per-application keys, budgets, and fallback to a second Spark or a cloud model, and the rest of the chain stays the same:

llm = ChatOpenAI(
    model="spark-qwen",
    base_url="http://localhost:4000/v1",
    api_key="sk-your-virtual-key",
)

What actually breaks

LangGraph and other LangChain-adjacent tools

LangGraph uses the same chat model objects as LangChain, so the ChatOpenAI instance above works as a node in a LangGraph graph with no extra wiring. The same goes for any library that accepts a LangChain BaseChatModel instead of shipping its own API client: point that one instance at your Spark and everything built on it follows.

See alsoPut a gateway in front for keys and budgets See alsovLLM on a DGX Spark
First top-up: pay $20, get $30 in credit

Run your chains against a dedicated Spark.

Deploy a dedicated DGX Spark in Europe, North America or East Asia and point LangChain at it in the time it takes to read this page.

Create account