Guides
Longer arguments about why any of this is worth doing on dedicated hardware. Written by Samuel Seidel, who runs the GPUwerk fleet, and opinionated where there is a view to hold.
Migrating and comparing costs
Whether self-hosting is worth it, and how to move if it is.
On-premise LLM cost vs cloud API cost: how to actually compare them
Flat hourly infrastructure vs per-token billing, and the crossover-point method worked with GPUwerk's own hourly rate.
DGX Spark rental vs building your own AI workstation
A multi-GPU workstation costs more than the graphics cards suggest, and VRAM per card caps what you can run. When building wins and when renting does.
Private AI
What leaves the building when a team uses someone else's model, and what does not have to.
Private AI customer support without sending customer data to a third party
Routing tickets through a third-party AI API means every name, email and order number in them passes through that vendor. What a dedicated self-hosted model changes.
Private AI document translation for sensitive documents
Legal, medical, and financial documents often can't go through a consumer translation tool. Open-weight translation models running on your own hardware, and how to set that up.
Building a private AI internal knowledge base
Turning internal wikis, tickets and documents into a searchable assistant without sending any of it to a third-party API.
Private AI for HR and recruiting
Resumes, performance reviews and candidate notes carry data an external AI vendor shouldn't see. What changes on a self-hosted setup.
Self-hosted AI for financial analysis
Running models over spreadsheets, forecasts and deal data without routing any of it through an outside API.
AI-assisted contract review without sending contracts to a third party
Clause extraction and redline suggestions on a dedicated model instead of a third-party AI API.
Private AI for contract negotiation support
Using a self-hosted LLM to draft redlines and negotiating positions during an active deal, without sending your fallback terms to a third party.
Private AI as an executive assistant for scheduling and correspondence
Using a self-hosted LLM to draft calendar coordination and executive correspondence without sending board and deal scheduling context to a third party.
Private AI for product requirements and spec drafting
Using a self-hosted LLM to turn raw product notes into PRDs and specs without sending unreleased roadmap and positioning to a third party.
Private AI for investor relations and board reporting
Using a self-hosted LLM to draft board decks and investor updates without sending material non-public information to a third party.
Private AI for compliance monitoring and audit preparation
Using a self-hosted LLM to assist internal audit prep and policy-gap analysis drafts without sending control evidence to a third party.
Private AI for warranty claims review without sending claim data to a third party
Using a self-hosted LLM to draft warranty claim summaries and flag anomalies for an adjuster, instead of routing product and customer data through a third-party AI API.
Private AI for shift scheduling without sending employee data to a third party
Drafting a first-pass shift schedule with a self-hosted LLM instead of sending employee availability and pay data through a third-party AI API. A manager still finalizes it.
Private AI for vendor onboarding without sending supplier data to a third party
Using a self-hosted LLM to extract and check vendor onboarding documents instead of routing supplier contracts and banking details through a third-party AI API.
Private AI for property listing descriptions without sending client data to a third party
Drafting property listing copy with a self-hosted LLM instead of sending client and property details through a third-party AI API. An agent still reviews every listing.
Private AI for return and refund processing without sending order data to a third party
Using a self-hosted LLM to draft return and refund case summaries instead of routing order and payment data through a third-party AI API. It doesn't approve refunds on its own.
Private AI for QA and software testing
Using a self-hosted LLM for test case generation, code review, and bug triage without sending a proprietary codebase to a third-party API.
Private AI for DevOps and runbook automation
Using a self-hosted LLM to draft runbooks, summarize incident logs, and assist on-call, keeping infrastructure details off third-party servers.
Private AI for meeting transcription and notes
Self-hosted transcription and summarization for meetings with sensitive content, legal, HR, M&A, where the recording can't sit on a third-party server.
Private AI for procurement and RFP response drafting
Using a self-hosted LLM to draft RFP responses and analyze vendor proposals without exposing bid strategy or vendor pricing to a third party.
Private AI for internal communications and email drafting
Using a self-hosted LLM to draft internal comms and HR-sensitive email before an announcement is public, without the draft passing through a third party.
Private AI for sales and CRM data enrichment
Using a self-hosted LLM to enrich CRM records and draft outreach without sending prospect and customer data to a third-party AI vendor.
Private AI for translation and localization
Using a self-hosted LLM for ongoing UI string and marketing copy localization without routing every string change through a third-party AI vendor.
Private AI for customer support ticket triage
Using a self-hosted LLM to classify, route, and draft responses to support tickets without sending customer PII to a third-party AI vendor.
Private AI for research and literature review
Using a self-hosted LLM to summarize papers and internal R&D documents without exposing unpublished findings to a third-party AI vendor.
Private AI for data labeling and annotation
Using a self-hosted LLM to pre-label and check annotator consistency on sensitive training data without sending it to a third-party annotation vendor.
Private AI for M&A due diligence document review
A data room full of unsigned material non-public information is a bad place to route through a third-party AI API. What changes when the review model runs on a dedicated machine.
Private AI for patent drafting and IP documentation
An unfiled invention disclosure sent to a third-party AI vendor is disclosure to a party outside your organization. What changes when the drafting model runs on a dedicated machine before you file.
Private AI for crisis communications drafting
A holding statement drafted before a crisis is confirmed shouldn't sit on a third-party AI vendor's servers. What a self-hosted drafting model changes for comms teams.
Private AI for employee onboarding and training content
Internal process documentation reveals how a company actually operates. What changes when the drafting model runs on a dedicated machine instead of a vendor's API.
Private AI for competitive intelligence analysis
Analysis built from public sources still reveals your strategy once you've synthesized it. What changes when the summarizing model runs on a dedicated machine instead of a vendor's API.
Private AI for financial modeling
Asking an AI assistant to help build or stress-test a financial model means describing unreleased numbers to it. Why that belongs on a dedicated model, not a shared AI service.
Private AI for support response drafting
Drafting support replies from your knowledge base with AI usually means sending customer names, order details and account history to a third party. Keeping the model in-house avoids that.
Private AI for technical documentation
Generating and keeping internal docs current from your own codebase with AI usually means sending that codebase to a third party. Running the model in-house avoids that trade.
Private AI for vendor risk assessment
Drafting vendor security questionnaires and risk assessments with AI usually means describing your own vendor list, and its weaknesses, to an outside model. Keeping that in-house avoids the irony.
Private AI for performance review drafting
Performance review drafts contain the most sensitive opinions a manager writes down about a named employee. What changes when the drafting model runs on a dedicated machine instead of a vendor's API.
Private AI for grant proposal drafting
A grant proposal draft often contains unpublished preliminary data and a strategy the funder shouldn't see reused elsewhere. What changes when the drafting model runs on a dedicated machine instead of a vendor's API.
Private AI for incident postmortem drafting
A postmortem draft is built from logs, stack traces and internal system names, exactly the detail an attacker would want. What changes when the drafting model runs on a dedicated machine, not a shared AI API.
Private AI for sales call summaries
A sales call transcript contains a prospect's pricing objections, budget and competitor comparisons in their own words. What changes when the summarizing model runs on a dedicated machine, not a shared AI API.
Private AI for internal audit workpapers
Internal audit workpapers document exactly where a company's financial controls are weak, before management has fixed anything. What changes when the drafting model runs on a dedicated machine, not a shared AI API.
Private AI for board meeting prep
Board decks and briefing packets carry pre-announcement numbers, pending litigation and personnel changes months before any public filing. What changes when the drafting model runs on a dedicated machine, not a shared AI API.
Private AI for technical interview question banks
A leaked interview question bank stops working the moment it's leaked. Why generating and storing a question bank on a shared AI service creates a copy of it you don't control.
Private AI for press release drafting
A press release is a document written to be secret right up until it isn't. What changes when the drafting model handling pre-announcement language runs on a dedicated machine, not a shared AI API.
Private AI for internal wiki search
An internal wiki accumulates architecture notes, incident writeups and process docs that are useful precisely because they're candid. What changes when the search model runs on a dedicated machine instead of a vendor's API.
Private AI for expense report review
Expense reports reveal travel patterns, vendor relationships and spending habits down to the individual employee. What changes when the model flagging policy violations runs on a dedicated machine, not a shared AI API.
Private AI for resume screening without sending resumes to a third party
Running a first-pass resume screen on a self-hosted LLM instead of a third-party AI API. What it's good for, what it isn't, and where the recruiter still has to decide.
Private AI for insurance claims processing without sending claims to a third party
Using a self-hosted LLM to summarize and triage insurance claims instead of routing medical and financial claim data through a third-party AI API. Not a fraud-detection system.
Private AI for social media content drafting without sending strategy to a third party
Drafting social posts and campaign copy on a self-hosted LLM instead of a third-party AI API, so unreleased plans and brand voice data stay off vendor infrastructure.
Private AI for podcast show notes without sending recordings to a third party
Transcribing and summarizing podcast episodes into show notes on a self-hosted LLM instead of a third-party transcription API, so unreleased audio and guest details stay away from a vendor's API.
Private AI for fraud detection review without sending transaction data to a third party
Using a self-hosted LLM to help analysts write up flagged transactions instead of routing account and transaction data through a third-party AI API. Augments a fraud system, doesn't replace it.
AI-assisted loan underwriting support without sending applicant financials to a third party
Using a self-hosted LLM to summarize loan files and flag items for a human underwriter, keeping applicant financial data away from third-party AI APIs. Not a credit decision tool.
AI-assisted lease abstraction without sending lease contracts to a third party
Using a self-hosted LLM to extract key terms from commercial leases and flag items for a human reviewer, keeping lease data away from third-party AI APIs. A drafting aid, not a substitute for counsel.
AI-assisted safety incident reports without sending incident details to a third party
Using a self-hosted LLM to draft and structure workplace safety incident reports, keeping incident details away from third-party AI APIs. A drafting aid, not a substitute for regulatory filing by qualified staff.
AI-assisted job description writing without sending req details to a third party
Using a self-hosted LLM to draft job descriptions from internal req details, keeping salary bands and hiring plans away from third-party AI APIs. A drafting aid, not a substitute for HR and legal review.
AI-assisted customer churn analysis without sending transaction data to a third party
Using a self-hosted LLM to summarize churn signals from customer transaction and usage data, keeping it away from third-party AI APIs. A drafting aid for the analysis, not a replacement for a retention strategy.
Private AI for e-discovery document review
Litigation document sets contain privileged material and the other side's confidential business records. What changes when the model doing first-pass review runs on a dedicated machine, not a shared AI API.
Private AI for change management communications
Reorg plans, layoff timelines and merger messaging are among the most sensitive drafts a company writes before they're announced. What changes when the drafting model runs on a dedicated machine, not a shared AI API.
Private AI for security policy drafting
A security policy draft describes exactly how the company defends itself, gaps included, before the controls are in place. What changes when the drafting model runs on a dedicated machine, not a shared AI API.
Private AI for equipment maintenance logs
Maintenance logs map out exactly where a facility's equipment is aging, unmonitored or overdue for service. What changes when the model summarizing them runs on a dedicated machine, not a shared AI API.
Private AI for payroll query support
A payroll question touches salary, benefits and tax data for a named employee. What changes when the assistant answering those questions runs on a dedicated machine, not a shared AI API.
Agents and automation
The work that pays for the machine is usually unattended, not chat.
An AI employee needs an office. Give it one you own.
An agent with your mail, files and credentials is a staff member with root. House it accordingly.
Automate the boring 80%, with an LLM that never bills per token
On a flat-rate machine the marginal token costs nothing, so it pays to run the model over everything.
Self-hosted AI agent infrastructure
Agent loops call a model dozens of times per task, which is why per-token API bills go unpredictable. What running agents against your own endpoint changes.
Self-hosted coding assistant vs GitHub Copilot
Running Qwen3-Coder on your own Spark against Copilot's cloud model: what stays private, what it costs at team scale, and where each one wins.
Building with models
The technical decisions behind putting a model to work on your own data.
RAG vs agents: what's the actual difference
RAG injects context into one generation call, agents make multi-step decisions and take actions, and most real systems use both.
Best GPU for running open-source LLMs privately
Memory capacity vs bandwidth, single-user vs shared serving, and where a DGX Spark, a consumer GPU, a datacenter card, or a Mac Studio each fit.
Open-weight vs closed-weight models: how to choose
When to use a closed API and when to self-host an open-weight model: data control, cost at your volume, and customization.
Batch vs interactive AI workloads and why it changes hardware choice
Prefill-bound and decode-bound workloads stress a GPU differently, so a model can be excellent at one and mediocre at the other on identical hardware.
Self-hosted speech and voice AI
Running speech-to-text and text-to-speech models on your own hardware instead of streaming audio to a cloud API.
What is token-based pricing?
Token-based pricing bills per input and output token consumed, so cost scales with how much text goes in and out of a model. How it works, and how it compares with a flat hourly rate.
What is a context window?
A context window is the maximum number of tokens a model can attend to at once, prompt and reply combined. What sets that limit, and what happens when a conversation grows past it.
What is an open-weight model?
An open-weight model is one whose trained parameters are published for anyone to download and run, regardless of whether the training data, code, or license are also open.
Inference vs training: what's the difference?
Training produces a model's weights; inference uses those weights to answer a specific input. The two phases have different hardware needs, and most self-hosted deployments only ever do the second.
What is a system prompt?
A system prompt is the instruction text given to a language model before a conversation begins, setting its behavior, role, and constraints ahead of anything the user asks.
What is an LLM agent?
An LLM agent is a model wrapped in a loop that can call tools, observe the results, and decide its next step across multiple turns, instead of producing one response and stopping.
What is temperature in LLM sampling?
Temperature is a setting that controls how much randomness an LLM uses when picking its next word. Low temperature favors the most likely token; high temperature spreads probability across more options.
What is a token (in LLMs)?
A token is the chunk of text a language model actually reads and generates, usually a word piece a few characters long rather than a whole word or a single letter.
What is a GPU node?
A GPU node is a single machine, physical or virtual, that has one or more GPUs attached and is treated as one unit of compute in a cluster or rental fleet.
Latency vs throughput in LLM inference
Latency is how long a single request takes to get a response; throughput is how many requests a system can handle over a given period. Optimizing one can work against the other.
What is a model weight?
A model weight is a single numeric parameter inside a neural network, learned during training, that scales how strongly one signal influences the next as data passes through the model.
What is a model checkpoint?
A model checkpoint is a saved snapshot of a neural network's weights, and often its training state, at one specific point, letting training resume or a specific version be reloaded later.
What is prompt injection?
Prompt injection is an attack where text fed into an LLM, whether typed directly or hidden in a document or webpage the model reads, tricks it into ignoring its original instructions.
What is model distillation?
Model distillation is training a smaller student model to mimic a larger teacher model's outputs, aiming to approximate its behavior at a fraction of the size and compute cost.
What is Model Context Protocol (MCP)?
Model Context Protocol (MCP) is an open specification for connecting an LLM to external tools and data sources through a standard interface, instead of a one-off integration per tool.
What is an LLM hallucination?
An LLM hallucination is a confident, fluent output that is factually wrong or unsupported by the input, produced because the model is predicting plausible text, not verifying claims.
What is zero-shot vs few-shot prompting?
Zero-shot prompting asks a model to perform a task with no examples given; few-shot prompting includes a handful of worked examples in the prompt to steer the model's output format and style.
What is an inference server?
An inference server is the software that loads a trained model into memory and exposes it over an API, handling requests, batching, and scheduling so clients don't talk to the model directly.
What is speculative decoding?
Speculative decoding uses a small draft model to guess several tokens ahead, then lets the larger model verify them in one pass, turning several slow steps into one.
What is batching in LLM inference?
Batching groups multiple requests so one read of the model's weights from memory serves all of them, which is how a bandwidth-bound GPU turns idle capacity into aggregate throughput.
Capacity planning for LLM workloads
How many concurrent users one Spark serves and when you need a second node: bandwidth arithmetic and measured concurrency knees for a 30B MoE and a dense 70B.
What is flash attention?
Flash attention is a fused, memory-IO-aware way of computing attention that avoids materializing the full attention matrix, cutting memory traffic and enabling longer context at the same memory budget.
What is a reasoning model?
A reasoning model is trained to produce an extended chain of intermediate steps before its final answer, trading more inference-time compute for better accuracy on harder problems.
What is NVLink?
NVLink is NVIDIA's high-bandwidth GPU-to-GPU interconnect, used to link multiple discrete GPUs so they can share memory and work together on one model much faster than over PCIe.
What are guardrails in LLM apps?
Guardrails are input and output filtering layers wrapped around an LLM: content moderation, PII redaction, schema validation, and jailbreak detection, separate from the model's own training.
What is a draft model?
A draft model is the small, fast model in speculative decoding that proposes candidate tokens for a larger target model to verify in one batched pass.
What is model serving?
Model serving is the discipline of running a trained model behind an API for inference at scale: batching, routing, autoscaling, and health checks, as opposed to a one-off inference script.
What is paged attention?
Paged attention manages the KV cache in fixed-size blocks instead of one contiguous allocation per sequence, cutting the memory wasted on fragmentation and over-reservation during LLM serving.
What is a transformer?
A transformer is the neural network architecture behind almost every current LLM, built from stacked attention and feed-forward layers that let every token weigh every other token directly.
What is rotary position embedding (RoPE)?
Rotary position embedding encodes a token's position by rotating its query and key vectors, so attention scores naturally depend on the relative distance between tokens rather than their absolute positions.
What is grouped-query attention?
Grouped-query attention has multiple query heads share a smaller set of key and value heads, cutting KV cache size and memory bandwidth per token with little effect on output quality.
Want the hands-on version?
The docs cover the commands: SSH keys, vLLM flags, model sizing and benchmarks measured on GPUwerk's own fleet.
Read the docs Create accountFirst top-up: pay $10, get $20 in credit, about 25 hours on a dedicated Spark.