Fine-tune a model on your company's data, without your data leaving the building

Inference sends one prompt at a time. Fine-tuning sends everything, every support ticket, every contract, your whole codebase, through someone else's machine, and returns weights that have memorised it. It is the workload where the privacy stakes are highest, and the one people are most casual about. This guide covers what actually fits in 128 GB of unified memory, what it costs you in wall-clock time, and when you shouldn't be fine-tuning at all.

First, the uncomfortable question: should you fine-tune?

Most teams who ask us about fine-tuning should be doing retrieval instead. The distinction that holds up in practice: RAG changes what the model knows; fine-tuning changes how the model behaves. Winder.ai's 2026 decision framework puts it as retrieval handling knowledge that changes over time, and fine-tuning handling behaviour that shouldn't. BigDataBoutique is blunter, most teams should fix their prompts, build a real retrieval pipeline, and write evals, in that order, before touching a training run.

In production these are not rivals. The common shape, and the one Atlan, Winder and the rest of the 2026 guidance converge on, is fine-tune for form, retrieve for fact: a LoRA adapter that has internalised your output structure and vocabulary, fed current documents at inference time. That combination is also the one that keeps your most sensitive corpus on hardware you control, because both halves run locally.

What actually fits: the memory table

Three methods, in ascending order of memory appetite. QLoRA quantises the frozen base model to 4-bit and trains small adapters on top. LoRA keeps the base in 16-bit and trains the same adapters, better fidelity, roughly 4× the base-weight footprint. Full fine-tuning updates every parameter and needs room for optimizer states too, which is why it stays small.

The numbers below are Unsloth's published minimum VRAM requirements per model size, their documentation flags them as absolute minimums at modest sequence lengths, so treat them as a floor, not a budget. The verdict column is against the DGX Spark's 128 GB of coherent unified memory.

The headline for anyone shopping: a 70B-class model is QLoRA-tunable on one machine that plugs into a wall socket. Unsloth's floor is 41 GB, and NVIDIA's own developer blog notes plainly that these fine-tuning workloads cannot run on a 32 GB consumer GPU. The 128 GB is not there to be filled by the base weights, it's there so that after the weights you still have room for gradient checkpointing off, longer sequences, and a batch size above one, which is where training runs actually get fast.

One caveat worth internalising: these floors assume short sequences. Multiple 2026 sizing guides (Spheron, VRLA Tech) note that the published minima assume sequence lengths around 512 with batch size 1 and gradient checkpointing on, and that doubling to 1024 tokens costs roughly 1.5–2× the activation memory. If your training examples are long contracts or whole files, size for your real sequence length, not the table.

Choosing a toolchain

• Unsloth: hand-written Triton kernels for the hot paths. The fastest way to a working LoRA run on a single box, and the project publishes the memory table above. There is community work on GB10 specifically, including Triton kernels for sm_121a reported as bit-identical in loss curve to stock. Start here.
• Hugging Face PEFT + TRL: the reference implementation. Slower than Unsloth, but it is what every paper and blog post assumes, and every adapter it produces loads anywhere. Use it when you want boring and portable.
• Axolotl: YAML-configured training. When you want the run to be a file in git rather than a notebook someone ran once, this is the ergonomic win.
• torchtune: PyTorch-native recipes, minimal dependency surface. Good fit for teams who already live in PyTorch and dislike framework magic.
• LLaMA-Factory: worth naming because it's what shows up in DGX Spark literature: a 2026 arXiv paper on LoRA fine-tuning for merchant-information extraction trained its models with LLaMA-Factory on a GB10 Spark, logging 462 GPU-hours of retained training plus 195 hours of checkpoint evaluation on that single machine.

Worked example: LoRA on a 32B, on one node

A concrete run, a 32B base, 16-bit LoRA, on your own support ticket history. Per the table that's around 76 GB of base and adapter state, leaving real room inside 128 GB for sequence length and batch.

Your dataset is a JSONL file of chat-formatted examples. It lives on the node's local NVMe, under /workspace, and is never uploaded anywhere:

# /data/tickets.jsonl, one resolved ticket per line
{"messages":[{"role":"user","content":"Invoice 8812 shows VAT twice…"},
             {"role":"assistant","content":"Duplicate VAT lines come from…"}]}

# train.py, Unsloth LoRA on a 32B base
from unsloth import FastLanguageModel
from trl import SFTTrainer, SFTConfig
from datasets import load_dataset

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name    = "Qwen/Qwen3-32B",
    max_seq_length = 4096,
    load_in_4bit  = False,      # 16-bit LoRA, we have the memory
)

model = FastLanguageModel.get_peft_model(
    model,
    r = 32, lora_alpha = 32, lora_dropout = 0.0,
    target_modules = ["q_proj","k_proj","v_proj","o_proj",
                      "gate_proj","up_proj","down_proj"],
    use_gradient_checkpointing = "unsloth",
)

SFTTrainer(
    model = model, tokenizer = tokenizer,
    train_dataset = load_dataset("json", data_files="/data/tickets.jsonl")["train"],
    args = SFTConfig(
        per_device_train_batch_size = 2,
        gradient_accumulation_steps = 8,   # effective batch 16
        num_train_epochs = 3,
        learning_rate = 1e-4,
        warmup_ratio = 0.03,
        lr_scheduler_type = "cosine",
        bf16 = True,
        output_dir = "/data/ckpt/tickets-32b",
        save_steps = 200,
    ),
).train()

model.save_pretrained("/data/adapters/tickets-32b")   # a few hundred MB

# run it overnight, detached, and watch memory as it warms up
nohup python train.py > /data/logs/run1.log 2>&1 &
watch -n5 nvidia-smi

# next morning: serve the base + adapter
vllm serve Qwen/Qwen3-32B --enable-lora \
  --lora-modules tickets=/data/adapters/tickets-32b \
  --host 0.0.0.0 --port 8000

Two details that matter more than the hyperparameters. First, the adapter is small, hundreds of megabytes against a 60+ GB base, which means your fine-tune is a portable artefact you can version, diff and revert without moving model weights around. Second, if you hit an out-of-memory error on unified memory, the community reports on GB10 consistently point at the same first moves: lower per_device_train_batch_size and raise gradient accumulation to hold the effective batch, cut max_seq_length to what your data actually needs, and keep gradient checkpointing on. Unified memory removes the offloading gymnastics a discrete GPU forces on you, but it is not infinite.

Honest expectations on speed

NVIDIA's developer blog publishes fine-tuning throughput for the Spark: a peak of 5,079.4 tokens per second tuning Llama 3.3 70B with QLoRA, 53,657.6 tokens per second for LoRA on Llama 3.1 8B, and 82,739.2 tokens per second full fine-tuning a Llama 3.2 3B. Treat these as NVIDIA's best-case peaks, the same blog's own summary table lists lower sustained figures, so plan capacity on the conservative side. Community results line up with the general order of magnitude, one reported Gemma-3-4B LoRA run finished 3 epochs over 10,000 examples at batch size 4 in 10–12 hours.

Set expectations accordingly. A rented H100 will finish the same run faster; that is not in dispute and it is not the point. The relevant comparison is against your calendar, not against a datacenter GPU:

• A fine-tune is an overnight job, not an interactive one. You start it at 18:00 and evaluate at 09:00. Whether the machine was 2× or 4× faster is invisible at that granularity, you were asleep either way.
• The machine is yours for the whole night. There is no queue, no spot preemption killing you at step 4,000, and no per-hour meter making you rush the run you should have repeated.
• Iteration count beats step speed. Most fine-tuning projects fail on data quality and evaluation, not on throughput. A box that lets you run a bad experiment on Tuesday and a better one on Wednesday, at no marginal cost, gets you to a good adapter sooner than a faster GPU you're reluctant to book.
• Memory capacity is the binary constraint. Speed is a gradient; fitting is a yes or no. A 32 GB consumer card cannot run the 70B QLoRA job at any speed.

The privacy architecture

Here is the part that has nothing to do with benchmarks. When you fine-tune through a hosted API service, you upload the training set. Read that again in terms of what your training set actually is: your resolved support tickets, with customer names in them. Your contract archive. Your private repository. It is not a query, it is the corpus. And the resulting weights are a lossy but real encoding of it, sitting on infrastructure you don't administer.

The equivalent problem on rented shared GPUs is quieter but the same family: a multi-tenant node means your dataset is written to storage and memory on hardware whose other tenants and whose operators you have no visibility into, under whatever jurisdiction the region actually sits in.

What we build instead, in both delivery models:

• The data never leaves the box. Dataset, tokenizer cache, intermediate checkpoints, final adapter, all of it lives on the node's local NVMe. Training reads from local disk and writes to local disk. There is no upload step in the workflow above, because there is nothing to upload to. The one exception is stopping the instance, which copies /workspace to our storage in EU-Central so the node can be handed back to the pool, and restores it when you start again.
• On GPUwerk cloud: a dedicated physical node, in EU-Central. Not a VM slice, not a partitioned GPU, the whole Spark, root over SSH, no other tenant on the silicon. GDPR applies to a machine in the EU because it is in the EU, not because of a contractual clause about it.
• On-premise: it never touches any cloud at all. We deliver and install the Spark in your office or rack. It can run on a network segment with no route to the internet. For teams whose legal position is "the training data cannot leave the premises", that is the only architecture that is literally true rather than contractually true.
• The adapter is yours and it is portable. A few hundred megabytes you can back up, escrow, or destroy. No vendor holds a copy of a model derived from your crown jewels, because no vendor ever saw the data.

The convenient part is that the two arguments point the same way. The workload with the highest privacy stakes is also the one whose defining constraint, memory capacity, not raw speed, is exactly what a 128 GB unified-memory machine is good at, and whose duty cycle, a long job overnight, then idle, is exactly what makes a dedicated box economical rather than wasteful.

A sensible first project

• Start at 8B, not 70B. Prove the pipeline and the data format on a model that trains in an hour. Scale up only when a run is boring.
• Write the eval before the training script. Fifty held-out examples and a rubric. Without it you cannot tell a good adapter from a confident one.
• Clean 1,000 examples rather than dumping 100,000. Every practitioner report says the same thing, and it remains the highest-leverage hour you'll spend.
• Try it on a rented node first. The run above is identical on a GPUwerk node and on a Spark in your own rack, same GB10, same DGX OS. One overnight job answers "does fine-tuning help us" for the price of a dinner, and if the answer is yes, you already know exactly which machine to buy.