Backup and disaster recovery for your DGX Spark workload
Backing up a rented Spark is your job, and this page covers what to back up, where to send it, and how often. GPUwerk keeps no copy of /workspace, not even one to recover from a failed drive. The pricing page says so plainly: that is how the system is built, not a gap to work around.
Why there's no platform backup
A Spark instance is one workspace on one node, and that node's drive holds the only copy GPUwerk has. If the drive fails, the workspace is lost. The pricing page's callout is equally direct: "Protect your results... Keep your own backup." If your balance hits zero or you hit a monthly budget, the instance is stopped and its machine released. The workspace stays on that machine for up to 48 hours: topping up and pressing start in that time picks up where you left off, unless someone else has rented the machine, which deletes it sooner. After that it is deleted for good, per the storage and persistence guide. A manual terminate deletes the workspace straight away, with no 48-hour window and no copy kept.
What's actually worth backing up
Not everything in /workspace is equally valuable. Sort it into two piles:
- Irreplaceable, back it up: fine-tuned or LoRA weights you trained yourself, configuration files (LiteLLM's
config.yaml, systemd units, environment files), any dataset you curated or cleaned that isn't a straight copy of a public source, and conversation or usage logs if you need them for audit or debugging later. - Reproducible, skip it: base model downloads from Hugging Face or similar (re-pulling them costs bandwidth and time, not data), and installed packages and system dependencies, which belong in a setup script or Dockerfile you keep in version control, not in the backup.
The practical test: if losing it means redoing a training run or a curation pass you can't easily repeat, back it up. If losing it means re-running a script, write the script down instead of backing up its output.
Where to send it
GPUwerk doesn't provide or manage a backup destination, so you're picking your own. Reasonable options, all reachable from the Spark's terminal:
- S3-compatible object storage (AWS S3, Backblaze B2, or a self-hosted MinIO instance elsewhere) via
aws s3 syncorrclone, which handles incremental syncs well for large model weight directories. - rsync or scp to a machine you control: the simplest to set up, and fine for infrequent manual backups of a moderate amount of data.
- A git remote for config files, scripts and small datasets, which benefit from version history in a way that large binary weight files don't.
Model weight files can run into tens of gigabytes, so factor transfer and storage cost into the choice. GPUwerk includes egress in the hourly rate (see the pricing page), but your destination may charge for storage or incoming data.
How often
Match frequency to how much rework a loss would cost you, not to a fixed calendar schedule:
- A fine-tuning run in progress: checkpoint every few hours, or at each meaningful loss milestone, so a crash costs you an hour, not a day.
- A stable inference deployment: back up config after each change, and weights once, after training completes, not on a recurring schedule.
- Before any terminate: a final sync. Treat this one as non-optional, because terminate deletes the workspace immediately and the console has no undo.
Practical checklist
- Assume nothing outside your own backup destination survives a terminate, or a zero-balance stop: its workspace is gone once someone else rents the machine, and after 48 hours at the latest.
- Back up fine-tuned weights, config and curated data; skip base downloads and packages and keep those reproducible instead.
- Pick an S3-compatible target or a remote you control, and automate the sync rather than relying on remembering to run it.
- Always sync before you terminate, since the delete on confirm is immediate.
- Keep auto top-up on so an unplanned zero-balance stop doesn't catch you without a recent backup.
See the storage and persistence guide for the full stop-versus-terminate mechanics, or a first engagement if you want help designing a backup pipeline around a specific workload.