Get your files out of a rented Spark

GPUwerk keeps no backup of /workspace. Terminating an instance deletes it, and so does a failed drive. Copy anything you want to keep to your own machine or to Hugging Face before you terminate, and from time to time during a long run.

When your files are deleted

• Terminate: /workspace is deleted at once, for good. Billing ends.
• Stop (from the console): the machine stays held for you at 75% of the hourly rate and nothing is deleted.
• Powering off from inside (poweroff, shutdown): the instance stops but is not terminated, and nothing is charged from then on. If you do not start it again within about 30 minutes, the machine is released and the workspace kept on it for up to 48 hours, as below.
• Credit runs out or your budget is reached: the instance is stopped and the machine released. The workspace stays on it for up to 48 hours, and is deleted sooner if someone else rents that machine.
• A partner Spark is withdrawn or goes offline: the workspace is kept for 24 hours so you can copy it off if the machine is reachable, then wiped.
• The drive fails: the workspace is lost. There is no copy.

Copy a folder with scp

Run this on your own computer, not on the instance. It copies /workspace/results into the current folder:

scp -r root@spark-abcd.gpuwerk.com:/workspace/results ./

To pack many small files first, which is faster over the network:

ssh root@spark-abcd.gpuwerk.com 'tar -C /workspace -czf - results' > results.tar.gz

Sync large or changing folders with rsync

rsync copies only what changed and can resume after a dropped connection, so it suits checkpoints and long runs. It has to be installed on both ends. On the instance:

apt-get update && apt-get install -y rsync

Then, on your own computer:

rsync -avz --partial --progress \
    root@spark-abcd.gpuwerk.com:/workspace/checkpoints/ ./checkpoints/

Run it again later and it fetches only new or changed files. Leave out -z for files that are already compressed, such as .safetensors or .gguf.

Push models and datasets to Hugging Face

For a fine-tuned model or a large dataset, uploading straight from the instance to a private Hugging Face repository is usually faster than downloading to a laptop. On the instance:

pip install -U huggingface_hub
hf auth login                      # paste a token with write access
hf upload your-name/my-finetune /workspace/out . --private

For a dataset, add --repo-type dataset. The token is stored under ~/.cache/huggingface on the instance; log out with hf auth logout before you hand the machine back, or create a token you can revoke afterwards.

Check before you terminate

• Compare sizes: du -sh /workspace/results on the instance against the copy you made.
• Remember files outside /workspace: packages you installed, files in /root and, on a system container, units in /etc/systemd. Write down or copy anything you will need to set up again.
• Terminate from the console only when the copy is complete. It asks you to type the instance name because it cannot be undone.