Train Models on Jobs

This page shows how to fine-tune and train models on Jobs, with a library’s script or your own. Every example is capped to a short run that finishes in minutes on a single A10G, for around $0.10 of compute. The Job runs on Hugging Face’s machines, so it keeps going if you close your terminal or your laptop. If you have not run a Job before, Quickstart covers installing the CLI, logging in and the credits a Job needs.

A first run

This command fine-tunes an image classifier on 2,000 Food-101 images and pushes it to your namespace on the Hub, in about three minutes:

hf jobs uv run --flavor a10g-small --timeout 30m -s HF_TOKEN -- \
  https://raw.githubusercontent.com/huggingface/transformers/main/examples/pytorch/image-classification/run_image_classification.py \
  --model_name_or_path google/vit-base-patch16-224-in21k \
  --dataset_name ethz/food101 \
  --do_train --do_eval \
  --remove_unused_columns False \
  --max_train_samples 2000 --max_eval_samples 500 --num_train_epochs 1 \
  --output_dir vit-food101 \
  --push_to_hub

The Transformers section covers the full run and the other example scripts.

How a training Job is put together

The command above has five parts, and every command on this page has the same ones.

• What runs. Either a uv script, launched with hf jobs uv run: a Python script that declares its dependencies in a comment block near the top (# /// script, the PEP 723 format), which uv installs into a fresh environment. The TRL section shows one. Or a library’s Docker image, launched with hf jobs run, which uses the library already installed in the image. Start with a uv script when the library installs with pip. Use the image when the library ships one with compiled dependencies, or when its docs say to. See Using Docker images for the trade-off.
• A token. Jobs get no Hugging Face token by default. -s HF_TOKEN forwards yours as a secret, so the run can push its model and read gated or private inputs. Other secrets travel the same way, for example -s WANDB_API_KEY.
• Hardware and time. --flavor picks the GPU. --timeout sets the time limit, which defaults to 30 minutes. A run that hits the timeout is stopped, so set --timeout above your expected run time. See Hardware flavor and Timeout.
• A -- between the hf flags and the script. Flags before -- are for hf jobs. After it come the script path and the script’s own arguments. Without it, a script argument that shares a name with an hf flag, such as --timeout or --token, is taken by hf. In the image form, what follows -- is the command to run in the container.
• Where the output goes. Every library below can push the finished model to a Hub repo, and each example shows the option. For Transformers and TRL the repo takes the --output_dir name. A run can also write to a mounted bucket as it goes, using -v to mount one into the container. See After it ends.

Your own training code. Use the simplest form that fits:

• One file. Declare its dependencies in a script header and run hf jobs uv run train.py. The TRL section shows one.

• A project folder with local imports, a pyproject.toml or config files. hf jobs uv run uploads only the script file, so mount the folder instead. The mount is read-only, so the command copies the project to a writable directory and runs it there, as you would locally:

  Copied

  hf jobs run --flavor a10g-small --timeout 30m -s HF_TOKEN \
    -v ./my-project:/code ghcr.io/astral-sh/uv:python3.12-bookworm -- \
    bash -c "cp -r /code /tmp/project && cd /tmp/project && uv run train.py --config configs/run.yaml"

  uv run installs dependencies from pyproject.toml or the script’s header, and local imports and relative paths work unchanged. See Local directories.

• Code that needs system packages or a CUDA toolkit. Build an image once and run it with hf jobs run. See Build your own image with a Docker Space.

A script can also carry its own launch config in a [tool.hf-jobs] table in the script header, as the TRL section shows. See Define the launch config in the script. The same commands are available from Python as run_uv_job() and run_job(), covered in Configuration.

Checks before a long run

Before you remove the caps, a few checks save a wasted run.

• Smoke-test first. Run the command with a step cap on a small flavor, as the examples on this page do. It proves the dependencies install, the data loads, the model fits and the push works. Then remove the cap and launch the full run.
• Check the data fits. Each flavor has a fixed disk, listed in the Ephemeral Storage column on Pricing and Billing. Weights, dataset and saved checkpoints share it. A dataset too big for the disk can be streamed or mounted instead of downloaded. See Process Large Datasets.
• Estimate the run time. The smoke test’s final train_* metrics include train_steps_per_second, and the trainer prints the total step count when it starts. Together they give the full run’s training time. Set --timeout comfortably above it, since the Job also spends time installing dependencies and downloading the model. Rates per flavor are on Pricing and Billing.
• On a multi-GPU flavor, start one process per GPU. Flavors ending in x2, x4 or x8 give several GPUs on one machine. Make sure the way you launch uses them. Transformers and TRL need accelerate launch, as the TRL section shows. Axolotl does it by itself. A plain python train.py uses one GPU, and a Trainer falls back to DataParallel, which is slower than one process per GPU.
• Pin if you will rerun. Pin a script URL to a commit instead of main, and an image to a specific tag instead of latest. A rerun then gets the same software. The script URLs below track main, so pin the commit before you rerun one.
• Checkpoint long runs to a mounted bucket, so a timeout or a crash does not lose the run. See After it ends.

While it runs

Both hf jobs run and hf jobs uv run stream the logs and hold your terminal until the run ends. Ctrl+C stops only the log stream. The Job keeps running until it finishes or you stop it with hf jobs cancel <job_id>.

For longer runs, pass -d (detach) to get the Job ID back straight away, then follow the run with hf jobs logs -f <job_id> and confirm the GPU is busy with hf jobs stats <job_id>. After a detached run, hf jobs wait <job_id> blocks until the Job ends and exits non-zero if it failed, which is what a script or an agent loop needs. A non-detached run already does this. See Manage Jobs.

Job logs print loss values as text. For curves, point the trainer at an experiment tracker such as trackio and, for a hosted tracker such as Weights & Biases, pass its key as a second secret.

After it ends

A Job’s disk is discarded when the Job ends, whether it finished, failed or timed out. Anything you want to keep has to leave the container before then.

Push the model to a Hub repo. Every library on this page has an option for it: --push_to_hub for Transformers and TRL, --output-repo for the Unsloth scripts, hub_model_id in an Axolotl YAML. At the end of the run the library uploads the weights, the tokenizer and a generated model card recording the base model and the training arguments. The repo is created if it does not exist. To make it private, create it first with hf repos create <name> --private. A fine-grained token needs write and create access to the model repo. Without it, the run trains to the end and then fails on the upload.

Write to a bucket as you go. For a run that takes hours, mount an existing Storage Bucket read-write (create one with hf buckets create) and point the library’s output directory at it. Checkpoints land in the bucket as they are saved, so a timeout or a crash does not lose the run, and the next Job can resume from them. The same route works for evaluation outputs, logs and anything else that is not a model.

hf jobs uv run --flavor a10g-large --timeout 8h -s HF_TOKEN \
  -v hf://buckets/your-username/checkpoints:/ckpt -- \
  train.py --output_dir /ckpt/run-01

Transformers and TRL scripts take --output_dir. Axolotl takes output_dir in the YAML. If you also pass --push_to_hub, set --hub_model_id too, or the repo is named after the last part of the output path (run-01). To continue an interrupted run, mount the same bucket again and pass the library’s resume option, such as --resume_from_checkpoint for a Transformers Trainer. See Volumes for the mount options.

Read a failed run. A Job that fails keeps its logs: hf jobs logs <job_id> works after it ends, and hf jobs inspect <job_id> gives the final status and error message. hf jobs logs -f returns when the log stream ends whether the run succeeded or not, so check inspect before assuming it worked.

Transformers

The example scripts in the Transformers repository declare their dependencies in a script header, so they run on Jobs straight from their GitHub URL. Arguments after the URL go to the script. A first run uses the image-classification script.

For the full run, drop --max_train_samples 2000 --max_eval_samples 500 --num_train_epochs 1 from that command: three epochs over the 75,000 Food-101 training images take about an hour on a10g-small, around $1 at that flavor’s rate, and reach 90% accuracy. Raise --timeout to 2h before you launch it. --push_to_hub uploads the model under your namespace using the output directory name. Scripts exist for text classification, summarization, translation, token classification, speech recognition and more.

TRL

TRL has a ready-to-run script for each trainer (SFT, DPO, GRPO and others), each declaring its own dependencies, so they run on Jobs straight from their URL, like the Transformers scripts. This command fine-tunes a small model on a chat dataset:

hf jobs uv run --flavor a10g-small --timeout 30m -s HF_TOKEN -- \
  https://raw.githubusercontent.com/huggingface/trl/refs/heads/main/trl/scripts/sft.py \
  --model_name_or_path Qwen/Qwen2-0.5B-Instruct \
  --dataset_name trl-lib/Capybara \
  --max_steps 100 \
  --output_dir Qwen2-0.5B-SFT \
  --push_to_hub

This finishes in about six minutes. Remove --max_steps for the full run: three epochs of Capybara, the script’s default, take about 2 h 20 min on a10g-small, so raise --timeout with it. TRL’s own docs use a100-large, which is faster.

The full guide, including writing your own TRL script and running the huggingface/trl image, is Training with Jobs in the TRL docs.

For several GPUs, switch to the TRL image, which ships accelerate, and let it start one process per GPU:

hf jobs run --flavor a10g-largex2 --timeout 30m -s HF_TOKEN huggingface/trl -- \
  accelerate launch --num_processes 2 -m trl.scripts.sft \
  --model_name_or_path Qwen/Qwen2-0.5B-Instruct \
  --dataset_name trl-lib/Capybara \
  --max_steps 100 \
  --output_dir Qwen2-0.5B-SFT \
  --push_to_hub

Without --max_steps, the full run (three epochs of Capybara, the script’s default) takes about 1 h 35 min on two A10Gs.

When you write your own TRL script, the launch config can travel with it. A [tool.hf-jobs] table in the header sets the flavor, timeout and secrets:

# /// script
# dependencies = ["trl"]
#
# [tool.hf-jobs]
# flavor  = "a10g-small"
# timeout = "1h"
# secrets = ["HF_TOKEN"]
# ///
from trl import SFTConfig, SFTTrainer
...

hf jobs uv run train.py then needs no flags, and a flag you do pass still wins. See Define the launch config in the script.

Unsloth

Unsloth provides ready-to-run scripts in the unsloth/jobs dataset, one per model family. They install Unsloth from their script header and take the dataset and output repo as arguments:

hf jobs uv run --flavor a10g-small --timeout 30m -s HF_TOKEN -- \
  https://huggingface.co/datasets/unsloth/jobs/resolve/main/sft-lfm2.5.py \
  --dataset mlabonne/FineTome-100k \
  --max-steps 50 \
  --output-repo your-username/lfm-finetuned

This finishes in about five minutes and pushes a LoRA adapter. For a full epoch, replace --max-steps 50 with --num-epochs 1 and raise --timeout. The walkthrough is Fine-tune with Unsloth on Jobs on the Hugging Face blog.

Axolotl

Axolotl takes a YAML config and runs from its own Docker image, so this section uses hf jobs run with a pinned tag and syncs the config in from a local directory with -v. Save a config from the Axolotl examples as ./configs/lora.yml and add the keys shown after the command. This example uses examples/phi/lora-3.5.yaml, a LoRA fine-tune of Phi-3.5-mini, otherwise unchanged. The image already contains the axolotl command.

hf jobs run --flavor a10g-small --timeout 30m -s HF_TOKEN \
  -v ./configs:/configs \
  axolotlai/axolotl:0.19.0-py3.12-cu130-2.13.0 -- \
  axolotl train /configs/lora.yml

-v ./configs:/configs uploads the local configs directory to your private jobs-artifacts bucket (created for you on first use) and mounts it read-only in the container, so the YAML on your disk is the YAML the run uses. Output is set in the YAML. These keys push the model at the end of the run, and max_steps caps this trial run:

hub_model_id: your-username/my-adapter
hub_strategy: end
max_steps: 20

This finishes in about seven minutes and pushes the adapter to hub_model_id, as a private repo. Remove max_steps for the full run. The image tag formats are described in Axolotl’s Docker guide.

For more GPUs, change the flavor and nothing else: on a10g-largex4, axolotl train starts one process per GPU by itself. DeepSpeed and FSDP are then a matter of YAML keys, covered in Axolotl’s multi-GPU guide.

Going further

• Serve Models to put the model you trained behind a temporary endpoint, for an evaluation run or a demo. Inference Endpoints runs one that stays up.
• Configuration for secrets, environment variables, volumes and the [tool.hf-jobs] table that lets a script carry its own flavor and timeout.
• Manage Jobs for listing, inspecting, debugging and cancelling Jobs.
• Process Large Datasets for streaming and mounting data that does not fit the disk.
• Schedule Jobs to run a training command on a timer.
• Examples & Tutorials for community write-ups, including vision-language fine-tuning and streaming large datasets into a training run.
• Use Jobs from a coding agent: the hf CLI skill lets Claude Code, Codex and Cursor launch and watch these runs for you.