How to fine-tune an open model with Unsloth on a rented GPU

Unsloth fine-tunes open models on a single rented GPU. By its own table, an 8B model needs about 6 GB with QLoRA (4-bit) and 22 GB with LoRA (16-bit).

A 24 GB card such as the RTX 4090 therefore trains models up to about 27B with QLoRA. Unsloth's guide for Qwen3.8-27B says QLoRA works with 24 GB.

The steps below follow Unsloth's documentation, read on 27 October 2026. The short training script is our own, built on the functions Unsloth documents.

What you will build

Fine-tuning with LoRA trains a small set of extra weights, an adapter, on top of an open model. The original model stays as it is; the adapter changes how it answers.

QLoRA does the same with the base model loaded at 4-bit, which cuts memory roughly fourfold, by Unsloth's guide. At the end you export a GGUF file for Ollama or llama.cpp, or a merged 16-bit model for vLLM.

What you need

• A dataset of example conversations: at least 100 rows, and over 1,000 for better results, by Unsloth's datasets guide.
• A rented NVIDIA GPU on Linux. Unsloth supports cards from 2018 onwards, from the T4 and V100 to the A100, L40 and H100.
• Ubuntu 20.04 or newer, the NVIDIA driver, CUDA toolkit 12.4 or newer (12.8 for RTX 50 cards) and Python 3.11 to 3.13.

Unsloth calls these figures the absolute minimum; some models need more. If training runs out of memory, its first advice is a batch size of 1 to 3.

Step 1: prepare the dataset

Save your examples as a JSONL file, one conversation per line, in the user and assistant format Unsloth's datasets guide shows:

{"messages": [{"role": "user", "content": "Summarise this ticket: printer offline since Monday."},
              {"role": "assistant", "content": "Issue: printer offline. Since: Monday. Next step: check the network cable."}]}

Every answer should show exactly what you want back: the format, the tone and the level of detail. Keep 20 or so examples aside to test the model later.

Step 2: rent the GPU and install Unsloth

Rent a card with enough memory from the table above, then connect over SSH. Unsloth offers a web interface, Unsloth Studio, and a Python library, Unsloth Core.

Option A: Unsloth Studio in your browser

Install it with Unsloth's script and start it on the server itself. An SSH tunnel then shows it in your own browser without opening a port:

curl -fsSL https://unsloth.ai/install.sh | sh
unsloth studio -H 127.0.0.1 -p 8888
# on your own computer:
ssh -L 8888:127.0.0.1:8888 user@your-server    # then open http://localhost:8888

In the Train tab, pick the model and your dataset, set the training options and start. Unsloth's guide says the training loss should fall steadily as it runs.

Option B: Unsloth Core in Python

curl -LsSf https://astral.sh/uv/install.sh | sh
uv venv unsloth_env --python 3.13 && source unsloth_env/bin/activate
uv pip install unsloth --torch-backend=auto

Step 3: train with QLoRA

This script fine-tunes an 8B model with the settings Unsloth's guide suggests to start with. Save it as train.py next to train.jsonl and run python train.py:

from unsloth import FastLanguageModel
from datasets import load_dataset
from trl import SFTTrainer, SFTConfig

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="unsloth/Qwen3-8B-unsloth-bnb-4bit",   # 4-bit base for QLoRA
    max_seq_length=2048, load_in_4bit=True)
model = FastLanguageModel.get_peft_model(
    model, r=16, lora_alpha=16, use_gradient_checkpointing="unsloth",
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"])

data = load_dataset("json", data_files="train.jsonl", split="train")
data = data.map(lambda row: {"text": tokenizer.apply_chat_template(row["messages"], tokenize=False)})

trainer = SFTTrainer(model=model, tokenizer=tokenizer, train_dataset=data, args=SFTConfig(
    dataset_text_field="text", per_device_train_batch_size=2, gradient_accumulation_steps=4,
    max_steps=60,            # a quick test; for the real run use num_train_epochs=1
    learning_rate=2e-4, optim="adamw_8bit", logging_steps=5, output_dir="outputs", report_to="none"))
trainer.train()

model.save_pretrained_gguf("my_model_gguf", tokenizer, quantization_method="q4_k_m")

The 60 steps are a quick test, as Unsloth's guide suggests. For the real run, replace them with one epoch; the guide recommends one to three, to avoid overfitting.

For Qwen3.8-27B on a 24 GB card, Unsloth's Qwen3.8 page uses FastModel and the base unsloth/Qwen3.8-27B-unsloth-bnb-4bit instead, with a batch size of 1. Our Qwen3.8-27B guide covers running the model.

Step 4: test it against the original

Ask the original model and the tuned one the 20 examples you kept aside, and compare the answers. Unsloth Studio's Compare Mode shows the adapter and the original model side by side.

If the tuned model repeats the training answers word for word, it has memorised them; Unsloth's guide calls this overfitting. Its remedies include fewer epochs and a non-zero lora_dropout.

Step 5: export and run it

The last line of the script writes a 4-bit GGUF file. Unsloth also writes the Modelfile Ollama needs, so you can load it into Ollama and chat with it:

ls my_model_gguf/                      # the .gguf file and the Modelfile
ollama create my-model -f my_model_gguf/Modelfile
ollama run my-model

For vLLM, save a merged 16-bit copy with model.save_pretrained_merged("merged", tokenizer, save_method="merged_16bit") and serve that folder. Our serving engine guide shows both engines.

Unsloth's Qwen3.8 page asks you to serve the model with the same chat template and end token it was trained with.

Which card to rent

• Models up to 14B with QLoRA: a 16 or 24 GB card, such as an L4 or RTX 4090.
• An 8B model with 16-bit LoRA, or Qwen3.8-27B with QLoRA: a 24 GB card.
• 32B models with QLoRA: a 32 GB RTX 5090 or a 48 GB card.
• 70B models with QLoRA: a 48 GB card at the very least, 80 GB for room to spare.

A run costs its GPU hours times the hourly price, plus storage while the server exists. Start with the 60-step test to see how long each step takes, then multiply for the full run.

Our guide to what fine-tuning costs adds run times others have reported and the cost of keeping the tuned model online.

Unsloth or Axolotl?

Axolotl is the other well-known open-source tool for this job, also under the Apache 2.0 licence. Its README describes one YAML file that drives the whole run, from data preparation to training and export.

Unsloth suits one GPU and people who prefer a web interface or a short script. Axolotl's README highlights training across several GPUs, with FSDP and DeepSpeed.

How we checked

We read Unsloth's requirements, installation and fine-tuning guides, its Qwen3.8 page and its datasets guide, and Axolotl's README, on 27 October 2026. The memory figures are Unsloth's own.

Our own measurements will be added here, with their dates, when our test bench runs them.

Back to Fine-tuning or a chatbot