vLLM, Ollama, llama.cpp or SGLang: which to run on a rented GPU

For one person on a rented GPU, Ollama or llama.cpp is the simplest choice. For many users at once, pick vLLM or SGLang, which keep the card busy with many requests together.

In Red Hat's test on one A100, vLLM peaked at 793 tokens a second against Ollama's 41. In a July 2026 test on a rented RTX 4090, SGLang and vLLM came out level.

The figures come from those two published tests and from each engine's own documentation. We read them all on 27 October 2026.

What a serving engine does

A serving engine loads the model onto the GPU and answers requests over the network. All four offer an API in OpenAI's format, so most apps and agent tools can connect to any of them.

They differ in how they share the card between requests. By their own documentation, vLLM and SGLang batch requests together and reuse the shared beginnings of prompts, which suits many users.

Ollama and llama.cpp run compressed GGUF files and suit one machine with a few users. Ollama's README names llama.cpp as its backend, wrapped in a simpler tool with its own model library.

Many users at once: two published tests

Red Hat compared vLLM 0.9.1 with Ollama 0.9.2 on one A100 with 40 GB, using Llama 3.1 8B at 16-bit. It raised the number of requests at once from 1 to 256.

Ollama handled four requests at a time by default then, and Red Hat raised its limit to 32. Its FAQ now gives a default of one request at a time per model.

Red Hat's article dates from August 2025, and both engines have changed since. Its conclusion: Ollama for one user and development, vLLM for many users.

The second test, by ComputingForGeeks, ran SGLang 0.5.15 and vLLM 0.25.1 on an RTX 4090 rented from vast.ai in July 2026. Both served Qwen2.5-7B-Instruct to 64 requests at once.

That test used unrelated prompts, so it could not show SGLang's cache for prompts that share a beginning, as its author notes.

One user at a time

With one request at a time, the card's memory speed and the size of the model file set the pace. Our speed figures by card show this across many cards.

Ollama and llama.cpp usually run 4-bit GGUF files, about a third of the 16-bit size. By that rule they write faster for one user on the same card, unless vLLM or SGLang also get a 4-bit file.

Set up each engine on a rented GPU

The commands below serve the same small model, Qwen2.5-7B-Instruct, on Ubuntu with an NVIDIA card and its driver installed. They follow each engine's own documentation.

Ollama

Ollama installs with one script and pulls a 4.7 GB file at 4-bit. It answers on port 11434, with an OpenAI-style API under /v1.

curl -fsSL https://ollama.com/install.sh | sh
ollama pull qwen2.5:7b
curl http://localhost:11434/v1/chat/completions -H "Content-Type: application/json" \
  -d '{"model": "qwen2.5:7b", "messages": [{"role": "user", "content": "Hello!"}]}'

For more than one user, raise OLLAMA_NUM_PARALLEL as our Ollama guide shows. Its FAQ warns that memory use grows with each extra request.

llama.cpp

Build llama.cpp with CUDA as our speed figures page shows. Then download a 4-bit file and start the server with four slots, so it answers four requests at once:

wget https://huggingface.co/lmstudio-community/Qwen2.5-7B-Instruct-GGUF/resolve/main/Qwen2.5-7B-Instruct-Q4_K_M.gguf
./build/bin/llama-server -m Qwen2.5-7B-Instruct-Q4_K_M.gguf -ngl 99 -c 32768 -np 4 --port 8080

Its API answers at http://localhost:8080/v1. The server listens only on the machine itself unless you add --host.

vLLM

By its documentation, vLLM needs Linux, Python 3.11 to 3.14 and an NVIDIA card from the T4 and RTX 20 series onwards. Install it in its own Python environment with uv:

sudo apt install -y ninja-build
curl -LsSf https://astral.sh/uv/install.sh | sh
uv venv --python 3.12 --seed && source .venv/bin/activate
uv pip install vllm --torch-backend=auto
vllm download-kernels
vllm serve Qwen/Qwen2.5-7B-Instruct --api-key choose-a-long-key

It downloads the 15.2 GB model at 16-bit and answers at http://localhost:8000/v1. Callers must then send the key you chose.

In the RTX 4090 test above, vLLM kept 6.5 GiB of the card for conversations, enough for about 122,000 tokens.

SGLang

SGLang also needs Linux, and its current version needs a driver that supports CUDA 13, by its documentation. Install it the same way:

sudo apt install -y ninja-build
uv venv --python 3.12 && source .venv/bin/activate
uv pip install --prerelease=allow sglang
sglang serve Qwen/Qwen2.5-7B-Instruct --port 30000

Its API answers at http://localhost:30000/v1. The RTX 4090 test found that both SGLang and vLLM need the ninja-build package; without it, the first request fails.

Whichever engine you pick, keep its port closed to the internet or put a proxy with a password in front. Both vLLM and llama.cpp can also require an API key.

Which engine for which job

• One person, a chat window or an agent on the same server: Ollama. For a chat window your whole team can use, see our guide to a private chat assistant for a team.
• One person or a few who want full control: llama.cpp. It can also keep part of a model on the processor when the card is too small.
• A team or an app with many users at once: vLLM.
• Many users whose prompts share a long beginning, such as one system prompt: SGLang or vLLM, which both cache that part.
• A model on its release day: check its model card. For Qwen3.8, Qwen recommends SGLang, vLLM or TokenSpeed, as our Qwen3.8-27B guide notes.

Settings that matter most

• Conversation length: each engine reserves memory for it. Set what you need with OLLAMA_CONTEXT_LENGTH, llama.cpp's -c or vLLM's --max-model-len.
• Requests at once: Ollama's OLLAMA_NUM_PARALLEL and llama.cpp's -np. vLLM and SGLang batch on their own; vLLM caps the number with --max-num-seqs.
• Share of the card's memory: vLLM and SGLang claim most of it at start-up (--gpu-memory-utilization, --mem-fraction-static), so run one engine per card.

How we checked

We read Red Hat's and ComputingForGeeks' published tests, each engine's README and documentation, and their latest releases on GitHub, on 27 October 2026. The ratio between vLLM and Ollama is our calculation.

Our own measurements will be added here, with their dates, when our test bench runs them.

Back to Where to run an open model