How fast open models run on rented GPUs: tokens per second by card

For one user, an NVIDIA H100 writes about 330 tokens a second with a small 7B model at 4-bit. An RTX 4090 writes about 190, and an L40 about 150.

Bigger models run slower, roughly in proportion to their file size. On cards rented from RunPod, Llama 3 70B at 4-bit wrote about 25 tokens a second on an H100 and 15 on an L40S.

These figures come from two public benchmarks of the llama.cpp engine: its own CUDA scoreboard on GitHub, and an independent benchmark run on rented cards. We read both on 27 October 2026.

Speed by card: a small model, one user

The llama.cpp project keeps a scoreboard where users post the results of one fixed test. It runs Llama 2 7B at 4-bit (Q4_0, a 3.8 GB file) with the whole model on one card.

Writing speed is how fast the answer appears. Prompt reading is how fast the card takes in your question and any documents before it starts to write.

The H100 SXM moves 3.35 TB a second to and from its memory and the L40 864 GB, by NVIDIA's figures. That is nearly four times as much, yet the H100 wrote only about twice as fast here.

A 3.8 GB model is too small to keep the fastest memory busy. With bigger models, the gap between cards grows, as the 70B results below show.

• The scoreboard lists the L40, not the L40S that most clouds rent. Both have 48 GB at 864 GB/s by NVIDIA's figures, so their writing speeds should be close (our estimate).
• Neither source tests the L4, a common 24 GB card. NVIDIA gives it 300 GB/s, close to the T4's 320 GB/s, and the T4 wrote 44 tokens a second here.
• The engine keeps getting faster. Between September and November 2025, the same poster's H100 result rose from 281 to 329 tokens a second on newer builds.

Why bigger models write more slowly

For each new token, the card reads the whole model from its memory once. Double the file size, and the speed for one user roughly halves.

One poster in the scoreboard thread ran five models on the same card, an RTX PRO 4500 Blackwell with 32 GB, on 14 November 2025:

The exception is a mixture-of-experts model. It stores many expert networks but uses only a few for each token. OpenAI's model card says gpt-oss-20b uses 3.6 billion of its 20.9 billion parameters per token.

So it writes like a much smaller model while answering like a larger one. Our gpt-oss-120b guide covers its bigger sibling, which needs an 80 GB card.

70B models on one rented card

For larger models, the GitHub benchmark GPU-Benchmarks-on-LLM-Inference ran Llama 3 8B and 70B on cards rented from RunPod in May 2024, using llama.cpp. Each figure is the average speed while writing 1,024 tokens.

The gap between cards grows with the model. With the 8B model, the H100 PCIe wrote about a quarter faster than the L40S; with the 70B model, about 60% faster (our calculation).

On the L40S, the 70B model wrote about a seventh as fast as the 8B one. Two RTX 4090 cards together held the 70B model and wrote about as fast as one RTX 6000 Ada.

These runs are from May 2024, and llama.cpp has been sped up since. Treat them as a floor rather than today's speed.

One user against many users at once

Every figure above is for one request at a time. Serving software such as vLLM or SGLang handles many requests together, so the total speed grows while each user's share falls.

NVIDIA's own benchmark of Llama 3.3 70B on two H100 cards shows the effect. One user got about 52 tokens a second; 50 users at once got about 814 in total, or about 16 each.

Our comparison of vLLM, Ollama, llama.cpp and SGLang covers which engine suits which job. Our guide to Llama 3.3 70B on one rented GPU works through one model in detail.

Which card for which model

• Models up to 8B at 4-bit (about 5 GB): any card from 16 GB. An RTX 4090 wrote Llama 3 8B at about 128 tokens a second.
• Models of 27B to 32B at 4-bit (16 to 20 GB): a 24 GB card for short conversations, 32 to 48 GB for long ones. An RTX PRO 4500 Blackwell with 32 GB wrote them at about 40 tokens a second. Our guide to Qwen3.8-27B on one GPU works through one such model.
• Models of 70B at 4-bit (about 40 to 43 GB): a 48 GB card for short conversations, 80 GB for long ones. Expect 12 to 25 tokens a second on one card.
• gpt-oss-120b (65 GB): one 80 GB card, as our gpt-oss-120b guide explains.

Check the memory first with our GPU memory guide. A model that spills from the card into the server's normal memory runs many times slower.

To compare cards, work out the cost per million tokens. Divide the hourly price by (tokens a second x 3,600), then multiply by 1,000,000.

A card that writes twice as fast at less than twice the price costs less per token. Our guide to API, rented GPU or your own hardware sets this against model APIs.

Run the same test on a card you rent

You can check a rented card against the scoreboard in one session. These steps follow llama.cpp's build guide and the scoreboard's own instructions, on Ubuntu.

First, check that the server sees its GPU. The command lists the card, its memory and the driver version.

nvidia-smi

Then build llama.cpp with CUDA support. The build needs NVIDIA's CUDA toolkit; nvcc --version shows whether it is installed.

sudo apt install -y git cmake build-essential
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j

Download the scoreboard's test model, a 3.8 GB file, and run the test with flash attention off and on:

wget https://huggingface.co/TheBloke/Llama-2-7B-GGUF/resolve/main/llama-2-7b.Q4_0.gguf
./build/bin/llama-bench -m llama-2-7b.Q4_0.gguf -ngl 99 -fa off,on

The tg128 rows give the writing speed and the pp512 rows the prompt reading speed. Each is the average of five runs, the tool's default.

Compare them with the table above. The scoreboard notes that the driver, the operating system and the card's maker can all change a result.

To time the model you plan to use, point -m at its GGUF file instead. Adding -p 4096 times a long prompt as well.

Limits of these figures

• Users post the scoreboard results from their own machines, with different drivers and builds of llama.cpp.
• The 70B benchmark dates from May 2024; newer builds of llama.cpp are faster.
• Every figure is for one request at a time, with the whole model on the card.
• Neither source ran the exact servers in our price boxes.

How we checked

We read llama.cpp's CUDA scoreboard and the results posted under it, the GPU-Benchmarks-on-LLM-Inference repository and NVIDIA's product pages on 27 October 2026. The ratios between cards are our calculation.

Our own measurements will be added here, with their dates, when our test bench runs them.

Back to Where to run an open model