How much GPU memory an LLM needs, model by model

A large language model (LLM) needs about 2 GB of GPU memory (VRAM) for every billion parameters to hold its weights at 16-bit precision. At 4-bit it needs a little over 0.5 GB per billion, and the conversation needs room on top.

Llama 3.3 70B, for example, needs about 141 GB for its weights at 16-bit: too much for one 80 GB card. At 4-bit the weights take about 40 GB. A conversation of 32K tokens (words and parts of words) adds about 11 GB. The total of about 50 GB fits on one 80 GB card.

The answer changes with each model's design and with the length of the conversation. We worked it out for 11 current open models from the files and settings their makers publish, checked on 25 October 2026. The tables show what each model needs, and the price boxes show which of the GPU plans we list can run it.

The rule of thumb: about 2 GB per billion parameters at 16-bit

A model's parameters are the numbers it learned in training, and its weights are those numbers stored in files. Each parameter takes a fixed amount of memory, set by the precision at which it is stored. At 16-bit precision a parameter takes 2 bytes, so the weights need about 2 GB of GPU memory for every billion parameters.

Storing the weights at lower precision is called quantisation. At 8-bit the weights need about 1 GB per billion parameters. At 4-bit they need a little over 0.5 GB per billion. A 4-bit copy can answer a little differently from the full model; how much that matters depends on the model and the task.

Two things come on top of the weights. The key-value (KV) cache holds the conversation so far: for every token the model reads or writes, its layers store a key and a value. The software that serves the model also needs working space of its own. Our rule: the weights and the KV cache together should use no more than 90% of the card's memory.

The formula: weights plus KV cache, within 90% of the card

GPU memory needed = weights + KV cache + working space. Work out each part in turn.

1. Weights: multiply the number of parameters by the number of bytes each one takes. That is 2 bytes at 16-bit, 1 byte at 8-bit and 0.5625 bytes (4.5 bits) at 4-bit.
2. KV cache per token: take 2 (a key and a value) and multiply by the number of full-attention layers, which keep the full conversation. Then multiply by the number of key-value heads, by the head size and by 2 bytes for a 16-bit cache.
3. KV cache in total: multiply the cache per token by the length of the conversation in tokens. Then multiply by the number of users served at once.
4. Fit: the weights and the KV cache should come to no more than 90% of the card's memory. The last 10% is the working space.

memory needed = weights + KV cache + working space
weights       = parameters × bytes per parameter (2, 1 or 0.5625)
KV cache      = 2 × full-attention layers × KV heads × head size × 2 bytes × tokens × users
working space = 10% of the card, so weights + KV cache ≤ 90% of the card

A layer is one of the stacked steps that every token passes through. Each layer splits its keys and values into several key-value heads, and the head size is how many numbers each head's key or value holds. These numbers are in each model's configuration file (config.json) on its Hugging Face page.

For 4-bit weights we count 4.5 bits, not 4. Common 4-bit formats store a scaling factor for each small block of weights, and files often keep some parts at higher precision. OpenAI's gpt-oss-120b, released in a 4-bit format called MXFP4, does both. Its files take 65.2 GB for 116.8 billion parameters, about 4.5 bits each.

The 90% rule is ours. It is close to the default in vLLM, software for serving models. By default, vLLM uses 92% of a card's memory (vLLM's engine settings, checked 25 October 2026).

Memory each model needs at 16-bit, 8-bit and 4-bit

At 4-bit, five of the six models with up to 36 billion parameters fit on one 24 GB card, with room for a 32K-token conversation. The sixth, Gemma 4 31B, needs a 32 GB card in Google's own 4-bit version for vLLM. Llama 3.3 70B and gpt-oss-120b need an 80 GB card, and Mistral Medium 3.5 needs a 94 GB card. DeepSeek-V4.1-Flash needs eight 80 GB cards for its files alone, and Kimi K3 needs more than that.

Every figure in this table comes from the published files and our own sums.

How to read the table

• For the three Qwen models and Gemma 4 31B, the parameter count on Hugging Face includes the vision encoder, which reads images. The Qwen files also hold an extra layer that predicts several tokens ahead. That is why these counts are slightly above the number in the model's name.
• A dense model uses all its parameters for every token. BF16 is a 16-bit format and FP8 an 8-bit one.
• Where a maker releases its own 4-bit or 8-bit version, the table shows the size of those files instead of our estimate. For Gemma 4, we list Google's 4-bit files for vLLM.
• DeepSeek gives 552 billion parameters for the backbone of DeepSeek-V4.1-Flash. The 763.2 billion that Hugging Face lists covers all its files, which also hold a vision encoder and extra prediction layers. We therefore size this model by its files as released.

The KV cache: longer conversations and more users need more memory

The KV cache grows with every token the model reads or writes. A long document, a long chat or several users at once can need as much memory as the model itself. Llama 3.3 70B stores about 0.33 MB per token at 16-bit. A 128K-token conversation therefore takes about 43 GB of cache, more than the model's 4-bit weights.

Each user needs a cache of their own. Eight people using Llama 3.3 70B at once, each in a 32K-token conversation, need about 86 GB of cache: eight times 10.7 GB.

Newer designs cut this cost sharply. Qwen3.8-27B keeps the full conversation in only 16 of its 64 layers. The other 48 use linear attention, which holds a small state of fixed size instead. Gemma 4 12B and 31B keep only the last 1,024 tokens in five of every six layers, called sliding-window layers. In gpt-oss, every other layer is a sliding-window layer that keeps only the last 128 tokens.

Gemma 4 and gpt-oss also need a small, fixed amount for their sliding-window layers; the last column includes it. The three Qwen models can be stretched to about one million tokens with settings their model cards describe. Their cache then grows to 20.7 GB (Qwen3.6-35B-A3B), 33.1 GB (Qwen3.5-9B) and 65.5 GB (Qwen3.8-27B). Serving software can store the cache at 8-bit, which halves it; vLLM, for example, has an fp8 option for its cache.

Mixture-of-experts models must fit whole

A mixture-of-experts model splits much of its network into many smaller sub-networks, called experts, and uses only a few of them for each token. For example, gpt-oss-120b has 116.8 billion parameters but uses about 5.1 billion for each token, according to its model card. That cuts the work per token, not the memory.

Any expert can be called on for the next token, so all of them have to be in GPU memory. Size the GPU for the total parameter count, never the active count.

Running Kimi K3 or DeepSeek-V4.1-Flash yourself is a far bigger project, since their files alone need eight 80 GB cards or more.

Which GPU sizes fit each model

The GPU plans we list have one card of 24, 32, 48, 80, 94 or 141 GB, or several cards in one machine. The table gives the smallest size that holds each model with a 32K-token conversation, at each precision. Where no single card is enough, we try machines of 2 × 80, 4 × 48, 4 × 80 and 8 × 80 GB.

A model can also be split across several GPUs in one machine; vLLM and similar software do this with tensor parallelism. Each card then holds a share of the weights and the cache. Base your sums on the combined memory of all the cards, less 10% of each.

NVIDIA's pages, checked 25 October 2026, give 24 GB for the L4 and GeForce RTX 4090, and 32 GB for the GeForce RTX 5090. They give 48 GB for the L40S and RTX A6000, and 80 GB for the H100 SXM and A100 80GB. The H100 NVL has 94 GB and the H200 141 GB. GPU Mart's own pages give 80 GB for its H100 and 40 GB for each A100 in its four-card plan.

The GPU plans we list, by size

The boxes below show the GPU plans we list at each size. To compare an hourly plan with a monthly one, multiply the hourly price by the hours you need; a full month is about 730 hours. UpCloud and Unihost show their prices without VAT. DigitalOcean, GPU Mart and Liquid Web do not make clear on the pages we read whether VAT is added.

The comparison table on our homepage lists more plans: choose GPU servers, then filter by GPU memory, billing and location.

Three worked examples

Qwen3.8-27B on one 24 GB card

Hugging Face lists 27.8 billion parameters for Qwen3.8-27B, a figure that includes its vision encoder. At 4-bit, the weights take 27.8 × 0.5625 = 15.6 GB. Its configuration lists 16 full-attention layers, four key-value heads and a head size of 256. Each token therefore needs 2 × 16 × 4 × 256 × 2 = 65,536 bytes of cache.

A 32K-token conversation (32,768 tokens) adds 2.1 GB, for a total of 17.8 GB. That is under 21.6 GB, the 90% of a 24 GB card our rule allows. At 16-bit the weights alone take 55.6 GB, so the same model needs an 80 GB card. Running Qwen3.8-27B on one 24 GB or 32 GB GPU comes down to two choices: the file's precision and the conversation length.

Llama 3.3 70B at 4-bit, 8-bit and 16-bit

Llama 3.3 70B has 70.6 billion parameters. Meta's own llama-models code lists its settings: 80 layers, eight key-value heads and 64 attention heads across a model width of 8,192. The head size is therefore 8,192 ÷ 64 = 128. Each token needs 2 × 80 × 8 × 128 × 2 = 327,680 bytes of cache.

At 4-bit, the weights take 39.7 GB. An 8K-token conversation (8,192 tokens) adds 2.7 GB. The total of 42.4 GB only just fits within the 43.2 GB our rule allows on a 48 GB card. A 32K-token conversation adds 10.7 GB instead, and the total of 50.4 GB needs an 80 GB card.

At 8-bit, the weights take 70.6 GB. With a 32K-token conversation the total is 81.3 GB, within the 84.6 GB our rule allows on a 94 GB card. At 16-bit, the 141.1 GB of weights need several cards. Two 80 GB cards hold them only with an 8K-token conversation: 143.8 GB, against 144 GB allowed.

On a single card, its memory sets how long the conversation can be:

• an 80 GB card holds the 4-bit weights with up to 98,000 tokens of conversation
• a 94 GB card holds the 8-bit weights with up to 42,000 tokens
• a 141 GB card holds the 8-bit weights with the full 131,072 tokens

Running Llama 3.3 70B on one GPU is therefore a trade between precision and conversation length.

gpt-oss-120b on one 80 GB card

OpenAI's gpt-oss-120b is a mixture-of-experts model released in 4-bit form: its files on Hugging Face take 65.2 GB. Half of its 36 layers keep only the last 128 tokens, so it has only 18 full-attention layers. With eight key-value heads of size 64, these need 2 × 18 × 8 × 64 × 2 = 36,864 bytes per token.

Even its longest supported conversation, 131,072 tokens, adds only 4.8 GB. The total of about 70 GB stays under the 72 GB our rule allows on an 80 GB card. That matches the model card, which says it runs on a single 80 GB GPU. Choosing where to run gpt-oss-120b then comes down to the 80 GB plans on offer and their prices.

How we calculated these figures

We read each model's page on Hugging Face on 25 October 2026. The parameter count is the exact total Hugging Face lists for the model's weight files, and the file size is the total of those files. Where a repository holds two copies of the weights, we counted one. For DeepSeek-V4.1-Flash we read its 1017 release, which DeepSeek calls a minor update.

Layer, head and head-size numbers come from each model's config.json. Llama 3.3 70B's come from Meta's llama-models code: on Hugging Face, its configuration file opens only to users who agree to share their account details. A gigabyte (GB) here is 1,000,000,000 bytes and a megabyte (MB) 1,000,000 bytes; 8K, 32K and 128K tokens mean 8,192, 32,768 and 131,072 tokens.

The KV cache is counted at 16-bit for one conversation. Sliding-window layers, which keep only recent tokens, are counted at their fixed size. Gemma 4's full-attention layers use one set of numbers for both keys and values. We still count keys and values separately, so serving software that stores them once may need less cache than we show.

We did not calculate the KV cache of DeepSeek-V4.1-Flash and Kimi K3. Both use new attention designs whose cache size we could not work out reliably from their configuration files, so the table shows their weights only.

Back to Where to run an open model