Most self-hosted models are compressed, or quantised, and the software around them often differs from the maker's own set-up. Both can make the same model answer worse.
In llama.cpp's own measurements on Llama 3 8B, the 8-bit file picked the same next word as the full model 97.7% of the time. The common 4-bit file managed 91.9%, and a 2-bit file only 71.1%.
Settings matter as much as the file. A short conversation window, the wrong chat template or the wrong sampling settings can each make a good model look poor. The sources, all read on 27 October 2026, are listed below.
Who this is for
- your own copy of a model gives weaker answers than the same model online
- you are choosing between 4-bit, 6-bit and 8-bit files
- you want a checklist of settings to try before you blame the model
Not for you if you need to know which files fit your card: see our GPU memory guide.
Key numbers
- 97.7% of next words the same as the full model, for Llama 3 8B at 8-bit, in llama.cpp's tests Official page,
- 91.9% for the usual 4-bit file (Q4_K_M) in the same tests Official page,
- 4,096 tokens Ollama's default conversation length on cards with less than 24 GiB of memory Official page,
What quantisation does
A model stores billions of numbers, its weights. Quantisation, spelled with a z in American English, stores each one in fewer bits. An 8-bit file takes about half the memory of the original 16-bit one, and a 4-bit file about a third.
The smaller file fits a smaller card and runs faster, because the card reads less for each word. The price is some loss of detail, which grows as the bits shrink. Our GPU memory guide shows the sizes.
How much quality each level loses
The llama.cpp project measured this on Llama 3 8B, comparing each compressed file with the full 16-bit model on the same text. Two of its measures are easy to read:
- Perplexity rise: how much less sure the model is of the right next word. Lower is better.
- Same top word: how often the compressed file picks the same most likely next word as the full model.
8-bit and 6-bit are almost identical to the full model; 4-bit loses a little; 2-bit loses a lot.
| Format | Plain name | Bits per weight | Size (GiB) | Perplexity rise | Same top word |
|---|---|---|---|---|---|
| Q8_0 | 8-bit | 8.5 | 7.96 | 0.04% | 97.7% |
| Q6_K | 6-bit | 6.6 | 6.14 | 0.35% | 96.0% |
| Q4_K_M | 4-bit, the usual default | 4.9 | 4.58 | 2.8% | 91.9% |
| Q2_K | 2-bit | 3.2 | 2.96 | 56% | 71.1% |
At 4-bit the top word differs about once in every 12 words (our calculation). Once one word differs, the rest of the answer can take a different path.
Newer models can lose more. In the same tests, Llama 2 7B at 4-bit rose 1.4% in perplexity, about half the 2.8% of Llama 3 8B.
Settings that change the answers
- The conversation window. Ollama gives 4,096 tokens by default on cards below 24 GiB, by its documentation, and the model forgets what falls outside. Ollama advises at least 64,000 for agents and coding tools.
- The chat template. Each model expects its messages in a set format. Unsloth's guide warns that templates uploaded with some models contain errors; serve a model with the template it was trained on.
- Sampling. Makers publish the settings their models were tuned for. Qwen's card for Qwen3.8, for example, gives one set for thinking mode and another for direct answers, plus a penalty against endless repetition.
- Thinking mode. Qwen3.8 thinks before it answers by default; turned off, it answers directly, and its card gives different settings for each mode.
- A compressed cache. Ollama's FAQ says an 8-bit cache loses very little, while a 4-bit cache loses more, especially in long conversations.
Our Qwen3.8-27B guide shows these settings for one model, and our serving engine comparison covers the software.
Why an online service can differ too
Online services run models in compressed formats as well. Qwen, for example, publishes an official 8-bit (FP8) release of Qwen3.8-27B next to the 16-bit one.
A service may also cap the conversation length or set its own sampling. Some state the precision they serve; if yours does not, ask before you compare it with your own copy.
A checklist before you blame the model
- Check the file: use 8-bit or 6-bit if the card has room, 4-bit if it does not, and avoid 2-bit for serious work.
- Check the conversation length: in Ollama, raise OLLAMA_CONTEXT_LENGTH; in llama.cpp, set
-c. - Use the maker's chat template, as the engine reads it from the model file or the maker's page.
- Use the maker's sampling settings from the model card.
- Compare like with like: the same model size, the same thinking setting and the same prompt.
| Plan | Price | New-customer offer | Link |
|---|---|---|---|
| GPU Mart Enterprise Dedicated GPU Server - RTX 5090 32 GB GPU | $479/month monthly rate of a 24-month term, paid in advance · VAT not stated checked 25 Oct 2026 | A free 24-hour trial of a VPS, dedicated or GPU server for new clients, on request checked 29 Oct 2026 | Visit |
| DigitalOcean GPU Droplet, 1 x NVIDIA L40S 48 GB GPU | $1.57/hour (about $1,146/month) on demand, billed per second · VAT not stated checked 25 Oct 2026 | New accounts get a $5 credit that lasts 90 days and covers every product except SaaS add-ons checked 29 Oct 2026 | Visit |
| UpCloud GPU Server, 1 x NVIDIA L40S, 8 cores, 64 GB 48 GB GPU | €1.11/hour (about €746/month; no charge after 28 days of use)30-day month: the last 2 days are free.31-day month: the last 3 days are free.February: no free days (one in a leap year).Started mid-month: it may never reach the cap that month. on demand, billed by the starting hour; monthly cost capped at 28 days of use checked 25 Oct 2026 | No offer | Visit |
| GPU Mart Enterprise Dedicated GPU Server - H100 80 GB GPU | $2,099/month monthly rate of a 24-month term, paid in advance · VAT not stated checked 25 Oct 2026 | A free 24-hour trial of a VPS, dedicated or GPU server for new clients, on request checked 29 Oct 2026 | Visit |
Monthly costs for hourly prices assume the server runs all month (730 hours), or less where a provider caps the monthly bill.
A 32 or 48 GB card holds 8-bit files that would not fit on 24 GB.
Our speed figures by card show the other side of the trade: smaller files run faster.
How we checked
We read llama.cpp's documentation on perplexity and on its file formats, Ollama's documentation, Qwen's model card and Unsloth's guides on 27 October 2026. The once-in-12 figure is our calculation.
Our own measurements will be added here, with their dates, when our test bench runs them.
Mistakes to avoid
Blaming 4-bit for a cut-off conversation
Instead: Check the context length first; Ollama's default can be 4,096 tokens.
Using 2-bit files for real work
Instead: In llama.cpp's tests they changed the top word almost 3 times in 10.
Ignoring the maker's sampling settings
Instead: Copy them from the model card.
Comparing a small local model with a large online one
Instead: Compare the same model size and settings.
Other questions
Is Q4_K_M good enough?
For most chat and writing, yes: in llama.cpp's tests it picked the same next word as the full Llama 3 8B 91.9% of the time. For exact reasoning, 6-bit or 8-bit is safer.
Q4 or Q8: which should I use?
Q8 is almost identical to the full model but takes about 1.7 times the memory of Q4_K_M. Use Q8 when it fits, Q4 when it does not.
Why does my Ollama model forget the start of the chat?
Its default conversation length can be as short as 4,096 tokens, by Ollama's documentation. Raise it with OLLAMA_CONTEXT_LENGTH.
Sources and test notes
- llama.cpp: perplexity and KL-divergence tables (Llama 3 8B, with a comparison against Llama 2), read
- llama.cpp: file formats and bits per weight, read
- Ollama documentation: context length, read
- Ollama FAQ (cache quantisation), read
- Qwen3.8-27B model card (sampling settings, thinking mode), read
- Qwen3.8-27B-FP8 on Hugging Face, read
- Unsloth: datasets guide (chat templates), read
- Unsloth: Qwen3.8 fine-tuning guide (serve with the same template), read
Every page read on 27 October 2026. The quality figures are llama.cpp's own; the once-in-12 figure is our calculation.