Running Qwen3.8-27B on one 24 GB or 32 GB GPU

Qwen3.8-27B fits on one 24 GB card at 4-bit. The 17.6 GB file leaves room for about 62,000 tokens of conversation, by our calculation. A 32 GB card holds a 6-bit file, or the 4-bit one with about 172,000 tokens.

By our estimate from public llama.cpp results, expect 40 to 50 tokens a second on an RTX 4090. An RTX 5090 should reach 65 to 80, and one published run of this model on RTX 3090 cards reached 39.

The figures come from Qwen's model card, Unsloth's and Ollama's files, vLLM's recipe and llama.cpp's benchmark thread. We read them all on 27 October 2026.

Which version and which file

Qwen released Qwen3.8-27B on Hugging Face on 5 August 2026, under the Apache 2.0 licence. On 27 October 2026 it was still Qwen's newest model of this size.

It is a dense model that also reads images and video. Thinking is on by default, so answers start with a reasoning section; the model card shows how to turn it off per request.

Qwen's own files are 16-bit (55.6 GB) and 8-bit FP8 (30.9 GB). Neither leaves room on a 24 or 32 GB card, so a single card needs a 4-bit to 6-bit file:

• Unsloth's GGUF files for llama.cpp, from 3-bit (13.1 GB) to 8-bit (29.0 GB). Unsloth's guide says the 4-bit files need 16 to 19 GB.
• Ollama's qwen3.8:27b, which is 4-bit and takes 18 GB with the image part included.
• NVFP4 files for vLLM, which only NVIDIA's newest Blackwell cards, such as the RTX 5090, can run.

What fits in 24 GB and in 32 GB

The cache that holds the conversation is small for a model this size. Only 16 of its 64 layers keep it; the other 48 keep a fixed-size summary instead.

Each token therefore needs 65,536 bytes at 16-bit, so 32,000 tokens take about 2.1 GB. Our GPU memory guide explains the sum.

We keep 10% of the card free for the engine's working memory. Reading images adds the 0.9 GB vision file, about 14,000 tokens less.

An 8-bit cache roughly halves the conversation memory, by Ollama's FAQ, which doubles the room above. In llama.cpp the settings are -ctk q8_0 -ctv q8_0; in Ollama, OLLAMA_KV_CACHE_TYPE=q8_0.

How fast it runs

One user posted a run of this model to llama.cpp's public benchmark thread on 7 September 2026. It used the 5-bit file (19.8 GB) on four RTX 3090 cards.

With the model split layer by layer, the cards work in turn and wrote 39 tokens a second. Split so the cards work together, they reached about 77.

For the cards you can rent, we scaled each card's result on llama.cpp's scoreboard to this model's 4-bit file. The one published run came out a quarter faster than that sum, so we give a range.

The L40S estimate uses the L40's scoreboard result; the two cards share the same memory speed. The L4 estimate uses the T4, which has similar memory bandwidth, by NVIDIA's figures.

Qwen3.8 can also guess several tokens ahead and check them in one step, which Qwen calls multi-token prediction (MTP). Unsloth says it needs 1 to 2 GB more memory; vLLM's recipe shows it working on one RTX 5090.

Our speed figures by card list the scoreboard results behind these estimates.

Step by step on a rented card

The quickest start is Ollama. Its default qwen3.8:27b file is 4-bit and includes the image part.

curl -fsSL https://ollama.com/install.sh | sh
sudo systemctl edit ollama.service
# add, under [Service]:
#   Environment="OLLAMA_CONTEXT_LENGTH=32768"
sudo systemctl daemon-reload && sudo systemctl restart ollama
ollama run qwen3.8:27b "Explain what a reverse proxy does, in three sentences."
ollama ps        # PROCESSOR should say 100% GPU

Ollama sets the conversation length from the card's memory: 4K tokens below 24 GiB and 32K from 24 GiB, by its documentation. It advises at least 64,000 tokens for agents and coding tools, so set the length yourself.

For more control, use llama.cpp's own server. Build it as our GPU speed guide shows, then download Unsloth's 4-bit file (17.6 GB) and start the server:

wget https://huggingface.co/unsloth/Qwen3.8-27B-GGUF/resolve/main/Qwen3.8-27B-UD-Q4_K_XL.gguf
./build/bin/llama-server -m Qwen3.8-27B-UD-Q4_K_XL.gguf -ngl 99 -fa on -c 49152 \
  --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0 --port 8080

The sampling settings are the ones Qwen's model card gives for thinking mode. On a 32 GB card, raise -c towards 131072, or use the 6-bit file.

The server listens only on the machine itself unless you add --host. Before you open it to the internet, add --api-key, or put a proxy with a password in front, as our Ollama guide explains.

On an RTX 5090, vLLM can serve several users at once from an NVFP4 file. The recipe from vLLM gives this command for one card, and says it starts only with CUDA graphs turned off:

vllm serve Inferact/Qwen3.8-27B-NVFP4 \
  --tensor-parallel-size 1 --max-model-len 32768 \
  --kv-cache-dtype fp8 --enforce-eager --reasoning-parser qwen3

The recipe reports 31.4 GiB usable on the card, and room for about 91,000 tokens of conversation across all users at once. Our comparison of vLLM, Ollama, llama.cpp and SGLang explains when each engine fits.

Which card to rent

• For one person or a small agent: a 24 GB RTX 4090 with the 4-bit file.
• For long documents, a higher-quality file or a few users at once: a 32 GB RTX 5090.
• For full-length 262,144-token conversations at 4-bit, or the 8-bit file with shorter ones: a 48 GB card such as the L40S.

When you outgrow 27B, the next step is a 70B model on a 48 or 80 GB card. Our guide to Llama 3.3 70B on one GPU covers it.

To teach the model your own format or field, Unsloth's QLoRA fits Qwen3.8-27B on a 24 GB card. Our Unsloth fine-tuning guide has the steps.

How we checked

We read Qwen's model card and file list, Unsloth's files and guide, and Ollama's library and documentation. We also read vLLM's recipe and llama.cpp's benchmark thread, all on 27 October 2026.

The memory and speed sums are ours, set out above.

Our own measurements will be added here, with their dates, when our test bench runs them.

Back to Where to run an open model