How we test
This page explains how George tests the services we compare: how the test accounts are opened, what we record and the limits of every result.
Last updated
What we test
We test what you would do yourself after renting a service: set it up, run real work on it, measure it and note what goes wrong. Every result names the date, the provider, the plan and the versions it was measured with.
We never describe a test that did not take place. When a page states a fact we did not measure, it names the source, such as the provider's own pricing page. It also gives the date we read it.
How we open test accounts
George opens every test account himself, as an ordinary customer, directly on the provider's own website. He never signs up through the Visit buttons or any other link on our website.
What we record
For every test we keep a log with:
- the date and time of each run
- the provider, the plan and the data-centre region
- the operating system and the version of each app and model
- the settings we changed
- the commands we ran and their raw output
An article shows a "How we tested" box only when such a log exists.
How we measure speed and memory
A single run can be lucky or unlucky, so every speed we publish is the median of at least 3 runs: the middle value when the results are put in order. We log memory and processor use throughout the test, rather than reading them once.
The limits of every result
Every result states its limits, which are usually one region, one week and one version of each app or model. A provider can change its hardware, prices or software after we test it, so every result carries its date.
We run each bench again once a year, and sooner when a new version or a changed plan could affect the result.
Screenshots
Our screenshots come only from George's own accounts. Before we publish one, we crop it to what matters and remove personal details, IP addresses, keys and e-mail addresses.
Who runs the tests
George runs every test. He opens the accounts, starts and stops the servers, runs the test scripts and keeps the raw output. You can find out more about him on About us.
How we use AI tools
AI tools help us with research, drafting and fact-checking. George runs every test and approves every article before it is published.
Our test benches
A bench is a set of rented servers or services and the tests we run on them. Several articles share each bench, and every "How we tested" box links to the bench it used.
T1 Agent bench
We have not run this bench yet.
We rent servers and managed plans for AI agents, each for one month. Every server runs n8n, OpenClaw and Hermes Agent with the same five workflows and three agent jobs.
We log memory and processor use every 5 minutes for 7 days and check that webhooks arrive. We also scan our own servers for open ports from outside, then try one update, one rollback and one restore from a backup.
T2 CPU model bench
We have not run this bench yet.
We run small open models on the processor, with no GPU. Servers with 8, 16 and 32 GB of memory run Ollama with four small models, each at two levels of quantisation. We measure tokens per second and memory use.
T3 GPU bench
We have not run this bench yet.
We rent GPUs with 24, 48 and 80 GB of memory and run one benchmark script on each. It runs models of three sizes (8B, 27B to 32B, and 70B or gpt-oss-120b) on two engines: llama.cpp (or Ollama) and vLLM.
We test each model with one user and then with eight users at once, and repeat every run three times.
T4 Fine-tuning bench
We have not run this bench yet.
We run LoRA and QLoRA fine-tuning with Unsloth on models of 8B, 27B to 32B and 70B, all with the same dataset. Then we compare the answers to 20 test prompts before and after training.
T5 Chatbot bench
We have not run this bench yet.
We give several document chatbot services the same 30 documents and the same 40 questions, so their answers can be compared side by side.
T6 Image and video bench
We have not run this bench yet.
One fixed ComfyUI workflow makes 100 images and a few short clips. It runs on a rented GPU and on two ComfyUI services that you use in the browser, one of them billed by the hour.
Two video models run on two kinds of GPU, and we time one Flux LoRA training run. We log the peak GPU memory of every job.
T7 Monthly GPU server
We have not run this bench yet.
We rent a GPU server by the month and time how long the order takes to be ready. Then we run the same benchmark script as on bench T3.
T8 Spending controls
We have not run this bench yet.
On the accounts opened for benches T1 and T3, we set each provider's spending limit, alert or prepaid balance. Then we check whether spending really stops when it reaches a small limit.
Tests we have published
We have not published any tests yet.