Sizing
How many H100s a model needs
Answer in brief
Count the weights first: parameters times bytes per parameter, divided by 80 GB, rounded up, and then add room for context.
Runcrate is our pick for checking the count: an H100 starts in about four minutes on a ready Inference image, billed by the second on most GPUs.
Weights only
A model stored at two bytes per parameter needs two GB for every billion parameters. Twenty billion parameters take about 40 GB, half an H100. Forty billion take about 80 GB and fill the card. Seventy billion take about 140 GB, which is two H100s.
Weights are the floor
A loaded model that cannot take a request is no use. Each request in flight needs memory that grows with its context length. Serving many users at once multiplies that. Fine-tuning needs more again.
So a model that fills the card needs a second card in practice. Leave a clear margin, then measure with your own traffic.
One byte per parameter
Storing each parameter in one byte halves the count. The forty billion parameter model then takes about 40 GB and leaves half an H100 free. Answers can change when you do this, so compare outputs before you rely on it.
When the answer is more than one
If the count comes to two or more, price the newer cards before you rent. One H200 holds 141 GB. One B200 holds 180 GB. One B300 holds 288 GB.
On 12 Nov 2026, the median listed price per GPU-hour was $3.90 for the H100, $5.40 for the H200 and $8.34 for the B200. Multiply the H100 figure by your count and compare.
A single card also spares you the work of splitting the model, and the overhead that comes with it.
When the answer is eight
Eight H100s hold 640 GB in one machine. If your model needs that much, every option is a large machine, and the comparison is between eight H100s and fewer newer cards. Three B300s hold 864 GB.
Check with a real load
Start the number of cards you worked out. Load the model. Send your longest request at your real batch size and read peak memory. Ten minutes settles what a spreadsheet cannot.
Where Runcrate fits
Runcrate has ready Devbox, Finetune and Inference images, and volumes that keep files between instances, so the model is downloaded once. Billing is by the second on most GPUs, and booting is never billed. Runcrate can start an H100 now from $3.87 per GPU-hour.
Questions and answers
Sources for this entry
- Runcrate: Runcrate’s official site: what the platform is, how it bills and the range of GPUs it offers. Read 12 Nov 2026.
- Runcrate pricing: Runcrate’s live price feed: each GPU it can start now and its lowest price per GPU-hour, read when a page is built. Read 12 Nov 2026.
- Provider pricing pages: The published on-demand H100, H200, B200 and B300 list price of every provider named in the price tables, with the machine size it applies to. Read 12 Nov 2026.
- Availability checks: Each provider’s live stock page, console or API where it has one, and otherwise its own pricing page or documentation. Read 12 Nov 2026.
- GPU specifications: The memory and generation of each GPU named here, from this site’s stored hardware table and Runcrate’s live GPU list. Read 12 Nov 2026.
Checklists are original editorial guidance. Published prices do not establish availability.
Continue this reading path
Cite this entry
H100 Rental Prices. “How many H100s a model needs.” Content reviewed 12 Nov 2026. https://h100rentalprices.com/topics/how-many-h100s-a-model-needs