How Much GPU Memory Do You Need for LLM Fine-Tuning and Inference?
A practical way to estimate GPU memory by model size, precision and training method — with example scenarios from 7B to 70B and GPU recommendations.
Memory is the first factor that determines how many GPUs you need to run or train a language model. Fortunately, a rough estimate is easy.
The basic rule: bytes per parameter
The space taken by model weights is the number of parameters multiplied by bytes per parameter:
| Precision | Per parameter | 7B model | 70B model |
| --- | --- | --- | --- |
| FP32 | 4 bytes | ~28 GB | ~280 GB |
| BF16 / FP16 | 2 bytes | ~14 GB | ~140 GB |
| FP8 / INT8 | 1 byte | ~7 GB | ~70 GB |
| INT4 / FP4 | 0.5 bytes | ~3.5 GB | ~35 GB |
That covers the weights only; real workloads need extra memory.
For inference
During inference, the KV cache consumes memory on top of the weights. It grows with context length, the number of concurrent requests (batch) and the model architecture. In practice, plan at least 20–30% headroom above the weights — and much more for long-context, high-concurrency services.
For fine-tuning
Full fine-tuning
Full fine-tuning with the AdamW optimizer and mixed precision needs roughly 16 bytes per parameter (weights, gradients and optimizer states), not counting activations. That's ~112 GB for a 7B model and ~1.1 TB for 70B, which is why full fine-tuning is usually done across multiple GPUs with distributed techniques such as ZeRO/FSDP.
LoRA
LoRA freezes the base model weights and trains only small adapter layers. Memory needs are roughly the BF16 weights plus activations.
QLoRA
QLoRA quantizes the base model to 4 bits, cutting memory once more. With this method a 70B model can be fine-tuned on a single 80 GB GPU.
Example scenarios and GPU recommendations
- 7B–8B inference: RTX 4090, L4 or L40S
- 7B–8B LoRA fine-tuning: L40S, RTX PRO 6000 or a single H100
- 70B inference (BF16): 2–4× H100, 1–2× H200 or a single B200
- 70B QLoRA: A single H100 or H200
- 70B full fine-tuning: A multi-node H100/H200 cluster, or an 8× B200 / B300 server
- Models with hundreds of billions of parameters: Multi-node B200 / B300 clusters
These estimates are a starting point; batch size, context length and your framework can change the numbers. Share your workload with us and we'll find the right configuration together.