GPU VRAM Calculator for Local LLMs
A 7B model at 4 bit needs about 3.3 GiB for its weights, and a Llama 3.1 8B class model adds about 1 GiB of KV cache for every 8,192 tokens of context.
How to use
- Pick a model preset to fill in parameters, layers, KV heads and head dimension, or choose Custom and type your own numbers from the model's config.json.
- Set the quantization bits of the weights (4 for Q4, 16 for FP16) and the KV cache precision.
- Enter the context length in tokens and the batch size, meaning how many sequences run at once.
- Read the total and the split into weights, KV cache and overhead. The table shows which common GPU sizes it fits.
How it works
Weight memory = parameters × bits ÷ 8 bytes. A 7 billion parameter model at 4 bits is 7,000,000,000 × 0.5 = 3.5 GB, or 3.26 GiB. At 16 bits the same model is 14 GB, which matches the rule of 2 GB per billion parameters in bfloat16 given in the Hugging Face guide "Optimizing LLMs for Speed and Memory".
KV cache = 2 × layers × KV heads × head dimension × context length × batch size × bytes per value. The 2 is for keys and values. Hugging Face gives the per token form 2 × 2 × layers × KV heads × head dimension for 16 bit values, in the blog post "Unlocking Longer Generation with Key-Value Cache Quantization".
Overhead is entered as a percentage of weights plus KV cache. It stands in for the CUDA context, activation buffers and allocator fragmentation. The default of 10 percent is an assumption, not a measured value.
All results are in GiB (1 GiB = 1,073,741,824 bytes). GPU makers print "24 GB" for a card with 24 GiB, so the fit table compares the total directly with 8, 12, 16, 24, 48 and 80. This is an estimate. Real usage differs by framework (llama.cpp, vLLM, Ollama, Transformers), by kernel and by how the cache is allocated.
What fills the VRAM
Two things dominate. The weights are fixed once the model and quantization are chosen. The KV cache grows with every token in the context and with every sequence in the batch, so long contexts and many parallel users are where a model that "fits" suddenly does not.
Models with grouped query attention (GQA) keep fewer KV heads than attention heads, which cuts the cache several times over. That is why the KV heads field matters more than the number of attention heads. Check the num_key_value_heads value in the model's config.json.
Weight size per billion parameters
| Bits per weight | GiB per 1B parameters | Example: 8B model |
|---|---|---|
| 16 (FP16 / BF16) | 1.86 | 14.90 GiB |
| 8 (INT8 / Q8) | 0.93 | 7.45 GiB |
| 6 (Q6) | 0.70 | 5.59 GiB |
| 5 (Q5) | 0.58 | 4.66 GiB |
| 4 (Q4) | 0.47 | 3.73 GiB |
| 3 (Q3) | 0.35 | 2.79 GiB |
| 2 (Q2) | 0.23 | 1.86 GiB |
Where the presets come from
- config.json of each repository: huggingface.co/Qwen/Qwen2.5-7B (also -14B, -32B, -72B), huggingface.co/Qwen/Qwen3-8B (also -32B), huggingface.co/mistralai/Mistral-7B-v0.3, huggingface.co/mistralai/Mistral-Nemo-Base-2407 and huggingface.co/microsoft/phi-4.
- Llama 3.1 8B and 70B: the Meta config.json files are gated on Hugging Face, so the values come from Table 3 of the Meta paper "The Llama 3 Herd of Models" (arXiv 2407.21783). Gemma 2 9B and 27B: Table 1 of the Google report "Gemma 2" (arXiv 2408.00118).
- Parameter counts are the safetensors totals shown by the Hugging Face API for each repository. Where config.json has no head_dim, the tool uses hidden_size ÷ attention heads.
FAQ
How much VRAM does a 7B model need at 4 bit?
About 3.3 GiB for the weights. With an 8,192 token context on a Llama 3.1 8B class model the KV cache adds about 1 GiB, so plan on roughly 5 GiB including overhead. An 8 GB card runs it, with room for a longer context.
Can a 24 GB card run a 32B model?
At 4 bit, yes. Qwen2.5 32B has about 32.8 billion parameters, so the weights take about 15.3 GiB. At an 8,192 token context the KV cache is 2 GiB, which brings the total near 19 GiB with 10 percent overhead.
Why does my real usage differ from the estimate?
Frameworks differ in overhead. vLLM pre-allocates most of the GPU for its KV cache and preempts requests when that space runs out, llama.cpp allocates the cache for the full context up front, and GGUF K-quants store extra scale values so their effective bits are higher than the nominal number. Raise the bits or the overhead if you see out of memory errors.
Does it work for mixture of experts models?
The weights still have to sit in memory, so enter the total parameter count, not the active count. The KV cache depends only on the attention layers, so use the layers and KV heads from the config. The presets here are dense models.
Gemma 2 uses a sliding window. Is the estimate lower?
Gemma 2 alternates local layers with a 4,096 token window and global layers. The tool counts the full context for every layer, so it gives an upper bound for Gemma 2 when the context is longer than the window.