LLM Quantization Size Calculator
GGUF file size = parameters × bits per weight ÷ 8. With llama.cpp's measured 4.89 bits per weight, Llama 3.1 8B in Q4_K_M is 4.58 GiB, against 7.95 GiB for Q8_0 and 14.96 GiB for F16 (llama.cpp quantize README, checked 16 October 2026).
How to use
- Pick a model to fill in parameters, layers, KV heads and head dimension, or type your own values from config.json.
- Set the context length, the KV cache precision and an overhead percentage.
- Read the table: bits per weight, file size and total memory for each quantization. The percentage next to a name is its size relative to F16.
How it works
File size = parameters × bits per weight ÷ 8. The tool uses the effective bits per weight that llama.cpp measured on Llama 3.1 8B: F16 16.00, Q8_0 8.50, Q6_K 6.56, Q5_K_M 5.70, Q4_K_M 4.89, Q3_K_M 4.00, Q2_K 3.16. Source: tools/quantize/README.md in the ggml-org/llama.cpp repository, checked 16 October 2026.
These are higher than the number in the name because K-quants store a scale and a minimum for each block of weights, and some tensors such as the embeddings and the output layer stay at higher precision. With 8.03 billion parameters the formula gives 4.58 GiB for Q4_K_M, the same as the README.
Memory = file size + KV cache + overhead. KV cache = 2 × layers × KV heads × head dimension × context × bytes per value, the same formula as the GPU VRAM calculator. Overhead is a percentage you set for buffers and the runtime.
Results are in GiB (2^30 bytes). Other models have a different mix of tensors, so their real files can be a few percent off.
File size by model
| Quantization | Llama 3.1 8B | Qwen2.5 32B | Llama 3.1 70B |
|---|---|---|---|
| F16 | 14.96 GiB | 61.02 GiB | 131.41 GiB |
| Q8_0 | 7.95 GiB | 32.42 GiB | 69.82 GiB |
| Q6_K | 6.14 GiB | 25.03 GiB | 53.91 GiB |
| Q5_K_M | 5.33 GiB | 21.75 GiB | 46.84 GiB |
| Q4_K_M | 4.58 GiB | 18.67 GiB | 40.20 GiB |
| Q3_K_M | 3.74 GiB | 15.24 GiB | 32.82 GiB |
| Q2_K | 2.95 GiB | 12.05 GiB | 25.95 GiB |
Which one to pick
- Q4_K_M is the common default: about 31 percent of F16 in size.
- Q5_K_M and Q6_K cost 16 and 34 percent more memory than Q4_K_M and stay closer to the original model.
- Q8_0 is about half of F16 and the closest to it in quality.
- Q3_K_M and Q2_K fit large models into small cards, with more loss in quality. The llama.cpp README lists perplexity for each type.
FAQ
How big is a 7B model in Q4_K_M?
About 0.57 GiB per billion parameters, so 7 billion parameters come to about 3.99 GiB. Mistral 7B v0.3 with 7.25 billion is about 4.13 GiB.
Why is Q4_K_M more than 4 bits?
Each block of weights stores its own scale values, and a few tensors keep higher precision. llama.cpp measured 4.89 bits per weight on Llama 3.1 8B.
Does a 70B model fit in 48 GB at Q4_K_M?
The file is about 40.2 GiB. With an 8,192 token context the KV cache adds 2.5 GiB, and with 10 percent overhead the total is about 47 GiB, so it is very tight.
How is this different from the GPU VRAM calculator?
It uses the same memory formula but puts every GGUF type in one table with measured bits per weight, so you can compare them at a glance.