Estimate the VRAM an LLM needs and check whether it fits on your GPU.
Compare per-request and monthly API costs across OpenAI, Anthropic and Gemini, with the price check date.
Open →Paste a document to see RAG chunking and embedding cost estimates live. Your text is never uploaded.
Open →Convert between units across 12 categories, including length, weight, temperature, area and volume.
Open →No comments yet. Be the first!
The LLM VRAM Calculator estimates how much memory you need to RUN a large language model locally or self-hosted. Total ≈ weights + KV cache + runtime overhead, where weights = parameters × bits per weight ÷ 8 and KV cache = 2 × layers × KV heads × head dimension × bytes per element × context × concurrent sequences. Choose a model, quantization and device and it tells you whether it fits, the headroom, the longest context that fits and a generation-speed ceiling.
Total ≈ weights + KV cache + runtime overhead. Weights = parameters × bits per weight ÷ 8; for example an 8-billion-parameter model needs about 15 GiB in FP16 (16 bits) and about 4.6 GiB in Q4_K_M (about 4.9 bits). The KV cache grows with context length and the number of concurrent sequences, so set both.
For every token processed the model stores each layer's Key and Value so it need not recompute them while generating. Size = 2 × layers × KV heads × head dimension × bytes per element × context length × sequences, so it is proportional to context length. Models with grouped-query attention (fewer KV heads) or sliding windows need much less.
Fewer bits make the weights smaller and usually faster, at the cost of quality. Q8_0 is close to the original, Q5_K_M and Q6_K are common compromises, Q4_K_M is the usual starting point when memory is tight, and quality loss becomes noticeable below Q3. The table shows whether each level fits under your current settings.
Every expert has to be loaded, so weights use the TOTAL parameters, not the parameters active for each token. Generation speed, however, depends on the ACTIVE parameters, which is why MoE models often use lots of memory yet are not especially slow.
Usually not. By default the system lets the GPU use only part of the unified memory (roughly two-thirds to three-quarters), so choosing "Apple unified memory" treats 75% as usable. Pick "Custom usable share" to match your actual setting.
The weights and KV cache are assumed to split evenly across cards and every card pays its own fixed runtime overhead, so the total is slightly higher than on one card. The actual split (tensor or pipeline parallelism) and communication overhead depend on the engine, so leave extra margin.
It is a planning estimate for INFERENCE only: weights, KV cache and an adjustable overhead. Real use varies with the engine (vLLM, llama.cpp and others), batching, CUDA version and paged-KV settings. Training and fine-tuning need far more memory (gradients, optimizer state, activations) and are outside this tool.
Generating each token reads the active weights once, so at batch size 1 the ceiling is roughly "memory bandwidth ÷ weight bytes read per token". It is a theoretical limit that ignores compute limits, KV-cache reads and software overhead, so real speed is usually lower. It only appears after you enter your device's bandwidth.
| Quantization | bpw | Weights | Total | Fits? |
|---|---|---|---|---|
| FP32 | 32 | 29.92 GiB | 34.91 GiB | No |
| FP16 | 16 | 14.96 GiB | 18.45 GiB | Yes |
| Q8_0 | 8.5008 | 7.95 GiB | 10.74 GiB | Yes |
| FP8 | 8 | 7.48 GiB | 10.23 GiB | Yes |
| Q6_K | 6.5633 | 6.14 GiB | 8.75 GiB | Yes |
| Q5_K_M | 5.7036 | 5.33 GiB | 7.87 GiB | Yes |
| Q5_K_S | 5.5704 | 5.21 GiB | 7.73 GiB | Yes |
| Q4_K_M | 4.8944 | 4.58 GiB | 7.03 GiB | Yes |
| Q4_K_S | 4.6672 | 4.36 GiB | 6.80 GiB | Yes |
| Q3_K_L | 4.2979 | 4.02 GiB | 6.42 GiB | Yes |
| Q3_K_M | 3.996 | 3.74 GiB | 6.11 GiB | Yes |
| Q3_K_S | 3.6429 | 3.41 GiB | 5.75 GiB | Yes |
| Q2_K | 3.1593 | 2.95 GiB | 5.25 GiB | Yes |
Model specs come from Hugging Face config.json files and safetensors metadata (verified 2026-10-20); quantization bpw values are llama.cpp measurements on Llama-3.1-8B and differ slightly for other models.
This is a planning estimate of the memory needed to RUN (infer) a model, not to train or fine-tune it. Real use varies with the engine (vLLM, llama.cpp, TensorRT-LLM…), batching and CUDA version, so leave some margin.
