Will it run this LLM?
Pick a model and quantization to see which GPUs run it locally — with fit verdicts, estimated speed tiers, and the best-value card if you're shopping.
Llama 3.1 8B @ Q4 needs ~5.6 GB of memory (weights + 8K-context KV cache).
| GPU | Memory | Verdict | Est. speed |
|---|---|---|---|
| NVIDIA RTX 5090 32GBreview | 32 GB | Fits | Blazing (>50 tok/s) |
| NVIDIA RTX 4090 24GB | 24 GB | Fits | Blazing (>50 tok/s) |
| NVIDIA RTX 5080 16GBreview | 16 GB | Fits | Blazing (>50 tok/s) |
| AMD RX 7900 XTX 24GB | 24 GB | Fits | Blazing (>50 tok/s) |
| NVIDIA RTX 3090 24GB | 24 GB | Fits | Blazing (>50 tok/s) |
| NVIDIA RTX 5070 Ti 16GB | 16 GB | Fits | Blazing (>50 tok/s) |
| NVIDIA RTX 4080 Super | 16 GB | Fits | Blazing (>50 tok/s) |
| NVIDIA RTX 4070 Ti Super | 16 GB | Fits | Blazing (>50 tok/s) |
| AMD RX 9070 XT 16GBreview | 16 GB | Fits | Blazing (>50 tok/s) |
| Mac Studio M4 Max 128GB | 96 GB (unified) | Fits | Blazing (>50 tok/s) |
| NVIDIA RTX 3060 12GB | 12 GB | Fits | Fast (25–50 tok/s) |
| Mac Mini M4 Pro 64GB | 48 GB (unified) | Fits | Fast (25–50 tok/s) |
| Ryzen AI Max+ 395 (Strix Halo) 128GB | 96 GB (unified) | Fits | Fast (25–50 tok/s) |
| NVIDIA RTX 4060 Ti 16GB | 16 GB | Fits | Fast (25–50 tok/s) |
| NVIDIA RTX 4060 8GB | 8 GB | Fits | Fast (25–50 tok/s) |
NVIDIA RTX 3060 12GB
$289 · Price as of Jul 14, 2026
Speeds are bandwidth-derived estimates (● = measured). Unified-memory machines list usable GPU-allocatable memory, not total RAM.
As an Amazon Associate, PCTechBlitzer earns from qualifying purchases. Full disclosure →
Memory required = model weights at the chosen quantization (Q4 ≈ 0.58 bytes/parameter through FP16 = 2 bytes/parameter) plus a KV-cache allowance for ~8K context. A GPU fits when its memory covers that; fits with offload when it's within 1.5×.
Speed is memory-bandwidth-bound during decoding, so we estimate tokens/sec as bandwidth ÷ model size × an efficiency factor, bucketed into coarse tiers. Measured numbers from our reviews (marked ●) override estimates. We update the dataset as we review new hardware.
How much VRAM does a 70B model need?
At Q4 quantization, Llama 3.1 70B needs about 43 GB of memory (weights plus an 8K-context KV cache); at Q8 that grows to about 77 GB. No single consumer GPU fits it fully at Q4 — you need a unified-memory machine, multiple GPUs, or partial CPU offload.
Can an RTX 4090 run Llama 70B?
No — Llama 3.1 70B at Q4 needs about 43 GB, far beyond the NVIDIA RTX 4090 24GB's 24 GB, even allowing for CPU offload.
What quantization should I use?
Q4 is the sweet spot for local inference: roughly a quarter of the FP16 memory footprint with minimal quality loss for most tasks. Use Q8 when you have VRAM to spare and want quality headroom; FP16 is rarely worth it locally.
Can I run LLMs on a Mac?
Yes. Apple Silicon's unified memory lets the GPU address most of system RAM — a Mac Studio M4 Max 128GB exposes ~96 GB to models, enough to run Llama 3.1 70B at Q4 fully in memory at slow speeds. Bandwidth, not capacity, is the constraint versus a discrete GPU.
What does 'fits with offload' mean?
The model is bigger than your GPU memory, but close enough (within 1.5×) that llama.cpp/Ollama can keep most layers on the GPU and run the rest on the CPU. It works, but expect the speed tier to drop to Slow regardless of how fast the card is.
How accurate are the tokens-per-second estimates?
LLM decoding is memory-bandwidth-bound, so we estimate speed as bandwidth ÷ model size × an efficiency factor (~0.6 for discrete GPUs, ~0.7 for unified memory), then bucket into coarse tiers. Where we have measured numbers from our own reviews, those override the estimate and are marked ●.