Training AI models locally: how much VRAM do you really need?

VRAM is not system RAM, and it does not automatically add up across GPUs

Every GPU retains its own memory. For execution or training, weights, activations, caches and buffers have to be available where compute needs them. A two-GPU workstation can be excellent for correctly parallelised jobs, but it is not generally interpreted as one GPU with the simple sum of its gigabytes.

NVIDIA’s LLM guidance offers a useful starting point: weight memory depends on parameter count, bytes per parameter and parallelism. Real projects need more than weights, though: context cache, activations, communication buffers, CUDA and operating headroom. A serious estimate includes that margin.

Define what you are doing with the model first

There is a substantial difference between occasional inference, development, adapter fine-tuning, training from scratch and serving an internal team. Quantisation can lower weight memory for local inference; fine-tuning and training introduce gradients, optimiser states and activations. For computer vision, resolution, batch size and architecture matter as much as class count.

Before choosing a GPU, we ask for framework and version, model family, target precision, context length, batch, dataset, training or inference, expected GPU count, and whether the desk will also render or run CAD. That list avoids expensive proposals that fail on a forgotten memory detail.

A sensible VRAM map

16 GB supports development, compact-model inference and well-sized vision or moderate fine-tuning. 32 GB opens substantially more room for models, datasets and contexts that do not settle into 16 GB. 96 GB is for professional AI projects that genuinely require that capacity, ECC memory and a broader platform. These are starting points to test against the exact profile, not marketing tiers.

If a requirement exceeds one GPU, the whole system design matters: a board with sufficient slots and PCIe lanes, PSU headroom, chassis, sustained temperatures, distribution software and the way cards communicate. Multi-GPU performance depends on software and topology; no fixed multiplication should be promised.

CPU, RAM and storage still shape the workflow

The GPU runs kernels, but the system must feed data, prepare samples, write checkpoints and remain stable. 64 GB RAM is a practical basis for many AI workstations; 128 GB or more may be needed for large datasets, preprocessing, several GPUs or parallel tooling. Fast NVMe helps with active datasets and checkpoints, while a second volume or network storage can solve capacity and backup.

A powerful desktop CPU serves many one-GPU workflows. Threadripper and Threadripper PRO make sense when you need many PCIe lanes, ECC RDIMM, higher capacity and a platform designed to grow. The right choice is not always the biggest one; it is the one that leaves margin for the next experiment without paying for unused capacity.

Catalog configurations to start the conversation

Estimate the reproducible way

Begin from a repeatable test: one model, one configuration, batch, context or resolution, and measured consumption. Then add iteration headroom rather than rounding to the limit. If production is planned, define concurrency and expected growth too. That distinguishes a development workstation from a machine expected to serve sustained load.