Home
Hardware
Configurator
Services
OEM Guides About us Warranty Contact
ESENDEFRITPT
Private AI · Models

Open AI models to run locally: sizes, memory and the hardware each one needs

There are dozens of open models and a new one appears every month. To choose hardware you do not need to know them all: it is enough to know what size the one you need is. This guide sorts models by size and says how much memory each one requires.

ACD&CO technical team10 November 20266 min read

The families and what sets them apart

The most widely used open models come from a handful of families: Llama, Mistral, Qwen, Gemma, DeepSeek and Phi. Each publishes several sizes of the same model. Within a family, the larger size reasons better and the smaller one answers faster.

Size is measured in billions of parameters. It is the number that decides the memory, and therefore the hardware.

Quantisation: the same model in less memory

A model can be stored with less precision in each number. That is called quantising. At 4 bits, a model takes about four times less than the original and loses very little quality. It is what almost everyone uses locally.

The rule: at 4 bits, about 0.6 GB per billion parameters, plus a margin for the conversation.

What memory and hardware each size needs

Model sizes, memory at 4 bits and starting hardware
Model sizeMemory it takesWhat it is forStarting hardware
Up to 8 billionAbout 5 GBClassifying, extracting data, summarising short textsAny 8 GB graphics card
12 to 32 billion8 to 20 GBWork assistant, drafting, documents16 to 24 GB graphics card or Compact AI 32
About 70 billionAbout 42 GBFiner reasoning, long textsCompact AI 64 or two 24 GB graphics cards
More than 100 billionMore than 65 GBThe most capable open modelsCompact AI 128 or a 96 GB graphics card

A well-chosen mid-sized model handles most office work. Large ones pay off when questions are complex or documents very long.

Context: how much text it reads at once

Context is the text the model has in front of it: the question, the documents and the conversation. A larger context lets it read a whole contract, but it takes memory on top of the model. That is why a machine that only just loads the model falls short as soon as you give it a long document.

The licence: not all can be used the same way

“Open” does not mean the same in every case. Some models are published under licences such as Apache 2.0 or MIT, which allow commercial use with no strings. Others carry the maker’s own conditions. Before building a product on a model, read its licence.

Try two or three models with your own questions and in your own language before deciding. General rankings say little about how a model answers on your documents.

Frequently asked questions

Which is the best open model?

It depends on the job and changes every few months. What stays stable is size: a mid-sized model for everyday work and a large one for the hard cases. With well-sized hardware, changing model means downloading another file.

Does a model lose quality at 4 bits?

Very little in normal use. If the work demands maximum fidelity, it is used at 8 bits, which takes twice as much. That is why spare memory is worth having.

Do they work well in languages other than English?

The mid-sized and large models of the main families answer well in the major European languages. The smallest ones are weaker. It is worth testing them with real texts.

Can I use them in a product I sell?

With many, yes, but it depends on each model’s licence. Apache 2.0 and MIT licences allow it. Others have conditions that have to be read.

Shall we do the sums for your company?

Tell us how many people will use it and with which documents. We will reply with the hardware, the installation and a fixed price within 12 working hours.

Ask for a proposal

A proposal within 12 working hours

Tell us what you need

Tell us what it is for and we will reply with a configuration and its price. If you already have a quote from another brand, send it over and we will prepare the equivalent.

  • A written reply within 12 working hours
  • A single point of contact, from quote to warranty
  • No obligation: we only use your details to reply

+34 613 17 68 06info@acdcompany.es

This field is required.
Check the email: something is missing.

Or call us: +34 613 17 68 06