The families and what sets them apart
The most widely used open models come from a handful of families: Llama, Mistral, Qwen, Gemma, DeepSeek and Phi. Each publishes several sizes of the same model. Within a family, the larger size reasons better and the smaller one answers faster.
Size is measured in billions of parameters. It is the number that decides the memory, and therefore the hardware.
Quantisation: the same model in less memory
A model can be stored with less precision in each number. That is called quantising. At 4 bits, a model takes about four times less than the original and loses very little quality. It is what almost everyone uses locally.
The rule: at 4 bits, about 0.6 GB per billion parameters, plus a margin for the conversation.
What memory and hardware each size needs
| Model size | Memory it takes | What it is for | Starting hardware |
|---|---|---|---|
| Up to 8 billion | About 5 GB | Classifying, extracting data, summarising short texts | Any 8 GB graphics card |
| 12 to 32 billion | 8 to 20 GB | Work assistant, drafting, documents | 16 to 24 GB graphics card or Compact AI 32 |
| About 70 billion | About 42 GB | Finer reasoning, long texts | Compact AI 64 or two 24 GB graphics cards |
| More than 100 billion | More than 65 GB | The most capable open models | Compact AI 128 or a 96 GB graphics card |
A well-chosen mid-sized model handles most office work. Large ones pay off when questions are complex or documents very long.
Context: how much text it reads at once
Context is the text the model has in front of it: the question, the documents and the conversation. A larger context lets it read a whole contract, but it takes memory on top of the model. That is why a machine that only just loads the model falls short as soon as you give it a long document.
The licence: not all can be used the same way
“Open” does not mean the same in every case. Some models are published under licences such as Apache 2.0 or MIT, which allow commercial use with no strings. Others carry the maker’s own conditions. Before building a product on a model, read its licence.
Related hardware and guides
Frequently asked questions
Which is the best open model?
It depends on the job and changes every few months. What stays stable is size: a mid-sized model for everyday work and a large one for the hard cases. With well-sized hardware, changing model means downloading another file.
Does a model lose quality at 4 bits?
Very little in normal use. If the work demands maximum fidelity, it is used at 8 bits, which takes twice as much. That is why spare memory is worth having.
Do they work well in languages other than English?
The mid-sized and large models of the main families answer well in the major European languages. The smallest ones are weaker. It is worth testing them with real texts.
Can I use them in a product I sell?
With many, yes, but it depends on each model’s licence. Apache 2.0 and MIT licences allow it. Others have conditions that have to be read.
Shall we do the sums for your company?
Tell us how many people will use it and with which documents. We will reply with the hardware, the installation and a fixed price within 12 working hours.