Two different limits
The first is about users: the model fits and answers well, but with many people at once the queue grows. The second is about size: you want a larger model, or faster answers from the same one, and the machine’s memory has no more to give.
It is worth knowing which of the two you have before buying anything.
What adding compacts does solve
Several identical compacts, each with its own copy of the model and a request balancer in front, serve more people at once. Three machines serve roughly three times as many. And if one fails, the others carry on: there is redundancy without paying for a duplicate server.
You grow in steps, buying another identical machine when needed, and they can be spread across sites if the company has several.
What it does not solve
Adding machines does not make a model bigger. A model can be split across several computers over the network, and it fits, but the network is far slower than memory and the answer suffers. For a larger model you need a machine with more memory, not more machines.
When it is time for a server with graphics cards
A server with NVIDIA graphics cards serves many simultaneous users on the same model, because it groups requests and processes them together. It is also the machine for tuning models with your data and for software that requires CUDA. In exchange, it uses more power, makes more noise and needs a rack cabinet or a room.
| Option | What it gives | Price |
|---|---|---|
| 1 Compact AI 64 | A large model for a small team | €2,940 |
| 3 Compact AI 64 | Three times the users, plus redundancy | €8,820 |
| 1 Compact AI 128 | Larger models or several models at once | €4,820 |
| AI Rack server, 24 GB RTX PRO 4000 | Many users on mid-sized models, in the rack | €6,407 |
| AI Threadripper PRO server, 96 GB RTX PRO 6000 | Large models for the whole company, expandable to three cards | €32,578 |
How to decide
If the problem is that there are more of you and the current model is good enough, add compacts. If the problem is that the model is too small, move to the 128 GB one. If you need both, or training, it is time for the server.
In many companies they coexist: the server at headquarters and a compact at each branch or on board.
Related hardware and guides
Frequently asked questions
Can I join several mini PCs to run a larger model?
You can, by splitting the model over the network, but the answer is much slower than on a single machine with enough memory. For a larger model, a machine with more memory is the better buy.
How many people can one compact serve?
It depends on the model and on how much they use it. As a guide, one compact serves a small team that queries it now and then. If many people type at once, the wait is noticeable and it is worth adding machines or moving to a dedicated graphics card.
Which uses more power, three compacts or a server?
Usually the server with graphics cards, especially under load. The compacts have low-power processors. If power or noise is a problem, that weighs in favour of the compacts.
Can compacts and a server be mixed?
Yes, and it is common: the server at headquarters for large models and training, and compacts at branches, on site or on board, where there is no rack and no reliable connection.
What will you use it for?
Tell us which models you want to run, how many people will use it and where it will work. We will propose the compact, with or without a professional graphics card, and its price within 12 working hours.