Why three nodes
A cluster needs a majority to decide who is alive. With two nodes, if one goes down, the other cannot tell whether it is the one that failed. With three, the two that remain are still a majority and start the third’s machines.
Ceph, the distributed storage, keeps three copies of each piece of data by default, one on each node. That is why three is the reasonable minimum.
The network: the first thing to fall short
Ceph replicates every write over the network. With 1 GbE the cluster works, but it is slow and chokes when rebuilding after a failure. The serious minimum is 10 GbE, and with NVMe drives 25 GbE is advisable.
Ask for separate networks: one for storage, another for the virtual machines and a separate link, even if only 1 GbE, for the cluster’s internal communication. With three nodes, the storage network can be a direct mesh between them, with no switch.
Drives: server SSDs, no RAID
Ceph manages the drives itself, so no RAID controller is used: the drives are connected directly. And they have to be enterprise-class SSDs with power-loss protection. With consumer SSDs, write performance collapses and drive life is shortened.
As there are three copies, usable capacity is roughly a third of the total, and it is best not to go above 80% full.
Memory and processor
Besides the memory of the virtual machines, each Ceph drive reserves several gigabytes of memory for its service. With four drives per node, allow about 16 to 32 GB for storage alone. Always ECC.
In the processor, cores are what count: those of the virtual machines plus one or two per drive. And remember that the cluster must hold up with one node fewer: the other two have to cope with everything.
Identical nodes
The three nodes should be identical: same processor, same memory, same drives. That way machines move from one to another without surprises and the storage stays balanced. We build them in series, with the same parts and the same firmware version, and test them together.
| Platform | For which cluster | From |
|---|---|---|
| Ryzen server | Small cluster: a few virtual machines and 10 GbE | €1,285 |
| EPYC server | 16 to 64 cores per node, six-channel memory and IPMI | €3,561 |
| EPYC 9005 server | Up to 96 cores and 1.5 TB per node, for dozens of machines | €6,152 |
The price of each node depends on the memory, the drives and the network cards. We give you a fixed price with the complete configuration.
If the budget does not stretch: two nodes and replication
For a small company there is a simpler option: two nodes with ZFS replication every few minutes and a minimal third machine that only votes. It is not instant high availability, because the minutes since the last replication are lost, but it costs far less and is easy to maintain.
Related hardware and guides
Frequently asked questions
Can I build a cluster with two nodes?
Yes, with a small third machine acting as arbiter and ZFS replication instead of Ceph. Ceph needs three real nodes to work safely.
Ceph or ZFS?
Ceph gives real high availability: if a node goes down, the machines carry on with their data up to date. It needs three nodes, fast networking and server SSDs. ZFS with replication is simpler and cheaper, and loses the last few minutes if a node fails.
Is 1 GbE networking enough?
For Ceph storage, not in production. It is fine for the cluster’s internal link and for management. Storage needs 10 GbE at least.
Can I mix different servers?
It works, but it complicates daily life: machines do not always live-migrate between different processors and the storage ends up unbalanced. Three identical nodes save a lot of trouble.
Shall we size your server?
Tell us how many people you are, which software you use and how much data you hold. We will propose the server in writing, with a price, within 12 working hours.