Home
Hardware
Configurator
Services
OEM Guides About us Warranty Contact
ESENDEFRITPT
Servers · Proxmox cluster

3-node Proxmox cluster with Ceph: what hardware to order for it to work well

A three-node cluster with Ceph keeps the virtual machines running even if an entire server goes down. It works very well when the hardware is right and very badly when it is not. This guide sums up what to ask of each node.

ACD&CO technical team10 November 20266 min read

Why three nodes

A cluster needs a majority to decide who is alive. With two nodes, if one goes down, the other cannot tell whether it is the one that failed. With three, the two that remain are still a majority and start the third’s machines.

Ceph, the distributed storage, keeps three copies of each piece of data by default, one on each node. That is why three is the reasonable minimum.

The network: the first thing to fall short

Ceph replicates every write over the network. With 1 GbE the cluster works, but it is slow and chokes when rebuilding after a failure. The serious minimum is 10 GbE, and with NVMe drives 25 GbE is advisable.

Ask for separate networks: one for storage, another for the virtual machines and a separate link, even if only 1 GbE, for the cluster’s internal communication. With three nodes, the storage network can be a direct mesh between them, with no switch.

Drives: server SSDs, no RAID

Ceph manages the drives itself, so no RAID controller is used: the drives are connected directly. And they have to be enterprise-class SSDs with power-loss protection. With consumer SSDs, write performance collapses and drive life is shortened.

As there are three copies, usable capacity is roughly a third of the total, and it is best not to go above 80% full.

Memory and processor

Besides the memory of the virtual machines, each Ceph drive reserves several gigabytes of memory for its service. With four drives per node, allow about 16 to 32 GB for storage alone. Always ECC.

In the processor, cores are what count: those of the virtual machines plus one or two per drive. And remember that the cluster must hold up with one node fewer: the other two have to cope with everything.

Identical nodes

The three nodes should be identical: same processor, same memory, same drives. That way machines move from one to another without surprises and the storage stays balanced. We build them in series, with the same parts and the same firmware version, and test them together.

Starting platforms for each node (excluding VAT, without the cluster’s memory and drives)
PlatformFor which clusterFrom
Ryzen serverSmall cluster: a few virtual machines and 10 GbE€1,285
EPYC server16 to 64 cores per node, six-channel memory and IPMI€3,561
EPYC 9005 serverUp to 96 cores and 1.5 TB per node, for dozens of machines€6,152

The price of each node depends on the memory, the drives and the network cards. We give you a fixed price with the complete configuration.

If the budget does not stretch: two nodes and replication

For a small company there is a simpler option: two nodes with ZFS replication every few minutes and a minimal third machine that only votes. It is not instant high availability, because the minutes since the last replication are lost, but it costs far less and is easy to maintain.

Frequently asked questions

Can I build a cluster with two nodes?

Yes, with a small third machine acting as arbiter and ZFS replication instead of Ceph. Ceph needs three real nodes to work safely.

Ceph or ZFS?

Ceph gives real high availability: if a node goes down, the machines carry on with their data up to date. It needs three nodes, fast networking and server SSDs. ZFS with replication is simpler and cheaper, and loses the last few minutes if a node fails.

Is 1 GbE networking enough?

For Ceph storage, not in production. It is fine for the cluster’s internal link and for management. Storage needs 10 GbE at least.

Can I mix different servers?

It works, but it complicates daily life: machines do not always live-migrate between different processors and the storage ends up unbalanced. Three identical nodes save a lot of trouble.

Shall we size your server?

Tell us how many people you are, which software you use and how much data you hold. We will propose the server in writing, with a price, within 12 working hours.

Ask for a proposal

A proposal within 12 working hours

Tell us what you need

Tell us what it is for and we will reply with a configuration and its price. If you already have a quote from another brand, send it over and we will prepare the equivalent.

  • A written reply within 12 working hours
  • A single point of contact, from quote to warranty
  • No obligation: we only use your details to reply

+34 613 17 68 06info@acdcompany.es

This field is required.
Check the email: something is missing.

Or call us: +34 613 17 68 06