Build a Ceph Tentacle 20.2 Cluster on Proxmox VE 9.2 with NVMe OSDs

Running VMs on local disks is risky. If one server fails, the VMs on it stop, and you may lose data. Ceph fixes this by keeping copies of your data on several servers. Proxmox VE has built-in tools (pveceph) that install and manage Ceph, so you do not need a separate storage system. In this guide, you will learn to build a Ceph Tentacle Cluster on Proxmox.

By the end of this guide, you will have a working three-node Proxmox cluster with shared Ceph storage on NVMe disks. Your VMs will keep running even if one node fails. 

• 3 Proxmox VE 9.2 nodes in one cluster.
• 3 Ceph Monitors (MON) and 3 Ceph Managers (MGR).
• 1 or more dedicated NVMe disks per node as OSDs.
• 1 replicated pool (3 copies, minimum 2) used as Proxmox RBD storage.
• A test VM with HA, live migration, and a node failure test.

Here are the example values for this guide; remember to change these names, IPs, and disk paths to match your servers.

Before you start to build a Ceph Tentacle Cluster on Proxmox, make sure your servers, disks, and network are ready.

You need at least three servers. Use servers that are the same or very close in size. A good base for each node:

• 8 or more CPU cores. Plan about 1 core for each Ceph service. NVMe OSDs can use 4 to 6 threads each under heavy load.
• 32 GB RAM or more. Ceph recommends at least 8 GiB of memory for each OSD for good performance. The OSD default is 4 GiB.
• One small disk or mirrored disks for the Proxmox OS.
• One or more empty NVMe disks for Ceph. Do not use the OS disk.
• No hardware RAID. Ceph needs the raw disk. Use an HBA or direct NVMe.
• 10 Gbps network at least. For NVMe, use 25 Gbps or more, because one NVMe disk can fill a 10 Gbps link.

For this setup, you can use bare-metal machines. Check the NVMe dedicated server hosting from PerLod if you need servers with NVMe disks and fast private networking.

Ceph sends a lot of data between nodes, so you must have a good networking plan. Splitting the traffic keeps your cluster fast and stable. Use three separate networks if you can:

1. Management and Corosync (1 Gbps is fine). This carries the Proxmox cluster traffic.
2. Ceph public network (10+ Gbps). This carries traffic between VMs, Proxmox, and Ceph services.
3. Ceph cluster network (25+ Gbps). This carries OSD copy and recovery traffic.

Note: Ceph recovery traffic is heavy. If it shares a link with Corosync, the Proxmox cluster can lose quorum. Remember to keep them apart.

Ceph on Proxmox needs a working Proxmox cluster. If you have not built one yet, follow the Proxmox HA cluster, Corosync, and quorum guide.

You must verify the cluster health on any node with the command below:

In the output, you must see Quorate: Yes and Nodes: 3.

To build a Ceph Tentacle Cluster on Proxmox, you must update Proxmox, set the Ceph network IPs, and check the network and time. Run all of these steps on all three nodes.

Start by updating all packages on each node. This makes sure every node runs the latest Proxmox 9.2 fixes before you install Ceph:

Then, check the version; the output should start with pve-manager/9.2:

If the kernel was updated, reboot the nodes one by one. Wait for each node to come back before you reboot the next one:

Next, you must give each node an IP address on the Ceph public network and on the Ceph cluster network. You must add these in the network file on every node, using a different IP on each one. Open the network file:

Leave your existing vmbr0 (management) lines as they are. Add the Ceph networks at the end of the file. The example below is for pve1. Change the interface names and IPs for your server and for each node:

Do not add a gateway to these two interfaces. Save the file and apply it:

On pve2 use 10.10.10.12 and 10.10.20.12. On pve3 use 10.10.10.13 and 10.10.20.13. You can find your real interface names with:

Before you install Ceph, you must check that the nodes can reach each other on both Ceph networks. A broken link will cause hard problems later:

Also, you can run a speed test with iperf3. Install it on all nodes:

On pve2 start the server, and on pve1 run the client for each network:

The result should be close to your link speed. Stop the server on pve2 with Ctrl+C.

Note: If you use the Proxmox firewall, you must allow Ceph traffic with the built-in Ceph firewall macro.

Ceph needs the same time on all nodes. If the clocks are different, Ceph shows health warnings. Proxmox VE uses chrony for time sync by default, so you only need to check it:

Now you can install the Ceph packages on every node. Proxmox has a built-in command that does this for you, so you don't need to add the Ceph repository manually.

If you do not have a Proxmox subscription, you can use the no-subscription repository:

If you have a subscription, use the enterprise repository:

Type y when it asks to continue. In Proxmox VE 9.2, Tentacle is already the default version, but --version setting makes it clear and safe.

Once you are done, check the repository file that was created:

For the no-subscription repository, it should look like this:

If you need to create or fix this file manually, you can open it with the command below, paste the lines above, save, and run the system update:

The no-subscription repository is for testing and non-production use. For production, use the enterprise repository. 

Check the Ceph version; you must see a version that starts with ceph version 20.2 and ends with tentacle (stable).

At this point, you must create the main Ceph config file. You tell Ceph which networks to use for public traffic and for cluster traffic. Only do this once, on the first node.

Run this only on pve1. It creates the Ceph config and sets the two networks:

This creates /etc/pve/ceph.conf. Proxmox shares this file with all nodes automatically, and it links it to /etc/ceph/ceph.conf. Check it with:

Look for these lines; your values must match your networks:

Now you must create the Ceph monitors and managers. Monitors keep the cluster healthy and in sync, and managers track the cluster status.

You must create three monitors and three managers, one on each node. On pve1, run:

This also creates the first manager. Then, run the same command on pve2 and on pve3:

Next, run the command below to create extra managers on pve2 and pve3:

Only one manager is active. The other two wait as standby and take over if the active one fails. 

Check the status on any node with the following commands:

You should see 3 monitors in quorum and one active manager with two standby. The health will show HEALTH_WARN because there are no OSDs yet. This is normal.

An OSD is the Ceph service that stores your data on a disk. You must turn the empty NVMe disk on each node into an OSD. First, find the right disk:

For more details, you can install nvme-cli and check:

Pick the empty NVMe disk, for example /dev/nvme1n1. Do not pick the disk that has /, /boot, or the Proxmox LVM on it.

Then, you must clean the disk only if it was used before. The pveceph osd create refuses disks that have old partitions, ZFS, or old Ceph data. To clean the disk, run:

Warning: This deletes all data on the disk. Check the disk name twice.

Next, use the command below to create the OSD:

Ceph finds NVMe disks and sets their class to nvme on its own. Run the same command on every node. If a node has more NVMe disks, run it for each one too, for example, /dev/nvme2n1.

Optional: Very fast NVMe disks can run more than one OSD each. For example, to run two OSDs on one disk:

Use this only if you have enough CPU and RAM. It cannot be combined with separate DB or WAL disks.

Once you are done, check the OSDs; all of them must be up and in, with the class nvme. Wait until ceph -s shows HEALTH_OK.

Optional: Ceph uses 4 GiB of memory for each OSD by default. If your nodes have enough RAM, you can raise it to the 8 GiB that Proxmox recommends:

Run this on any one node. It creates a replicated pool with 3 copies and a minimum of 2, and it adds the pool to Proxmox as storage:

Never set min_size to 1. That can cause data loss.

In a normal setup, a Ceph Tentacle Cluster on Proxmox does not need extra keyring files. Proxmox copies them for you. You can check the pool and storage:

In /etc/pve/storage.cfg you should see a block like this:

If you created the pool without --add_storages, you can add the storage manually:

Also, you can do it in the web UI by navigating to Datacenter > Storage > Add > RBD.

For an optional quick speed test, you can use the command below and then remove the test data:

At this point, you can create a test VM that stores its disk on Ceph. You will use this VM in the next steps to test live migration and node failure. We use a Debian 13 cloud image. Run this on pve1:

Change the password. If your network has no DHCP, set a fixed IP instead, for example, --ipconfig0 ip=192.168.10.50/24,gw=192.168.10.1.

Check that the disk is on Ceph with the commands below:

You should see vm-100-disk-0 in the list. To open the VM console and find its IP, run:

Log in as debian, run ip -br addr, then exit the console with Ctrl+O.

Because the VM disk is on shared Ceph storage, Proxmox only moves the VM memory. It does not copy the disk. You can start a ping to the VM from your laptop or another server with the VM IP:

In another terminal on pve1, migrate the VM while it is running:

Check where the VM is, and the ping should continue with little or no loss:

You can move it back with qm migrate 100 pve1 --online, or move it to pve3 the same way.

In this last test, you must shut down one node on purpose. You will see how Ceph and Proxmox HA keep your VM running. Remember to use a test cluster only. Do not run this test on a production cluster.

First, add the VM to Proxmox HA. With HA on, Proxmox restarts the VM on another node if its node fails:

Then, move the VM to pve3, so you can fail that node:

A normal poweroff is a clean shutdown, so HA may just move the VM first. For a real crash test, cut power with the server BMC (IPMI), or on pve3 run the command below:

Now check how Ceph and Proxmox react to the lost node. Run these commands on a node that is still online. On pve1, run:

In another terminal, run the command below to check the status:

• Ceph shows HEALTH_WARN. One host, its OSDs, and one monitor are down. The cluster still has 2 of 3 monitors, so quorum stays.
• Placement groups show active+undersized+degraded. Because min_size is 2 and two copies remain, reads and writes keep working.
• After a few minutes, Proxmox HA sees that pve3 is down and starts VM 100 on pve1 or pve2. Wait about 2 to 3 minutes.

Next, use the commands below to check the VM and the health details:

By default, Ceph marks down OSDs as out after 10 minutes. With only three nodes and 3 copies, there is no other host to rebuild the missing copy, so the data stays degraded until pve3 comes back.

At this point, you can power on the failed node and let Ceph heal. Ceph will copy the missing data back to the node's OSDs. This can take some time. Power on pve3 and watch the recovery with:

Wait until all OSDs are up and in, all PGs are active+clean, and the health is HEALTH_OK. You can move the VM back if you want:

To clean up the test, remove the VM from HA and delete it:

At this point, you built a full Ceph Tentacle Cluster on Proxmox. With three nodes, separate public and cluster networks, three monitors, managers, dedicated NVMe OSDs, a replicated pool, and RBD storage. Also, you have tested live migration and a node failure, and saw that the VM kept running on the other nodes.

For production, use more NVMe disks per node, 25 Gbps or faster links, a subscription repository, and monitoring. You can build this on multiple PerLod NVMe dedicated servers to get a fast and resilient Proxmox and Ceph platform.

We hope you enjoy this guide. For more detailed information, you can check the Proxmox VE Hyper-Converged Ceph Cluster documentation.