Setting Up a Ceph Reef Storage Cluster on Bare Metal
Ceph Reef storage cluster setup on bare metal lets you turn a group of servers into a unified, highly available storage platform. Instead of managing separate disks or RAID arrays on each machine, Ceph pools all your drives together and automatically handles replication, fault tolerance, and data distribution across the cluster.
In this guide, we will build a three‑node Ceph Reef storage cluster on Ubuntu 24.04 bare‑metal servers from PerLod Hosting, configure OSDs, create a replicated pool, and test RBD block storage.
Table of Contents
Architecture and Requirements for Ceph Reef Storage Cluster Setup
We want to deploy a Ceph cluster, which:
- All nodes include Ubuntu 24.04 LTS OS and are fully updated.
- Ceph version 18.x, which is deployed via cephadm and containers.
- Minimum nodes include 3 bare metal servers.
All nodes can host monitors (MON), managers (MGR), and OSD (Object Storage Daemon) to keep the setup simple.
If you are looking for reliable and high-performance bare metal servers, PerLod provides flexible plans that are best suited for your workload.
For example:
| Hostname | IP address | Role |
|---|---|---|
| ceph-admin | 10.0.0.11 | Admin, MON, MGR, OSD |
| ceph-node1 | 10.0.0.12 | MON, MGR, OSD |
| ceph-node2 | 10.0.0.13 | OSD |
| ceph-node3 | 10.0.0.14 | OSD (optional expansion) |
Minimum hardware requirements per node include:
- 4 CPU cores and 8–16 GB RAM.
- 1 system disk for Ubuntu (40–80 GB).
- 1 or more empty raw disks for OSDs (HDD or SSD).
For the network, at least 1 GbE and 10 GbE is recommended for serious workloads.
Note: Make sure all nodes can resolve each other by hostname via DNS or /etc/hosts and have stable IP addresses.
Step 1. Prepare OS on Each Node for Ceph Storage Cluster
The first step is to prepare Ubuntu 24.04 OS on every node, including ceph-admin and every ceph-nodeX.
Run the system update and upgrade with the commands below:
sudo apt update
sudo apt full-upgrade -yReboot the system to apply the new kernel and packages:
sudo rebootAfter reboot, install the required tools for Ceph with the command below:
sudo apt update
sudo apt install podman chrony lvm2 openssh-server -y- Docker or Podman is required because cephadm runs each Ceph daemon in a container.
- Chrony keeps node clocks in sync, which is essential for monitoring quorum and cluster health.
- LVM2 is used for OSD data volumes.
Once you are done, enable and start the services with the commands below:
sudo systemctl enable --now podman
sudo systemctl enable --now chronyd || sudo systemctl enable --now chrony
sudo systemctl enable --now sshAll nodes must have a running container runtime, NTP, and SSH at boot.
Also, you must set the hostname on each node. For example, for ceph-node1, run the command below:
sudo hostnamectl set-hostname ceph-node1Add the static entries to /etc/hosts on all nodes:
10.0.0.11 ceph-admin
10.0.0.12 ceph-node1
10.0.0.13 ceph-node2
10.0.0.14 ceph-node3This guarantees name resolution without relying on external DNS while you build the cluster.
If you are using the UFW firewall, adjust the firewall rules with the commands below:
sudo ufw allow 6789/tcp # MON
sudo ufw allow 3300/tcp # MON v2
sudo ufw allow 8443/tcp # DashboardThese are the standard ports for Ceph monitors and the web dashboard.
Step 2. Install cephadm and ceph-common on Admin Node
The admin node needs the Ceph tooling that will manage the entire cluster. You must install cephadm on the ceph-admin server and point it to the Reef release, so it knows which Ceph images and packages to use.
Ubuntu 24.04 provides a cephadm package suitable for managing Ceph clusters via containers. Install cephadm with the commands below:
sudo apt update
sudo apt install cephadm -y
cephadm --versionConfigure cephadm to use the Reef release and install CLI utilities with the commands below:
sudo cephadm add-repo --release reef
sudo cephadm install ceph-commonVerify the Ceph client:
ceph --versionIn your output, you must see a Reef (18.x) build line consistent with the cluster you plan to deploy.
Step 3. Bootstrap the Ceph Cluster on Admin Node
At this point, you can create the cluster and bring the first core services online.
Bootstrapping deploys the initial monitor and manager daemons on ceph-admin, enables the web dashboard with your chosen credentials, and writes the base configuration and keyrings that all other nodes will use.
On ceph-admin, run the initial bootstrap and dashboard setup with the command below:
sudo cephadm bootstrap \
--mon-ip 10.0.0.11 \
--initial-dashboard-user admin \
--initial-dashboard-password 'StrongPass123!'This generates /etc/ceph/ceph.conf, admin keyrings, and an SSH key for cephadm at /etc/ceph/ceph.pub.
Check cluster health with the command below:
sudo ceph -sNow you must have a live but empty Ceph Reef cluster that is ready for additional nodes, OSDs, and data.
Step 4. Configure SSH Access from Ceph Admin to All Nodes
Before you can add hosts or deploy daemons, the admin node needs passwordless SSH access to every server in the cluster.
On the ceph-admin node, run the command below to display the SSH public key created during bootstrap:
sudo cat /etc/ceph/ceph.pubCopy the key to all nodes from ceph-admin with the commands below:
ssh-copy-id -f -i /etc/ceph/ceph.pub root@ceph-node1
ssh-copy-id -f -i /etc/ceph/ceph.pub root@ceph-node2
ssh-copy-id -f -i /etc/ceph/ceph.pub root@ceph-node3Verify the SSH access with the commands below:
ssh root@ceph-node1 hostname
ssh root@ceph-node2 hostname
ssh root@ceph-node3 hostnameEach command should return the hostname without asking for a password.
Step 5. Add All Nodes to Ceph Admin
In this step, you must add each server to the cluster so cephadm can schedule services on it. To do this, run the commands below from the ceph-admin node:
sudo ceph orch host add ceph-node1 10.0.0.12
sudo ceph orch host add ceph-node2 10.0.0.13
sudo ceph orch host add ceph-node3 10.0.0.14The ceph orch host command adds registers to each host with the cephadm orchestrator, making it eligible to run Ceph daemons.
List hosts with the command below:
sudo ceph orch host lsYou should see ceph-admin and all ceph-nodeX entries with the Online status.
Step 6. Deploy Ceph Monitoring and Manager Daemons for High Availability
In this step, you can deploy the monitors (MON) and managers (MGR) daemons, so they run on multiple servers, which keeps the cluster healthy even if one host goes down.
For a flexible cluster, you can target three monitors and two managers with the commands below:
sudo ceph orch apply mon --placement="3"
sudo ceph orch apply mgr --placement="2"- Ceph automatically schedules these daemons across available hosts, typically one MON per node, until it reaches three.
- Two managers (one active, one standby) ensure the orchestrator and dashboard remain available.
Verify the status with the command below:
sudo ceph -sIn the output, you must see mon: 3 daemons, quorum, and mgr: 2 daemons, along with continued warnings about missing OSDs.
Step 7. Prepare Disks and Create OSDs for Ceph Cluster
At this point, your Ceph cluster still does not have any usable storage. In this step, you can pick which disks on each node will be used for data, and let cephadm automatically turn every free drive into an OSD.
On all nodes, run the command below to identify Ceph data disks:
lsblkConfirm which devices, like /dev/sdb or /dev/sdc, are unused and reserved for Ceph OSDs. These disks must not contain partitions or filesystems if you want to use the automatic method.
From the ceph-admin node, use the command below to deploy OSDs automatically:
sudo ceph orch apply osd --all-available-devicesCephadm scans each node for free raw disks and creates one OSD per eligible device.
Ceph sets up LVM and mounts the OSD data into containers for you, so in standard setups, you don’t have to create or manage disk partitions manually.
Check the OSD status with the commands below:
sudo ceph orch ps --daemon-type=osd
sudo ceph osd treeThen, run the command below:
sudo ceph -sThe command output will list all OSDs as “up” and “in,” and the overall cluster health should get close to HEALTH_OK, except for some minor tuning suggestions about PGs.
Step 8. Access Ceph Storage Dashboard
At this point, you can access Ceph’s web interface so you can see and manage the cluster in a browser.
From the Ceph admin, use the command below to display the dashboard URL:
sudo ceph mgr servicesThis command provides the dashboard URL, usually:
https://10.0.0.11:8443/Open the URL in your desired browser:
- Accept the self‑signed certificate.
- Log in with the admin credentials you set.
From the dashboard, you can:
- View overall health and performance.
- Check hosts, OSDs, pools, PGs, and utilization.
- Create pools, enable RGW (S3), NFS, or iSCSI services.
Step 9. Create a Test Pool and Block Volume with Ceph RBD
Now that the OSDs are online, you can turn the raw storage into something clients can actually use.
You can create a replicated pool for RBD, create a small virtual disk in that pool, attach it to your server, and then format and mount it like a regular disk.
From the ceph-admin node, create an RBD pool with the commands below:
sudo ceph osd pool create rbdpool 128
sudo ceph osd pool set rbdpool size 3
sudo ceph osd pool application enable rbdpool rbdThe rbdpool is a replicated pool with 128 placement groups and a replication size of 3.
Then, create and map an RDP image with the commands below:
sudo rbd create rbdpool/testimg --size 1G
sudo rbd map rbdpool/testimg
lsblk- The rbd create defines a 1 GB image inside rbdpool.
- The rbd map exposes it as a device like /dev/rbd0 on the host.
Finally, format and mount the RBD block device with the following commands:
sudo mkfs.ext4 /dev/rbd0
sudo mkdir -p /mnt/ceph-test
sudo mount /dev/rbd0 /mnt/ceph-test
df -h /mnt/ceph-testThis creates an ext4 filesystem on the RBD device and mounts it under /mnt/ceph-test.
When you see the new filesystem mounted with about 1 GB of space, you know your Ceph Reef cluster is serving block storage correctly.
Useful Ceph Management Commands
After the initial setup, you will mostly work with a set of commands to check cluster health, manage OSDs, and grow the cluster with new nodes.
To check the Ceph cluster health summary and a detailed list of any warnings or errors, you can use the commands below:
sudo ceph -s
sudo ceph health detailLists all Ceph hosts and all containerized daemons with the following commands:
sudo ceph orch host ls
sudo ceph orch psDisplay host and OSD layout and capacity, and utilization per OSD with the commands below:
sudo ceph osd tree
sudo ceph osd dfUse the “osd out” command before maintenance on a node to migrate data away:
sudo ceph osd out OSD_IDUse the “osd in” command to reintroduce the OSD to active service afterwards:
sudo ceph osd in OSD_IDTo add a new node later, repeat Step 1 for it, including packages, chrony, SSH, and hostnames. Then, from the Ceph admin, run the commands below:
ssh-copy-id -f -i /etc/ceph/ceph.pub root@ceph-node4
sudo ceph orch host add ceph-node4 10.0.0.15
sudo ceph orch apply osd --all-available-devicesThe new node will join the cluster, and any free disks will become OSDs, automatically increasing capacity and performance.
Quick Troubleshooting for Common Ceph Cluster Issues
Even a healthy Ceph cluster will sometimes show warnings or performance issues, so it helps to know a few quick checks.
1. Cluster shows HEALTH_WARN about PGs or OSDs:
Run the command below to extract messages:
sudo ceph health detailUse the commands below to verify all OSDs are up and not overfull.:
sudo ceph osd tree
sudo ceph osd df2. An OSD is down:
Check OSD on the node with:
sudo systemctl status ce******@**.service
or
sudo cephadm logs --name osd.IDIf the disk failed, replace it and rerun:
ceph orch apply osd --all-available-devicesOr use a specific OSD spec.
3. Slow performance:
- Verify network: Check link speed, latency, and packet loss.
- Look for OSDs that are almost full, because Ceph will automatically slow down recovery and client I/O as disks get close to their “near full” and “full” limits.
FAQs
How many nodes do I need for a Ceph Reef cluster on bare metal?
For a reliable setup, use at least three servers so the monitors can agree on cluster state and your data can be safely copied across several OSDs.
Can I run the Ceph admin node and OSDs on the same server?
Yes. In a small cluster, you can use one server as both the admin node and a storage node running MON, MGR, and OSD, as long as it has enough CPU, RAM, and disk.
What happens if one of the Ceph nodes fails?
If you run at least three MONs and use a replication size of 3, the cluster can keep working, and data stays available as long as enough OSDs are still online to meet the replica count and voting rules.
Conclusion
With this setup guide, you can turn a small group of bare metal servers into a unified Ceph Reef storage cluster with a strong control layer and safely replicated data across several OSDs.
We hope you enjoy this Ceph Reef storage cluster setup guide.
Subscribe to our X and Facebook channels to get the latest updates and articles on the storage platforms.
For further reading:
