Ceph Reef storage cluster setup on bare metal lets you turn a group of servers into a unified, highly available storage platform. Instead of managing separate disks or RAID arrays on each machine, Ceph pools all your drives together and automatically handles replication, fault tolerance, and data distribution across the cluster.
In this guide, we will build a three-node Ceph Reef storage cluster on Ubuntu 24.04 bare-metal servers from PerLod Hosting, configure OSDs, create a replicated pool, and test RBD block storage.
Architecture and Requirements for Ceph Reef Storage Cluster Setup
We want to deploy a Ceph cluster, which:
All nodes include Ubuntu 24.04 LTS OS and are fully updated.
Ceph version 18.x, which is deployed via cephadm and containers.
Minimum nodes include 3 bare metal servers.
All nodes can host monitors (MON), managers (MGR), and OSD (Object Storage Daemon) to keep the setup simple.
If you are looking for reliable and high-performance bare metal servers, PerLod provides flexible plans that are best suited for your workload.
For example:
Hostname
IP address
Role
ceph-admin
10.0.0.11
Admin, MON, MGR, OSD
ceph-node1
10.0.0.12
MON, MGR, OSD
ceph-node2
10.0.0.13
OSD
ceph-node3
10.0.0.14
OSD (optional expansion)
Minimum hardware requirements per node include:
4 CPU cores and 8–16 GB RAM.
1 system disk for Ubuntu (40–80 GB).
1 or more empty raw disks for OSDs (HDD or SSD).
For the network, at least 1 GbE and 10 GbE is recommended for serious workloads.
Note: Make sure all nodes can resolve each other by hostname via DNS or /etc/hosts and have stable IP addresses.
Step 1. Prepare OS on Each Node for Ceph Storage Cluster
The first step is to prepare Ubuntu 24.04 OS on every node, including ceph-admin and every ceph-nodeX.
Run the system update and upgrade with the commands below:
Bash
sudoaptupdatesudoaptfull-upgrade-y
Reboot the system to apply the new kernel and packages:
Bash
sudoreboot
After reboot, install the required tools for Ceph with the command below:
These are the standard ports for Ceph monitors and the web dashboard.
Step 2. Install cephadm and ceph-common on Admin Node
The admin node needs the Ceph tooling that will manage the entire cluster. You must install cephadm on the ceph-admin server and point it to the Reef release, so it knows which Ceph images and packages to use.
Ubuntu 24.04 provides a cephadm package suitable for managing Ceph clusters via containers. Install cephadm with the commands below:
In your output, you must see a Reef (18.x) build line consistent with the cluster you plan to deploy.
Step 3. Bootstrap the Ceph Cluster on Admin Node
At this point, you can create the cluster and bring the first core services online.
Bootstrapping deploys the initial monitor and manager daemons on ceph-admin, enables the web dashboard with your chosen credentials, and writes the base configuration and keyrings that all other nodes will use.
On ceph-admin, run the initial bootstrap and dashboard setup with the command below:
Each command should return the hostname without asking for a password.
Step 5. Add All Nodes to Ceph Admin
In this step, you must add each server to the cluster so cephadm can schedule services on it. To do this, run the commands below from the ceph-admin node:
The ceph orch host command adds registers to each host with the cephadm orchestrator, making it eligible to run Ceph daemons.
List hosts with the command below:
Bash
sudocephorchhostls
You should see ceph-admin and all ceph-nodeX entries with the Online status.
Step 6. Deploy Ceph Monitoring and Manager Daemons for High Availability
In this step, you can deploy the monitors (MON) and managers (MGR) daemons, so they run on multiple servers, which keeps the cluster healthy even if one host goes down.
For a flexible cluster, you can target three monitors and two managers with the commands below:
Ceph automatically schedules these daemons across available hosts, typically one MON per node, until it reaches three.
Two managers (one active, one standby) ensure the orchestrator and dashboard remain available.
Verify the status with the command below:
Bash
sudoceph-s
In the output, you must see mon: 3 daemons, quorum, and mgr: 2 daemons, along with continued warnings about missing OSDs.
Step 7. Prepare Disks and Create OSDs for Ceph Cluster
At this point, your Ceph cluster still does not have any usable storage. In this step, you can pick which disks on each node will be used for data, and let cephadm automatically turn every free drive into an OSD.
On all nodes, run the command below to identify Ceph data disks:
Bash
lsblk
Confirm which devices, like /dev/sdb or /dev/sdc, are unused and reserved for Ceph OSDs. These disks must not contain partitions or filesystems if you want to use the automatic method.
From the ceph-admin node, use the command below to deploy OSDs automatically:
Bash
sudocephorchapplyosd--all-available-devices
Cephadm scans each node for free raw disks and creates one OSD per eligible device.
Ceph sets up LVM and mounts the OSD data into containers for you, so in standard setups, you don’t have to create or manage disk partitions manually.
Check the OSD status with the commands below:
Bash
sudocephorchps--daemon-type=osdsudocephosdtree
Then, run the command below:
Bash
sudoceph-s
The command output will list all OSDs as “up” and “in,” and the overall cluster health should get close to HEALTH_OK, except for some minor tuning suggestions about PGs.
Step 8. Access Ceph Storage Dashboard
At this point, you can access Ceph’s web interface so you can see and manage the cluster in a browser.
From the Ceph admin, use the command below to display the dashboard URL:
Bash
sudocephmgrservices
This command provides the dashboard URL, usually:
Bash
https://10.0.0.11:8443/
Open the URL in your desired browser:
Accept the self-signed certificate.
Log in with the admin credentials you set.
From the dashboard, you can:
View overall health and performance.
Check hosts, OSDs, pools, PGs, and utilization.
Create pools, enable RGW (S3), NFS, or iSCSI services.
Step 9. Create a Test Pool and Block Volume with Ceph RBD
Now that the OSDs are online, you can turn the raw storage into something clients can actually use.
You can create a replicated pool for RBD, create a small virtual disk in that pool, attach it to your server, and then format and mount it like a regular disk.
From the ceph-admin node, create an RBD pool with the commands below:
Verify network: Check link speed, latency, and packet loss.
Look for OSDs that are almost full, because Ceph will automatically slow down recovery and client I/O as disks get close to their “near full” and “full” limits.
Conclusion
With this setup guide, you can turn a small group of bare metal servers into a unified Ceph Reef storage cluster with a strong control layer and safely replicated data across several OSDs.
We hope you enjoy this Ceph Reef storage cluster setup guide.
Subscribe to our X and Facebook channels to get the latest updates and articles on the storage platforms.
For a reliable setup, use at least three servers so the monitors can agree on cluster state and your data can be safely copied across several OSDs.
Yes. In a small cluster, you can use one server as both the admin node and a storage node running MON, MGR, and OSD, as long as it has enough CPU, RAM, and disk.
If you run at least three MONs and use a replication size of 3, the cluster can keep working, and data stays available as long as enough OSDs are still online to meet the replica count and voting rules.
We use cookies to run PerLod and to understand how the site is used. You can accept everything, or open Manage to turn individual categories off. Cookie policy.