How to Fix Ceph HEALTH_WARN and HEALTH_ERR on Proxmox VE

Proxmox Ceph problems usually come from capacity, disk, network, or placement group (PG) state. If you know which one you have, you can pick a safe fix and avoid the commands that destroy data. This guide is a runbook. You can start with a few read-only checks, find the cause, and then apply the fix in a safe order. 

These rules keep Proxmox Ceph troubleshooting safe during every incident:

1. Look first, change later. Run read-only commands before you fix anything.
2. Change one thing at once. Then, check ceph -s again.
3. Do not remove disks or nodes while PGs are degraded. Fewer copies means less safety.
4. Never set min_size to 1 to make a pool work again. It risks data loss.
5. Avoid these commands unless support tells you to use them: ceph osd purge, ceph osd lost, ceph pg force-create-pg, ceph-objectstore-tool, and pool delete.
6. Wait until Ceph is already recovering. Many warnings go away on their own when recovery ends.

Before you start, make sure your cluster is built correctly. If you are setting up a new one, follow our guide to build a Ceph Tentacle cluster on Proxmox with NVMe.

Before you fix anything, you need to know what is wrong. Start with these commands:

• ceph -s shows the overall health, the monitors, the OSD count, the PG states, and the client I/O.
•  ceph health detail explains each warning and names the OSDs, PGs, or monitors involved.

Also, you can open the Proxmox web interface and go to Node > Ceph. The health levels mean:

Next, you can check more details with the following commands:

As we said, most Ceph warnings come from capacity, disk, network, time, or PG state. You can use the following table to match your warning to its cause. This helps you pick the right fix and avoid commands that can harm your data.

Ceph has three fill limits. Each one is a percentage of how full an OSD is, and each one triggers a different action. These are the defaults:

You can check the current values and the OSD usage with the commands below:

Look at the %USE and VAR columns. If one OSD is far above the others, the data is not balanced. If all OSDs are high, the cluster is simply running out of space.

Sometimes the cluster has enough free space, but a few OSDs hold more data than the others. This can trigger nearfull warnings on those OSDs. The fix is to move some data to the OSDs that have more free space. 

Turn on the balancer. It slowly moves PGs so all OSDs have a similar amount of data:

The upmap mode needs all Ceph clients to be version Luminous or newer. Current Proxmox VE meets this. Also, you can lower the weight of the fullest OSDs, so they hold less data. Run a dry test:

Also, check the PG count. If a pool has too few PGs, the data cannot spread well across OSDs:

Sometimes all OSDs are close to full, not just a few. In this case, moving data will not help, because there is no free space left to use. You need to free space or add more storage.

• Delete data you do not need, such as old VM disks, old backups, and old snapshots.
• Add more OSDs or more nodes. This is the real fix.
• Do not add a new OSD to a cluster that is already full and unstable. Stabilize it first.

If you see backfill_toofull in PG states, the target OSDs are too full. You can raise the backfillfull limit a little to let recovery finish:

Do this only as a short fix, and set it back to the default after recovery ends:

When an OSD reaches the full limit, Ceph stops client writes to protect your data. VMs that use this storage can freeze or show I/O errors. This is an emergency, so you must free space. 

You can raise the full ratio by a very small amount, remove data, then set it back:

Do not keep the higher value. Ceph needs free space to recover from failures.

A slow ops warning in Proxmox Ceph means some requests take too long. It looks like this:

The cause is usually a bad disk, a busy or broken network, or an OSD that is stuck.

Before you fix slow ops, you need to know where they come from. To do this, run the commands below:

One OSD with a much higher value than the others points to a disk problem. Many OSDs with high values on the same host or switch point to a network problem.

On the node that hosts the slow OSD, you must look at the requests:

These show what each request is waiting for. Waiting for a "sub op" or "replica" usually means another OSD or the network is slow.

If one OSD is much slower than the others, its disk may be failing. Run the following commands on the node that hosts the slow OSD:

Look for media errors, reallocated sectors, high temperature, or I/O errors. If the disk is failing, go to Step 4.

Sometimes the disk is fine, but the OSD is slow because the node is busy. Run the commands below to check for other load:

Heavy recovery, a big backup job, or a deep scrub can slow client I/O. If recovery is the cause, you must give VMs a higher priority. On current Ceph releases, you can use the mClock profile:

After the cluster is healthy, set it back with the command below:

Restarting an OSD can make things worse, because it starts peering again. Restart one OSD only when its logs show it is stuck, and only when the cluster has enough healthy copies:

Ceph needs a fast and stable network, so network checks are a key part of Proxmox Ceph troubleshooting. A bad network can cause slow ops, OSDs that keep going down and up, and slow recovery.

First, check the Ceph network settings in the config file:

• The public network is used by clients, monitors, and OSDs.
• The cluster network is used by OSDs to copy data, recover data, and check on each other.

For best results, you can use a separate and fast storage network. If you run Ceph on rented hardware, dedicated servers with a private storage network keep Ceph traffic away from VM and internet traffic.

A slow or broken link between nodes can make Ceph slow. These tests show if the link has errors, runs at the right speed, and can carry enough traffic:

Replace eno1 and the IP with your storage interface and a peer node. Look for errors, drops, and wrong speed.

Then, test the speed between nodes with iperf3:

A 10 Gbit link should give close to 9.4 Gbit/s. A much lower result means a bad cable, port, or switch setting.

A wrong MTU is a common hidden problem. If you use jumbo frames (MTU 9000), every node and every switch port must use it. Test it with:

If this fails but a normal ping works, the MTU does not match somewhere. Fix the MTU on all nodes and switch ports, or go back to 1500 everywhere.

Ceph services talk to each other on set network ports. If a firewall blocks them, nodes cannot connect, and you may see slow ops, OSDs going down, or monitor problems. 

Ceph monitors use ports 3300 and 6789. OSDs and MGRs use the range 6800 to 7568. Make sure your firewall allows them between all Ceph nodes.

If VM traffic, backups, and Ceph recovery share one link, a large backup can cause Ceph slow ops. You must move Ceph to its own network, or at least to its own VLAN with enough speed. NVMe disks need 25 Gbit/s or more to use their speed in many setups.

An OSD can fail because of a bad disk, a crashed service, or a network problem. Also, it can keep going down and up again. Both cases can lead to degraded PGs and slow ops.

In this step, you check the OSD, try to bring it back, and replace the disk only if it is really bad. First, you must look at the status:

Sometimes an OSD is down, but the disk has no errors. The service may have stopped or crashed. In this case, you can try to start it again:

If the service fails, read the log for the reason. Common reasons are a missing disk, a full root disk, or a LVM problem. Check with:

Sometimes an OSD goes down, comes back, and then goes down again. This is usually caused by a network problem or a slow disk. To stop data from moving while you work, you can set these flags for a short time:

noout stops Ceph from marking down OSDs as out, so it does not start rebalancing. nodown stops monitors from marking OSDs down. Use nodown only for a short test, because it hides real failures.

Always remove the flags once you are finished:

By default, Ceph marks a down OSD as out after 10 minutes (600 seconds), and then it starts to rebuild the data on other OSDs.

If the disk has errors or has failed, the OSD must be replaced. Do this slowly and in the right order, so you do not lose data.

First, make sure the rest of the cluster is healthy enough. Other PGs should be active.

Then, mark the OSD out so data moves away from it; if it is still readable, this is the safest way:

Wait until all PGs are active+clean. Watch with ceph -s. Next, check that it is safe to remove it:

Now you can stop and destroy the OSD with Proxmox tools:

Finally, you can replace the disk and create a new OSD:

If the disk has already failed and cannot be read, Ceph rebuilds the data from the other copies. Do not wait for active+clean in that case, but make sure you still have enough copies before you remove anything.

Only do one disk at once. Do not replace a second disk until the cluster is active+clean again.

After a crash, you can check what happened and clear the warning with:

Use archive-all only after you read the crash reports.

Ceph monitors must agree with each other, and their clocks must be the same. If a monitor goes offline or the time is off, Ceph can show warnings or stop working. More than half of the monitors must be online. If you have 3 monitors, at least 2 must work.

If the ceph commands hang, quorum may be lost. You can add a timeout to see the error:

You can ask one monitor about its own state, even if there is no quorum. Run this on the monitor node:

If the state is probing, electing, or synchronizing, the monitor is out of quorum. A monitor stays in probing until it finds enough other monitors.

Ceph monitors need clocks that are the same. If the time differs by more than 0.05 seconds, Ceph shows a clock skew warning. The fix is to make all nodes use the same time source. Check the time on every node:

Proxmox VE uses chrony by default. You must make sure the service runs and the nodes can reach their time servers:

chronyc makestep forces a fast time correction. Use the same time servers on all nodes. You can set them in /etc/chrony/chrony.conf or in a file in /etc/chrony/sources.d/. If your nodes are virtual machines, their time can go wrong more easily, so check the time on the host too.

The warning may take a few minutes to go away after the time is fixed. Do not raise mon_clock_drift_allowed to hide it. Fix the time instead.

If a monitor is down, the cluster can still work as long as most monitors are online. But you should fix it fast, because losing one more monitor can stop the cluster. First, check the reason with:

A full disk is a common reason. Ceph warns with MON_DISK_LOW when free space on the monitor disk is below 30%. You can free some space, then start the service:

If the monitor data is broken and the other monitors still have quorum, you can remove and create it again with Proxmox tools:

Do this only when the other monitors are working and have quorum. Never remove a monitor if it would leave less than half of the monitors online.

PGs are groups of objects that Ceph places on OSDs. A healthy PG is active+clean. You can list problem PGs with:

Every PG has a state that tells you if your data is safe and available. The state is shown in ceph -s and ceph pg stat. Use the table below to see what each state means, if your data is at risk, and what to do.

Degraded and misplaced are different. Degraded means fewer copies than wanted. Misplaced means the data is safe but not yet in its final place.

Degraded and undersized PGs mean Ceph has fewer copies of your data than the pool needs. Your data is still there, but you have less safety. This usually happens when an OSD is down, or the cluster does not have enough hosts for the pool. The steps below show how to find the cause and fix it.

1. Find which OSDs are down with ceph osd tree.
2. Bring them back (Step 4).
3. Watch recovery with ceph -s.

If PGs stay undersized with all OSDs up, check the pool and CRUSH rule:

For example, a size 3 pool with a rule that spreads copies over hosts needs at least 3 hosts. With only 2 hosts, PGs stay undersized.

Inactive and down PGs are more serious, because they can block reads and writes. VMs that use these PGs may freeze. Usually the cause is that the OSDs for these PGs are offline or cannot be reached. Query one PG to see why it is stuck:

Look at recovery_state. It tells you which OSDs it is waiting for blocked_by, down_osds_we_would_probe. In most cases, the fix is to start those OSDs or to fix the network between them.

For a pool at min_size 2 with size 3, a PG stops I/O when only one copy is online. This protects your data. Do not lower min_size to force it online.

If recovery is slow, you must check the network and the disks first. You can control recovery speed with the mClock profile:

Use this only when your VMs can run with slower disk speed. When recovery is done, go back to balanced or high_client_ops.

For planned work, such as a node reboot, you can stop Ceph from rebalancing:

Also, you can pause data movement for a short time:

Remove them with ceph osd unset <flag>. If you forget them, the cluster will stay unhealthy. Check ceph osd dump | grep flags to see the active flags.

After you fix the problem, check that the cluster is really healthy. Run the commands below to confirm that all OSDs, monitors, and PGs are fine, and that no old flags are left.

If a warning is harmless and you need to hide it for a while, you can mute it for a set time instead of ignoring it:

Most Ceph warnings have a simple cause. Look before you act, avoid risky commands, and give the cluster time to recover. Follow this runbook, and Proxmox Ceph troubleshooting becomes much easier.

We hope you enjoy this guide. For more information, you can check the official Ceph docs.