Kubernetes Disaster Recovery Drill: Restore Test With Velero/Longhorn

Updated on Sep 14, 2026
Mila H
10 MINS READ
Table of Contents
Kubernetes Disaster Recovery

Losing your main server is scary, but a real recovery test is the only way to know your backups actually work. This guide provides a full Kubernetes disaster recovery drill with Velero and Longhorn. 

This guide is not about setting up backups; it is the actual test. We will break the old cluster, set up new servers, and bring everything back with proof it worked. If you plan to set up backups, you can check this guide on backing up K3s with Velero.

What the Cluster Looked Like Before the Crash

Before we break anything, here is the setup we are protecting. A small and real-world K3s cluster with:

  • One main K3s server node, which stores all cluster data.
  • Two or three worker nodes running the app.
  • Longhorn installed to copy storage across the workers, following our Longhorn on K3s with NVMe setup guide.
  • Velero installed with an S3-style storage bucket to hold backups.
  • A small test app, a web app with a database, using storage from Longhorn.

This is why teams test Kubernetes disaster recovery with Velero and Longhorn together. The main server holds the cluster's memory, Velero holds the app files and settings, and Longhorn holds the real data on disk. You need all three to come back safely. None of them can protect you alone.

Prerequisites and Tools You Need

For the Kubernetes disaster recovery with Velero and Longhorn drill, you will need:

  • Root or sudo access on all servers.
  • At least one new master server and two new worker servers, on the same network as, or reachable from, the old cluster's DNS/load balancer.
  • A remote, S3-compatible bucket that survived the disaster. This is where your etcd snapshot, Velero backups, and Longhorn backups already live.
  • kubectl and helm installed on your workstation.
  • The Velero CLI matching the server version.
  • A basic understanding of Linux systemd services is helpful, since K3s runs as a systemd unit.

If you are rebuilding for real, not just practicing, use fresh, clean servers instead of reusing the old ones. Using dedicated servers for hosting gives you steady CPU, RAM, and NVMe storage for both the main server and the Longhorn copies.

Step 1: Take the Backups You Will Restore From

If you already run scheduled backups, skip to Step 2. If not, take fresh ones now so the drill has something real to restore.

Save an etcd snapshot: On the master node, run:

Bash
sudo k3s etcd-snapshot save --name pre-disaster-snapshot

Push it to S3 so it survives even if the whole server disappears:

Bash
sudo k3s etcd-snapshot save \  --name pre-disaster-snapshot \  --s3 \  --s3-bucket=k3s-dr-backups \  --s3-endpoint=s3.amazonaws.com \  --s3-region=us-east-1 \  --s3-access-key="$AWS_ACCESS_KEY" \  --s3-secret-key="$AWS_SECRET_KEY"

List snapshots to confirm it:

Bash
sudo k3s etcd-snapshot ls

Take a Velero backup with the commands below:

Bash
velero backup create pre-disaster-backup --waitvelero backup describe pre-disaster-backup

Take a Longhorn backup: Open the Longhorn UI or use kubectl and run a backup for each volume. Or create a full Longhorn System Backup so you can restore the whole storage state in one shot:

Bash
kubectl apply -f - <<EOFapiVersion: longhorn.io/v1beta2kind: SystemBackupmetadata:  name: pre-disaster-system-backupspec:  volumeBackupPolicy: IfNotPresentEOF

Check that it is finished:

Bash
kubectl get systembackup pre-disaster-system-backup -n longhorn-system

At this point, you have everything you need to run a real Kubernetes disaster recovery test with Velero and Longhorn. Write down the exact time of each backup. You will compare this to when your restore finishes to measure your RPO and RTO.

Step 2: Break the Main Server on Purpose

To make the test real, actually turn off the main server:

Bash
sudo systemctl stop k3ssudo /usr/local/bin/k3s-uninstall.sh

If you want to be extra sure, boot into a rescue image and wipe the disk. Or just turn the server off and leave it off for the rest of the test. Do not touch the worker nodes yet. We will rebuild the whole cluster from scratch anyway.

Step 3: Prepare New Servers for the Rebuild

At this point, you must set up new servers to replace the old ones. Give each Longhorn copy its own NVMe or SSD disk, since Longhorn works best with fast local disks, not shared network storage.

If you need servers fast, PerLod's Dedicated Server is the quickest way to get real CPU, RAM, and NVMe you fully control, with no wait for cloud setup.

For this drill, we use three new machines:

  • new-master-1: Will become the master node.
  • new-worker-1 and new-worker-2: Will run the application pods and Longhorn replicas.

Make sure all three can reach each other over the network, and update your DNS record or load balancer to point at the new master node IP once it is ready.

Step 4: Restore etcd on the New Master Node

This is the main part of the Kubernetes disaster recovery drill with Velero and Longhorn: get Kubernetes running again before touching your apps.

First, you must copy the etcd snapshot to new-master-1, or pull it directly from S3 during the restore.

Install K3s in a stopped state so it does not try to form a cluster before the restore:

Bash
curl -sfL https://get.k3s.io | INSTALL_K3S_CHANNEL=stable INSTALL_K3S_SKIP_START=true sh -

This installs the current stable K3s release line with the matching embedded etcd version.

Restore from the local snapshot file:

Bash
sudo k3s server \  --cluster-init \  --cluster-reset \  --cluster-reset-restore-path=/var/lib/rancher/k3s/server/db/snapshots/pre-disaster-snapshot \  --token="$OLD_CLUSTER_TOKEN"

Or restore straight from S3 if you did not copy the file locally:

Bash
sudo k3s server \  --cluster-init \  --cluster-reset \  --cluster-reset-restore-path=pre-disaster-snapshot \  --etcd-s3 \  --etcd-s3-bucket=k3s-dr-backups \  --etcd-s3-endpoint=s3.amazonaws.com \  --etcd-s3-region=us-east-1 \  --etcd-s3-access-key="$AWS_ACCESS_KEY" \  --etcd-s3-secret-key="$AWS_SECRET_KEY" \  --token="$OLD_CLUSTER_TOKEN"

Once the reset process finishes, K3s exits by itself. Start it normally as a service:

Bash
sudo systemctl start k3s

Confirm the cluster is alive and remembers the old state:

Bash
sudo k3s kubectl get nodessudo k3s kubectl get namespaces

You should see the old namespaces already listed, because etcd holds the full Kubernetes object database, even though the actual Pods are not running yet. 

Now join the two new workers using the same server URL and token:

Bash
curl -sfL https://get.k3s.io | INSTALL_K3S_CHANNEL=stable K3S_URL=https://new-master-1:6443 K3S_TOKEN="$OLD_CLUSTER_TOKEN" sh -

Run this on both new-worker-1 and new-worker-2.

Step 5: Restore Namespaces and Resources with Velero

The etcd restore already brings back most of the cluster's data. But in most real disasters, you start etcd fresh, a clean --cluster-init with no snapshot, and let Velero bring back all your app files and settings. Either way, this step checks that Velero can put everything back correctly.

Install Velero on the new cluster, pointing at the same backup storage location:

Bash
velero install \  --provider aws \  --plugins velero/velero-plugin-for-aws:v1.11.0 \  --bucket k3s-dr-backups \  --backup-location-config region=us-east-1,s3Url=https://s3.amazonaws.com \  --secret-file ./credentials-velero \  --use-node-agent

Check the backup storage location is reachable and set it to read-only so nothing overwrites your backup while restoring:

Bash
kubectl get backupstoragelocation -n velerokubectl patch backupstoragelocation default \  --namespace velero \  --type merge \  --patch '{"spec":{"accessMode":"ReadOnly"}}'

List the available backups and restore the last one:

Bash
velero backup getvelero restore create --from-backup pre-disaster-backup --wait

Watch the restore progress and confirm it finished without errors:

Bash
velero restore describe pre-disaster-backup-<timestamp>kubectl get pods -A

Switch the backup storage location back to read-write once the restore is confirmed:

Bash
kubectl patch backupstoragelocation default \  --namespace velero \  --type merge \  --patch '{"spec":{"accessMode":"ReadWrite"}}'

This is where a Kubernetes disaster recovery drill with Velero and Longhorn really shows up. You find out fast if your labels, settings, passwords, and app files come back the way you expect.

Step 6: Restore Longhorn Volumes on the New Cluster

Pods will come back from Velero, but they will not have data yet because Longhorn volumes need their own restore step.

Install Longhorn on the new cluster with Helm, matching the setup guide:

Bash
helm repo add longhorn https://charts.longhorn.iohelm repo updatehelm install longhorn longhorn/longhorn \  --namespace longhorn-system \  --create-namespace \  --version 1.10.2

Point Longhorn at the same backup target used before:

Bash
kubectl -n longhorn-system patch setting.longhorn.io backup-target \  --type merge \  --patch '{"value":"s3://longhorn-dr-backups@us-east-1/"}'kubectl -n longhorn-system patch setting.longhorn.io backup-target-credential-secret \  --type merge \  --patch '{"value":"longhorn-backup-secret"}'

If you took a full System Backup earlier, restore everything at once with a SystemRestore resource:

Bash
kubectl apply -f - <<EOFapiVersion: longhorn.io/v1beta2kind: SystemRestoremetadata:  name: post-disaster-restorespec:  systemBackup: pre-disaster-system-backupEOF

If you backed up individual volumes instead, restore each one by creating a new volume from its backup:

Bash
kubectl apply -f - <<EOFapiVersion: longhorn.io/v1beta1kind: Volumemetadata:  name: app-data-restored  namespace: longhorn-systemspec:  size: "20Gi"  numberOfReplicas: 3  fromBackup: "s3://longhorn-dr-backups@us-east-1/?backup=backup-xxxx&volume=app-data"EOF

Then re-point the application's PersistentVolumeClaim to the restored volume, or restore the PVC/PV objects through Velero and let Longhorn's CSI driver bind them automatically.

Check volume health once restored:

Bash
kubectl get volumes.longhorn.io -n longhorn-systemkubectl get pvc -A

Restart any pods that were stuck waiting for storage:

Bash
kubectl rollout restart deployment -n <your-app-namespace>

Step 7: Reissue Certificates and Fix Networking

New servers usually mean new security certificates. K3s makes its own internal certificates again on its own when you run a fresh --cluster-init, so you usually don't need to fix those. Here is what you do need to check:

  • If you use cert-manager for your app's HTTPS, its saved certificates come back through the Velero restore. But you may need new ones if the DNS check now points to a different IP address.
  • Update your DNS records or load balancer to point to the new server IPs.
  • If you use your own private certificate authority, copy its key files onto the new servers before starting anything that needs them.
  • Run kubectl get certificate -A and kubectl describe certificate <name> to check that cert-manager made new certificates for anything that expired or pointed to an old IP.

A good Kubernetes disaster recovery plan with Velero and Longhorn should always note which certificates fix themselves and which ones need manual work after a DNS change.

Step 8: Validate RPO and RTO

Getting pods to Running is not the finish line. You must check that your data is really correct, and time how long the whole recovery took.

Check application data matches what you expect from the last backup:

Bash
kubectl exec -it <app-pod> -- <command to check data, e.g. a row count or file checksum>

Compare this with the note you took before the drill or the backup time to calculate your real RPO.

To calculate your RTO, compare the time you shut down the old cluster in Step 2 with the time your app became fully healthy again:

Bash
kubectl get pods -A -o widekubectl top nodes

Run your normal smoke tests: Load the app in a browser, hit its health endpoint, check that background jobs resume:

Bash
curl -I https://your-app.example.com/health

Write down both numbers. If your RTO is too slow for your business, try keeping backup servers ready, or turn Steps 3 to 7 into scripts to cut down manual work. This full test is the only sure way to know your Kubernetes disaster recovery plan with Velero and Longhorn really meets your goals.

Conclusion

A backup you have never restored is just a guess. By breaking your main server on purpose and rebuilding it on new servers, you get real proof that your Kubernetes disaster recovery plan with Velero and Longhorn works. Also, you get real RPO and RTO numbers you can share with your team.

We hope you enjoy this guide. For more information on the restore workflow and the read-only backup storage location pattern, you can check the Velero disaster recovery documentation.

Prev PostThe Complete Longhorn Troubleshooting Guide for Kubernetes Storage
Category:Server Setup & Administration
Tag:

We use cookies to run PerLod and to understand how the site is used. You can accept everything, or open Manage to turn individual categories off. Cookie policy.