How to Use S3-Compatible Storage with Proxmox Backup Server 4.2

Proxmox Backup Server (PBS) 4.2 can store backups straight in an S3 bucket. In this guide, you will learn how to use S3 datastore with Proxmox backup server, from the first bucket to the last restore test. We use only the built-in PBS tools, with no mounts and no extra software.

An S3 datastore keeps backup data and metadata in your bucket. Also, it needs a local cache on disk. The cache cuts the number of S3 requests and makes listing and restores faster. Keep these facts in mind before you start:

• All objects in the bucket start with the datastore name, so you can create more than one datastore in the same bucket.
• PBS does not create buckets and does not manage access keys. First, you must set both up at your S3 provider.
• PBS talks to S3 over HTTPS only. Plain HTTP is not supported.
• One S3 datastore can be used by only one PBS server at once.
• The cache must be on local and persistent storage. RAM-only cache is not possible.
• Your provider may charge for storage, API requests, and traffic out. Watch these numbers.

Before you begin, make sure you have the follwoing things ready:

• A working PBS 4.2 server. If you do not have one yet, you can follow our Proxmox Backup Server setup guide.
• An S3-compatible provider, such as AWS S3, Backblaze B2, Wasabi, Cloudflare R2, or your own Ceph or MinIO-style server.
• One empty disk, partition, or ZFS dataset for the cache.
• Root access to the PBS server.

Check your PBS version; it must show version 4.2 or newer:

If it is older, you can update it with the command below:

At this point, you must follow the steps below in order. You will create the bucket and keys at your S3 provider, prepare the local cache disk, and add the S3 endpoint.

Then, you will create the datastore, set up a backup user, and watch the request and traffic counters. Last, you will run a backup and test a restore.

Remember that you must do this in your S3 provider panel, not in PBS.

1. Create a new bucket, for example pbs-s3-bucket.
2. Create an access key and a secret key for it.
3. Give the key permission to get, put, list, and delete objects in that bucket. PBS needs all four.
4. Write down the endpoint address and the region name.

Note: Use one bucket only for PBS. Do not turn on features that delete or change objects on their own, such as lifecycle rules, because they can damage the datastore.

The cache must be on its own disk, partition, or ZFS dataset. You cannot use an existing normal datastore as the cache. The PBS suggests 64 GiB to 128 GiB.

First, list your disks with the command below:

Then, create a filesystem on the free disk. This example uses sdb and mounts it under /mnt/datastore/s3cache:

If you use ZFS, you must create a dataset with a size limit instead:

Replace rpool with your own pool name. Remember to use only one of these two options.

The endpoint holds your credentials and your provider address.

In the web interface, you must go to Configuration > Remotes > S3 Endpoints and click Add. Fill in the ID, access key, secret key, endpoint, and region.

Also, you can do it from the command line. For example, for AWS S3, you can use:

The {{bucket}} and {{region}} parts are templates. PBS fills them in, so with a bucket named my-pbs-bucket in eu-central-1, the address becomes my-pbs-bucket.s3.eu-central-1.amazonaws.com.

You can check the result with the command below:

Some providers need different settings:

• Path-style addressing: PBS prefers vhost-style addresses, where the bucket name is part of the host name. If your provider needs the bucket in the path, add --path-style true.
• Cloudflare R2: use path-style addressing and set the region to auto. Otherwise, request signing may fail.
• Self-hosted S3 with a self-signed certificate: you must add the certificate fingerprint to the endpoint, or PBS will refuse to connect. You can also set a custom port, which is common for Ceph Rados Gateway.

For example, a Cloudflare R2 endpoint looks like this:

Also, PBS has a provider setting for S3 servers that cannot delete many objects in one request. It makes PBS delete objects one by one. Turn it on only if you see delete errors. You can also set upload and download speed limits per endpoint, which helps if backups fill your internet line.

You can use this same method to use S3 datastore with Proxmox backup server, with any S3 provider.

Now you must link the endpoint, the bucket, and the cache path. Note that the path in this command is the local cache path, not the place where data is stored:

Also, you can do this from the web interface. Go to Datastore > Add Datastore, then pick S3 as the backend, choose your endpoint, enter the bucket name, and enter the cache path.

Once you are done, check the result with the command below:

The settings are saved in /etc/proxmox-backup/s3.cfg and /etc/proxmox-backup/datastore.cfg. You can set a prune and garbage collection schedule so old data is removed:

Garbage collection on S3 can send many requests, so run it once a week, not every day.

You should not use the root account for backups. You must create a user and an API token with only backup rights on this datastore:

The second command prints the token secret once. Save it now. If you want to run restore tests with this user too, you must give it the DatastoreReader role in the same way, or use a separate token for that.

PBS 4.2 counts the requests and the traffic of every S3 datastore. You can see the request counters on the datastore Summary page in the web interface. Also, the dashboard labels the usage column as Local Cache Usage for S3 datastores, because it shows the cache and not the bucket.

You can set limits for each request type and for traffic. If a counter goes over its limit, PBS sends a notification. The request types are s3-get, s3-put, s3-head, s3-post, and s3-delete. The traffic types are s3-download and s3-upload.

The --counter-reset-schedule option sets zero on the counters at the time you choose. With monthly, you can compare each month with your provider bill. Pick numbers that match your plan, then check the saved values:

PBS needs a notification target, such as email, Gotify, or a webhook, to send alerts. You can add one under Configuration > Notifications. After your first backup, check how the counters grow and adjust the limits.

When you use S3 datastore with Proxmox backup server, check these counters weekly at first. A rise in GET requests means more cache misses, which can mean your cache is too small.

From Proxmox VE, you must add the datastore as storage on your PVE node. First, get the PBS fingerprint on the PBS server:

Then, on the PVE node, run this command to add your S3 datastore as PBS storage:

Now run a test backup of VM 100 to the new S3 datastore. Change 100 to your own VM ID:

To back up a normal Linux server, you can use the PBS backup client. First, set the repository and token, then run the backup. This example backs up the /etc folder:

Run the same backup again. The second run should be much faster, because PBS only sends new data. Then, open the Summary page to see how many requests the backup used.

A backup is only useful if you can restore it. Test a restore before you trust the setup, and repeat the test every few months.

First, list the snapshots with the following command:

Then, restore one folder from the host backup and measure the time:

To restore a VM, first find the backup ID, then restore it to a new VM ID. Do not restore over a running VM during a test:

Write down three things: the restore time, the number of GET requests, and the amount of data downloaded.

First, test a new snapshot, then an old one. The old snapshot is more likely to be missing from the cache, so it shows the slowest case. Run the same restore again to see the speed with a warm cache.

To check the data, start the restored VM and make sure it boots and works. For file backups, compare the restored files with the source.

Every S3 request has a delay. A local disk answers in microseconds, but an S3 request over the internet can take tens or hundreds of milliseconds.

PBS splits data into chunks of about 4 MiB, so 1 TiB of data can make up to 262,144 chunks. Deduplication and compression lower this number, but the first backup still sends many objects. That is why the first backup takes the longest. Later backups only send new chunks, so they are much faster.

The cache helps in two ways. It keeps metadata on the local disk, so listing backups needs no S3 requests. Also, it keeps recent chunks, so restoring fresh data does not need a new download. When the cache is full, PBS removes the chunks used least recently first.

Use these tips to pick the cache size and location:

• Start with 64 GiB to 128 GiB. This is a good start for most small and medium setups.
• If your restore tests show too many GET requests for fresh snapshots, make the cache bigger.
• If you back up a lot of data each day, make sure the cache can hold at least a few days of new chunks.
• Use fast local NVMe or SSD storage. A slow cache disk makes every backup and restore slower.
• Use a separate disk, partition, or a ZFS dataset with a quota, so the cache can never fill the system disk.
• Put the PBS server near your Proxmox VE nodes on a fast local network, and pick an S3 region that is close to the PBS server. A short path to S3 means lower latency.

Restore speed from S3 depends on your internet speed and on how many chunks are missing from the cache. Restoring old data is slower than restoring from a local datastore. Plan your recovery time with the numbers from your own test in Step 8.

If your PBS server is lost, you can build a new one and reuse the same bucket. Install PBS on the new server and add the same S3 endpoint. Then, create the datastore again with the same name, a new cache path, and the --reuse-datastore true --overwrite-in-use true options:

If you change objects in the bucket manually, or if the bucket ran out of space during a write, refresh the cache from S3:

PBS 4.2 makes off-site backups simple. At this point, you know how to use S3 datastore with Proxmox backup server. You just create the bucket, prepare a local cache, add the endpoint, create the datastore, watch the request counters, run backups, and test restores.

The native backend is better than mounting a bucket, because PBS controls every request. For the best results, use a fast local cache, set traffic limits early, and test your restores often.

You can run PBS on a dedicated server with fast local NVMe for the cache, and send your off-site copy to an S3-compatible target. With this, you can get quick local backups and restores, plus a safe copy in another place.

For more detailed information, you can check the official Proxmox Backup Server for Backup Storage documentation.