Training large AI models requires a significant amount of time and power. When you train on a single GPU, you might wait days or weeks to see results. Horovod solves this by letting you spread your training job across many GPUs and servers. This guide will show you the Horovod Distributed Training Setup environment.
You will learn how to install Horovod with MPI and NCCL, configure your PyTorch or TensorFlow training code, and run training across multiple GPUs and servers.
By the end of this guide, your team will have a scalable AI training platform that is faster, cheaper, and easier to manage than cloud alternatives.
Many teams struggle to find affordable, reliable GPU hosting for distributed training. PerLod Hosting solves this by providing dedicated GPU servers with transparent pricing and no oversubscription.
What is Horovod and How It Works?
Horovod is an open-source distributed training framework created by Uber for TensorFlow, Keras, PyTorch, and Apache MXNet.
It takes your single-GPU training script and scales it to many GPUs with minimal code changes.
Instead of using a central parameter server to coordinate training, Horovod uses the ring-allreduce algorithm, which means each GPU communicates only with its two neighbors in a ring, passing gradients around the circle.
To understand it better, imagine 4 GPUs in a circle. Each GPU:
Sends its gradients to the GPU on the right.
Receives gradients from the GPU on the left.
Adds them together and passes them on.
After going around the ring twice, every GPU has the average of all gradients. This is much faster than having all GPUs talk to a central server.
Key features of Horovod include:
Ring-AllReduce: Efficient gradient averaging without parameter servers.
Multi-Framework: Works with PyTorch, TensorFlow, Keras, and MXNet.
Scalable: From 2 GPUs to hundreds of GPUs.
Fault Tolerant: Recovers from GPU or node failures with Elastic mode.
Easy Integration: Only 5-6 lines of code changes needed.
Horovod achieves 90% scaling efficiency across 128 servers with 4 GPUs each. This means if you double your GPUs, your training speed almost doubles too, with no wasted resources.
Prerequisites for Horovod Distributed Training Setup
Before you begin Horovod Distributed Training setup, you need these:
Operating System: Ubuntu 22.04 or Ubuntu 24.04 LTS.
Multiple GPUs: 2+ NVIDIA GPUs (can be on the same server or multiple servers).
NVIDIA CUDA: CUDA 11.0 or newer (12.0+ recommended).
MPI (Open MPI): Version 4.0+ (avoid 3.1.3, which has bugs).
PerLod Hosting offers dedicated GPU servers with multiple GPUs, full root access, and pre-installed Ubuntu OS. This makes them perfect for Horovod distributed training setups.
Step 1. Install Dependencies For Horovod Setup
The first step is to update your system and install the dependencies and required packages for Horovod setup.
Run the system update and upgrade with the command below:
Bash
sudoaptupdate&&sudoaptupgrade-y
Install essential build tools and Open MPI with the following command:
The driver should be version 450+ for best compatibility.
Check CUDA version with:
Bash
nvcc--version
CUDA 11.0+ is required. CUDA 12.0+ is recommended for the latest GPUs.
Step 3. Install NCCL for Horovod Distributed Training
NCCL (NVIDIA Collective Communications Library) is what makes GPU-to-GPU communication fast. NCCL 2 introduced the ability to run ring-allreduce across multiple machines.
You can install it in two ways: using pip and the NVIDIA Repository.
Option A: Install via pip
You can easily use the command below to install the NCCL 2:
Bash
pipinstallnvidia-nccl-cu12
This installs NCCL 2.x for CUDA 12. For CUDA 11, you can use nvidia-nccl-cu11.
Option B: Install via NVIDIA Repository (Recommended for System-Wide)
Add the NVIDIA repository with the commands below:
Make sure [X] NCCL and [X] PyTorch or TensorFlow are enabled.
Step 6: Set Up SSH Between Servers for Horovod (Multi-Server Setup Only)
If you are using multiple servers, they need to communicate without passwords. You can skip this step if you are using a single server with multiple GPUs.
On your main server, create an SSH key with the following command:
All GPUs should show high utilization; 70-95% is normal.
Advanced Communication and Performance Tuning for Horovod
As you add more GPUs, they spend a lot of time communicating with each other to sync up data. If this communication is inefficient, it can slow down your entire training process.
In this step, you will learn some performance optimization for your Horovod setup, including batch small updates together, reducing the traffic between servers, and compressing data to save bandwidth. We will also show you how to use a special profiling tool to see exactly where the delays are happening so you can remove them.
Enable Tensor Fusion: Horovod can batch small tensors together before communication. This will set a 64MB fusion threshold:
Horovod makes distributed training accessible to everyone. You can scale from one GPU to hundreds with just a few code line changes. The ring-allreduce algorithm in Horovod ensures efficient communication without bottlenecks, and NCCL makes GPU-to-GPU transfers super fast.
With Horovod on PerLod's bare-metal GPU servers, your team gains full control over infrastructure, achieves better scaling efficiency, and pays predictable monthly costs instead of surprise cloud bills.
We hope you enjoy this guide. Subscribe to our X and Facebook channels to get the latest updates on GPU and AI hosting.
We use cookies to run PerLod and to understand how the site is used. You can accept everything, or open Manage to turn individual categories off. Cookie policy.