Deploying a PyTorch model for inference on a Virtual Private Server allows you to make your trained models available for real-time predictions from anywhere. This tutorial provides a complete PyTorch Model Inference Setup on VPS.
In this guide, you will learn to set up a clean Ubuntu environment, build a Docker-based alternative, and optimize performance for real-world use. You’ll also learn how to productionize your deployment using FastAPI with Uvicorn or Gunicorn, manage it as a systemd service, and configure Nginx as a reverse proxy for secure and efficient request handling.
If you want a reliable and stable VPS environment to set up a PyTorch inference model, you can check PerLod Hosting.
Let's dive into the details.
Complete Guide To PyTorch Model Inference Setup on VPS
This guide will show you the complete process of setting up a production-ready PyTorch model on an Ubuntu VPS. We will package a model using ResNet18 as our example into a FastAPI web service, and then use Nginx as a reverse proxy to handle public traffic securely. We will configure the application to run on 127.0.0.1:8000 and set up Nginx to proxy requests from ports 80 and 443.
The setup is designed to work on a CPU by default, but includes optional steps to leverage an NVIDIA GPU for accelerated inference.
We assume that you have an Ubuntu 22.04 or 24.04 VPS with root or sudo access. Now, follow the steps below to complete the guide.
Prepare VPS for Deploying PyTorch Model Inference
The first step is to prepare your system. Run the system update and upgrade with the following command:
Bash
sudoaptupdate&&sudoaptupgrade-y
Then, install the required packages on your server:
Enabling GPU Acceleration with NVIDIA CUDA for PyTorch
You can skip this section if you are deploying on a CPU-only server. These steps are only necessary to enable GPU-accelerated inference for a performance boost.
First, you need to install the proper drivers for your NVIDIA GPU. You can identify available drivers with the command below:
Bash
ubuntu-driversdevices
This command will list the recommended driver for your GPU. For example, nvidia-driver-535. Install the recommended driver with the command below:
Bash
sudoaptinstallnvidia-driver-535-y
Reboot your system to load the new drivers:
Bash
sudoreboot
After rebooting, log back in and run:
Bash
nvidia-smi
You must see a table showing your GPU details, confirming it is recognized and ready.
Now, you must install a PyTorch version built to leverage your GPU's CUDA cores. First, create and activate a Python virtual environment:
PyTorch Model Inference Project Structure and Setup
At this point, you must create a clean and organized project directory to hold your application code, model, and configuration files. First, create the main project folder and navigate into it with:
CPU only: Tune with --workers 2-4 (based on cores)
Multi-GPU: Use one process per GPU (see production section)
Set up Systemd PyTorch Inference Service for Production
Also, you can create a systemd service for automatic restarts and process management. To do this, you can run:
Bash
sudotee/etc/systemd/system/pytorch-infer.service>/dev/null<<'UNIT'[Unit]Description=PyTorchInferenceAPIAfter=network.target[Service]User=inferuserGroup=inferuserWorkingDirectory=/home/inferuser/apps/pytorch-inferEnvironment="PATH=/home/inferuser/venvs/infer/bin"ExecStart=/home/inferuser/apps/pytorch-infer/run.shRestart=alwaysRestartSec=3StandardOutput=journalStandardError=journal# For GPU: uncomment if you need nvidia libs on PATH# Environment="LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu:/usr/local/cuda/lib64"[Install]WantedBy=multi-user.targetUNIT
Configure Nginx as a Reverse Proxy for PyTorch Model Inference
Nginx acts as a public-facing web server that forwards requests to your FastAPI application, providing better security, performance, and enabling HTTPS. First, install Nginx with the command below:
Security: Nginx handles public traffic; your app runs locally.
Performance: Static file serving and load balancing.
Reliability: Better timeout and request handling.
HTTPS: Easy SSL/TLS setup with Certbot.
Setting Up UFW Firewall Rules
Also, you must configure the UFW to secure your server by allowing only necessary network traffic. Install UFW with:
Bash
sudoaptinstallufw-y
Then, allow the required ports and services and enable the UFW:
Bash
sudoufwallowOpenSSHsudoufwallow80/tcpsudoufwallow443/tcp# if using TLSsudoufwenable
Check your UFW firewall rules with:
Bash
sudoufwstatus
Scaling Strategies for PyTorch Inference Model
This step will show you how to scale your inference service for maximum performance across different hardware configurations, from multi-core CPUs to single and multi-GPU setups.
For CPU Scaling (No GPU):
Increase Uvicorn workers: --workers N (start with cores/2 to cores).
Keep one model per process (automatically handled by Uvicorn workers).
For Single GPU:
Use --workers 1 to avoid GPU memory contention.
If CPU preprocessing becomes a bottleneck, consider 2 workers with lazy model loading in only one worker, or move preprocessing to a threadpool with a single worker.
For Multi-GPU Setup:
Run one service instance per GPU using CUDA_VISIBLE_DEVICES in separate processes, then load balance via Nginx.
Example override for GPU0:
Bash
sudosystemctleditpytorch-infer
Add this:
Bash
[Service]Environment="CUDA_VISIBLE_DEVICES=0"
Repeat for GPU1 with a second unit on a different port.
Optimizing PyTorch Inference Performance Safely and Effectively
To ensure your PyTorch model runs as efficiently as possible in production, you can apply several safe and verifiable optimization techniques. These methods improve inference speed, reduce latency, and make better use of your hardware without changing the model’s predictions.
Performance Tips To Consider:
Use inference mode (already in code): torch.inference_mode().
cudnn.benchmark for static-sized inputs (already set on GPU).
Batching: consider accepting multiple images to increase GPU utilization.
Once your model server is running, it’s important to keep track of its health and performance. Adding a /health endpoint, monitoring logs, and checking Nginx access logs help ensure your system is running smoothly and make debugging easier when issues arise.
Before going live, it’s a good idea to benchmark your setup. Simple tools like hey can simulate user traffic and measure how fast your API responds under load. This helps you identify bottlenecks and confirm that your VPS and model can handle real-world usage.
hey-n500-c10-mPOST-T"multipart/form-data"\-D<(printf'%s'"$(curl -s -F "file=@tests/sample.jpg" -X POST http://127.0.0.1/predict -v 2>&1 | sed -n '/^>/,$p' | sed -n '/^$/,${p;q}' )")\http://127.0.0.1/predict
For realistic tests, you can write a small script to POST images in a loop.
Containerizing PyTorch Inference Server with Docker
Docker provides an easy, consistent way to package and deploy your PyTorch inference server. It simplifies environment setup, ensures reproducibility, and allows you to run your model in isolated containers, whether on CPU or GPU, with just a few commands.
Note: For GPU Docker, it requires NVIDIA Container Toolkit:
Docker
# Install NVIDIA container toolkit (Ubuntu)distribution=$(./etc/os-release;echo$ID$VERSION_ID)curl-fsSLhttps://nvidia.github.io/libnvidia-container/gpgkey|sudogpg--dearmor-o/usr/share/keyrings/nvidia-container-toolkit.gpgcurl-fsSLhttps://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list\|sudotee/etc/apt/sources.list.d/nvidia-container-toolkit.listsudoaptupdate&&sudoapt-yinstallnvidia-container-toolkitsudonvidia-ctkruntimeconfiguresudosystemctlrestartdocker# Build with CUDA wheels inside Docker (change requirements accordingly)dockerrun-d--gpusall-p127.0.0.1:8000:8000pytorch-infer:gpu
Protecting PyTorch API in Production
Security is essential when exposing your model to the internet. Restricting network access, adding API keys, limiting request sizes, and keeping your software updated will protect your VPS and ensure your model is only accessible to authorized users.
Once the base setup works, you can easily replace the example model with your own. Just upload your trained weights, adjust the model definition and preprocessing steps, and make sure the input/output formats match your use case.
You can replace ResNet18Classifier with your model:
Copy your .pt/.pth weights to the project folder.
In model.py, define your architecture and load_state_dict.
PyTorch model inference setup on VPS gives you full control, flexibility, and cost efficiency. With the combination of FastAPI, Uvicorn, and systemd, you can create a reliable, production-grade inference API that runs smoothly on CPU or GPU environments.
If you’re looking for high-performance and affordable infrastructure to host your deep learning models, PerLod VPS Hosting Plans provides optimized VPS instances for AI workloads.
We hope you enjoy this guide. Subscribe to X and Facebook channels to get the latest updates and articles.
No. PyTorch supports both CPU and GPU inference. If your model is lightweight or your traffic volume is moderate, CPU-only VPS instances perform well. For heavy workloads or deep learning models, choose a VPS with NVIDIA GPU support.
This depends on your model size and concurrency. For small models like ResNet18, 2–4 GB RAM and 2 vCPUs are enough.
Yes. Docker provides consistent environments and easier scaling. The tutorial includes both native and Docker-based setups.
We use cookies to run PerLod and to understand how the site is used. You can accept everything, or open Manage to turn individual categories off. Cookie policy.