Deploy Edge AI on dedicated servers means running AI models directly on servers which is located close to the users or data sources, which brings faster responses, better privacy, and less internet dependence. Benefits of AI deployments in dedicated servers include low latency, data privacy and security, also it works even with a limited internet connection.
This tutorial assumes access to a dedicated server with root privileges. If you need ready-to-use infrastructure, PerLod GPU Dedicated Servers include options with NVIDIA GPUs, NVMe storage, and high-bandwidth networking, which is ideal for local AI inference and model serving.
Deploy Edge AI on Dedicated Servers Architecture
To deploy AI Edge, we use a dedicated server that can be CPU-only or include NVIDIA, AMD, or Intel accelerators. Be sure your dedicated server has a fast NVMe SSD for storage. Also, we will run the AI services inside Docker containers.
Inference Servers Used: Depends on Hardware:
- NVIDIA GPUs: Use NVIDIA Triton Inference Server, which works with TensorRT, ONNX, PyTorch, and TensorFlow.
- Intel CPUs/iGPUs/NPUs: Use OpenVINO Model Server (OVMS) or ONNX Runtime with the OpenVINO plugin.
- AMD GPUs: Use ROCm with ONNX Runtime, vLLM, or other ROCm-supported backends.
This guide uses an Ubuntu LTS OS with a sudo user.
Initial Server Setup for Edge AI Deployment
You must run the system update and install the required tools to prepare your server for Edge AI deployments:
sudo apt update && sudo apt upgrade -ysudo apt install build-essential curl gnupg2 ca-certificates lsb-release jq unzip ufw -y
For more security, it is recommended to create a dedicated user to manage the node:
sudo useradd -m -s /bin/bash edge && sudo usermod -aG sudo,docker edge
Then, implement some basic system hardening with the commands below:
echo -e "net.ipv4.ip_forward=1\nnet.ipv4.conf.all.rp_filter=1\nnet.ipv4.tcp_syncookies=1" | sudo tee /etc/sysctl.d/99-edge.conf sudo sysctl --system
Configure firewall rules to deny all incoming traffic by default, and only allow necessary ports:
sudo ufw default deny incomingsudo ufw default allow outgoingsudo ufw allow 22/tcp
Enable the firewall with the command below:
Also, you must install Docker on your server. To do this, you can check the official Docker page and install Docker on Ubuntu via the APT repository.
Once you are done with the Docker installation, verify it with the command below:
docker run --rm hello-world
Configuring Hardware Acceleration
To increase your server performance, you must configure hardware acceleration appropriate to your setup. You can choose one of the following paths that match your hardware, including NVIDIA GPUs, Intel processors, or AMD graphics cards.
Path 1. NVIDIA GPUs Acceleration
To use NVIDIA hardware acceleration, you must install the necessary drivers and container toolkit, which allows Docker containers to access your GPU for optimal performance.
To install the proper drivers and container toolkit, run the commands below:
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | \ sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg curl -fsSL https://nvidia.github.io/libnvidia-container/$(. /etc/os-release; \ echo $ID$VERSION_ID)/libnvidia-container.list | \ sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#' | \ sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list sudo apt updatesudo apt install nvidia-container-toolkit -ysudo nvidia-ctk runtime configure --runtime=dockersudo systemctl restart docker
To verify that Docker containers can successfully detect and use your NVIDIA GPU, you can run:
docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi
Path 2: Intel Acceleration OpenVINO
For Intel-based acceleration, OpenVINO provides optimized performance across CPUs, integrated graphics, and AI accelerators without complex driver setups. You can check the next deployment steps.
Path 3: AMD GPU Acceleration ROCm
AMD users can utilize the ROCm platform for GPU acceleration, which enables high-performance computing on supported AMD graphics cards.
Prepare and Optimize AI Model Inference
Before deployment, you need to export and optimize your trained model for production inference, which ensures maximum performance on your specific hardware.
1. Export to ONNX: Convert your model from its original training framework, like PyTorch or TensorFlow, into this universal format. This allows the model to run on many different types of hardware and software, which makes it ready for production deployment.
2. Hardware-Specific Optimization: Different accelerators require different optimized model formats. Choose the conversion path below that matches your target hardware to achieve the best possible speed and efficiency:
- NVIDIA GPUs: Convert your models to TensorRT (FP16 or INT8) for faster performance, or let Triton automatically run ONNX models using the TensorRT backend.
- Intel CPUs/iGPUs/NPUs: Convert models to OpenVINO IR format, and optionally quantize to INT8 for better speed and efficiency.
- AMD GPUs: Keep models in ONNX format and use ROCm-enabled runtimes for inference.
3. Creating the Model Repository Structure: Inference servers require a specific directory structure to load your models used by both Triton and OVMS:
/opt/models/my_model/1/model.onnx config.pbtxt
Using versioned model directories enables seamless rollbacks and zero-downtime updates, which is essential for maintaining reliable production services.
Deploying an AI Model for Inference
In this step, you must deploy your optimized model to a production-ready inference server. Choose the path below that matches your hardware to start serving predictions.
NVIDIA Triton GPU Serving
For NVIDIA GPU users, Triton Inference Server provides high-performance model serving with support for multiple frameworks and concurrent execution.
Use the commands below to create a model directory and run Triton:
sudo mkdir -p /opt/models/my_model/1sudo chown -R $USER:$USER /opt/models docker run --rm --gpus all \ -p 8000:8000 -p 8001:8001 -p 8002:8002 \ -v /opt/models:/models \ nvcr.io/nvidia/tritonserver:24.10-py3 \ tritonserver --model-repository=/models
Ports used:
- 8000: HTTP
- 8001: gRPC
- 8002: Metrics
To check server is up and running, you can run:
curl -s http://localhost:8000/v2/health/ready
Now you can customize how your model runs by defining its input/output specifications, batch size, and GPU instance allocation in a configuration file.
Create the following example file for ONNX:
Add the following configuration to the file:
name: "my_model"platform: "onnxruntime_onnx"max_batch_size: 16input [ { name: "input", data_type: TYPE_FP32, dims: [3, 224, 224] }]output [ { name: "logits", data_type: TYPE_FP32, dims: [1000] }]instance_group [{ kind: KIND_GPU, count: 1 }]
Use Triton's built-in performance analyzer to test your model's throughput and latency under various load conditions:
docker run --rm --net=host nvcr.io/nvidia/tritonserver:24.10-py3 \ perf_analyzer -m my_model -u localhost:8000 -i HTTP --concurrency-range 1:64
OpenVINO Intel Acceleration Serving
For Intel hardware, OVMS delivers optimized inference across CPUs, iGPUs, and NPUs with minimal configuration. You can start the OpenVINO Model Server container to serve your optimized IR or ONNX models through gRPC and REST endpoints:
docker run --rm \ -p 9000:9000 -p 8001:8001 \ -v /opt/models:/models \ openvino/model_server:latest \ --model_path /models/my_model \ --port 9000 --rest_port 8001 --target_device AUTO
12from ovmsclient import make_grpc_client3cli = make_grpc_client("localhost:9000")4outputs = cli.predict({"input": your_numpy_batch}, "my_model")
ONNX Runtime with OpenVINO
For lightweight Python deployments, use ONNX Runtime with the OpenVINO execution provider for flexible CPU/iGPU/NPU inference.
Install ONNX runtime in a Python environment shell:
python -m venv venv && source venv/bin/activatepip install onnxruntime-openvino onnx
Load and run the AI model with:
1import onnxruntime as ort2 3sess = ort.InferenceSession(4 "model.onnx",5 providers=[("OpenVINOExecutionProvider", {"device_type":"AUTO"})]6) 78res = sess.run(None, {"input": your_numpy_batch})
AMD ROCm Runtime
AMD users can use ROCm-supported runtimes like ONNX Runtime or PyTorch for GPU-accelerated inference on compatible hardware.
You can follow AMD’s ROCm installation guide for the correct version pairing with your kernel and OS.
Secure AI Inference Service with Nginx Reverse Proxy
To safely display your AI service to the network, you can set up Nginx as a secure gateway that provides TLS encryption and controlled access to your inference endpoints.
Install Nginx and generate TLS certificates with the commands below:
sudo apt install nginx -ysudo openssl req -x509 -nodes -days 365 -newkey rsa:2048 \ -keyout /etc/nginx/self.key -out /etc/nginx/self.crt -subj "/CN=edge.local"
Create the Nginx configuration file with the following command:
sudo nano /etc/nginx/sites-available/edge-ai.conf
Add the following config to the file:
server { listen 443 ssl; ssl_certificate /etc/nginx/self.crt; ssl_certificate_key /etc/nginx/self.key; location /triton/ { proxy_pass http://127.0.0.1:8000/; proxy_set_header Host $host; } location /ovms/ { proxy_pass http://127.0.0.1:8001/; proxy_set_header Host $host; }}
Enable your configuration, check for syntax errors, and reload Nginx to apply the changes:
sudo ln -s /etc/nginx/sites-available/edge-ai.conf /etc/nginx/sites-enabled/sudo nginx -tsudo systemctl reload nginx
Tip: For production environments, consider adding authentication layers like JWT tokens or API keys at the proxy level for more restricted access to your inference services.
Conclusion
Deploying Edge AI on dedicated servers means using powerful enterprise hardware to run AI models locally, close to where data is created. By picking the right hardware, optimizing models for your platform, and using modern tools like Triton or OpenVINO for deployment, organizations can get ultra-low latency, keep their data private, reduce internet dependence, and build scalable, easy-to-manage systems.
We hope you enjoy this guide. Subscribe to X and Facebook channels to get the latest Edge AI articles.
For readers who would like a more detailed guide on packaging and deploying AI inference services via containers, Containerized AI Serving with Docker on Bare Metal provides step-by-step instructions.
Also, if you are interested in the upstream part of the AI workflow, you can check out the Automated ML Workflow via Dedicated Infrastructure tutorial.
