Containerized AI Serving with Docker on Bare Metal

Containerized AI serving with Docker means packaging an AI model and all its dependencies into a Docker container so it can run consistently on any machine. The container exposes an API so applications can send data to the model and get predictions. Deploying AI Serving with Docker on a Dedicated Server requires a powerful, scalable, and secure environment.

This guide will teach you how to set up a fresh Ubuntu server as a powerful AI inference platform, which is capable of serving both Large Language Models (LLMs) and multi-framework vision models using industry-standard tools like vLLM and NVIDIA Triton.

We will also cover both GPU-accelerated and CPU-only setups.

If you are looking for pre-configured hosting solutions optimized for AI workloads, explore our services at PerLod Hosting.

Building an AI server needs a strong foundation, the right tools, and a clear design. In this guide, we will set up a secure host system that runs Dockerized services that provide OpenAI-compatible endpoints for LLMs and a high-performance inference server for many model types, all accessible through a secure reverse proxy.

Instead of manually managing Python environments and model files, we will use Docker containers to create isolated, reproducible, and portable environments for each service.

The core components include:

• Docker Engine and Docker Compose: The runtime and automation layer that will manage your AI applications.
• NVIDIA Stack for GPU setups: The drivers and container toolkit for GPU setups.
• vLLM: A highly optimized library and server for LLM inference, which provides an OpenAI-compatible API.
• NVIDIA Triton Inference Server: A powerful server that can host models from multiple frameworks like PyTorch, TensorFlow, ONNX, and TensorRT in one place. This is ideal for vision models, embeddings, or any non-LLM task.
• Caddy: A modern reverse proxy that will handle HTTPS termination, routing, and security.

Now that you have understood what we are going to build, proceed to the next step to prepare your system.

You must be sure your Ubuntu server is patched, verified, and ready for the AI inference setup. Run the system update and upgrade with the command below:

Reboot the system to load new settings:

After rebooting, verify the system is running the latest kernel version:

Also, list all block storage devices like SSDs and NVMe drives with their filesystems by running the command below:

In this step, you must install Docker, which is the cornerstone of modern application deployment, like AI serving. It allows you to define and manage multi-container applications with a simple YAML file.

Use the following commands to add Docker's official GPG key and install Docker and Compose plugin on your system:

To verify your Docker and Compose installation, check the version:

Also, you can use the command below to verify that Docker is working correctly:

Security note: Avoid adding the user to the Docker group on multi-tenant servers. While convenient, adding your user to the Docker group grants root-equivalent privileges, as Docker can mount any host directory. Using sudo for every Docker command is safer.

At this point, you must set up the NVIDIA drivers and NVIDIA container toolkit on your system, which lets Docker containers seamlessly use the GPU resources.

Use the following Ubuntu automated tool for detecting your NVIDIA GPU and installing the recommended driver:

Reboot the system to load the new driver kernel module:

After rebooting, confirm he driver is loaded, shows GPU model, utilization, temperature, and memory usage:

Then, install the NVIDIA container toolkit with the commands below, which enables --gpus all in Docker:

Use the command below to run a minimal CUDA container and execute nvidia-smi inside the container:

If the output is the same GPU info as the host command, your containerized applications now have full GPU access.

Now you must secure the server's network access and implement a key OS tuning parameter to prevent common container failures.

If your container needs to use folders from your computer, you need to create a special user inside the container that cannot log in. Otherwise, if you don’t mount any folders, you can just let those folders stay owned by root, which is safer and simpler.

Install UFW and allow the necessary rules and ports with the commands below:

Enable the UFW firewall and check its status with:

Tip: While basic firewall setup provides essential network security, you can consider implementing Multi-Factor Authentication (MFA) and Role-Based Access Control (RBAC) to secure server access. For a complete setup, you can check the MFA and RBAC Server Setup Guide.

Note: If your AI model or tokenizer crashes because it runs out of shared memory, give the container more shared memory. In this guide, we will fix this by setting --shm-size=2g, which means 2 gigabytes of shared memory for each container.

In this step, we want to deploy vLLM as a container, which provides a fully compliant OpenAI API endpoint. This means any application that works with OpenAI's API can be pointed to your own private and high-performance server.

The vLLM is GPU-oriented, for CPU-on;y, consider Ollama or TGI with quantized models. We will discuss it in the next steps.

You must choose a model first, and your choice depends on your GPU's VRAM. For example, the "meta-llama/Llama-3.1-8B-Instruct" model typically requires ~16GB of VRAM for comfortable operation.

The HF_TOKEN environment variable is only needed if you are pulling a gated model from Hugging Face Hub.:

To serve an OpenAI-compatible endpoint on port 8000, you can run the following command:

Notes:

• Swap --model with the name (ID) of the Hugging Face model you want to use.
• In production, don’t use “latest” for your Docker image; use a specific version tag or digest so the image doesn’t change unexpectedly.

Now you can run a quick test with the curl command that sends a Chat Completions request and a simpler Completions request.

For Chat Completions, you can run:

For Completions, you can run:

Performance tips you can consider:

• Start with --gpu-memory-utilization 0.85, which safely uses about 85% of your GPU memory.
• You can also set limits like --max-model-len, --max-num-seqs, and --tensor-parallel-size.
• Make sure you’re using CUDA version 12 or higher.

You can create a Docker Compose YAML file for vLLM, which is better for version control and reproducibility.

Create the Compose file with the command below:

Add the following configuration to the file:

Then, launch the compose file with the following commands:

Tip: For a comprehensive tutorial on deploying general PyTorch models in production environments, including VPS setups, you can check this guide on PyTorch Model Inference Setup on VPS.

The AI ecosystem is multi-lingual, with models trained in PyTorch, TensorFlow, and exported to ONNX. NVIDIA Triton Inference Server can serve them all simultaneously, with advanced features like dynamic batching and models. In this step, we want to set up Triton with a sample model repository.

Triton requires a specific filesystem layout. You can create a host directory and set the correct permissions for it with the following commands:

For adding an ONNX model, for example, ResNet50, the directory layout looks like this:

Create a minimal config.pbtxt file with:

You can adjust the names to your model. Also, you can export models from PyTorch to ONNX with torch.onnx.export.

Run the Triton for GPU with the following command:

Run a health check by using the command below:

And model list with the command below:

Notes:

• You can send inference requests via HTTP/gRPC using Triton’s v2 protocol; Python and C++ clients exist.
• For CPU-only, choose an image variant that uses ONNX Runtime CPU and omit --gpus all.

Tip: For a focused guide on production-ready TensorFlow Serving with performance optimization techniques, check this guide on Setting up TensorFlow Serving on Dedicated Servers.

At this point, you can set up lightweight alternatives like Ollama and custom FastAPI services that are optimized for CPU execution.

Ollama is a simple tool that manages the model files and the inference backend, which provides a clean REST API. To do this, you can run the commands below:

Text Generation Inference (TGI): While vLLM is GPU-first, TGI has better support for CPU and quantization backends, which makes it a powerful alternative for CPU-based LLM serving.

FastAPI custom microservice for classic ML: You can wrap any Python model, for example, from scikit-learn, PyTorch, XGBoost, in a REST API. You can use the following script to create the model:

A reverse proxy acts as a secure gateway, managing incoming traffic, terminating TLS for HTTPS, and routing requests to the correct backend service. Caddy simplifies this process by automatically handling TLS certificates and renewal from Let's Encrypt.

We display vLLM on llm.example.com and Triton on triton.example.com, which point to the server's IP address. Run the Docker Caddy with:

Create the Caddyfile with the command below:

Add the following config to the file:

Reload caddy to apply the changes:

That's it, you are done with deploying AI Serving with Docker on a Dedicated Server.

At this point, you have learned to deploy AI Serving with Docker on a Dedicated Server, which is capable of serving both LLMs and multi-framework vision models using tools like vLLM and NVIDIA Triton for both GPU-accelerated and CPU-only setups.

PerLod hosting offers pre-configured GPU servers with Docker, NVIDIA stack, and optional model deployment services, which let you focus on your AI applications rather than infrastructure.

We hope you enjoy this guide. Subscribe to X and Facebook channels to get the latest articles in AI serving and Machine learning.