TensorFlow Serving is a powerful, flexible system designed for serving machine learning models in production environments. By deploying TensorFlow Serving on dedicated servers, you can gain better performance, scalability, and security for your AI-powered applications. This setup allows you to host trained TensorFlow models, handle real-time predictions, and seamlessly integrate machine learning into your web or mobile platforms.
A dedicated server gives you full control over resources, enables smooth model updates, faster response times, and the ability to handle higher traffic without downtime. Whether you’re running a small AI project or a large-scale machine learning service, combining TensorFlow Serving with a reliable dedicated hosting solution ensures a professional and production-ready environment.
Now proceed to the rest of the article to deploy TensorFlow Serving on a dedicated server. In this guide to show you the steps we will use on a fresh Ubuntu 22.04 or Ubuntu 24.04 dedicated server (CPU-only or NVIDIA GPU).
Steps To Deploy TensorFlow Serving on Dedicated Servers
This guide covers model export, directory structure, Docker and native installs, config for multi-model and versioning, batching, performance tuning, monitoring, security, reverse proxy, systemd, and client examples.
Let's dive into the guide steps by preparing the server.
Prepare a Dedicated Server for TensorFlow Serving
Every successful deployment starts with a properly configured server. You must install essential system packages, set up firewall rules, and create a directory where all your models will be stored.
First, update the server and install the required packages with the following commands:
sudo apt updatesudo apt install ca-certificates curl jq unzip ufw -ysudo ufw allow OpenSSHsudo ufw enable
This ensures your system is secure, up-to-date, and ready to run TensorFlow Serving.
Then, you must create a Model Repository. TensorFlow Serving expects models to live in a central folder. We will keep models in /srv/tfmodels:
sudo mkdir -p /srv/tfmodelssudo chown $USER:$USER /srv/tfmodels
This directory will hold all model versions in the proper format.
Next, you must export a TensorFlow Model, for example, in Python. Here’s an example of exporting a MobileNetV2 model. This creates the versioned SavedModel layout expected by TF Serving:
12import tensorflow as tf, pathlib3model = tf.keras.applications.MobileNetV2(weights=None, classes=10)4model.build([None, 224, 224, 3])5model.save("mobilenetv2/1")
Once you are done, run the script:
python3 save_model.pymv mobilenetv2 /srv/tfmodels/ls -R /srv/tfmodels/mobilenetv2
That layout is the standard SavedModel on disk.
Install TensorFlow Serving on Dedicated Servers | Docker or Native Install
TensorFlow Serving can be installed in two main ways:
- Using Docker is recommended for flexibility and portability.
- Via a native APT package, which is useful for minimal setups.
Here we will explain both methods; you can choose the right method depending on your needs.
Method 1. Installing TensorFlow Serving with Docker (Recommended)
This method is recommended because Docker simplifies updates and management. First, set up Docker with the following commands:
curl -fsSL https://get.docker.com | sudo shsudo usermod -aG docker $USERnewgrp docker
Note: If your server has an NVIDIA GPU, you must install the appropriate NVIDIA driver for your card and install NVIDIA Container Toolkit:
distribution=$(. /etc/os-release;echo $ID$VERSION_ID) curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg curl -fsSL https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list \ | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' \ | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list sudo apt update && sudo apt install nvidia-container-toolkit -ysudo nvidia-ctk runtime configure --runtime=dockersudo systemctl restart docker
Then, check your GPU access with the commands below:
nvidia-smi docker run --rm --gpus all nvidia/cuda:12.2.0-base-ubuntu22.04 nvidia-smi
For Docker GPU, you only need the NVIDIA driver and container toolkit; the CUDA toolkit on the host isn’t required. You can use the --gpus flag.
Once you are done with Docker setup, you can pull the TensorFlow Serving image with the following Docker command:
docker pull tensorflow/serving:latest
For GPU, you can run:
docker pull tensorflow/serving:latest-gpu
After pulling the image, you can run a single model for a quick start.
For CPU, you can run:
docker run -d --name tfs-mobilenetv2 \-p 8500:8500 -p 8501:8501 \-v /srv/tfmodels/mobilenetv2:/models/mobilenetv2 \-e MODEL_NAME=mobilenetv2 \tensorflow/serving:latest
docker run -d --name tfs-mobilenetv2-gpu \--gpus all \-p 8500:8500 -p 8501:8501 \-v /srv/tfmodels/mobilenetv2:/models/mobilenetv2 \-e MODEL_NAME=mobilenetv2 \tensorflow/serving:latest-gpu
At this point, TensorFlow Serving is running and ready to serve predictions.
Method 2. Installing TensorFlow Serving with APT
If you don't want to use Docker, you can natively install TensorFlow Serving with APT. To do this, you can run the commands below:
echo "deb [arch=amd64] http://storage.googleapis.com/tensorflow-serving-apt stable tensorflow-model-server tensorflow-model-server-universal" | sudo tee /etc/apt/sources.list.d/tensorflow-serving.list curl -fsSL https://storage.googleapis.com/tensorflow-serving-apt/tensorflow-serving.release.pub.gpg | sudo apt-key add - sudo apt update && sudo apt install tensorflow-model-server -y
Binary will be available at:
/usr/bin/tensorflow_model_server
Set up TensorFlow Serving for Multiple Models and Versions
Real-world applications often require serving multiple models or multiple versions of the same model. In this step, you’ll learn how to set up TensorFlow Serving to handle more than one model using a models.config file with automatic hot reloading.
First, prepare model directories with the command below. This is an example model with two versions. Each version goes into its own numbered folder:
mkdir -p /srv/tfmodels/resnet50/{1,2}
Copy your SavedModel into each version folder accordingly.
Then, create the models.config file which tells TensorFlow Serving to load both models and watch for updates:
cat >/srv/tfmodels/models.config <<'EOF'model_config_list { config { name: "mobilenetv2" base_path: "/models/mobilenetv2" model_platform: "tensorflow" model_version_policy { all: {} } } config { name: "resnet50" base_path: "/models/resnet50" model_platform: "tensorflow" model_version_policy { all: {} } }}EOF
This is the standard Model Server config format (ModelConfigList) for multiple models and version policies.
Next, launch with config and hot reload:
docker run -d --name tfs-multi \ -p 8500:8500 -p 8501:8501 \ -v /srv/tfmodels/mobilenetv2:/models/mobilenetv2 \ -v /srv/tfmodels/resnet50:/models/resnet50 \ -v /srv/tfmodels/models.config:/models/models.config \ tensorflow/serving:latest \ --model_config_file=/models/models.config \ --model_config_file_poll_wait_seconds=60
Now you can run multiple models with automatic reload when versions change.
Enable Server-Side Batching for Better Throughput
When handling high traffic, batching requests can improve performance by processing them together instead of one by one. Here, we want to configure server-side batching for better throughput without sacrificing response time.
First, create a batching config with the following command:
sudo tee /srv/tfmodels/batching.config >/dev/null <<'EOF'max_batch_size { value: 32 }batch_timeout_micros { value: 1000 } num_batch_threads { value: 4 }max_enqueued_batches { value: 1000 }EOF
docker run -d --name tfs-batched \ -p 8500:8500 -p 8501:8501 \ -v /srv/tfmodels/mobilenetv2:/models/mobilenetv2 \ -v /srv/tfmodels/batching.config:/etc/tfs/batching.config \ -e MODEL_NAME=mobilenetv2 \ tensorflow/serving:latest \ --enable_batching=true \ --batching_parameters_file=/etc/tfs/batching.config
This config is ideal for high-traffic applications.
Set up TensorFlow Serving for Optimizing Resource Usage
In this step, you’ll learn how to adjust TensorFlow Serving options to maximize the performance of your CPU or GPU resources. You can fine-tune TensorFlow Serving with flags. For example:
--rest_api_num_threads=48 \--tensorflow_intra_op_parallelism=8 \--tensorflow_inter_op_parallelism=4
This is adjusted based on CPU cores and latency needs. The flags above are supported by tensorflow_model_server and commonly tuned.
Enable TensorFlow Serving Prometheus Endpoint
As you must know, monitoring is essential for maintaining reliable machine learning services. This step shows you how to enable TensorFlow Serving’s Prometheus endpoint so you can track metrics like request counts, latencies, and errors in real time.
First, create a monitoring config with:
sudo tee /srv/tfmodels/monitoring.config >/dev/null <<'EOF'prometheus_config { enable: true path: "/monitoring/prometheus/metrics"}EOF
Then, start TensorFlow Serving with monitoring enabled:
docker run -d --name tfs-mon \ -p 8500:8500 -p 8501:8501 \ -v /srv/tfmodels/mobilenetv2:/models/mobilenetv2 \ -v /srv/tfmodels/monitoring.config:/etc/tfs/monitoring.config \ -e MODEL_NAME=mobilenetv2 \ tensorflow/serving:latest \ --rest_api_port=8501 \ --monitoring_config_file=/etc/tfs/monitoring.config
Verify metrics with the command below:
curl -s localhost:8501/monitoring/prometheus/metrics | head
Now Prometheus can scrape performance metrics.
Interact with TensorFlow Serving: Accessing Models via REST and gRPC
Once the server is running, clients need to send prediction requests. This step introduces the two main ways of interacting with TensorFlow Serving:
- REST: Simple and universal.
- gRPC: Faster and more efficient for high-performance systems.
For REST predict:
curl -s -X POST \ -H "Content-Type: application/json" \ -d '{"instances":[[0,0, ... image/inputs ... ]]}' \ http://localhost:8501/v1/models/mobilenetv2:predict | jq curl -s -X POST -H "Content-Type: application/json" \ -d '{"instances":[[...]]}' \ http://localhost:8501/v1/models/mobilenetv2/versions/1:predict
Note: REST endpoints, request shape, and versioned URLs are defined in the REST API docs.
For gRPC Python:
12import grpc, json3from tensorflow_serving.apis import predict_pb2, prediction_service_pb2_grpc4from tensorflow.core.framework import tensor_pb2, types_pb2, tensor_shape_pb25 6channel = grpc.insecure_channel("localhost:8500")7stub = prediction_service_pb2_grpc.PredictionServiceStub(channel)8 9req = predict_pb2.PredictRequest()10req.model_spec.name = "mobilenetv2" 11 1213vals = [0.0] * (224*224*3)14tensor = tensor_pb2.TensorProto(15 dtype=types_pb2.DT_FLOAT,16 tensor_shape=tensor_shape_pb2.TensorShapeProto(dim=[17 tensor_shape_pb2.TensorShapeProto.Dim(size=1),18 tensor_shape_pb2.TensorShapeProto.Dim(size=224),19 tensor_shape_pb2.TensorShapeProto.Dim(size=224),20 tensor_shape_pb2.TensorShapeProto.Dim(size=3),21 ]),22 float_val=vals23)24req.inputs["input_1"].CopyFrom(tensor) 25 26res = stub.Predict(req, timeout=10.0)27print(res.outputs.keys())
Tip: Inspect your model’s SignatureDefs to know exact input and output names and dtypes:
saved_model_cli show --dir /srv/tfmodels/mobilenetv2/1 --all
Secure TensorFlow Serving with Nginx Reverse Proxy and TLS
Displaying TensorFlow Serving ports directly is insecure. You can use Nginx as a reverse proxy with HTTPS/TLS encryption for more security. Now you will learn to set up a secure domain endpoint for safe model access.
First, bind the TensorFlow Serving locally:
docker run -d --name tfs-internal \ -p 127.0.0.1:8500:8500 -p 127.0.0.1:8501:8501 \ -v /srv/tfmodels/mobilenetv2:/models/mobilenetv2 \ -e MODEL_NAME=mobilenetv2 \ tensorflow/serving:latest
Then, set up Nginx and Certbot:
sudo apt install nginx -ysudo ufw allow 80,443/tcpsudo certbot --nginx -d api.yourdomain.com
Next, create a minimal proxy file:
sudo nano /etc/nginx/sites-available/tfs.conf
server { listen 443 ssl http2; server_name api.yourdomain.com; location /v1/ { proxy_pass http://127.0.0.1:8501; proxy_set_header Host $host; proxy_http_version 1.1; } location /monitoring/prometheus/metrics { allow 10.0.0.0/8; deny all; proxy_pass http://127.0.0.1:8501/monitoring/prometheus/metrics; }}
Enable the configuration, check for syntax errors, and reload Nginx to apply the changes:
sudo ln -s /etc/nginx/sites-available/tfs.conf /etc/nginx/sites-enabled/sudo nginx -t sudo systemctl reload nginx
Run TensorFlow Serving Under Systemd
If you want TensorFlow Serving to start automatically on boot, you must run TensorFlow Serving under systemd. To do this, create a systemd unit file for TensorFlow Serving:
sudo nano /etc/systemd/system/tfs@.service
Add the following content to the file:
[Unit]Description=TensorFlow Serving (%i)After=docker.serviceRequires=docker.service [Service]Restart=alwaysExecStart=/usr/bin/docker run --rm \ --name tfs-%i \ -p 127.0.0.1:8500:8500 -p 127.0.0.1:8501:8501 \ -v /srv/tfmodels/%i:/models/%i \ -e MODEL_NAME=%i \ tensorflow/serving:latestExecStop=/usr/bin/docker stop -t 10 tfs-%i [Install]WantedBy=multi-user.target
Then, enable and start the service with the commands below:
sudo systemctl daemon-reloadsudo systemctl enable --now tfs@mobilenetv2sudo systemctl status tfs@mobilenetv2
Perform Health Checks | Verify TensorFlow Serving Model Status
Before clients rely on your service, you need to verify that models are loading correctly. In this step, you’ll learn how to perform health checks, read logs, and confirm that TensorFlow Serving is responding as expected.
To verify model status (REST), you can run:
curl -s http://127.0.0.1:8501/v1/models/mobilenetv2 | jqcurl -s http://127.0.0.1:8501/v1/models/mobilenetv2/versions/1 | jq
Check logs and preparations with:
docker logs -f tfs-mobilenetv2
For rotating to a new version:
- Write new SavedModel into /srv/tfmodels/mobilenetv2/2/.
- TensorFlow Serving will detect the new version automatically (if using base_path) or via
--model_config_file_poll_wait_seconds if using config.
- Clients can hit the latest or explicitly …/versions/2:predict.
Note: You can also use labels in config to shift traffic without changing clients.
Security Hardening Tips For TensorFlow Serving
Security is essential in production environments. Remember not to expose ports publicly; instead, use Nginx as a reverse proxy with TLS encryption. Also, lock the metrics path to internal networks.
Use firewall rules:
sudo ufw deny 8500/tcpsudo ufw deny 8501/tcpsudo ufw allow 80,443/tcp
Consider rate limits and auth in Nginx for the REST API, and keep Docker and base images updated.
Troubleshooting Common TensorFlow Serving Issues
Deployments don’t always go smoothly. This section covers common issues like model version errors, GPU not being detected, request timeouts, input/output mismatches, and how to fix them quickly.
- Shape/dtype mismatch: Check
saved_model_cli to match input names and shapes.
- No versions available: Verify the versioned folder contains
saved_model.pb and variables/. Also, verify permissions are readable by the container.
- GPU not used: Confirm
nvidia-smi works in the host and inside the container via --gpus all. Ensure you pulled the latest GPU.
- Timeouts/thread starvation: Increase
--rest_api_num_threads tune --rest_api_timeout_in_ms and add batching parameters.
- Prometheus endpoint empty: Ensure
--monitoring_config_file is set and the REST port is enabled; hit /monitoring/prometheus/metrics.
Minimal Docker Compose Setup for TensorFlow Serving
For teams that prefer Docker Compose, this step provides a minimal Compose configuration. It simplifies managing TensorFlow Serving with a single YAML file that handles models, configurations, and restarts.
12services:3 tfserving:4 image: tensorflow/serving:latest5 container_name: tfserving6 ports:7 - "8500:8500"8 - "8501:8501"9 volumes:10 - /srv/tfmodels/mobilenetv2:/models/mobilenetv211 - /srv/tfmodels/models.config:/models/models.config12 - /srv/tfmodels/batching.config:/etc/tfs/batching.config13 - /srv/tfmodels/monitoring.config:/etc/tfs/monitoring.config14 command: >15 --model_config_file=/models/models.config16 --model_config_file_poll_wait_seconds=6017 --enable_batching=true18 --batching_parameters_file=/etc/tfs/batching.config19 --rest_api_num_threads=4820 --rest_api_timeout_in_ms=3000021 --monitoring_config_file=/etc/tfs/monitoring.config22 restart: unless-stopped
Note: For a GPU with Compose, run the container using docker run --gpus all … as shown earlier. The official images support GPU when started with the NVIDIA runtime.
Testing TensorFlow Serving Setup
Finally, it’s time to confirm everything works. This step shows you how to send a test prediction request to your TensorFlow Serving instance to ensure your deployment is ready for real-world use.
cat >payload.json <<'EOF'{"instances": [[[0,0,0],[0,0,0],...[trim to match your model]]] }EOF curl -s -X POST -H "Content-Type: application/json" \ -d @payload.json \ http://localhost:8501/v1/models/mobilenetv2:predict | jq
It confirms that TensorFlow Serving is working.
Conclusion
Deploying TensorFlow Serving on dedicated servers gives you complete control over performance, scalability, and security. With these steps, your machine learning models can be deployed in a production-grade environment that is stable, secure, and ready to handle real-world workloads.
If you’re hosting on PerLod Hosting, you’ll benefit from the reliability, dedicated resources, and scalability required to serve AI models with confidence.
We hope you enjoy this guide. Subscribe to X and Facebook channels to get the latest updates and articles.
For further reading:
Full setup guide for OSSEC and Fail2Ban on Linux Server
Monitoring Linux Server with Prometheus and Node Exporter
How to secure a dedicated server - SSH Hardening Techniques
