TensorFlow Serving is a powerful, flexible system designed for serving machine learning models in production environments. By deploying TensorFlow Serving on dedicated servers, you can gain better performance, scalability, and security for your AI-powered applications. This setup allows you to host trained TensorFlow models, handle real-time predictions, and seamlessly integrate machine learning into your web or mobile platforms.
A dedicated server gives you full control over resources, enables smooth model updates, faster response times, and the ability to handle higher traffic without downtime. Whether you’re running a small AI project or a large-scale machine learning service, combining TensorFlow Serving with a reliable dedicated hosting solution ensures a professional and production-ready environment.
Now proceed to the rest of the article to deploy TensorFlow Serving on a dedicated server. In this guide to show you the steps we will use on a fresh Ubuntu 22.04 or Ubuntu 24.04 dedicated server (CPU-only or NVIDIA GPU).
Steps To Deploy TensorFlow Serving on Dedicated Servers
This guide covers model export, directory structure, Docker and native installs, config for multi-model and versioning, batching, performance tuning, monitoring, security, reverse proxy, systemd, and client examples.
Let's dive into the guide steps by preparing the server.
Prepare a Dedicated Server for TensorFlow Serving
Every successful deployment starts with a properly configured server. You must install essential system packages, set up firewall rules, and create a directory where all your models will be stored.
First, update the server and install the required packages with the following commands:
This directory will hold all model versions in the proper format.
Next, you must export a TensorFlow Model, for example, in Python. Here’s an example of exporting a MobileNetV2 model. This creates the versioned SavedModel layout expected by TF Serving:
Python
1# save_model.py2import tensorflow as tf, pathlib3model = tf.keras.applications.MobileNetV2(weights=None, classes=10)4model.build([None,224,224,3])5model.save("mobilenetv2/1")# version "1"
Once you are done, run the script:
Bash
python3save_model.pymvmobilenetv2/srv/tfmodels/ls-R/srv/tfmodels/mobilenetv2# -> 1/saved_model.pb and variables/
That layout is the standard SavedModel on disk.
Install TensorFlow Serving on Dedicated Servers | Docker or Native Install
TensorFlow Serving can be installed in two main ways:
Using Docker is recommended for flexibility and portability.
Via a native APT package, which is useful for minimal setups.
Here we will explain both methods; you can choose the right method depending on your needs.
Method 1. Installing TensorFlow Serving with Docker (Recommended)
This method is recommended because Docker simplifies updates and management. First, set up Docker with the following commands:
Set up TensorFlow Serving for Multiple Models and Versions
Real-world applications often require serving multiple models or multiple versions of the same model. In this step, you’ll learn how to set up TensorFlow Serving to handle more than one model using a models.config file with automatic hot reloading.
First, prepare model directories with the command below. This is an example model with two versions. Each version goes into its own numbered folder:
Bash
mkdir-p/srv/tfmodels/resnet50/{1,2}
Copy your SavedModel into each version folder accordingly.
Then, create the models.config file which tells TensorFlow Serving to load both models and watch for updates:
Bash
cat>/srv/tfmodels/models.config<<'EOF'model_config_list{config{name:"mobilenetv2"base_path:"/models/mobilenetv2"model_platform:"tensorflow"model_version_policy{all:{}}# serve all discovered versions}config{name:"resnet50"base_path:"/models/resnet50"model_platform:"tensorflow"model_version_policy{all:{}}}}EOF
This is the standard Model Server config format (ModelConfigList) for multiple models and version policies.
Now you can run multiple models with automatic reload when versions change.
Enable Server-Side Batching for Better Throughput
When handling high traffic, batching requests can improve performance by processing them together instead of one by one. Here, we want to configure server-side batching for better throughput without sacrificing response time.
First, create a batching config with the following command:
This config is ideal for high-traffic applications.
Set up TensorFlow Serving for Optimizing Resource Usage
In this step, you’ll learn how to adjust TensorFlow Serving options to maximize the performance of your CPU or GPU resources. You can fine-tune TensorFlow Serving with flags. For example:
This is adjusted based on CPU cores and latency needs. The flags above are supported by tensorflow_model_server and commonly tuned.
Enable TensorFlow Serving Prometheus Endpoint
As you must know, monitoring is essential for maintaining reliable machine learning services. This step shows you how to enable TensorFlow Serving’s Prometheus endpoint so you can track metrics like request counts, latencies, and errors in real time.
Note: REST endpoints, request shape, and versioned URLs are defined in the REST API docs.
For gRPC Python:
Python
1# client_grpc.py2import grpc, json3from tensorflow_serving.apis import predict_pb2, prediction_service_pb2_grpc4from tensorflow.core.framework import tensor_pb2, types_pb2, tensor_shape_pb256channel = grpc.insecure_channel("localhost:8500")7stub = prediction_service_pb2_grpc.PredictionServiceStub(channel)89req = predict_pb2.PredictRequest()10req.model_spec.name ="mobilenetv2"# or set version: req.model_spec.version.value = 11112# Build a TensorProto input named according to your SavedModel signature13vals =[0.0]*(224*224*3)14tensor = tensor_pb2.TensorProto(15 dtype=types_pb2.DT_FLOAT,16 tensor_shape=tensor_shape_pb2.TensorShapeProto(dim=[17 tensor_shape_pb2.TensorShapeProto.Dim(size=1),18 tensor_shape_pb2.TensorShapeProto.Dim(size=224),19 tensor_shape_pb2.TensorShapeProto.Dim(size=224),20 tensor_shape_pb2.TensorShapeProto.Dim(size=3),21]),22 float_val=vals23)24req.inputs["input_1"].CopyFrom(tensor)# change name to match your signature2526res = stub.Predict(req, timeout=10.0)27print(res.outputs.keys())
Tip: Inspect your model’s SignatureDefs to know exact input and output names and dtypes:
Secure TensorFlow Serving with Nginx Reverse Proxy and TLS
Displaying TensorFlow Serving ports directly is insecure. You can use Nginx as a reverse proxy with HTTPS/TLS encryption for more security. Now you will learn to set up a secure domain endpoint for safe model access.
If you want TensorFlow Serving to start automatically on boot, you must run TensorFlow Serving under systemd. To do this, create a systemd unit file for TensorFlow Serving:
Perform Health Checks | Verify TensorFlow Serving Model Status
Before clients rely on your service, you need to verify that models are loading correctly. In this step, you’ll learn how to perform health checks, read logs, and confirm that TensorFlow Serving is responding as expected.
Write new SavedModel into /srv/tfmodels/mobilenetv2/2/.
TensorFlow Serving will detect the new version automatically (if using base_path) or via --model_config_file_poll_wait_seconds if using config.
Clients can hit the latest or explicitly …/versions/2:predict.
Note: You can also use labels in config to shift traffic without changing clients.
Security Hardening Tips For TensorFlow Serving
Security is essential in production environments. Remember not to expose ports publicly; instead, use Nginx as a reverse proxy with TLS encryption. Also, lock the metrics path to internal networks.
Consider rate limits and auth in Nginx for the REST API, and keep Docker and base images updated.
Troubleshooting Common TensorFlow Serving Issues
Deployments don’t always go smoothly. This section covers common issues like model version errors, GPU not being detected, request timeouts, input/output mismatches, and how to fix them quickly.
Shape/dtype mismatch: Check saved_model_cli to match input names and shapes.
No versions available: Verify the versioned folder contains saved_model.pb and variables/. Also, verify permissions are readable by the container.
GPU not used: Confirm nvidia-smi works in the host and inside the container via --gpus all. Ensure you pulled the latest GPU.
Timeouts/thread starvation: Increase --rest_api_num_threads tune --rest_api_timeout_in_ms and add batching parameters.
Prometheus endpoint empty: Ensure --monitoring_config_file is set and the REST port is enabled; hit /monitoring/prometheus/metrics.
Minimal Docker Compose Setup for TensorFlow Serving
For teams that prefer Docker Compose, this step provides a minimal Compose configuration. It simplifies managing TensorFlow Serving with a single YAML file that handles models, configurations, and restarts.
Note: For a GPU with Compose, run the container using docker run --gpus all … as shown earlier. The official images support GPU when started with the NVIDIA runtime.
Testing TensorFlow Serving Setup
Finally, it’s time to confirm everything works. This step shows you how to send a test prediction request to your TensorFlow Serving instance to ensure your deployment is ready for real-world use.
Deploying TensorFlow Serving on dedicated servers gives you complete control over performance, scalability, and security. With these steps, your machine learning models can be deployed in a production-grade environment that is stable, secure, and ready to handle real-world workloads.
If you’re hosting on PerLod Hosting, you’ll benefit from the reliability, dedicated resources, and scalability required to serve AI models with confidence.
We hope you enjoy this guide. Subscribe to X and Facebook channels to get the latest updates and articles.
Docker is recommended because it makes upgrades, GPU support, and dependency management much easier. Native install is lightweight but less flexible for production.
No. TensorFlow Serving works fine on CPU, especially for smaller models or low-traffic environments. GPU is recommended only for large-scale inference or very deep models.
Yes. TensorFlow Serving can also serve models converted to TensorFlow SavedModel format.
We use cookies to run PerLod and to understand how the site is used. You can accept everything, or open Manage to turn individual categories off. Cookie policy.