Building an AI app is easy; building one that serves thousands of users without issues is hard. If you are building an AI SaaS (Software as a Service), your GPU backend needs to do two things including scale up in high traffic and scale down in low traffic to save costs. This tutorial provides a complete setup for a scalable GPU backend for AI SaaS.
We will use vLLM for serving, Kubernetes with KEDA for scaling, and Semantic Caching to reduce costs.
Many developers are now moving their workloads to dedicated GPU providers like Perlod Hosting, which offers bare-metal performance for AI hosting and saves costs.
Architecture Review: Build a Scalable GPU Backend for AI SaaS
Your GPU backend needs a smart architecture that can handle many users at once, keep latency low, and avoid wasting expensive GPU time.
The architecture below shows how a single request travels through your system, from the user to the GPU and back. A cache layer like Redis sits in front of your models to answer repeated or similar questions instantly, while a load balancer routes the rest to GPU Dedicated Servers running vLLM.
Additionally, an autoscaling engine such as KEDA watches the traffic and spins GPUs up or down based on demand, so your AI app stays fast, stable, and cost-efficient even as usage grows.
Here is a simple architecture example to build the GPU backend for AI SaaS:
User sends a request, for example, Summarize this text.
Redis Cache Layer checks: Have we answered a similar question recently? Yes? Return the saved answer (Cost: $0, Time: 10ms). No? Send it to the GPU.
The Load Balancer sends the request to a GPU node.
vLLM Engine runs the model and generates text.
KEDA Scaler watches the line of waiting users. If the line gets too long, it adds more GPUs automatically.
Now that you have understood the architecture, proceed to the following steps to build a scalable GPU backend for AI SaaS.
Step1. Prerequisites for SaaS AI GPU Hosting
Before deploying vLLM, you must ensure your server has the right physical hardware and the software to let Docker talk to your GPU.
Hardware requirements include:
NVIDIA GPU: You need at least 8GB VRAM to run a quantized 7B model. For production, 24GB+, like an A10G or A100, is recommended.
Drivers: Ensure CUDA-compatible drivers are installed. The version must match your GPU. For a quick check, you can run:
Bash
nvidia-smi
If you see your GPU details, your drivers are working.
Tip: If you are unsure whether to use consumer cards or enterprise-grade, check out this comparison on RTX 4090 vs A100 for AI Training to see which fits your budget and workload best.
Operating System: Ubuntu 22.04 or Ubuntu 24.04 LTS is the industry standard for AI serving.
Also, you need Python 3.8+ installed on your system.
Software Tools You Need to Install:
Docker and NVIDIA Container Toolkit
Kubernetes.
KEDA Autoscaler for Kubernetes.
Prometheus Metrics collector for KEDA.
Redis Stack.
Python libraries: sentence-transformers, redis, and numpy.
By default, Docker containers cannot see your GPU. You need the NVIDIA Container Toolkit to do this.
If you haven't installed Docker yet, run these commands to get the latest version:
Bash
# Add Docker's official GPG key:sudoaptupdatesudoaptinstallca-certificatescurl-ysudoinstall-m0755-d/etc/apt/keyringssudocurl-fsSLhttps://download.docker.com/linux/ubuntu/gpg-o/etc/apt/keyrings/docker.ascsudochmoda+r/etc/apt/keyrings/docker.asc# Add the repository to Apt sources:echo\"deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \ $(. /etc/os-release && echo "$VERSION_CODENAME") stable"|\sudotee/etc/apt/sources.list.d/docker.list>/dev/nullsudoaptupdate# Install Dockersudoaptinstalldocker-cedocker-ce-clicontainerd.iodocker-buildx-plugindocker-compose-plugin-y
Then, install the NVIDIA Container Toolkit with the commands below:
Bash
# Add the repositorycurl-fsSLhttps://nvidia.github.io/libnvidia-container/gpgkey|sudogpg--dearmor-o/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg\&&curl-s-Lhttps://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list|\sed's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g'|\sudotee/etc/apt/sources.list.d/nvidia-container-toolkit.list# Update and Installsudoaptupdatesudoaptinstallnvidia-container-toolkit-y# Configure the Docker runtimesudonvidia-ctkruntimeconfigure--runtime=dockersudosystemctlrestartdocker
Now, verify that Docker can actually see your GPU with the command below:
Step2. Set up vLLM Serving Engine for SaaS AI Apps
The first step in your GPU backend is to set up a fast and reliable serving engine, and vLLM is one of the best options for this. It is a high-throughput, open-source engine that uses smart memory tricks like PagedAttention to keep your GPUs busy and your latency low.
In this step, we will turn vLLM into an HTTP server that looks and behaves like the OpenAI API, so your SaaS can plug into it with simple REST calls. We package everything in a Dockerfile, which the vLLM setup runs consistently across the local machine, staging, and production clusters.
Create your project folder and navigate to it:
Bash
sudomkdirai-saas-backendcdai-saas-backend
Create a Dockerfile from your project directory with the following command:
Bash
sudonanoDockerfile
Add the following content to the file:
Docker
# Use the official vLLM image from NVIDIA/vLLM (Verified stable version)FROMvllm/vllm-openai:latest# Set the working directory inside the containerWORKDIR/app# Install dependencies needed for the middleware script# NOTE: We add sentence-transformers and numpy here so they are available inside the containerRUNpipinstallredissentence-transformersnumpy# Expose port 8000 for API trafficEXPOSE8000# The command to start the server# --model: The name of the model (e.g., Mistral or Llama-3)# --quantization: specific compression to make it faster (awq)ENTRYPOINT["python3","-m","vllm.entrypoints.openai.api_server",\"--model","TheBloke/Mistral-7B-Instruct-v0.2-AWQ",\"--quantization","awq",\"--dtype","float16",\"--port","8000"]
Flags Explanations:
FROM: Pull the base system with all NVIDIA drivers pre-installed.
ENTRYPOINT: This command runs automatically. We use api_server to make it look exactly like GPT-4's API.
--quantization awq: This tells the GPU to use a compressed version of the model. It uses less memory and runs faster with almost no loss in quality.
Build the Docker image with the command below:
Bash
# Build the image (from inside your ai-saas-backend folder)dockerbuild-tmy-ai-backend:latest.
You should see logs like "Uvicorn running on 0.0.0.0:8000" and "Started vLLM engine successfully."
In another terminal, you can test the API with the command below:
Bash
curl-XPOSThttp://localhost:8000/v1/completions\-H"Content-Type: application/json"\-d'{"model":"Mistral-7B-Instruct-v0.2","prompt":"Explain quantum computing in simple terms:","max_tokens":100}'
If you see generated text, vLLM is working.
Note: This step assumes you already have a model ready to serve. If you are still in the training phase and looking for a cost-effective way to fine-tune your models before deploying them here, check out this guide on Kaggle GPU Training with Best Configs.
Step 3. Implement Semantic Caching for Cost Savings in SaaS AI Apps
Semantic caching is a game-changer for AI SaaS apps because it goes beyond simple text matching to understand the intent behind user questions. This means your system can reuse answers for similar queries without firing up expensive GPUs every time, which cuts costs and speeds up responses.
In this step, we build a cache layer using Redis with vector search, where queries get turned into embeddings for quick similarity checks.
First, start Redis in Docker with the commands below:
Bash
# Run Redis Stack (includes vector search capabilities)dockerrun-d--nameredis-stack-p6379:6379redis/redis-stack-server:latest# Verify Redis is runningdockerexecredis-stackredis-cliping# Expected output: PONG
Create a middleware.py file that acts as a middle layer between user requests and vLLM; check the cache first, then forward if needed.
Bash
sudonanomiddleware.py
Add the following sample script to the file:
Python
1import time2import numpy as np3from redis import Redis4from redis.commands.search.field import VectorField5from redis.commands.search.indexDefinition import IndexDefinition, IndexType6from sentence_transformers import SentenceTransformer78# 1. Connect to Redis9# Use 'localhost' for local testing, 'redis-service' for Kubernetes10REDIS_HOST ='localhost'11redis_client =Redis(host=REDIS_HOST, port=6379, decode_responses=False)1213# 2. Load Embedding Model14embedder =SentenceTransformer('all-MiniLM-L6-v2')1516definit_redis_index():17"""Initialize Redis vector index if it doesn't exist."""18try:19 redis_client.ft("cache_idx").info()20print("Index already exists.")21except:22# Define schema for vector search23 schema =(VectorField("embedding","FLAT",{24"TYPE":"FLOAT32",25"DIM":384,26"DISTANCE_METRIC":"COSINE"27}),)28 definition =IndexDefinition(prefix=["cache:"], index_type=IndexType.HASH)29 redis_client.ft("cache_idx").create_index(schema, definition=definition)30print("Created new Redis vector index.")3132defget_embedding(text):33"""Turns text into a 384-dim vector."""34return embedder.encode(text).astype(np.float32).tobytes()3536defcheck_cache(user_query):37"""Checks if a similar question exists."""38 vector =get_embedding(user_query)3940# Search for closest vector41 query = f"*=>[KNN 1 @embedding $vec AS score]"4243try:44 result = redis_client.ft("cache_idx").search(45 query,46 query_params={"vec": vector}47)4849# Threshold: 0.1 distance means very similar (90%+)50if result.total >0andfloat(result.docs[0].score)<0.1:51# The answer is stored in the 'answer' field of the hash52return result.docs[0].answer.decode('utf-8')53except Exception as e:54print(f"Cache search error: {e}")5556returnNone5758defsave_to_cache(user_query, llm_answer):59"""Saves new Q&A pair with embedding."""60 vector =get_embedding(user_query)61 key = f"cache:{int(time.time()*1000)}"6263# Store vector and answer in a Hash64 redis_client.hset(key, mapping={65"embedding": vector,66"answer": llm_answer67})68# Expire after 24 hours (optional)69 redis_client.expire(key,86400)7071# --- TEST SECTION ---72if __name__ =="__main__":73init_redis_index()7475print("Saving test Q&A...")76save_to_cache("Who is the president of USA?","The current president is Donald Trump.")7778print("Checking cache for similar question...")79# Different wording, same meaning80 answer =check_cache("Who runs the United States?")8182if answer:83print(f"Cache HIT! Answer: {answer}")84else:85print("Cache MISS.")
If 20% of your users ask similar questions, this script reduces your GPU bill by 20% and makes your app feel instant.
Test the Caching Script with:
Bash
# Install dependenciespipinstallsentence-transformersredisnumpy# Run the testpythonmiddleware.py
In your output, you must see something similar to this:
Advanced Tip: This tutorial focuses on the infrastructure for serving standard LLMs. However, if you are building a system that needs to read your own data (RAG), you can check this guide on Building RAG Pipelines Using GPU Servers.
Step 4. Deploy vLLM with Kubernetes for SaaS AI Apps
Once your Docker image is ready, the next step is to run it in a reliable, scalable environment, and Kubernetes is the standard choice for that.
Before deploying, push your image to Docker Hub or a private registry:
Bash
# Log in to Docker Hubdockerlogin# Tag your imagedockertagmy-ai-backend:latestyourusername/my-ai-backend:latest# Push the imagedockerpushyourusername/my-ai-backend:latest
Create a Deployment YAML file with the command below:
Bash
sudonanodeployment.yaml
Add the following content to the file:
YAML
1apiVersion: apps/v12kind: Deployment3metadata:4name: llm-inference5spec:6replicas:17selector:8matchLabels:9app: llm-inference10template:11metadata:12labels:13app: llm-inference14spec:15containers:16-name: vllm-container17# REPLACE 'yourusername' with your actual Docker Hub username18image: yourusername/my-ai-backend:latest19resources:20limits:21nvidia.com/gpu:1# Request 1 GPU22ports:23-containerPort:800024livenessProbe:25httpGet:26path: /health27port:800028initialDelaySeconds:6029---30apiVersion: v131kind: Service32metadata:33name: llm-service34spec:35selector:36app: llm-inference37ports:38-protocol: TCP39port:800040targetPort:800041type: LoadBalancer
The most important line is nvidia.com/gpu: 1, which tells Kubernetes to schedule this pod only on nodes that have a free NVIDIA GPU available, which prevents crashes and wasted deployments.
Step 5. Smart GPU Autoscaling with KEDA for AI SaaS Products
Traditional Kubernetes autoscaling is built around CPU metrics, which do not work well for AI inference workloads where GPUs do the heavy lifting and CPUs often stay almost idle. To keep your AI SaaS responsive and cost-efficient, you need to scale based on real demand signals, such as how many user requests are currently waiting in the queue.
KEDA (Kubernetes Event-Driven Autoscaling) solves this by reading metrics from Prometheus and dynamically adjusting the number of vLLM pods based on queue depth instead of CPU usage.
Install KEDA and Helm package manager with the commands below:
Bash
# Add KEDA to your clusterkubectlapply-fhttps://github.com/kedacore/keda/releases/download/v2.12.1/keda-2.12.1.yaml# Verify KEDA installedkubectlgetpods-nkeda# Add Helmcurlhttps://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3|bash
Then, create a YAML file for KEDA Scaler with the command below:
Bash
sudonanokeda-scaler.yaml
Add the following script to the file:
YAML
1apiVersion: keda.sh/v1alpha12kind: ScaledObject3metadata:4name: gpu-scaler5spec:6scaleTargetRef:7name: llm-inference8minReplicaCount:19maxReplicaCount:1010triggers:11-type: prometheus12metadata:13# Default Prometheus service address in K8s14serverAddress: http://prometheus-kube-prometheus-prometheus.default:909015# Query: Scale if requests per second > 516query: sum(rate(vllm_request_duration_seconds_count[1m]))17threshold:"5"
This gives you automatic, demand-based GPU scaling without manual intervention, which keeps latency low while controlling cloud costs.
Deploy the Scaler with the command below:
Bash
kubectlapply-fkeda-scaler.yaml
Monitor scaling and watch pods scale up or down in real time:
Bash
kubectlgetpods--watch
You now have a fully functional, production-ready GPU backend for AI SaaS.
Conclusion
Building a scalable AI SaaS is about architecture. In this guide, you have learned to build a scalable GPU backend for AI SaaS that serves fast using vLLM's PagedAttention, saves money using Redis Semantic Caching, and scales automatically using KEDA to add resources only when users are actually waiting.
We hope you enjoy this guide on SaaS AI GPU Hosting. Subscribe to our X and Facebook channels to get the latest articles and updates on AI and GPU hosting.
Redis runs entirely in memory (RAM), which makes it much faster than disk-based databases for checking cache.
For a 7B parameter model like Mistral or Llama 2/3:
FP16 (Full precision): ~16GB VRAM. Requires an A10G or A100.
Quantized (AWQ/GPTQ): ~6-8GB VRAM. Runs on cheaper T4 or consumer GPUs.
By using KEDA and setting minReplicaCount: 0, you can save 100% of your GPU costs when no one is using the app. However, when the first user arrives, it will take 1-2 minutes for the new Pod to spin up and load the model weights. For a SaaS, it is usually better to keep minReplicaCount: 1 so the first user gets an instant response.
We use cookies to run PerLod and to understand how the site is used. You can accept everything, or open Manage to turn individual categories off. Cookie policy.