Building smart AI systems often starts with Retrieval-Augmented Generation (RAG). But as your data grows and your users expect faster answers, a basic RAG setup is no longer enough. This is where GPU-powered embeddings and model serving make a big difference. In this guide, you will learn to build RAG Pipelines using GPU Servers.
By moving heavy tasks like generating embeddings and running large language models onto GPUs, you can speed up response times, handle larger workloads, and get much better accuracy.
For developers and companies looking for reliable hardware, PerLod Hosting offers the best GPs, which are optimized for AI workloads.
Building RAG Pipelines Using GPU Servers
In this guide, we assume you are running Ubuntu 22.04 or 24.04 on a server that has an NVIDIA GPU.
Our goal is to build a self-hosted RAG system that takes full advantage of the GPU Dedicated Server, which means running embeddings on the GPU, running the LLM on the GPU, and using a vector database (Qdrant) to store and retrieve documents.
Here are the three main parts we want to use:
Use Hugging Face’s Text Embeddings Inference (TEI) to create embeddings on the GPU.
Serve the large language model using vLLM, which gives us an OpenAI-compatible API and very fast GPU inference.
Store the vectors in Qdrant and access them through its Python client.
This setup gives you a fast, flexible, and fully self-hosted RAG system that takes full advantage of your GPU hardware.
Prerequisites for Setting Up a Fully Self-hosted RAG System
In this setup guide, we want to configure the Ubuntu server to run GPU-accelerated applications within Docker containers. You can easily create a robust environment that enables containerized software to access and utilize your GPU resources.
You must install NVIDIA drivers and CUDA by using the official docs or your cloud provider image. Just remember that the driver version must be compatible with CUDA 12.2+ because TEI images are built around that.
To verify your GPU model and driver, you can run the command below:
Bash
nvidia-smi
Install Docker on your server by using the following commands:
Deploy a GPU-Powered Embedding Service with Text Embedding Inference (TEI)
Once your server is configured with the necessary GPU and Docker tools, you can deploy a high-performance service for generating text embeddings.
In this step, we want to launch Hugging Face's Text Embeddings Inference (TEI) server, which is a specialized tool designed for fast and efficient batch embedding generation on GPU hardware.
The first step is to choose the model and create the model directory. We use the popular BAAI/bge-large-en-v1.5 model.
Create a project directory and navigate to it with the commands below:
Bash
sudomkdir-p~/tei_datacd~/tei_data
Choose the model and create the directory with the commands below:
The first run will download model weights into /data to avoid re-download.
Once it is running, you can send text to its API endpoint and receive back embeddings, which are the fundamental building blocks for RAG. To do this, you can run the command below:
Bash
sudocurl127.0.0.1:8080/embed\-XPOST\-H"Content-Type: application/json"\-d'{"inputs":"What is Retrieval Augmented Generation?"}'
You should receive a JSON response that includes an array of numbers called embeddings.
Deploy a High-Performance GPU LLM Server with vLLM
The next essential component of a RAG backend is a high-performance Large Language Model (LLM) server. We want to deploy vLLM, which is a powerful inference engine known for its efficiency and OpenAI-compatible API.
You can easily create an isolated Python environment, install vLLM, and launch a server that provides standard OpenAI endpoints like /v1/chat/completions.
Set up the Python environment with the commands below:
Bash
sudomkdir-p~/rag_llm&&cd~/rag_llmpython3-mvenv.venvsource.venv/bin/activatepipinstall--upgradepippipinstall"vllm[all]"# or `pip install vllm` if you prefer minimal extras
You must choose a model that fits your GPU; for a 24 GB GPU, you can often run a 7B instruct model; for smaller VRAM, pick a smaller model.
For example, we deploy vLLM to serve the Qwen/Qwen3-8B model on port 8001 with the command below:
Bash
vllmserveQwen/Qwen3-8B\--port8001\--host0.0.0.0
This downloads the model from Hugging Face on the first run and starts the server at http://0.0.0.0:8001/v1.
Now, verify the LLM server with the following curl command:
Bash
curlhttp://127.0.0.1:8001/v1/chat/completions\-H"Content-Type: application/json"\-d'{"model":"Qwen/Qwen3-8B","messages":[{"role":"user","content":"Say hello from a PerLod GPU optimized RAG backend."}],"max_tokens":128}'
You should see a JSON response with a choices array containing the model output.
Start Qdrant Vector Database
At this point, you need a dedicated database to store and query the vector embeddings generated by the TEI server. We want to use Qdrant, which is a high-performance and production-ready vector database.
Qdrant acts as the long-term memory for your application, efficiently storing vector embeddings and enabling lightning-fast similarity searches. When a user asks a question, the RAG pipeline will query Qdrant to find the most relevant information before the LLM generates a final answer.
Run Qdrant vector database in Docker with a persistent volume to ensure your data survives when the container restarts:
The Web Dashboard is at: http://localhost:6333/dashboard
gRPC is available at: localhost:6334
Build the RAG System with Python: Ingestion and Query
In this step, you must make the services work together. You can create a RAG system with a Python service for the entire workflow.
Create the RAG Ingestion Service
This Python service will ingest your documents by chunking and embedding them with the TEI server, store the results in Qdrant, and then answer questions by retrieving relevant context and generating responses with the vLLM LLM server.
You must install the libraries inside the ~/rag_llm venv you have created, or you can create a separate env on your choice:
Bash
cd~/rag_llmsource.venv/bin/activatepipinstallqdrant-clienthttpxfastapiuvicorntiktokenpipinstallopenai# for OpenAI compatible client
Here we use the ~/rag_llm directory as the project directory. Create a basic configuration file with the command below:
Bash
nano~/rag_llm/config.py
Add the following configuration to the file:
Python
1# config.py2TEI_URL ="http://127.0.0.1:8080"# TEI endpoint3QDRANT_URL ="http://127.0.0.1:6333"# Qdrant REST4QDRANT_COLLECTION ="rag_docs"56VLLM_BASE_URL ="http://127.0.0.1:8001/v1"# vLLM OpenAI compatible7VLLM_MODEL ="Qwen/Qwen3-8B"89EMBEDDING_DIM =1024# match your TEI model dimension (check HF model card)10TOP_K =511CHUNK_SIZE =51212CHUNK_OVERLAP =64
Note: Check the embedding size on the model’s page. For example, BGE Large often uses 1024, but you should confirm this on Hugging Face.
Then, create a chunking file with the command below:
Bash
nano~/rag_llm/chunking.py
Add the following chunking configuration to the file:
Python
1# chunking.py2from typing import List34defsimple_chunk(text: str, chunk_size:int=512, overlap:int=64)-> List[str]:5"""Simple word based chunking."""6 words = text.split()7ifnot words:8return[]910 chunks =[]11 start =012while start <len(words):13 end =min(len(words), start + chunk_size)14 chunk =" ".join(words[start:end])15 chunks.append(chunk)16if end ==len(words):17break18 start = end - overlap19return chunks
Now you must create the Clients' wrappers, including TEI, Qdrant, and LLM:
Place your .txt files inside the ./documents folder:
Bash
mkdir-p~/rag_llm/documents
Now you can run the ingestion with the command below:
Bash
cd~/rag_llmpythoningest.py
This script will:
Make sure the Qdrant collection exists with the correct vector size.
Split your documents into chunks and create embeddings using TEI.
Save the vectors and metadata into Qdrant.
Create the RAG Query Service
To make the RAG system accessible to frontend applications and other services, you need a standardized interface. You can create a FastAPI service that wraps the entire RAG pipeline into a clean HTTP endpoint.
Create the RAG query API file with the command below:
Bash
nano~/rag_llm/rag_api.py
Add the following configuration to create a /rag/query endpoint that takes a question and returns an answer with supporting context:
Python
1# rag_api.py2from typing import List34from fastapi import FastAPI5from pydantic import BaseModel67from config import TOP_K, VLLM_MODEL8from clients import EmbeddingClient, QdrantVectorStore, LLMClient910app =FastAPI()1112emb_client =EmbeddingClient()13store =QdrantVectorStore()14llm_client =LLMClient()1516classQueryRequest(BaseModel):17 question: str18 top_k:int|None=None1920classQueryResponse(BaseModel):21 answer: str22 context: List[str]2324@app.post("/rag/query", response_model=QueryResponse)25asyncdefrag_query(req: QueryRequest):26 top_k = req.top_k or TOP_K2728# 1. Embed question29 vectors =await emb_client.embed([req.question])30 query_vec = vectors[0]3132# 2. Search in Qdrant33 result = store.search(query_vec, top_k=top_k)3435 context_chunks =[]36for hit in result:37 payload = hit.payload or{}38 context_chunks.append(payload.get("text",""))3940 context_text ="\n\n".join(41 f"- {chunk}"for chunk in context_chunks if chunk.strip()42)4344# 3. Build prompt45 system_prompt =(46"You are a helpful assistant that answers using the provided context.\n"47"If the context does not contain the answer, say you are not sure.\n"48)4950 user_prompt =(51 f"Context:\n{context_text}\n\n"52 f"Question: {req.question}\n\n"53"Answer in a concise and accurate way based only on the context."54)5556# 4. Call LLM via vLLM OpenAI compatible API57 answer = llm_client.chat(58 model=VLLM_MODEL,59 messages=[60{"role":"system","content": system_prompt},61{"role":"user","content": user_prompt},62],63 max_tokens=512,64)6566returnQueryResponse(answer=answer, context=context_chunks)
Run the API with the following command:
Bash
uvicornrag_api:app--host0.0.0.0--port9000
You can test it with the following curl command:
Bash
curlhttp://127.0.0.1:9000/rag/query\-H"Content-Type: application/json"\-d'{"question": "What is this project about?"}'
You should get JSON with the answer and the context array.
Conclusion
Building a Retrieval-Augmented Generation (RAG) system with GPU-accelerated embeddings and model serving gives you the best combination of speed, accuracy, and scalability.
By using Hugging Face TEI for embeddings, vLLM for high-speed LLM serving, and Qdrant for vector search, you can create a modern, production-ready system that handles real workloads efficiently.
This RAG Pipelines Using GPU Servers Setup allows you to:
Quickly ingest large knowledge bases.
Get fast, low-latency LLM responses.
Make full use of your GPU resources.
Build a flexible and extendable retrieval pipeline.
We hope you enjoy this guide. Subscribe to our X and Facebook channels to get the latest updates and articles on GPU hosting.
Retrieval-Augmented Generation (RAG) is an AI architecture that combines a vector database, retriever, and generative model. When a user asks a question, the system finds the most relevant documents and sends them to the LLM to generate accurate and well-informed answers.
Yes. This is recommended for performance:
GPU 0: Embeddings (TEI)
GPU 1: LLM (vLLM)
This prevents resource contention and increases throughput.
Most common high-performance embedding models include BAAI/bge-large-en-v1.5, gte-large, and nomic-embed-text. All perform exceptionally when served through HuggingFace TEI.
We use cookies to run PerLod and to understand how the site is used. You can accept everything, or open Manage to turn individual categories off. Cookie policy.