GPU-Optimized RAG Pipelines for Fast and Scalable AI

Updated on Apr 16, 2026
Mathew M
11 MINS READ
Table of Contents
build RAG Pipelines using GPU Servers

Building smart AI systems often starts with Retrieval-Augmented Generation (RAG). But as your data grows and your users expect faster answers, a basic RAG setup is no longer enough. This is where GPU-powered embeddings and model serving make a big difference. In this guide, you will learn to build RAG Pipelines using GPU Servers.

By moving heavy tasks like generating embeddings and running large language models onto GPUs, you can speed up response times, handle larger workloads, and get much better accuracy.

For developers and companies looking for reliable hardware, PerLod Hosting offers the best GPs, which are optimized for AI workloads.

Building RAG Pipelines Using GPU Servers

In this guide, we assume you are running Ubuntu 22.04 or 24.04 on a server that has an NVIDIA GPU.

Our goal is to build a self-hosted RAG system that takes full advantage of the GPU Dedicated Server, which means running embeddings on the GPU, running the LLM on the GPU, and using a vector database (Qdrant) to store and retrieve documents.

Here are the three main parts we want to use:

  • Use Hugging Face’s Text Embeddings Inference (TEI) to create embeddings on the GPU.
  • Serve the large language model using vLLM, which gives us an OpenAI-compatible API and very fast GPU inference.
  • Store the vectors in Qdrant and access them through its Python client.

This setup gives you a fast, flexible, and fully self-hosted RAG system that takes full advantage of your GPU hardware.

Prerequisites for Setting Up a Fully Self-hosted RAG System

In this setup guide, we want to configure the Ubuntu server to run GPU-accelerated applications within Docker containers. You can easily create a robust environment that enables containerized software to access and utilize your GPU resources.

You must install NVIDIA drivers and CUDA by using the official docs or your cloud provider image. Just remember that the driver version must be compatible with CUDA 12.2+ because TEI images are built around that.

To verify your GPU model and driver, you can run the command below:

Bash
nvidia-smi

Install Docker on your server by using the following commands:

Bash
sudo apt updatesudo apt install ca-certificates curl gnupg -y sudo install -m 0755 -d /etc/apt/keyringscurl -fsSL https://download.docker.com/linux/ubuntu/gpg \  | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg echo \  "deb [arch=$(dpkg --print-architecture) \  signed-by=/etc/apt/keyrings/docker.gpg] \  https://download.docker.com/linux/ubuntu \  $(lsb_release -cs) stable" \| sudo tee /etc/apt/sources.list.d/docker.list >/dev/null sudo apt updatesudo apt install docker-ce docker-ce-cli containerd.io -ysudo usermod -aG docker $USER

To apply that your user is in the Docker group, log out and log in again.

To set up the NVIDIA container toolkit, you can run the commands below:

Bash
distribution=$(. /etc/os-release;echo $ID$VERSION_ID) curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \  | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg curl -s -L https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list \  | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' \  | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list sudo apt updatesudo apt install nvidia-container-toolkit -y sudo nvidia-ctk runtime configure --runtime=dockersudo systemctl restart docker

To confirm everything is working correctly, run nvidia-smi from inside a container:

Bash
docker run --rm --gpus all nvidia/cuda:12.2.0-base-ubuntu22.04 nvidia-smi

Deploy a GPU-Powered Embedding Service with Text Embedding Inference (TEI)

Once your server is configured with the necessary GPU and Docker tools, you can deploy a high-performance service for generating text embeddings.

In this step, we want to launch Hugging Face's Text Embeddings Inference (TEI) server, which is a specialized tool designed for fast and efficient batch embedding generation on GPU hardware.

The first step is to choose the model and create the model directory. We use the popular BAAI/bge-large-en-v1.5 model.

Create a project directory and navigate to it with the commands below:

Bash
sudo mkdir -p ~/tei_datacd ~/tei_data

Choose the model and create the directory with the commands below:

Bash
export TEI_MODEL="BAAI/bge-large-en-v1.5"export TEI_VOLUME="$PWD/data"mkdir -p "$TEI_VOLUME"

Then, containerize it with Docker, and expose it as a web service on port 8080 with the command below:

Bash
docker run --gpus all \  -p 8080:80 \  -v "$TEI_VOLUME":/data \  --pull always \  ghcr.io/huggingface/text-embeddings-inference:1.8 \  --model-id "$TEI_MODEL"

The first run will download model weights into /data to avoid re-download.

Once it is running, you can send text to its API endpoint and receive back embeddings, which are the fundamental building blocks for RAG. To do this, you can run the command below:

Bash
sudo curl 127.0.0.1:8080/embed \  -X POST \  -H "Content-Type: application/json" \  -d '{"inputs":"What is Retrieval Augmented Generation?"}'

You should receive a JSON response that includes an array of numbers called embeddings.

Deploy a High-Performance GPU LLM Server with vLLM

The next essential component of a RAG backend is a high-performance Large Language Model (LLM) server. We want to deploy vLLM, which is a powerful inference engine known for its efficiency and OpenAI-compatible API.

You can easily create an isolated Python environment, install vLLM, and launch a server that provides standard OpenAI endpoints like /v1/chat/completions.

Set up the Python environment with the commands below:

Bash
sudo mkdir -p ~/rag_llm && cd ~/rag_llmpython3 -m venv .venvsource .venv/bin/activate pip install --upgrade pippip install "vllm[all]" # or `pip install vllm` if you prefer minimal extras

You must choose a model that fits your GPU; for a 24 GB GPU, you can often run a 7B instruct model; for smaller VRAM, pick a smaller model.

For example, we deploy vLLM to serve the Qwen/Qwen3-8B model on port 8001 with the command below:

Bash
vllm serve Qwen/Qwen3-8B \  --port 8001 \  --host 0.0.0.0

This downloads the model from Hugging Face on the first run and starts the server at http://0.0.0.0:8001/v1.

Now, verify the LLM server with the following curl command:

Bash
curl http://127.0.0.1:8001/v1/chat/completions \  -H "Content-Type: application/json" \  -d '{    "model": "Qwen/Qwen3-8B",    "messages": [      {"role":"user","content":"Say hello from a PerLod GPU optimized RAG backend."}    ],    "max_tokens": 128  }'

You should see a JSON response with a choices array containing the model output.

Start Qdrant Vector Database

At this point, you need a dedicated database to store and query the vector embeddings generated by the TEI server. We want to use Qdrant, which is a high-performance and production-ready vector database.

Qdrant acts as the long-term memory for your application, efficiently storing vector embeddings and enabling lightning-fast similarity searches. When a user asks a question, the RAG pipeline will query Qdrant to find the most relevant information before the LLM generates a final answer.

Run Qdrant vector database in Docker with a persistent volume to ensure your data survives when the container restarts:

Bash
sudo mkdir -p ~/qdrant_storage docker run -p 6333:6333 -p 6334:6334 \  -v "$HOME/qdrant_storage:/qdrant/storage:z" \  qdrant/qdrant
  • REST API is available at: http://localhost:6333
  • The Web Dashboard is at: http://localhost:6333/dashboard
  • gRPC is available at: localhost:6334

Build the RAG System with Python: Ingestion and Query

In this step, you must make the services work together. You can create a RAG system with a Python service for the entire workflow.

Create the RAG Ingestion Service

This Python service will ingest your documents by chunking and embedding them with the TEI server, store the results in Qdrant, and then answer questions by retrieving relevant context and generating responses with the vLLM LLM server.

You must install the libraries inside the ~/rag_llm venv you have created, or you can create a separate env on your choice:

Bash
cd ~/rag_llmsource .venv/bin/activate pip install qdrant-client httpx fastapi uvicorn tiktokenpip install openai # for OpenAI compatible client

Here we use the ~/rag_llm directory as the project directory. Create a basic configuration file with the command below:

Bash
nano ~/rag_llm/config.py

Add the following configuration to the file:

Python
# config.pyTEI_URL = "http://127.0.0.1:8080"          # TEI endpointQDRANT_URL = "http://127.0.0.1:6333"       # Qdrant RESTQDRANT_COLLECTION = "rag_docs" VLLM_BASE_URL = "http://127.0.0.1:8001/v1" # vLLM OpenAI compatibleVLLM_MODEL = "Qwen/Qwen3-8B" EMBEDDING_DIM = 1024  # match your TEI model dimension (check HF model card)TOP_K = 5CHUNK_SIZE = 512CHUNK_OVERLAP = 64

Note: Check the embedding size on the model’s page. For example, BGE Large often uses 1024, but you should confirm this on Hugging Face.

Then, create a chunking file with the command below:

Bash
nano ~/rag_llm/chunking.py

Add the following chunking configuration to the file:

Python
# chunking.pyfrom typing import List def simple_chunk(text: str, chunk_size: int = 512, overlap: int = 64) -> List[str]:    """Simple word based chunking."""    words = text.split()    if not words:        return []     chunks = []    start = 0    while start < len(words):        end = min(len(words), start + chunk_size)        chunk = " ".join(words[start:end])        chunks.append(chunk)        if end == len(words):            break        start = end - overlap    return chunks

Now you must create the Clients' wrappers, including TEI, Qdrant, and LLM:

Bash
nano ~/rag_llm/clients.py

Add the following configuration to the file:

Python
# clients.pyimport httpxfrom typing import List, Dict, Anyfrom qdrant_client import QdrantClient, modelsfrom openai import OpenAI from config import TEI_URL, QDRANT_URL, QDRANT_COLLECTION, EMBEDDING_DIM, VLLM_BASE_URL class EmbeddingClient:    def __init__(self, base_url: str = TEI_URL):        self.base_url = base_url.rstrip("/")     async def embed(self, texts: List[str]) -> List[List[float]]:        async with httpx.AsyncClient(timeout=60.0) as client:            response = await client.post(                f"{self.base_url}/embed",                json={"inputs": texts},            )            response.raise_for_status()            data = response.json()            # TEI returns {"embeddings": [[...], ...]}            return data["embeddings"] class QdrantVectorStore:    def __init__(self, url: str = QDRANT_URL, collection: str = QDRANT_COLLECTION):        self.collection = collection        self.client = QdrantClient(url=url)     def ensure_collection(self, dim: int):        if not self.client.collection_exists(self.collection):            self.client.create_collection(                collection_name=self.collection,                vectors_config=models.VectorParams(                    size=dim,                    distance=models.Distance.COSINE                ),            )     def upsert_points(self, vectors: List[List[float]], payloads: List[Dict[str, Any]]):        assert len(vectors) == len(payloads)        points = [            models.PointStruct(                id=i,                vector=v,                payload=payloads[i],            )            for i, v in enumerate(vectors)        ]        self.client.upsert(            collection_name=self.collection,            wait=True,            points=points,        )     def search(self, query_vector: List[float], top_k: int = 5):        result = self.client.query_points(            collection_name=self.collection,            query=query_vector,            limit=top_k,        )        return result class LLMClient:    def __init__(self, base_url: str = VLLM_BASE_URL, api_key: str = "EMPTY"):        # OpenAI client, but pointed to vLLM        self.client = OpenAI(            api_key=api_key,            base_url=base_url,        )     def chat(self, model: str, messages: List[Dict[str, str]], max_tokens: int = 512):        response = self.client.chat.completions.create(            model=model,            messages=messages,            max_tokens=max_tokens,        )        return response.choices[0].message.content

Note: The Qdrant client code follows the official examples, using VectorParams for setting up the collection and query_points for searching.

Finally, create the ingestion script file with the following command:

Bash
nano ~/rag_llm/ingest.py

Add the following script to the file:

Python
# ingest.pyimport asynciofrom pathlib import Pathfrom typing import List from config import EMBEDDING_DIM, QDRANT_COLLECTION, TOP_K, CHUNK_SIZE, CHUNK_OVERLAPfrom chunking import simple_chunkfrom clients import EmbeddingClient, QdrantVectorStore DOCS_DIR = Path("./documents") async def ingest_directory():    emb_client = EmbeddingClient()    store = QdrantVectorStore()     store.ensure_collection(EMBEDDING_DIM)     all_chunks: List[str] = []    payloads = []     for path in DOCS_DIR.glob("*.txt"):        text = path.read_text(encoding="utf8", errors="ignore")        chunks = simple_chunk(text, CHUNK_SIZE, CHUNK_OVERLAP)        for idx, chunk in enumerate(chunks):            all_chunks.append(chunk)            payloads.append(                {                    "source": str(path.name),                    "chunk_index": idx,                    "text": chunk,                }            )     # Embed in batches    batch_size = 64    vectors: List[List[float]] = []    for i in range(0, len(all_chunks), batch_size):        batch = all_chunks[i : i + batch_size]        batch_vectors = await emb_client.embed(batch)        vectors.extend(batch_vectors)     store.upsert_points(vectors, payloads)     print(f"Ingested {len(all_chunks)} chunks into collection '{QDRANT_COLLECTION}'.") if __name__ == "__main__":    asyncio.run(ingest_directory())

Place your .txt files inside the ./documents folder:

Bash
mkdir -p ~/rag_llm/documents

Now you can run the ingestion with the command below:

Bash
cd ~/rag_llmpython ingest.py

This script will:

  • Make sure the Qdrant collection exists with the correct vector size.
  • Split your documents into chunks and create embeddings using TEI.
  • Save the vectors and metadata into Qdrant.

Create the RAG Query Service

To make the RAG system accessible to frontend applications and other services, you need a standardized interface. You can create a FastAPI service that wraps the entire RAG pipeline into a clean HTTP endpoint.

Create the RAG query API file with the command below:

Bash
nano ~/rag_llm/rag_api.py

Add the following configuration to create a /rag/query endpoint that takes a question and returns an answer with supporting context:

Python
# rag_api.pyfrom typing import List from fastapi import FastAPIfrom pydantic import BaseModel from config import TOP_K, VLLM_MODELfrom clients import EmbeddingClient, QdrantVectorStore, LLMClient app = FastAPI() emb_client = EmbeddingClient()store = QdrantVectorStore()llm_client = LLMClient() class QueryRequest(BaseModel):    question: str    top_k: int | None = None class QueryResponse(BaseModel):    answer: str    context: List[str] @app.post("/rag/query", response_model=QueryResponse)async def rag_query(req: QueryRequest):    top_k = req.top_k or TOP_K     # 1. Embed question    vectors = await emb_client.embed([req.question])    query_vec = vectors[0]     # 2. Search in Qdrant    result = store.search(query_vec, top_k=top_k)     context_chunks = []    for hit in result:        payload = hit.payload or {}        context_chunks.append(payload.get("text", ""))     context_text = "\n\n".join(        f"- {chunk}" for chunk in context_chunks if chunk.strip()    )     # 3. Build prompt    system_prompt = (        "You are a helpful assistant that answers using the provided context.\n"        "If the context does not contain the answer, say you are not sure.\n"    )     user_prompt = (        f"Context:\n{context_text}\n\n"        f"Question: {req.question}\n\n"        "Answer in a concise and accurate way based only on the context."    )     # 4. Call LLM via vLLM OpenAI compatible API    answer = llm_client.chat(        model=VLLM_MODEL,        messages=[            {"role": "system", "content": system_prompt},            {"role": "user", "content": user_prompt},        ],        max_tokens=512,    )     return QueryResponse(answer=answer, context=context_chunks)

Run the API with the following command:

Bash
uvicorn rag_api:app --host 0.0.0.0 --port 9000

You can test it with the following curl command:

Bash
curl http://127.0.0.1:9000/rag/query \  -H "Content-Type: application/json" \  -d '{"question": "What is this project about?"}'

You should get JSON with the answer and the context array.

Conclusion

Building a Retrieval-Augmented Generation (RAG) system with GPU-accelerated embeddings and model serving gives you the best combination of speed, accuracy, and scalability.

By using Hugging Face TEI for embeddings, vLLM for high-speed LLM serving, and Qdrant for vector search, you can create a modern, production-ready system that handles real workloads efficiently.

This RAG Pipelines Using GPU Servers Setup allows you to:

  • Quickly ingest large knowledge bases.
  • Get fast, low-latency LLM responses.
  • Make full use of your GPU resources.
  • Build a flexible and extendable retrieval pipeline.

We hope you enjoy this guide. Subscribe to our X and Facebook channels to get the latest updates and articles on GPU hosting.

For further reading:

Explore GPU Hosting Alternatives to AWS for AI Training

Retrieval-Augmented Generation (RAG) is an AI architecture that combines a vector database, retriever, and generative model. When a user asks a question, the system finds the most relevant documents and sends them to the LLM to generate accurate and well-informed answers.

Yes. This is recommended for performance: GPU 0: Embeddings (TEI) GPU 1: LLM (vLLM) This prevents resource contention and increases throughput.

Most common high-performance embedding models include BAAI/bge-large-en-v1.5, gte-large, and nomic-embed-text. All perform exceptionally when served through HuggingFace TEI.

Prev PostHow to Deploy Stable Diffusion on Dedicated GPU Servers for Maximum Performance
Next PostThe Ultimate Guide to Hosting Models with FlashAttention
Category:AI & GPU Infrastructure
Tag:

We use cookies to run PerLod and to understand how the site is used. You can accept everything, or open Manage to turn individual categories off. Cookie policy.