Running large language models (LLMs) on your own GPU gives you full control, better privacy, and can save money in the long run. This guide will show you how to set up an OpenAI Compatible API server using your own hardware or a hosting provider like PerLod Hosting, which allows any application that works with OpenAI's API to connect to your local models instead.
When we say a model server is "OpenAI-compatible," it means the server uses the same format and endpoints as OpenAI's official API. This allows you to use the same code, tools, and applications you would use with OpenAI, but pointed at your own server instead.
Applications like LangChain, LibreChat, and Open WebUI can connect to your local server without modification.
Hardware and Software Requirements for OpenAI Compatible Hosting
The most important setup is your GPU. If you don’t want to manage the hardware yourself, you can also use AI hosting from PerLod, which provides ready-to-use GPU servers already configured for OpenAI-compatible model serving.
Here is what you need for OpenAI Compatible Hosting:
1. Operating system: Ubuntu 22.04 or Ubuntu 24.04 LTS. 2. GPU: NVIDIA GPU with compute capability 7.0 or higher. 3. GPU Memory (VRAM): Depends on model size.
Small models (3B-7B parameters): 8-12GB VRAM
Medium models (13B-20B parameters): 16-24GB VRAM
Large models (70B+ parameters): 48GB+ VRAM or multiple GPUs
4. System RAM: At least 16GB. 5. NVIDIA Drivers: Version 535 or newer. 6. CUDA: Version 11.8 or 12.1+.
The --max-model-len sets the maximum context length means how much text the model can process at once.
Test OpenAI Compatible Serving
Once your server is up and running, you can test it to make sure everything works correctly.
You can use the following curl command in a new terminal, which keeps the server running in the first one:
Bash
curlhttp://localhost:8000/v1/chat/completions\-H"Content-Type: application/json"\-H"Authorization: Bearer sk-your-secret-key"\-d'{"model":"meta-llama/Llama-3.2-3B-Instruct","messages":[{"role":"system","content":"You are a helpful assistant."},{"role":"user","content":"Hello! Can you explain what you are?"}],"max_tokens":100,"temperature":0.7}'
Note: If you didn't set an API key, remove the Authorization header line. If you set a different API key, replace sk-your-secret-key with your actual key.
You should receive a JSON response with the model's answer.
Also, you can test your server using Python with the OpenAI library. Install the OpenAI Python library with the command below:
Bash
pipinstallopenai
Create a test script file with the command below:
Bash
nanotest_api.py
Add the following script to the file:
Python
1from openai import OpenAI23# Point the client to your local server4client =OpenAI(5 base_url="http://localhost:8000/v1",6 api_key="sk-your-secret-key"# Use your actual key, or any string if no key is set7)89# Make a chat completion request10response = client.chat.completions.create(11 model="meta-llama/Llama-3.2-3B-Instruct",# Must match the model you're serving12 messages=[13{"role":"system","content":"You are a helpful assistant."},14{"role":"user","content":"What is the capital of France?"}15],16 max_tokens=100,17 temperature=0.718)1920print(response.choices[0].message.content)
Then, run the script:
Bash
pythontest_api.py
If everything is working, you'll see the model's response printed to your terminal.
Use OpenAI-compatible API Server with Existing Applications
One of the amazing features of an OpenAI-compatible API is that you can use it with any tool that supports OpenAI.
For example, you can use it with LangChain, which is a popular framework for building LLM applications. Here's how to connect it to your local server:
Python
1from langchain_openai import ChatOpenAI23llm =ChatOpenAI(4 base_url="http://localhost:8000/v1",5 api_key="sk-your-secret-key",6 model="meta-llama/Llama-3.2-3B-Instruct"7)89response = llm.invoke("Tell me a joke about programming")10print(response.content)
Another example is to use it with Open WebUI, which is a user-friendly interface for chatting with LLMs. To connect it to your local server:
1. Install Docker and Docker Compose. 2. Set up Open WebUI following their documentation. 3. In Open WebUI settings, go to Admin Settings, Connections, and OpenAI. 4. Add a new connection:
API URL: http://localhost:8000/v1
API Key: Your key (or leave empty if you didn't set one)
5. Save and start chatting.
Also, you can use it with Environment Variables. Many applications read OpenAI credentials from environment variables. You can set these to point to your local server:
Check the service status with the following command:
Bash
sudosystemctlstatusvllm-server
View logs with:
Bash
sudojournalctl-uvllm-server-f
Now your server will start automatically when your system boots.
Alternative Option: Use Ollama for OpenAI Compatible Hosting
If you prefer a simpler setup with a user-friendly interface, you can consider using Ollama, which is easier to set up but offers less customization than vLLM.
Install Ollama with the command below:
Bash
curl-fsSLhttps://ollama.com/install.sh|sh
Run a model with Ollama:
Bash
ollamarunllama3.2:3b
Then, expose as an OpenAI-compatible API. Ollama automatically provides an OpenAI-compatible API on port 11434. To use it, you can run:
Python
1from openai import OpenAI23client =OpenAI(4 base_url="http://localhost:11434/v1",5 api_key="not-needed"# Ollama doesn't require an API key6)78response = client.chat.completions.create(9 model="llama3.2:3b",10 messages=[{"role":"user","content":"Hello!"}]11)1213print(response.choices[0].message.content)
Ollama is great for getting started quickly, but vLLM offers better performance and more configuration options for production use.
Fix Common Issues in OpenAI Compatible Serving
During setting up OpenAI-compatible API Server, you may encounter some errors and issues. Here are the most common issues you may face and their solutions:
1. CUDA out of memory error: The model is too large for your GPU. Try:
Using a smaller model.
Reducing "--max-model-len" to decrease context size.
Lowering "--gpu-memory-utilization" to 0.8 or 0.7.
Using quantized models such as 4-bit or 8-bit versions.
2. Server is slow or unresponsive error:
Check if you have enough VRAM.
Ensure no other programs are using the GPU.
Try reducing batch size or concurrent requests.
Make sure you're using --dtype auto for optimal performance.
3. Cannot connect from other devices:
Use "--host 0.0.0.0" instead of localhost.
Check your firewall settings: sudo ufw allow 8000.
Verify the server is running: curl http://localhost:8000/v1/models
4. Model downloading is very slow:
Models can be 10-50GB in size, so downloads take time.
Consider downloading manually first using huggingface-cli download.
Check your internet connection.
Use --download-dir to specify a download location with enough space.
Performance Optimization Tips for OpenAI Compatible Serving
Good performance makes your local AI server feel fast and stable for every request. Here are some simple ways to speed up your OpenAI-compatible server, like using more than one GPU, turning on flash attention, changing batch size, and choosing light (quantized) models.
These tips help you get more speed from your GPU and avoid common memory errors while serving many users.
1. Use tensor parallelism for large models: If you have multiple GPUs, use --tensor-parallel-size to split the model across them.
4. Use quantized models: 4-bit or 8-bit quantized models use less memory and run faster with minimal quality loss. Look for models with "GPTQ," "AWQ," or "GGUF" in their name.
5. Monitor GPU usage: Keep nvidia-smi running in a separate terminal to watch GPU utilization and memory usage.
Running your own models can be much cheaper than using cloud APIs when you send a lot of requests every month. With cloud providers, you pay for every million tokens you send and receive, so your bill grows as your traffic grows.
With a self-hosted setup, you mainly pay once for the GPU hardware and then for power and cooling, while there are no per-token fees. This means that after the initial hardware cost, busy workloads can become far more cost-effective on your own server.
In addition to running your own hardware, you can also choose GPU dedicated servers, which offer high-performance GPUs. These servers give you the benefits of self-hosting without the upfront hardware purchase.
OpenAI GPT-4 Pricing:
Input: $2.50 per million tokens
Output: $10.00 per million tokens
Self-Hosted Costs:
One-time GPU purchase: $500–$5,000, depending on the GPU.
Or GPU dedicated servers: monthly or hourly rental (no per-token fees), ideal if you want full control without buying hardware.
Conclusion
Hosting your own OpenAI compatible API server gives you full control over your AI infrastructure, reduces costs, and improves privacy. By using tools like vLLM or Ollama, you can run powerful language models directly on your GPU, integrate them into existing applications, and customize performance in ways that cloud providers do not allow.
We hope you enjoy this guide. Subscribe to our X and Facebook channels to get the latest updates and articles on GPU and AI Hosting.
Yes. Although small models can run on modest GPUs (8–12GB VRAM), larger models require 24GB, 48GB, or even multiple GPUs. If you don’t own a GPU, you can use GPU dedicated servers or AI hosting from PerLod, which provides preconfigured environments ready for OpenAI-compatible serving.
Yes. Any tool that uses OpenAI’s environment variables will automatically connect to your local or hosted server.
An OpenAI-compatible API server uses the same API format, endpoints, and request structure as OpenAI’s official API. This allows any application to connect to your own self-hosted model instead of OpenAI with no code changes.
We use cookies to run PerLod and to understand how the site is used. You can accept everything, or open Manage to turn individual categories off. Cookie policy.