How to Run llama.cpp as a Production OpenAI-Compatible Server

Running a large language model does not need a big cluster. With a single NVIDIA GPU, a quantized GGUF model, and llama-server, you can serve an OpenAI-compatible API for your apps, tools, and team. 

In this guide, you will set up llama.cpp speculative decoding on a real server and make it faster without buying more hardware. If you want a fast start, you can also look at AI hosting options built for this kind of work.

Before you start, make sure you have the items below. They are all simple, and you can set most of them up in a few minutes:

• A Linux server with Ubuntu 24.04 or newer and sudo access.
• One NVIDIA GPU. 12 GB of VRAM or more is a comfortable size for the models in this guide.
• At least 30 GB of free disk space.
• A domain name that points to your server. In this guide, we use llm.example.com.

If you do not have a GPU machine yet, a dedicated GPU server for LLM inference gives you full access to the card, so you can run all the steps below as they are.

llama.cpp needs a working NVIDIA driver to use your GPU. In this step, you must check that the driver is installed and that the system can see your card. Run the command below:

You should see your GPU name, the driver version, and the CUDA version. If the command is not found, you can install a driver with the commands below:

After the reboot, run nvidia-smi again to confirm that the GPU shows up.

The driver is enough to run models, but you need the CUDA toolkit to build llama.cpp. Run the commands below to add the NVIDIA repository and install the toolkit:

Then, add CUDA to your path and check that the compiler works:

Now you must install the build tools. libssl-dev is needed if you want HTTPS features such as downloading models from Hugging Face with -hf:

Then, download the source code. We use the v0.6.0 tag, which is the latest stable release at the time of writing.

Use the commands below to configure and build with CUDA turned on:

The build can take 10 to 20 minutes. If CMake cannot detect your GPU, you must set the architecture manually. For example, use 86 for RTX 30 series cards or 89 for RTX 40 series cards:

Check the build and confirm that llama.cpp can see your GPU:

Now copy the binaries to a fixed folder. We will use this path in the service file later:

In this step, you must download the two model files that the server will use. We use two models from the same family. The big one answers your users. The small one is the draft model that we add in Step 9.

• Main model: Qwen3 8B in Q4_K_M (about 5 GB).
• Draft model: Qwen3 0.6B in Q8_0 (about 0.6 GB).

Create a folder and download both files with the commands below:

A draft model must use the same tokenizer as the main model. Models from the same family, like Qwen3 8B and Qwen3 0.6B, match. Mixing different families will fail or give poor results.

Now you must create an API key to protect your server. Only people who send this key can use your models, so others cannot use your GPU.

Create a folder for the config files and generate a strong key:

Create a service user. It will run the server and read the key file:

Save the key in your shell so you can use it in tests:

The key file can hold many keys, one per line. Lines that start with # are comments.

For the first time, you must start the server manually. This is a baseline run, without a draft model:

Here is what each option does:

Wait until the log says the server is listening. Then open a second terminal and test:

Then, test that the key is really needed. This call has no key and should return a 401 error:

These two settings decide how many users you can serve and how much VRAM you need.

• -np sets the number of slots. Each slot handles one request at once.
• -c sets the total context. It is shared by the slots. With -c 16384 and -np 4, plan for about 4096 tokens per request.

You can pick the numbers like this:

1. Decide how many users will call the API at the same time. Set -np to that number.
2. Decide the longest prompt plus answer you need. Multiply it by -np. Set -c to the result.
3. Watch VRAM with nvidia-smi. If you run out of memory, lower -c or -np.

If VRAM is tight, you can store the KV cache in 8-bit. This cuts the cache size by about half compared to the default 16-bit. Add these options:

Keep Flash Attention on (-fa on) when you use a quantized KV cache. Test the quality of your answers after this change.

You can see what each slot is doing at any time with the command below:

Before we turn on llama.cpp speculative decoding, we need a baseline. You can create the benchmark script with:

Paste this code, then save the file (Ctrl+O, Enter, Ctrl+X):

The script uses temperature 0, so both runs with and without a draft model. This makes the comparison fair.

Run it and, in another terminal, watch GPU memory:

Write down the tok/s for each concurrency level and the highest memory.used value. Then stop the server with Ctrl+C.

Also, you can measure model speed without the server. llama-bench does this:

It reports prompt processing (pp) and text generation (tg) in tokens per second. Note that llama-bench does not test speculative decoding. Use the server script above for that.

Text generation is slow because the model creates one token at once. With llama.cpp speculative decoding, a small draft model guesses the next few tokens fast. Then, the big model checks all the guesses in one pass. Correct guesses are kept, so you get several tokens for the price of one big step. Wrong guesses are thrown away, so the output stays the same as the big model would give.

Start the server again with the draft model added:

These are the new options:

Old guides use --model-draft, -md, and --draft-max. In current llama.cpp, --draft-max and --draft-min were removed. You can use --spec-draft-n-max and --spec-draft-n-min instead. The -md short form still works. If you are unsure about an option on your build, run llama-server --help | grep spec.

Run the same benchmark again:

Now look at the server log. After each request, llama-server prints the draft acceptance rate, for example draft acceptance rate = 0.70312 ( 90 accepted / 128 generated). A higher rate means more speed. If the rate is low, the draft model is a poor match for your prompts.

Try a few values for --spec-draft-n-max, such as 4, 8, and 16. Pick the one that gives the best tok/s in your tests. Gains depend on your prompts, your GPU, and the model pair. Code and structured text usually gain more than free-form creative writing.

Use this table to judge if llama.cpp speculative decoding helps your load. Fill it in with your own numbers from Step 8 and Step 9:

Expect VRAM to go up a little, because the draft model and its own KV cache also live on the GPU. For the models in this guide, the draft weights are about 0.6 GB. Expect the biggest gain with 1 or 2 requests. When many requests run at once, the GPU is already busy, so the gain gets smaller. If speculation does not help at your load, you can remove the draft options.

Stop the manual server with Ctrl+C. Then, create the service file. It keeps llama.cpp speculative decoding on after every reboot:

Add the following content to the file:

Make the model files readable by the service user, start the service, then watch the logs:

To change any option later, edit the file, then run sudo systemctl daemon-reload and sudo systemctl restart llama-server.

Never open port 8080 to the internet. Keep the server on 127.0.0.1 and let Nginx handle outside traffic. For this purpose, install Nginx:

Create the site file with your desired text editor:

Paste this content. Replace llm.example.com with your domain:

Enable the site and reload Nginx with the following commands:

Now you must obtain a free TLS certificate with Certbot. Install Certbot and get your certificates with the commands below:

Then, open only the ports you need in the UFW firewall:

At this point, you have three layers of control, including the firewall, the Nginx rate limit, and the API key.

Now you must call the API through your domain. A request with the key must work:

A request without the key must fail, and the hidden endpoints must be blocked:

You can use the server with any OpenAI client. Set the base URL to https://llm.example.com/v1 and use your key. Here is an example with the Python SDK:

Now you have a complete single-GPU inference server. It uses a CUDA build of llama.cpp, a quantized GGUF model, API key protection, an Nginx proxy with HTTPS, and a draft model for faster output.

The benchmark script gives you real numbers, so you can decide if llama.cpp speculative decoding is worth it for your own load. You can find all server options in the llama.cpp server documentation.

You can run quantized models on a PerLod GPU server when VRAM efficiency matters more than maximum distributed throughput.