SGLang GPU Server Deployment: OpenAI API, systemd, Metrics, Benchmarks

SGLang lets you serve a language model from your own GPU server and connect apps through an OpenAI-compatible API. This guide shows how to deploy SGLang with an OpenAI API on a GPU server, protect requests with separate user and admin keys, add HTTPS, collect metrics, and measure speed.

Before you start to deploy SGLang with an OpenAI API on a GPU server, make sure have these requirements ready:

• A fresh Ubuntu 26.04 LTS x86_64 server or 24.04.
• An NVIDIA RTX 4090 with 24 GB VRAM, or another supported modern NVIDIA GPU with at least 16 GB VRAM for this small model and limited context.
• At least 32 GB system RAM; 64 GB is a useful starting choice.
• At least 100 GB free disk space for the container, model files, caches, and results.
• Root access or a sudo account.
• A domain such as llm.example.com with an A record pointing to the server.
• Public TCP ports 80 and 443. Keep SSH open only to trusted IPs where possible.

PerLod offers bare-metal NVIDIA options, including RTX 4090 servers. Remember to choose a suitable GPU dedicated server for this deployment.

First, you must run these commands from your sudo account to prepare your server:

If your NVIDIA driver works and meets the requirements, keep it. You can check it with:

If the driver is missing or too old, you can find Ubuntu's recommendation and install it:

Reconnect to the server and run nvidia-smi again. If Secure Boot blocks the driver, use the server console to approve the driver’s security key. Continue only when nvidia-smi shows your GPU correctly.

You do not need to install the CUDA Toolkit on the server. The SGLang container already includes CUDA and the Python packages it needs.

To deploy SGLang with an OpenAI API on a GPU server, you must use the official stable Docker repository and NVIDIA's production container repository. Install the latest stable Docker engine release with the commands below:

Once you are done, use the commands below to enable GPU access in Docker:

The toolkit adds NVIDIA support to Docker's runtime configuration.

At this point, you must download the stable SGLang image, then check its version and GPU access before starting the server.

Confirm that SGLang reports 0.5.21, CUDA is available, and the tensor command succeeds. A successful nvidia-smi alone does not prove the model runtime works.

Then, create the folders and save the image’s unique ID with the commands below:

The digest identifies the exact image you downloaded. The systemd service uses it to keep the same version, even if the image tag changes.

When you deploy SGLang with an OpenAI API on a GPU server, you must keep a fixed model snapshot so a later model update does not change your deployment.

Run sudo -i to switch to root for the rest of the setup. Keep this session open. If you disconnect, reconnect and run sudo -i again.

Download the current model revision and save its commit ID:

This model is public, so you do not need access approval. Download it before starting SGLang. The server will then load the model from the local folder.

SGLang provides --api-key for normal API access and --admin-api-key for control routes such as /server_info, cache operations, and weight updates. You must set up both. Remember that an admin key is not a customer key. 

To deploy SGLang with an OpenAI API on a GPU server safely, keep these keys different and never put the admin key in a web browser or public app.

Create the configuration and a private client environment file with the commands below:

Once clients use the service, run this code again only if you want to replace both API keys. To change non-secret settings, you can edit the file:

SGLang accepts a YAML configuration through --config. Here is what each setting in the configuration file does:

Check that your installed version supports these options:

Make sure all four options appear. This confirms your installed version supports them.

You can deploy SGLang with an OpenAI API on a GPU server and close your terminal. Systemd will start, stop, and restart the container. Also, do not set Docker to restart it.

Create a start script that uses the saved image digest:

This setup uses only GPU 0, not multiple GPUs. --shm-size=32g allows up to 32 GB of shared memory but does not use it all at once. 

Create the service with the command below and enable it:

Press Ctrl+C to leave the log view; the service keeps running. The model and GPU setup may take a few minutes.

Docker runs in the foreground and sends logs to the system journal. The container uses the image’s default user; this is not a rootless setup. Limit who has root or Docker access, as they can read the configuration and API keys.

active (running) means the process has started, but the model may not be ready yet. Check it with:

Check again after the model finishes loading. Make sure the port is available only at 127.0.0.1:30000, not 0.0.0.0:30000.

At this point, you must send a chat request to check that the API works. Then, test both keys. The user key should allow chat requests but block admin routes, while the admin key should allow access to those routes:

The response should include the model’s answer. Qwen recommends temperature: 0.7 and top_p: 0.8. We keep the response short for a quick API test.

Now you must check that requests with missing or wrong keys are blocked. Also, make sure the user key cannot access admin routes:

Requests to the model route must fail if the key is missing or wrong. The user key must be blocked from admin routes, while the admin key must work. Blocked requests may return 401 or 403. A user key must never get a successful response from an admin-only route.

Health and metrics routes do not require an API key. In this setup, /metrics stays private because SGLang is available only on localhost and Nginx does not expose that route.

To deploy SGLang with an OpenAI API on a GPU server for remote clients, you must add HTTPS before sharing the user key. Enter your own domain and email. If you reconnect, set these values again:

The A record must match the server's public IPv4 address. If you publish an AAAA record, it must also reach this server over IPv6. Remove an incorrect AAAA record before requesting a certificate.

At this point, you must only allow the required ports. If SSH uses a custom port, allow that port before enabling UFW:

Add the same rules to your hosting provider’s firewall. Do not open port 30000 to the internet. Docker ports can bypass UFW, so bind this port to 127.0.0.1 instead of relying only on UFW.

Create an Nginx configuration file to forward API requests to SGLang and keep other routes private. Only /v1/models and /v1/chat/completions are public. Metrics, admin routes, SGLang’s own generation routes, and documentation pages stay private.

Now install Certbot and get an SSL certificate for your domain to enable HTTPS:

Check that your API works over HTTPS and sends streamed responses as they are generated:

The chat request should return stream chunks. The metrics and admin requests must return 404 at Nginx. Test the public API from a separate machine too; a local test does not prove the external firewall is correct.

For apps that deploy SGLang with an OpenAI API on a GPU server, the main client changes are the base URL, local model name, and user key.

Run this on your application machine, not on the GPU host:

Use only the user key and keep /v1 at the end of the client’s base URL. The benchmark URL later does not need /v1 because the benchmark tool adds the API path itself.

This setup serves chat requests. It does not create embeddings or support every OpenAI feature.

When you deploy SGLang with an OpenAI API on a GPU server, you must collect metrics alongside client-side benchmark results. Server metrics help explain whether delay comes from queued work, prompt processing, or token generation.

Back in the root shell on the GPU host, run:

Send a few chat requests if the metrics do not have enough data yet. Useful metrics include:

The server’s first-token metric may stop before any text is visible. It can differ from how long a remote user waits to see the first text.

If Prometheus already runs directly on this server, add this job under scrape_configs:

This is a job example, not a replacement for an existing complete Prometheus configuration. If Prometheus runs in a container, 127.0.0.1 points to that container, not the GPU server. Use a private connection to reach SGLang, and do not make /metrics public through Nginx.

Use these PromQL queries after Prometheus has collected samples:

If the queue keeps growing, the server may be getting more requests than it can handle. Check GPU use and run benchmarks before allowing more requests to run at once.

Do not deploy SGLang with an OpenAI API on a GPU server based only on whether one request succeeds. You must measure streamed latency and throughput at the request sizes your app uses.

SGLang's bench_serving tool supports OpenAI chat requests, controlled concurrency, TTFT, token throughput, and JSONL result files. Its sglang-oai-chat backend calls /v1/chat/completions. 

Run inside the existing service container so the benchmark uses the same SGLang release:

Run these tests when no customers are using the service. The benchmark uses CPU and RAM, so run it from another machine for a more accurate capacity test.

Start with one request at a time to measure latency under a light load. Use these results as a baseline for the higher-concurrency tests:

Inside the container, 127.0.0.1:30000 connects directly to SGLang. --tokenizer /models uses the local tokenizer instead of trying to download one using the short API model name. OPENAI_API_KEY provides the key for API requests.

Keep streaming enabled; do not add --disable-stream. This lets the test measure the wait for the first token. The random dataset may download data on its first run. The final prompt may contain extra tokens added by the chat format.

Test several requests at once to measure throughput and see how latency changes as the load increases:

This test sends requests as fast as the concurrency limit allows. To test a steady rate, use --request-rate 2 for two requests per second and keep the other settings unchanged.

The user key may be blocked from the optional /server_info check. Review completed requests, errors, and saved latency results. Keep admin protection enabled.

Run the benchmark through HTTPS to include Nginx and TLS in the test:

This tests HTTPS and Nginx, but still runs locally. To measure network delay, repeat it from another machine near your users, using SGLang 0.5.21 and the same tokenizer. Do not add /v1 to --base-url.

Now you can check latency, throughput, and failed requests to see how the server performs at each load level. Record:

• Successful and failed requests.
• Mean, median, and p99 TTFT in milliseconds.
• Generated-token throughput in tokens per second.
• Request throughput in requests per second.
• Full request latency and time per generated token.
• GPU, driver, image digest, model revision, prompt size, generated size, concurrency, and cache state.

TTFT measures the wait for the first token. Generated-token throughput counts generated tokens across all requests. Total-token throughput also includes prompt tokens.

Run each test at least three times and use the middle result. Keep the model, request sizes, hardware, and cache settings the same when comparing runtimes. These tests keep prefix caching enabled, so they are not cold-cache tests.

Do not use --flush-cache with the user key; it needs admin access. For a fresh-cache test, restart the service during maintenance and wait until it is ready. Note that benchmark warmup adds data to the cache.

In this step, you can check logs and GPU use, fix common errors, and keep your keys and service up to date.

Check service logs and GPU use with the commands below:

For GPU memory errors, edit the configuration file:

Try these settings:

Restart and repeat readiness and benchmarks:

Lowering the memory share leaves more room for temporary GPU tasks, but less room for model weights and the KV cache. It will not help if the model weights are too large for your GPU.

If FlashInfer fails on a supported GPU, try these settings:

Use them only when needed and benchmark again; they can change speed.

If HTTPS returns 502, check that SGLang is ready and Nginx is using the correct port:

Check whether model loading finished, the service failed, or the proxy's port is wrong.

For authentication failures, confirm the correct key and model name:

Use qwen3-4b-instruct in API requests. Use the admin key only for local control routes.

Rotate keys: Replace your API keys when needed, then update the server settings and clients to use the new keys.

Edit both files so the matching key values stay in sync:

You can generate replacement values with:

Give your clients the new user key. The old key will stop working. The shared key does not provide billing or limits for each customer. You must add an API gateway with authentication if you need those controls.

Recover after repeated failures: Fix the cause shown in the logs, then reset the restart limit:

NVIDIA documents a container GPU-access issue related to systemctl daemon-reload on some systemd cgroup setups. If a running container loses GPU access after that command, restart this service and rerun the GPU checks.

Plan updates and rollback: Do not replace a production image with the latest and assume it is safe. Always save the current digest, configuration, and model revision:

To update, pull the new stable version’s exact tag, check GPU support and startup options, and save its digest in image.txt. Restart during maintenance, then test API keys, HTTPS, metrics, and performance before serving users.

To restore the saved image and configuration:

This restores only the image and settings. If you change the model files too, back up and restore the whole previous model folder, not just the revision file.

For a larger service, choose a server based on your test results, memory needs, and users’ locations. PerLod’s AI server hosting supports GPU inference and other AI tasks.

You can now deploy SGLang with an OpenAI API on a GPU server, with API keys, systemd, HTTPS, and private metrics. Test readiness, key security, HTTPS, and speed before serving users.

Deploy SGLang on a PerLod GPU server that fits your model, and use benchmark results to set request limits.

We hope you enjoy this guide. For more information, you can check the SGLang server arguments documentation.