SGLang Speculative Decoding: EAGLE-3, MTP, NGRAM Benchmarks and OOM Fixes

Updated on Oct 18, 2026
Mathew M
23 MINS READ
Table of Contents
SGLang Speculative Decoding

Speculative decoding can make LLM answers faster, but it does not help every setup. In this guide, you will run a fair test on one GPU server and measure each mode with the same model, the same prompts, and the same settings. This SGLang speculative decoding tuning guide uses SGLang 0.5.21.

If you are still choosing between inference engines, read our 
vLLM vs TGI vs Triton comparison. This guide is only about SGLang.

What You Will Test

Then, the big model checks all the guesses in one step. It keeps the correct guesses and drops the wrong ones. If many guesses are correct, you get more than one token per step.

Four terms are used in this guide:

  • Draft tokens: the guesses made by the helper.
  • Accept length: the average number of tokens you get from each target step. It includes one extra token that the target model always gives.
  • Acceptance rate: the share of draft tokens that the target model accepted.
  • Verification capacity: how many draft tokens the target model checks in one step.

You can change three SGLang settings to control how the draft tree works. Here is what each one does:

Setting What it controls Cost when you raise it
--speculative-num-steps Draft depth: how many tokens the helper guesses in a row More chance of wrong guesses at the end of the chain
--speculative-eagle-topk Branching: how many options the helper keeps at each step More memory and compute
--speculative-num-draft-tokens Verification capacity: the most draft tokens checked at once More GPU memory

These are the modes in this guide, based on the SGLang speculative decoding documentation:

Mode Extra model needed? Main flag Notes
EAGLE-3 Yes, an EAGLE-3 draft model --speculative-algorithm EAGLE3 The SGLang docs recommend it for best speed and quality
MTP Often no --speculative-algorithm EAGLE Uses the model's own multi-token layers. NEXTN is another name for EAGLE
Standalone Yes, a smaller model --speculative-algorithm STANDALONE Does not support DP attention
NGRAM No --speculative-algorithm NGRAM CUDA only. It turns off the overlap scheduler and mixed chunked prefill

Here is a public example. On one H100 with Llama 3.1 8B (MT-bench), normal decoding ran at 158.34 tokens/s. EAGLE-2 reached 244.10 tokens/s, and EAGLE-3 reached 373.25 tokens/s. These numbers will not match your setup, so test it yourself.

Requirements

You need these items before you start SGLang speculative decoding tuning:

  • A Linux server with an NVIDIA GPU that has 24 GB VRAM or more. This guide uses one RTX 4090 (24 GB). If you need a server, you can choose a 
    GPU dedicated server with enough VRAM for your model.
  • Ubuntu 24.04 or 26.04 (x86_64), a working NVIDIA driver, Docker, and the NVIDIA Container Toolkit.
  • At least 100 GB of free disk space and 32 GB of RAM.
  • Root access. Run sudo -i and keep that root shell open.

This SGLang speculative decoding tuning test uses the models below. Each model fits on a 24 GB GPU when you run it alone:

Test Target model Helper
Baseline, EAGLE-3, Standalone, NGRAM Qwen/Qwen3-8B (about 16 GB in bf16) EAGLE-3: thoughtworks/Qwen3-8B-Eagle3. Standalone: Qwen/Qwen3-0.6B. NGRAM: none
MTP XiaomiMiMo/MiMo-7B-RL Built-in MTP layer

MTP uses a different model, so it has its own baseline. Never compare MTP numbers with the Qwen3 numbers.

Step 1: Prepare the Server

For this SGLang speculative decoding tuning lab, you will use a separate folder, container name, and port. Your setup from the SGLang GPU server deployment guide stays the same. First, make sure Docker and the NVIDIA toolkit are ready. If they are not, follow steps 1 to 3 of that guide.

If you followed the deploy guide, a service named sglang is already using the GPU. You must stop it first, because 24 GB of VRAM is not enough for two servers:

Bash
sudo -isystemctl is-active sglangsystemctl stop sglangnvidia-smi --query-gpu=name,memory.total,memory.used --format=csv

If you do not have the sglang service, skip the two systemctl commands. The memory.used value should be close to zero. If it is not, find the process that uses the GPU and stop it.

Then, create the test folders and pull the same SGLang v0.5.21 image tag:

Bash
mkdir -p /opt/sglang-spec/bin /opt/sglang-spec/cache /opt/sglang-spec/resultsdocker pull lmsysorg/sglang:v0.5.21docker image inspect lmsysorg/sglang:v0.5.21 --format '{{index .RepoDigests 0}}' \  | tee /opt/sglang-spec/results/image-digest.txt

Check that this release has the flags you need:

Bash
docker run --rm --entrypoint python3 lmsysorg/sglang:v0.5.21 \  -m sglang.launch_server --help > /opt/sglang-spec/server-help.txt grep -E -- '--speculative-(algorithm|num-steps|eagle-topk|num-draft-tokens|draft-model-path)|--speculative-ngram|--cuda-graph-max-bs|--mem-fraction-static' \  /opt/sglang-spec/server-help.txt

Make sure --cuda-graph-max-bs-decode is in the list. In older releases, this flag was named --cuda-graph-max-bs. If your output only shows the old name, replace the new name in the scripts below.

Step 2: Download the Models and Data

Now you must download the models into the test cache. The command prints the folder name of each model, which includes its version hash. Save it for your records:

Bash
docker run --rm -i \  -e HF_HOME=/cache/huggingface \  -v /opt/sglang-spec/cache:/cache \  --entrypoint python3 lmsysorg/sglang:v0.5.21 - <<'PY' \  | tee /opt/sglang-spec/results/model-revisions.txtfrom huggingface_hub import snapshot_download for repo in [    "Qwen/Qwen3-8B",    "thoughtworks/Qwen3-8B-Eagle3",    "Qwen/Qwen3-0.6B",    "XiaomiMiMo/MiMo-7B-RL",]:    print(repo, "->", snapshot_download(repo_id=repo))PY

Download the ShareGPT prompt file. It has real chat prompts, which is better than random text for speculative decoding. Random text gives poor draft guesses and hides the real result:

Bash
curl -L -o /opt/sglang-spec/cache/sharegpt.json \  https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.jsonls -lh /opt/sglang-spec/cache/sharegpt.json

Step 3: Create the Test Scripts

You need three small files. The first file runs the whole test for you. It starts a new server, waits for it to be ready, records VRAM, runs the benchmark, saves the logs, and stops the server.

Each run needs a new server. SGLang shows the accept length as an average since the server started. If you reuse a server, the results from different tests get mixed. 

First, create the runner script with the command below:

Bash
nano /opt/sglang-spec/bin/run-test.sh

Paste this, then save with Ctrl+O, Enter, and exit with Ctrl+X:

Bash
#!/usr/bin/env bash# Usage: run-test.sh NAME CONCURRENCY [extra sglang server args...]# Example: run-test.sh q3-baseline 1set -uo pipefail NAME="$1"; C="$2"; shift 2 ROOT=/opt/sglang-specIMAGE="lmsysorg/sglang:v0.5.21"MODEL="${MODEL:-Qwen/Qwen3-8B}"WORKLOAD="${WORKLOAD:-sharegpt}"NUM_PROMPTS="${NUM_PROMPTS:-64}"OUTPUT_LEN="${OUTPUT_LEN:-256}"PORT=30001CNAME=sglang-specOUT="$ROOT/results/${NAME}-c${C}" rm -rf "$OUT"; mkdir -p "$OUT"echo START_FAILED > "$OUT/status.txt"docker rm -f "$CNAME" >/dev/null 2>&1 || true finish() {  [ -n "${SMI_PID:-}" ] && kill "$SMI_PID" 2>/dev/null  docker logs "$CNAME" > "$OUT/server.log" 2>&1  docker rm -f "$CNAME" >/dev/null 2>&1  if grep -qiE 'out of memory|OutOfMemoryError|Not enough memory' \       "$OUT/server.log" "$OUT/bench.log" 2>/dev/null; then    echo OOM > "$OUT/status.txt"  fi  echo "[$NAME c=$C] status: $(cat "$OUT/status.txt")"}trap finish EXIT # Common settings: keep these the same for every rundocker run -d --name "$CNAME" --gpus all --shm-size 16g \  -e CUDA_VISIBLE_DEVICES=0 \  -e HF_HOME=/cache/huggingface \  ${EXTRA_ENV:+-e $EXTRA_ENV} \  -v "$ROOT/cache:/cache" \  -v "$ROOT/results:/results" \  -p 127.0.0.1:${PORT}:30000 \  --entrypoint python3 "$IMAGE" \  -m sglang.launch_server \  --model-path "$MODEL" --host 0.0.0.0 --port 30000 \  --dtype bfloat16 --context-length 4096 \  --mem-fraction-static 0.80 \  --attention-backend flashinfer \  --max-running-requests 16 \  --cuda-graph-max-bs-decode 16 \  --enable-metrics \  "$@" > /dev/null READY=0for i in $(seq 1 180); do  if curl -sf --max-time 5 "http://127.0.0.1:${PORT}/health_generate" > /dev/null; then    READY=1; break  fi  if [ "$(docker inspect -f '{{.State.Running}}' "$CNAME" 2>/dev/null)" != "true" ]; then    break  fi  sleep 5done[ "$READY" = "1" ] || exit 1 echo BENCH_FAILED > "$OUT/status.txt"nvidia-smi --query-gpu=memory.used --format=csv,noheader,nounits | head -n 1 > "$OUT/vram_idle.txt"nvidia-smi --query-gpu=memory.used --format=csv,noheader,nounits -lms 500 > "$OUT/vram.csv" &SMI_PID=$! if [ "$WORKLOAD" = "json" ]; then  python3 "$ROOT/bin/json_load.py" --port "$PORT" --concurrency "$C" \    --num-prompts "$NUM_PROMPTS" --output-len "$OUTPUT_LEN" \    --out "$OUT/bench.jsonl" 2>&1 | tee "$OUT/bench.log"else  docker exec "$CNAME" python3 -m sglang.bench_serving \    --backend sglang --host 127.0.0.1 --port 30000 \    --model "$MODEL" \    --dataset-name sharegpt --dataset-path /cache/sharegpt.json \    --sharegpt-output-len "$OUTPUT_LEN" \    --num-prompts "$NUM_PROMPTS" --max-concurrency "$C" --request-rate inf \    --warmup-requests 3 \    --output-file "/results/${NAME}-c${C}/bench.jsonl" 2>&1 | tee "$OUT/bench.log"fiRC=${PIPESTATUS[0]} if [ "$RC" = "0" ] && [ -s "$OUT/bench.jsonl" ]; then  echo OK > "$OUT/status.txt"fi

The script uses these base settings for every run: bfloat16, 4096 context length, --mem-fraction-static 0.80, the flashinfer backend, and 16 running requests at most. You can change any of them. Just add the flag after the concurrency number.

If FlashInfer fails on your GPU, change --attention-backend flashinfer to triton in this file and use it for all runs.

Then, create the second script. It sends a repeat-heavy JSON workload, which we use for the NGRAM test:

Bash
nano /opt/sglang-spec/bin/json_load.py
Python
#!/usr/bin/env python3"""Repeat-heavy JSON workload for SGLang (native /generate, streaming)."""import argparse, json, time, urllib.requestfrom concurrent.futures import ThreadPoolExecutor p = argparse.ArgumentParser()p.add_argument("--port", type=int, default=30001)p.add_argument("--concurrency", type=int, default=1)p.add_argument("--num-prompts", type=int, default=64)p.add_argument("--output-len", type=int, default=256)p.add_argument("--out", required=True)a = p.parse_args()base = f"http://127.0.0.1:{a.port}" def prompt(i):    return (        "Return only a JSON array with 40 objects. Each object has the fields "        f'"id", "name", "email", and "active". Start the ids at {i * 100 + 1}.\n'    ) def call(i):    body = {        "text": prompt(i),        "stream": True,        "sampling_params": {"temperature": 0, "max_new_tokens": a.output_len, "ignore_eos": True},    }    req = urllib.request.Request(        base + "/generate", data=json.dumps(body).encode(),        headers={"Content-Type": "application/json"},    )    t0 = time.perf_counter(); ttft = None; tokens = 0    with urllib.request.urlopen(req, timeout=600) as r:        for raw in r:            line = raw.decode().strip()            if not line.startswith("data:") or line.endswith("[DONE]"):                continue            chunk = json.loads(line[5:])            if ttft is None:                ttft = time.perf_counter() - t0            tokens = chunk["meta_info"]["completion_tokens"]    total = time.perf_counter() - t0    tpot = (total - ttft) / (tokens - 1) if tokens > 1 else 0.0    return tokens, ttft, tpot call(0)  # warmupstart = time.perf_counter()with ThreadPoolExecutor(max_workers=a.concurrency) as ex:    res = list(ex.map(call, range(1, a.num_prompts + 1)))wall = time.perf_counter() - start tokens = sum(r[0] for r in res)info = json.load(urllib.request.urlopen(base + "/server_info", timeout=30))accept = info.get("internal_states", [{}])[0].get("avg_spec_accept_length") out = {    "workload": "json",    "completed": len(res),    "concurrency": a.concurrency,    "total_output_tokens": tokens,    "output_throughput": tokens / wall,    "mean_ttft_ms": 1000 * sum(r[1] for r in res) / len(res),    "mean_tpot_ms": 1000 * sum(r[2] for r in res) / len(res),    "accept_length": accept,}with open(a.out, "a") as f:    f.write(json.dumps(out) + "\n")print(json.dumps(out, indent=2))

Next, create the third script. It builds one table from all your runs and saves a summary.csv file:

Bash
nano /opt/sglang-spec/bin/summarize.py
Python
#!/usr/bin/env python3"""Build one table from all runs in /opt/sglang-spec/results."""import csv, json, refrom pathlib import Path ROOT = Path("/opt/sglang-spec")RES = ROOT / "results" def pick(d, *keys):    for k in keys:        if d.get(k) is not None:            return d[k]    return None def last_json(path):    lines = [l for l in path.read_text().splitlines() if l.strip()]    return json.loads(lines[-1]) if lines else {} def first_int(path):    try:        return int(path.read_text().split()[0])    except Exception:        return None def peak(path):    try:        vals = [int(x) for x in path.read_text().split() if x.isdigit()]        return max(vals) if vals else None    except Exception:        return None rows = []for d in sorted(p for p in RES.iterdir() if p.is_dir()):    m = re.match(r"^(.*)-c(\d+)$", d.name)    if not m:        continue    cfg, conc = m.group(1), int(m.group(2))    status = (d / "status.txt").read_text().strip() if (d / "status.txt").exists() else "UNKNOWN"    r = {}    if status == "OK" and (d / "bench.jsonl").exists():        r = last_json(d / "bench.jsonl")    rows.append({        "config": cfg, "conc": conc, "status": status,        "tok_s": pick(r, "output_throughput"),        "tpot_ms": pick(r, "mean_tpot_ms"),        "ttft_ms": pick(r, "mean_ttft_ms"),        "accept_len": pick(r, "accept_length"),        "vram_idle": first_int(d / "vram_idle.txt"),        "vram_peak": peak(d / "vram.csv"),    }) base = {(r["config"], r["conc"]): r["tok_s"] for r in rows}def fmt(v, n=1):    return "n/a" if v is None else (f"{v:.{n}f}" if isinstance(v, float) else str(v)) for r in rows:    group = r["config"].split("-")[0]    b = base.get((group + "-baseline", r["conc"]))    r["speedup"] = (r["tok_s"] / b) if (r["tok_s"] and b) else None cols = ["config", "conc", "status", "tok_s", "speedup", "tpot_ms", "ttft_ms",        "accept_len", "vram_idle", "vram_peak"]with open(ROOT / "summary.csv", "w", newline="") as f:    w = csv.DictWriter(f, fieldnames=cols)    w.writeheader()    for r in rows:        w.writerow({k: r[k] for k in cols}) print("| Config | Conc | Status | Out tok/s | vs base | TPOT ms | TTFT ms | Accept len | VRAM idle MiB | VRAM peak MiB |")print("|---|---|---|---|---|---|---|---|---|---|")for r in rows:    sp = "n/a" if r["speedup"] is None else f"{r['speedup']:.2f}x"    print(f"| {r['config']} | {r['conc']} | {r['status']} | {fmt(r['tok_s'])} | {sp} | "          f"{fmt(r['tpot_ms'])} | {fmt(r['ttft_ms'])} | {fmt(r['accept_len'], 2)} | "          f"{fmt(r['vram_idle'])} | {fmt(r['vram_peak'])} |")

Once you are done, make the scripts ready to run:

Bash
chmod +x /opt/sglang-spec/bin/run-test.sh /opt/sglang-spec/bin/json_load.py /opt/sglang-spec/bin/summarize.pybash -n /opt/sglang-spec/bin/run-test.sh && python3 -m py_compile /opt/sglang-spec/bin/json_load.py /opt/sglang-spec/bin/summarize.py && echo "scripts ok"

Step 4: Run the Baseline

The baseline is normal SGLang decoding with no speculative mode. Every other result is compared with it, so a clean baseline is the base of all SGLang speculative decoding tuning work. First, run one test:

Bash
cd /opt/sglang-spec/bin./run-test.sh q3-baseline 1

The name format is group-config. The part before the first dash (q3) is the group. The summary script compares each run with the -baseline run of the same group and the same concurrency.

Check the result file and confirm that the field names match the summary script:

Bash
tail -n 1 /opt/sglang-spec/results/q3-baseline-c1/bench.jsonl | python3 -m json.tool | head -n 60

You should see output_throughput, mean_tpot_ms, and mean_ttft_ms. If a name is different in your release, you must edit the pick(...) lines in summarize.py.

Now run the baseline at the concurrency levels you care about. Concurrency is the number of requests running at the same time:

Bash
for C in 4 8; do ./run-test.sh q3-baseline $C; done

Use the same concurrency levels for every mode. One user and one chat window is 1. A busy API is 8 or more.

Step 5: Test EAGLE-3

EAGLE-3 needs a draft model that was trained for your target model. Here, we use thoughtworks/Qwen3-8B-Eagle3. Start with the small and safe setting, including 3 steps, top-k 1, and 4 draft tokens.

Bash
cd /opt/sglang-spec/bin./run-test.sh q3-eagle3-s3k1d4 1 \  --speculative-algorithm EAGLE3 \  --speculative-draft-model-path thoughtworks/Qwen3-8B-Eagle3 \  --speculative-num-steps 3 \  --speculative-eagle-topk 1 \  --speculative-num-draft-tokens 4

Two rules for these three numbers:

  • Set all three together, or leave all three unset so SGLang picks values for you. When you tune, always set all three.
  • With topk equal to 1, the draft tokens are steps + 1. With a higher topk, keep num-draft-tokens at or below steps × topk + 1. SGLang's own tuning script skips combinations above this limit.

Look at the summary with the command below:

Bash
python3 /opt/sglang-spec/bin/summarize.py

Step 6: Test a Standalone Draft Model

Standalone mode uses a smaller normal model as the helper. The helper must use the same tokenizer as the target model. Qwen3-0.6B and Qwen3-8B are from the same family. The SGLang defaults for this mode are 3 steps, top-k 1, and 4 draft tokens.

Bash
cd /opt/sglang-spec/bin./run-test.sh q3-standalone-s3k1d4 1 \  --speculative-algorithm STANDALONE \  --speculative-draft-model-path Qwen/Qwen3-0.6B \  --speculative-num-steps 3 \  --speculative-eagle-topk 1 \  --speculative-num-draft-tokens 4

The helper uses extra VRAM for its weights. A helper that is not trained to copy the big model usually gives a lower accept length than an EAGLE-3 draft model. Your table will show if that is true here.

Step 7: Test NGRAM

NGRAM needs no extra model. It builds a cache of token patterns from text that was already generated, and it uses that cache to guess the next tokens. It works best when the output repeats itself, such as JSON, code, or templates.

Use the command below to run the default setting on the normal chat prompts:

Bash
cd /opt/sglang-spec/bin./run-test.sh q3-ngram-d12 1 \  --speculative-algorithm NGRAM

If you leave --speculative-num-draft-tokens unset, NGRAM uses 12. Next, you can test the setting from the SGLang docs, which uses 16 draft tokens and a wider search:

Bash
./run-test.sh q3-ngram-d16b10 1 \  --speculative-algorithm NGRAM \  --speculative-num-draft-tokens 16 \  --speculative-ngram-max-bfs-breadth 10

Also, these NGRAM settings are available:

Flag Default Meaning
--speculative-ngram-min-bfs-breadth 1 Smallest search width
--speculative-ngram-max-bfs-breadth 10 Largest search width
--speculative-ngram-match-type BFS BFS uses recent text. PROB uses how often a pattern appears
--speculative-ngram-max-trie-depth 18 Longest pattern that is stored
--speculative-ngram-capacity 10,000,000 Cache size in entries

Next, try the frequency mode once with the command below:

Bash
./run-test.sh q3-ngram-prob 1 \  --speculative-algorithm NGRAM \  --speculative-ngram-match-type PROB

Now run the repeat-heavy JSON workload. This test shows where NGRAM can win. Run a baseline and an NGRAM test with the same script. The WORKLOAD=json setting switches the benchmark:

Bash
WORKLOAD=json ./run-test.sh json-baseline 1WORKLOAD=json ./run-test.sh json-ngram-d12 1 --speculative-algorithm NGRAM

Do not compare the json- rows with the q3- rows. They use different prompts.

Step 8: Test MTP

Some models have extra layers that predict more than one next token. This is called multi-token prediction (MTP). SGLang runs MTP through the speculative decoding path. The docs use XiaomiMiMo/MiMo-7B-RL as the example and the EAGLE algorithm name. They do not need a draft model path for this model.

DeepSeek models use the same path with their own NextN layers. The docs list a default of 3 steps, top-k 1, and 4 draft tokens for DeepSeek, and 1, 1, and 2 as the smallest setup. DeepSeek models need a multi-GPU server, so we test MiMo here.

Run a MiMo baseline and then MTP with the commands below:

Bash
cd /opt/sglang-spec/binMODEL=XiaomiMiMo/MiMo-7B-RL ./run-test.sh mimo-baseline 1 --trust-remote-code MODEL=XiaomiMiMo/MiMo-7B-RL ./run-test.sh mimo-mtp-s1k1d2 1 \  --trust-remote-code \  --speculative-algorithm EAGLE \  --speculative-num-steps 1 \  --speculative-eagle-topk 1 \  --speculative-num-draft-tokens 2 MODEL=XiaomiMiMo/MiMo-7B-RL ./run-test.sh mimo-mtp-s3k1d4 1 \  --trust-remote-code \  --speculative-algorithm EAGLE \  --speculative-num-steps 3 \  --speculative-eagle-topk 1 \  --speculative-num-draft-tokens 4

If your model has MTP layers, you must check its SGLang page for the right flags. Not every model supports this mode.

Step 9: Tune Draft Depth and Verification Size

Now run the full test set. This is the main part of SGLang speculative decoding tuning. Every run starts a new server, so it takes some time. Begin with concurrency 1, and run the other levels later.

Run the EAGLE-3 tests. It covers a short chain, longer chains, and two branching settings:

Bash
cd /opt/sglang-spec/binE3="--speculative-algorithm EAGLE3 --speculative-draft-model-path thoughtworks/Qwen3-8B-Eagle3" for C in 1 4 8; do  for CFG in "3 1 4" "5 1 6" "7 1 8" "4 2 8" "3 4 12"; do    read S K D <<< "$CFG"    ./run-test.sh q3-eagle3-s${S}k${K}d${D} $C $E3 \      --speculative-num-steps $S \      --speculative-eagle-topk $K \      --speculative-num-draft-tokens $D  donedone

Run the same loops for the other modes:

Bash
SA="--speculative-algorithm STANDALONE --speculative-draft-model-path Qwen/Qwen3-0.6B" for C in 1 4 8; do  for CFG in "3 1 4" "5 1 6" "4 2 7"; do    read S K D <<< "$CFG"    ./run-test.sh q3-standalone-s${S}k${K}d${D} $C $SA \      --speculative-num-steps $S \      --speculative-eagle-topk $K \      --speculative-num-draft-tokens $D  done  for D in 8 12 16; do    ./run-test.sh q3-ngram-d${D} $C \      --speculative-algorithm NGRAM --speculative-num-draft-tokens $D  donedone

Follow this order when you read the results:

  1. Raise steps first (3, 5, 7) with topk 1. Stop when tokens per second stop going up. A deeper chain only helps while the late guesses are still correct.
  2. Then try topk 2 to 4 with a larger num-draft-tokens. This can raise the accept length, but it costs more VRAM and compute.
  3. Repeat at each concurrency level. A large tree that wins at concurrency 1 can lose at concurrency 8, because the GPU has less space for the extra checking work.

Also, you can try two more options from the SGLang docs as extra rows. The first is Speculative Decoding V2 (the overlap scheduler). It needs topk 1 and works with EAGLE, EAGLE3, and STANDALONE. The second is adaptive speculative decoding, which changes the draft depth while the server runs and also needs topk 1. You can test V2 like this:

Bash
EXTRA_ENV=SGLANG_ENABLE_SPEC_V2=True ./run-test.sh q3-eagle3-v2-s3k1d4 1 $E3 \  --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4

SGLang also has a script that tests these settings for you. It is called bench_speculative.py, and it is in the SGLang repository under scripts/playground. Use it to double-check your results.

To see the accept length while a test is running, start the server with --enable-metrics. Then, read the live values with this command:

Bash
curl -s http://127.0.0.1:30001/metrics | grep -E 'sglang:spec_accept_(rate|length)'

Step 10: Build the Results Table

Create the summary table from all runs with the command below:

Bash
python3 /opt/sglang-spec/bin/summarize.py

The table has these columns. The same data is saved to /opt/sglang-spec/summary.csv.

Column Meaning
Out tok/s Output tokens per second for all requests together
vs base Out tok/s divided by the baseline at the same concurrency
TPOT ms Time per output token after the first token. Lower is better
TTFT ms Time to the first token
Accept len Average tokens per target step (the first token you get for free counts as 1)
VRAM idle / peak GPU memory in MiB after the server is ready, and the highest value during the test
Status OK, OOM, START_FAILED, or BENCH_FAILED

Copy your numbers into a table like this one for your report. Run each test three times, and use the middle result:

Mode and setting Conc Out tok/s vs base TPOT ms Accept len VRAM peak MiB Status
Baseline 1 your result 1.00x your result n/a your result your result
EAGLE-3 3/1/4 1 your result your result your result your result your result your result
EAGLE-3 5/1/6 1 your result your result your result your result your result your result
Standalone 3/1/4 1 your result your result your result your result your result your result
NGRAM d12 1 your result your result your result your result your result your result

Step 11: SGLang Speculative Decoding Tuning Rules

The math is simple. A speculative step takes more time than a normal step, but it can give you more than one token. It only helps when the extra tokens are worth the extra time. To get the speedup, divide the baseline TPOT by the speculative TPOT.

Here is an example. The accept length is 2.5, and a speculative step takes 1.4 times as long as a normal step. The speedup is 2.5 ÷ 1.4 = 1.79, so it is about 1.8 times faster.

Now take a weak case. The accept length is only 1.4, and a step takes 1.3 times as long. The speedup is 1.4 ÷ 1.3 = 1.08, which is only about 8% faster. That gain is too small to matter.

Keep a speculative setting only if all of these are true in your table:

  1. Output tokens per second is at least 10% above the baseline at your real concurrency. Smaller gains are usually just noise.
  2. TPOT is lower than the baseline, and TTFT did not get worse than you can accept.
  3. The status is OK in all three runs, and the VRAM peak leaves at least 1 GiB free.
  4. The accept length is above about 1.5. A lower value rarely pays for the helper.

Use this table to pick what to test first:

Your case Try first Reason
One user or a few users, chat or coding EAGLE-3 Fast decoding at low load is where drafts help most
Model has built-in MTP layers MTP No extra model to load
Output repeats a lot (JSON, code, templates) NGRAM No extra model, and repeated text is easy to guess
No EAGLE-3 draft exists for your model Standalone or NGRAM You can use any small model with the same tokenizer
Busy server with high concurrency Baseline first The GPU is already busy, so extra checking can slow it down
Many different prompts and high temperature Measure before you decide Guesses are wrong more often, so accept length may drop

If your table shows a loss, that is a valid result. Turn speculative decoding off for that workload.

Step 12: Fix OOM Errors

Speculative decoding uses more GPU memory than normal decoding. The draft model, the draft tree, the CUDA graphs, and the checking buffers all need VRAM. Most failures in SGLang speculative decoding tuning come from a draft tree that is too big for the memory left over.

First, find which kind of error you have. Open the log of the failed run:

Bash
grep -iE 'out of memory|OutOfMemory|Not enough memory' /opt/sglang-spec/results/*/server.log | head
  • "Not enough memory. Please try to increase --mem-fraction-static" at startup: the cache pool is too small. Raise --mem-fraction-static a little.
  • CUDA out of memory during startup graph capture or while running: too much memory is reserved. Lower --mem-fraction-static.

--mem-fraction-static is the share of GPU memory for model weights and the KV cache. The rest is left for graphs and temporary work.

To see an OOM, run the large tree. It may fail on a 24 GB GPU:

Bash
cd /opt/sglang-spec/bin./run-test.sh q3-eagle3-aggr 8 $E3 \  --speculative-num-steps 5 \  --speculative-eagle-topk 8 \  --speculative-num-draft-tokens 64 \  --mem-fraction-static 0.88

Fix it in this order.  Change one thing at once and run the test again:

  1. Lower the static memory: --mem-fraction-static 0.70. This is the most effective fix.
  2. Reduce the CUDA graph size: --cuda-graph-max-bs-decode 4, or 2 when memory is very tight.
  3. Make the draft tree smaller: from 5 / 8 / 64 to 3 / 1 / 4 (steps, top-k, draft tokens).
  4. Limit running requests: --max-running-requests 4.
  5. If OOM happens during long prompts, lower --chunked-prefill-size to 4096 or 2048.
  6. Make sure no other process uses the GPU. Run nvidia-smi and check that the production sglang service is stopped.

Here is the quick recovery recipe. It uses the smallest setting with our model:

Bash
./run-test.sh q3-eagle3-safe 8 $E3 \  --speculative-num-steps 3 \  --speculative-eagle-topk 1 \  --speculative-num-draft-tokens 4 \  --cuda-graph-max-bs-decode 2 \  --mem-fraction-static 0.50 \  --max-running-requests 4

When this works, raise the settings one by one: draft tokens, then top-k, then the CUDA graph size. Raise --mem-fraction-static last, and only when the run is stable.

If the error is during CUDA graph capture, you can also remove --enable-torch-compile if you added it. As a last choice, add --disable-cuda-graph. It works, but speed drops.

Step 13: Apply the Best Settings to Your Production Server

Move a setup to production only if it passed the rules in Step 11. For SGLang speculative decoding tuning results to be valid in production, the model must be the same one you tested. The deploy guide serves Qwen3-4B-Instruct-2507 by default, which is a different model. Do not copy the Qwen3-8B draft model to it. Use a draft that was made for your exact target model, and run the whole test again.

Start the production service again after the tests:

Bash
systemctl start sglang

To add the winning setting to the deploy guide setup, first back up the files, using the same names as the rollback step in that guide:

Bash
cp -a /etc/sglang/server.yaml /etc/sglang/server.previous.yamlcp -a /usr/local/sbin/start-sglang /usr/local/sbin/start-sglang.previous

Download the draft model into a folder that you will mount into the container:

Bash
install -d -m 0700 /var/lib/sglang/draftIMAGE=$(cat /etc/sglang/image.txt)docker run --rm -i \  -e HF_HOME=/cache/huggingface \  -v /var/lib/sglang/draft:/draft \  -v /var/lib/sglang/cache:/cache \  --entrypoint python3 "$IMAGE" - <<'PY'from huggingface_hub import snapshot_downloadsnapshot_download(repo_id="thoughtworks/Qwen3-8B-Eagle3", local_dir="/draft")PY

Edit the start script and add one mount line after the /models mount:

Bash
nano /usr/local/sbin/start-sglang
Bash
  -v /var/lib/sglang/model:/models:ro \  -v /var/lib/sglang/draft:/draft:ro \

Edit the config file and add your winning values at the end:

Bash
nano /etc/sglang/server.yaml
Bash
speculative-algorithm: EAGLE3speculative-draft-model-path: /draftspeculative-num-steps: 3speculative-eagle-topk: 1speculative-num-draft-tokens: 4

Restart the service and check the logs:

Bash
systemctl restart sglangjournalctl -u sglang -f

When the server is ready, check that speculative decoding is active. The /server_info route needs the admin key, and /metrics stays on localhost:

Bash
source /root/sglang-client.envcurl -s -H "Authorization: Bearer $SGLANG_ADMIN_KEY" http://127.0.0.1:30000/server_info \  | python3 -m json.tool | grep -i speccurl -s http://127.0.0.1:30000/metrics | grep -E 'sglang:spec_accept_(rate|length)'

To roll back, restore the saved files and restart:

Bash
cp -a /etc/sglang/server.previous.yaml /etc/sglang/server.yamlcp -a /usr/local/sbin/start-sglang.previous /usr/local/sbin/start-sglangsystemctl restart sglang

Conclusion

Speculative decoding is not always faster, so always compare it with your baseline. Test EAGLE-3 first, then MTP, NGRAM, or a standalone helper if they fit your model. Keep a setup only if it gives better speed without OOM errors. This SGLang speculative decoding tuning method gives you results you can repeat on your own hardware.

You can use PerLod GPU servers to test high-throughput inference configurations on dedicated GPUs.

We hope you enjoy this guide.

Prev PostInstall Proxmox Backup Server 4.2 and Back Up a Proxmox VE Cluster
Next PostSet Up an Ephemeral GitHub Actions Self-Hosted Runner on a Linux VPS
Category:Performance & Scaling
Tag:

We use cookies to run PerLod and to understand how the site is used. You can accept everything, or open Manage to turn individual categories off. Cookie policy.