bpftrace Troubleshooting: Find Slow CPU, Disk, Syscalls, and Network

In this guide, you will learn a simple path for Linux performance troubleshooting with bpftrace. You can start with top, iostat, and ss, then move to short bpftrace scripts that show the exact cause. Also, you will learn how to set safe limits and how to remove probes cleanly.

You need a Linux server with root or sudo access, a modern kernel, and a test window. Tracing at kernel level needs full control of the machine, so dedicated server hosting is a good option. Many shared or managed platforms block this kind of access.

First, you must check your kernel and BTF data. BTF is a small data block in the kernel that lets bpftrace read kernel structures:

If the second command shows the file, you are ready. Then, install bpftrace with the commands below based on your distro:

Check the version and what your kernel supports:

Try a quick test with the command below, which prints one line and exits:

Also, you need need these common tools. On Ubuntu or Debian, sysstat gives you iostat and pidstat:

Note: bpftrace needs root, or the CAP_BPF and CAP_PERFMON capabilities. Newer versions may ask for a few more capabilities, so using sudo is the easiest way.

Remember to do not start with bpftrace. Start with simple tools, find which area is slow, and then trace only that area. This keeps the load low and the answer clear. This fits well with your other troubleshooting and monitoring tools.

Start with a quick look at the whole server. Use top and vmstat to see if the problem is CPU, memory, swap, or disk wait. This takes less than a minute and shows you where to look next.

In the outout, look for these values:

• %us and %sy: time in user code and in the kernel.
• %wa: time the CPU waits for disk I/O.
• %st: time stolen by the host.
• r in vmstat: tasks waiting to run. b: tasks blocked in uninterruptible sleep, often waiting for disk.
• si and so: swap in and swap out.

Next, you must check the disk. Use iostat to see how long reads and writes wait. If the wait time is high, the disk may be the reason for the slowdown.

Look at r_await and w_await (average wait time in ms), aqu-sz (average queue size), and %util. On NVMe drives, %util can be unclear, because these drives handle many requests at the same time. Trust the wait times more.

Now you must check the network. Use ss and nstat to see open connections, retransmits, and delays. If you see many retransmits, the network path may be the problem.

ss -ti shows per-connection details such as retrans and rtt. nstat shows system-wide counters.

At this point, you can check each process on its own. Use pidstat to see which process uses the most CPU or disk. This helps you find the exact process behind the slowdown.

Now it is time to use bpftrace. The tools above show what is slow, but not why. Use the table below to match your symptom to the right script, then run it for a short time. This is the main step in Linux performance troubleshooting with bpftrace.

bpftrace is safe by design. The kernel checks every eBPF program before it runs, so a bad script cannot crash the kernel. But a script can still slow down a busy server if it traces very frequent events, like every context switch or every packet. You can follow these rules:

1. Run short. Use timeout 30 or add interval:s:30 { exit(); } to the script.
2. Filter early. Use -p PID, a comm check, or a predicate like /pid == 1234/.
3. Print less. Use maps (count(), hist(), sum()) instead of printf on every event. Print only slow events.
4. Avoid wide wildcards. kprobe:* tries to attach to a huge number of functions. Count first with bpftrace -l 'kprobe:vfs_*' | wc -l.
5. Test first on staging, or on a quiet moment on production.
6. Never use --unsafe unless you know why. It allows calls like system(), which can run commands as root.

Before you run a script, check that the probe exists on your server. Use these commands to list the probes and see the fields you can read:

The -v flag shows the fields you can read through args. Tracepoints are stable between kernel versions, but fields can change a little, so always check on your own kernel.

bpftrace has limits for the number of probes, programs, and map keys. The defaults depend on the version. For example, 4096 map keys and 1024 probes in recent versions, 512 probes in older ones. You can change them with environment variables:

On older versions, the variable is named BPFTRACE_MAP_KEYS_MAX.

A low BPFTRACE_MAX_PROBES value protects you if a wildcard matches too many probes. Raise the map key limit only if your script stores one item per thread or per request and the map gets full.

To check a script before you trace anything, you can use a dry run. It loads the script and shows errors, but it does not collect any data:

You can use this template for most of your runs. It is short, safe, and easy to reuse. It stops the script after a fixed time and saves the output to a file, so you can read it later:

The symptoms of this problem include: The app is slow, but CPU use is low. Load average is high. Requests wait, and nothing seems busy.

When a thread is not running on the CPU, it is off-CPU. It may wait for disk, a lock, a timer, or the network. If CPU time does not explain the delay, off-CPU time usually does. It is the first script to try in Linux performance troubleshooting with bpftrace.

The sched:sched_switch tracepoint runs when the CPU switches from one thread to another. The script saves a time when a thread goes to sleep, and adds up the sleep time when it comes back.

Use your desired text editor to create the script file:

Add the follwoing content to the file:

Once you are done, press Ctrl+O, Enter, and Ctrl+X to save and exit. Then, run the script with:

The output is sorted from small to large, so you can look at the last lines. Idle workers, like a thread pool that waits for work, will also show large numbers. This is normal. Compare the result with a normal time, or filter by the process you care about. The time is added across all threads, so the total can be bigger than the real clock time.

Now you know which process waits too much. Next, you must find out what it waits for. This command shows the kernel stack where its threads go to sleep, so you can see if it waits for the disk, a lock, or the network:

Here are some common stack names you may see:

• io_schedule: waiting for the disk.
• futex_wait: waiting for a lock.
• do_epoll_wait or poll: waiting for network events.
• jbd2_log_wait_commit: waiting for the file system journal to save data.

The top lines of the stack show what the thread waits for.

Note: sched_switch can run hundreds of thousands of times per second on a busy server. Keep the run short.

The symptom include: iostat shows high await. The database is slow, %wa is high, and vmstat shows tasks in the b column.

iostat gives averages. A few very slow requests can hide inside a good average. This script measures each request and prints only the slow ones, which makes Linux performance troubleshooting with bpftrace very useful for disk issues. After you find the slow device, read the NVMe optimization guide to tune your storage.

The block:block_rq_issue tracepoint runs when a request goes to the device. The block:block_rq_complete tracepoint runs when the request is done. The time between them is the device time. Create the script file:

Add the follwoing content to the file:

Save and exit. Then, run it and show requests slower than 10 ms:

Next, match the device numbers (major:minor) to a disk name:

Here is how to read the results:

• @lat_us: shows request times in microseconds. A long right tail means some requests are slow.
• op=: R means read, W means write, WS means sync write, and F means flush. Look for slow sync writes and flushes.
• issuer=: shows who sent the I/O, which may be a kernel worker, not your app. Treat it as a hint, not proof.
• Slow reads on one disk may point to drive trouble. Slow writes during backups may point to a busy queue.

Note: This measures time from issue to completion, not time spent waiting in the kernel queue. LVM and software RAID may show requests differently. Trace the physical disk, such as nvme0n1, for clearer results.

The symptoms include: An app freezes for a moment. Logs show random pauses. A database waits on fsync, a web server waits on open, or a client waits on connect.

A syscall is a request from a program to the kernel. If it takes too long, the program has to wait. This script measures how long common syscalls take and prints the slow ones. Create the script file with:

Add the follwoing content to the file:

Save and close the file. Then, run it for one process. The -p flag acts as a filter, so only that process is traced:

Or, you can trace the whole system with the command below:

The name in the output is the enter probe, for example, tracepoint:syscalls:sys_enter_fsync. The last part is the syscall name.

• Slow fsync or fdatasync usually means slow disk writes. Go back to Script 2.
• Slow openat can mean a slow network file system or a busy disk.
• Slow connect points to the network. Go to Script 4.
• Slow read and write on sockets or pipes can be normal, because the program may wait for data. Look at the file descriptor and the process role before you decide it is a problem.

Note: If you do not know which syscall to check, you can count them by process and syscall number:

You can translate a number into a name with ausyscall. It comes from the auditd package on Ubuntu and Debian:

The symptoms here include: Some API calls time out. Connections may take 1 or 3 seconds to open, and ss -ti shows more retransmits. If you see these signs, you can also read the Linux kernel network tuning guide for settings to check next.

A retransmit means the kernel sends TCP data again because it did not get a reply in time. A few are normal. Many may mean lost packets, a bad connection, or a busy remote server. Linux performance troubleshooting with bpftrace helps you find which remote address is affected.

You can use the tcp:tcp_retransmit_skb tracepoint, which gives the remote address and port. Create the script file with:

Add the follwoing content to the file:

Once you are done, save and close the file. Then, run the script:

Important Note: Count by remote address and port, not process name. Retransmits usually run from a kernel timer, so comm may show the wrong process. This script uses IPv4. For IPv6, use daddr_v6.

Also, check the system counters with the commands below:

If most retransmits go to one IP, check that server and its network path. If they affect all remote servers, check your network card, driver, and queue settings.

This script measures the TCP handshake, from the first SYN until the connection opens. It uses sock:inet_sock_set_state to track TCP state changes.

Unlike a kretprobe on tcp_v4_connect, it includes the wait for a reply, not just the time needed to send the SYN. Craete the script file with:

Add the follwoing content to the file:

Save and close the file. Then, run it and show connects slower than 100 ms:

The numbers used in the script are TCP state numbers: 1 is ESTABLISHED, 2 is SYN_SENT, and 6 is the TCP protocol number. Here is how to read the results:

• Delays near 1 second may mean a lost SYN. Around 3 seconds may mean another SYN was lost.
• @failed counts connections that left SYN_SENT without connecting, such as refusals or timeouts.
• The script saves the process name when sending the SYN. When the reply arrives, comm may no longer show your app.

In this case, the top or pidstat -u shows high CPU use by one process. You want to find which part of its code uses the CPU. Linux performance troubleshooting with bpftrace helps you find it without restarting the app or changing its code.

You can take short samples while the process runs to see which parts of its code use the most CPU. The script below collects these samples for 20 seconds. Create the script file with:

Save and close the file. Run it with the PID of the busy process:

Stacks with the highest counts show your hot paths. Kernel stacks show kernel work; user stacks show app code. If user stacks show only addresses or look incomplete, the app may lack symbols or frame pointers. You can use a build with these enabled, or try perf with another stack-reading method.

You can count the process’s syscalls to see which ones it uses most. The command below collects counts for 10 seconds:

At this point, you can see which processes make the most file operations. The command below counts reads, writes, opens, and syncs for 10 seconds:

A process with a huge vfs_fsync count, or many vfs_open calls, is a good suspect for slow disk response and high system CPU time.

Note: kprobes attach to kernel function names, and these names can change between kernel versions. If a probe is missing, check with bpftrace -l 'kprobe:vfs_*'.

bpftrace probes live only as long as the bpftrace process runs. When it exits, the kernel removes the probes and the maps. You do not need to unload anything manually. To stop a script correctly, follow these steps:

1. In the terminal, press Ctrl-C. bpftrace runs the END block, prints the maps, and detaches the probes.
2. For a background process, send SIGINT: sudo pkill -INT bpftrace.
3. If it does not stop, send SIGTERM: sudo pkill -TERM bpftrace.
4. Use kill -9 only as the last step. The kernel still cleans up the probes when the process dies, but you lose the output.

Verify that nothing is left with the commands below:

Compare the bpftool lists before and after tracing. They should match. The *_events files should be empty unless another tool uses them. If bpftool is missing, install it with your package manager. On Ubuntu, install linux-tools-common and the tools package for your kernel.

Also, check that the load is back to normal with the command below:

Tip: Keep scripts in /opt/bpftrace/ and run them with timeout. This makes Linux performance troubleshooting with bpftrace safer and easier for your team to repeat.

Linux performance troubleshooting with bpftrace starts with simple tools. Use top, vmstat, iostat, and ss to find where to look. Then, use the scripts to check waiting threads, slow disk requests, slow syscalls, TCP issues, and code that uses the most CPU.

Keep runs short, filter what you trace, and limit output. Check that probes are gone when you finish. Save your scripts so your team can reuse them next time.