//------------------------------------------------------------------- //-------------------------------------------------------------------
Fix Container CPU Throttling and Latency in Docker and Kubernetes

How to Fix Container CPU Throttling and Latency in Docker and Kubernetes

Container CPU throttling is a performance issue where the Linux kernel is forced to pause a container’s processes because they have consumed their assigned CPU quota for a specific time window, even if the host CPU is idle. This causes silent latency spikes where applications become unresponsive for milliseconds at a time without crashing.

In this guide, we want to detect if the Linux kernel is pausing your containers, explain the causes related to CFS quotas, and optimize the infrastructure with the best fixes. Whether you’re running locally or tuning high-performance workloads on Perlod Hosting, these fixes are essential for low latency and stability.

How to Detect CPU Throttling in Docker Containers and Kubernetes?

Standard tools like top or Docker stats are too slow to check the container CPU throttling. They show you a smooth average over a few seconds, which hides the tiny and millisecond pauses that are actually happening. To check these pauses, you must look for the Linux Control Group (cgroup) metrics directly.

Find the Docker Container ID

The first step is to find the specific location where Linux stores your container’s data. This requires the full Container ID because the short IDs shown in standard summaries will not work for filesystem paths.

List containers to find the ID with the command below:

docker ps

Get the full ID with the following command, which is required for file paths:

docker inspect --format="{{.Id}}" <container_name_or_short_id>

This command gives you the full 64-character hexadecimal ID of the container.

Example Output:
78d65261394c8672013f97204981e74163972236780371900138981440183121

Locate and Check cgroup Statistics

Once you have the Container ID, you need to find the specific kernel file where throttling data is recorded. This file, named cpu.stat, is located in a different directory depending on whether your Linux system is running the older cgroup v1 or the modern cgroup v2 architecture.

Use the path that matches your system version:

For cgroup v1 on older systems, the path is typically located under the CPU controller.

Syntax: /sys/fs/cgroup/cpu/docker/<long_container_id>/cpu.stat

To identify it, you can use the command below:

cat /sys/fs/cgroup/cpu/docker/78d6.../cpu.stat
Example Output:
nr_periods 1583
nr_throttled 24
throttled_time 2038472948

For cgroup v2 on modern Linux systems, the path is usually located under the system slice.

Syntax: /sys/fs/cgroup/system.slice/docker-<long_container_id>.scope/cpu.stat

To identify it, you can use the command below:

cat /sys/fs/cgroup/system.slice/docker-78d6....scope/cpu.stat
Example Output:
usage_usec 429548
user_usec 218394
system_usec 211154
nr_periods 1583
nr_throttled 24
throttled_usec 2038472

Analyze the Throttling Metrics from cgroup

Once you have identified the throttling metrics, you must focus on these two lines from the output:

1. nr_throttled: This is your primary throttling counter. It counts the total number of times the system has forced your container to stop because it used up all its allowed CPU time.

A value of 0 for this metric is perfect. If you see a non-zero number, for example, 24, t means throttling has occurred in the past. If you run the command again a few seconds later and this number has increased, your container is actively being throttled.

2. throttled_time or throttled_usec: This measures the total duration your container was frozen while waiting for the next CPU period.

This is basically time wasted waiting. The delay is caused by the system limits, not your app being slow. A high number here means your users are waiting longer for no reason.

Monitor CPU Throttling in Kubernetes

If you are running Kubernetes, you can not easily SSH into nodes to check files and identify container CPU throttling. You must use the built-in metrics provided by cAdvisor and Prometheus.

To detect throttling across your pods, run the following PromQL query in your monitoring dashboard:

rate(container_cpu_cfs_throttled_seconds_total[5m])

A positive rate represents active container CPU throttling.

What Causes Container CPU Throttling?

To solve throttling, you first need to understand the mechanism behind it. The root cause of throttling is the Completely Fair Scheduler (CFS) Quota mechanism used by Linux to enforce CPU limits. It does not act like a speed limit that slows your process down; it acts like a traffic light.

The CFS Period and Quota: Linux doesn’t just look at your CPU usage over a whole hour; it breaks time down into tiny chunks called Periods, which usually last for 100 milliseconds.

  • Quota: This is how much of that 100ms period your container is allowed to use.
  • Limit Logic: If you set a limit of 0.5 CPUs, you get 50% of the time. That means for every 100ms cycle, your container can run for 50ms. Once you use that up, you are cut off until the next cycle starts.

The Micro-Burst Problem: This system causes problems because real-world apps don’t run smoothly; they burst. Imagine a web request comes in that needs 20ms of power to finish quickly:

  • You have a low limit of 0.1 CPU. This gives you only 10ms of runtime per cycle.
  • Your app works hard for 10ms, but it needs 10ms more to finish the job.
  • The Throttle: The kernel pauses the container for the remaining 90ms of the period.
  • The request finally finishes in the next cycle. The user waited over 100ms for a task that should have taken only 20ms.

This mismatch explains why a container showing only 10% CPU usage in Docker stats can still be heavily throttled.

How to Fix Container CPU Throttling?

To fix CPU throttling, you usually need to change your CPU limits or remove them so the container can handle short CPU spikes without being forced to pause.

1. Remove CPU limits, which is recommended for latency:

The most effective solution for latency-sensitive applications like web servers is to remove the CPU limit completely.

For Docker, do not pass the –cpus or –cpu-quota flags:

# Correct (No limits):
docker run -d --name my-app my-image

# Avoid these flags:
# --cpus="0.5"
# --cpu-quota=50000

For Kubernetes, remove the limits.cpu field from your Pod spec and keeps only requests.cpu:

resources:
  requests:
    cpu: "500m"  # Keep this (0.5 CPU guaranteed)
  limits:
    memory: "512Mi" # Keep memory limits if needed
    # cpu: "1"   <-- DELETE THIS LINE (Remove CPU limit)

When you remove the hard limit, the system stops acting like a strict traffic light. Instead, it only restricts your container if the server is full and other apps need space. As long as there is spare power available on your VPS hosting server, your app can use it to run as fast as possible, which handles sudden bursts of traffic instantly without being paused.

2. Increase CPU limits: If you can’t remove the limit entirely, for example, to stop one broken app from slowing down the entire server, the next best solution is to increase CPU limits.

Set the CPU limit much higher than you think you need, usually 2 to 4 times higher than your average usage. This gives the app enough resources to handle sudden spikes without hitting the limit. You might end up with resources that are rarely used, which can be inefficient, but it keeps your app fast.

3. Tune the CFS Period: If you cannot raise limits but need faster refills of quota, you can decrease the CFS period.

For Docker, you can use the –cpu-period flag. Reducing this from 100ms to 10ms means the quota resets 10 times faster, reducing maximum pause duration:

# Example: 0.5 CPU limit (50% usage)
# Standard: period=100000 (100ms), quota=50000 (50ms)
# Tuned:    period=10000 (10ms),   quota=5000 (5ms)

docker run -d \
  --cpu-period=10000 \
  --cpu-quota=5000 \
  my-container

Note: If you only change –cpu-period without adjusting –cpu-quota, you will change the total CPU limit. Always scale both values together to keep the same CPU percentage.

Warning: This makes the system work harder because it has to check limits 10 times more often. Also, most Kubernetes clusters don’t let you change this setting easily; you would need deep administrative access to the server configuration.

FAQs

Can I get CPU throttling even if my CPU usage is low?

Yes. Throttling happens during millisecond bursts, not long-term averages.

Is it safe to remove CPU limits in Kubernetes?

Yes. Removing limits.cpu improves latency by allowing bursts into idle resources. Just ensure requests.cpu is set correctly so the scheduler can still place pods effectively.

Does the Docker stats command show CPU throttling?

No. The Docker stats command only shows usage percentage, not throttling. To see if your container is being paused, you must check the cgroup files or use a monitoring tool like Prometheus.

Conclusion

Container CPU throttling is a silent performance killer that often goes unnoticed by standard monitoring tools. By understanding the CFS quota mechanism and checking the correct cgroup metrics, you can identify why your applications suffer from latency spikes and apply the best fixes.

We hope you enjoy this guide. Subscribe to our X and Facebook channels to get the latest updates and articles.

For further reading:

VPS Performance and Latency Optimization

VPS Performance Testing JMeter Tutorial

Linux Kernel Tuning for Maximum Network Performance

Post Your Comment

PerLod delivers high-performance hosting with real-time support and unmatched reliability.

Contact us

Payment methods

payment gateway
Perlod Logo
Privacy Overview

This website uses cookies so that we can provide you with the best user experience possible. Cookie information is stored in your browser and performs functions such as recognising you when you return to our website and helping our team to understand which sections of the website you find most interesting and useful.