How to Run GPU Workloads on K3s Using NVIDIA GPU Operator 26.7

If you run AI or CUDA jobs on a real GPU server, you need Kubernetes to see the GPU and share it safely between pods. This guide shows you how to install NVIDIA GPU Operator 26.7 on K3s on a bare-metal server.

NVIDIA GPU Operator is a Kubernetes tool that automatically installs and manages everything a GPU needs inside your cluster. This includes the GPU driver, the NVIDIA Container Toolkit, the device plugin, GPU Feature Discovery, and the DCGM Exporter for monitoring. Instead of installing each of these manually on every server, the Operator does it for you as Kubernetes pods. 

K3s is a lightweight Kubernetes distribution. It is popular for bare-metal GPU servers because it is small, fast to install, and uses containerd as its container engine by default. 

K3s stores its containerd configuration files in a different folder than standard Kubernetes, so when you install NVIDIA GPU Operator 26.7 on K3s, you must tell the Operator where those K3s-specific files live. 

Before you install NVIDIA GPU Operator 26.7 on K3s, make sure you have:

• A bare-metal or dedicated server with an NVIDIA GPU.
• Ubuntu 24.04 or newer installed and updated.
• Root or sudo access over SSH.
• At least 8 GB RAM and 4 CPU cores free for the master node. If you run workloads on the same node, plan for more.
• Secure Boot turned off in BIOS, unless you plan to sign the NVIDIA kernel modules yourself. This avoids driver loading errors later.

If you need a ready GPU server instead of building one yourself, PerLod's GPU dedicated servers come with GPU hardware already racked and ready, so you can skip the hardware setup and go straight to this tutorial.

First, run the system update and reboot the server:

After the server restarts, log in again and confirm Linux can see your GPU card at the hardware level:

You should see a line like NVIDIA Corporation Device .... If nothing shows up, the GPU is not connected properly, or it is disabled in the BIOS. Fix that before continuing.

Here you have two choices. The default option is to let the GPU Operator install the driver inside a container automatically. Or, you can install the driver yourself on the host first and tell the Operator to skip driver installation. 

For bare-metal production servers, installing the driver on the host is more stable:

The last command lists the recommended driver package for your card. Install it with the command below; just replace the version number:

After reboot, check that the driver loaded correctly:

You should see a table with your GPU name, driver version, and memory. If this command fails, the driver did not install correctly, and you must fix it before moving to Kubernetes.

Now install K3s itself. This single command downloads and starts K3s as a systemd service:

Wait about a minute, then check that the node is ready:

You should see your node with a Ready status. To make kubectl easier to use without typing k3s kubectl every time, copy the config file and set an environment variable:

Now plain kubectl get nodes will work from your normal user account too.

The GPU Operator is installed using Helm, which is a package manager for Kubernetes. Install it with the official script:

Check that Helm installed correctly:

Add NVIDIA's official Helm chart repository and refresh it, so Helm knows where to find the GPU Operator chart:

Create a dedicated namespace and mark it as privileged, since GPU Operator pods need direct hardware access:

Because K3s stores its containerd config in a different location than standard Linux, we must pass extra --set options that point to the K3s paths. 

Since we already installed the driver on the host in Step 2, we set driver.enabled=false so the Operator does not try to install its own driver container on top of it:

This single command installs NVIDIA GPU Operator 26.7 on K3s with the correct settings for a bare-metal node. Helm will wait until all pods are running before returning your terminal prompt, which can take 3 to 6 minutes depending on your internet speed.

If you did not pre-install the driver in Step 2 and instead want the Operator to install it in a container, drop the driver.enabled=false line and simply run:

After you install NVIDIA GPU Operator 26.7 on K3s, do not skip verification. A pod stuck in CrashLoopBackOff here means your GPU workloads will silently fail later.

First, check that all pods in the gpu-operator namespace are running:

You are looking for output similar to this, where every pod says Running or Completed:

Next, check the ClusterPolicy object, which controls all Operator behavior. It must show ready:

Now confirm the GPU is registered as a schedulable resource in Kubernetes. This is the real proof that the Device Plugin is working correctly:

You should see your node name with a gpus count matching your physical cards, for example "gpus": "1". If this returns nothing, check the toolkit and driver pod logs with the command below to find the exact error:

Finally, check GPU Feature Discovery labels, which describe your GPU model and memory to Kubernetes as node labels:

DCGM Exporter collects live GPU health data, such as temperature, utilization, and memory usage, and exposes it as Prometheus metrics. Confirm it is running and reachable:

While that command runs, open a second terminal on the same server and pull the metrics:

If you see numbers coming back, DCGM Exporter is healthy, and you can later connect Prometheus or Grafana to this endpoint for long-term GPU monitoring.

Now that you have installed NVIDIA GPU Operator 26.7 on K3s and confirmed every component is healthy, it is time to prove the GPU actually works inside a container. Create a file to test it:

Paste this content into the file:

Apply the file to create the pod with the command below:

Watch it run, then read its output with the following commands:

A working setup prints a message ending in Test PASSED and Done. This means the pod successfully asked Kubernetes for a GPU, the Device Plugin gave it access, and CUDA ran real math on the physical card.

Clean up the test pod when you are done:

For a more realistic test, deploy a Jupyter Notebook container with GPU access, so you can interactively test PyTorch or TensorFlow code:

Apply it, then grab the access token from the logs:

Open http://<your-server-ip>:30001 in your browser and paste in the token shown in the logs. This gives you a real GPU-backed notebook running inside your K3s cluster, ready for real AI workloads.

A few issues come up on bare-metal K3s nodes, so check these first if something fails:

• Toolkit pod stuck pending or crashing: It means the CONTAINERD_CONFIG path is wrong. Double-check the K3s path exactly as shown in Step 6.
• nvidia-smi fails inside pods but works on the host: The container runtime is not using the NVIDIA runtime class yet. Restart K3s with sudo systemctl restart k3s after installing the Operator so containerd reloads its config.
• Driver pod crash loop: It is a Secure Boot issue. Disable Secure Boot in the BIOS or sign the kernel modules.
• ClusterPolicy stuck in notReady: Run kubectl describe clusterpolicy cluster-policy to see the exact blocking component.

Note: Testing on one server is fine for learning, but real AI workloads need strong GPU hardware, fast storage, and stable network. If you don't want to build your own GPU server, you can also try PerLod's AI hosting plans for fully managed AI infrastructure.

At this point, you know how to install NVIDIA GPU Operator 26.7 on K3s, starting from a clean Ubuntu server to a fully working and monitored GPU cluster. From here, you can add more GPU nodes, split GPUs between workloads with MIG, or connect Prometheus and Grafana to DCGM Exporter for full monitoring. 

We hope you enjoy this guide. For more detailed information, you can check the NVIDIA GPU Operator Official Documentation.