How to Deploy GitHub Actions Runner Controller Scale Sets on K3s

Running CI jobs on your own servers is cheap and private, but only if the runners scale on their own. Actions Runner Controller (ARC) does this for you. It creates a fresh runner pod for each job and deletes it when the job ends.

In this guide, you will learn how to run ARC scale sets on k3s. You will install ARC on an existing K3s cluster, deploy an ephemeral runner scale set with Helm, and set up GitHub authentication. 

ARC is a Kubernetes operator from GitHub. It runs a controller and a listener. The listener connects to GitHub and watches for queued jobs for your runner scale set. When jobs arrive, the controller creates runner pods. Each runner pod takes one job and is removed after that job ends. That is why the runners are called ephemeral.

Tips: If you only have one server and do not use Kubernetes, you can build a simpler one-job runner with systemd and Docker. You can check this guide on how to set up an ephemeral GitHub Actions self-hosted runner.

If you want automatic scaling on K3s, you must proceed to the rest of this article.

Here is how ARC scale sets on K3s work:

• A workflow job starts and uses runs-on: <scale-set-name>.
• GitHub puts the job in the queue for that scale set.
• The listener pod sees the job and tells the controller how many runners are needed.
• The controller creates runner pods in the runner namespace.
• Each pod runs one job and is deleted.
• When the queue is empty, the pods scale down to your minimum number.

You must use two namespaces. The controller and listener live in arc-systems. The runner pods live in arc-runners. It is recommended to keep runner pods in a different namespace than the controller, because workflows run any code a user writes.

Before you start with ARC scale sets on k3s, make sure you have:

• A working K3s cluster and shell access to a server node. 
• kubectl and Helm 3 on that server.
• A GitHub repository or organization where you can add runners.
• Outbound HTTPS access from the cluster to GitHub. ARC does not need any inbound ports.
• A default storage class. K3s ships with the local-path storage class, which is enough for this guide.

First, you must check the cluster. K3s saves its kubeconfig at /etc/rancher/k3s/k3s.yaml. Helm needs this path, so you must export it with the commands below:

All nodes should show Ready, and you should see local-path (default) in the storage class list.

Also, you must install Helm if it is not installed yet:

The controller is the first part of ARC scale sets on k3s, and it goes in the arc-systems namespace. We use a fixed chart version, so your install matches this guide. The latest version right now is 0.15.0. Use the same version for both the controller chart and the scale set chart:

Check that the controller pod is running:

You should see one controller pod with 1/1 Running. Also, the ARC install adds its custom resource definitions (CRDs) to the cluster.

ARC needs a credential to register runners with GitHub. You have two choices, including a GitHub App, which is best for organizations, or a personal access token (PAT).

First, you must create the runner namespace with the command below:

In this option, you must create a GitHub App that is owned by your organization. Use these settings:

• Homepage URL: https://github.com/actions/actions-runner-controller
• Repository permissions: Metadata read-only. Add Administration read and write only if you register runners at the repository level.
• Organization permissions: Self-hosted runners read and write.

Then, you must generate a private key (.pem file), install the app on your organization, and note the App ID and the Installation ID. You can find the Installation ID in the URL of the app installation page:

Then, create the Kubernetes secret in the arc-runners namespace:

Replace the IDs with your own values. You cannot use a GitHub App for enterprise-level runners.

With this option, you can create a classic PAT with the repo scope for repository runners, or the admin:org scope for organization runners. Then, create the secret with:

Both options use the same secret name, which is arc-github-secret. Always pass a secret name to Helm. Do not paste tokens in the helm install command, because they can leak in your shell history.

At this point, you must create a values file for the runner scale set. This keeps your settings for ARC scale sets on k3s in one place. Create the file with your desired text editor:

Add the following content to the file:

Once you are done, save and exit. What each line does:

• githubConfigUrl: The repository, organization, or enterprise that owns the runners. For one repository, use https://github.com/YOUR_ORG/YOUR_REPO.
• githubConfigSecret: The name of the secret from Step 2.
• runnerScaleSetName: The name you will put in runs-on. If you leave it empty, ARC uses the Helm release name.
• minRunners and maxRunners: The scaling range. We explain these in Step 4.
• containerMode: How the runner starts containers. We explain this in Step 5.

Then, install the scale set in the arc-runners namespace:

Verify the installation with the commands below:

You should see two Helm releases, arc and k3s-runners, both with status deployed. In arc-systems, you should see the controller pod and a new listener pod whose name ends in -listener. 

In GitHub, go to your organization or repository, Settings > Actions > Runners. The new scale set should be listed there.

The minRunners and maxRunners values set the smallest and largest number of runners. ARC adds minRunners and the number of assigned jobs to get the runner count. It never goes above maxRunners.

Here are the example settings and what each one does. It helps you pick the right values for your needs:

Keep these rules in mind when you tune ARC scale sets on k3s:

• minRunners can never be higher than maxRunners, unless you comment out maxRunners.
• ARC has no built-in schedule for min and max values. If you want different values at night, change the values with a cron job and helm upgrade.
• Always set a maxRunners value on a small K3s cluster. This stops a busy queue from using all CPU and memory on your nodes.

To change the values later, you can edit the values file and run:

ARC scales runner pods based on queued jobs, not on CPU or memory. Also, if you want to scale your own apps by events or metrics on the same cluster, check this guide on KEDA autoscaling on K3s.

You need a container mode only if your jobs use container jobs, service containers, or container actions, or if they run docker build. ARC gives you two options. Set one of them in containerMode.type.

In dind (Docker-in-Docker) mode, each runner pod gets a Docker daemon sidecar. Your workflow can run docker build and docker run like on a normal server. This is the easiest mode, and it is the one we used in Step 3.

The trade-off is that the dind container must run in privileged mode. Use it only for trusted private repositories.

In kubernetes mode, there is no Docker daemon. When a job uses a container, ARC creates a new pod in the same namespace to run it. The runner pod and the job pod share a work volume, so you need a storage class that can create volumes on demand. K3s already has local-path for this.

To switch, you must replace the containerMode block in arc-runner-values.yaml:

Then, run the same helm upgrade command from Step 4.

In Kubernetes mode, every job must use a container: value. A job without one fails with this error: Jobs without a job container are forbidden on this runner.

To allow jobs without a container, you can set the ACTIONS_RUNNER_REQUIRE_JOB_CONTAINER environment variable to "false" on the runner container in the template section of the values file.

Note that local-path volumes live on one node's disk. This is fine for a single node or a test. On a multi-node cluster, you must use a shared storage system such as Longhorn.

Here is a quick comparison between two modes:

Monitoring helps you see how ARC scale sets on k3s are working. ARC exposes Prometheus metrics from the controller and the listener. You need to turn them on in the controller values and the scale set values.

First, create the controller values file with the command below:

Add the following content to the file:

Once you are done, save and exit. Then, upgrade the controller:

Next, you must tell the listener which metrics to publish. If the controller publishes metrics but the listenerMetrics field is empty, the listener can fail to start on some chart versions. Create the file with:

Add the following content to the file:

Once you are done, save and exit. If you leave buckets empty, ARC uses its default histogram buckets. Upgrade the scale set with both values files:

Wait until the new listener pod is Running, then read the metrics with a port-forward:

Stop the port-forward when you finish with kill %1. These are the most useful metrics:

Point your Prometheus at port 8080 of the controller and listener pods to build dashboards and alerts. Listener counters reset when the listener pod restarts.

Now you must test that the scale set scales up and down. This test starts 8 jobs at once, but maxRunners is 5. You should see 5 runner pods first, and the other 3 jobs wait in the queue.

Create the test file inside your GitHub repository. The repository must be able to use your runner scale set:

Paste the following content to the file:

The runs-on value must match runnerScaleSetName exactly. In this guide, that is k3s-runners.

Open two terminals on your K3s server. In the first one, watch the runner pods:

In the second one, watch the scale set status:

Now go to the Actions tab in GitHub, choose ARC Autoscaling Test, and click Run workflow. You should see this order of events:

1. Before the run, there are no runner pods, because minRunners is 0.
2. A few seconds after the run starts, up to 5 runner pods appear and go to Running.
3. The other 3 jobs remain queued on GitHub until a pod becomes available.
4. As jobs finish, each pod is deleted, and new pods start for the queued jobs.
5. After all 8 jobs finish, the pod list is empty again.

While the test runs, check the metrics from Step 6. The gha_desired_runners value should go up to 5. The gha_busy_runners value should match the number of running pods. When the test ends, both values should go back to 0.

If you use dind mode, you can also test Docker in a second job:

If you use kubernetes mode, you can add a container to the job instead:

If no pods appear, check the listener logs and the controller logs with:

The most common causes are a wrong githubConfigUrl, a secret with wrong permissions, or a runs-on name that does not match the scale set name.

At this point, you have ARC scale sets on k3s running with separate controller and runner namespaces, secure GitHub authentication, and clear min and max limits. Also, you picked a job mode, enabled metrics, and proved with a real test that runners appear when jobs arrive and disappear when they end.

For the best results, it is recommended to run it on strong hardware. You can run ARC on a reliable dedicated server to get elastic private CI capacity with full control over your runners.