My laptop runner was too slow and not always there, so I added GitHub runners in Azure Container Apps (ACA): a Container Apps Job with the KEDA GitHub scaler starts one container per queued job, the container registers as an ephemeral runner, runs that one job and exits. The runners have the same labels as my laptop runner, so a workflow can run on either. Why I did it and which jobs go where is the previous part. This part is about what went wrong on the first runs, and why the image holds more tools than I expected: on ACA a job cannot install anything later.
The problems
No sudo
The first workflow I ran on ACA failed on the Playwright step:
Run npx playwright install --with-deps chromium
Switching to root user to install dependencies...
sudo: The "no new privileges" flag is set, which prevents sudo from running as root.
--with-deps installs system libraries with apt-get, through sudo. On my laptop the job container has sudo, so it worked there. A Container Apps container runs with "no new privileges", so sudo can never work. The same goes for the usual curl https://rclone.org/install.sh | sudo bash in another project's workflows.
The fix is in two places. The tools go into the image, and the workflow stops trying to install them when it cannot:
- name: Install Playwright browser
run: |
if sudo -n true 2>/dev/null; then
npx playwright install --with-deps chromium # laptop and GitHub-hosted
else
npx playwright install chromium # ACA: libraries are in the image
fi
For rclone the guard is command -v rclone >/dev/null || curl ... | sudo bash: hosted runners have no rclone and still install it, ACA finds it in the image.
No Docker
ACA does not allow privileged containers, so there is no Docker daemon inside a runner. A workflow that builds an image with the local Docker daemon has to stay on the laptop. The workflow that builds the runner image itself is one of them, so that one runs on the laptop.
The runner nobody needs
The scaler starts a container for every queued job it sees. If the laptop takes that job first, the container registers, finds nothing and waits. A runner has no idle timeout of its own, so it only ends when the job's replica timeout kills it, and you pay for 2 vCPU and 4 GiB while it waits.
Two settings fix the cost. The replica timeout (the longest a runner may live) is the ceiling for a job that is running, not what is billed: a runner exits when its job is done, so a 15-minute job is billed for 15 minutes. I set it to 45 minutes. And I added an idle exit to the container's entrypoint: a runner that has not started a job after 5 minutes stops and ends successfully. GitHub gives a queued job to an idle runner within seconds, so by then the job went elsewhere. I chose 5 minutes instead of a few seconds on purpose: a job that is queued a little later in the same workflow (behind a needs:) can still find a warm runner.
The token
A runner needs a token that may register runners, and the scaler needs one to read the queue. It is a fine-grained token for my repositories, stored as an ACA secret and never in the image. The entrypoint uses it once, to fetch the just-in-time configuration, and then removes it from the environment before the runner starts. The steps of a job inherit the runner's environment, so otherwise any step could read it. It is still a secret of the job in ACA, which is one more reason this is only for private repositories: anyone who can run a workflow in a repository can run code on its self-hosted runners.
What goes into the image, and why
On ACA I cannot install anything later, so whatever a job needs has to be in the image at build time. That is a longer list than I expected:
| In the image | Why |
|---|---|
| git, curl, jq, zip, unzip, xz | What ubuntu-latest has; xz unpacks the Node tarball |
| GitHub CLI | Scripts and workflows call gh |
| Azure CLI | azure/login and the az steps |
| rclone | The image backup workflows; its installer needs sudo |
| Node 22 and wrangler | Site builds and Cloudflare deploys |
| The same Node in the runner's tool cache | actions/setup-node finds it there instead of downloading it, which took 30 to 40 seconds per job |
| Chromium and its system libraries | Playwright checks; PLAYWRIGHT_BROWSERS_PATH points to a folder the runner user can write to, so a project that wants another Chromium revision downloads it there without root |
| .NET SDK (optional) | For the .NET projects, off by default to keep the image small |
| The actions the workflows use, cached | Every job is a new container, and the runner gives up on a slow download after 100 seconds |
| procps | pgrep, for the idle exit |
Each tool is a build argument, so a repository that does not need Chromium does not pay for it in image size, and a smaller image starts faster. Starting is the part to watch: a job on ACA takes roughly 30 to 60 seconds longer to begin than on the laptop (the scaler's check, the container start, the registration).
Lessons learned
- Decide what a job may install at runtime. On a runner without sudo, the answer is nothing, so put the tools in the image.
- Make the workflow steps that install things check first (
sudo -n true,command -v), so the same workflow runs on any kind of runner. - Set a replica timeout longer than your longest job, and an idle exit so a runner that lost the race does not sit there.
- Keep the workflows that need Docker on a runner that has it.
- Private repositories only.