ci: ignore .md file changes everywhere

* docs: adding docs for the chart values

* docs: updating the main docs

* docs: grammar fixes

* docs: updating proxy default

Co-authored-by: Callum James Tait <callum.tait@photobox.com>

.dockerignore

@@ -0,0 +1,12 @@
+Makefile
+acceptance
+runner
+hack
+test-assets
+config
+charts
+.github
+.envrc
+*.md
+*.txt
+*.sh

.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,36 @@
+---
+name: Bug report
+about: Create a report to help us improve
+title: ''
+assignees: ''
+
+---
+
+**Describe the bug**
+A clear and concise description of what the bug is.
+
+**Checks**
+
+- [ ] My actions-runner-controller version (v0.x.y) does support the feature
+- [ ] I'm using an unreleased version of the controller I built from HEAD of the default branch
+
+**To Reproduce**
+Steps to reproduce the behavior:
+1. Go to '...'
+2. Click on '....'
+3. Scroll down to '....'
+4. See error
+
+**Expected behavior**
+A clear and concise description of what you expected to happen.
+
+**Screenshots**
+If applicable, add screenshots to help explain your problem.
+
+**Environment (please complete the following information):**
+ - Controller Version [e.g. 0.18.2]
+ - Deployment Method [e.g. Helm and Kustomize ]
+ - Helm Chart Version [e.g. 0.11.0, if applicable]
+
+**Additional context**
+Add any other context about the problem here.

.github/ISSUE_TEMPLATE/feature_request.md

@@ -0,0 +1,19 @@
+---
+name: Feature request
+about: Suggest an idea for this project
+title: ''
+assignees: ''
+
+---
+
+**Is your feature request related to a problem? Please describe.**
+A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
+
+**Describe the solution you'd like**
+A clear and concise description of what you want to happen.
+
+**Describe alternatives you've considered**
+A clear and concise description of any alternative solutions or features you've considered.
+
+**Additional context**
+Add any other context or screenshots about the feature request here.

.github/RELEASE_NOTE_TEMPLATE.md

@@ -0,0 +1,34 @@
+# Release Note Template
+
+This is the template of actions-runner-controller's release notes.
+
+Whenever a new release is made, I start by manually copy-pasting this template onto the GitHub UI for creating the release.
+
+I then walk-through all the changes, take sometime to think abount best one-sentence explanations to tell the users about changes, write it all,
+and click the publish button.
+
+If you think you can improve future release notes in any way, please do submit a pull request to change the template below.
+
+Note that even though it looks like a Go template, I don't use any templating to generate the changelog.
+It's just that I'm used to reading and intepreting Go template by myself, not a computer program :)
+
+**Title**:
+
+```
+v{{ .Version }}: {{ .TitlesOfImportantChanges }}
+```
+
+**Body**:
+
+```
+**CAUTION:** If you're using the Helm chart, beware to review changes to CRDs and do manually upgrade CRDs! Helm installs CRDs only on installing a chart. It doesn't automatically upgrade CRDs. Otherwise you end up with troubles like #427, #467, and #468. Please refer to the [UPGRADING](charts/actions-runner-controller/docs/UPGRADING.md) docs for the latest process.
+
+This release includes the following changes from contributors. Thank you!
+
+- @{{ .GitHubUser }} fixed {{ .Feature }} to not break when ... (#{{ .PullRequestNumber }})
+- @{{ .GitHubUser }} enhanced {{ .Feature }} to ... (#{{ .PullRequestNumber }})
+- @{{ .GitHubUser }} added {{ .Feature }} for ... (#{{ .PullRequestNumber }})
+- @{{ .GitHubUser }} fixed {{ .Topic }} in the documentation so that ... (#{{ .PullRequestNumber }})
+- @{{ .GitHubUser }} added {{ .Topic }} to the documentation (#{{ .PullRequestNumber }})
+- @{{ .GitHubUser }} improved the documentation about {{ .Topic }} to also cover ... (#{{ .PullRequestNumber }})
+```

.github/stale.yml

@@ -0,0 +1,66 @@
+# Configuration for probot-stale - https://github.com/probot/stale
+
+# Number of days of inactivity before an Issue or Pull Request becomes stale
+daysUntilStale: 30
+
+# Number of days of inactivity before an Issue or Pull Request with the stale label is closed.
+# Set to false to disable. If disabled, issues still need to be closed manually, but will remain marked as stale.
+daysUntilClose: 14
+
+# Only issues or pull requests with all of these labels are check if stale. Defaults to `[]` (disabled)
+onlyLabels: []
+
+# Issues or Pull Requests with these labels will never be considered stale. Set to `[]` to disable
+exemptLabels:
+  - pinned
+  - security
+  - enhancement
+  - refactor
+  - documentation
+  - chore
+  - needs-investigation
+  - bug
+
+# Set to true to ignore issues in a project (defaults to false)
+exemptProjects: false
+
+# Set to true to ignore issues in a milestone (defaults to false)
+exemptMilestones: false
+
+# Set to true to ignore issues with an assignee (defaults to false)
+exemptAssignees: false
+
+# Label to use when marking as stale
+staleLabel: stale
+
+# Comment to post when marking as stale. Set to `false` to disable
+markComment: >
+  This issue has been automatically marked as stale because it has not had
+  recent activity. It will be closed if no further activity occurs. Thank you
+  for your contributions.
+
+# Comment to post when removing the stale label.
+# unmarkComment: >
+#   Your comment here.
+
+# Comment to post when closing a stale Issue or Pull Request.
+# closeComment: >
+#   Your comment here.
+
+# Limit the number of actions per hour, from 1-30. Default is 30
+limitPerRun: 30
+
+# Limit to only `issues` or `pulls`
+# only: issues
+
+# Optionally, specify configuration settings that are specific to just 'issues' or 'pulls':
+# pulls:
+#   daysUntilStale: 30
+#   markComment: >
+#     This pull request has been automatically marked as stale because it has not had
+#     recent activity. It will be closed if no further activity occurs. Thank you
+#     for your contributions.
+
+# issues:
+#   exemptLabels:
+#     - confirmed

.github/workflows/build-and-release-runners.yml

@@ -13,25 +13,31 @@ on:
     paths:
       - runner/patched/*
       - runner/Dockerfile
-      - runner/dindrunner.Dockerfile
+      - runner/Dockerfile.ubuntu.1804
+      - runner/Dockerfile.dindrunner
       - runner/entrypoint.sh
       - .github/workflows/build-and-release-runners.yml
 
 jobs:
   build:
     runs-on: ubuntu-latest
-    name: Build ${{ matrix.name }}
+    name: Build ${{ matrix.name }}-ubuntu-${{ matrix.os-version }}
     strategy:
       matrix:
         include:
           - name: actions-runner
+            os-version: 20.04
             dockerfile: Dockerfile
+          - name: actions-runner
+            os-version: 18.04
+            dockerfile: Dockerfile.ubuntu.1804
           - name: actions-runner-dind
-            dockerfile: dindrunner.Dockerfile
+            os-version: 20.04
+            dockerfile: Dockerfile.dindrunner
     env:
-      RUNNER_VERSION: 2.277.1
+      RUNNER_VERSION: 2.278.0
       DOCKER_VERSION: 19.03.12
-      DOCKERHUB_USERNAME: ${{ github.repository_owner }}
+      DOCKERHUB_USERNAME: ${{ secrets.DOCKER_USER }}
     steps:
       - name: Set outputs
         id: vars
@@ -52,10 +58,10 @@ jobs:
         uses: docker/login-action@v1
         if: ${{ github.event_name == 'push' || github.event_name == 'release' }}
         with:
-          username: ${{ github.repository_owner }}
+          username: ${{ secrets.DOCKER_USER }}
           password: ${{ secrets.DOCKER_ACCESS_TOKEN }}
 
-      - name: Build and Push
+      - name: Build and Push Versioned Tags
         uses: docker/build-push-action@v2
         with:
           context: ./runner
@@ -66,6 +72,52 @@ jobs:
             RUNNER_VERSION=${{ env.RUNNER_VERSION }}
             DOCKER_VERSION=${{ env.DOCKER_VERSION }}
           tags: |
-            ${{ env.DOCKERHUB_USERNAME }}/${{ matrix.name }}:v${{ env.RUNNER_VERSION }}
-            ${{ env.DOCKERHUB_USERNAME }}/${{ matrix.name }}:v${{ env.RUNNER_VERSION }}-${{ steps.vars.outputs.sha_short }}
+            ${{ env.DOCKERHUB_USERNAME }}/${{ matrix.name }}:v${{ env.RUNNER_VERSION }}-ubuntu-${{ matrix.os-version }}
+            ${{ env.DOCKERHUB_USERNAME }}/${{ matrix.name }}:v${{ env.RUNNER_VERSION }}-ubuntu-${{ matrix.os-version }}-${{ steps.vars.outputs.sha_short }}
+
+  latest-tags:
+    if: ${{ github.event_name == 'push' || github.event_name == 'release' }}
+    runs-on: ubuntu-latest
+    name: Build ${{ matrix.name }}-latest
+    strategy:
+      matrix:
+        include:
+          - name: actions-runner
+            dockerfile: Dockerfile
+          - name: actions-runner-dind
+            dockerfile: Dockerfile.dindrunner
+    env:
+      RUNNER_VERSION: 2.277.1
+      DOCKER_VERSION: 19.03.12
+      DOCKERHUB_USERNAME: ${{ secrets.DOCKER_USER }}
+    steps:
+
+      - name: Checkout
+        uses: actions/checkout@v2
+
+      - name: Set up QEMU
+        uses: docker/setup-qemu-action@v1
+
+      - name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@v1
+        with:
+          version: latest
+
+      - name: Login to DockerHub
+        uses: docker/login-action@v1
+        with:
+          username: ${{ secrets.DOCKER_USER }}
+          password: ${{ secrets.DOCKER_ACCESS_TOKEN }}
+
+      - name: Build and Push Latest Tag
+        uses: docker/build-push-action@v2
+        with:
+          context: ./runner
+          file: ./runner/${{ matrix.dockerfile }}
+          platforms: linux/amd64,linux/arm64
+          push: true
+          build-args: |
+            RUNNER_VERSION=${{ env.RUNNER_VERSION }}
+            DOCKER_VERSION=${{ env.DOCKER_VERSION }}
+          tags: |
             ${{ env.DOCKERHUB_USERNAME }}/${{ matrix.name }}:latest

.github/workflows/on-push-lint-charts.yml

@@ -4,9 +4,11 @@ on:
   push:
     paths:
       - 'charts/**'
+      - '!charts/actions-runner-controller/docs/**'
+      - '!charts/actions-runner-controller/*.md'
       - '.github/**'
+      - '!.github/*.md'
   workflow_dispatch:
-
 env:
   KUBE_SCORE_VERSION: 1.10.0
   HELM_VERSION: v3.4.1

.github/workflows/on-push-master-publish-chart.yml

@@ -7,7 +7,9 @@ on:
       - main # assume that the branch name may change in future
     paths:
       - 'charts/**'
+      - '!charts/actions-runner-controller/docs/**'
       - '.github/**'
+      - '!**.md'
   workflow_dispatch:
 
 env:

.github/workflows/release.yml

@@ -7,7 +7,7 @@ jobs:
     runs-on: ubuntu-latest
     name: Release
     env:
-      DOCKERHUB_USERNAME: ${{ github.repository_owner }}
+      DOCKERHUB_USERNAME: ${{ secrets.DOCKER_USER }}
     steps:
       - name: Set outputs
         id: vars
@@ -47,7 +47,7 @@ jobs:
       - name: Login to DockerHub
         uses: docker/login-action@v1
         with:
-          username: ${{ github.repository_owner }}
+          username: ${{ secrets.DOCKER_USER }}
           password: ${{ secrets.DOCKER_ACCESS_TOKEN }}
 
       - name: Build and Push

.github/workflows/test.yaml

@@ -7,6 +7,8 @@ on:
     paths-ignore:
     - 'runner/**'
     - .github/workflows/build-and-release-runners.yml
+    - '*.md'
+    - '.gitignore'
 
 jobs:
   test:

.github/workflows/wip.yml

@@ -4,13 +4,15 @@ on:
       - master
     paths-ignore:
       - "runner/**"
+      - "**.md"
+      - ".gitignore"
 
 jobs:
   build:
     runs-on: ubuntu-latest
     name: release-latest
     env:
-      DOCKERHUB_USERNAME: ${{ github.repository_owner }}
+      DOCKERHUB_USERNAME: ${{ secrets.DOCKER_USER }}
     steps:
       - name: Checkout
         uses: actions/checkout@v2
@@ -27,7 +29,7 @@ jobs:
       - name: Login to DockerHub
         uses: docker/login-action@v1
         with:
-          username: ${{ github.repository_owner }}
+          username: ${{ secrets.DOCKER_USER }}
           password: ${{ secrets.DOCKER_ACCESS_TOKEN }}
 
       # Considered unstable builds

CONTRIBUTING.md

@@ -0,0 +1,8 @@
+# Contributing
+
+### Helm Version Bumps
+
+**Chart Version :** When bumping the chart version follow semantic versioning https://semver.org/<br />
+**App Version :** When bumping the app version you will also need to bump the chart version too. Again, follow semantic versioning when bumping the chart.
+
+To determine if you need to bump the MAJOR, MINOR or PATCH versions you will need to review the changes between the previous app version and the new app version and / or ask for a maintainer to advise.

Makefile

@@ -1,5 +1,17 @@
-NAME ?= summerwind/actions-runner-controller
+ifdef DOCKER_USER
+	NAME ?= ${DOCKER_USER}/actions-runner-controller
+else
+	NAME ?= summerwind/actions-runner-controller
+endif
+DOCKER_USER ?= $(shell echo ${NAME} | cut -d / -f1)
 VERSION ?= latest
+RUNNER_NAME ?= ${DOCKER_USER}/actions-runner
+RUNNER_TAG  ?= ${VERSION}
+TEST_REPO ?= ${DOCKER_USER}/actions-runner-controller
+TEST_ORG ?=
+TEST_ORG_REPO ?=
+SYNC_PERIOD ?= 5m
+
 # From https://github.com/VictoriaMetrics/operator/pull/44
 YAML_DROP=$(YQ) delete --inplace
 YAML_DROP_PREFIX=spec.validation.openAPIV3Schema.properties.spec.properties
@@ -14,6 +26,8 @@ else
 GOBIN=$(shell go env GOBIN)
 endif
 
+TEST_ASSETS=$(PWD)/test-assets
+
 # default list of platforms for which multiarch image is built
 ifeq (${PLATFORMS}, )
 	export PLATFORMS="linux/amd64,linux/arm64"
@@ -22,8 +36,8 @@ endif
 # if IMG_RESULT is unspecified, by default the image will be pushed to registry
 ifeq (${IMG_RESULT}, load)
 	export PUSH_ARG="--load"
-    # if load is specified, image will be built only for the build machine architecture.
-    export PLATFORMS="local"
+	# if load is specified, image will be built only for the build machine architecture.
+	export PLATFORMS="local"
 else ifeq (${IMG_RESULT}, cache)
 	# if cache is specified, image will only be available in the build cache, it won't be pushed or loaded
 	# therefore no PUSH_ARG will be specified
@@ -37,6 +51,13 @@ all: manager
 test: generate fmt vet manifests
 	go test ./... -coverprofile cover.out
 
+test-with-deps: kube-apiserver etcd kubectl
+	# See https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/envtest#pkg-constants
+	TEST_ASSET_KUBE_APISERVER=$(KUBE_APISERVER_BIN) \
+	TEST_ASSET_ETCD=$(ETCD_BIN) \
+	TEST_ASSET_KUBECTL=$(KUBECTL_BIN) \
+	  make test
+
 # Build manager binary
 manager: generate fmt vet
 	go build -o bin/manager main.go
@@ -96,12 +117,9 @@ generate: controller-gen
 	$(CONTROLLER_GEN) object:headerFile=./hack/boilerplate.go.txt paths="./..."
 
 # Build the docker image
-docker-build: test
+docker-build:
 	docker build . -t ${NAME}:${VERSION}
-
-# Push the docker image
-docker-push:
-	docker push ${NAME}:${VERSION}
+	docker build runner -t ${RUNNER_NAME}:${RUNNER_TAG} --build-arg TARGETPLATFORM=$(shell arch)
 
 docker-buildx:
 	export DOCKER_CLI_EXPERIMENTAL=enabled
@@ -115,6 +133,11 @@ docker-buildx:
 		-f Dockerfile \
 		. ${PUSH_ARG}
 
+# Push the docker image
+docker-push:
+	docker push ${NAME}:${VERSION}
+	docker push ${RUNNER_NAME}:${RUNNER_TAG}
+
 # Generate the release manifest file
 release: manifests
 	cd config/manager && kustomize edit set image controller=${NAME}:${VERSION}
@@ -126,19 +149,41 @@ release/clean:
 	rm -rf release
 
 .PHONY: acceptance
-acceptance: release/clean docker-build docker-push release
-	ACCEPTANCE_TEST_SECRET_TYPE=token make acceptance/kind acceptance/setup acceptance/tests acceptance/teardown
-	ACCEPTANCE_TEST_SECRET_TYPE=app make acceptance/kind acceptance/setup acceptance/tests acceptance/teardown
-	ACCEPTANCE_TEST_DEPLOYMENT_TOOL=helm ACCEPTANCE_TEST_SECRET_TYPE=token make acceptance/kind acceptance/setup acceptance/tests acceptance/teardown
-	ACCEPTANCE_TEST_DEPLOYMENT_TOOL=helm ACCEPTANCE_TEST_SECRET_TYPE=app make acceptance/kind acceptance/setup acceptance/tests acceptance/teardown
+acceptance: release/clean acceptance/pull docker-build release
+	ACCEPTANCE_TEST_SECRET_TYPE=token make acceptance/run
+	ACCEPTANCE_TEST_SECRET_TYPE=app make acceptance/run
+	ACCEPTANCE_TEST_DEPLOYMENT_TOOL=helm ACCEPTANCE_TEST_SECRET_TYPE=token make acceptance/run
+	ACCEPTANCE_TEST_DEPLOYMENT_TOOL=helm ACCEPTANCE_TEST_SECRET_TYPE=app make acceptance/run
+
+acceptance/run: acceptance/kind acceptance/load acceptance/setup acceptance/deploy acceptance/tests acceptance/teardown
 
 acceptance/kind:
-	kind create cluster --name acceptance
+	kind create cluster --name acceptance --config acceptance/kind.yaml
+
+# Set TMPDIR to somewhere under $HOME when you use docker installed with Ubuntu snap
+# Otherwise `load docker-image` fail while running `docker save`.
+# See https://kind.sigs.k8s.io/docs/user/known-issues/#docker-installed-with-snap
+acceptance/load:
+	kind load docker-image ${NAME}:${VERSION} --name acceptance
+	kind load docker-image quay.io/brancz/kube-rbac-proxy:v0.10.0 --name acceptance
+	kind load docker-image ${RUNNER_NAME}:${RUNNER_TAG} --name acceptance
+	kind load docker-image docker:dind --name acceptance
+	kind load docker-image quay.io/jetstack/cert-manager-controller:v1.0.4 --name acceptance
+	kind load docker-image quay.io/jetstack/cert-manager-cainjector:v1.0.4 --name acceptance
+	kind load docker-image quay.io/jetstack/cert-manager-webhook:v1.0.4 --name acceptance
 	kubectl cluster-info --context kind-acceptance
 
+# Pull the docker images for acceptance
+acceptance/pull:
+	docker pull quay.io/brancz/kube-rbac-proxy:v0.10.0
+	docker pull docker:dind
+	docker pull quay.io/jetstack/cert-manager-controller:v1.0.4
+	docker pull quay.io/jetstack/cert-manager-cainjector:v1.0.4
+	docker pull quay.io/jetstack/cert-manager-webhook:v1.0.4
+
 acceptance/setup:
 	kubectl apply --validate=false -f https://github.com/jetstack/cert-manager/releases/download/v1.0.4/cert-manager.yaml	#kubectl create namespace actions-runner-system
-	kubectl -n cert-manager wait deploy/cert-manager-cainjector --for condition=available --timeout 60s
+	kubectl -n cert-manager wait deploy/cert-manager-cainjector --for condition=available --timeout 90s
 	kubectl -n cert-manager wait deploy/cert-manager-webhook --for condition=available --timeout 60s
 	kubectl -n cert-manager wait deploy/cert-manager --for condition=available --timeout 60s
 	kubectl create namespace actions-runner-system || true
@@ -148,8 +193,12 @@ acceptance/setup:
 acceptance/teardown:
 	kind delete cluster --name acceptance
 
-acceptance/tests:
+acceptance/deploy:
+	NAME=${NAME} DOCKER_USER=${DOCKER_USER} VERSION=${VERSION} RUNNER_NAME=${RUNNER_NAME} RUNNER_TAG=${RUNNER_TAG} TEST_REPO=${TEST_REPO} \
+	TEST_ORG=${TEST_ORG} TEST_ORG_REPO=${TEST_ORG_REPO} SYNC_PERIOD=${SYNC_PERIOD} \
 	acceptance/deploy.sh
+
+acceptance/tests:
 	acceptance/checks.sh
 
 # Upload release file to GitHub.
@@ -191,3 +240,77 @@ ifeq (, $(wildcard $(GOBIN)/yq))
 	}
 endif
 YQ=$(GOBIN)/yq
+
+OS_NAME := $(shell uname -s | tr A-Z a-z)
+
+# find or download etcd
+etcd:
+ifeq (, $(shell which etcd))
+ifeq (, $(wildcard $(TEST_ASSETS)/etcd))
+	@{ \
+	set -xe ;\
+	INSTALL_TMP_DIR=$$(mktemp -d) ;\
+	cd $$INSTALL_TMP_DIR ;\
+	wget https://github.com/kubernetes-sigs/kubebuilder/releases/download/v2.3.2/kubebuilder_2.3.2_$(OS_NAME)_amd64.tar.gz ;\
+	mkdir -p $(TEST_ASSETS) ;\
+	tar zxvf kubebuilder_2.3.2_$(OS_NAME)_amd64.tar.gz ;\
+	mv kubebuilder_2.3.2_$(OS_NAME)_amd64/bin/etcd $(TEST_ASSETS)/etcd ;\
+	mv kubebuilder_2.3.2_$(OS_NAME)_amd64/bin/kube-apiserver $(TEST_ASSETS)/kube-apiserver ;\
+	mv kubebuilder_2.3.2_$(OS_NAME)_amd64/bin/kubectl $(TEST_ASSETS)/kubectl ;\
+	rm -rf $$INSTALL_TMP_DIR ;\
+	}
+ETCD_BIN=$(TEST_ASSETS)/etcd
+else
+ETCD_BIN=$(TEST_ASSETS)/etcd
+endif
+else
+ETCD_BIN=$(shell which etcd)
+endif
+
+# find or download kube-apiserver
+kube-apiserver:
+ifeq (, $(shell which kube-apiserver))
+ifeq (, $(wildcard $(TEST_ASSETS)/kube-apiserver))
+	@{ \
+	set -xe ;\
+	INSTALL_TMP_DIR=$$(mktemp -d) ;\
+	cd $$INSTALL_TMP_DIR ;\
+	wget https://github.com/kubernetes-sigs/kubebuilder/releases/download/v2.3.2/kubebuilder_2.3.2_$(OS_NAME)_amd64.tar.gz ;\
+	mkdir -p $(TEST_ASSETS) ;\
+	tar zxvf kubebuilder_2.3.2_$(OS_NAME)_amd64.tar.gz ;\
+	mv kubebuilder_2.3.2_$(OS_NAME)_amd64/bin/etcd $(TEST_ASSETS)/etcd ;\
+	mv kubebuilder_2.3.2_$(OS_NAME)_amd64/bin/kube-apiserver $(TEST_ASSETS)/kube-apiserver ;\
+	mv kubebuilder_2.3.2_$(OS_NAME)_amd64/bin/kubectl $(TEST_ASSETS)/kubectl ;\
+	rm -rf $$INSTALL_TMP_DIR ;\
+	}
+KUBE_APISERVER_BIN=$(TEST_ASSETS)/kube-apiserver
+else
+KUBE_APISERVER_BIN=$(TEST_ASSETS)/kube-apiserver
+endif
+else
+KUBE_APISERVER_BIN=$(shell which kube-apiserver)
+endif
+
+# find or download kubectl
+kubectl:
+ifeq (, $(shell which kubectl))
+ifeq (, $(wildcard $(TEST_ASSETS)/kubectl))
+	@{ \
+	set -xe ;\
+	INSTALL_TMP_DIR=$$(mktemp -d) ;\
+	cd $$INSTALL_TMP_DIR ;\
+	wget https://github.com/kubernetes-sigs/kubebuilder/releases/download/v2.3.2/kubebuilder_2.3.2_$(OS_NAME)_amd64.tar.gz ;\
+	mkdir -p $(TEST_ASSETS) ;\
+	tar zxvf kubebuilder_2.3.2_$(OS_NAME)_amd64.tar.gz ;\
+	mv kubebuilder_2.3.2_$(OS_NAME)_amd64/bin/etcd $(TEST_ASSETS)/etcd ;\
+	mv kubebuilder_2.3.2_$(OS_NAME)_amd64/bin/kube-apiserver $(TEST_ASSETS)/kube-apiserver ;\
+	mv kubebuilder_2.3.2_$(OS_NAME)_amd64/bin/kubectl $(TEST_ASSETS)/kubectl ;\
+	rm -rf $$INSTALL_TMP_DIR ;\
+	}
+KUBECTL_BIN=$(TEST_ASSETS)/kubectl
+else
+KUBECTL_BIN=$(TEST_ASSETS)/kubectl
+endif
+else
+KUBECTL_BIN=$(shell which kubectl)
+endif

README.md

@@ -8,25 +8,28 @@ ToC:
 
 - [Motivation](#motivation)
 - [Installation](#installation)
-  - [GitHub Enterprise support](#github-enterprise-support)
-- [Setting up authentication with GitHub API](#setting-up-authentication-with-github-api)
-  - [Deploying using GitHub App Authentication](#deploying-using-github-app-authentication)
-  - [Deploying using PAT Authentication](#deploying-using-pat-authentication)
+  - [GitHub Enterprise Support](#github-enterprise-support)
+- [Setting Up Authentication with GitHub API](#setting-up-authentication-with-github-api)
+  - [Deploying Using GitHub App Authentication](#deploying-using-github-app-authentication)
+  - [Deploying Using PAT Authentication](#deploying-using-pat-authentication)
 - [Usage](#usage)
   - [Repository Runners](#repository-runners)
   - [Organization Runners](#organization-runners)
+  - [Enterprise Runners](#enterprise-runners)
   - [Runner Deployments](#runnerdeployments)
-    - [Autoscaling](#autoscaling)
-      - [Faster Autoscaling with GitHub Webhook](#faster-autoscaling-with-github-webhook)
+    - [Note on scaling to/from 0](#note-on-scaling-tofrom-0)
+  - [Autoscaling](#autoscaling)
+    - [Faster Autoscaling with GitHub Webhook](#faster-autoscaling-with-github-webhook)
+    - [Autoscaling to/from 0](#autoscaling-tofrom-0)
+    - [Scheduled Overrides](#scheduled-overrides)
   - [Runner with DinD](#runner-with-dind)
-  - [Additional tweaks](#additional-tweaks)
-  - [Runner labels](#runner-labels)
-  - [Runner groups](#runner-groups)
-  - [Using EKS IAM role for service accounts](#using-eks-iam-role-for-service-accounts)
-  - [Software installed in the runner image](#software-installed-in-the-runner-image)
-  - [Common errors](#common-errors)
-- [Developing](#developing)
-- [Alternatives](#alternatives)
+  - [Additional Tweaks](#additional-tweaks)
+  - [Runner Labels](#runner-labels)
+  - [Runner Groups](#runner-groups)
+  - [Using IRSA (IAM Roles for Service Accounts) in EKS](#using-irsa-iam-roles-for-service-accounts-in-eks)
+  - [Software Installed in the Runner Image](#software-installed-in-the-runner-image)
+  - [Common Errors](#common-errors)
+- [Contributing](#contributing)
 
 ## Motivation
 
@@ -42,87 +45,83 @@ actions-runner-controller uses [cert-manager](https://cert-manager.io/docs/insta
 
 Install the custom resource and actions-runner-controller with `kubectl` or `helm`. This will create actions-runner-system namespace in your Kubernetes and deploy the required resources.
 
-`kubectl`:
+**Kubectl Deployment:**
 
 ```shell
-# REPLACE "v0.17.0" with the version you wish to deploy
-kubectl apply -f https://github.com/summerwind/actions-runner-controller/releases/download/v0.17.0/actions-runner-controller.yaml
+# REPLACE "v0.18.2" with the version you wish to deploy
+kubectl apply -f https://github.com/actions-runner-controller/actions-runner-controller/releases/download/v0.18.2/actions-runner-controller.yaml
 ```
 
-`helm`:
+**Helm Deployment:**
+
+__**Note: For all configuration options for the Helm chart see the chart's [README](./charts/actions-runner-controller/README.md)
 
 ```shell
-helm repo add actions-runner-controller https://summerwind.github.io/actions-runner-controller
-helm upgrade --install -n actions-runner-system actions-runner-controller/actions-runner-controller
+helm repo add actions-runner-controller https://actions-runner-controller.github.io/actions-runner-controller
+helm upgrade --install --namespace actions-runner-system --create-namespace \ 
+             --wait actions-runner-controller actions-runner-controller/actions-runner-controller
 ```
 
-### Github Enterprise support
+### GitHub Enterprise Support
 
-If you use either Github Enterprise Cloud or Server, you can use **actions-runner-controller**  with those, too.
-Authentication works same way as with public Github (repo and organization level).
-The minimum version of Github Enterprise Server is 3.0.0 (or rc1/rc2).
-__**NOTE : The maintainers do not have an Enterprise environment to be able to test changes and so are reliant on the community for testing, support is a best endeavors basis only and is community driven**__
+The solution supports both GitHub Enterprise Cloud and Server editions as well as regular GitHub. Both PAT (personal access token) and GitHub App authentication works for installations that will be deploying either repository level and / or organization level runners. If you need to deploy enterprise level runners then you are restricted to PAT based authentication as GitHub doesn't support GitHub App based authentication for enterprise runners currently.
+
+If you are deploying this solution into a GitHub Enterprise Server environment then you will need version >= [3.0.0](https://docs.github.com/en/enterprise-server@3.0/admin/release-notes#3.0.0).
+
+When deploying the solution for a GitHub Enterprise Server environment you need to provide an additional environment variable as part of the controller deployment:
 
 ```shell
 kubectl set env deploy controller-manager -c manager GITHUB_ENTERPRISE_URL=<GHEC/S URL> --namespace actions-runner-system
 ```
 
-#### Enterprise runners usage
+__**Note: The repository maintainers do not have an enterprise environment (cloud or server). Support for the enterprise specific feature set is community driven and on a best effort basis. PRs from the community are welcomed to add features and maintain support.**__
 
-In order to use enterprise runners you must have Admin access to Github Enterprise and you should do Personal Access Token (PAT)
-with `enterprise:admin` access. Enterprise runners are not possible to run with Github APP or any other permission.
-
-When you use enterprise runners those will get access to Github Organisations. However, access to the repositories is **NOT**
-allowed by default. Each Github Organisation must allow Enterprise runner groups to be used in repositories.
-This is needed only one time and is permanent after that.
-
-Example:
-
-```yaml
-apiVersion: actions.summerwind.dev/v1alpha1
-kind: RunnerDeployment
-metadata:
-  name: ghe-runner-deployment
-spec:
-  replicas: 2
-  template:
-    spec:
-      enterprise: your-enterprise-name
-      dockerdWithinRunnerContainer: true
-      resources:
-        limits:
-          cpu: "4000m"
-          memory: "2Gi"
-        requests:
-          cpu: "200m"
-          memory: "200Mi"
-      volumeMounts:
-      - mountPath: /runner
-        name: runner
-      volumes:
-      - name: runner
-        emptyDir: {}
-
-```
-
-## Setting up authentication with GitHub API
+## Setting Up Authentication with GitHub API
 
 There are two ways for actions-runner-controller to authenticate with the GitHub API (only 1 can be configured at a time however):
 
-1. Using GitHub App.
-2. Using Personal Access Token.
+1. Using a GitHub App (not supported for enterprise level runners due to lack of support from GitHub)
+2. Using a PAT
 
-Functionality wise there isn't a difference between the 2 authentication methods. There are however some benefits to using a GitHub App for authentication over a PAT such as an [increased API quota](https://docs.github.com/en/developers/apps/rate-limits-for-github-apps), if you run into rate limiting consider deploying this solution using GitHub App authentication instead.
+Functionality wise, there isn't much of a difference between the 2 authentication methods. The primarily benefit of authenticating via a GitHub App is an [increased API quota](https://docs.github.com/en/developers/apps/rate-limits-for-github-apps).
 
-### Deploying using GitHub App Authentication
+If you are deploying the solution for a GitHub Enterprise Server environment you are able to [configure your rate limiting settings](https://docs.github.com/en/enterprise-server@3.0/admin/configuration/configuring-rate-limits) making the main benefit irrelevant. If you're deploying the solution for a GitHub Enterprise Cloud or regular GitHub environment and you run into rate limiting issues, consider deploying the solution using the GitHub App authentication method instead.
 
-You can create a GitHub App for either your account or any organization. If you want to create a GitHub App for your account, open the following link to the creation page, enter any unique name in the "GitHub App name" field, and hit the "Create GitHub App" button at the bottom of the page.
+### Deploying Using GitHub App Authentication
 
-- [Create GitHub Apps on your account](https://github.com/settings/apps/new?url=http://github.com/summerwind/actions-runner-controller&webhook_active=false&public=false&administration=write&actions=read)
+You can create a GitHub App for either your user account or any organization, below are the app permissions required for each supported type of runner:
+
+_Note: Links are provided further down to create an app for your logged in user account or an organisation with the permissions for all runner types set in each link's query string_
+
+**Required Permissions for Repository Runners:**<br />
+**Repository Permissions**
+
+* Actions (read)
+* Administration (read / write)
+* Metadata (read)
+
+**Required Permissions for Organisation Runners:**<br />
+**Repository Permissions**
+
+* Actions (read)
+* Metadata (read)
+
+**Organization Permissions**
+* Self-hosted runners (read / write)
+
+_Note: All API routes mapped to their permissions can be found [here](https://docs.github.com/en/rest/reference/permissions-required-for-github-apps) if you wish to review_
+
+---
+
+**Setup Steps**
+
+If you want to create a GitHub App for your account, open the following link to the creation page, enter any unique name in the "GitHub App name" field, and hit the "Create GitHub App" button at the bottom of the page.
+
+- [Create GitHub Apps on your account](https://github.com/settings/apps/new?url=http://github.com/actions-runner-controller/actions-runner-controller&webhook_active=false&public=false&administration=write&actions=read)
 
 If you want to create a GitHub App for your organization, replace the `:org` part of the following URL with your organization name before opening it. Then enter any unique name in the "GitHub App name" field, and hit the "Create GitHub App" button at the bottom of the page to create a GitHub App.
 
-- [Create GitHub Apps on your organization](https://github.com/organizations/:org/settings/apps/new?url=http://github.com/summerwind/actions-runner-controller&webhook_active=false&public=false&administration=write&organization_self_hosted_runners=write&actions=read)
+- [Create GitHub Apps on your organization](https://github.com/organizations/:org/settings/apps/new?url=http://github.com/actions-runner-controller/actions-runner-controller&webhook_active=false&public=false&administration=write&organization_self_hosted_runners=write&actions=read)
 
 You will see an *App ID* on the page of the GitHub App you created as follows, the value of this App ID will be used later.
 
@@ -141,8 +140,11 @@ When the installation is complete, you will be taken to a URL in one of the foll
 - `https://github.com/settings/installations/${INSTALLATION_ID}`
 - `https://github.com/organizations/eventreactor/settings/installations/${INSTALLATION_ID}`
 
+
 Finally, register the App ID (`APP_ID`), Installation ID (`INSTALLATION_ID`), and downloaded private key file (`PRIVATE_KEY_FILE_PATH`) to Kubernetes as Secret.
 
+**Kubectl Deployment:**
+
 ```shell
 $ kubectl create secret generic controller-manager \
     -n actions-runner-system \
@@ -151,29 +153,41 @@ $ kubectl create secret generic controller-manager \
     --from-file=github_app_private_key=${PRIVATE_KEY_FILE_PATH}
 ```
 
-### Deploying using PAT Authentication
+**Helm Deployment:**
 
-Personal Acess Token can be used to register a self-hosted runner by *actions-runner-controller*.
+Configure your values.yaml, see the chart's [README](./charts/actions-runner-controller/README.md) for deploying the secret via Helm
 
-Self-hosted runners in GitHub can either be connected to a single repository, or to a GitHub organization (so they are available to all repositories in the organization). How you plan on using the runner will affect what scopes are needed for the token.
+### Deploying Using PAT Authentication
+
+Personal Access Tokens can be used to register a self-hosted runner by *actions-runner-controller*.
 
 Log-in to a GitHub account that has `admin` privileges for the repository, and [create a personal access token](https://github.com/settings/tokens/new) with the appropriate scopes listed below:
 
-**Scopes for a Repository Runner**
+**Required Scopes for Repository Runners**
 
 * repo (Full control)
 
-**Scopes for a Organisation Runner**
+**Required Scopes for Organization Runners**
 
 * repo (Full control)
 * admin:org (Full control)
-* admin:public_key - read:public_key
-* admin:repo_hook - read:repo_hook
-* admin:org_hook
-* notifications
-* workflow
+* admin:public_key (read:public_key)
+* admin:repo_hook (read:repo_hook)
+* admin:org_hook (Full control)
+* notifications (Full control)
+* workflow (Full control)
 
-Once you have created the appropriate token, deploy it as a secret to your kubernetes cluster that you are going to deploy the solution on:
+**Required Scopes for Enterprise Runners**
+
+* enterprise:admin (Full control)
+
+_Note: When you deploy enterprise runners they will get access to organisations, however, access to the repositories themselves is **NOT** allowed by default. Each GitHub organisation must allow enterprise runner groups to be used in repositories as an initial one time configuration step, this  only needs to be done once after which it is permanent for that runner group._
+
+---
+
+Once you have created the appropriate token, deploy it as a secret to your Kubernetes cluster that you are going to deploy the solution on:
+
+**Kubectl Deployment:**
 
 ```shell
 kubectl create secret generic controller-manager \
@@ -181,8 +195,17 @@ kubectl create secret generic controller-manager \
     --from-literal=github_token=${GITHUB_TOKEN}
 ```
 
+**Helm Deployment:**
+
+Configure your values.yaml, see the chart's [README](./charts/actions-runner-controller/README.md) for deploying the secret via Helm
+
 ## Usage
 
+[GitHub self-hosted runners can be deployed at various levels in a management hierarchy](https://docs.github.com/en/actions/hosting-your-own-runners/about-self-hosted-runners#about-self-hosted-runners):
+- The repository level
+- The organization level
+- The enterprise level
+
 There are two ways to use this controller:
 
 - Manage runners one by one with `Runner`.
@@ -190,7 +213,7 @@ There are two ways to use this controller:
 
 ### Repository Runners
 
-To launch a single self-hosted runner, you need to create a manifest file includes *Runner* resource as follows. This example launches a self-hosted runner with name *example-runner* for the *summerwind/actions-runner-controller* repository.
+To launch a single self-hosted runner, you need to create a manifest file includes `Runner` resource as follows. This example launches a self-hosted runner with name *example-runner* for the *actions-runner-controller/actions-runner-controller* repository.
 
 ```yaml
 # runner.yaml
@@ -248,6 +271,22 @@ spec:
 
 Now you can see the runner on the organization level (if you have organization owner permissions).
 
+### Enterprise Runners
+
+To add the runner to an enterprise, you only need to replace the `repository` field with `enterprise`, so the runner will register itself to the enterprise.
+
+```yaml
+# runner.yaml
+apiVersion: actions.summerwind.dev/v1alpha1
+kind: Runner
+metadata:
+  name: example-enterprise-runner
+spec:
+  enterprise: your-enterprise-name
+```
+
+Now you can see the runner on the enterprise level (if you have enterprise access permissions).
+
 ### RunnerDeployments
 
 There are `RunnerReplicaSet` and `RunnerDeployment` that corresponds to `ReplicaSet` and `Deployment` but for `Runner`.
@@ -271,7 +310,7 @@ spec:
 Apply the manifest file to your cluster:
 
 ```shell
-$ kubectl apply -f runner.yaml
+$ kubectl apply -f runnerdeployment.yaml
 runnerdeployment.actions.summerwind.dev/example-runnerdeploy created
 ```
 
@@ -284,9 +323,37 @@ example-runnerdeploy2475h595fr   mumoshu/actions-runner-controller-ci   Running
 example-runnerdeploy2475ht2qbr   mumoshu/actions-runner-controller-ci   Running
 ```
 
-#### Autoscaling
+##### Note on scaling to/from 0
 
-A `RunnerDeployment` can scale the number of runners between `minReplicas` and `maxReplicas` fields based the chosen scaling metric as defined in the `metrics` attribute
+> This is a documentation about a unreleased version of actions-runner-controller.
+>
+> It would be great if you could try building the latest controller image following https://github.com/actions-runner-controller/actions-runner-controller#contributing if you are eager to test it early and help
+> developers by reporting any bugs :smile:
+
+You can either delete the runner deployment, or update it to have `replicas: 0`, so that there will be 0 runner pods in the cluster. This, in combination with e.g. `cluster-autoscaler`, enables you to save your infrastructure cost when there's no need to run Actions jobs.
+
+```yaml
+# runnerdeployment.yaml
+apiVersion: actions.summerwind.dev/v1alpha1
+kind: RunnerDeployment
+metadata:
+  name: example-runnerdeploy
+spec:
+  replicas: 0
+```
+
+The implication of setting `replicas: 0` instead of deleting the runner deployment is that you can let GitHub Actions queue jobs until there will be one or more runners. See [#465](https://github.com/actions-runner-controller/actions-runner-controller/pull/465) for more information.
+
+Also note that the controller creates a "registration-only" runner per RunnerReplicaSet on it's being scaled to zero,
+and retains it until there are one or more runners available.
+
+This, in combination with a correctly configured HorizontalRunnerAutoscaler, allows you to automatically [scale to/from 0](#autoscaling-tofrom-0)
+
+### Autoscaling
+
+__**IMPORTANT : Due to limitations / a bug with GitHub's [routing engine](https://docs.github.com/en/actions/hosting-your-own-runners/using-self-hosted-runners-in-a-workflow#routing-precedence-for-self-hosted-runners) autoscaling does NOT work correctly with RunnerDeployments that target the enterprise level. Scaling activity works as expected however jobs fail to get assigned to the scaled out replicas. This was explored in issue [#470](https://github.com/actions-runner-controller/actions-runner-controller/issues/470). Once GitHub resolves the issue with their backend service we expect the solution to be able to support autoscaled enterprise runnerdeploments without any additional changes.**__
+
+A `RunnerDeployment` (excluding enterprise runners) can scale the number of runners between `minReplicas` and `maxReplicas` fields based the chosen scaling metric as defined in the `metrics` attribute
 
 **Scaling Metrics**
 
@@ -298,21 +365,22 @@ With this scaling metric we are required to define a list of repositories within
 The scale out performance is controlled via the manager containers startup `--sync-period` argument. The default value is set to 10 minutes to prevent default deployments rate limiting themselves from the GitHub API.
 
 **Kustomize Config :** The period can be customised in the `config/default/manager_auth_proxy_patch.yaml` patch<br />
-**Helm Config :** `syncPeriod`
 
 **Benefits of this metric**
 1. Supports named repositories allowing you to restrict the runner to a specified set of repositories server side.
-2. Scales the runner count based on the actual queue depth of the jobs meaning a more 1:1 scaling of runners to queued jobs.
-3. Like all scaling metrics, you can manage workflow allocation to the RunnerDeployment through the use of [Github labels](#runner-labels).
+2. Scales the runner count based on the actual queue depth of the jobs meaning a more 1:1 scaling of runners to queued jobs (caveat, see drawback #4)
+3. Like all scaling metrics, you can manage workflow allocation to the RunnerDeployment through the use of [GitHub labels](#runner-labels).
 
 **Drawbacks of this metric**
 1. Repositories must be named within the scaling metric, maintaining a list of repositories may not be viable in larger environments or self-serve environments.
 2. May not scale quick enough for some users needs. This metric is pull based and so the queue depth is polled as configured by the sync period, as a result scaling performance is bound by this sync period meaning there is a lag to scaling activity.
 3. Relatively large amounts of API requests required to maintain this metric, you may run in API rate limiting issues depending on the size of your environment and how aggressive your sync period configuration is
+4. The GitHub API doesn't provide a way to filter workflow jobs to just those targeting self-hosted runners. If your environment's workflows target both self-hosted and GitHub hosted runners then the queue depth this metric scales against isn't a true 1:1 mapping of queue depth to required runner count. As a result of this, this metric may scale too aggressively for your actual self-hosted runner count needs.
 
 
-Example `RunnerDeployment` backed by a `HorizontalRunnerAutoscaler`
+Example `RunnerDeployment` backed by a `HorizontalRunnerAutoscaler`:
 
+_Important!!! We no longer include the attribute `replicas` in our `RunnerDeployment` if we are configuring autoscaling!_
 
 ```yaml
 apiVersion: actions.summerwind.dev/v1alpha1
@@ -352,13 +420,12 @@ spec:
 The `HorizontalRunnerAutoscaler` will poll GitHub based on the configuration sync period for the number of busy runners which live in the RunnerDeployment's namespace and scale based on the settings
 
 **Kustomize Config :** The period can be customised in the `config/default/manager_auth_proxy_patch.yaml` patch<br />
-**Helm Config :** `syncPeriod`
 
 **Benefits of this metric**
-1. Supports named repositories server side the same as the `TotalNumberOfQueuedAndInProgressWorkflowRuns` metric [#313](https://github.com/summerwind/actions-runner-controller/pull/313)
-2. Supports GitHub organisation wide scaling without maintaining an explicit list of repositories, this is especially useful for those that are working at a larger scale. [#223](https://github.com/summerwind/actions-runner-controller/pull/223)
-3. Like all scaling metrics, you can manage workflow allocation to the RunnerDeployment through the use of [Github labels](#runner-labels)
-4. Supports scaling desired runner count on both a percentage increase / decrease basis as well as on a fixed increase / decrease count basis [#223](https://github.com/summerwind/actions-runner-controller/pull/223) [#315](https://github.com/summerwind/actions-runner-controller/pull/315)
+1. Supports named repositories server side the same as the `TotalNumberOfQueuedAndInProgressWorkflowRuns` metric [#313](https://github.com/actions-runner-controller/actions-runner-controller/pull/313)
+2. Supports GitHub organization wide scaling without maintaining an explicit list of repositories, this is especially useful for those that are working at a larger scale. [#223](https://github.com/actions-runner-controller/actions-runner-controller/pull/223)
+3. Like all scaling metrics, you can manage workflow allocation to the RunnerDeployment through the use of [GitHub labels](#runner-labels)
+4. Supports scaling desired runner count on both a percentage increase / decrease basis as well as on a fixed increase / decrease count basis [#223](https://github.com/actions-runner-controller/actions-runner-controller/pull/223) [#315](https://github.com/actions-runner-controller/actions-runner-controller/pull/315)
 
 **Drawbacks of this metric**
 1. May not scale quick enough for some users needs. This metric is pull based and so the number of busy runners are polled as configured by the sync period, as a result scaling performance is bound by this sync period meaning there is a lag to scaling activity.
@@ -368,6 +435,8 @@ The `HorizontalRunnerAutoscaler` will poll GitHub based on the configuration syn
 Examples of each scaling type implemented with a `RunnerDeployment` backed by a `HorizontalRunnerAutoscaler`:
 
 
+_Important!!! We no longer include the attribute `replicas` in our `RunnerDeployment` if we are configuring autoscaling!_
+
 ```yaml
 ---
 apiVersion: actions.summerwind.dev/v1alpha1
@@ -415,15 +484,17 @@ spec:
 
 #### Faster Autoscaling with GitHub Webhook
 
+__**IMPORTANT : Due to missing webhook events, webhook based scaling is not available for enterprise level RunnerDeployments. This was explored in issue [#470](https://github.com/actions-runner-controller/actions-runner-controller/issues/470).**__
+
 > This feature is an ADVANCED feature which may require more work to set up.
 > Please get prepared to put some time and effort to learn and leverage this feature!
 
 `actions-runner-controller` has an optional Webhook server that receives GitHub Webhook events and scale
-[`RunnerDeployment`s](#runnerdeployments) by updating corresponding [`HorizontalRunnerAutoscaler`s](#autoscaling).
+[`RunnerDeployments`](#runnerdeployments) by updating corresponding [`HorizontalRunnerAutoscalers`](#autoscaling).
 
 Today, the Webhook server can be configured to respond GitHub `check_run`, `pull_request`, and `push` events
 by scaling up the matching `HorizontalRunnerAutoscaler` by N replica(s), where `N` is configurable within
-`HorizontalRunerAutoscaler`'s `Spec`.
+`HorizontalRunnerAutoscaler's` `Spec`.
 
 More concretely, you can configure the targeted GitHub event types and the `N` in
 `scaleUpTriggers`:
@@ -452,7 +523,7 @@ In contrast, the standard autoscaling requires you to wait next sync period to a
 insufficient runners. You can definitely shorten the sync period to make the standard autoscaling more responsive.
 But doing so eventually result in the controller not functional due to GitHub API rate limit.
 
-> You can learn the implementation details in #282
+> You can learn the implementation details in [#282](https://github.com/actions-runner-controller/actions-runner-controller/pull/282)
 
 To enable this feature, you firstly need to install the webhook server.
 
@@ -525,6 +596,107 @@ spec:
 
 See ["activity types"](https://docs.github.com/en/actions/reference/events-that-trigger-workflows#pull_request) for the list of valid values for `scaleUpTriggers[].githubEvent.pullRequest.types`.
 
+
+#### Autoscaling to/from 0
+
+> This is a documentation about a unreleased version of actions-runner-controller.
+>
+> It would be great if you could try building the latest controller image following https://github.com/actions-runner-controller/actions-runner-controller#contributing if you are eager to test it early and help
+> developers by reporting any bugs :smile:
+
+Previously, we've discussed about [how to scale a RunnerDeployment to/from 0](#note-on-scaling-tofrom-0)
+
+To automate the process of scaling to/from 0, you can use `HorizontalRunnerAutoscaler` with a caveat.
+
+That is, you need to choose one of the following configuration for metrics and triggers:
+
+- `TotalNumberOfQueuedAndInProgressWorkflowRuns`
+- `PercentageRunnersBusy` + `TotalNumberOfQueuedAndInProgressWorkflowRuns`
+- `PercentageRunnersBusy` + Webhook-based autoscaling
+
+This is due to that `PercentageRunnersBusy`, by its definition, needs one or more GitHub runners that can become `busy`, which cannot happen at all when you have 0 active runners.
+
+If and only if HorizontalRunnerAutoscaler is configured to have a secondary metric of `TotalNumberOfQueuedAndInProgressWorkflowRuns` and the controller sees the primary metric of `PercentageRunnersBusy` returned 0 desired replicas, it uses the secondary metric for calculating the desired replicas once again.
+
+A correctly configured `TotalNumberOfQueuedAndInProgressWorkflowRuns` can return non-zero desired replicas even when there are no runners other than [registration-only runners](#note-on-scaling-tofrom-0), hence the `PercentageRunnersBusy` + `TotalNumberOfQueuedAndInProgressWorkflowRuns` configuration makes scaling from zero possible.
+
+Similarly, Webhook-based autoscaling works regardless of there are active runners, hence `PercentageRunnersBusy` + Webhook-based autoscaling configuration makes scaling from zero, too.
+
+#### Scheduled Overrides
+
+> This is a documentation about a unreleased version of actions-runner-controller.
+>
+> It would be great if you could try building the latest controller image following https://github.com/actions-runner-controller/actions-runner-controller#contributing if you are eager to test it early and help
+> developers by reporting any bugs :smile:
+
+`Scheduled Overrides` allows you to configure HorizontalRunnerAutoscaler so that its Spec gets updated only during a certain period of time.
+
+usually, this feature is used for following scenarios:
+
+- You want to pay for your infrastructure cost running runners only in business hours
+- You want to prepare for scheduled spikes in workloads
+
+For the first scenario, you might consider configuration like the below:
+
+```
+apiVersion: actions.summerwind.dev/v1alpha1
+kind: HorizontalRunnerAutoscaler
+metadata:
+  name: example-runner-deployment-autoscaler
+spec:
+  scaleTargetRef:
+    name: example-runner-deployment
+  scheduledOverrides:
+  # Override minReplicas to 0 only between 0am sat to 0am mon
+  - startTime: "2021-05-01T00:00:00+09:00"
+    endTime: "2021-05-03T00:00:00+09:00"
+    recurrenceRule:
+      frequency: Weekly
+      untilTime: "2022-05-01T00:00:00+09:00"
+    minReplicas: 0
+  minReplicas: 1
+```
+
+For the second scenario, you might consider something like the below:
+
+```
+apiVersion: actions.summerwind.dev/v1alpha1
+kind: HorizontalRunnerAutoscaler
+metadata:
+  name: example-runner-deployment-autoscaler
+spec:
+  scaleTargetRef:
+    name: example-runner-deployment
+  scheduledOverrides:
+  # Override minReplicas to 100 only between 2021-06-01T00:00:00+09:00 and 2021-06-03T00:00:00+09:00
+  - startTime: "2021-06-01T00:00:00+09:00"
+    endTime: "2021-06-03T00:00:00+09:00"
+    minReplicas: 100
+  minReplicas: 1
+```
+
+The most basic usage of this feature is actually the second scenario mentioned above.
+A scheduled override without `recurrenceRule` is considered a one-off override, that is active between `startTime` and `endTime`. In the second scenario, it overrides `minReplicas` to `100` only between `2021-06-01T00:00:00+09:00` and `2021-06-03T00:00:00+09:00`.
+
+A scheduled override with `recurrenceRule` is considered a recurring override. A recurring override is initially active between `startTime` and `endTime`, and then it repeatedly get activated after a certain period of time denoted by `frequency`.
+
+`frequecy` can take one of the following values:
+
+- `Daily`
+- `Weekly`
+- `Monthly`
+- `Yearly`
+
+By default, a scheduled override repeats forever. If you want it to repeat until a specific point in time, define `untilTime`. The controller create the last recurrence of the override until the recurrence's `startTime` is equal or earlier than `untilTime`.
+
+Do note that you have enough slack for `untilTime`, so that a delayed or offline `actions-runner-controller` is much less likely to miss the last recurrence. For example, you might want to set `untilTime` to `M` minutes after the last recurrence's `startTime`, so that `actions-runner-controller` being offline up to `M` minutes doesn't miss the last recurrence.
+
+**Combining Multiple Scheduled Overrides**:
+
+In case you have a more complex scenarios, try writing two or more entries under `scheduledOverrides`.
+
+The earlier entry is prioritized higher than later entries. So you usually define one-time overrides in the top of your list, then yearly, monthly, weekly, and lastly daily overrides.
+
 ### Runner with DinD
 
 When using default runner, runner pod starts up 2 containers: runner and DinD (Docker-in-Docker). This might create issues if there's `LimitRange` set to namespace.
@@ -547,7 +719,7 @@ spec:
 
 This also helps with resources, as you don't need to give resources separately to docker and runner.
 
-### Additional tweaks
+### Additional Tweaks
 
 You can pass details through the spec selector. Here's an eg. of what you may like to do:
 
@@ -560,10 +732,22 @@ metadata:
 spec:
   replicas: 2
   template:
+    metadata:
+      annotations:
+        cluster-autoscaler.kubernetes.io/safe-to-evict: "true"
     spec:
       nodeSelector:
         node-role.kubernetes.io/test: ""
-
+      
+      securityContext:
+        #All level/role/type/user values will vary based on your SELinux policies.
+        #See https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux_atomic_host/7/html/container_security_guide/docker_selinux_security_policy for information about SELinux with containers
+        seLinuxOptions: 
+          level: "s0"
+          role: "system_r"
+          type: "super_t"
+          user: "system_u"
+          
       tolerations:
       - effect: NoSchedule
         key: node-role.kubernetes.io/test
@@ -585,12 +769,29 @@ spec:
           operator: "Exists"
           effect: "NoExecute"
           tolerationSeconds: 10
+      # true (default) = The runner restarts after running jobs, to ensure a clean and reproducible build environment
+      # false = The runner is persistent across jobs and doesn't automatically restart
+      # This directly controls the behaviour of `--once` flag provided to the github runner
+      ephemeral: false 
       # true (default) = A privileged docker sidecar container is included in the runner pod.
       # false = A docker sidecar container is not included in the runner pod and you can't use docker.
       # If set to false, there are no privileged container and you cannot use docker.
       dockerEnabled: false
+      # Optional Docker containers network MTU
+      # If your network card MTU is smaller than Docker's default 1500, you might encounter Docker networking issues.
+      # To fix these issues, you should setup Docker MTU smaller than or equal to that on the outgoing network card.
+      # More information:
+      # - https://mlohr.com/docker-mtu/
+      dockerMTU: 1500
+      # Optional Docker registry mirror
+      # Docker Hub has enabled rate-limiting for free plans.
+      # To avoid disruptions in your CI/CD pipelines, you might want to setup an external or on-premises Docker registry mirror.
+      # More information:
+      # - https://docs.docker.com/docker-hub/download-rate-limit/
+      # - https://cloud.google.com/container-registry/docs/pulling-cached-images
+      dockerRegistryMirror: https://mirror.gcr.io/
       # false (default) = Docker support is provided by a sidecar container deployed in the runner pod.
-      # true = No docker sidecar container is deployed in the runner pod but docker can be used within teh runner container instead. The image summerwind/actions-runner-dind is used by default.
+      # true = No docker sidecar container is deployed in the runner pod but docker can be used within the runner container instead. The image summerwind/actions-runner-dind is used by default.
       dockerdWithinRunnerContainer: true
       # Docker sidecar container image tweaks examples below, only applicable if dockerdWithinRunnerContainer = false
       dockerdContainerResources:
@@ -613,9 +814,24 @@ spec:
       # You can customise this setting allowing you to change the default working directory location
       # for example, the below setting is the same as on the ubuntu-18.04 image
       workDir: /home/runner/work
+      # You can mount some of the shared volumes to the dind container using dockerVolumeMounts, like any other volume mounting.
+      # NOTE: in case you want to use an hostPath like the following example, make sure that Kubernetes doesn't schedule more than one runner
+      # per physical host. You can achieve that by setting pod anti-affinity rules and/or resource requests/limits.
+      volumes:
+        - name: docker-extra
+          hostPath:
+            path: /mnt/docker-extra
+            type: DirectoryOrCreate
+      dockerVolumeMounts:
+        - mountPath: /var/lib/docker
+          name: docker-extra
+      # Optional name of the container runtime configuration that should be used for pods.
+      # This must match the name of a RuntimeClass resource available on the cluster.
+      # More info: https://kubernetes.io/docs/concepts/containers/runtime-class
+      runtimeClassName: "runc"
 ```
 
-### Runner labels
+### Runner Labels
 
 To run a workflow job on a self-hosted runner, you can use the following syntax in your workflow:
 
@@ -654,7 +870,7 @@ Note that if you specify `self-hosted` in your workflow, then this will run your
 
 ### Runner Groups
 
-Runner groups can be used to limit which repositories are able to use the GitHub Runner at an Organisation level. Runner groups have to be [created in GitHub first](https://docs.github.com/en/actions/hosting-your-own-runners/managing-access-to-self-hosted-runners-using-groups) before they can be referenced.
+Runner groups can be used to limit which repositories are able to use the GitHub Runner at an organization level. Runner groups have to be [created in GitHub first](https://docs.github.com/en/actions/hosting-your-own-runners/managing-access-to-self-hosted-runners-using-groups) before they can be referenced.
 
 To add the runner to the group `NewGroup`, specify the group in your `Runner` or `RunnerDeployment` spec.
 
@@ -671,15 +887,14 @@ spec:
       group: NewGroup
 ```
 
-### Using EKS IAM role for service accounts
+### Using IRSA (IAM Roles for Service Accounts) in EKS
 
-`actions-runner-controller` v0.15.0 or later has support for EKS IAM role for service accounts.
+`actions-runner-controller` v0.15.0 or later has support for IRSA in EKS.
 
 As similar as for regular pods and deployments, you firstly need an existing service account with the IAM role associated.
 Create one using e.g. `eksctl`. You can refer to [the EKS documentation](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html) for more details.
 
-Once you set up the service account, all you need is to add `serviceAccountName` and `fsGroup` to any pods that uses
-the IAM-role enabled service account.
+Once you set up the service account, all you need is to add `serviceAccountName` and `fsGroup` to any pods that uses the IAM-role enabled service account.
 
 For `RunnerDeployment`, you can set those two fields under the runner spec at `RunnerDeployment.Spec.Template`:
 
@@ -694,17 +909,21 @@ spec:
       repository: USER/REO
       serviceAccountName: my-service-account
       securityContext:
-        fsGroup: 1447
+        fsGroup: 1000
 ```
 
-### Software installed in the runner image
+### Software Installed in the Runner Image
 
-The GitHub hosted runners include a large amount of pre-installed software packages. For Ubuntu 18.04, this list can be found at <https://github.com/actions/virtual-environments/blob/master/images/linux/Ubuntu1804-README.md>
+**Cloud Tooling**<br />
+The project supports being deployed on the various cloud Kubernetes platforms (e.g. EKS), it does not however aim to go beyond that. No cloud specific tooling is bundled in the base runner, this is an active decision to keep the overhead of maintaining the solution manageable.
 
-The container image is based on Ubuntu 18.04, but it does not contain all of the software installed on the GitHub runners. It contains the following subset of packages from the GitHub runners:
+**Bundled Software**<br />
+The GitHub hosted runners include a large amount of pre-installed software packages. GitHub maintain a list in README files at <https://github.com/actions/virtual-environments/tree/main/images/linux>
+
+This solution maintains a few runner images with `latest` aligning with GitHub's Ubuntu version. Older images are maintained whilst GitHub also provides them as an option. These images do not contain all of the software installed on the GitHub runners. It contains the following subset of packages from the GitHub runners:
 
 - Basic CLI packages
-- git (2.26)
+- git
 - docker
 - build-essentials
 
@@ -743,10 +962,68 @@ spec:
 **Solutions**<br />
 Your base64'ed PAT token has a new line at the end, it needs to be created without a `\n` added
 * `echo -n $TOKEN | base64`
-* Create the secret as described in the docs using the shell and documeneted flags
+* Create the secret as described in the docs using the shell and documented flags
 
-# Developing
+#### Runner coming up before network available
 
+If you're running your action runners on a service mesh like Istio, you might
+have problems with runner configuration accompanied by logs like:
+
+```
+....
+runner Starting Runner listener with startup type: service
+runner Started listener process
+runner An error occurred: Not configured
+runner Runner listener exited with error code 2
+runner Runner listener exit with retryable error, re-launch runner in 5 seconds.
+....
+```
+
+This is because the `istio-proxy` has not completed configuring itself when the
+configuration script tries to communicate with the network.
+
+**Solution**<br />
+
+> This feature is experimental and will be dropped once maintainers think that
+> everyone has already migrated to use Istio's `holdApplicationUntilProxyStarts` ([istio/istio#11130](https://github.com/istio/istio/issues/11130)).
+>
+> Please read the discussion in #592 for more information.
+
+You can add a delay to the entrypoint script by setting the `STARTUP_DELAY` environment
+variable. This will cause the script to sleep `STARTUP_DELAY` seconds.
+
+*Example `Runner` with a 2 second startup delay:*
+```yaml
+apiVersion: actions.summerwind.dev/v1alpha1
+kind: Runner
+metadata:
+  name: example-runner-with-sleep
+spec:
+  env:
+    - name: STARTUP_DELAY
+      value: "2" # Remember! env var values must be strings.  
+```
+
+*Example `RunnerDeployment` with a 2 second startup delay:*
+```yaml
+apiVersion: actions.summerwind.dev/v1alpha1
+kind: RunnerDeployment
+metadata:
+  name: example-runnerdeployment-with-sleep
+spec:
+  template:
+    spec:
+      env:
+        - name: STARTUP_DELAY
+          value: "2" # Remember! env var values must be strings.
+```
+
+
+# Contributing
+
+For more details about any requirements or process, please check out [Getting Started with Contributing](CONTRIBUTING.md).
+
+**The Controller**<br />
 If you'd like to modify the controller to fork or contribute, I'd suggest using the following snippet for running
 the acceptance test:
 
@@ -754,14 +1031,23 @@ the acceptance test:
 # This sets `VERSION` envvar to some appropriate value
 . hack/make-env.sh
 
-NAME=$DOCKER_USER/actions-runner-controller \
+DOCKER_USER=*** \
   GITHUB_TOKEN=*** \
   APP_ID=*** \
   PRIVATE_KEY_FILE_PATH=path/to/pem/file \
   INSTALLATION_ID=*** \
-  make docker-build docker-push acceptance
+  make acceptance
 ```
 
+> **Notes for Ubuntu 20.04+ users**
+>
+> If you're using Ubuntu 20.04 or greater, you might have installed `docker` with `snap`.
+>
+> If you want to stick with `snap`-provided `docker`, do not forget to set `TMPDIR` to
+> somewhere under `$HOME`.
+> Otherwise `kind load docker-image` fail while running `docker save`.
+> See https://kind.sigs.k8s.io/docs/user/known-issues/#docker-installed-with-snap for more information.
+
 Please follow the instructions explained in [Using Personal Access Token](#using-personal-access-token) to obtain
 `GITHUB_TOKEN`, and those in [Using GitHub App](#using-github-app) to obtain `APP_ID`, `INSTALLATION_ID`, and
 `PRIAVTE_KEY_FILE_PATH`.
@@ -770,29 +1056,103 @@ The test creates a one-off `kind` cluster, deploys `cert-manager` and `actions-r
 creates a `RunnerDeployment` custom resource for a public Git repository to confirm that the
 controller is able to bring up a runner pod with the actions runner registration token installed.
 
+**Rerunning a failed test**
+
+When one of tests run by `make acceptance` failed, you'd probably like to rerun only the failed one.
+
+It can be done by `make acceptance/run` and by setting the combination of `ACCEPTANCE_TEST_DEPLOYMENT_TOOL` and `ACCEPTANCE_TEST_SECRET_TYPE` values that failed.
+
+In the example below, we rerun the test for the combination `ACCEPTANCE_TEST_DEPLOYMENT_TOOL=helm ACCEPTANCE_TEST_SECRET_TYPE=token` only:
+
+```
+DOCKER_USER=*** \
+  GITHUB_TOKEN=*** \
+  APP_ID=*** \
+  PRIVATE_KEY_FILE_PATH=path/to/pem/file \
+  INSTALLATION_ID=*** \
+  ACCEPTANCE_TEST_DEPLOYMENT_TOOL=helm ACCEPTANCE_TEST_SECRET_TYPE=token \
+  make acceptance/run
+```
+
+**Testing in a non-kind cluster**
+
 If you prefer to test in a non-kind cluster, you can instead run:
 
 ```shell script
 KUBECONFIG=path/to/kubeconfig \
-NAME=$DOCKER_USER/actions-runner-controller \
+  DOCKER_USER=*** \
   GITHUB_TOKEN=*** \
   APP_ID=*** \
   PRIVATE_KEY_FILE_PATH=path/to/pem/file \
   INSTALLATION_ID=*** \
   ACCEPTANCE_TEST_SECRET_TYPE=token \
-  make docker-build docker-push \
-       acceptance/setup acceptance/tests
+  make docker-build acceptance/setup \
+       acceptance/deploy \
+       acceptance/tests
 ```
-# Alternatives
 
-The following is a list of alternative solutions that may better fit you depending on your use-case:
+**Development Tips**
 
-- <https://github.com/evryfs/github-actions-runner-operator/>
-- <https://github.com/philips-labs/terraform-aws-github-runner/>
+Rerunning the whole acceptance test suite from scratch on every little change to the controller, the runner, and the chart would be counter-productive.
 
-Although the situation can change over time, as of writing this sentence, the benefits of using `actions-runner-controller` over the alternatives are:
+To make your development cycle faster, use the below command to update deploy and update all the three:
 
-- `actions-runner-controller` has the ability to autoscale runners based on number of pending/progressing jobs (#99)
-- `actions-runner-controller` is able to gracefully stop runners (#103)
-- `actions-runner-controller` has ARM support
-- `actions-runner-controller` has GitHub Enterprise support (see [GitHub Enterprise support](#github-enterprise-support) section for caveats)
+```
+# Let assume we have all other envvars like DOCKER_USER, GITHUB_TOKEN already set,
+# The below command will (re)build `actions-runner-controller:controller1` and `actions-runner:runner1`,
+# load those into kind nodes, and then rerun kubectl or helm to install/upgrade the controller,
+# and finally upgrade the runner deployment to use the new runner image.
+#
+# As helm 3 and kubectl is unable to recreate a pod when no tag change,
+# you either need to bump VERSION and RUNNER_TAG on each run,
+# or manually run `kubectl delete pod $POD` on respective pods for changes to actually take effect.
+VERSION=controller1 \
+  RUNNER_TAG=runner1 \
+  make docker-build acceptance/load acceptance/deploy
+```
+
+If you've already deployed actions-runner-controller and only want to recreate pods to use the newer image, you can run:
+
+```
+NAME=$DOCKER_USER/actions-runner-controller \
+  make docker-build acceptance/load && \
+  kubectl -n actions-runner-system delete po $(kubectl -n actions-runner-system get po -ojsonpath={.items[*].metadata.name})
+```
+
+Similarly, if you'd like to recreate runner pods with the newer runner image,
+
+```
+NAME=$DOCKER_USER/actions-runner make \
+  -C runner docker-{build,push}-ubuntu && \
+  (kubectl get po -ojsonpath={.items[*].metadata.name} | xargs -n1 kubectl delete po)
+```
+
+**Runner Tests**<br />
+A set of example pipelines (./acceptance/pipelines) are provided in this repository which you can use to validate your runners are working as expected. When raising a PR please run the relevant suites to prove your change hasn't broken anything.
+
+**Running Ginkgo Tests**
+
+You can run the integration test suite that is written in Ginkgo with:
+
+```bash
+make test-with-deps
+```
+
+This will firstly install a few binaries required to setup the integration test environment and then runs `go test` to start the Ginkgo test.
+
+If you don't want to use `make`, like when you're running tests from your IDE, install required binaries to `/usr/local/kubebuilder/bin`. That's the directory in which controller-runtime's `envtest` framework locates the binaries.
+
+```bash
+sudo mkdir -p /usr/local/kubebuilder/bin
+make kube-apiserver etcd
+sudo mv test-assets/{etcd,kube-apiserver} /usr/local/kubebuilder/bin/
+go test -v -run TestAPIs github.com/summerwind/actions-runner-controller/controllers
+```
+
+To run Ginkgo tests selectively, set the pattern of target test names to `GINKGO_FOCUS`.
+All the Ginkgo test that matches `GINKGO_FOCUS` will be run.
+
+```bash
+GINKGO_FOCUS='[It] should create a new Runner resource from the specified template, add a another Runner on replicas increased, and removes all the replicas when set to 0' \
+  go test -v -run TestAPIs github.com/summerwind/actions-runner-controller/controllers
+```

acceptance/checks.sh

@@ -12,6 +12,9 @@ done
 
 echo Found runner ${runner_name}.
 
+# Wait a bit to make sure the runner pod is created before looking for it.
+sleep 2
+
 pod_name=
 
 while [ -z "${pod_name}" ]; do
@@ -24,6 +27,6 @@ echo Found pod ${pod_name}.
 
 echo Waiting for pod ${runner_name} to become ready... 1>&2
 
-kubectl wait pod/${runner_name} --for condition=ready --timeout 180s
+kubectl wait pod/${runner_name} --for condition=ready --timeout 270s
 
 echo All tests passed. 1>&2

acceptance/deploy.sh

@@ -4,10 +4,14 @@ set -e
 
 tpe=${ACCEPTANCE_TEST_SECRET_TYPE}
 
+VALUES_FILE=${VALUES_FILE:-$(dirname $0)/values.yaml}
+
 if [ "${tpe}" == "token" ]; then
-  kubectl create secret generic controller-manager \
-	  -n actions-runner-system \
-	  --from-literal=github_token=${GITHUB_TOKEN:?GITHUB_TOKEN must not be empty}
+  if ! kubectl get secret controller-manager -n actions-runner-system >/dev/null; then
+    kubectl create secret generic controller-manager \
+      -n actions-runner-system \
+      --from-literal=github_token=${GITHUB_TOKEN:?GITHUB_TOKEN must not be empty}
+  fi
 elif [ "${tpe}" == "app" ]; then
   kubectl create secret generic controller-manager \
     -n actions-runner-system \
@@ -26,17 +30,37 @@ if [ "${tool}" == "helm" ]; then
     charts/actions-runner-controller \
     -n actions-runner-system \
     --create-namespace \
-    --set syncPeriod=5m
-  kubectl -n actions-runner-system wait deploy/actions-runner-controller --for condition=available
+    --set syncPeriod=${SYNC_PERIOD} \
+    --set authSecret.create=false \
+    --set image.repository=${NAME} \
+    --set image.tag=${VERSION} \
+    -f ${VALUES_FILE}
+  kubectl -n actions-runner-system wait deploy/actions-runner-controller --for condition=available --timeout 60s
 else
   kubectl apply \
     -n actions-runner-system \
     -f release/actions-runner-controller.yaml
-  kubectl -n actions-runner-system wait deploy/controller-manager --for condition=available --timeout 60s
+  kubectl -n actions-runner-system wait deploy/controller-manager --for condition=available --timeout 120s
 fi
 
 # Adhocly wait for some time until actions-runner-controller's admission webhook gets ready
 sleep 20
 
-kubectl apply \
-  -f acceptance/testdata/runnerdeploy.yaml
+if [ -n "${TEST_REPO}" ]; then
+  cat acceptance/testdata/runnerdeploy.yaml | envsubst | kubectl apply -f -
+  cat acceptance/testdata/hra.yaml | envsubst | kubectl apply -f -
+else
+  echo 'Skipped deploying runnerdeployment and hra. Set TEST_REPO to "yourorg/yourrepo" to deploy.'
+fi
+
+if [ -n "${TEST_ORG}" ]; then
+  cat acceptance/testdata/org.runnerdeploy.yaml | envsubst | kubectl apply -f -
+
+  if [ -n "${TEST_ORG_REPO}" ]; then
+    cat acceptance/testdata/org.hra.yaml | envsubst | kubectl apply -f -
+  else
+    echo 'Skipped deploying organizational hra. Set TEST_ORG_REPO to "yourorg/yourrepo" to deploy.'
+  fi
+else
+  echo 'Skipped deploying organizational runnerdeployment. Set TEST_ORG to deploy.'
+fi

acceptance/kind.yaml

@@ -0,0 +1,10 @@
+apiVersion: kind.x-k8s.io/v1alpha4
+kind: Cluster
+nodes:
+- role: control-plane
+  extraPortMappings:
+  - containerPort: 31000
+    hostPort: 31000
+    listenAddress: "0.0.0.0"
+    protocol: tcp
+#- role: worker

acceptance/pipelines/eks-integration-tests.yaml

@@ -0,0 +1,36 @@
+name: EKS Integration Tests
+
+on:
+  workflow_dispatch:
+
+env:
+  IRSA_ROLE_ARN:
+  ASSUME_ROLE_ARN: 
+  AWS_REGION: 
+
+jobs:
+  assume-role-in-runner-test:
+    runs-on: ['self-hosted', 'Linux']
+    steps:
+      - name: Test aws-actions/configure-aws-credentials Action
+        uses: aws-actions/configure-aws-credentials@v1
+        with:
+          aws-region: ${{ env.AWS_REGION }}
+          role-to-assume: ${{ env.ASSUME_ROLE_ARN }}
+          role-duration-seconds: 900
+  assume-role-in-container-test:
+    runs-on: ['self-hosted', 'Linux']
+    container: 
+      image: amazon/aws-cli
+      env:
+        AWS_WEB_IDENTITY_TOKEN_FILE: /var/run/secrets/eks.amazonaws.com/serviceaccount/token
+        AWS_ROLE_ARN: ${{ env.IRSA_ROLE_ARN }}
+      volumes:
+        - /var/run/secrets/eks.amazonaws.com/serviceaccount/token:/var/run/secrets/eks.amazonaws.com/serviceaccount/token
+    steps:
+      - name: Test aws-actions/configure-aws-credentials Action in container
+        uses: aws-actions/configure-aws-credentials@v1
+        with:
+          aws-region: ${{ env.AWS_REGION }}
+          role-to-assume: ${{ env.ASSUME_ROLE_ARN }}
+          role-duration-seconds: 900

acceptance/pipelines/runner-integration-tests.yaml

@@ -0,0 +1,83 @@
+name: Runner Integration Tests
+
+on:
+  workflow_dispatch:
+
+env:
+  ImageOS: ubuntu18 # Used by ruby/setup-ruby action | Update me for the runner OS version you are testing against
+
+jobs:
+  run-step-in-container-test:
+    runs-on: ['self-hosted', 'Linux']
+    container: 
+      image: alpine
+    steps:
+      - name: Test we are working in the container
+        run: |
+          if [[ $(sed -n '2p' < /etc/os-release | cut -d "=" -f2) != "alpine" ]]; then
+              echo "::error ::Failed OS detection test, could not match /etc/os-release with alpine. Are we really running in the container?"
+              echo "/etc/os-release below:"
+              cat /etc/os-release
+              exit 1
+          fi
+  setup-python-test:
+    runs-on: ['self-hosted', 'Linux']
+    steps:
+      - name: Print native Python environment
+        run: |
+          which python
+          python --version
+      - uses: actions/setup-python@v2
+        with:
+          python-version: 3.9
+      - name: Test actions/setup-python works
+        run: |
+          VERSION=$(python --version 2>&1 | cut -d ' ' -f2 | cut -d '.' -f1-2)
+          if [[ $VERSION != '3.9' ]]; then
+            echo "Python version detected : $(python --version 2>&1)"
+            echo "::error ::Detected python failed setup version test, could not match version with version specified in the setup action"
+            exit 1
+          else
+            echo "Python version detected : $(python --version 2>&1)"
+          fi
+  setup-node-test:
+    runs-on: ['self-hosted', 'Linux']
+    steps:
+      - uses: actions/setup-node@v2
+        with:
+          node-version: '12'
+      - name: Test actions/setup-node works 
+        run: |
+          VERSION=$(node --version | cut -c 2- | cut -d '.' -f1)
+          if [[ $VERSION != '12' ]]; then
+            echo "Node version detected : $(node --version 2>&1)"
+            echo "::error ::Detected node failed setup version test, could not match version with version specified in the setup action"
+            exit 1
+          else
+            echo "Node version detected : $(node --version 2>&1)"
+          fi
+  setup-ruby-test:
+    runs-on: ['self-hosted', 'Linux']
+    steps:
+      - uses: ruby/setup-ruby@v1
+        with:
+          ruby-version: 3.0
+          bundler-cache: true
+      - name: Test ruby/setup-ruby works 
+        run: |
+          VERSION=$(ruby --version | cut -d ' ' -f2 | cut -d '.' -f1-2)
+          if [[ $VERSION != '3.0' ]]; then
+              echo "Ruby version detected : $(ruby --version 2>&1)"
+              echo "::error ::Detected ruby failed setup version test, could not match version with version specified in the setup action"
+              exit 1
+          else
+              echo "Ruby version detected : $(ruby --version 2>&1)"
+          fi
+  python-shell-test:
+    runs-on: ['self-hosted', 'Linux']
+    steps:      
+      - name: Test Python shell works
+        run: |
+          import os
+          print(os.environ['PATH'])
+        shell: python

acceptance/testdata/hra.yaml

@@ -0,0 +1,25 @@
+apiVersion: actions.summerwind.dev/v1alpha1
+kind: HorizontalRunnerAutoscaler
+metadata:
+  name: actions-runner-aos-autoscaler
+spec:
+  scaleTargetRef:
+    name: example-runnerdeploy
+  scaleUpTriggers:
+  - githubEvent:
+      checkRun:
+        types: ["created"]
+        status: "queued"
+    amount: 1
+    duration: "1m"
+  minReplicas: 0
+  maxReplicas: 5
+  metrics:
+  - type: PercentageRunnersBusy
+    scaleUpThreshold: '0.75'
+    scaleDownThreshold: '0.3'
+    scaleUpFactor: '2'
+    scaleDownFactor: '0.5'
+  - type: TotalNumberOfQueuedAndInProgressWorkflowRuns
+    repositoryNames:
+    - ${TEST_REPO}

acceptance/testdata/org.hra.yaml

@@ -0,0 +1,35 @@
+apiVersion: actions.summerwind.dev/v1alpha1
+kind: HorizontalRunnerAutoscaler
+metadata:
+  name: org
+spec:
+  scaleTargetRef:
+    name: org-runnerdeploy
+  scaleUpTriggers:
+  - githubEvent:
+      checkRun:
+        types: ["created"]
+        status: "queued"
+    amount: 1
+    duration: "1m"
+  scheduledOverrides:
+  - startTime: "2021-05-11T16:05:00+09:00"
+    endTime: "2021-05-11T16:40:00+09:00"
+    minReplicas: 2
+  - startTime: "2021-05-01T00:00:00+09:00"
+    endTime: "2021-05-03T00:00:00+09:00"
+    recurrenceRule:
+      frequency: Weekly
+      untilTime: "2022-05-01T00:00:00+09:00"
+    minReplicas: 0
+  minReplicas: 0
+  maxReplicas: 5
+  metrics:
+  - type: PercentageRunnersBusy
+    scaleUpThreshold: '0.75'
+    scaleDownThreshold: '0.3'
+    scaleUpFactor: '2'
+    scaleDownFactor: '0.5'
+  - type: TotalNumberOfQueuedAndInProgressWorkflowRuns
+    repositoryNames:
+    - ${TEST_ORG_REPO}

acceptance/testdata/org.runnerdeploy.yaml

@@ -0,0 +1,37 @@
+apiVersion: actions.summerwind.dev/v1alpha1
+kind: RunnerDeployment
+metadata:
+  name: org-runnerdeploy
+spec:
+#  replicas: 1
+  template:
+    spec:
+      organization: ${TEST_ORG}
+
+      #
+      # Custom runner image
+      #
+      image: ${RUNNER_NAME}:${RUNNER_TAG}
+      imagePullPolicy: IfNotPresent
+
+      #
+      # dockerd within runner container
+      #
+      ## Replace `mumoshu/actions-runner-dind:dev` with your dind image
+      #dockerdWithinRunnerContainer: true
+      #image: mumoshu/actions-runner-dind:dev
+
+      #
+      # Set the MTU used by dockerd-managed network interfaces (including docker-build-ubuntu)
+      #
+      #dockerMTU: 1450
+
+      #Runner group
+      # labels:
+      # - "mylabel 1"
+      # - "mylabel 2"
+
+      #
+      # Non-standard working directory
+      #
+      # workDir: "/"

acceptance/testdata/runnerdeploy.yaml

@@ -6,4 +6,32 @@ spec:
 #  replicas: 1
   template:
     spec:
-      repository: mumoshu/actions-runner-controller-ci
+      repository: ${TEST_REPO}
+
+      #
+      # Custom runner image
+      #
+      image: ${RUNNER_NAME}:${RUNNER_TAG}
+      imagePullPolicy: IfNotPresent
+
+      #
+      # dockerd within runner container
+      #
+      ## Replace `mumoshu/actions-runner-dind:dev` with your dind image
+      #dockerdWithinRunnerContainer: true
+      #image: mumoshu/actions-runner-dind:dev
+
+      #
+      # Set the MTU used by dockerd-managed network interfaces (including docker-build-ubuntu)
+      #
+      #dockerMTU: 1450
+
+      #Runner group
+      # labels:
+      # - "mylabel 1"
+      # - "mylabel 2"
+
+      #
+      # Non-standard working directory
+      #
+      # workDir: "/"

acceptance/values.yaml

@@ -0,0 +1,20 @@
+# Set actions-runner-controller settings for testing
+githubAPICacheDuration: 10s
+githubWebhookServer:
+  enabled: true
+  labels: {}
+  replicaCount: 1
+  syncPeriod: 10m
+  secret:
+    create: true
+    name: "github-webhook-server"
+    ### GitHub Webhook Configuration
+    #github_webhook_secret_token: ""
+  service:
+    type: NodePort
+    ports:
+      - port: 80
+        targetPort: http
+        protocol: TCP
+        name: http
+        nodePort: 31000

api/v1alpha1/horizontalrunnerautoscaler_types.go

@@ -54,6 +54,12 @@ type HorizontalRunnerAutoscalerSpec struct {
 	ScaleUpTriggers []ScaleUpTrigger `json:"scaleUpTriggers,omitempty"`
 
 	CapacityReservations []CapacityReservation `json:"capacityReservations,omitempty" patchStrategy:"merge" patchMergeKey:"name"`
+
+	// ScheduledOverrides is the list of ScheduledOverride.
+	// It can be used to override a few fields of HorizontalRunnerAutoscalerSpec on schedule.
+	// The earlier a scheduled override is, the higher it is prioritized.
+	// +optional
+	ScheduledOverrides []ScheduledOverride `json:"scheduledOverrides,omitempty"`
 }
 
 type ScaleUpTrigger struct {
@@ -144,6 +150,40 @@ type MetricSpec struct {
 	ScaleDownAdjustment int `json:"scaleDownAdjustment,omitempty"`
 }
 
+// ScheduledOverride can be used to override a few fields of HorizontalRunnerAutoscalerSpec on schedule.
+// A schedule can optionally be recurring, so that the correspoding override happens every day, week, month, or year.
+type ScheduledOverride struct {
+	// StartTime is the time at which the first override starts.
+	StartTime metav1.Time `json:"startTime"`
+
+	// EndTime is the time at which the first override ends.
+	EndTime metav1.Time `json:"endTime"`
+
+	// MinReplicas is the number of runners while overriding.
+	// If omitted, it doesn't override minReplicas.
+	// +optional
+	// +nullable
+	// +kubebuilder:validation:Minimum=0
+	MinReplicas *int `json:"minReplicas,omitempty"`
+
+	// +optional
+	RecurrenceRule RecurrenceRule `json:"recurrenceRule,omitempty"`
+}
+
+type RecurrenceRule struct {
+	// Frequency is the name of a predefined interval of each recurrence.
+	// The valid values are "Daily", "Weekly", "Monthly", and "Yearly".
+	// If empty, the corresponding override happens only once.
+	// +optional
+	// +kubebuilder:validation:Enum=Daily;Weekly;Monthly;Yearly
+	Frequency string `json:"frequency,omitempty"`
+
+	// UntilTime is the time of the final recurrence.
+	// If empty, the schedule recurs forever.
+	// +optional
+	UntilTime metav1.Time `json:"untilTime,omitempty"`
+}
+
 type HorizontalRunnerAutoscalerStatus struct {
 	// ObservedGeneration is the most recent generation observed for the target. It corresponds to e.g.
 	// RunnerDeployment's generation, which is updated on mutation by the API Server.
@@ -161,6 +201,11 @@ type HorizontalRunnerAutoscalerStatus struct {
 
 	// +optional
 	CacheEntries []CacheEntry `json:"cacheEntries,omitempty"`
+
+	// ScheduledOverridesSummary is the summary of active and upcoming scheduled overrides to be shown in e.g. a column of a `kubectl get hra` output
+	// for observability.
+	// +optional
+	ScheduledOverridesSummary *string `json:"scheduledOverridesSummary,omitempty"`
 }
 
 const CacheEntryKeyDesiredReplicas = "desiredReplicas"
@@ -176,6 +221,7 @@ type CacheEntry struct {
 // +kubebuilder:printcolumn:JSONPath=".spec.minReplicas",name=Min,type=number
 // +kubebuilder:printcolumn:JSONPath=".spec.maxReplicas",name=Max,type=number
 // +kubebuilder:printcolumn:JSONPath=".status.desiredReplicas",name=Desired,type=number
+// +kubebuilder:printcolumn:JSONPath=".status.scheduledOverridesSummary",name=Schedule,type=string
 
 // HorizontalRunnerAutoscaler is the Schema for the horizontalrunnerautoscaler API
 type HorizontalRunnerAutoscaler struct {

api/v1alpha1/runner_types.go

@@ -19,6 +19,8 @@ package v1alpha1
 import (
 	"errors"
 
+	"k8s.io/apimachinery/pkg/api/resource"
+
 	corev1 "k8s.io/api/core/v1"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 )
@@ -43,11 +45,16 @@ type RunnerSpec struct {
 	// +optional
 	Group string `json:"group,omitempty"`
 
+	// +optional
+	Ephemeral *bool `json:"ephemeral,omitempty"`
+
 	// +optional
 	Containers []corev1.Container `json:"containers,omitempty"`
 	// +optional
 	DockerdContainerResources corev1.ResourceRequirements `json:"dockerdContainerResources,omitempty"`
 	// +optional
+	DockerVolumeMounts []corev1.VolumeMount `json:"dockerVolumeMounts,omitempty"`
+	// +optional
 	Resources corev1.ResourceRequirements `json:"resources,omitempty"`
 	// +optional
 	VolumeMounts []corev1.VolumeMount `json:"volumeMounts,omitempty"`
@@ -94,6 +101,17 @@ type RunnerSpec struct {
 	DockerEnabled *bool `json:"dockerEnabled,omitempty"`
 	// +optional
 	DockerMTU *int64 `json:"dockerMTU,omitempty"`
+	// +optional
+	DockerRegistryMirror *string `json:"dockerRegistryMirror,omitempty"`
+	// +optional
+	HostAliases []corev1.HostAlias `json:"hostAliases,omitempty"`
+	// +optional
+	VolumeSizeLimit *resource.Quantity `json:"volumeSizeLimit,omitempty"`
+
+	// RuntimeClassName is the container runtime configuration that containers should run under.
+	// More info: https://kubernetes.io/docs/concepts/containers/runtime-class
+	// +optional
+	RuntimeClassName *string `json:"runtimeClassName,omitempty"`
 }
 
 // ValidateRepository validates repository field.
@@ -121,13 +139,17 @@ func (rs *RunnerSpec) ValidateRepository() error {
 
 // RunnerStatus defines the observed state of Runner
 type RunnerStatus struct {
+	// +optional
 	Registration RunnerStatusRegistration `json:"registration"`
-	Phase        string                   `json:"phase"`
-	Reason       string                   `json:"reason"`
-	Message      string                   `json:"message"`
-
-	//+optional
-	LastRegistrationCheckTime *metav1.Time `json:"lastRegistrationCheckTime"`
+	// +optional
+	Phase string `json:"phase,omitempty"`
+	// +optional
+	Reason string `json:"reason,omitempty"`
+	// +optional
+	Message string `json:"message,omitempty"`
+	// +optional
+	// +nullable
+	LastRegistrationCheckTime *metav1.Time `json:"lastRegistrationCheckTime,omitempty"`
 }
 
 // RunnerStatusRegistration contains runner registration status
@@ -147,6 +169,7 @@ type RunnerStatusRegistration struct {
 // +kubebuilder:printcolumn:JSONPath=".spec.repository",name=Repository,type=string
 // +kubebuilder:printcolumn:JSONPath=".spec.labels",name=Labels,type=string
 // +kubebuilder:printcolumn:JSONPath=".status.phase",name=Status,type=string
+// +kubebuilder:printcolumn:name="Age",type="date",JSONPath=".metadata.creationTimestamp"
 
 // Runner is the Schema for the runners API
 type Runner struct {

api/v1alpha1/runner_webhook.go

@@ -34,7 +34,7 @@ func (r *Runner) SetupWebhookWithManager(mgr ctrl.Manager) error {
 		Complete()
 }
 
-// +kubebuilder:webhook:path=/mutate-actions-summerwind-dev-v1alpha1-runner,verbs=create;update,mutating=true,failurePolicy=fail,groups=actions.summerwind.dev,resources=runners,versions=v1alpha1,name=mutate.runner.actions.summerwind.dev
+// +kubebuilder:webhook:path=/mutate-actions-summerwind-dev-v1alpha1-runner,verbs=create;update,mutating=true,failurePolicy=fail,groups=actions.summerwind.dev,resources=runners,versions=v1alpha1,name=mutate.runner.actions.summerwind.dev,sideEffects=None
 
 var _ webhook.Defaulter = &Runner{}
 
@@ -43,7 +43,7 @@ func (r *Runner) Default() {
 	// Nothing to do.
 }
 
-// +kubebuilder:webhook:path=/validate-actions-summerwind-dev-v1alpha1-runner,verbs=create;update,mutating=false,failurePolicy=fail,groups=actions.summerwind.dev,resources=runners,versions=v1alpha1,name=validate.runner.actions.summerwind.dev
+// +kubebuilder:webhook:path=/validate-actions-summerwind-dev-v1alpha1-runner,verbs=create;update,mutating=false,failurePolicy=fail,groups=actions.summerwind.dev,resources=runners,versions=v1alpha1,name=validate.runner.actions.summerwind.dev,sideEffects=None
 
 var _ webhook.Validator = &Runner{}
 

api/v1alpha1/runnerdeployment_types.go

@@ -38,20 +38,41 @@ type RunnerDeploymentSpec struct {
 }
 
 type RunnerDeploymentStatus struct {
-	AvailableReplicas int `json:"availableReplicas"`
-	ReadyReplicas     int `json:"readyReplicas"`
+	// See K8s deployment controller code for reference
+	// https://github.com/kubernetes/kubernetes/blob/ea0764452222146c47ec826977f49d7001b0ea8c/pkg/controller/deployment/sync.go#L487-L505
 
-	// Replicas is the total number of desired, non-terminated and latest pods to be set for the primary RunnerSet
+	// AvailableReplicas is the total number of available runners which have been successfully registered to GitHub and still running.
+	// This corresponds to the sum of status.availableReplicas of all the runner replica sets.
+	// +optional
+	AvailableReplicas *int `json:"availableReplicas"`
+
+	// ReadyReplicas is the total number of available runners which have been successfully registered to GitHub and still running.
+	// This corresponds to the sum of status.readyReplicas of all the runner replica sets.
+	// +optional
+	ReadyReplicas *int `json:"readyReplicas"`
+
+	// ReadyReplicas is the total number of available runners which have been successfully registered to GitHub and still running.
+	// This corresponds to status.replicas of the runner replica set that has the desired template hash.
+	// +optional
+	UpdatedReplicas *int `json:"updatedReplicas"`
+
+	// DesiredReplicas is the total number of desired, non-terminated and latest pods to be set for the primary RunnerSet
 	// This doesn't include outdated pods while upgrading the deployment and replacing the runnerset.
 	// +optional
-	Replicas *int `json:"desiredReplicas,omitempty"`
+	DesiredReplicas *int `json:"desiredReplicas"`
+
+	// Replicas is the total number of replicas
+	// +optional
+	Replicas *int `json:"replicas"`
 }
 
 // +kubebuilder:object:root=true
 // +kubebuilder:subresource:status
 // +kubebuilder:printcolumn:JSONPath=".spec.replicas",name=Desired,type=number
-// +kubebuilder:printcolumn:JSONPath=".status.availableReplicas",name=Current,type=number
-// +kubebuilder:printcolumn:JSONPath=".status.readyReplicas",name=Ready,type=number
+// +kubebuilder:printcolumn:JSONPath=".status.replicas",name=Current,type=number
+// +kubebuilder:printcolumn:JSONPath=".status.updatedReplicas",name=Up-To-Date,type=number
+// +kubebuilder:printcolumn:JSONPath=".status.availableReplicas",name=Available,type=number
+// +kubebuilder:printcolumn:name="Age",type="date",JSONPath=".metadata.creationTimestamp"
 
 // RunnerDeployment is the Schema for the runnerdeployments API
 type RunnerDeployment struct {

api/v1alpha1/runnerreplicaset_types.go

@@ -33,8 +33,19 @@ type RunnerReplicaSetSpec struct {
 }
 
 type RunnerReplicaSetStatus struct {
-	AvailableReplicas int `json:"availableReplicas"`
-	ReadyReplicas     int `json:"readyReplicas"`
+	// See K8s replicaset controller code for reference
+	// https://github.com/kubernetes/kubernetes/blob/ea0764452222146c47ec826977f49d7001b0ea8c/pkg/controller/replicaset/replica_set_utils.go#L101-L106
+
+	// Replicas is the number of runners that are created and still being managed by this runner replica set.
+	// +optional
+	Replicas *int `json:"replicas"`
+
+	// ReadyReplicas is the number of runners that are created and Runnning.
+	ReadyReplicas *int `json:"readyReplicas"`
+
+	// AvailableReplicas is the number of runners that are created and Runnning.
+	// This is currently same as ReadyReplicas but perserved for future use.
+	AvailableReplicas *int `json:"availableReplicas"`
 }
 
 type RunnerTemplate struct {
@@ -46,8 +57,9 @@ type RunnerTemplate struct {
 // +kubebuilder:object:root=true
 // +kubebuilder:subresource:status
 // +kubebuilder:printcolumn:JSONPath=".spec.replicas",name=Desired,type=number
-// +kubebuilder:printcolumn:JSONPath=".status.availableReplicas",name=Current,type=number
+// +kubebuilder:printcolumn:JSONPath=".status.replicas",name=Current,type=number
 // +kubebuilder:printcolumn:JSONPath=".status.readyReplicas",name=Ready,type=number
+// +kubebuilder:printcolumn:name="Age",type="date",JSONPath=".metadata.creationTimestamp"
 
 // RunnerReplicaSet is the Schema for the runnerreplicasets API
 type RunnerReplicaSet struct {

api/v1alpha1/runnerreplicaset_webhook.go

@@ -34,7 +34,7 @@ func (r *RunnerReplicaSet) SetupWebhookWithManager(mgr ctrl.Manager) error {
 		Complete()
 }
 
-// +kubebuilder:webhook:path=/mutate-actions-summerwind-dev-v1alpha1-runnerreplicaset,verbs=create;update,mutating=true,failurePolicy=fail,groups=actions.summerwind.dev,resources=runnerreplicasets,versions=v1alpha1,name=mutate.runnerreplicaset.actions.summerwind.dev
+// +kubebuilder:webhook:path=/mutate-actions-summerwind-dev-v1alpha1-runnerreplicaset,verbs=create;update,mutating=true,failurePolicy=fail,groups=actions.summerwind.dev,resources=runnerreplicasets,versions=v1alpha1,name=mutate.runnerreplicaset.actions.summerwind.dev,sideEffects=None
 
 var _ webhook.Defaulter = &RunnerReplicaSet{}
 
@@ -43,7 +43,7 @@ func (r *RunnerReplicaSet) Default() {
 	// Nothing to do.
 }
 
-// +kubebuilder:webhook:path=/validate-actions-summerwind-dev-v1alpha1-runnerreplicaset,verbs=create;update,mutating=false,failurePolicy=fail,groups=actions.summerwind.dev,resources=runnerreplicasets,versions=v1alpha1,name=validate.runnerreplicaset.actions.summerwind.dev
+// +kubebuilder:webhook:path=/validate-actions-summerwind-dev-v1alpha1-runnerreplicaset,verbs=create;update,mutating=false,failurePolicy=fail,groups=actions.summerwind.dev,resources=runnerreplicasets,versions=v1alpha1,name=validate.runnerreplicaset.actions.summerwind.dev,sideEffects=None
 
 var _ webhook.Validator = &RunnerReplicaSet{}
 

api/v1alpha1/zz_generated.deepcopy.go

@@ -212,6 +212,13 @@ func (in *HorizontalRunnerAutoscalerSpec) DeepCopyInto(out *HorizontalRunnerAuto
 			(*in)[i].DeepCopyInto(&(*out)[i])
 		}
 	}
+	if in.ScheduledOverrides != nil {
+		in, out := &in.ScheduledOverrides, &out.ScheduledOverrides
+		*out = make([]ScheduledOverride, len(*in))
+		for i := range *in {
+			(*in)[i].DeepCopyInto(&(*out)[i])
+		}
+	}
 }
 
 // DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new HorizontalRunnerAutoscalerSpec.
@@ -243,6 +250,11 @@ func (in *HorizontalRunnerAutoscalerStatus) DeepCopyInto(out *HorizontalRunnerAu
 			(*in)[i].DeepCopyInto(&(*out)[i])
 		}
 	}
+	if in.ScheduledOverridesSummary != nil {
+		in, out := &in.ScheduledOverridesSummary, &out.ScheduledOverridesSummary
+		*out = new(string)
+		**out = **in
+	}
 }
 
 // DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new HorizontalRunnerAutoscalerStatus.
@@ -315,6 +327,22 @@ func (in *PushSpec) DeepCopy() *PushSpec {
 	return out
 }
 
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *RecurrenceRule) DeepCopyInto(out *RecurrenceRule) {
+	*out = *in
+	in.UntilTime.DeepCopyInto(&out.UntilTime)
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new RecurrenceRule.
+func (in *RecurrenceRule) DeepCopy() *RecurrenceRule {
+	if in == nil {
+		return nil
+	}
+	out := new(RecurrenceRule)
+	in.DeepCopyInto(out)
+	return out
+}
+
 // DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
 func (in *Runner) DeepCopyInto(out *Runner) {
 	*out = *in
@@ -430,6 +458,26 @@ func (in *RunnerDeploymentSpec) DeepCopy() *RunnerDeploymentSpec {
 // DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
 func (in *RunnerDeploymentStatus) DeepCopyInto(out *RunnerDeploymentStatus) {
 	*out = *in
+	if in.AvailableReplicas != nil {
+		in, out := &in.AvailableReplicas, &out.AvailableReplicas
+		*out = new(int)
+		**out = **in
+	}
+	if in.ReadyReplicas != nil {
+		in, out := &in.ReadyReplicas, &out.ReadyReplicas
+		*out = new(int)
+		**out = **in
+	}
+	if in.UpdatedReplicas != nil {
+		in, out := &in.UpdatedReplicas, &out.UpdatedReplicas
+		*out = new(int)
+		**out = **in
+	}
+	if in.DesiredReplicas != nil {
+		in, out := &in.DesiredReplicas, &out.DesiredReplicas
+		*out = new(int)
+		**out = **in
+	}
 	if in.Replicas != nil {
 		in, out := &in.Replicas, &out.Replicas
 		*out = new(int)
@@ -485,7 +533,7 @@ func (in *RunnerReplicaSet) DeepCopyInto(out *RunnerReplicaSet) {
 	out.TypeMeta = in.TypeMeta
 	in.ObjectMeta.DeepCopyInto(&out.ObjectMeta)
 	in.Spec.DeepCopyInto(&out.Spec)
-	out.Status = in.Status
+	in.Status.DeepCopyInto(&out.Status)
 }
 
 // DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new RunnerReplicaSet.
@@ -567,6 +615,21 @@ func (in *RunnerReplicaSetSpec) DeepCopy() *RunnerReplicaSetSpec {
 // DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
 func (in *RunnerReplicaSetStatus) DeepCopyInto(out *RunnerReplicaSetStatus) {
 	*out = *in
+	if in.Replicas != nil {
+		in, out := &in.Replicas, &out.Replicas
+		*out = new(int)
+		**out = **in
+	}
+	if in.ReadyReplicas != nil {
+		in, out := &in.ReadyReplicas, &out.ReadyReplicas
+		*out = new(int)
+		**out = **in
+	}
+	if in.AvailableReplicas != nil {
+		in, out := &in.AvailableReplicas, &out.AvailableReplicas
+		*out = new(int)
+		**out = **in
+	}
 }
 
 // DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new RunnerReplicaSetStatus.
@@ -587,6 +650,11 @@ func (in *RunnerSpec) DeepCopyInto(out *RunnerSpec) {
 		*out = make([]string, len(*in))
 		copy(*out, *in)
 	}
+	if in.Ephemeral != nil {
+		in, out := &in.Ephemeral, &out.Ephemeral
+		*out = new(bool)
+		**out = **in
+	}
 	if in.Containers != nil {
 		in, out := &in.Containers, &out.Containers
 		*out = make([]v1.Container, len(*in))
@@ -595,6 +663,13 @@ func (in *RunnerSpec) DeepCopyInto(out *RunnerSpec) {
 		}
 	}
 	in.DockerdContainerResources.DeepCopyInto(&out.DockerdContainerResources)
+	if in.DockerVolumeMounts != nil {
+		in, out := &in.DockerVolumeMounts, &out.DockerVolumeMounts
+		*out = make([]v1.VolumeMount, len(*in))
+		for i := range *in {
+			(*in)[i].DeepCopyInto(&(*out)[i])
+		}
+	}
 	in.Resources.DeepCopyInto(&out.Resources)
 	if in.VolumeMounts != nil {
 		in, out := &in.VolumeMounts, &out.VolumeMounts
@@ -699,6 +774,28 @@ func (in *RunnerSpec) DeepCopyInto(out *RunnerSpec) {
 		*out = new(int64)
 		**out = **in
 	}
+	if in.DockerRegistryMirror != nil {
+		in, out := &in.DockerRegistryMirror, &out.DockerRegistryMirror
+		*out = new(string)
+		**out = **in
+	}
+	if in.HostAliases != nil {
+		in, out := &in.HostAliases, &out.HostAliases
+		*out = make([]v1.HostAlias, len(*in))
+		for i := range *in {
+			(*in)[i].DeepCopyInto(&(*out)[i])
+		}
+	}
+	if in.VolumeSizeLimit != nil {
+		in, out := &in.VolumeSizeLimit, &out.VolumeSizeLimit
+		x := (*in).DeepCopy()
+		*out = &x
+	}
+	if in.RuntimeClassName != nil {
+		in, out := &in.RuntimeClassName, &out.RuntimeClassName
+		*out = new(string)
+		**out = **in
+	}
 }
 
 // DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new RunnerSpec.
@@ -804,3 +901,26 @@ func (in *ScaleUpTrigger) DeepCopy() *ScaleUpTrigger {
 	in.DeepCopyInto(out)
 	return out
 }
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *ScheduledOverride) DeepCopyInto(out *ScheduledOverride) {
+	*out = *in
+	in.StartTime.DeepCopyInto(&out.StartTime)
+	in.EndTime.DeepCopyInto(&out.EndTime)
+	if in.MinReplicas != nil {
+		in, out := &in.MinReplicas, &out.MinReplicas
+		*out = new(int)
+		**out = **in
+	}
+	in.RecurrenceRule.DeepCopyInto(&out.RecurrenceRule)
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new ScheduledOverride.
+func (in *ScheduledOverride) DeepCopy() *ScheduledOverride {
+	if in == nil {
+		return nil
+	}
+	out := new(ScheduledOverride)
+	in.DeepCopyInto(out)
+	return out
+}

charts/actions-runner-controller/.helmignore

@@ -21,3 +21,5 @@
 .idea/
 *.tmproj
 .vscode/
+# Docs
+docs/

charts/actions-runner-controller/Chart.yaml

@@ -15,7 +15,10 @@ type: application
 # This is the chart version. This version number should be incremented each time you make changes
 # to the chart and its templates, including the app version.
 # Versions are expected to follow Semantic Versioning (https://semver.org/)
-version: 0.10.2
+version: 0.11.0
+
+# Used as the default manager tag value when no tag property is provided in the values.yaml
+appVersion: 0.18.2
 
 home: https://github.com/summerwind/actions-runner-controller
 

charts/actions-runner-controller/README.md

@@ -0,0 +1,81 @@
+## Docs
+
+All additional docs are kept in the `docs/` folder, this README is solely for documenting the values.yaml keys and values
+
+## Values
+
+_The values are documented as of HEAD_
+
+_Default values are the defaults set in the charts values.yaml, some properties have default configurations in the code for when the property is omitted or invalid_
+
+| Key                                                      | Description                                                                                                                | Default                                                              |
+|----------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------|
+| `labels`                                                 | Set labels to apply to all resources in the chart                                                                          |                                                                      |
+| `replicaCount`                                           | Set the number of controller pods                                                                                          | 1                                                                    |
+| `syncPeriod`                                             | Set the period in which the controler reconciles the desired runners count                                                 | 10m                                                                  |
+| `githubAPICacheDuration`                                 | Set the cache period for API calls                                                                                         |                                                                      |
+| `logLevel`                                               | Set the log level of the controller container                                                                              |                                                                      |
+| `authSecret.create`                                      | Deploy the controller auth secret                                                                                          | true                                                                 |
+| `authSecret.name`                                        | Set the name of the auth secret                                                                                            | controller-manager                                                   |
+| `authSecret.github_app_id`                               | The ID of your GitHub App. **This can't be set at the same time as `authSecret.github_token`**                             |                                                                      |
+| `authSecret.github_app_installation_id`                  | The ID of your GitHub App installation. **This can't be set at the same time as `authSecret.github_token`**                |                                                                      |
+| `authSecret.github_app_private_key`                      | The multiline string of your GitHub App's private key. **This can't be set at the same time as `authSecret.github_token`** |                                                                      |
+| `authSecret.github_token`                                | Your chosen GitHub PAT token. **This can't be set at the same time as the `authSecret.github_app_*`**                      |                                                                      |
+| `image.repository`                                       | The "repository/image" of the controller container                                                                         | summerwind/actions-runner-controller                                 |
+| `image.tag`                                              | The tag of the controller container                                                                                        |                                                                      |
+| `image.dindSidecarRepositoryAndTag`                      | The "repository/image" of the dind sidecar container                                                                       | docker:dind                                                          |
+| `image.pullPolicy`                                       | The pull policy of the controller image                                                                                    | IfNotPresent                                                         |
+| `metrics.serviceMonitor`                                 | Deploy serviceMonitor kind for for use with prometheus-operator CRDs                                                       | false                                                                |
+| `metrics.port`                                           | Set port of metrics service                                                                                                | 8443                                                                 |
+| `metrics.proxy.enabled`                                  | Deploy kube-rbac-proxy container in controller pod                                                                         | true                                                                 |
+| `metrics.proxy.image.repository`                         | The "repository/image" of the kube-proxy container                                                                         | quay.io/brancz/kube-rbac-proxy                                       |
+| `metrics.proxy.image.tag`                                | The tag of the kube-proxy image to use when pulling the container                                                          | v0.10.0                                                              |
+| `imagePullSecrets`                                       | Specifies the secret to be used when pulling the controller pod containers                                                 |                                                                      |
+| `fullNameOverride`                                       | Override the full resource names	                                                                                        |                                                                      |
+| `nameOverride`                                           | Override the resource name prefix	                                                                                        |                                                                      |
+| `serviceAccont.annotations`                              | Set annotations to the service account                                                                                     |                                                                      |
+| `serviceAccount.create`                                  | Deploy the controller pod under a service account                                                                          | true                                                                 |
+| `podAnnotations`                                         | Set annotations for the controller pod                                                                                     |                                                                      |
+| `podLabels`                                              | Set labels for the controller pod                                                                                          |                                                                      |
+| `serviceAccount.name`                                    | Set the name of the service account                                                                                        |                                                                      |
+| `securityContext`                                        | Set the security context for each container in the controller pod                                                          |                                                                      |
+| `podSecurityContext`                                     | Set the security context to controller pod                                                                                 |                                                                      |
+| `service.port`                                           | Set controller service type                                                                                                |                                                                      |
+| `service.type`                                           | Set controller service ports                                                                                               |                                                                      |
+| `topologySpreadConstraints`                              | Set the controller pod topologySpreadConstraints                                                                           |                                                                      |
+| `nodeSelector`                                           | Set the controller pod nodeSelector                                                                                        |                                                                      |
+| `resources`                                              | Set the controller pod resources                                                                                           |                                                                      |
+| `affinity`                                               | Set the controller pod affinity rules                                                                                      |                                                                      |
+| `tolerations`                                            | Set the controller pod tolerations                                                                                         |                                                                      |
+| `env`                                                    | Set environment variables for the controller container                                                                     |                                                                      |
+| `priorityClassName`                                      | Set the controller pod priorityClassName                                                                                   |                                                                      |
+| `scope.watchNamespace`                                   | Tells the controller which namespace to watch if `scope.singleNamespace` is true                                           |                                                                      |
+| `scope.singleNamespace`                                  | Limit the controller to watch a single namespace                                                                           | false                                                                |
+| `githubWebhookServer.logLevel`                           | Set the log level of the githubWebhookServer container                                                                     |                                                                      |
+| `githubWebhookServer.replicaCount`                       | Set the number of webhook server pods                                                                                      | 1                                                                    |
+| `githubWebhookServer.enabled`                            | Deploy the webhook server pod                                                                                              | false                                                                |
+| `githubWebhookServer.secret.create`                      | Deploy the webhook hook secret                                                                                             | true                                                                 |
+| `githubWebhookServer.secret.name`                        | Set the name of the webhook hook secret                                                                                    | github-webhook-server                                                |
+| `githubWebhookServer.secret.github_webhook_secret_token` | Set the webhook secret token value                                                                                         |                                                                      |
+| `githubWebhookServer.imagePullSecrets`                   | Specifies the secret to be used when pulling the githubWebhookServer pod containers                                        |                                                                      |
+| `githubWebhookServer.nameOveride`                        | Override the resource name prefix	                                                                                        |                                                                      |
+| `githubWebhookServer.fullNameOveride`                    | Override the full resource names	                                                                                        |                                                                      |
+| `githubWebhookServer.serviceAccount.create`              | Deploy the githubWebhookServer under a service account                                                                     | true                                                                 |
+| `githubWebhookServer.serviceAccount.annotations`         | Set annotations for the service account                                                                                    |                                                                      |
+| `githubWebhookServer.serviceAccount.name`                | Set the service account name                                                                                               |                                                                      |
+| `githubWebhookServer.podAnnotations`                     | Set annotations for the githubWebhookServer pod                                                                            |                                                                      |
+| `githubWebhookServer.podLabels`                          | Set labels for the githubWebhookServer pod                                                                                 |                                                                      |
+| `githubWebhookServer.podSecurityContext`                 | Set the security context to githubWebhookServer pod                                                                        |                                                                      |
+| `githubWebhookServer.securityContext`                    | Set the security context for each container in the githubWebhookServer pod                                                 |                                                                      |
+| `githubWebhookServer.resources`                          | Set the githubWebhookServer pod resources                                                                                  |                                                                      |
+| `githubWebhookServer.topologySpreadConstraints`          | Set the githubWebhookServer pod topologySpreadConstraints                                                                  |                                                                      |
+| `githubWebhookServer.nodeSelector`                       | Set the githubWebhookServer pod nodeSelector                                                                               |                                                                      |
+| `githubWebhookServer.tolerations`                        | Set the githubWebhookServer pod tolerations                                                                                |                                                                      |
+| `githubWebhookServer.affinity`                           | Set the githubWebhookServer pod affinity rules                                                                             |                                                                      |
+| `githubWebhookServer.priorityClassName`                  | Set the githubWebhookServer pod priorityClassName                                                                          |                                                                      |
+| `githubWebhookServer.service.type`                       | Set githubWebhookServer service type                                                                                       |                                                                      |
+| `githubWebhookServer.service.ports`                      | Set githubWebhookServer service ports                                                                                      | `[{"port":80, "targetPort:"http", "protocol":"TCP", "name":"http"}]` |
+| `githubWebhookServer.ingress.enabled`                    | Deploy an ingress kind for the githubWebhookServer                                                                         | false                                                                |
+| `githubWebhookServer.ingress.annotations`                | Set annotations for the ingress kind                                                                                       |                                                                      |
+| `githubWebhookServer.ingress.hosts`                      | Set hosts configuration for ingress                                                                                        | `[{"host": "chart-example.local", "paths": []}]`                     |
+| `githubWebhookServer.ingress.tls`                        | Set tls configuration for ingress                                                                                          |                                                                      |

charts/actions-runner-controller/crds/actions.summerwind.dev_horizontalrunnerautoscalers.yaml

@@ -18,6 +18,9 @@ spec:
   - JSONPath: .status.desiredReplicas
     name: Desired
     type: number
+  - JSONPath: .status.scheduledOverridesSummary
+    name: Schedule
+    type: string
   group: actions.summerwind.dev
   names:
     kind: HorizontalRunnerAutoscaler
@@ -185,6 +188,56 @@ spec:
                     type: object
                 type: object
               type: array
+            scheduledOverrides:
+              description: ScheduledOverrides is the list of ScheduledOverride. It
+                can be used to override a few fields of HorizontalRunnerAutoscalerSpec
+                on schedule. The earlier a scheduled override is, the higher it is
+                prioritized.
+              items:
+                description: ScheduledOverride can be used to override a few fields
+                  of HorizontalRunnerAutoscalerSpec on schedule. A schedule can optionally
+                  be recurring, so that the correspoding override happens every day,
+                  week, month, or year.
+                properties:
+                  endTime:
+                    description: EndTime is the time at which the first override ends.
+                    format: date-time
+                    type: string
+                  minReplicas:
+                    description: MinReplicas is the number of runners while overriding.
+                      If omitted, it doesn't override minReplicas.
+                    minimum: 0
+                    nullable: true
+                    type: integer
+                  recurrenceRule:
+                    properties:
+                      frequency:
+                        description: Frequency is the name of a predefined interval
+                          of each recurrence. The valid values are "Daily", "Weekly",
+                          "Monthly", and "Yearly". If empty, the corresponding override
+                          happens only once.
+                        enum:
+                        - Daily
+                        - Weekly
+                        - Monthly
+                        - Yearly
+                        type: string
+                      untilTime:
+                        description: UntilTime is the time of the final recurrence.
+                          If empty, the schedule recurs forever.
+                        format: date-time
+                        type: string
+                    type: object
+                  startTime:
+                    description: StartTime is the time at which the first override
+                      starts.
+                    format: date-time
+                    type: string
+                required:
+                - endTime
+                - startTime
+                type: object
+              type: array
           type: object
         status:
           properties:
@@ -215,6 +268,11 @@ spec:
                 which is updated on mutation by the API Server.
               format: int64
               type: integer
+            scheduledOverridesSummary:
+              description: ScheduledOverridesSummary is the summary of active and
+                upcoming scheduled overrides to be shown in e.g. a column of a `kubectl
+                get hra` output for observability.
+              type: string
           type: object
       type: object
   version: v1alpha1

charts/actions-runner-controller/crds/actions.summerwind.dev_runnerdeployments.yaml

@@ -10,12 +10,18 @@ spec:
     - JSONPath: .spec.replicas
       name: Desired
       type: number
-    - JSONPath: .status.availableReplicas
+    - JSONPath: .status.replicas
       name: Current
       type: number
-    - JSONPath: .status.readyReplicas
-      name: Ready
+    - JSONPath: .status.updatedReplicas
+      name: Up-To-Date
       type: number
+    - JSONPath: .status.availableReplicas
+      name: Available
+      type: number
+    - JSONPath: .metadata.creationTimestamp
+      name: Age
+      type: date
   group: actions.summerwind.dev
   names:
     kind: RunnerDeployment
@@ -436,6 +442,35 @@ spec:
                     dockerMTU:
                       format: int64
                       type: integer
+                    dockerRegistryMirror:
+                      type: string
+                    dockerVolumeMounts:
+                      items:
+                        description: VolumeMount describes a mounting of a Volume within a container.
+                        properties:
+                          mountPath:
+                            description: Path within the container at which the volume should be mounted.  Must not contain ':'.
+                            type: string
+                          mountPropagation:
+                            description: mountPropagation determines how mounts are propagated from the host to container and the other way around. When not set, MountPropagationNone is used. This field is beta in 1.10.
+                            type: string
+                          name:
+                            description: This must match the Name of a Volume.
+                            type: string
+                          readOnly:
+                            description: Mounted read-only if true, read-write otherwise (false or unspecified). Defaults to false.
+                            type: boolean
+                          subPath:
+                            description: Path within the volume from which the container's volume should be mounted. Defaults to "" (volume's root).
+                            type: string
+                          subPathExpr:
+                            description: Expanded path within the volume from which the container's volume should be mounted. Behaves similarly to SubPath but environment variable references $(VAR_NAME) are expanded using the container's environment. Defaults to "" (volume's root). SubPathExpr and SubPath are mutually exclusive. This field is beta in 1.15.
+                            type: string
+                        required:
+                          - mountPath
+                          - name
+                        type: object
+                      type: array
                     dockerdContainerResources:
                       description: ResourceRequirements describes the compute resource requirements.
                       properties:
@@ -571,6 +606,8 @@ spec:
                             type: object
                         type: object
                       type: array
+                    ephemeral:
+                      type: boolean
                     ephemeralContainers:
                       items:
                         description: An EphemeralContainer is a container that may be added temporarily to an existing pod for user-initiated activities such as debugging. Ephemeral containers have no resource or scheduling guarantees, and they will not be restarted when they exit or when a pod is removed or restarted. If an ephemeral container causes a pod to exceed its resource allocation, the pod may be evicted. Ephemeral containers may not be added by directly updating the pod spec. They must be added via the pod's ephemeralcontainers subresource, and they will appear in the pod spec once added. This is an alpha feature enabled by the EphemeralContainers feature flag.
@@ -580,6 +617,20 @@ spec:
                       type: array
                     group:
                       type: string
+                    hostAliases:
+                      items:
+                        description: HostAlias holds the mapping between IP and hostnames that will be injected as an entry in the pod's hosts file.
+                        properties:
+                          hostnames:
+                            description: Hostnames for the above IP address.
+                            items:
+                              type: string
+                            type: array
+                          ip:
+                            description: IP address of the host file entry.
+                            type: string
+                        type: object
+                      type: array
                     image:
                       type: string
                     imagePullPolicy:
@@ -637,6 +688,9 @@ spec:
                           description: 'Requests describes the minimum amount of compute resources required. If Requests is omitted for a container, it defaults to Limits if that is explicitly specified, otherwise to an implementation-defined value. More info: https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/'
                           type: object
                       type: object
+                    runtimeClassName:
+                      description: 'RuntimeClassName is the container runtime configuration that containers should run under. More info: https://kubernetes.io/docs/concepts/containers/runtime-class'
+                      type: string
                     securityContext:
                       description: PodSecurityContext holds pod-level security attributes and common container settings. Some fields are also present in container.securityContext.  Field values of container.securityContext take precedence over field values of PodSecurityContext.
                       properties:
@@ -768,6 +822,12 @@ spec:
                           - name
                         type: object
                       type: array
+                    volumeSizeLimit:
+                      anyOf:
+                        - type: integer
+                        - type: string
+                      pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
+                      x-kubernetes-int-or-string: true
                     volumes:
                       items:
                         description: Volume represents a named volume in a pod that may be accessed by any container in the pod.
@@ -1580,15 +1640,20 @@ spec:
         status:
           properties:
             availableReplicas:
+              description: AvailableReplicas is the total number of available runners which have been successfully registered to GitHub and still running. This corresponds to the sum of status.availableReplicas of all the runner replica sets.
               type: integer
             desiredReplicas:
-              description: Replicas is the total number of desired, non-terminated and latest pods to be set for the primary RunnerSet This doesn't include outdated pods while upgrading the deployment and replacing the runnerset.
+              description: DesiredReplicas is the total number of desired, non-terminated and latest pods to be set for the primary RunnerSet This doesn't include outdated pods while upgrading the deployment and replacing the runnerset.
               type: integer
             readyReplicas:
+              description: ReadyReplicas is the total number of available runners which have been successfully registered to GitHub and still running. This corresponds to the sum of status.readyReplicas of all the runner replica sets.
+              type: integer
+            replicas:
+              description: Replicas is the total number of replicas
+              type: integer
+            updatedReplicas:
+              description: ReadyReplicas is the total number of available runners which have been successfully registered to GitHub and still running. This corresponds to status.replicas of the runner replica set that has the desired template hash.
               type: integer
-          required:
-            - availableReplicas
-            - readyReplicas
           type: object
       type: object
   version: v1alpha1

charts/actions-runner-controller/crds/actions.summerwind.dev_runnerreplicasets.yaml

@@ -10,12 +10,15 @@ spec:
     - JSONPath: .spec.replicas
       name: Desired
       type: number
-    - JSONPath: .status.availableReplicas
+    - JSONPath: .status.replicas
       name: Current
       type: number
     - JSONPath: .status.readyReplicas
       name: Ready
       type: number
+    - JSONPath: .metadata.creationTimestamp
+      name: Age
+      type: date
   group: actions.summerwind.dev
   names:
     kind: RunnerReplicaSet
@@ -436,6 +439,35 @@ spec:
                     dockerMTU:
                       format: int64
                       type: integer
+                    dockerRegistryMirror:
+                      type: string
+                    dockerVolumeMounts:
+                      items:
+                        description: VolumeMount describes a mounting of a Volume within a container.
+                        properties:
+                          mountPath:
+                            description: Path within the container at which the volume should be mounted.  Must not contain ':'.
+                            type: string
+                          mountPropagation:
+                            description: mountPropagation determines how mounts are propagated from the host to container and the other way around. When not set, MountPropagationNone is used. This field is beta in 1.10.
+                            type: string
+                          name:
+                            description: This must match the Name of a Volume.
+                            type: string
+                          readOnly:
+                            description: Mounted read-only if true, read-write otherwise (false or unspecified). Defaults to false.
+                            type: boolean
+                          subPath:
+                            description: Path within the volume from which the container's volume should be mounted. Defaults to "" (volume's root).
+                            type: string
+                          subPathExpr:
+                            description: Expanded path within the volume from which the container's volume should be mounted. Behaves similarly to SubPath but environment variable references $(VAR_NAME) are expanded using the container's environment. Defaults to "" (volume's root). SubPathExpr and SubPath are mutually exclusive. This field is beta in 1.15.
+                            type: string
+                        required:
+                          - mountPath
+                          - name
+                        type: object
+                      type: array
                     dockerdContainerResources:
                       description: ResourceRequirements describes the compute resource requirements.
                       properties:
@@ -571,6 +603,8 @@ spec:
                             type: object
                         type: object
                       type: array
+                    ephemeral:
+                      type: boolean
                     ephemeralContainers:
                       items:
                         description: An EphemeralContainer is a container that may be added temporarily to an existing pod for user-initiated activities such as debugging. Ephemeral containers have no resource or scheduling guarantees, and they will not be restarted when they exit or when a pod is removed or restarted. If an ephemeral container causes a pod to exceed its resource allocation, the pod may be evicted. Ephemeral containers may not be added by directly updating the pod spec. They must be added via the pod's ephemeralcontainers subresource, and they will appear in the pod spec once added. This is an alpha feature enabled by the EphemeralContainers feature flag.
@@ -580,6 +614,20 @@ spec:
                       type: array
                     group:
                       type: string
+                    hostAliases:
+                      items:
+                        description: HostAlias holds the mapping between IP and hostnames that will be injected as an entry in the pod's hosts file.
+                        properties:
+                          hostnames:
+                            description: Hostnames for the above IP address.
+                            items:
+                              type: string
+                            type: array
+                          ip:
+                            description: IP address of the host file entry.
+                            type: string
+                        type: object
+                      type: array
                     image:
                       type: string
                     imagePullPolicy:
@@ -637,6 +685,9 @@ spec:
                           description: 'Requests describes the minimum amount of compute resources required. If Requests is omitted for a container, it defaults to Limits if that is explicitly specified, otherwise to an implementation-defined value. More info: https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/'
                           type: object
                       type: object
+                    runtimeClassName:
+                      description: 'RuntimeClassName is the container runtime configuration that containers should run under. More info: https://kubernetes.io/docs/concepts/containers/runtime-class'
+                      type: string
                     securityContext:
                       description: PodSecurityContext holds pod-level security attributes and common container settings. Some fields are also present in container.securityContext.  Field values of container.securityContext take precedence over field values of PodSecurityContext.
                       properties:
@@ -768,6 +819,12 @@ spec:
                           - name
                         type: object
                       type: array
+                    volumeSizeLimit:
+                      anyOf:
+                        - type: integer
+                        - type: string
+                      pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
+                      x-kubernetes-int-or-string: true
                     volumes:
                       items:
                         description: Volume represents a named volume in a pod that may be accessed by any container in the pod.
@@ -1580,8 +1637,13 @@ spec:
         status:
           properties:
             availableReplicas:
+              description: AvailableReplicas is the number of runners that are created and Runnning. This is currently same as ReadyReplicas but perserved for future use.
               type: integer
             readyReplicas:
+              description: ReadyReplicas is the number of runners that are created and Runnning.
+              type: integer
+            replicas:
+              description: Replicas is the number of runners that are created and still being managed by this runner replica set.
               type: integer
           required:
             - availableReplicas

charts/actions-runner-controller/crds/actions.summerwind.dev_runners.yaml

@@ -22,6 +22,9 @@ spec:
     - JSONPath: .status.phase
       name: Status
       type: string
+    - JSONPath: .metadata.creationTimestamp
+      name: Age
+      type: date
   group: actions.summerwind.dev
   names:
     kind: Runner
@@ -401,6 +404,35 @@ spec:
             dockerMTU:
               format: int64
               type: integer
+            dockerRegistryMirror:
+              type: string
+            dockerVolumeMounts:
+              items:
+                description: VolumeMount describes a mounting of a Volume within a container.
+                properties:
+                  mountPath:
+                    description: Path within the container at which the volume should be mounted.  Must not contain ':'.
+                    type: string
+                  mountPropagation:
+                    description: mountPropagation determines how mounts are propagated from the host to container and the other way around. When not set, MountPropagationNone is used. This field is beta in 1.10.
+                    type: string
+                  name:
+                    description: This must match the Name of a Volume.
+                    type: string
+                  readOnly:
+                    description: Mounted read-only if true, read-write otherwise (false or unspecified). Defaults to false.
+                    type: boolean
+                  subPath:
+                    description: Path within the volume from which the container's volume should be mounted. Defaults to "" (volume's root).
+                    type: string
+                  subPathExpr:
+                    description: Expanded path within the volume from which the container's volume should be mounted. Behaves similarly to SubPath but environment variable references $(VAR_NAME) are expanded using the container's environment. Defaults to "" (volume's root). SubPathExpr and SubPath are mutually exclusive. This field is beta in 1.15.
+                    type: string
+                required:
+                  - mountPath
+                  - name
+                type: object
+              type: array
             dockerdContainerResources:
               description: ResourceRequirements describes the compute resource requirements.
               properties:
@@ -536,6 +568,8 @@ spec:
                     type: object
                 type: object
               type: array
+            ephemeral:
+              type: boolean
             ephemeralContainers:
               items:
                 description: An EphemeralContainer is a container that may be added temporarily to an existing pod for user-initiated activities such as debugging. Ephemeral containers have no resource or scheduling guarantees, and they will not be restarted when they exit or when a pod is removed or restarted. If an ephemeral container causes a pod to exceed its resource allocation, the pod may be evicted. Ephemeral containers may not be added by directly updating the pod spec. They must be added via the pod's ephemeralcontainers subresource, and they will appear in the pod spec once added. This is an alpha feature enabled by the EphemeralContainers feature flag.
@@ -545,6 +579,20 @@ spec:
               type: array
             group:
               type: string
+            hostAliases:
+              items:
+                description: HostAlias holds the mapping between IP and hostnames that will be injected as an entry in the pod's hosts file.
+                properties:
+                  hostnames:
+                    description: Hostnames for the above IP address.
+                    items:
+                      type: string
+                    type: array
+                  ip:
+                    description: IP address of the host file entry.
+                    type: string
+                type: object
+              type: array
             image:
               type: string
             imagePullPolicy:
@@ -602,6 +650,9 @@ spec:
                   description: 'Requests describes the minimum amount of compute resources required. If Requests is omitted for a container, it defaults to Limits if that is explicitly specified, otherwise to an implementation-defined value. More info: https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/'
                   type: object
               type: object
+            runtimeClassName:
+              description: 'RuntimeClassName is the container runtime configuration that containers should run under. More info: https://kubernetes.io/docs/concepts/containers/runtime-class'
+              type: string
             securityContext:
               description: PodSecurityContext holds pod-level security attributes and common container settings. Some fields are also present in container.securityContext.  Field values of container.securityContext take precedence over field values of PodSecurityContext.
               properties:
@@ -733,6 +784,12 @@ spec:
                   - name
                 type: object
               type: array
+            volumeSizeLimit:
+              anyOf:
+                - type: integer
+                - type: string
+              pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
+              x-kubernetes-int-or-string: true
             volumes:
               items:
                 description: Volume represents a named volume in a pod that may be accessed by any container in the pod.
@@ -1543,6 +1600,7 @@ spec:
           properties:
             lastRegistrationCheckTime:
               format: date-time
+              nullable: true
               type: string
             message:
               type: string
@@ -1572,11 +1630,6 @@ spec:
                 - expiresAt
                 - token
               type: object
-          required:
-            - message
-            - phase
-            - reason
-            - registration
           type: object
       type: object
   version: v1alpha1

charts/actions-runner-controller/docs/UPGRADING.md

@@ -0,0 +1,39 @@
+## Upgrading
+
+This project makes extensive use of CRDs to provide much of its functionality. Helm unfortunately does not support [managing](https://helm.sh/docs/chart_best_practices/custom_resource_definitions/) CRDs by design:
+
+_The full breakdown as to how they came to this decision and why they have taken the approach they have for dealing with CRDs can be found in [Helm Improvement Proposal 11](https://github.com/helm/community/blob/main/hips/hip-0011.md)_
+
+```
+There is no support at this time for upgrading or deleting CRDs using Helm. This was an explicit decision after much 
+community discussion due to the danger for unintentional data loss. Furthermore, there is currently no community 
+consensus around how to handle CRDs and their lifecycle. As this evolves, Helm will add support for those use cases.
+```
+
+Helm will do an initial install of CRDs but it will not touch them afterwards (update or delete).
+
+Additionally, because the project leverages CRDs so extensively you **MUST** run the matching controller app container with its matching CRDs i.e. always redeploy your CRDs if you are changing the app version.
+
+Due to the above you can't just do a `helm upgrade` to release the latest version of the chart, the best practice steps are recorded below:
+
+## Steps
+
+1. Upgrade CRDs
+
+```shell
+CHART_VERSION=0.11.0
+
+curl -L https://github.com/actions-runner-controller/actions-runner-controller/releases/download/actions-runner-controller-${CHART_VERSION}/actions-runner-controller-${CHART_VERSION}.tgz | tar zxv --strip 1 actions-runner-controller/crds
+
+kubectl apply -f crds/
+```
+
+2. Upgrade the Helm release
+
+```shell
+helm upgrade --install \
+  --namespace actions-runner-system \
+  --version ${CHART_VERSION} \
+  actions-runner-controller/actions-runner-controller \
+  actions-runner-controller
+```

charts/actions-runner-controller/templates/_github_webhook_server_helpers.tpl

@@ -54,3 +54,7 @@ Create the name of the service account to use
 {{- define "actions-runner-controller-github-webhook-server.roleName" -}}
 {{- include "actions-runner-controller-github-webhook-server.fullname" . }}
 {{- end }}
+
+{{- define "actions-runner-controller-github-webhook-server.serviceMonitorName" -}}
+{{- include "actions-runner-controller-github-webhook-server.fullname" . | trunc 47 }}-service-monitor
+{{- end }}

charts/actions-runner-controller/templates/_helpers.tpl

@@ -92,10 +92,14 @@ Create the name of the service account to use
 {{- include "actions-runner-controller.fullname" . | trunc 55 }}-webhook
 {{- end }}
 
-{{- define "actions-runner-controller.authProxyServiceName" -}}
+{{- define "actions-runner-controller.metricsServiceName" -}}
 {{- include "actions-runner-controller.fullname" . | trunc 47 }}-metrics-service
 {{- end }}
 
+{{- define "actions-runner-controller.serviceMonitorName" -}}
+{{- include "actions-runner-controller.fullname" . | trunc 47 }}-service-monitor
+{{- end }}
+
 {{- define "actions-runner-controller.selfsignedIssuerName" -}}
 {{- include "actions-runner-controller.fullname" . }}-selfsigned-issuer
 {{- end }}

charts/actions-runner-controller/templates/auth_proxy_role.yaml

@@ -1,3 +1,4 @@
+{{- if .Values.metrics.proxy.enabled }}
 apiVersion: rbac.authorization.k8s.io/v1
 kind: ClusterRole
 metadata:
@@ -11,3 +12,4 @@ rules:
   resources:
   - subjectaccessreviews
   verbs: ["create"]
+{{- end }}

charts/actions-runner-controller/templates/auth_proxy_role_binding.yaml

@@ -1,3 +1,4 @@
+{{- if .Values.metrics.proxy.enabled }}
 apiVersion: rbac.authorization.k8s.io/v1
 kind: ClusterRoleBinding
 metadata:
@@ -10,3 +11,4 @@ subjects:
 - kind: ServiceAccount
   name: {{ include "actions-runner-controller.serviceAccountName" . }}
   namespace: {{ .Release.Namespace }}
+{{- end }}

charts/actions-runner-controller/templates/controller.metrics.service.yaml

@@ -3,12 +3,12 @@ kind: Service
 metadata:
   labels:
     {{- include "actions-runner-controller.labels" . | nindent 4 }}
-  name: {{ include "actions-runner-controller.authProxyServiceName" . }}
+  name: {{ include "actions-runner-controller.metricsServiceName" . }}
   namespace: {{ .Release.Namespace }}
 spec:
   ports:
-  - name: https
-    port: 8443
-    targetPort: https
+  - name: metrics-port
+    port: {{ .Values.metrics.port }}
+    targetPort: metrics-port
   selector:
     {{- include "actions-runner-controller.selectorLabels" . | nindent 4 }}

charts/actions-runner-controller/templates/controller.metrics.serviceMonitor.yaml

@@ -0,0 +1,15 @@
+{{- if .Values.metrics.serviceMonitor }}
+apiVersion: monitoring.coreos.com/v1
+kind: ServiceMonitor
+metadata:
+  labels:
+    {{- include "actions-runner-controller.labels" . | nindent 4 }}
+  name: {{ include "actions-runner-controller.serviceMonitorName" . }}
+spec:
+  endpoints:
+    - path: /metrics
+      port: metrics-port
+  selector:
+    matchLabels:
+      {{- include "actions-runner-controller.selectorLabels" . | nindent 6 }}
+{{- end }}

charts/actions-runner-controller/templates/deployment.yaml

@@ -18,6 +18,9 @@ spec:
       {{- end }}
       labels:
         {{- include "actions-runner-controller.selectorLabels" . | nindent 8 }}
+      {{- with .Values.podLabels }}
+        {{- toYaml . | nindent 8 }}
+      {{- end }}
     spec:
       {{- with .Values.imagePullSecrets }}
       imagePullSecrets:
@@ -31,13 +34,21 @@ spec:
       {{- end }}
       containers:
       - args:
-        - "--metrics-addr=127.0.0.1:8080"
+        {{- $metricsHost := .Values.metrics.proxy.enabled | ternary "127.0.0.1" "0.0.0.0" }}
+        {{- $metricsPort := .Values.metrics.proxy.enabled | ternary "8080" .Values.metrics.port }}
+        - "--metrics-addr={{ $metricsHost }}:{{ $metricsPort }}"
         - "--enable-leader-election"
         - "--sync-period={{ .Values.syncPeriod }}"
         - "--docker-image={{ .Values.image.dindSidecarRepositoryAndTag }}"
         {{- if .Values.scope.singleNamespace }}
         - "--watch-namespace={{ default .Release.Namespace .Values.scope.watchNamespace }}"
         {{- end }}
+        {{- if .Values.githubAPICacheDuration }}
+        - "--github-api-cache-duration={{ .Values.githubAPICacheDuration }}"
+        {{- end }}
+        {{- if .Values.logLevel }}
+        - "--log-level={{ .Values.logLevel }}"
+        {{- end }}
         command:
         - "/manager"
         env:
@@ -65,13 +76,18 @@ spec:
         - name: {{ $key }}
           value: {{ $val | quote }}
         {{- end }}
-        image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
+        image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default (cat "v" .Chart.AppVersion | replace " " "") }}"
         name: manager
         imagePullPolicy: {{ .Values.image.pullPolicy }}
         ports:
         - containerPort: 9443
           name: webhook-server
           protocol: TCP
+        {{- if not .Values.metrics.proxy.enabled }}
+        - containerPort: {{ .Values.metrics.port }}
+          name: metrics-port
+          protocol: TCP
+        {{- end }}
         resources:
           {{- toYaml .Values.resources | nindent 12 }}
         securityContext:
@@ -85,21 +101,23 @@ spec:
         - mountPath: /tmp/k8s-webhook-server/serving-certs
           name: cert
           readOnly: true
+      {{- if .Values.metrics.proxy.enabled }}
       - args:
-        - "--secure-listen-address=0.0.0.0:8443"
+        - "--secure-listen-address=0.0.0.0:{{ .Values.metrics.port }}"
         - "--upstream=http://127.0.0.1:8080/"
         - "--logtostderr=true"
         - "--v=10"
-        image: "{{ .Values.kube_rbac_proxy.image.repository }}:{{ .Values.kube_rbac_proxy.image.tag }}"
+        image: "{{ .Values.metrics.proxy.image.repository }}:{{ .Values.metrics.proxy.image.tag }}"
         name: kube-rbac-proxy
         imagePullPolicy: {{ .Values.image.pullPolicy }}
         ports:
-        - containerPort: 8443
-          name: https
+        - containerPort: {{ .Values.metrics.port }}
+          name: metrics-port
         resources:
           {{- toYaml .Values.resources | nindent 12 }}
         securityContext:
           {{- toYaml .Values.securityContext | nindent 12 }}
+      {{- end }}
       terminationGracePeriodSeconds: 10
       volumes:
       - name: secret
@@ -123,3 +141,7 @@ spec:
       tolerations:
         {{- toYaml . | nindent 8 }}
       {{- end }}
+      {{- with .Values.topologySpreadConstraints }}
+      topologySpreadConstraints:
+        {{- toYaml . | nindent 8 }}
+      {{- end }}

charts/actions-runner-controller/templates/githubwebhook.deployment.yaml

@@ -19,6 +19,9 @@ spec:
       {{- end }}
       labels:
         {{- include "actions-runner-controller-github-webhook-server.selectorLabels" . | nindent 8 }}
+      {{- with .Values.githubWebhookServer.podLabels }}
+        {{- toYaml . | nindent 8 }}
+      {{- end }}
     spec:
       {{- with .Values.githubWebhookServer.imagePullSecrets }}
       imagePullSecrets:
@@ -32,8 +35,13 @@ spec:
       {{- end }}
       containers:
       - args:
-        - "--metrics-addr=127.0.0.1:8080"
+        {{- $metricsHost := .Values.metrics.proxy.enabled | ternary "127.0.0.1" "0.0.0.0" }}
+        {{- $metricsPort := .Values.metrics.proxy.enabled | ternary "8080" .Values.metrics.port }}
+        - "--metrics-addr={{ $metricsHost }}:{{ $metricsPort }}"
         - "--sync-period={{ .Values.githubWebhookServer.syncPeriod }}"
+        {{- if .Values.githubWebhookServer.logLevel }}
+        - "--log-level={{ .Values.githubWebhookServer.logLevel }}"
+        {{- end }}
         command:
         - "/github-webhook-server"
         env:
@@ -47,32 +55,39 @@ spec:
         - name: {{ $key }}
           value: {{ $val | quote }}
         {{- end }}
-        image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
+        image: "{{ .Values.image.repository }}:{{ .Values.image.tag  | default (cat "v" .Chart.AppVersion | replace " " "") }}"
         name: github-webhook-server
         imagePullPolicy: {{ .Values.image.pullPolicy }}
         ports:
         - containerPort: 8000
           name: http
           protocol: TCP
+        {{- if not .Values.metrics.proxy.enabled }}
+        - containerPort: {{ .Values.metrics.port }}
+          name: metrics-port
+          protocol: TCP
+        {{- end }}
         resources:
           {{- toYaml .Values.githubWebhookServer.resources | nindent 12 }}
         securityContext:
           {{- toYaml .Values.githubWebhookServer.securityContext | nindent 12 }}
+      {{- if .Values.metrics.proxy.enabled }}
       - args:
-        - "--secure-listen-address=0.0.0.0:8443"
+        - "--secure-listen-address=0.0.0.0:{{ .Values.metrics.port }}"
         - "--upstream=http://127.0.0.1:8080/"
         - "--logtostderr=true"
         - "--v=10"
-        image: "{{ .Values.kube_rbac_proxy.image.repository }}:{{ .Values.kube_rbac_proxy.image.tag }}"
+        image: "{{ .Values.metrics.proxy.image.repository }}:{{ .Values.metrics.proxy.image.tag }}"
         name: kube-rbac-proxy
         imagePullPolicy: {{ .Values.image.pullPolicy }}
         ports:
-        - containerPort: 8443
-          name: https
+        - containerPort: {{ .Values.metrics.port }}
+          name: metrics-port
         resources:
           {{- toYaml .Values.resources | nindent 12 }}
         securityContext:
           {{- toYaml .Values.securityContext | nindent 12 }}
+      {{- end }}
       terminationGracePeriodSeconds: 10
       {{- with .Values.githubWebhookServer.nodeSelector }}
       nodeSelector:
@@ -86,4 +101,8 @@ spec:
       tolerations:
         {{- toYaml . | nindent 8 }}
       {{- end }}
+      {{- with .Values.githubWebhookServer.topologySpreadConstraints }}
+      topologySpreadConstraints:
+        {{- toYaml . | nindent 8 }}
+      {{- end }}
 {{- end }}

charts/actions-runner-controller/templates/githubwebhook.ingress.yaml

@@ -1,6 +1,6 @@
 {{- if .Values.githubWebhookServer.ingress.enabled -}}
 {{- $fullName := include "actions-runner-controller-github-webhook-server.fullname" . -}}
-{{- $svcPort := .Values.githubWebhookServer.service.port -}}
+{{- $svcPort := (index .Values.githubWebhookServer.service.ports 0).port -}}
 {{- if semverCompare ">=1.14-0" .Capabilities.KubeVersion.GitVersion -}}
 apiVersion: networking.k8s.io/v1beta1
 {{- else -}}

charts/actions-runner-controller/templates/githubwebhook.service.yaml

@@ -12,6 +12,11 @@ spec:
     {{ range $_, $port := .Values.githubWebhookServer.service.ports -}}
     - {{ $port | toYaml | nindent 6 }}
     {{- end }}
+    {{- if .Values.metrics.serviceMonitor }}
+    - name: metrics-port
+      port: {{ .Values.metrics.port }}
+      targetPort: metrics-port
+    {{- end }}
   selector:
     {{- include "actions-runner-controller-github-webhook-server.selectorLabels" . | nindent 4 }}
 {{- end }}

charts/actions-runner-controller/templates/githubwebhook.serviceMonitor.yaml

@@ -0,0 +1,15 @@
+{{- if and .Values.githubWebhookServer.enabled .Values.metrics.serviceMonitor }}
+apiVersion: monitoring.coreos.com/v1
+kind: ServiceMonitor
+metadata:
+  labels:
+    {{- include "actions-runner-controller.labels" . | nindent 4 }}
+  name: {{ include "actions-runner-controller-github-webhook-server.serviceMonitorName" . }}
+spec:
+  endpoints:
+    - path: /metrics
+      port: metrics-port
+  selector:
+    matchLabels:
+      {{- include "actions-runner-controller-github-webhook-server.selectorLabels" . | nindent 6 }}
+{{- end }}

charts/actions-runner-controller/templates/webhook_configs.yaml

@@ -26,6 +26,7 @@ webhooks:
     - UPDATE
     resources:
     - runners
+  sideEffects: None
 - clientConfig:
     caBundle: Cg==
     service:
@@ -44,6 +45,7 @@ webhooks:
     - UPDATE
     resources:
     - runnerdeployments
+  sideEffects: None
 - clientConfig:
     caBundle: Cg==
     service:
@@ -62,7 +64,7 @@ webhooks:
     - UPDATE
     resources:
     - runnerreplicasets
-
+  sideEffects: None
 ---
 apiVersion: admissionregistration.k8s.io/v1beta1
 kind: ValidatingWebhookConfiguration
@@ -90,6 +92,7 @@ webhooks:
     - UPDATE
     resources:
     - runners
+  sideEffects: None
 - clientConfig:
     caBundle: Cg==
     service:
@@ -108,6 +111,7 @@ webhooks:
     - UPDATE
     resources:
     - runnerdeployments
+  sideEffects: None
 - clientConfig:
     caBundle: Cg==
     service:
@@ -126,3 +130,4 @@ webhooks:
     - UPDATE
     resources:
     - runnerreplicasets
+  sideEffects: None

charts/actions-runner-controller/values.yaml

@@ -8,6 +8,11 @@ replicaCount: 1
 
 syncPeriod: 10m
 
+# The controller tries its best not to repeat the duplicate GitHub API call
+# within this duration.
+# Defaults to syncPeriod - 10s.
+#githubAPICacheDuration: 30s
+
 # Only 1 authentication method can be deployed at a time
 # Uncomment the configuration you are applying and fill in the details
 authSecret:
@@ -22,15 +27,9 @@ authSecret:
 
 image:
   repository: summerwind/actions-runner-controller
-  tag: "v0.17.0"
   dindSidecarRepositoryAndTag: "docker:dind"
   pullPolicy: IfNotPresent
 
-kube_rbac_proxy:
-  image:
-    repository: quay.io/brancz/kube-rbac-proxy
-    tag: v0.8.0
-
 imagePullSecrets: []
 nameOverride: ""
 fullnameOverride: ""
@@ -46,6 +45,8 @@ serviceAccount:
 
 podAnnotations: {}
 
+podLabels: {}
+
 podSecurityContext:
   {}
   # fsGroup: 2000
@@ -63,6 +64,15 @@ service:
   type: ClusterIP
   port: 443
 
+metrics:
+  serviceMonitor: false
+  port: 8443
+  proxy:
+    enabled: true
+    image:
+      repository: quay.io/brancz/kube-rbac-proxy
+      tag: v0.10.0
+
 resources:
   {}
   # We usually recommend not to specify default resources and to leave this as a conscious
@@ -76,13 +86,6 @@ resources:
   #   cpu: 100m
   #   memory: 128Mi
 
-autoscaling:
-  enabled: false
-  minReplicas: 1
-  maxReplicas: 100
-  targetCPUUtilizationPercentage: 80
-  # targetMemoryUtilizationPercentage: 80
-
 nodeSelector: {}
 
 tolerations: []
@@ -109,9 +112,7 @@ scope:
 
 githubWebhookServer:
   enabled: false
-  labels: {}
   replicaCount: 1
-  syncPeriod: 10m
   secret:
     create: true
     name: "github-webhook-server"
@@ -129,6 +130,7 @@ githubWebhookServer:
     # If not set and create is true, a name is generated using the fullname template
     name: ""
   podAnnotations: {}
+  podLabels: {}
   podSecurityContext: {}
   # fsGroup: 2000
   securityContext: {}

cmd/githubwebhookserver/main.go

@@ -27,6 +27,7 @@ import (
 
 	actionsv1alpha1 "github.com/summerwind/actions-runner-controller/api/v1alpha1"
 	"github.com/summerwind/actions-runner-controller/controllers"
+	zaplib "go.uber.org/zap"
 	"k8s.io/apimachinery/pkg/runtime"
 	clientgoscheme "k8s.io/client-go/kubernetes/scheme"
 	_ "k8s.io/client-go/plugin/pkg/client/auth/exec"
@@ -42,6 +43,13 @@ var (
 	setupLog = ctrl.Log.WithName("setup")
 )
 
+const (
+	logLevelDebug = "debug"
+	logLevelInfo  = "info"
+	logLevelWarn  = "warn"
+	logLevelError = "error"
+)
+
 func init() {
 	_ = clientgoscheme.AddToScheme(scheme)
 
@@ -63,6 +71,7 @@ func main() {
 
 		enableLeaderElection bool
 		syncPeriod           time.Duration
+		logLevel             string
 	)
 
 	webhookSecretToken = os.Getenv("GITHUB_WEBHOOK_SECRET_TOKEN")
@@ -73,6 +82,7 @@ func main() {
 	flag.BoolVar(&enableLeaderElection, "enable-leader-election", false,
 		"Enable leader election for controller manager. Enabling this will ensure there is only one active controller manager.")
 	flag.DurationVar(&syncPeriod, "sync-period", 10*time.Minute, "Determines the minimum frequency at which K8s resources managed by this controller are reconciled. When you use autoscaling, set to a lower value like 10 minute, because this corresponds to the minimum time to react on demand change")
+	flag.StringVar(&logLevel, "log-level", logLevelDebug, `The verbosity of the logging. Valid values are "debug", "info", "warn", "error". Defaults to "debug".`)
 	flag.Parse()
 
 	if webhookSecretToken == "" {
@@ -86,7 +96,19 @@ func main() {
 	}
 
 	logger := zap.New(func(o *zap.Options) {
-		o.Development = true
+		switch logLevel {
+		case logLevelDebug:
+			o.Development = true
+		case logLevelInfo:
+			lvl := zaplib.NewAtomicLevelAt(zaplib.InfoLevel)
+			o.Level = &lvl
+		case logLevelWarn:
+			lvl := zaplib.NewAtomicLevelAt(zaplib.WarnLevel)
+			o.Level = &lvl
+		case logLevelError:
+			lvl := zaplib.NewAtomicLevelAt(zaplib.ErrorLevel)
+			o.Level = &lvl
+		}
 	})
 
 	ctrl.SetLogger(logger)

config/crd/bases/actions.summerwind.dev_horizontalrunnerautoscalers.yaml

@@ -18,6 +18,9 @@ spec:
   - JSONPath: .status.desiredReplicas
     name: Desired
     type: number
+  - JSONPath: .status.scheduledOverridesSummary
+    name: Schedule
+    type: string
   group: actions.summerwind.dev
   names:
     kind: HorizontalRunnerAutoscaler
@@ -185,6 +188,56 @@ spec:
                     type: object
                 type: object
               type: array
+            scheduledOverrides:
+              description: ScheduledOverrides is the list of ScheduledOverride. It
+                can be used to override a few fields of HorizontalRunnerAutoscalerSpec
+                on schedule. The earlier a scheduled override is, the higher it is
+                prioritized.
+              items:
+                description: ScheduledOverride can be used to override a few fields
+                  of HorizontalRunnerAutoscalerSpec on schedule. A schedule can optionally
+                  be recurring, so that the correspoding override happens every day,
+                  week, month, or year.
+                properties:
+                  endTime:
+                    description: EndTime is the time at which the first override ends.
+                    format: date-time
+                    type: string
+                  minReplicas:
+                    description: MinReplicas is the number of runners while overriding.
+                      If omitted, it doesn't override minReplicas.
+                    minimum: 0
+                    nullable: true
+                    type: integer
+                  recurrenceRule:
+                    properties:
+                      frequency:
+                        description: Frequency is the name of a predefined interval
+                          of each recurrence. The valid values are "Daily", "Weekly",
+                          "Monthly", and "Yearly". If empty, the corresponding override
+                          happens only once.
+                        enum:
+                        - Daily
+                        - Weekly
+                        - Monthly
+                        - Yearly
+                        type: string
+                      untilTime:
+                        description: UntilTime is the time of the final recurrence.
+                          If empty, the schedule recurs forever.
+                        format: date-time
+                        type: string
+                    type: object
+                  startTime:
+                    description: StartTime is the time at which the first override
+                      starts.
+                    format: date-time
+                    type: string
+                required:
+                - endTime
+                - startTime
+                type: object
+              type: array
           type: object
         status:
           properties:
@@ -215,6 +268,11 @@ spec:
                 which is updated on mutation by the API Server.
               format: int64
               type: integer
+            scheduledOverridesSummary:
+              description: ScheduledOverridesSummary is the summary of active and
+                upcoming scheduled overrides to be shown in e.g. a column of a `kubectl
+                get hra` output for observability.
+              type: string
           type: object
       type: object
   version: v1alpha1

config/crd/bases/actions.summerwind.dev_runnerdeployments.yaml

@@ -10,12 +10,18 @@ spec:
     - JSONPath: .spec.replicas
       name: Desired
       type: number
-    - JSONPath: .status.availableReplicas
+    - JSONPath: .status.replicas
       name: Current
       type: number
-    - JSONPath: .status.readyReplicas
-      name: Ready
+    - JSONPath: .status.updatedReplicas
+      name: Up-To-Date
       type: number
+    - JSONPath: .status.availableReplicas
+      name: Available
+      type: number
+    - JSONPath: .metadata.creationTimestamp
+      name: Age
+      type: date
   group: actions.summerwind.dev
   names:
     kind: RunnerDeployment
@@ -436,6 +442,35 @@ spec:
                     dockerMTU:
                       format: int64
                       type: integer
+                    dockerRegistryMirror:
+                      type: string
+                    dockerVolumeMounts:
+                      items:
+                        description: VolumeMount describes a mounting of a Volume within a container.
+                        properties:
+                          mountPath:
+                            description: Path within the container at which the volume should be mounted.  Must not contain ':'.
+                            type: string
+                          mountPropagation:
+                            description: mountPropagation determines how mounts are propagated from the host to container and the other way around. When not set, MountPropagationNone is used. This field is beta in 1.10.
+                            type: string
+                          name:
+                            description: This must match the Name of a Volume.
+                            type: string
+                          readOnly:
+                            description: Mounted read-only if true, read-write otherwise (false or unspecified). Defaults to false.
+                            type: boolean
+                          subPath:
+                            description: Path within the volume from which the container's volume should be mounted. Defaults to "" (volume's root).
+                            type: string
+                          subPathExpr:
+                            description: Expanded path within the volume from which the container's volume should be mounted. Behaves similarly to SubPath but environment variable references $(VAR_NAME) are expanded using the container's environment. Defaults to "" (volume's root). SubPathExpr and SubPath are mutually exclusive. This field is beta in 1.15.
+                            type: string
+                        required:
+                          - mountPath
+                          - name
+                        type: object
+                      type: array
                     dockerdContainerResources:
                       description: ResourceRequirements describes the compute resource requirements.
                       properties:
@@ -571,6 +606,8 @@ spec:
                             type: object
                         type: object
                       type: array
+                    ephemeral:
+                      type: boolean
                     ephemeralContainers:
                       items:
                         description: An EphemeralContainer is a container that may be added temporarily to an existing pod for user-initiated activities such as debugging. Ephemeral containers have no resource or scheduling guarantees, and they will not be restarted when they exit or when a pod is removed or restarted. If an ephemeral container causes a pod to exceed its resource allocation, the pod may be evicted. Ephemeral containers may not be added by directly updating the pod spec. They must be added via the pod's ephemeralcontainers subresource, and they will appear in the pod spec once added. This is an alpha feature enabled by the EphemeralContainers feature flag.
@@ -580,6 +617,20 @@ spec:
                       type: array
                     group:
                       type: string
+                    hostAliases:
+                      items:
+                        description: HostAlias holds the mapping between IP and hostnames that will be injected as an entry in the pod's hosts file.
+                        properties:
+                          hostnames:
+                            description: Hostnames for the above IP address.
+                            items:
+                              type: string
+                            type: array
+                          ip:
+                            description: IP address of the host file entry.
+                            type: string
+                        type: object
+                      type: array
                     image:
                       type: string
                     imagePullPolicy:
@@ -637,6 +688,9 @@ spec:
                           description: 'Requests describes the minimum amount of compute resources required. If Requests is omitted for a container, it defaults to Limits if that is explicitly specified, otherwise to an implementation-defined value. More info: https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/'
                           type: object
                       type: object
+                    runtimeClassName:
+                      description: 'RuntimeClassName is the container runtime configuration that containers should run under. More info: https://kubernetes.io/docs/concepts/containers/runtime-class'
+                      type: string
                     securityContext:
                       description: PodSecurityContext holds pod-level security attributes and common container settings. Some fields are also present in container.securityContext.  Field values of container.securityContext take precedence over field values of PodSecurityContext.
                       properties:
@@ -768,6 +822,12 @@ spec:
                           - name
                         type: object
                       type: array
+                    volumeSizeLimit:
+                      anyOf:
+                        - type: integer
+                        - type: string
+                      pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
+                      x-kubernetes-int-or-string: true
                     volumes:
                       items:
                         description: Volume represents a named volume in a pod that may be accessed by any container in the pod.
@@ -1580,15 +1640,20 @@ spec:
         status:
           properties:
             availableReplicas:
+              description: AvailableReplicas is the total number of available runners which have been successfully registered to GitHub and still running. This corresponds to the sum of status.availableReplicas of all the runner replica sets.
               type: integer
             desiredReplicas:
-              description: Replicas is the total number of desired, non-terminated and latest pods to be set for the primary RunnerSet This doesn't include outdated pods while upgrading the deployment and replacing the runnerset.
+              description: DesiredReplicas is the total number of desired, non-terminated and latest pods to be set for the primary RunnerSet This doesn't include outdated pods while upgrading the deployment and replacing the runnerset.
               type: integer
             readyReplicas:
+              description: ReadyReplicas is the total number of available runners which have been successfully registered to GitHub and still running. This corresponds to the sum of status.readyReplicas of all the runner replica sets.
+              type: integer
+            replicas:
+              description: Replicas is the total number of replicas
+              type: integer
+            updatedReplicas:
+              description: ReadyReplicas is the total number of available runners which have been successfully registered to GitHub and still running. This corresponds to status.replicas of the runner replica set that has the desired template hash.
               type: integer
-          required:
-            - availableReplicas
-            - readyReplicas
           type: object
       type: object
   version: v1alpha1

config/crd/bases/actions.summerwind.dev_runnerreplicasets.yaml

@@ -10,12 +10,15 @@ spec:
     - JSONPath: .spec.replicas
       name: Desired
       type: number
-    - JSONPath: .status.availableReplicas
+    - JSONPath: .status.replicas
       name: Current
       type: number
     - JSONPath: .status.readyReplicas
       name: Ready
       type: number
+    - JSONPath: .metadata.creationTimestamp
+      name: Age
+      type: date
   group: actions.summerwind.dev
   names:
     kind: RunnerReplicaSet
@@ -436,6 +439,35 @@ spec:
                     dockerMTU:
                       format: int64
                       type: integer
+                    dockerRegistryMirror:
+                      type: string
+                    dockerVolumeMounts:
+                      items:
+                        description: VolumeMount describes a mounting of a Volume within a container.
+                        properties:
+                          mountPath:
+                            description: Path within the container at which the volume should be mounted.  Must not contain ':'.
+                            type: string
+                          mountPropagation:
+                            description: mountPropagation determines how mounts are propagated from the host to container and the other way around. When not set, MountPropagationNone is used. This field is beta in 1.10.
+                            type: string
+                          name:
+                            description: This must match the Name of a Volume.
+                            type: string
+                          readOnly:
+                            description: Mounted read-only if true, read-write otherwise (false or unspecified). Defaults to false.
+                            type: boolean
+                          subPath:
+                            description: Path within the volume from which the container's volume should be mounted. Defaults to "" (volume's root).
+                            type: string
+                          subPathExpr:
+                            description: Expanded path within the volume from which the container's volume should be mounted. Behaves similarly to SubPath but environment variable references $(VAR_NAME) are expanded using the container's environment. Defaults to "" (volume's root). SubPathExpr and SubPath are mutually exclusive. This field is beta in 1.15.
+                            type: string
+                        required:
+                          - mountPath
+                          - name
+                        type: object
+                      type: array
                     dockerdContainerResources:
                       description: ResourceRequirements describes the compute resource requirements.
                       properties:
@@ -571,6 +603,8 @@ spec:
                             type: object
                         type: object
                       type: array
+                    ephemeral:
+                      type: boolean
                     ephemeralContainers:
                       items:
                         description: An EphemeralContainer is a container that may be added temporarily to an existing pod for user-initiated activities such as debugging. Ephemeral containers have no resource or scheduling guarantees, and they will not be restarted when they exit or when a pod is removed or restarted. If an ephemeral container causes a pod to exceed its resource allocation, the pod may be evicted. Ephemeral containers may not be added by directly updating the pod spec. They must be added via the pod's ephemeralcontainers subresource, and they will appear in the pod spec once added. This is an alpha feature enabled by the EphemeralContainers feature flag.
@@ -580,6 +614,20 @@ spec:
                       type: array
                     group:
                       type: string
+                    hostAliases:
+                      items:
+                        description: HostAlias holds the mapping between IP and hostnames that will be injected as an entry in the pod's hosts file.
+                        properties:
+                          hostnames:
+                            description: Hostnames for the above IP address.
+                            items:
+                              type: string
+                            type: array
+                          ip:
+                            description: IP address of the host file entry.
+                            type: string
+                        type: object
+                      type: array
                     image:
                       type: string
                     imagePullPolicy:
@@ -637,6 +685,9 @@ spec:
                           description: 'Requests describes the minimum amount of compute resources required. If Requests is omitted for a container, it defaults to Limits if that is explicitly specified, otherwise to an implementation-defined value. More info: https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/'
                           type: object
                       type: object
+                    runtimeClassName:
+                      description: 'RuntimeClassName is the container runtime configuration that containers should run under. More info: https://kubernetes.io/docs/concepts/containers/runtime-class'
+                      type: string
                     securityContext:
                       description: PodSecurityContext holds pod-level security attributes and common container settings. Some fields are also present in container.securityContext.  Field values of container.securityContext take precedence over field values of PodSecurityContext.
                       properties:
@@ -768,6 +819,12 @@ spec:
                           - name
                         type: object
                       type: array
+                    volumeSizeLimit:
+                      anyOf:
+                        - type: integer
+                        - type: string
+                      pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
+                      x-kubernetes-int-or-string: true
                     volumes:
                       items:
                         description: Volume represents a named volume in a pod that may be accessed by any container in the pod.
@@ -1580,8 +1637,13 @@ spec:
         status:
           properties:
             availableReplicas:
+              description: AvailableReplicas is the number of runners that are created and Runnning. This is currently same as ReadyReplicas but perserved for future use.
               type: integer
             readyReplicas:
+              description: ReadyReplicas is the number of runners that are created and Runnning.
+              type: integer
+            replicas:
+              description: Replicas is the number of runners that are created and still being managed by this runner replica set.
               type: integer
           required:
             - availableReplicas

config/crd/bases/actions.summerwind.dev_runners.yaml

@@ -22,6 +22,9 @@ spec:
     - JSONPath: .status.phase
       name: Status
       type: string
+    - JSONPath: .metadata.creationTimestamp
+      name: Age
+      type: date
   group: actions.summerwind.dev
   names:
     kind: Runner
@@ -401,6 +404,35 @@ spec:
             dockerMTU:
               format: int64
               type: integer
+            dockerRegistryMirror:
+              type: string
+            dockerVolumeMounts:
+              items:
+                description: VolumeMount describes a mounting of a Volume within a container.
+                properties:
+                  mountPath:
+                    description: Path within the container at which the volume should be mounted.  Must not contain ':'.
+                    type: string
+                  mountPropagation:
+                    description: mountPropagation determines how mounts are propagated from the host to container and the other way around. When not set, MountPropagationNone is used. This field is beta in 1.10.
+                    type: string
+                  name:
+                    description: This must match the Name of a Volume.
+                    type: string
+                  readOnly:
+                    description: Mounted read-only if true, read-write otherwise (false or unspecified). Defaults to false.
+                    type: boolean
+                  subPath:
+                    description: Path within the volume from which the container's volume should be mounted. Defaults to "" (volume's root).
+                    type: string
+                  subPathExpr:
+                    description: Expanded path within the volume from which the container's volume should be mounted. Behaves similarly to SubPath but environment variable references $(VAR_NAME) are expanded using the container's environment. Defaults to "" (volume's root). SubPathExpr and SubPath are mutually exclusive. This field is beta in 1.15.
+                    type: string
+                required:
+                  - mountPath
+                  - name
+                type: object
+              type: array
             dockerdContainerResources:
               description: ResourceRequirements describes the compute resource requirements.
               properties:
@@ -536,6 +568,8 @@ spec:
                     type: object
                 type: object
               type: array
+            ephemeral:
+              type: boolean
             ephemeralContainers:
               items:
                 description: An EphemeralContainer is a container that may be added temporarily to an existing pod for user-initiated activities such as debugging. Ephemeral containers have no resource or scheduling guarantees, and they will not be restarted when they exit or when a pod is removed or restarted. If an ephemeral container causes a pod to exceed its resource allocation, the pod may be evicted. Ephemeral containers may not be added by directly updating the pod spec. They must be added via the pod's ephemeralcontainers subresource, and they will appear in the pod spec once added. This is an alpha feature enabled by the EphemeralContainers feature flag.
@@ -545,6 +579,20 @@ spec:
               type: array
             group:
               type: string
+            hostAliases:
+              items:
+                description: HostAlias holds the mapping between IP and hostnames that will be injected as an entry in the pod's hosts file.
+                properties:
+                  hostnames:
+                    description: Hostnames for the above IP address.
+                    items:
+                      type: string
+                    type: array
+                  ip:
+                    description: IP address of the host file entry.
+                    type: string
+                type: object
+              type: array
             image:
               type: string
             imagePullPolicy:
@@ -602,6 +650,9 @@ spec:
                   description: 'Requests describes the minimum amount of compute resources required. If Requests is omitted for a container, it defaults to Limits if that is explicitly specified, otherwise to an implementation-defined value. More info: https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/'
                   type: object
               type: object
+            runtimeClassName:
+              description: 'RuntimeClassName is the container runtime configuration that containers should run under. More info: https://kubernetes.io/docs/concepts/containers/runtime-class'
+              type: string
             securityContext:
               description: PodSecurityContext holds pod-level security attributes and common container settings. Some fields are also present in container.securityContext.  Field values of container.securityContext take precedence over field values of PodSecurityContext.
               properties:
@@ -733,6 +784,12 @@ spec:
                   - name
                 type: object
               type: array
+            volumeSizeLimit:
+              anyOf:
+                - type: integer
+                - type: string
+              pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
+              x-kubernetes-int-or-string: true
             volumes:
               items:
                 description: Volume represents a named volume in a pod that may be accessed by any container in the pod.
@@ -1543,6 +1600,7 @@ spec:
           properties:
             lastRegistrationCheckTime:
               format: date-time
+              nullable: true
               type: string
             message:
               type: string
@@ -1572,11 +1630,6 @@ spec:
                 - expiresAt
                 - token
               type: object
-          required:
-            - message
-            - phase
-            - reason
-            - registration
           type: object
       type: object
   version: v1alpha1

config/default/manager_auth_proxy_patch.yaml

@@ -10,7 +10,7 @@ spec:
     spec:
       containers:
       - name: kube-rbac-proxy
-        image: quay.io/brancz/kube-rbac-proxy:v0.8.0
+        image: quay.io/brancz/kube-rbac-proxy:v0.10.0
         args:
         - "--secure-listen-address=0.0.0.0:8443"
         - "--upstream=http://127.0.0.1:8080/"

config/manager/kustomization.yaml

@@ -4,5 +4,5 @@ apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 images:
 - name: controller
-  newName: summerwind/actions-runner-controller
-  newTag: latest
+  newName: mumoshu/actions-runner-controller
+  newTag: dev

config/webhook/manifests.yaml

@@ -24,6 +24,7 @@ webhooks:
     - UPDATE
     resources:
     - runners
+  sideEffects: None
 - clientConfig:
     caBundle: Cg==
     service:
@@ -60,6 +61,7 @@ webhooks:
     - UPDATE
     resources:
     - runnerreplicasets
+  sideEffects: None
 
 ---
 apiVersion: admissionregistration.k8s.io/v1beta1
@@ -86,6 +88,7 @@ webhooks:
     - UPDATE
     resources:
     - runners
+  sideEffects: None
 - clientConfig:
     caBundle: Cg==
     service:
@@ -122,3 +125,4 @@ webhooks:
     - UPDATE
     resources:
     - runnerreplicasets
+  sideEffects: None

controllers/autoscaling.go

@@ -34,7 +34,7 @@ func getValueAvailableAt(now time.Time, from, to *time.Time, reservedValue int)
 	return &reservedValue
 }
 
-func (r *HorizontalRunnerAutoscalerReconciler) getDesiredReplicasFromCache(hra v1alpha1.HorizontalRunnerAutoscaler) *int {
+func (r *HorizontalRunnerAutoscalerReconciler) fetchSuggestedReplicasFromCache(hra v1alpha1.HorizontalRunnerAutoscaler) *int {
 	var entry *v1alpha1.CacheEntry
 
 	for i := range hra.Status.CacheEntries {
@@ -63,7 +63,7 @@ func (r *HorizontalRunnerAutoscalerReconciler) getDesiredReplicasFromCache(hra v
 	return nil
 }
 
-func (r *HorizontalRunnerAutoscalerReconciler) determineDesiredReplicas(rd v1alpha1.RunnerDeployment, hra v1alpha1.HorizontalRunnerAutoscaler) (*int, error) {
+func (r *HorizontalRunnerAutoscalerReconciler) suggestDesiredReplicas(rd v1alpha1.RunnerDeployment, hra v1alpha1.HorizontalRunnerAutoscaler) (*int, error) {
 	if hra.Spec.MinReplicas == nil {
 		return nil, fmt.Errorf("horizontalrunnerautoscaler %s/%s is missing minReplicas", hra.Namespace, hra.Name)
 	} else if hra.Spec.MaxReplicas == nil {
@@ -71,25 +71,68 @@ func (r *HorizontalRunnerAutoscalerReconciler) determineDesiredReplicas(rd v1alp
 	}
 
 	metrics := hra.Spec.Metrics
-	if len(metrics) == 0 {
+	numMetrics := len(metrics)
+	if numMetrics == 0 {
 		if len(hra.Spec.ScaleUpTriggers) == 0 {
-			return r.calculateReplicasByQueuedAndInProgressWorkflowRuns(rd, hra)
+			return r.suggestReplicasByQueuedAndInProgressWorkflowRuns(rd, hra, nil)
 		}
 
-		return hra.Spec.MinReplicas, nil
-	} else if metrics[0].Type == v1alpha1.AutoscalingMetricTypeTotalNumberOfQueuedAndInProgressWorkflowRuns {
-		return r.calculateReplicasByQueuedAndInProgressWorkflowRuns(rd, hra)
-	} else if metrics[0].Type == v1alpha1.AutoscalingMetricTypePercentageRunnersBusy {
-		return r.calculateReplicasByPercentageRunnersBusy(rd, hra)
-	} else {
-		return nil, fmt.Errorf("validting autoscaling metrics: unsupported metric type %q", metrics[0].Type)
+		return nil, nil
+	} else if numMetrics > 2 {
+		return nil, fmt.Errorf("Too many autoscaling metrics configured: It must be 0 to 2, but got %d", numMetrics)
 	}
+
+	primaryMetric := metrics[0]
+	primaryMetricType := primaryMetric.Type
+
+	var (
+		suggested *int
+		err       error
+	)
+
+	switch primaryMetricType {
+	case v1alpha1.AutoscalingMetricTypeTotalNumberOfQueuedAndInProgressWorkflowRuns:
+		suggested, err = r.suggestReplicasByQueuedAndInProgressWorkflowRuns(rd, hra, &primaryMetric)
+	case v1alpha1.AutoscalingMetricTypePercentageRunnersBusy:
+		suggested, err = r.suggestReplicasByPercentageRunnersBusy(rd, hra, primaryMetric)
+	default:
+		return nil, fmt.Errorf("validting autoscaling metrics: unsupported metric type %q", primaryMetric)
+	}
+
+	if err != nil {
+		return nil, err
+	}
+
+	if suggested != nil && *suggested > 0 {
+		return suggested, nil
+	}
+
+	if len(metrics) == 1 {
+		// This is never supposed to happen but anyway-
+		// Fall-back to `minReplicas + capacityReservedThroughWebhook`.
+		return nil, nil
+	}
+
+	// At this point, we are sure that there are exactly 2 Metrics entries.
+
+	fallbackMetric := metrics[1]
+	fallbackMetricType := fallbackMetric.Type
+
+	if primaryMetricType != v1alpha1.AutoscalingMetricTypePercentageRunnersBusy ||
+		fallbackMetricType != v1alpha1.AutoscalingMetricTypeTotalNumberOfQueuedAndInProgressWorkflowRuns {
+
+		return nil, fmt.Errorf(
+			"invalid HRA Spec: Metrics[0] of %s cannot be combined with Metrics[1] of %s: The only allowed combination is 0=PercentageRunnersBusy and 1=TotalNumberOfQueuedAndInProgressWorkflowRuns",
+			primaryMetricType, fallbackMetricType,
+		)
+	}
+
+	return r.suggestReplicasByQueuedAndInProgressWorkflowRuns(rd, hra, &fallbackMetric)
 }
 
-func (r *HorizontalRunnerAutoscalerReconciler) calculateReplicasByQueuedAndInProgressWorkflowRuns(rd v1alpha1.RunnerDeployment, hra v1alpha1.HorizontalRunnerAutoscaler) (*int, error) {
+func (r *HorizontalRunnerAutoscalerReconciler) suggestReplicasByQueuedAndInProgressWorkflowRuns(rd v1alpha1.RunnerDeployment, hra v1alpha1.HorizontalRunnerAutoscaler, metrics *v1alpha1.MetricSpec) (*int, error) {
 
 	var repos [][]string
-	metrics := hra.Spec.Metrics
 	repoID := rd.Spec.Template.Spec.Repository
 	if repoID == "" {
 		orgName := rd.Spec.Template.Spec.Organization
@@ -100,15 +143,15 @@ func (r *HorizontalRunnerAutoscalerReconciler) calculateReplicasByQueuedAndInPro
 		// In case it's an organizational runners deployment without any scaling metrics defined,
 		// we assume that the desired replicas should always be `minReplicas + capacityReservedThroughWebhook`.
 		// See https://github.com/summerwind/actions-runner-controller/issues/377#issuecomment-793372693
-		if len(metrics) == 0 {
-			return hra.Spec.MinReplicas, nil
+		if metrics == nil {
+			return nil, nil
 		}
 
-		if len(metrics[0].RepositoryNames) == 0 {
+		if len(metrics.RepositoryNames) == 0 {
 			return nil, errors.New("validating autoscaling metrics: spec.autoscaling.metrics[].repositoryNames is required and must have one more more entries for organizational runner deployment")
 		}
 
-		for _, repoName := range metrics[0].RepositoryNames {
+		for _, repoName := range metrics.RepositoryNames {
 			repos = append(repos, []string{orgName, repoName})
 		}
 	} else {
@@ -178,28 +221,10 @@ func (r *HorizontalRunnerAutoscalerReconciler) calculateReplicasByQueuedAndInPro
 		}
 	}
 
-	minReplicas := *hra.Spec.MinReplicas
-	maxReplicas := *hra.Spec.MaxReplicas
 	necessaryReplicas := queued + inProgress
 
-	var desiredReplicas int
-
-	if necessaryReplicas < minReplicas {
-		desiredReplicas = minReplicas
-	} else if necessaryReplicas > maxReplicas {
-		desiredReplicas = maxReplicas
-	} else {
-		desiredReplicas = necessaryReplicas
-	}
-
-	rd.Status.Replicas = &desiredReplicas
-	replicas := desiredReplicas
-
 	r.Log.V(1).Info(
-		"Calculated desired replicas",
-		"computed_replicas_desired", desiredReplicas,
-		"spec_replicas_min", minReplicas,
-		"spec_replicas_max", maxReplicas,
+		fmt.Sprintf("Suggested desired replicas of %d by TotalNumberOfQueuedAndInProgressWorkflowRuns", necessaryReplicas),
 		"workflow_runs_completed", completed,
 		"workflow_runs_in_progress", inProgress,
 		"workflow_runs_queued", queued,
@@ -209,14 +234,11 @@ func (r *HorizontalRunnerAutoscalerReconciler) calculateReplicasByQueuedAndInPro
 		"horizontal_runner_autoscaler", hra.Name,
 	)
 
-	return &replicas, nil
+	return &necessaryReplicas, nil
 }
 
-func (r *HorizontalRunnerAutoscalerReconciler) calculateReplicasByPercentageRunnersBusy(rd v1alpha1.RunnerDeployment, hra v1alpha1.HorizontalRunnerAutoscaler) (*int, error) {
+func (r *HorizontalRunnerAutoscalerReconciler) suggestReplicasByPercentageRunnersBusy(rd v1alpha1.RunnerDeployment, hra v1alpha1.HorizontalRunnerAutoscaler, metrics v1alpha1.MetricSpec) (*int, error) {
 	ctx := context.Background()
-	minReplicas := *hra.Spec.MinReplicas
-	maxReplicas := *hra.Spec.MaxReplicas
-	metrics := hra.Spec.Metrics[0]
 	scaleUpThreshold := defaultScaleUpThreshold
 	scaleDownThreshold := defaultScaleDownThreshold
 	scaleUpFactor := defaultScaleUpFactor
@@ -363,21 +385,13 @@ func (r *HorizontalRunnerAutoscalerReconciler) calculateReplicasByPercentageRunn
 		desiredReplicas = *rd.Spec.Replicas
 	}
 
-	if desiredReplicas < minReplicas {
-		desiredReplicas = minReplicas
-	} else if desiredReplicas > maxReplicas {
-		desiredReplicas = maxReplicas
-	}
-
 	// NOTES for operators:
 	//
 	// - num_runners can be as twice as large as replicas_desired_before while
 	//   the runnerdeployment controller is replacing RunnerReplicaSet for runner update.
 
 	r.Log.V(1).Info(
-		"Calculated desired replicas",
-		"replicas_min", minReplicas,
-		"replicas_max", maxReplicas,
+		fmt.Sprintf("Suggested desired replicas of %d by PercentageRunnersBusy", desiredReplicas),
 		"replicas_desired_before", desiredReplicasBefore,
 		"replicas_desired", desiredReplicas,
 		"num_runners", numRunners,
@@ -391,8 +405,5 @@ func (r *HorizontalRunnerAutoscalerReconciler) calculateReplicasByPercentageRunn
 		"repository", repository,
 	)
 
-	rd.Status.Replicas = &desiredReplicas
-	replicas := desiredReplicas
-
-	return &replicas, nil
+	return &desiredReplicas, nil
 }

controllers/autoscaling_test.go

@@ -209,7 +209,7 @@ func TestDetermineDesiredReplicas_RepositoryRunner(t *testing.T) {
 					Replicas: tc.fixed,
 				},
 				Status: v1alpha1.RunnerDeploymentStatus{
-					Replicas: tc.sReplicas,
+					DesiredReplicas: tc.sReplicas,
 				},
 			}
 
@@ -224,7 +224,12 @@ func TestDetermineDesiredReplicas_RepositoryRunner(t *testing.T) {
 				},
 			}
 
-			got, err := h.computeReplicas(rd, hra)
+			minReplicas, _, _, err := h.getMinReplicas(log, metav1Now.Time, hra)
+			if err != nil {
+				t.Fatalf("unexpected error: %v", err)
+			}
+
+			got, _, _, err := h.computeReplicasWithCache(log, metav1Now.Time, rd, hra, minReplicas)
 			if err != nil {
 				if tc.err == "" {
 					t.Fatalf("unexpected error: expected none, got %v", err)
@@ -234,12 +239,8 @@ func TestDetermineDesiredReplicas_RepositoryRunner(t *testing.T) {
 				return
 			}
 
-			if got == nil {
-				t.Fatalf("unexpected value of rs.Spec.Replicas: nil")
-			}
-
-			if *got != tc.want {
-				t.Errorf("%d: incorrect desired replicas: want %d, got %d", i, tc.want, *got)
+			if got != tc.want {
+				t.Errorf("%d: incorrect desired replicas: want %d, got %d", i, tc.want, got)
 			}
 		})
 	}
@@ -424,6 +425,8 @@ func TestDetermineDesiredReplicas_OrganizationalRunner(t *testing.T) {
 		_ = v1alpha1.AddToScheme(scheme)
 
 		t.Run(fmt.Sprintf("case %d", i), func(t *testing.T) {
+			t.Helper()
+
 			server := fake.NewServer(
 				fake.WithListRepositoryWorkflowRunsResponse(200, tc.workflowRuns, tc.workflowRuns_queued, tc.workflowRuns_in_progress),
 				fake.WithListWorkflowJobsResponse(200, tc.workflowJobs),
@@ -461,7 +464,7 @@ func TestDetermineDesiredReplicas_OrganizationalRunner(t *testing.T) {
 					Replicas: tc.fixed,
 				},
 				Status: v1alpha1.RunnerDeploymentStatus{
-					Replicas: tc.sReplicas,
+					DesiredReplicas: tc.sReplicas,
 				},
 			}
 
@@ -485,7 +488,12 @@ func TestDetermineDesiredReplicas_OrganizationalRunner(t *testing.T) {
 				},
 			}
 
-			got, err := h.computeReplicas(rd, hra)
+			minReplicas, _, _, err := h.getMinReplicas(log, metav1Now.Time, hra)
+			if err != nil {
+				t.Fatalf("unexpected error: %v", err)
+			}
+
+			got, _, _, err := h.computeReplicasWithCache(log, metav1Now.Time, rd, hra, minReplicas)
 			if err != nil {
 				if tc.err == "" {
 					t.Fatalf("unexpected error: expected none, got %v", err)
@@ -495,12 +503,8 @@ func TestDetermineDesiredReplicas_OrganizationalRunner(t *testing.T) {
 				return
 			}
 
-			if got == nil {
-				t.Fatalf("unexpected value of rs.Spec.Replicas: nil, wanted %v", tc.want)
-			}
-
-			if *got != tc.want {
-				t.Errorf("%d: incorrect desired replicas: want %d, got %d", i, tc.want, *got)
+			if got != tc.want {
+				t.Errorf("%d: incorrect desired replicas: want %d, got %d", i, tc.want, got)
 			}
 		})
 	}

controllers/horizontal_runner_autoscaler_webhook.go

@@ -20,13 +20,14 @@ import (
 	"context"
 	"fmt"
 	"io/ioutil"
-	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
-	"k8s.io/apimachinery/pkg/types"
 	"net/http"
-	"sigs.k8s.io/controller-runtime/pkg/reconcile"
 	"strings"
 	"time"
 
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/apimachinery/pkg/types"
+	"sigs.k8s.io/controller-runtime/pkg/reconcile"
+
 	"github.com/go-logr/logr"
 	gogithub "github.com/google/go-github/v33/github"
 	"k8s.io/apimachinery/pkg/runtime"
@@ -330,6 +331,8 @@ func (autoscaler *HorizontalRunnerAutoscalerGitHubWebhook) getScaleTarget(ctx co
 		return nil, err
 	}
 
+	autoscaler.Log.V(1).Info(fmt.Sprintf("Found %d HRAs by key", len(hras)), "key", name)
+
 	targets := autoscaler.searchScaleTargets(hras, f)
 
 	n := len(targets)
@@ -362,14 +365,16 @@ func (autoscaler *HorizontalRunnerAutoscalerGitHubWebhook) getScaleUpTarget(ctx
 	repositoryRunnerKey := owner + "/" + repo
 
 	if target, err := autoscaler.getScaleTarget(ctx, repositoryRunnerKey, f); err != nil {
-		autoscaler.Log.Info("finding repository-wide runner", "repository", repositoryRunnerKey)
+		log.Info("finding repository-wide runner", "repository", repositoryRunnerKey)
 		return nil, err
 	} else if target != nil {
-		autoscaler.Log.Info("scale up target is repository-wide runners", "repository", repo)
+		log.Info("scale up target is repository-wide runners", "repository", repo)
 		return target, nil
 	}
 
 	if ownerType == "User" {
+		log.V(1).Info("no repository runner found", "organization", owner)
+
 		return nil, nil
 	}
 
@@ -379,6 +384,11 @@ func (autoscaler *HorizontalRunnerAutoscalerGitHubWebhook) getScaleUpTarget(ctx
 	} else if target != nil {
 		log.Info("scale up target is organizational runners", "organization", owner)
 		return target, nil
+	} else {
+		log.V(1).Info("no repository runner or organizational runner found",
+			"repository", repositoryRunnerKey,
+			"organization", owner,
+		)
 	}
 
 	return nil, nil

controllers/horizontalrunnerautoscaler_controller.go

@@ -19,8 +19,11 @@ package controllers
 import (
 	"context"
 	"fmt"
+	"reflect"
 	"time"
 
+	corev1 "k8s.io/api/core/v1"
+
 	"github.com/summerwind/actions-runner-controller/github"
 	"k8s.io/apimachinery/pkg/types"
 
@@ -30,10 +33,10 @@ import (
 	ctrl "sigs.k8s.io/controller-runtime"
 	"sigs.k8s.io/controller-runtime/pkg/client"
 
-	corev1 "k8s.io/api/core/v1"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 
 	"github.com/summerwind/actions-runner-controller/api/v1alpha1"
+	"github.com/summerwind/actions-runner-controller/controllers/metrics"
 )
 
 const (
@@ -52,6 +55,8 @@ type HorizontalRunnerAutoscalerReconciler struct {
 	Name          string
 }
 
+const defaultReplicas = 1
+
 // +kubebuilder:rbac:groups=actions.summerwind.dev,resources=runnerdeployments,verbs=get;list;watch;update;patch
 // +kubebuilder:rbac:groups=actions.summerwind.dev,resources=horizontalrunnerautoscalers,verbs=get;list;watch;create;update;patch;delete
 // +kubebuilder:rbac:groups=actions.summerwind.dev,resources=horizontalrunnerautoscalers/finalizers,verbs=get;list;watch;create;update;patch;delete
@@ -71,6 +76,8 @@ func (r *HorizontalRunnerAutoscalerReconciler) Reconcile(req ctrl.Request) (ctrl
 		return ctrl.Result{}, nil
 	}
 
+	metrics.SetHorizontalRunnerAutoscalerSpec(hra.ObjectMeta, hra.Spec)
+
 	var rd v1alpha1.RunnerDeployment
 	if err := r.Get(ctx, types.NamespacedName{
 		Namespace: req.Namespace,
@@ -83,42 +90,26 @@ func (r *HorizontalRunnerAutoscalerReconciler) Reconcile(req ctrl.Request) (ctrl
 		return ctrl.Result{}, nil
 	}
 
-	var replicas *int
-
-	replicasFromCache := r.getDesiredReplicasFromCache(hra)
-
-	if replicasFromCache != nil {
-		replicas = replicasFromCache
-	} else {
-		var err error
-
-		replicas, err = r.computeReplicas(rd, hra)
-		if err != nil {
-			r.Recorder.Event(&hra, corev1.EventTypeNormal, "RunnerAutoscalingFailure", err.Error())
-
-			log.Error(err, "Could not compute replicas")
-
-			return ctrl.Result{}, err
-		}
-	}
-
-	const defaultReplicas = 1
-
-	currentDesiredReplicas := getIntOrDefault(rd.Spec.Replicas, defaultReplicas)
-	newDesiredReplicas := getIntOrDefault(replicas, defaultReplicas)
-
 	now := time.Now()
 
-	for _, reservation := range hra.Spec.CapacityReservations {
-		if reservation.ExpirationTime.Time.After(now) {
-			newDesiredReplicas += reservation.Replicas
-		}
+	minReplicas, active, upcoming, err := r.getMinReplicas(log, now, hra)
+	if err != nil {
+		log.Error(err, "Could not compute min replicas")
+
+		return ctrl.Result{}, err
 	}
 
-	if hra.Spec.MaxReplicas != nil && *hra.Spec.MaxReplicas < newDesiredReplicas {
-		newDesiredReplicas = *hra.Spec.MaxReplicas
+	newDesiredReplicas, computedReplicas, computedReplicasFromCache, err := r.computeReplicasWithCache(log, now, rd, hra, minReplicas)
+	if err != nil {
+		r.Recorder.Event(&hra, corev1.EventTypeNormal, "RunnerAutoscalingFailure", err.Error())
+
+		log.Error(err, "Could not compute replicas")
+
+		return ctrl.Result{}, err
 	}
 
+	currentDesiredReplicas := getIntOrDefault(rd.Spec.Replicas, defaultReplicas)
+
 	// Please add more conditions that we can in-place update the newest runnerreplicaset without disruption
 	if currentDesiredReplicas != newDesiredReplicas {
 		copy := rd.DeepCopy()
@@ -129,11 +120,9 @@ func (r *HorizontalRunnerAutoscalerReconciler) Reconcile(req ctrl.Request) (ctrl
 		}
 	}
 
-	var updated *v1alpha1.HorizontalRunnerAutoscaler
+	updated := hra.DeepCopy()
 
 	if hra.Status.DesiredReplicas == nil || *hra.Status.DesiredReplicas != newDesiredReplicas {
-		updated = hra.DeepCopy()
-
 		if (hra.Status.DesiredReplicas == nil && newDesiredReplicas > 1) ||
 			(hra.Status.DesiredReplicas != nil && newDesiredReplicas > *hra.Status.DesiredReplicas) {
 
@@ -143,11 +132,7 @@ func (r *HorizontalRunnerAutoscalerReconciler) Reconcile(req ctrl.Request) (ctrl
 		updated.Status.DesiredReplicas = &newDesiredReplicas
 	}
 
-	if replicasFromCache == nil {
-		if updated == nil {
-			updated = hra.DeepCopy()
-		}
-
+	if computedReplicasFromCache == nil {
 		cacheEntries := getValidCacheEntries(updated, now)
 
 		var cacheDuration time.Duration
@@ -160,14 +145,39 @@ func (r *HorizontalRunnerAutoscalerReconciler) Reconcile(req ctrl.Request) (ctrl
 
 		updated.Status.CacheEntries = append(cacheEntries, v1alpha1.CacheEntry{
 			Key:            v1alpha1.CacheEntryKeyDesiredReplicas,
-			Value:          *replicas,
+			Value:          computedReplicas,
 			ExpirationTime: metav1.Time{Time: time.Now().Add(cacheDuration)},
 		})
 	}
 
-	if updated != nil {
+	var overridesSummary string
+
+	if (active != nil && upcoming == nil) || (active != nil && upcoming != nil && active.Period.EndTime.Before(upcoming.Period.StartTime)) {
+		after := defaultReplicas
+		if hra.Spec.MinReplicas != nil && *hra.Spec.MinReplicas >= 0 {
+			after = *hra.Spec.MinReplicas
+		}
+
+		overridesSummary = fmt.Sprintf("min=%d time=%s", after, active.Period.EndTime)
+	}
+
+	if active == nil && upcoming != nil || (active != nil && upcoming != nil && active.Period.EndTime.After(upcoming.Period.StartTime)) {
+		if upcoming.ScheduledOverride.MinReplicas != nil {
+			overridesSummary = fmt.Sprintf("min=%d time=%s", *upcoming.ScheduledOverride.MinReplicas, upcoming.Period.StartTime)
+		}
+	}
+
+	if overridesSummary != "" {
+		updated.Status.ScheduledOverridesSummary = &overridesSummary
+	} else {
+		updated.Status.ScheduledOverridesSummary = nil
+	}
+
+	if !reflect.DeepEqual(hra.Status, updated.Status) {
+		metrics.SetHorizontalRunnerAutoscalerStatus(updated.ObjectMeta, updated.Status)
+
 		if err := r.Status().Patch(ctx, updated, client.MergeFrom(&hra)); err != nil {
-			return ctrl.Result{}, fmt.Errorf("patching horizontalrunnerautoscaler status to add cache entry: %w", err)
+			return ctrl.Result{}, fmt.Errorf("patching horizontalrunnerautoscaler status: %w", err)
 		}
 	}
 
@@ -200,14 +210,132 @@ func (r *HorizontalRunnerAutoscalerReconciler) SetupWithManager(mgr ctrl.Manager
 		Complete(r)
 }
 
-func (r *HorizontalRunnerAutoscalerReconciler) computeReplicas(rd v1alpha1.RunnerDeployment, hra v1alpha1.HorizontalRunnerAutoscaler) (*int, error) {
-	var computedReplicas *int
+type Override struct {
+	ScheduledOverride v1alpha1.ScheduledOverride
+	Period            Period
+}
 
-	replicas, err := r.determineDesiredReplicas(rd, hra)
-	if err != nil {
-		return nil, err
+func (r *HorizontalRunnerAutoscalerReconciler) matchScheduledOverrides(log logr.Logger, now time.Time, hra v1alpha1.HorizontalRunnerAutoscaler) (*int, *Override, *Override, error) {
+	var minReplicas *int
+	var active, upcoming *Override
+
+	for _, o := range hra.Spec.ScheduledOverrides {
+		log.V(1).Info(
+			"Checking scheduled override",
+			"now", now,
+			"startTime", o.StartTime,
+			"endTime", o.EndTime,
+			"frequency", o.RecurrenceRule.Frequency,
+			"untilTime", o.RecurrenceRule.UntilTime,
+		)
+
+		a, u, err := MatchSchedule(
+			now, o.StartTime.Time, o.EndTime.Time,
+			RecurrenceRule{
+				Frequency: o.RecurrenceRule.Frequency,
+				UntilTime: o.RecurrenceRule.UntilTime.Time,
+			},
+		)
+		if err != nil {
+			return minReplicas, nil, nil, err
+		}
+
+		// Use the first when there are two or more active scheduled overrides,
+		// as the spec defines that the earlier scheduled override is prioritized higher than later ones.
+		if a != nil && active == nil {
+			active = &Override{Period: *a, ScheduledOverride: o}
+
+			if o.MinReplicas != nil {
+				minReplicas = o.MinReplicas
+
+				log.V(1).Info(
+					"Found active scheduled override",
+					"activeStartTime", a.StartTime,
+					"activeEndTime", a.EndTime,
+					"activeMinReplicas", minReplicas,
+				)
+			}
+		}
+
+		if u != nil && (upcoming == nil || u.StartTime.Before(upcoming.Period.StartTime)) {
+			upcoming = &Override{Period: *u, ScheduledOverride: o}
+
+			log.V(1).Info(
+				"Found upcoming scheduled override",
+				"upcomingStartTime", u.StartTime,
+				"upcomingEndTime", u.EndTime,
+				"upcomingMinReplicas", o.MinReplicas,
+			)
+		}
 	}
 
+	return minReplicas, active, upcoming, nil
+}
+
+func (r *HorizontalRunnerAutoscalerReconciler) getMinReplicas(log logr.Logger, now time.Time, hra v1alpha1.HorizontalRunnerAutoscaler) (int, *Override, *Override, error) {
+	minReplicas := defaultReplicas
+	if hra.Spec.MinReplicas != nil && *hra.Spec.MinReplicas >= 0 {
+		minReplicas = *hra.Spec.MinReplicas
+	}
+
+	m, active, upcoming, err := r.matchScheduledOverrides(log, now, hra)
+	if err != nil {
+		return 0, nil, nil, err
+	} else if m != nil {
+		minReplicas = *m
+	}
+
+	return minReplicas, active, upcoming, nil
+}
+
+func (r *HorizontalRunnerAutoscalerReconciler) computeReplicasWithCache(log logr.Logger, now time.Time, rd v1alpha1.RunnerDeployment, hra v1alpha1.HorizontalRunnerAutoscaler, minReplicas int) (int, int, *int, error) {
+	var suggestedReplicas int
+
+	suggestedReplicasFromCache := r.fetchSuggestedReplicasFromCache(hra)
+
+	var cached *int
+
+	if suggestedReplicasFromCache != nil {
+		cached = suggestedReplicasFromCache
+
+		if cached == nil {
+			suggestedReplicas = minReplicas
+		} else {
+			suggestedReplicas = *cached
+		}
+	} else {
+		v, err := r.suggestDesiredReplicas(rd, hra)
+		if err != nil {
+			return 0, 0, nil, err
+		}
+
+		if v == nil {
+			suggestedReplicas = minReplicas
+		} else {
+			suggestedReplicas = *v
+		}
+	}
+
+	var reserved int
+
+	for _, reservation := range hra.Spec.CapacityReservations {
+		if reservation.ExpirationTime.Time.After(now) {
+			reserved += reservation.Replicas
+		}
+	}
+
+	newDesiredReplicas := suggestedReplicas + reserved
+
+	if newDesiredReplicas < minReplicas {
+		newDesiredReplicas = minReplicas
+	} else if hra.Spec.MaxReplicas != nil && newDesiredReplicas > *hra.Spec.MaxReplicas {
+		newDesiredReplicas = *hra.Spec.MaxReplicas
+	}
+
+	//
+	// Delay scaling-down for ScaleDownDelaySecondsAfterScaleUp or DefaultScaleDownDelay
+	//
+
 	var scaleDownDelay time.Duration
 
 	if hra.Spec.ScaleDownDelaySecondsAfterScaleUp != nil {
@@ -216,17 +344,50 @@ func (r *HorizontalRunnerAutoscalerReconciler) computeReplicas(rd v1alpha1.Runne
 		scaleDownDelay = DefaultScaleDownDelay
 	}
 
-	now := time.Now()
+	var scaleDownDelayUntil *time.Time
 
 	if hra.Status.DesiredReplicas == nil ||
-		*hra.Status.DesiredReplicas < *replicas ||
-		hra.Status.LastSuccessfulScaleOutTime == nil ||
-		hra.Status.LastSuccessfulScaleOutTime.Add(scaleDownDelay).Before(now) {
+		*hra.Status.DesiredReplicas < newDesiredReplicas ||
+		hra.Status.LastSuccessfulScaleOutTime == nil {
 
-		computedReplicas = replicas
+	} else if hra.Status.LastSuccessfulScaleOutTime != nil {
+		t := hra.Status.LastSuccessfulScaleOutTime.Add(scaleDownDelay)
+
+		// ScaleDownDelay is not passed
+		if t.After(now) {
+			scaleDownDelayUntil = &t
+			newDesiredReplicas = *hra.Status.DesiredReplicas
+		}
 	} else {
-		computedReplicas = hra.Status.DesiredReplicas
+		newDesiredReplicas = *hra.Status.DesiredReplicas
 	}
 
-	return computedReplicas, nil
+	//
+	// Logs various numbers for monitoring and debugging purpose
+	//
+
+	kvs := []interface{}{
+		"suggested", suggestedReplicas,
+		"reserved", reserved,
+		"min", minReplicas,
+	}
+
+	if cached != nil {
+		kvs = append(kvs, "cached", *cached)
+	}
+
+	if scaleDownDelayUntil != nil {
+		kvs = append(kvs, "last_scale_up_time", *hra.Status.LastSuccessfulScaleOutTime)
+		kvs = append(kvs, "scale_down_delay_until", scaleDownDelayUntil)
+	}
+
+	if maxReplicas := hra.Spec.MaxReplicas; maxReplicas != nil {
+		kvs = append(kvs, "max", *maxReplicas)
+	}
+
+	log.V(1).Info(fmt.Sprintf("Calculated desired replicas of %d", newDesiredReplicas),
+		kvs...,
+	)
+
+	return newDesiredReplicas, suggestedReplicas, suggestedReplicasFromCache, nil
 }

controllers/integration_test.go

@@ -71,7 +71,9 @@ func SetupIntegrationTest(ctx context.Context) *testEnvironment {
 		err := k8sClient.Create(ctx, ns)
 		Expect(err).NotTo(HaveOccurred(), "failed to create test namespace")
 
-		mgr, err := ctrl.NewManager(cfg, ctrl.Options{})
+		mgr, err := ctrl.NewManager(cfg, ctrl.Options{
+			Namespace: ns.Name,
+		})
 		Expect(err).NotTo(HaveOccurred(), "failed to create manager")
 
 		responses := &fake.FixedResponses{}
@@ -97,6 +99,21 @@ func SetupIntegrationTest(ctx context.Context) *testEnvironment {
 			return fmt.Sprintf("%s%s", ns.Name, name)
 		}
 
+		runnerController := &RunnerReconciler{
+			Client:                      mgr.GetClient(),
+			Scheme:                      scheme.Scheme,
+			Log:                         logf.Log,
+			Recorder:                    mgr.GetEventRecorderFor("runnerreplicaset-controller"),
+			GitHubClient:                env.ghClient,
+			RunnerImage:                 "example/runner:test",
+			DockerImage:                 "example/docker:test",
+			Name:                        controllerName("runner"),
+			RegistrationRecheckInterval: time.Millisecond,
+			RegistrationRecheckJitter:   time.Millisecond,
+		}
+		err = runnerController.SetupWithManager(mgr)
+		Expect(err).NotTo(HaveOccurred(), "failed to setup runner controller")
+
 		replicasetController := &RunnerReplicaSetReconciler{
 			Client:       mgr.GetClient(),
 			Scheme:       scheme.Scheme,
@@ -106,7 +123,7 @@ func SetupIntegrationTest(ctx context.Context) *testEnvironment {
 			Name:         controllerName("runnerreplicaset"),
 		}
 		err = replicasetController.SetupWithManager(mgr)
-		Expect(err).NotTo(HaveOccurred(), "failed to setup controller")
+		Expect(err).NotTo(HaveOccurred(), "failed to setup runnerreplicaset controller")
 
 		deploymentsController := &RunnerDeploymentReconciler{
 			Client:   mgr.GetClient(),
@@ -116,7 +133,7 @@ func SetupIntegrationTest(ctx context.Context) *testEnvironment {
 			Name:     controllerName("runnnerdeployment"),
 		}
 		err = deploymentsController.SetupWithManager(mgr)
-		Expect(err).NotTo(HaveOccurred(), "failed to setup controller")
+		Expect(err).NotTo(HaveOccurred(), "failed to setup runnerdeployment controller")
 
 		autoscalerController := &HorizontalRunnerAutoscalerReconciler{
 			Client:        mgr.GetClient(),
@@ -128,7 +145,7 @@ func SetupIntegrationTest(ctx context.Context) *testEnvironment {
 			Name:          controllerName("horizontalrunnerautoscaler"),
 		}
 		err = autoscalerController.SetupWithManager(mgr)
-		Expect(err).NotTo(HaveOccurred(), "failed to setup controller")
+		Expect(err).NotTo(HaveOccurred(), "failed to setup autoscaler controller")
 
 		autoscalerWebhook := &HorizontalRunnerAutoscalerGitHubWebhook{
 			Client:    mgr.GetClient(),
@@ -429,9 +446,6 @@ var _ = Context("INTEGRATION: Inside of a new namespace", func() {
 				ExpectCreate(ctx, rd, "test RunnerDeployment")
 				ExpectRunnerSetsCountEventuallyEquals(ctx, ns.Name, 1)
 				ExpectRunnerSetsManagedReplicasCountEventuallyEquals(ctx, ns.Name, 1)
-			}
-
-			{
 				env.ExpectRegisteredNumberCountEventuallyEquals(1, "count of fake list runners")
 			}
 
@@ -475,10 +489,9 @@ var _ = Context("INTEGRATION: Inside of a new namespace", func() {
 
 				ExpectRunnerSetsCountEventuallyEquals(ctx, ns.Name, 1)
 				ExpectRunnerSetsManagedReplicasCountEventuallyEquals(ctx, ns.Name, 3)
-			}
-
-			{
 				env.ExpectRegisteredNumberCountEventuallyEquals(3, "count of fake list runners")
+				env.SyncRunnerRegistrations()
+				ExpectRunnerCountEventuallyEquals(ctx, ns.Name, 3)
 			}
 
 			// Scale-up to 4 replicas on first check_run create webhook event
@@ -486,19 +499,19 @@ var _ = Context("INTEGRATION: Inside of a new namespace", func() {
 				env.SendOrgCheckRunEvent("test", "valid", "pending", "created")
 				ExpectRunnerSetsCountEventuallyEquals(ctx, ns.Name, 1, "runner sets after webhook")
 				ExpectRunnerSetsManagedReplicasCountEventuallyEquals(ctx, ns.Name, 4, "runners after first webhook event")
-			}
-
-			{
 				env.ExpectRegisteredNumberCountEventuallyEquals(4, "count of fake list runners")
+				env.SyncRunnerRegistrations()
+				ExpectRunnerCountEventuallyEquals(ctx, ns.Name, 4)
 			}
 
 			// Scale-up to 5 replicas on second check_run create webhook event
 			{
 				env.SendOrgCheckRunEvent("test", "valid", "pending", "created")
 				ExpectRunnerSetsManagedReplicasCountEventuallyEquals(ctx, ns.Name, 5, "runners after second webhook event")
+				env.ExpectRegisteredNumberCountEventuallyEquals(5, "count of fake list runners")
+				env.SyncRunnerRegistrations()
+				ExpectRunnerCountEventuallyEquals(ctx, ns.Name, 5)
 			}
-
-			env.ExpectRegisteredNumberCountEventuallyEquals(5, "count of fake list runners")
 		})
 
 		It("should create and scale organization's repository runners only on check_run event", func() {
@@ -538,9 +551,6 @@ var _ = Context("INTEGRATION: Inside of a new namespace", func() {
 				ExpectCreate(ctx, rd, "test RunnerDeployment")
 				ExpectRunnerSetsCountEventuallyEquals(ctx, ns.Name, 1)
 				ExpectRunnerSetsManagedReplicasCountEventuallyEquals(ctx, ns.Name, 1)
-			}
-
-			{
 				env.ExpectRegisteredNumberCountEventuallyEquals(1, "count of fake list runners")
 			}
 
@@ -579,9 +589,6 @@ var _ = Context("INTEGRATION: Inside of a new namespace", func() {
 
 				ExpectRunnerSetsCountEventuallyEquals(ctx, ns.Name, 1)
 				ExpectRunnerSetsManagedReplicasCountEventuallyEquals(ctx, ns.Name, 1)
-			}
-
-			{
 				env.ExpectRegisteredNumberCountEventuallyEquals(1, "count of fake list runners")
 			}
 
@@ -590,9 +597,6 @@ var _ = Context("INTEGRATION: Inside of a new namespace", func() {
 				env.SendOrgCheckRunEvent("test", "valid", "pending", "created")
 				ExpectRunnerSetsCountEventuallyEquals(ctx, ns.Name, 1, "runner sets after webhook")
 				ExpectRunnerSetsManagedReplicasCountEventuallyEquals(ctx, ns.Name, 2, "runners after first webhook event")
-			}
-
-			{
 				env.ExpectRegisteredNumberCountEventuallyEquals(2, "count of fake list runners")
 			}
 
@@ -600,9 +604,8 @@ var _ = Context("INTEGRATION: Inside of a new namespace", func() {
 			{
 				env.SendOrgCheckRunEvent("test", "valid", "pending", "created")
 				ExpectRunnerSetsManagedReplicasCountEventuallyEquals(ctx, ns.Name, 3, "runners after second webhook event")
+				env.ExpectRegisteredNumberCountEventuallyEquals(3, "count of fake list runners")
 			}
-
-			env.ExpectRegisteredNumberCountEventuallyEquals(3, "count of fake list runners")
 		})
 
 		It("should create and scale user's repository runners on pull_request event", func() {
@@ -868,9 +871,6 @@ var _ = Context("INTEGRATION: Inside of a new namespace", func() {
 				ExpectCreate(ctx, rd, "test RunnerDeployment")
 				ExpectRunnerSetsCountEventuallyEquals(ctx, ns.Name, 1)
 				ExpectRunnerSetsManagedReplicasCountEventuallyEquals(ctx, ns.Name, 1)
-			}
-
-			{
 				env.ExpectRegisteredNumberCountEventuallyEquals(1, "count of fake list runners")
 			}
 
@@ -914,9 +914,6 @@ var _ = Context("INTEGRATION: Inside of a new namespace", func() {
 
 				ExpectRunnerSetsCountEventuallyEquals(ctx, ns.Name, 1)
 				ExpectRunnerSetsManagedReplicasCountEventuallyEquals(ctx, ns.Name, 3)
-			}
-
-			{
 				env.ExpectRegisteredNumberCountEventuallyEquals(3, "count of fake list runners")
 			}
 
@@ -925,9 +922,6 @@ var _ = Context("INTEGRATION: Inside of a new namespace", func() {
 				env.SendUserCheckRunEvent("test", "valid", "pending", "created")
 				ExpectRunnerSetsCountEventuallyEquals(ctx, ns.Name, 1, "runner sets after webhook")
 				ExpectRunnerSetsManagedReplicasCountEventuallyEquals(ctx, ns.Name, 4, "runners after first webhook event")
-			}
-
-			{
 				env.ExpectRegisteredNumberCountEventuallyEquals(4, "count of fake list runners")
 			}
 
@@ -935,9 +929,8 @@ var _ = Context("INTEGRATION: Inside of a new namespace", func() {
 			{
 				env.SendUserCheckRunEvent("test", "valid", "pending", "created")
 				ExpectRunnerSetsManagedReplicasCountEventuallyEquals(ctx, ns.Name, 5, "runners after second webhook event")
+				env.ExpectRegisteredNumberCountEventuallyEquals(5, "count of fake list runners")
 			}
-
-			env.ExpectRegisteredNumberCountEventuallyEquals(5, "count of fake list runners")
 		})
 
 		It("should create and scale user's repository runners only on check_run event", func() {
@@ -1029,9 +1022,6 @@ var _ = Context("INTEGRATION: Inside of a new namespace", func() {
 				env.SendUserCheckRunEvent("test", "valid", "pending", "created")
 				ExpectRunnerSetsCountEventuallyEquals(ctx, ns.Name, 1, "runner sets after webhook")
 				ExpectRunnerSetsManagedReplicasCountEventuallyEquals(ctx, ns.Name, 2, "runners after first webhook event")
-			}
-
-			{
 				env.ExpectRegisteredNumberCountEventuallyEquals(2, "count of fake list runners")
 			}
 
@@ -1039,9 +1029,8 @@ var _ = Context("INTEGRATION: Inside of a new namespace", func() {
 			{
 				env.SendUserCheckRunEvent("test", "valid", "pending", "created")
 				ExpectRunnerSetsManagedReplicasCountEventuallyEquals(ctx, ns.Name, 3, "runners after second webhook event")
+				env.ExpectRegisteredNumberCountEventuallyEquals(3, "count of fake list runners")
 			}
-
-			env.ExpectRegisteredNumberCountEventuallyEquals(3, "count of fake list runners")
 		})
 
 	})
@@ -1228,6 +1217,44 @@ func ExpectRunnerSetsCountEventuallyEquals(ctx context.Context, ns string, count
 		time.Second*10, time.Millisecond*500).Should(BeEquivalentTo(count), optionalDescription...)
 }
 
+func ExpectRunnerCountEventuallyEquals(ctx context.Context, ns string, count int, optionalDescription ...interface{}) {
+	runners := actionsv1alpha1.RunnerList{Items: []actionsv1alpha1.Runner{}}
+
+	EventuallyWithOffset(
+		1,
+		func() int {
+			err := k8sClient.List(ctx, &runners, client.InNamespace(ns))
+			if err != nil {
+				logf.Log.Error(err, "list runner sets")
+			}
+
+			var running int
+
+			for _, r := range runners.Items {
+				if r.Status.Phase == string(corev1.PodRunning) {
+					running++
+				} else {
+					var pod corev1.Pod
+					if err := k8sClient.Get(ctx, types.NamespacedName{Namespace: ns, Name: r.Name}, &pod); err != nil {
+						logf.Log.Error(err, "simulating pod controller")
+						continue
+					}
+
+					copy := pod.DeepCopy()
+					copy.Status.Phase = corev1.PodRunning
+
+					if err := k8sClient.Status().Patch(ctx, copy, client.MergeFrom(&pod)); err != nil {
+						logf.Log.Error(err, "simulating pod controller")
+						continue
+					}
+				}
+			}
+
+			return running
+		},
+		time.Second*10, time.Millisecond*500).Should(BeEquivalentTo(count), optionalDescription...)
+}
+
 func ExpectRunnerSetsManagedReplicasCountEventuallyEquals(ctx context.Context, ns string, count int, optionalDescription ...interface{}) {
 	runnerSets := actionsv1alpha1.RunnerReplicaSetList{Items: []actionsv1alpha1.RunnerReplicaSet{}}
 

controllers/metrics/horizontalrunnerautoscaler.go

@@ -0,0 +1,67 @@
+package metrics
+
+import (
+	"github.com/prometheus/client_golang/prometheus"
+	"github.com/summerwind/actions-runner-controller/api/v1alpha1"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+)
+
+const (
+	hraName      = "horizontalrunnerautoscaler"
+	hraNamespace = "namespace"
+)
+
+var (
+	horizontalRunnerAutoscalerMetrics = []prometheus.Collector{
+		horizontalRunnerAutoscalerMinReplicas,
+		horizontalRunnerAutoscalerMaxReplicas,
+		horizontalRunnerAutoscalerDesiredReplicas,
+	}
+)
+
+var (
+	horizontalRunnerAutoscalerMinReplicas = prometheus.NewGaugeVec(
+		prometheus.GaugeOpts{
+			Name: "horizontalrunnerautoscaler_spec_min_replicas",
+			Help: "minReplicas of HorizontalRunnerAutoscaler",
+		},
+		[]string{hraName, hraNamespace},
+	)
+	horizontalRunnerAutoscalerMaxReplicas = prometheus.NewGaugeVec(
+		prometheus.GaugeOpts{
+			Name: "horizontalrunnerautoscaler_spec_max_replicas",
+			Help: "maxReplicas of HorizontalRunnerAutoscaler",
+		},
+		[]string{hraName, hraNamespace},
+	)
+	horizontalRunnerAutoscalerDesiredReplicas = prometheus.NewGaugeVec(
+		prometheus.GaugeOpts{
+			Name: "horizontalrunnerautoscaler_status_desired_replicas",
+			Help: "desiredReplicas of HorizontalRunnerAutoscaler",
+		},
+		[]string{hraName, hraNamespace},
+	)
+)
+
+func SetHorizontalRunnerAutoscalerSpec(o metav1.ObjectMeta, spec v1alpha1.HorizontalRunnerAutoscalerSpec) {
+	labels := prometheus.Labels{
+		hraName:      o.Name,
+		hraNamespace: o.Namespace,
+	}
+	if spec.MaxReplicas != nil {
+		horizontalRunnerAutoscalerMaxReplicas.With(labels).Set(float64(*spec.MaxReplicas))
+	}
+	if spec.MinReplicas != nil {
+		horizontalRunnerAutoscalerMinReplicas.With(labels).Set(float64(*spec.MinReplicas))
+	}
+}
+
+func SetHorizontalRunnerAutoscalerStatus(o metav1.ObjectMeta, status v1alpha1.HorizontalRunnerAutoscalerStatus) {
+	labels := prometheus.Labels{
+		hraName:      o.Name,
+		hraNamespace: o.Namespace,
+	}
+	if status.DesiredReplicas != nil {
+		horizontalRunnerAutoscalerDesiredReplicas.With(labels).Set(float64(*status.DesiredReplicas))
+	}
+}

controllers/metrics/metrics.go

@@ -0,0 +1,14 @@
+// Package metrics provides the metrics of custom resources such as HRA.
+//
+// This depends on the metrics exporter of kubebuilder.
+// See https://book.kubebuilder.io/reference/metrics.html for details.
+package metrics
+
+import (
+	"sigs.k8s.io/controller-runtime/pkg/metrics"
+)
+
+func init() {
+	metrics.Registry.MustRegister(runnerDeploymentMetrics...)
+	metrics.Registry.MustRegister(horizontalRunnerAutoscalerMetrics...)
+}

controllers/metrics/runnerdeployment.go

@@ -0,0 +1,37 @@
+package metrics
+
+import (
+	"github.com/prometheus/client_golang/prometheus"
+	"github.com/summerwind/actions-runner-controller/api/v1alpha1"
+)
+
+const (
+	rdName      = "runnerdeployment"
+	rdNamespace = "namespace"
+)
+
+var (
+	runnerDeploymentMetrics = []prometheus.Collector{
+		runnerDeploymentReplicas,
+	}
+)
+
+var (
+	runnerDeploymentReplicas = prometheus.NewGaugeVec(
+		prometheus.GaugeOpts{
+			Name: "runnerdeployment_spec_replicas",
+			Help: "replicas of RunnerDeployment",
+		},
+		[]string{rdName, rdNamespace},
+	)
+)
+
+func SetRunnerDeployment(rd v1alpha1.RunnerDeployment) {
+	labels := prometheus.Labels{
+		rdName:      rd.Name,
+		rdNamespace: rd.Namespace,
+	}
+	if rd.Spec.Replicas != nil {
+		runnerDeploymentReplicas.With(labels).Set(float64(*rd.Spec.Replicas))
+	}
+}

controllers/runner_controller.go

@@ -20,11 +20,12 @@ import (
 	"context"
 	"errors"
 	"fmt"
+	"strings"
+	"time"
+
 	gogithub "github.com/google/go-github/v33/github"
 	"github.com/summerwind/actions-runner-controller/hash"
 	"k8s.io/apimachinery/pkg/util/wait"
-	"strings"
-	"time"
 
 	"github.com/go-logr/logr"
 	kerrors "k8s.io/apimachinery/pkg/api/errors"
@@ -47,17 +48,23 @@ const (
 	LabelKeyPodTemplateHash = "pod-template-hash"
 
 	retryDelayOnGitHubAPIRateLimitError = 30 * time.Second
+
+	// This is an annotation internal to actions-runner-controller and can change in backward-incompatible ways
+	annotationKeyRegistrationOnly = "actions-runner-controller/registration-only"
 )
 
 // RunnerReconciler reconciles a Runner object
 type RunnerReconciler struct {
 	client.Client
-	Log          logr.Logger
-	Recorder     record.EventRecorder
-	Scheme       *runtime.Scheme
-	GitHubClient *github.Client
-	RunnerImage  string
-	DockerImage  string
+	Log                         logr.Logger
+	Recorder                    record.EventRecorder
+	Scheme                      *runtime.Scheme
+	GitHubClient                *github.Client
+	RunnerImage                 string
+	DockerImage                 string
+	Name                        string
+	RegistrationRecheckInterval time.Duration
+	RegistrationRecheckJitter   time.Duration
 }
 
 // +kubebuilder:rbac:groups=actions.summerwind.dev,resources=runners,verbs=get;list;watch;create;update;patch;delete
@@ -141,6 +148,34 @@ func (r *RunnerReconciler) Reconcile(req ctrl.Request) (ctrl.Result, error) {
 		return ctrl.Result{}, nil
 	}
 
+	registrationOnly := metav1.HasAnnotation(runner.ObjectMeta, annotationKeyRegistrationOnly)
+	if registrationOnly && runner.Status.Phase != "" {
+		// At this point we are sure that the registration-only runner has successfully configured and
+		// is of `offline` status, because we set runner.Status.Phase to that of the runner pod only after
+		// successful registration.
+
+		var pod corev1.Pod
+		if err := r.Get(ctx, req.NamespacedName, &pod); err != nil {
+			if !kerrors.IsNotFound(err) {
+				log.Info(fmt.Sprintf("Retrying soon as we failed to get registration-only runner pod: %v", err))
+
+				return ctrl.Result{Requeue: true}, nil
+			}
+		} else if err := r.Delete(ctx, &pod); err != nil {
+			if !kerrors.IsNotFound(err) {
+				log.Info(fmt.Sprintf("Retrying soon as we failed to delete registration-only runner pod: %v", err))
+
+				return ctrl.Result{Requeue: true}, nil
+			}
+		}
+
+		log.Info("Successfully deleted egistration-only runner pod to free node and cluster resource")
+
+		// Return here to not recreate the deleted pod, because recreating it is the waste of cluster and node resource,
+		// and also defeats the original purpose of scale-from/to-zero we're trying to implement by using the registration-only runner.
+		return ctrl.Result{}, nil
+	}
+
 	var pod corev1.Pod
 	if err := r.Get(ctx, req.NamespacedName, &pod); err != nil {
 		if !kerrors.IsNotFound(err) {
@@ -164,9 +199,11 @@ func (r *RunnerReconciler) Reconcile(req ctrl.Request) (ctrl.Result, error) {
 				// Gracefully handle pod-already-exists errors due to informer cache delay.
 				// Without this we got a few errors like the below on new runner pod:
 				// 2021-03-16T00:23:10.116Z        ERROR   controller-runtime.controller   Reconciler error      {"controller": "runner-controller", "request": "default/example-runnerdeploy-b2g2g-j4mcp", "error": "pods \"example-runnerdeploy-b2g2g-j4mcp\" already exists"}
-				log.Info("Runner pod already exists. Probably this pod has been already created in previous reconcilation but the new pod is not yet cached.")
+				log.Info(
+					"Failed to create pod due to AlreadyExists error. Probably this pod has been already created in previous reconcilation but is still not in the informer cache. Will retry on pod created. If it doesn't repeat, there's no problem",
+				)
 
-				return ctrl.Result{RequeueAfter: 10 * time.Second}, nil
+				return ctrl.Result{}, nil
 			}
 
 			log.Error(err, "Failed to create pod resource")
@@ -184,7 +221,7 @@ func (r *RunnerReconciler) Reconcile(req ctrl.Request) (ctrl.Result, error) {
 
 			if deletionDidTimeout {
 				log.Info(
-					"Pod failed to delete itself in a timely manner. "+
+					fmt.Sprintf("Failed to delete pod within %s. ", deletionTimeout)+
 						"This is typically the case when a Kubernetes node became unreachable "+
 						"and the kube controller started evicting nodes. Forcefully deleting the pod to not get stuck.",
 					"podDeletionTimestamp", pod.DeletionTimestamp,
@@ -215,20 +252,33 @@ func (r *RunnerReconciler) Reconcile(req ctrl.Request) (ctrl.Result, error) {
 
 		// If pod has ended up succeeded we need to restart it
 		// Happens e.g. when dind is in runner and run completes
-		restart := pod.Status.Phase == corev1.PodSucceeded
+		stopped := pod.Status.Phase == corev1.PodSucceeded
 
-		if pod.Status.Phase == corev1.PodRunning {
-			for _, status := range pod.Status.ContainerStatuses {
-				if status.Name != containerName {
-					continue
-				}
+		if !stopped {
+			if pod.Status.Phase == corev1.PodRunning {
+				for _, status := range pod.Status.ContainerStatuses {
+					if status.Name != containerName {
+						continue
+					}
 
-				if status.State.Terminated != nil && status.State.Terminated.ExitCode == 0 {
-					restart = true
+					if status.State.Terminated != nil && status.State.Terminated.ExitCode == 0 {
+						stopped = true
+					}
 				}
 			}
 		}
 
+		restart := stopped
+
+		if registrationOnly && stopped {
+			restart = false
+
+			log.Info(
+				"Observed that registration-only runner for scaling-from-zero has successfully stopped. " +
+					"Unlike other pods, this one will be recreated only when runner spec changes.",
+			)
+		}
+
 		if updated, err := r.updateRegistrationToken(ctx, runner); err != nil {
 			return ctrl.Result{}, err
 		} else if updated {
@@ -241,13 +291,26 @@ func (r *RunnerReconciler) Reconcile(req ctrl.Request) (ctrl.Result, error) {
 			return ctrl.Result{}, err
 		}
 
+		if registrationOnly {
+			newPod.Spec.Containers[0].Env = append(
+				newPod.Spec.Containers[0].Env,
+				corev1.EnvVar{
+					Name:  "RUNNER_REGISTRATION_ONLY",
+					Value: "true",
+				},
+			)
+		}
+
 		var registrationRecheckDelay time.Duration
 
 		// all checks done below only decide whether a restart is needed
 		// if a restart was already decided before, there is no need for the checks
-		// saving API calls and scary{ log messages
+		// saving API calls and scary log messages
 		if !restart {
 			registrationCheckInterval := time.Minute
+			if r.RegistrationRecheckInterval > 0 {
+				registrationCheckInterval = r.RegistrationRecheckInterval
+			}
 
 			// We want to call ListRunners GitHub Actions API only once per runner per minute.
 			// This if block, in conjunction with:
@@ -255,15 +318,28 @@ func (r *RunnerReconciler) Reconcile(req ctrl.Request) (ctrl.Result, error) {
 			// achieves that.
 			if lastCheckTime := runner.Status.LastRegistrationCheckTime; lastCheckTime != nil {
 				nextCheckTime := lastCheckTime.Add(registrationCheckInterval)
-				if nextCheckTime.After(time.Now()) {
+				now := time.Now()
+
+				// Requeue scheduled by RequeueAfter can happen a bit earlier (like dozens of milliseconds)
+				// so to avoid excessive, in-effective retry, we heuristically ignore the remaining delay in case it is
+				// shorter than 1s
+				requeueAfter := nextCheckTime.Sub(now) - time.Second
+				if requeueAfter > 0 {
 					log.Info(
-						fmt.Sprintf("Skipping registration check because it's deferred until %s", nextCheckTime),
+						fmt.Sprintf("Skipped registration check because it's deferred until %s. Retrying in %s at latest", nextCheckTime, requeueAfter),
+						"lastRegistrationCheckTime", lastCheckTime,
+						"registrationCheckInterval", registrationCheckInterval,
 					)
 
-					// Note that we don't need to explicitly requeue on this reconcilation because
-					// the requeue should have been already scheduled previsouly
-					// (with `return ctrl.Result{RequeueAfter: registrationRecheckDelay}, nil` as noted above and coded below)
-					return ctrl.Result{}, nil
+					// Without RequeueAfter, the controller may not retry on scheduled. Instead, it must wait until the
+					// next sync period passes, which can be too much later than nextCheckTime.
+					//
+					// We need to requeue on this reconcilation even though we have already scheduled the initial
+					// requeue previously with `return ctrl.Result{RequeueAfter: registrationRecheckDelay}, nil`.
+					// Apparently, the workqueue used by controller-runtime seems to deduplicate and resets the delay on
+					// other requeues- so the initial scheduled requeue may have been reset due to requeue on
+					// spec/status change.
+					return ctrl.Result{RequeueAfter: requeueAfter}, nil
 				}
 			}
 
@@ -334,7 +410,14 @@ func (r *RunnerReconciler) Reconcile(req ctrl.Request) (ctrl.Result, error) {
 					)
 				}
 			} else if offline {
-				if registrationDidTimeout {
+				if registrationOnly {
+					log.Info(
+						"Observed that registration-only runner for scaling-from-zero has successfully been registered.",
+						"podCreationTimestamp", pod.CreationTimestamp,
+						"currentTime", currentTime,
+						"configuredRegistrationTimeout", registrationTimeout,
+					)
+				} else if registrationDidTimeout {
 					log.Info(
 						"Already existing GitHub runner still appears offline . "+
 							"Recreating the pod to see if it resolves the issue. "+
@@ -353,8 +436,13 @@ func (r *RunnerReconciler) Reconcile(req ctrl.Request) (ctrl.Result, error) {
 				}
 			}
 
-			if (notFound || offline) && !registrationDidTimeout {
-				registrationRecheckDelay = registrationCheckInterval + wait.Jitter(10*time.Second, 0.1)
+			if (notFound || (offline && !registrationOnly)) && !registrationDidTimeout {
+				registrationRecheckJitter := 10 * time.Second
+				if r.RegistrationRecheckJitter > 0 {
+					registrationRecheckJitter = r.RegistrationRecheckJitter
+				}
+
+				registrationRecheckDelay = registrationCheckInterval + wait.Jitter(registrationRecheckJitter, 0.1)
 			}
 		}
 
@@ -476,9 +564,11 @@ func (r *RunnerReconciler) updateRegistrationToken(ctx context.Context, runner v
 
 func (r *RunnerReconciler) newPod(runner v1alpha1.Runner) (corev1.Pod, error) {
 	var (
-		privileged      bool = true
-		dockerdInRunner bool = runner.Spec.DockerdWithinRunnerContainer != nil && *runner.Spec.DockerdWithinRunnerContainer
-		dockerEnabled   bool = runner.Spec.DockerEnabled == nil || *runner.Spec.DockerEnabled
+		privileged                bool = true
+		dockerdInRunner           bool = runner.Spec.DockerdWithinRunnerContainer != nil && *runner.Spec.DockerdWithinRunnerContainer
+		dockerEnabled             bool = runner.Spec.DockerEnabled == nil || *runner.Spec.DockerEnabled
+		ephemeral                 bool = runner.Spec.Ephemeral == nil || *runner.Spec.Ephemeral
+		dockerdInRunnerPrivileged bool = dockerdInRunner
 	)
 
 	runnerImage := runner.Spec.Image
@@ -537,6 +627,18 @@ func (r *RunnerReconciler) newPod(runner v1alpha1.Runner) (corev1.Pod, error) {
 			Name:  "RUNNER_WORKDIR",
 			Value: workDir,
 		},
+		{
+			Name:  "RUNNER_EPHEMERAL",
+			Value: fmt.Sprintf("%v", ephemeral),
+		},
+	}
+
+	if metav1.HasAnnotation(runner.ObjectMeta, annotationKeyRegistrationOnly) {
+		env = append(env, corev1.EnvVar{
+			Name:  "RUNNER_REGISTRATION_ONLY",
+			Value: "true",
+		},
+		)
 	}
 
 	env = append(env, runner.Spec.Env...)
@@ -573,6 +675,15 @@ func (r *RunnerReconciler) newPod(runner v1alpha1.Runner) (corev1.Pod, error) {
 		r.GitHubClient.GithubBaseURL,
 	)
 
+	var seLinuxOptions *corev1.SELinuxOptions
+	if runner.Spec.SecurityContext != nil {
+		seLinuxOptions = runner.Spec.SecurityContext.SELinuxOptions
+		if seLinuxOptions != nil {
+			privileged = false
+			dockerdInRunnerPrivileged = false
+		}
+	}
+
 	pod := corev1.Pod{
 		ObjectMeta: metav1.ObjectMeta{
 			Name:        runner.Name,
@@ -591,7 +702,7 @@ func (r *RunnerReconciler) newPod(runner v1alpha1.Runner) (corev1.Pod, error) {
 					EnvFrom:         runner.Spec.EnvFrom,
 					SecurityContext: &corev1.SecurityContext{
 						// Runner need to run privileged if it contains DinD
-						Privileged: runner.Spec.DockerdWithinRunnerContainer,
+						Privileged: &dockerdInRunnerPrivileged,
 					},
 					Resources: runner.Spec.Resources,
 				},
@@ -608,45 +719,72 @@ func (r *RunnerReconciler) newPod(runner v1alpha1.Runner) (corev1.Pod, error) {
 		}...)
 	}
 
-	if !dockerdInRunner && dockerEnabled {
-		runnerVolumeName := "runner"
-		runnerVolumeMountPath := "/runner"
-
-		pod.Spec.Volumes = []corev1.Volume{
+	if mirror := runner.Spec.DockerRegistryMirror; mirror != nil && dockerdInRunner {
+		pod.Spec.Containers[0].Env = append(pod.Spec.Containers[0].Env, []corev1.EnvVar{
 			{
+				Name:  "DOCKER_REGISTRY_MIRROR",
+				Value: *runner.Spec.DockerRegistryMirror,
+			},
+		}...)
+	}
+
+	//
+	// /runner must be generated on runtime from /runnertmp embedded in the container image.
+	//
+	// When you're NOT using dindWithinRunner=true,
+	// it must also be shared with the dind container as it seems like required to run docker steps.
+	//
+
+	runnerVolumeName := "runner"
+	runnerVolumeMountPath := "/runner"
+	runnerVolumeEmptyDir := &corev1.EmptyDirVolumeSource{}
+
+	if runner.Spec.VolumeSizeLimit != nil {
+		runnerVolumeEmptyDir.SizeLimit = runner.Spec.VolumeSizeLimit
+	}
+
+	pod.Spec.Volumes = append(pod.Spec.Volumes,
+		corev1.Volume{
+			Name: runnerVolumeName,
+			VolumeSource: corev1.VolumeSource{
+				EmptyDir: runnerVolumeEmptyDir,
+			},
+		},
+	)
+
+	pod.Spec.Containers[0].VolumeMounts = append(pod.Spec.Containers[0].VolumeMounts,
+		corev1.VolumeMount{
+			Name:      runnerVolumeName,
+			MountPath: runnerVolumeMountPath,
+		},
+	)
+
+	if !dockerdInRunner && dockerEnabled {
+		pod.Spec.Volumes = append(pod.Spec.Volumes,
+			corev1.Volume{
 				Name: "work",
 				VolumeSource: corev1.VolumeSource{
 					EmptyDir: &corev1.EmptyDirVolumeSource{},
 				},
 			},
-			{
-				Name: runnerVolumeName,
-				VolumeSource: corev1.VolumeSource{
-					EmptyDir: &corev1.EmptyDirVolumeSource{},
-				},
-			},
-			{
+			corev1.Volume{
 				Name: "certs-client",
 				VolumeSource: corev1.VolumeSource{
 					EmptyDir: &corev1.EmptyDirVolumeSource{},
 				},
 			},
-		}
-		pod.Spec.Containers[0].VolumeMounts = []corev1.VolumeMount{
-			{
+		)
+		pod.Spec.Containers[0].VolumeMounts = append(pod.Spec.Containers[0].VolumeMounts,
+			corev1.VolumeMount{
 				Name:      "work",
 				MountPath: workDir,
 			},
-			{
-				Name:      runnerVolumeName,
-				MountPath: runnerVolumeMountPath,
-			},
-			{
+			corev1.VolumeMount{
 				Name:      "certs-client",
 				MountPath: "/certs/client",
 				ReadOnly:  true,
 			},
-		}
+		)
 		pod.Spec.Containers[0].Env = append(pod.Spec.Containers[0].Env, []corev1.EnvVar{
 			{
 				Name:  "DOCKER_HOST",
@@ -661,23 +799,31 @@ func (r *RunnerReconciler) newPod(runner v1alpha1.Runner) (corev1.Pod, error) {
 				Value: "/certs/client",
 			},
 		}...)
-		pod.Spec.Containers = append(pod.Spec.Containers, corev1.Container{
-			Name:  "docker",
-			Image: r.DockerImage,
-			VolumeMounts: []corev1.VolumeMount{
-				{
-					Name:      "work",
-					MountPath: workDir,
-				},
-				{
-					Name:      runnerVolumeName,
-					MountPath: runnerVolumeMountPath,
-				},
-				{
-					Name:      "certs-client",
-					MountPath: "/certs/client",
-				},
+
+		// Determine the volume mounts assigned to the docker sidecar. In case extra mounts are included in the RunnerSpec, append them to the standard
+		// set of mounts. See https://github.com/summerwind/actions-runner-controller/issues/435 for context.
+		dockerVolumeMounts := []corev1.VolumeMount{
+			{
+				Name:      "work",
+				MountPath: workDir,
 			},
+			{
+				Name:      runnerVolumeName,
+				MountPath: runnerVolumeMountPath,
+			},
+			{
+				Name:      "certs-client",
+				MountPath: "/certs/client",
+			},
+		}
+		if extraDockerVolumeMounts := runner.Spec.DockerVolumeMounts; extraDockerVolumeMounts != nil {
+			dockerVolumeMounts = append(dockerVolumeMounts, extraDockerVolumeMounts...)
+		}
+
+		pod.Spec.Containers = append(pod.Spec.Containers, corev1.Container{
+			Name:         "docker",
+			Image:        r.DockerImage,
+			VolumeMounts: dockerVolumeMounts,
 			Env: []corev1.EnvVar{
 				{
 					Name:  "DOCKER_TLS_CERTDIR",
@@ -685,20 +831,32 @@ func (r *RunnerReconciler) newPod(runner v1alpha1.Runner) (corev1.Pod, error) {
 				},
 			},
 			SecurityContext: &corev1.SecurityContext{
-				Privileged: &privileged,
+				Privileged:     &privileged,
+				SELinuxOptions: seLinuxOptions,
 			},
 			Resources: runner.Spec.DockerdContainerResources,
 		})
 
 		if mtu := runner.Spec.DockerMTU; mtu != nil {
 			pod.Spec.Containers[1].Env = append(pod.Spec.Containers[1].Env, []corev1.EnvVar{
+				// See https://docs.docker.com/engine/security/rootless/
 				{
 					Name:  "DOCKERD_ROOTLESS_ROOTLESSKIT_MTU",
 					Value: fmt.Sprintf("%d", *runner.Spec.DockerMTU),
 				},
 			}...)
+
+			pod.Spec.Containers[1].Args = append(pod.Spec.Containers[1].Args,
+				"--mtu",
+				fmt.Sprintf("%d", *runner.Spec.DockerMTU),
+			)
 		}
 
+		if mirror := runner.Spec.DockerRegistryMirror; mirror != nil {
+			pod.Spec.Containers[1].Args = append(pod.Spec.Containers[1].Args,
+				fmt.Sprintf("--registry-mirror=%s", *runner.Spec.DockerRegistryMirror),
+			)
+		}
 	}
 
 	if len(runner.Spec.Containers) != 0 {
@@ -759,6 +917,14 @@ func (r *RunnerReconciler) newPod(runner v1alpha1.Runner) (corev1.Pod, error) {
 		pod.Spec.TerminationGracePeriodSeconds = runner.Spec.TerminationGracePeriodSeconds
 	}
 
+	if len(runner.Spec.HostAliases) != 0 {
+		pod.Spec.HostAliases = runner.Spec.HostAliases
+	}
+
+	if runner.Spec.RuntimeClassName != nil {
+		pod.Spec.RuntimeClassName = runner.Spec.RuntimeClassName
+	}
+
 	if err := ctrl.SetControllerReference(&runner, &pod, r.Scheme); err != nil {
 		return pod, err
 	}
@@ -768,6 +934,9 @@ func (r *RunnerReconciler) newPod(runner v1alpha1.Runner) (corev1.Pod, error) {
 
 func (r *RunnerReconciler) SetupWithManager(mgr ctrl.Manager) error {
 	name := "runner-controller"
+	if r.Name != "" {
+		name = r.Name
+	}
 
 	r.Recorder = mgr.GetEventRecorderFor(name)
 

controllers/runnerdeployment_controller.go

@@ -38,6 +38,7 @@ import (
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 
 	"github.com/summerwind/actions-runner-controller/api/v1alpha1"
+	"github.com/summerwind/actions-runner-controller/controllers/metrics"
 )
 
 const (
@@ -77,6 +78,8 @@ func (r *RunnerDeploymentReconciler) Reconcile(req ctrl.Request) (ctrl.Result, e
 		return ctrl.Result{}, nil
 	}
 
+	metrics.SetRunnerDeployment(rd)
+
 	var myRunnerReplicaSetList v1alpha1.RunnerReplicaSetList
 	if err := r.List(ctx, &myRunnerReplicaSetList, client.InNamespace(req.Namespace), client.MatchingFields{runnerSetOwnerKey: req.Name}); err != nil {
 		return ctrl.Result{}, err
@@ -185,21 +188,35 @@ func (r *RunnerDeploymentReconciler) Reconcile(req ctrl.Request) (ctrl.Result, e
 		return ctrl.Result{}, err
 	}
 
-	// Do we old runner replica sets that should eventually deleted?
+	// Do we have old runner replica sets that should eventually deleted?
 	if len(oldSets) > 0 {
-		readyReplicas := newestSet.Status.ReadyReplicas
+		var readyReplicas int
+		if newestSet.Status.ReadyReplicas != nil {
+			readyReplicas = *newestSet.Status.ReadyReplicas
+		}
 
-		if readyReplicas < currentDesiredReplicas {
-			log.WithValues("runnerreplicaset", types.NamespacedName{
+		oldSetsCount := len(oldSets)
+
+		logWithDebugInfo := log.WithValues(
+			"newest_runnerreplicaset", types.NamespacedName{
 				Namespace: newestSet.Namespace,
 				Name:      newestSet.Name,
-			}).
-				Info("Waiting until the newest runner replica set to be 100% available",
-					"ready", readyReplicas,
-					"desired", currentDesiredReplicas,
-				)
+			},
+			"newest_runnerreplicaset_replicas_ready", readyReplicas,
+			"newest_runnerreplicaset_replicas_desired", currentDesiredReplicas,
+			"old_runnerreplicasets_count", oldSetsCount,
+		)
 
-			return ctrl.Result{RequeueAfter: 10 * time.Second}, nil
+		if readyReplicas < currentDesiredReplicas {
+			logWithDebugInfo.
+				Info("Waiting until the newest runnerreplicaset to be 100% available")
+
+			return ctrl.Result{}, nil
+		}
+
+		if oldSetsCount > 0 {
+			logWithDebugInfo.
+				Info("The newest runnerreplicaset is 100% available. Deleting old runnerreplicasets")
 		}
 
 		for i := range oldSets {
@@ -217,14 +234,49 @@ func (r *RunnerDeploymentReconciler) Reconcile(req ctrl.Request) (ctrl.Result, e
 		}
 	}
 
-	if rd.Spec.Replicas == nil && desiredRS.Spec.Replicas != nil {
+	var replicaSets []v1alpha1.RunnerReplicaSet
+
+	replicaSets = append(replicaSets, *newestSet)
+	replicaSets = append(replicaSets, oldSets...)
+
+	var totalCurrentReplicas, totalStatusAvailableReplicas, updatedReplicas int
+
+	for _, rs := range replicaSets {
+		var current, available int
+
+		if rs.Status.Replicas != nil {
+			current = *rs.Status.Replicas
+		}
+
+		if rs.Status.AvailableReplicas != nil {
+			available = *rs.Status.AvailableReplicas
+		}
+
+		totalCurrentReplicas += current
+		totalStatusAvailableReplicas += available
+	}
+
+	if newestSet.Status.Replicas != nil {
+		updatedReplicas = *newestSet.Status.Replicas
+	}
+
+	var status v1alpha1.RunnerDeploymentStatus
+
+	status.AvailableReplicas = &totalStatusAvailableReplicas
+	status.ReadyReplicas = &totalStatusAvailableReplicas
+	status.DesiredReplicas = &newDesiredReplicas
+	status.Replicas = &totalCurrentReplicas
+	status.UpdatedReplicas = &updatedReplicas
+
+	if !reflect.DeepEqual(rd.Status, status) {
 		updated := rd.DeepCopy()
-		updated.Status.Replicas = desiredRS.Spec.Replicas
+		updated.Status = status
 
-		if err := r.Status().Update(ctx, updated); err != nil {
-			log.Error(err, "Failed to update runnerdeployment status")
-
-			return ctrl.Result{}, err
+		if err := r.Status().Patch(ctx, updated, client.MergeFrom(&rd)); err != nil {
+			log.Info("Failed to patch runnerdeployment status. Retrying immediately", "error", err.Error())
+			return ctrl.Result{
+				Requeue: true,
+			}, nil
 		}
 	}
 

controllers/runnerdeployment_controller_test.go

@@ -139,7 +139,9 @@ func SetupDeploymentTest(ctx context.Context) *corev1.Namespace {
 		err := k8sClient.Create(ctx, ns)
 		Expect(err).NotTo(HaveOccurred(), "failed to create test namespace")
 
-		mgr, err := ctrl.NewManager(cfg, ctrl.Options{})
+		mgr, err := ctrl.NewManager(cfg, ctrl.Options{
+			Namespace: ns.Name,
+		})
 		Expect(err).NotTo(HaveOccurred(), "failed to create manager")
 
 		controller := &RunnerDeploymentReconciler{
@@ -199,7 +201,7 @@ var _ = Context("Inside of a new namespace", func() {
 								},
 							},
 							Spec: actionsv1alpha1.RunnerSpec{
-								Repository: "foo/bar",
+								Repository: "test/valid",
 								Image:      "bar",
 								Env: []corev1.EnvVar{
 									{Name: "FOO", Value: "FOOVALUE"},
@@ -295,7 +297,7 @@ var _ = Context("Inside of a new namespace", func() {
 						Replicas: intPtr(1),
 						Template: actionsv1alpha1.RunnerTemplate{
 							Spec: actionsv1alpha1.RunnerSpec{
-								Repository: "foo/bar",
+								Repository: "test/valid",
 								Image:      "bar",
 								Env: []corev1.EnvVar{
 									{Name: "FOO", Value: "FOOVALUE"},
@@ -391,7 +393,7 @@ var _ = Context("Inside of a new namespace", func() {
 						Replicas: intPtr(1),
 						Template: actionsv1alpha1.RunnerTemplate{
 							Spec: actionsv1alpha1.RunnerSpec{
-								Repository: "foo/bar",
+								Repository: "test/valid",
 								Image:      "bar",
 								Env: []corev1.EnvVar{
 									{Name: "FOO", Value: "FOOVALUE"},

controllers/runnerreplicaset_controller.go

@@ -20,6 +20,7 @@ import (
 	"context"
 	"errors"
 	"fmt"
+	"reflect"
 	"time"
 
 	gogithub "github.com/google/go-github/v33/github"
@@ -88,20 +89,23 @@ func (r *RunnerReplicaSetReconciler) Reconcile(req ctrl.Request) (ctrl.Result, e
 	var myRunners []v1alpha1.Runner
 
 	var (
-		available int
+		current   int
 		ready     int
+		available int
 	)
 
 	for _, r := range allRunners.Items {
 		// This guard is required to avoid the RunnerReplicaSet created by the controller v0.17.0 or before
 		// to not treat all the runners in the namespace as its children.
-		if metav1.IsControlledBy(&r, &rs) {
+		if metav1.IsControlledBy(&r, &rs) && !metav1.HasAnnotation(r.ObjectMeta, annotationKeyRegistrationOnly) {
 			myRunners = append(myRunners, r)
 
-			available += 1
+			current += 1
 
 			if r.Status.Phase == string(corev1.PodRunning) {
 				ready += 1
+				// available is currently the same as ready, as we don't yet have minReadySeconds for runners
+				available += 1
 			}
 		}
 	}
@@ -114,13 +118,79 @@ func (r *RunnerReplicaSetReconciler) Reconcile(req ctrl.Request) (ctrl.Result, e
 		desired = 1
 	}
 
-	log.V(0).Info("debug", "desired", desired, "available", available)
+	registrationOnlyRunnerNsName := req.NamespacedName
+	registrationOnlyRunnerNsName.Name = registrationOnlyRunnerNameFor(rs.Name)
+	registrationOnlyRunner := v1alpha1.Runner{}
+	registrationOnlyRunnerExists := false
+	if err := r.Get(
+		ctx,
+		registrationOnlyRunnerNsName,
+		&registrationOnlyRunner,
+	); err != nil {
+		if !kerrors.IsNotFound(err) {
+			return ctrl.Result{}, err
+		}
+	} else {
+		registrationOnlyRunnerExists = true
+	}
 
-	if available > desired {
-		n := available - desired
+	// On scale to zero, we must have fully registered registration-only runner before we start deleting other runners, hence `desired == 0`
+	// On scale from zero, we must retain the registratoin-only runner until one or more other runners get registered, hence `registrationOnlyRunnerExists && available == 0`.
+	// On RunnerReplicaSet creation, it have always 0 replics and no registration-only runner.
+	// In this case We don't need to bother creating a registration-only runner which gets deleted soon after we have 1 or more available repolicas,
+	// hence it's not `available == 0`, but `registrationOnlyRunnerExists && available == 0`.
+	// See https://github.com/actions-runner-controller/actions-runner-controller/issues/516
+	registrationOnlyRunnerNeeded := desired == 0 || (registrationOnlyRunnerExists && current == 0)
+
+	if registrationOnlyRunnerNeeded {
+		if registrationOnlyRunnerExists {
+			if registrationOnlyRunner.Status.Phase == "" {
+				log.Info("Still waiting for the registration-only runner to be registered")
+
+				return ctrl.Result{}, nil
+			}
+		} else {
+			// A registration-only runner does not exist and is needed, hence create it.
+
+			runnerForScaleFromToZero, err := r.newRunner(rs)
+			if err != nil {
+				return ctrl.Result{}, fmt.Errorf("failed to create runner for scale from/to zero: %v", err)
+			}
+
+			runnerForScaleFromToZero.ObjectMeta.Name = registrationOnlyRunnerNsName.Name
+			runnerForScaleFromToZero.ObjectMeta.GenerateName = ""
+			runnerForScaleFromToZero.ObjectMeta.Labels = nil
+			metav1.SetMetaDataAnnotation(&runnerForScaleFromToZero.ObjectMeta, annotationKeyRegistrationOnly, "true")
+
+			if err := r.Client.Create(ctx, &runnerForScaleFromToZero); err != nil {
+				log.Error(err, "Failed to create runner for scale from/to zero")
+
+				return ctrl.Result{}, err
+			}
+
+			// We can continue to deleting runner pods only after the
+			// registration-only runner gets registered.
+			return ctrl.Result{}, nil
+		}
+	} else {
+		// A registration-only runner exists and is not needed, hence delete it.
+		if registrationOnlyRunnerExists {
+			if err := r.Client.Delete(ctx, &registrationOnlyRunner); err != nil {
+				log.Error(err, "Retrying soon because we failed to delete registration-only runner")
+
+				return ctrl.Result{Requeue: true}, nil
+			}
+		}
+	}
+
+	if current > desired {
+		n := current - desired
+
+		log.V(0).Info(fmt.Sprintf("Deleting %d runners", n), "desired", desired, "current", current, "ready", ready)
+
+		// get runners that are currently offline/not busy/timed-out to register
+		var deletionCandidates []v1alpha1.Runner
 
-		// get runners that are currently not busy
-		var notBusy []v1alpha1.Runner
 		for _, runner := range allRunners.Items {
 			busy, err := r.GitHubClient.IsRunnerBusy(ctx, runner.Spec.Enterprise, runner.Spec.Organization, runner.Spec.Repository, runner.Name)
 			if err != nil {
@@ -168,34 +238,38 @@ func (r *RunnerReplicaSetReconciler) Reconcile(req ctrl.Request) (ctrl.Result, e
 						"configuredRegistrationTimeout", registrationTimeout,
 					)
 
-					notBusy = append(notBusy, runner)
+					deletionCandidates = append(deletionCandidates, runner)
 				}
 
 				// offline runners should always be a great target for scale down
 				if offline {
-					notBusy = append(notBusy, runner)
+					deletionCandidates = append(deletionCandidates, runner)
 				}
 			} else if !busy {
-				notBusy = append(notBusy, runner)
+				deletionCandidates = append(deletionCandidates, runner)
 			}
 		}
 
-		if len(notBusy) < n {
-			n = len(notBusy)
+		if len(deletionCandidates) < n {
+			n = len(deletionCandidates)
 		}
 
+		log.V(0).Info(fmt.Sprintf("Deleting %d runner(s)", n), "desired", desired, "current", current, "ready", ready)
+
 		for i := 0; i < n; i++ {
-			if err := r.Client.Delete(ctx, &notBusy[i]); client.IgnoreNotFound(err) != nil {
+			if err := r.Client.Delete(ctx, &deletionCandidates[i]); client.IgnoreNotFound(err) != nil {
 				log.Error(err, "Failed to delete runner resource")
 
 				return ctrl.Result{}, err
 			}
 
-			r.Recorder.Event(&rs, corev1.EventTypeNormal, "RunnerDeleted", fmt.Sprintf("Deleted runner '%s'", notBusy[i].Name))
-			log.Info("Deleted runner", "runnerreplicaset", rs.ObjectMeta.Name)
+			r.Recorder.Event(&rs, corev1.EventTypeNormal, "RunnerDeleted", fmt.Sprintf("Deleted runner '%s'", deletionCandidates[i].Name))
+			log.Info("Deleted runner")
 		}
-	} else if desired > available {
-		n := desired - available
+	} else if desired > current {
+		n := desired - current
+
+		log.V(0).Info(fmt.Sprintf("Creating %d runner(s)", n), "desired", desired, "available", current, "ready", ready)
 
 		for i := 0; i < n; i++ {
 			newRunner, err := r.newRunner(rs)
@@ -213,13 +287,18 @@ func (r *RunnerReplicaSetReconciler) Reconcile(req ctrl.Request) (ctrl.Result, e
 		}
 	}
 
-	if rs.Status.AvailableReplicas != available || rs.Status.ReadyReplicas != ready {
-		updated := rs.DeepCopy()
-		updated.Status.AvailableReplicas = available
-		updated.Status.ReadyReplicas = ready
+	var status v1alpha1.RunnerReplicaSetStatus
 
-		if err := r.Status().Update(ctx, updated); err != nil {
-			log.Info("Failed to update status. Retrying immediately", "error", err.Error())
+	status.Replicas = &current
+	status.AvailableReplicas = &available
+	status.ReadyReplicas = &ready
+
+	if !reflect.DeepEqual(rs.Status, status) {
+		updated := rs.DeepCopy()
+		updated.Status = status
+
+		if err := r.Status().Patch(ctx, updated, client.MergeFrom(&rs)); err != nil {
+			log.Info("Failed to update runnerreplicaset status. Retrying immediately", "error", err.Error())
 			return ctrl.Result{
 				Requeue: true,
 			}, nil
@@ -262,3 +341,7 @@ func (r *RunnerReplicaSetReconciler) SetupWithManager(mgr ctrl.Manager) error {
 		Named(name).
 		Complete(r)
 }
+
+func registrationOnlyRunnerNameFor(rsName string) string {
+	return rsName + "-registration-only"
+}

controllers/runnerreplicaset_controller_test.go

@@ -2,15 +2,14 @@ package controllers
 
 import (
 	"context"
+	"fmt"
 	"math/rand"
 	"net/http/httptest"
 	"time"
 
-	"github.com/google/go-github/v33/github"
 	corev1 "k8s.io/api/core/v1"
 	"k8s.io/apimachinery/pkg/types"
 	"k8s.io/client-go/kubernetes/scheme"
-	"k8s.io/utils/pointer"
 	ctrl "sigs.k8s.io/controller-runtime"
 	logf "sigs.k8s.io/controller-runtime/pkg/log"
 
@@ -47,7 +46,9 @@ func SetupTest(ctx context.Context) *corev1.Namespace {
 		err := k8sClient.Create(ctx, ns)
 		Expect(err).NotTo(HaveOccurred(), "failed to create test namespace")
 
-		mgr, err := ctrl.NewManager(cfg, ctrl.Options{})
+		mgr, err := ctrl.NewManager(cfg, ctrl.Options{
+			Namespace: ns.Name,
+		})
 		Expect(err).NotTo(HaveOccurred(), "failed to create manager")
 
 		runnersList = fake.NewRunnersList()
@@ -127,7 +128,7 @@ var _ = Context("Inside of a new namespace", func() {
 								},
 							},
 							Spec: actionsv1alpha1.RunnerSpec{
-								Repository: "foo/bar",
+								Repository: "test/valid",
 								Image:      "bar",
 								Env: []corev1.EnvVar{
 									{Name: "FOO", Value: "FOOVALUE"},
@@ -167,15 +168,7 @@ var _ = Context("Inside of a new namespace", func() {
 							return -1
 						}
 
-						for i, runner := range runners.Items {
-							runnersList.Add(&github.Runner{
-								ID:     pointer.Int64Ptr(int64(i) + 1),
-								Name:   pointer.StringPtr(runner.Name),
-								OS:     pointer.StringPtr("linux"),
-								Status: pointer.StringPtr("online"),
-								Busy:   pointer.BoolPtr(false),
-							})
-						}
+						runnersList.Sync(runners.Items)
 
 						return len(runners.Items)
 					},
@@ -224,15 +217,7 @@ var _ = Context("Inside of a new namespace", func() {
 							logf.Log.Error(err, "list runners")
 						}
 
-						for i, runner := range runners.Items {
-							runnersList.Add(&github.Runner{
-								ID:     pointer.Int64Ptr(int64(i) + 1),
-								Name:   pointer.StringPtr(runner.Name),
-								OS:     pointer.StringPtr("linux"),
-								Status: pointer.StringPtr("online"),
-								Busy:   pointer.BoolPtr(false),
-							})
-						}
+						runnersList.Sync(runners.Items)
 
 						return len(runners.Items)
 					},
@@ -260,21 +245,35 @@ var _ = Context("Inside of a new namespace", func() {
 
 				Eventually(
 					func() int {
-						err := k8sClient.List(ctx, &runners, client.InNamespace(ns.Name))
-						if err != nil {
+						selector, err := metav1.LabelSelectorAsSelector(&metav1.LabelSelector{
+							MatchLabels: map[string]string{
+								"foo": "bar",
+							},
+						})
+						Expect(err).ToNot(HaveOccurred())
+
+						var regOnly actionsv1alpha1.Runner
+						if err := k8sClient.Get(ctx, types.NamespacedName{Namespace: ns.Name, Name: registrationOnlyRunnerNameFor(name)}, &regOnly); err != nil {
+							logf.Log.Info(fmt.Sprintf("Failed getting registration-only runner in test: %v", err))
+							return -1
+						} else {
+							updated := regOnly.DeepCopy()
+							updated.Status.Phase = "Completed"
+
+							if err := k8sClient.Status().Patch(ctx, updated, client.MergeFrom(&regOnly)); err != nil {
+								logf.Log.Info(fmt.Sprintf("Failed updating registration-only runner in test: %v", err))
+								return -1
+							}
+
+							runnersList.AddOffline([]actionsv1alpha1.Runner{*updated})
+						}
+
+						if err := k8sClient.List(ctx, &runners, client.InNamespace(ns.Name), client.MatchingLabelsSelector{Selector: selector}); err != nil {
 							logf.Log.Error(err, "list runners")
 							return -1
 						}
 
-						for i, runner := range runners.Items {
-							runnersList.Add(&github.Runner{
-								ID:     pointer.Int64Ptr(int64(i) + 1),
-								Name:   pointer.StringPtr(runner.Name),
-								OS:     pointer.StringPtr("linux"),
-								Status: pointer.StringPtr("online"),
-								Busy:   pointer.BoolPtr(false),
-							})
-						}
+						runnersList.Sync(runners.Items)
 
 						return len(runners.Items)
 					},

controllers/schedule.go

@@ -0,0 +1,122 @@
+package controllers
+
+import (
+	"fmt"
+	"time"
+
+	"github.com/teambition/rrule-go"
+)
+
+type RecurrenceRule struct {
+	Frequency string
+	UntilTime time.Time
+}
+
+type Period struct {
+	StartTime time.Time
+	EndTime   time.Time
+}
+
+func (r *Period) String() string {
+	if r == nil {
+		return ""
+	}
+
+	return r.StartTime.Format(time.RFC3339) + "-" + r.EndTime.Format(time.RFC3339)
+}
+
+func MatchSchedule(now time.Time, startTime, endTime time.Time, recurrenceRule RecurrenceRule) (*Period, *Period, error) {
+	return calculateActiveAndUpcomingRecurringPeriods(
+		now,
+		startTime,
+		endTime,
+		recurrenceRule.Frequency,
+		recurrenceRule.UntilTime,
+	)
+}
+
+func calculateActiveAndUpcomingRecurringPeriods(now, startTime, endTime time.Time, frequency string, untilTime time.Time) (*Period, *Period, error) {
+	var freqValue rrule.Frequency
+
+	var freqDurationDay int
+	var freqDurationMonth int
+	var freqDurationYear int
+
+	switch frequency {
+	case "Daily":
+		freqValue = rrule.DAILY
+		freqDurationDay = 1
+	case "Weekly":
+		freqValue = rrule.WEEKLY
+		freqDurationDay = 7
+	case "Monthly":
+		freqValue = rrule.MONTHLY
+		freqDurationMonth = 1
+	case "Yearly":
+		freqValue = rrule.YEARLY
+		freqDurationYear = 1
+	case "":
+		if now.Before(startTime) {
+			return nil, &Period{StartTime: startTime, EndTime: endTime}, nil
+		}
+
+		if now.Before(endTime) {
+			return &Period{StartTime: startTime, EndTime: endTime}, nil, nil
+		}
+
+		return nil, nil, nil
+	default:
+		return nil, nil, fmt.Errorf(`invalid freq %q: It must be one of "Daily", "Weekly", "Monthly", and "Yearly"`, frequency)
+	}
+
+	freqDurationLater := time.Date(
+		now.Year()+freqDurationYear,
+		time.Month(int(now.Month())+freqDurationMonth),
+		now.Day()+freqDurationDay,
+		now.Hour(), now.Minute(), now.Second(), now.Nanosecond(), now.Location(),
+	)
+
+	freqDuration := freqDurationLater.Sub(now)
+
+	overrideDuration := endTime.Sub(startTime)
+	if overrideDuration > freqDuration {
+		return nil, nil, fmt.Errorf("override's duration %s must be equal to sor shorter than the duration implied by freq %q (%s)", overrideDuration, frequency, freqDuration)
+	}
+
+	rrule, err := rrule.NewRRule(rrule.ROption{
+		Freq:    freqValue,
+		Dtstart: startTime,
+		Until:   untilTime,
+	})
+	if err != nil {
+		return nil, nil, err
+	}
+
+	overrideDurationBefore := now.Add(-overrideDuration + 1)
+	activeOverrideStarts := rrule.Between(overrideDurationBefore, now, true)
+
+	var active *Period
+
+	if len(activeOverrideStarts) > 1 {
+		return nil, nil, fmt.Errorf("[bug] unexpted number of active overrides found: %v", activeOverrideStarts)
+	} else if len(activeOverrideStarts) == 1 {
+		active = &Period{
+			StartTime: activeOverrideStarts[0],
+			EndTime:   activeOverrideStarts[0].Add(overrideDuration),
+		}
+	}
+
+	oneSecondLater := now.Add(1)
+	upcomingOverrideStarts := rrule.Between(oneSecondLater, freqDurationLater, true)
+
+	var next *Period
+
+	if len(upcomingOverrideStarts) > 0 {
+		next = &Period{
+			StartTime: upcomingOverrideStarts[0],
+			EndTime:   upcomingOverrideStarts[0].Add(overrideDuration),
+		}
+	}
+
+	return active, next, nil
+}

controllers/schedule_test.go

@@ -0,0 +1,607 @@
+package controllers
+
+import (
+	"testing"
+	"time"
+)
+
+func TestCalculateActiveAndUpcomingRecurringPeriods(t *testing.T) {
+	type recurrence struct {
+		Start string
+		End   string
+		Freq  string
+		Until string
+	}
+
+	type testcase struct {
+		now string
+
+		recurrence recurrence
+
+		wantActive   string
+		wantUpcoming string
+	}
+
+	check := func(t *testing.T, tc testcase) {
+		t.Helper()
+
+		_, err := time.Parse(time.RFC3339, "2021-05-08T00:00:00Z")
+		if err != nil {
+			t.Fatal(err)
+		}
+
+		now, err := time.Parse(time.RFC3339, tc.now)
+		if err != nil {
+			t.Fatal(err)
+		}
+
+		active, upcoming, err := parseAndMatchRecurringPeriod(now, tc.recurrence.Start, tc.recurrence.End, tc.recurrence.Freq, tc.recurrence.Until)
+		if err != nil {
+			t.Fatal(err)
+		}
+
+		if active.String() != tc.wantActive {
+			t.Errorf("unexpected active: want %q, got %q", tc.wantActive, active)
+		}
+
+		if upcoming.String() != tc.wantUpcoming {
+			t.Errorf("unexpected upcoming: want %q, got %q", tc.wantUpcoming, upcoming)
+		}
+	}
+
+	t.Run("onetime override about to start", func(t *testing.T) {
+		t.Helper()
+
+		check(t, testcase{
+			recurrence: recurrence{
+				Start: "2021-05-01T00:00:00+09:00",
+				End:   "2021-05-03T00:00:00+09:00",
+			},
+
+			now: "2021-04-30T23:59:59+09:00",
+
+			wantActive:   "",
+			wantUpcoming: "2021-05-01T00:00:00+09:00-2021-05-03T00:00:00+09:00",
+		})
+	})
+
+	t.Run("onetime override started", func(t *testing.T) {
+		t.Helper()
+
+		check(t, testcase{
+			recurrence: recurrence{
+				Start: "2021-05-01T00:00:00+09:00",
+				End:   "2021-05-03T00:00:00+09:00",
+			},
+
+			now: "2021-05-01T00:00:00+09:00",
+
+			wantActive:   "2021-05-01T00:00:00+09:00-2021-05-03T00:00:00+09:00",
+			wantUpcoming: "",
+		})
+	})
+
+	t.Run("onetime override about to end", func(t *testing.T) {
+		t.Helper()
+
+		check(t, testcase{
+			recurrence: recurrence{
+				Start: "2021-05-01T00:00:00+09:00",
+				End:   "2021-05-03T00:00:00+09:00",
+			},
+
+			now: "2021-05-02T23:59:59+09:00",
+
+			wantActive:   "2021-05-01T00:00:00+09:00-2021-05-03T00:00:00+09:00",
+			wantUpcoming: "",
+		})
+	})
+
+	t.Run("onetime override ended", func(t *testing.T) {
+		t.Helper()
+
+		check(t, testcase{
+			recurrence: recurrence{
+				Start: "2021-05-01T00:00:00+09:00",
+				End:   "2021-05-03T00:00:00+09:00",
+			},
+
+			now: "2021-05-03T00:00:00+09:00",
+
+			wantActive:   "",
+			wantUpcoming: "",
+		})
+	})
+
+	t.Run("weekly override about to start", func(t *testing.T) {
+		t.Helper()
+
+		check(t, testcase{
+			recurrence: recurrence{
+				Start: "2021-05-01T00:00:00+09:00",
+				End:   "2021-05-03T00:00:00+09:00",
+				Freq:  "Weekly",
+				Until: "2022-05-01T00:00:00+09:00",
+			},
+
+			now: "2021-04-30T23:59:59+09:00",
+
+			wantActive:   "",
+			wantUpcoming: "2021-05-01T00:00:00+09:00-2021-05-03T00:00:00+09:00",
+		})
+	})
+
+	t.Run("weekly override started", func(t *testing.T) {
+		t.Helper()
+
+		check(t, testcase{
+			recurrence: recurrence{
+				Start: "2021-05-01T00:00:00+09:00",
+				End:   "2021-05-03T00:00:00+09:00",
+				Freq:  "Weekly",
+				Until: "2022-05-01T00:00:00+09:00",
+			},
+
+			now: "2021-05-01T00:00:00+09:00",
+
+			wantActive:   "2021-05-01T00:00:00+09:00-2021-05-03T00:00:00+09:00",
+			wantUpcoming: "2021-05-08T00:00:00+09:00-2021-05-10T00:00:00+09:00",
+		})
+	})
+
+	t.Run("weekly override about to end", func(t *testing.T) {
+		t.Helper()
+
+		check(t, testcase{
+			recurrence: recurrence{
+				Start: "2021-05-01T00:00:00+09:00",
+				End:   "2021-05-03T00:00:00+09:00",
+				Freq:  "Weekly",
+				Until: "2022-05-01T00:00:00+09:00",
+			},
+
+			now: "2021-05-02T23:59:59+09:00",
+
+			wantActive:   "2021-05-01T00:00:00+09:00-2021-05-03T00:00:00+09:00",
+			wantUpcoming: "2021-05-08T00:00:00+09:00-2021-05-10T00:00:00+09:00",
+		})
+	})
+
+	t.Run("weekly override ended", func(t *testing.T) {
+		t.Helper()
+
+		check(t, testcase{
+			recurrence: recurrence{
+				Start: "2021-05-01T00:00:00+09:00",
+				End:   "2021-05-03T00:00:00+09:00",
+				Freq:  "Weekly",
+				Until: "2022-05-01T00:00:00+09:00",
+			},
+
+			now: "2021-05-03T00:00:00+09:00",
+
+			wantActive:   "",
+			wantUpcoming: "2021-05-08T00:00:00+09:00-2021-05-10T00:00:00+09:00",
+		})
+	})
+
+	t.Run("weekly override reccurrence about to start", func(t *testing.T) {
+		t.Helper()
+
+		check(t, testcase{
+			recurrence: recurrence{
+				Start: "2021-05-01T00:00:00+09:00",
+				End:   "2021-05-03T00:00:00+09:00",
+				Freq:  "Weekly",
+				Until: "2022-05-01T00:00:00+09:00",
+			},
+
+			now: "2021-05-07T23:59:59+09:00",
+
+			wantActive:   "",
+			wantUpcoming: "2021-05-08T00:00:00+09:00-2021-05-10T00:00:00+09:00",
+		})
+	})
+
+	t.Run("weekly override reccurrence started", func(t *testing.T) {
+		t.Helper()
+
+		check(t, testcase{
+			recurrence: recurrence{
+				Start: "2021-05-01T00:00:00+09:00",
+				End:   "2021-05-03T00:00:00+09:00",
+				Freq:  "Weekly",
+				Until: "2022-05-01T00:00:00+09:00",
+			},
+
+			now: "2021-05-08T00:00:00+09:00",
+
+			wantActive:   "2021-05-08T00:00:00+09:00-2021-05-10T00:00:00+09:00",
+			wantUpcoming: "2021-05-15T00:00:00+09:00-2021-05-17T00:00:00+09:00",
+		})
+	})
+
+	t.Run("weekly override reccurrence about to end", func(t *testing.T) {
+		t.Helper()
+
+		check(t, testcase{
+			recurrence: recurrence{
+				Start: "2021-05-01T00:00:00+09:00",
+				End:   "2021-05-03T00:00:00+09:00",
+				Freq:  "Weekly",
+				Until: "2022-05-01T00:00:00+09:00",
+			},
+
+			now: "2021-05-09T23:59:59+09:00",
+
+			wantActive:   "2021-05-08T00:00:00+09:00-2021-05-10T00:00:00+09:00",
+			wantUpcoming: "2021-05-15T00:00:00+09:00-2021-05-17T00:00:00+09:00",
+		})
+	})
+
+	t.Run("weekly override reccurrence ended", func(t *testing.T) {
+		t.Helper()
+
+		check(t, testcase{
+			recurrence: recurrence{
+				Start: "2021-05-01T00:00:00+09:00",
+				End:   "2021-05-03T00:00:00+09:00",
+				Freq:  "Weekly",
+				Until: "2022-05-01T00:00:00+09:00",
+			},
+
+			now: "2021-05-10T00:00:00+09:00",
+
+			wantActive:   "",
+			wantUpcoming: "2021-05-15T00:00:00+09:00-2021-05-17T00:00:00+09:00",
+		})
+	})
+
+	t.Run("weekly override's last reccurrence about to start", func(t *testing.T) {
+		t.Helper()
+
+		check(t, testcase{
+			recurrence: recurrence{
+				Start: "2021-05-01T00:00:00+09:00",
+				End:   "2021-05-03T00:00:00+09:00",
+				Freq:  "Weekly",
+				Until: "2022-05-01T00:00:00+09:00",
+			},
+
+			now: "2022-04-29T23:59:59+09:00",
+
+			wantActive:   "",
+			wantUpcoming: "2022-04-30T00:00:00+09:00-2022-05-02T00:00:00+09:00",
+		})
+	})
+
+	t.Run("weekly override reccurrence started", func(t *testing.T) {
+		t.Helper()
+
+		check(t, testcase{
+			recurrence: recurrence{
+				Start: "2021-05-01T00:00:00+09:00",
+				End:   "2021-05-03T00:00:00+09:00",
+				Freq:  "Weekly",
+				Until: "2022-05-01T00:00:00+09:00",
+			},
+
+			now: "2022-04-30T00:00:00+09:00",
+
+			wantActive:   "2022-04-30T00:00:00+09:00-2022-05-02T00:00:00+09:00",
+			wantUpcoming: "",
+		})
+	})
+
+	t.Run("weekly override reccurrence about to end", func(t *testing.T) {
+		t.Helper()
+
+		check(t, testcase{
+			recurrence: recurrence{
+				Start: "2021-05-01T00:00:00+09:00",
+				End:   "2021-05-03T00:00:00+09:00",
+				Freq:  "Weekly",
+				Until: "2022-05-01T00:00:00+09:00",
+			},
+
+			now: "2022-05-01T23:59:59+09:00",
+
+			wantActive:   "2022-04-30T00:00:00+09:00-2022-05-02T00:00:00+09:00",
+			wantUpcoming: "",
+		})
+	})
+
+	t.Run("weekly override reccurrence ended", func(t *testing.T) {
+		t.Helper()
+
+		check(t, testcase{
+			recurrence: recurrence{
+				Start: "2021-05-01T00:00:00+09:00",
+				End:   "2021-05-03T00:00:00+09:00",
+				Freq:  "Weekly",
+				Until: "2022-05-01T00:00:00+09:00",
+			},
+
+			now: "2022-05-02T00:00:00+09:00",
+
+			wantActive:   "",
+			wantUpcoming: "",
+		})
+	})
+
+	t.Run("weekly override repeated forever started", func(t *testing.T) {
+		t.Helper()
+
+		check(t, testcase{
+			recurrence: recurrence{
+				Start: "2021-05-01T00:00:00+09:00",
+				End:   "2021-05-03T00:00:00+09:00",
+				Freq:  "Weekly",
+			},
+
+			now: "2021-05-08T00:00:00+09:00",
+
+			wantActive:   "2021-05-08T00:00:00+09:00-2021-05-10T00:00:00+09:00",
+			wantUpcoming: "2021-05-15T00:00:00+09:00-2021-05-17T00:00:00+09:00",
+		})
+	})
+
+	t.Run("monthly override started", func(t *testing.T) {
+		t.Helper()
+
+		check(t, testcase{
+			recurrence: recurrence{
+				Start: "2021-05-01T00:00:00+09:00",
+				End:   "2021-05-03T00:00:00+09:00",
+				Freq:  "Monthly",
+				Until: "2022-05-01T00:00:00+09:00",
+			},
+
+			now: "2021-05-01T00:00:00+09:00",
+
+			wantActive:   "2021-05-01T00:00:00+09:00-2021-05-03T00:00:00+09:00",
+			wantUpcoming: "2021-06-01T00:00:00+09:00-2021-06-03T00:00:00+09:00",
+		})
+	})
+
+	t.Run("monthly override recurrence started", func(t *testing.T) {
+		t.Helper()
+
+		check(t, testcase{
+			recurrence: recurrence{
+				Start: "2021-05-01T00:00:00+09:00",
+				End:   "2021-05-03T00:00:00+09:00",
+				Freq:  "Monthly",
+				Until: "2022-05-01T00:00:00+09:00",
+			},
+
+			now: "2021-06-01T00:00:00+09:00",
+
+			wantActive:   "2021-06-01T00:00:00+09:00-2021-06-03T00:00:00+09:00",
+			wantUpcoming: "2021-07-01T00:00:00+09:00-2021-07-03T00:00:00+09:00",
+		})
+	})
+
+	t.Run("monthly override's last reccurence about to start", func(t *testing.T) {
+		t.Helper()
+
+		check(t, testcase{
+			recurrence: recurrence{
+				Start: "2021-05-01T00:00:00+09:00",
+				End:   "2021-05-03T00:00:00+09:00",
+				Freq:  "Monthly",
+				Until: "2022-05-01T00:00:00+09:00",
+			},
+
+			now: "2022-04-30T23:59:59+09:00",
+
+			wantActive:   "",
+			wantUpcoming: "2022-05-01T00:00:00+09:00-2022-05-03T00:00:00+09:00",
+		})
+	})
+
+	t.Run("monthly override's last reccurence started", func(t *testing.T) {
+		t.Helper()
+
+		check(t, testcase{
+			recurrence: recurrence{
+				Start: "2021-05-01T00:00:00+09:00",
+				End:   "2021-05-03T00:00:00+09:00",
+				Freq:  "Monthly",
+				Until: "2022-05-01T00:00:00+09:00",
+			},
+
+			now: "2022-05-01T00:00:00+09:00",
+
+			wantActive:   "2022-05-01T00:00:00+09:00-2022-05-03T00:00:00+09:00",
+			wantUpcoming: "",
+		})
+	})
+
+	t.Run("monthly override's last reccurence started", func(t *testing.T) {
+		t.Helper()
+
+		check(t, testcase{
+			recurrence: recurrence{
+				Start: "2021-05-01T00:00:00+09:00",
+				End:   "2021-05-03T00:00:00+09:00",
+				Freq:  "Monthly",
+				Until: "2022-05-01T00:00:00+09:00",
+			},
+
+			now: "2022-05-01T00:00:01+09:00",
+
+			wantActive:   "2022-05-01T00:00:00+09:00-2022-05-03T00:00:00+09:00",
+			wantUpcoming: "",
+		})
+	})
+
+	t.Run("monthly override's last reccurence ending", func(t *testing.T) {
+		t.Helper()
+
+		check(t, testcase{
+			recurrence: recurrence{
+				Start: "2021-05-01T00:00:00+09:00",
+				End:   "2021-05-03T00:00:00+09:00",
+				Freq:  "Monthly",
+				Until: "2022-05-01T00:00:00+09:00",
+			},
+
+			now: "2022-05-02T23:59:59+09:00",
+
+			wantActive:   "2022-05-01T00:00:00+09:00-2022-05-03T00:00:00+09:00",
+			wantUpcoming: "",
+		})
+	})
+
+	t.Run("monthly override's last reccurence ended", func(t *testing.T) {
+		t.Helper()
+
+		check(t, testcase{
+			recurrence: recurrence{
+				Start: "2021-05-01T00:00:00+09:00",
+				End:   "2021-05-03T00:00:00+09:00",
+				Freq:  "Monthly",
+				Until: "2022-05-01T00:00:00+09:00",
+			},
+
+			now: "2022-05-03T00:00:00+09:00",
+
+			wantActive:   "",
+			wantUpcoming: "",
+		})
+	})
+
+	t.Run("yearly override started", func(t *testing.T) {
+		t.Helper()
+
+		check(t, testcase{
+			recurrence: recurrence{
+				Start: "2021-05-01T00:00:00+09:00",
+				End:   "2021-05-03T00:00:00+09:00",
+				Freq:  "Yearly",
+				Until: "2022-05-01T00:00:00+09:00",
+			},
+
+			now: "2021-05-01T00:00:00+09:00",
+
+			wantActive:   "2021-05-01T00:00:00+09:00-2021-05-03T00:00:00+09:00",
+			wantUpcoming: "2022-05-01T00:00:00+09:00-2022-05-03T00:00:00+09:00",
+		})
+	})
+
+	t.Run("yearly override reccurrence started", func(t *testing.T) {
+		t.Helper()
+
+		check(t, testcase{
+			recurrence: recurrence{
+				Start: "2021-05-01T00:00:00+09:00",
+				End:   "2021-05-03T00:00:00+09:00",
+				Freq:  "Yearly",
+				Until: "2023-05-01T00:00:00+09:00",
+			},
+
+			now: "2022-05-01T00:00:00+09:00",
+
+			wantActive:   "2022-05-01T00:00:00+09:00-2022-05-03T00:00:00+09:00",
+			wantUpcoming: "2023-05-01T00:00:00+09:00-2023-05-03T00:00:00+09:00",
+		})
+	})
+
+	t.Run("yearly override's last recurrence about to start", func(t *testing.T) {
+		t.Helper()
+
+		check(t, testcase{
+			recurrence: recurrence{
+				Start: "2021-05-01T00:00:00+09:00",
+				End:   "2021-05-03T00:00:00+09:00",
+				Freq:  "Yearly",
+				Until: "2023-05-01T00:00:00+09:00",
+			},
+
+			now: "2023-04-30T23:59:59+09:00",
+
+			wantActive:   "",
+			wantUpcoming: "2023-05-01T00:00:00+09:00-2023-05-03T00:00:00+09:00",
+		})
+	})
+
+	t.Run("yearly override's last recurrence started", func(t *testing.T) {
+		t.Helper()
+
+		check(t, testcase{
+			recurrence: recurrence{
+				Start: "2021-05-01T00:00:00+09:00",
+				End:   "2021-05-03T00:00:00+09:00",
+				Freq:  "Yearly",
+				Until: "2023-05-01T00:00:00+09:00",
+			},
+
+			now: "2023-05-01T00:00:00+09:00",
+
+			wantActive:   "2023-05-01T00:00:00+09:00-2023-05-03T00:00:00+09:00",
+			wantUpcoming: "",
+		})
+	})
+
+	t.Run("yearly override's last recurrence ending", func(t *testing.T) {
+		t.Helper()
+
+		check(t, testcase{
+			recurrence: recurrence{
+				Start: "2021-05-01T00:00:00+09:00",
+				End:   "2021-05-03T00:00:00+09:00",
+				Freq:  "Yearly",
+				Until: "2023-05-01T00:00:00+09:00",
+			},
+
+			now: "2023-05-02T23:23:59+09:00",
+
+			wantActive:   "2023-05-01T00:00:00+09:00-2023-05-03T00:00:00+09:00",
+			wantUpcoming: "",
+		})
+	})
+
+	t.Run("yearly override's last recurrence ended", func(t *testing.T) {
+		t.Helper()
+
+		check(t, testcase{
+			recurrence: recurrence{
+				Start: "2021-05-01T00:00:00+09:00",
+				End:   "2021-05-03T00:00:00+09:00",
+				Freq:  "Yearly",
+				Until: "2023-05-01T00:00:00+09:00",
+			},
+
+			now: "2023-05-03T00:00:00+09:00",
+
+			wantActive:   "",
+			wantUpcoming: "",
+		})
+	})
+}
+
+func parseAndMatchRecurringPeriod(now time.Time, start, end, frequency, until string) (*Period, *Period, error) {
+	startTime, err := time.Parse(time.RFC3339, start)
+	if err != nil {
+		return nil, nil, err
+	}
+
+	endTime, err := time.Parse(time.RFC3339, end)
+	if err != nil {
+		return nil, nil, err
+	}
+
+	var untilTime time.Time
+
+	if until != "" {
+		ut, err := time.Parse(time.RFC3339, until)
+		if err != nil {
+			return nil, nil, err
+		}
+
+		untilTime = ut
+	}
+
+	return MatchSchedule(now, startTime, endTime, RecurrenceRule{Frequency: frequency, UntilTime: untilTime})
+}

controllers/suite_test.go

@@ -55,9 +55,17 @@ func TestAPIs(t *testing.T) {
 var _ = BeforeSuite(func(done Done) {
 	logf.SetLogger(zap.LoggerTo(GinkgoWriter, true))
 
+	var apiServerFlags []string
+
+	apiServerFlags = append(apiServerFlags, envtest.DefaultKubeAPIServerFlags...)
+	// Avoids the following error:
+	// 2021-03-19T15:14:11.673+0900    ERROR   controller-runtime.controller   Reconciler error      {"controller": "testns-tvjzjrunner", "request": "testns-gdnyx/example-runnerdeploy-zps4z-j5562", "error": "Pod \"example-runnerdeploy-zps4z-j5562\" is invalid: [spec.containers[1].image: Required value, spec.containers[1].securityContext.privileged: Forbidden: disallowed by cluster policy]"}
+	apiServerFlags = append(apiServerFlags, "--allow-privileged=true")
+
 	By("bootstrapping test environment")
 	testEnv = &envtest.Environment{
-		CRDDirectoryPaths: []string{filepath.Join("..", "config", "crd", "bases")},
+		CRDDirectoryPaths:  []string{filepath.Join("..", "config", "crd", "bases")},
+		KubeAPIServerFlags: apiServerFlags,
 	}
 
 	var err error

github/fake/runners.go

@@ -2,11 +2,12 @@ package fake
 
 import (
 	"encoding/json"
-	"github.com/summerwind/actions-runner-controller/api/v1alpha1"
 	"net/http"
 	"net/http/httptest"
 	"strconv"
 
+	"github.com/summerwind/actions-runner-controller/api/v1alpha1"
+
 	"github.com/google/go-github/v33/github"
 	"github.com/gorilla/mux"
 )
@@ -79,6 +80,18 @@ func (r *RunnersList) Sync(runners []v1alpha1.Runner) {
 	}
 }
 
+func (r *RunnersList) AddOffline(runners []v1alpha1.Runner) {
+	for i, want := range runners {
+		r.Add(&github.Runner{
+			ID:     github.Int64(int64(1000 + i)),
+			Name:   github.String(want.Name),
+			OS:     github.String("linux"),
+			Status: github.String("offline"),
+			Busy:   github.Bool(false),
+		})
+	}
+}
+
 func exists(runners []*github.Runner, runner *github.Runner) bool {
 	for _, r := range runners {
 		if *r.Name == *runner.Name {

go.mod

@@ -14,10 +14,11 @@ require (
 	github.com/onsi/gomega v1.5.0
 	github.com/prometheus/client_golang v0.9.2
 	github.com/stretchr/testify v1.4.0 // indirect
+	github.com/teambition/rrule-go v1.6.2
+	go.uber.org/zap v1.9.1
 	golang.org/x/oauth2 v0.0.0-20190604053449-0f29369cfe45
 	k8s.io/api v0.0.0-20190918155943-95b840bb6a1f
 	k8s.io/apimachinery v0.0.0-20190913080033-27d36303b655
 	k8s.io/client-go v0.0.0-20190918160344-1fbdaa4c8d90
-	k8s.io/utils v0.0.0-20190801114015-581e00157fb1
 	sigs.k8s.io/controller-runtime v0.4.0
 )

go.sum

@@ -112,7 +112,6 @@ github.com/golang/protobuf v1.3.2 h1:6nsPYzhq5kReh6QImI3k5qWzO4PEbvbIW2cwSfR/6xs
 github.com/golang/protobuf v1.3.2/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
 github.com/google/btree v0.0.0-20180813153112-4030bb1f1f0c/go.mod h1:lNA+9X1NB3Zf8V7Ke586lFgjr2dZNuvo3lPJSGZ5JPQ=
 github.com/google/go-cmp v0.2.0/go.mod h1:oXzfMopK8JAjlY9xF4vHSVASa0yLyX7SntLO5aqRK0M=
-github.com/google/go-cmp v0.3.0 h1:crn/baboCvb5fXaQ0IJ1SGTsTVrWpDsCWC8EGETZijY=
 github.com/google/go-cmp v0.3.0/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU=
 github.com/google/go-cmp v0.3.1 h1:Xye71clBPdm5HgqGwUkwhbynsUJZhDbS20FvLhQ2izg=
 github.com/google/go-cmp v0.3.1/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU=
@@ -216,7 +215,6 @@ github.com/prometheus/procfs v0.0.0-20181204211112-1dc9a6cbc91a h1:9a8MnZMP0X2nL
 github.com/prometheus/procfs v0.0.0-20181204211112-1dc9a6cbc91a/go.mod h1:c3At6R/oaqEKCNdg8wHV1ftS6bRYblBhIjjI8uT2IGk=
 github.com/remyoudompheng/bigfft v0.0.0-20170806203942-52369c62f446/go.mod h1:uYEyJGbgTkfkS4+E/PavXkNJcbFIpEtjt2B0KDQ5+9M=
 github.com/russross/blackfriday v1.5.2/go.mod h1:JO/DiYxRf+HjHt06OyowR9PTA263kcR/rfWxYHBV53g=
-github.com/sirupsen/logrus v1.4.2 h1:SPIRibHv4MatM3XXNO2BJeFLZwZ2LvZgfQ5+UNI2im4=
 github.com/sirupsen/logrus v1.4.2/go.mod h1:tLMulIdttU9McNUspp0xgXVQah82FyeX6MwdIuYE2rE=
 github.com/soheilhy/cmux v0.1.3/go.mod h1:IM3LyeVVIOuxMH7sFAkER9+bJ4dT7Ms6E4xg4kGIyLM=
 github.com/spf13/afero v1.1.2/go.mod h1:j4pytiNVoe2o6bmDsKpLACNPDBIoEAkihy7loJ1B0CQ=
@@ -233,10 +231,11 @@ github.com/stretchr/objx v0.1.1/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+
 github.com/stretchr/objx v0.2.0/go.mod h1:qt09Ya8vawLte6SNmTgCsAVtYtaKzEcn8ATUoHMkEqE=
 github.com/stretchr/testify v0.0.0-20151208002404-e3a8ff8ce365/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs=
 github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs=
-github.com/stretchr/testify v1.3.0 h1:TivCn/peBQ7UY8ooIcPgZFpTNSz0Q2U6UrFlUfqbe0Q=
 github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
 github.com/stretchr/testify v1.4.0 h1:2E4SXV/wtOkTonXsotYi4li6zVWxYlZuYNCXe9XRJyk=
 github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4=
+github.com/teambition/rrule-go v1.6.2 h1:keZiiijltBxYUuhQaySAEGyIFR0UOkAd7i+u6FM5/+I=
+github.com/teambition/rrule-go v1.6.2/go.mod h1:mBJ1Ht5uboJ6jexKdNUJg2NcwP8uUMNvStWXlJD3MvU=
 github.com/tmc/grpc-websocket-proxy v0.0.0-20170815181823-89b8d40f7ca8/go.mod h1:ncp9v5uamzpCO7NfCPTXjqaC+bZgJeR0sMTm6dMHP7U=
 github.com/ugorji/go/codec v0.0.0-20181204163529-d75b2dcb6bc8/go.mod h1:VFNgLljTbGfSG7qAOspJ7OScBnGdDN/yBr0sguwnwf0=
 github.com/xiang90/probing v0.0.0-20160813154853-07dd2e8dfe18/go.mod h1:UETIi67q53MR2AWcXfiuqkDkRtnGDLqkBTpCHuJHxtU=

main.go

@@ -27,6 +27,7 @@ import (
 	actionsv1alpha1 "github.com/summerwind/actions-runner-controller/api/v1alpha1"
 	"github.com/summerwind/actions-runner-controller/controllers"
 	"github.com/summerwind/actions-runner-controller/github"
+	zaplib "go.uber.org/zap"
 	"k8s.io/apimachinery/pkg/runtime"
 	clientgoscheme "k8s.io/client-go/kubernetes/scheme"
 	_ "k8s.io/client-go/plugin/pkg/client/auth/gcp"
@@ -38,11 +39,16 @@ import (
 const (
 	defaultRunnerImage = "summerwind/actions-runner:latest"
 	defaultDockerImage = "docker:dind"
+
+	logLevelDebug = "debug"
+	logLevelInfo  = "info"
+	logLevelWarn  = "warn"
+	logLevelError = "error"
 )
 
 var (
-	scheme   = runtime.NewScheme()
-	setupLog = ctrl.Log.WithName("setup")
+	scheme = runtime.NewScheme()
+	log    = ctrl.Log.WithName("actions-runner-controller")
 )
 
 func init() {
@@ -61,9 +67,12 @@ func main() {
 		enableLeaderElection bool
 		syncPeriod           time.Duration
 
+		gitHubAPICacheDuration time.Duration
+
 		runnerImage string
 		dockerImage string
 		namespace   string
+		logLevel    string
 
 		commonRunnerLabels commaSeparatedStringSlice
 	)
@@ -83,13 +92,27 @@ func main() {
 	flag.Int64Var(&c.AppID, "github-app-id", c.AppID, "The application ID of GitHub App.")
 	flag.Int64Var(&c.AppInstallationID, "github-app-installation-id", c.AppInstallationID, "The installation ID of GitHub App.")
 	flag.StringVar(&c.AppPrivateKey, "github-app-private-key", c.AppPrivateKey, "The path of a private key file to authenticate as a GitHub App")
-	flag.DurationVar(&syncPeriod, "sync-period", 10*time.Minute, "Determines the minimum frequency at which K8s resources managed by this controller are reconciled. When you use autoscaling, set to a lower value like 10 minute, because this corresponds to the minimum time to react on demand change")
+	flag.DurationVar(&gitHubAPICacheDuration, "github-api-cache-duration", 0, "The duration until the GitHub API cache expires. Setting this to e.g. 10m results in the controller tries its best not to make the same API call within 10m to reduce the chance of being rate-limited. Defaults to mostly the same value as sync-period. If you're tweaking this in order to make autoscaling more responsive, you'll probably want to tweak sync-period, too")
+	flag.DurationVar(&syncPeriod, "sync-period", 10*time.Minute, "Determines the minimum frequency at which K8s resources managed by this controller are reconciled. When you use autoscaling, set to a lower value like 10 minute, because this corresponds to the minimum time to react on demand change. . If you're tweaking this in order to make autoscaling more responsive, you'll probably want to tweak github-api-cache-duration, too")
 	flag.Var(&commonRunnerLabels, "common-runner-labels", "Runner labels in the K1=V1,K2=V2,... format that are inherited all the runners created by the controller. See https://github.com/summerwind/actions-runner-controller/issues/321 for more information")
 	flag.StringVar(&namespace, "watch-namespace", "", "The namespace to watch for custom resources. Set to empty for letting it watch for all namespaces.")
+	flag.StringVar(&logLevel, "log-level", logLevelDebug, `The verbosity of the logging. Valid values are "debug", "info", "warn", "error". Defaults to "debug".`)
 	flag.Parse()
 
 	logger := zap.New(func(o *zap.Options) {
-		o.Development = true
+		switch logLevel {
+		case logLevelDebug:
+			o.Development = true
+		case logLevelInfo:
+			lvl := zaplib.NewAtomicLevelAt(zaplib.InfoLevel)
+			o.Level = &lvl
+		case logLevelWarn:
+			lvl := zaplib.NewAtomicLevelAt(zaplib.WarnLevel)
+			o.Level = &lvl
+		case logLevelError:
+			lvl := zaplib.NewAtomicLevelAt(zaplib.ErrorLevel)
+			o.Level = &lvl
+		}
 	})
 
 	ghClient, err = c.NewClient()
@@ -109,13 +132,13 @@ func main() {
 		Namespace:          namespace,
 	})
 	if err != nil {
-		setupLog.Error(err, "unable to start manager")
+		log.Error(err, "unable to start manager")
 		os.Exit(1)
 	}
 
 	runnerReconciler := &controllers.RunnerReconciler{
 		Client:       mgr.GetClient(),
-		Log:          ctrl.Log.WithName("controllers").WithName("Runner"),
+		Log:          log.WithName("runner"),
 		Scheme:       mgr.GetScheme(),
 		GitHubClient: ghClient,
 		RunnerImage:  runnerImage,
@@ -123,64 +146,82 @@ func main() {
 	}
 
 	if err = runnerReconciler.SetupWithManager(mgr); err != nil {
-		setupLog.Error(err, "unable to create controller", "controller", "Runner")
+		log.Error(err, "unable to create controller", "controller", "Runner")
 		os.Exit(1)
 	}
 
 	runnerSetReconciler := &controllers.RunnerReplicaSetReconciler{
 		Client:       mgr.GetClient(),
-		Log:          ctrl.Log.WithName("controllers").WithName("RunnerReplicaSet"),
+		Log:          log.WithName("runnerreplicaset"),
 		Scheme:       mgr.GetScheme(),
 		GitHubClient: ghClient,
 	}
 
 	if err = runnerSetReconciler.SetupWithManager(mgr); err != nil {
-		setupLog.Error(err, "unable to create controller", "controller", "RunnerReplicaSet")
+		log.Error(err, "unable to create controller", "controller", "RunnerReplicaSet")
 		os.Exit(1)
 	}
 
 	runnerDeploymentReconciler := &controllers.RunnerDeploymentReconciler{
 		Client:             mgr.GetClient(),
-		Log:                ctrl.Log.WithName("controllers").WithName("RunnerDeployment"),
+		Log:                log.WithName("runnerdeployment"),
 		Scheme:             mgr.GetScheme(),
 		CommonRunnerLabels: commonRunnerLabels,
 	}
 
 	if err = runnerDeploymentReconciler.SetupWithManager(mgr); err != nil {
-		setupLog.Error(err, "unable to create controller", "controller", "RunnerDeployment")
+		log.Error(err, "unable to create controller", "controller", "RunnerDeployment")
 		os.Exit(1)
 	}
 
+	if gitHubAPICacheDuration == 0 {
+		gitHubAPICacheDuration = syncPeriod - 10*time.Second
+	}
+
+	if gitHubAPICacheDuration < 0 {
+		gitHubAPICacheDuration = 0
+	}
+
+	log.Info(
+		"Initializing actions-runner-controller",
+		"github-api-cahce-duration", gitHubAPICacheDuration,
+		"sync-period", syncPeriod,
+		"runner-image", runnerImage,
+		"docker-image", dockerImage,
+		"common-runnner-labels", commonRunnerLabels,
+		"watch-namespace", namespace,
+	)
+
 	horizontalRunnerAutoscaler := &controllers.HorizontalRunnerAutoscalerReconciler{
 		Client:        mgr.GetClient(),
-		Log:           ctrl.Log.WithName("controllers").WithName("HorizontalRunnerAutoscaler"),
+		Log:           log.WithName("horizontalrunnerautoscaler"),
 		Scheme:        mgr.GetScheme(),
 		GitHubClient:  ghClient,
-		CacheDuration: syncPeriod - 10*time.Second,
+		CacheDuration: gitHubAPICacheDuration,
 	}
 
 	if err = horizontalRunnerAutoscaler.SetupWithManager(mgr); err != nil {
-		setupLog.Error(err, "unable to create controller", "controller", "HorizontalRunnerAutoscaler")
+		log.Error(err, "unable to create controller", "controller", "HorizontalRunnerAutoscaler")
 		os.Exit(1)
 	}
 
 	if err = (&actionsv1alpha1.Runner{}).SetupWebhookWithManager(mgr); err != nil {
-		setupLog.Error(err, "unable to create webhook", "webhook", "Runner")
+		log.Error(err, "unable to create webhook", "webhook", "Runner")
 		os.Exit(1)
 	}
 	if err = (&actionsv1alpha1.RunnerDeployment{}).SetupWebhookWithManager(mgr); err != nil {
-		setupLog.Error(err, "unable to create webhook", "webhook", "RunnerDeployment")
+		log.Error(err, "unable to create webhook", "webhook", "RunnerDeployment")
 		os.Exit(1)
 	}
 	if err = (&actionsv1alpha1.RunnerReplicaSet{}).SetupWebhookWithManager(mgr); err != nil {
-		setupLog.Error(err, "unable to create webhook", "webhook", "RunnerReplicaSet")
+		log.Error(err, "unable to create webhook", "webhook", "RunnerReplicaSet")
 		os.Exit(1)
 	}
 	// +kubebuilder:scaffold:builder
 
-	setupLog.Info("starting manager")
+	log.Info("starting manager")
 	if err := mgr.Start(ctrl.SetupSignalHandler()); err != nil {
-		setupLog.Error(err, "problem running manager")
+		log.Error(err, "problem running manager")
 		os.Exit(1)
 	}
 }

runner/Dockerfile

@@ -1,65 +1,68 @@
-FROM ubuntu:18.04
+FROM ubuntu:20.04
 
 ARG TARGETPLATFORM
-ARG RUNNER_VERSION=2.274.2
+ARG RUNNER_VERSION=2.278.0
 ARG DOCKER_VERSION=19.03.12
 
 RUN test -n "$TARGETPLATFORM" || (echo "TARGETPLATFORM must be set" && false)
 
 ENV DEBIAN_FRONTEND=noninteractive
 RUN apt update -y \
-  && apt install -y software-properties-common \
-  && add-apt-repository -y ppa:git-core/ppa \
-  && apt update -y \
-  && apt install -y --no-install-recommends \
-  build-essential \
-  curl \
-  ca-certificates \
-  dnsutils \
-  ftp \
-  git \
-  iproute2 \
-  iputils-ping \
-  jq \
-  libunwind8 \
-  locales \
-  netcat \
-  openssh-client \
-  parallel \
-  rsync \
-  shellcheck \
-  sudo \
-  telnet \
-  time \
-  tzdata \
-  unzip \
-  upx \
-  wget \
-  zip \
-  zstd \
-  && cd /usr/bin && ln -sf python3 python \
-  && rm -rf /var/lib/apt/lists/*
+    && apt install -y software-properties-common \
+    && add-apt-repository -y ppa:git-core/ppa \
+    && apt update -y \
+    && apt install -y --no-install-recommends \
+    build-essential \
+    curl \
+    ca-certificates \
+    dnsutils \
+    ftp \
+    git \
+    iproute2 \
+    iputils-ping \
+    jq \
+    libunwind8 \
+    locales \
+    netcat \
+    openssh-client \
+    parallel \
+    python3-pip \
+    rsync \
+    shellcheck \
+    sudo \
+    telnet \
+    time \
+    tzdata \
+    unzip \
+    upx \
+    wget \
+    zip \
+    zstd \
+    && ln -sf /usr/bin/python3 /usr/bin/python \
+    && ln -sf /usr/bin/pip3 /usr/bin/pip \
+    && rm -rf /var/lib/apt/lists/*
 
 RUN export ARCH=$(echo ${TARGETPLATFORM} | cut -d / -f2) \
-  && curl -L -o /usr/local/bin/dumb-init https://github.com/Yelp/dumb-init/releases/download/v1.2.2/dumb-init_1.2.2_${ARCH} \
-  && chmod +x /usr/local/bin/dumb-init
+    && curl -L -o /usr/local/bin/dumb-init https://github.com/Yelp/dumb-init/releases/download/v1.2.2/dumb-init_1.2.2_${ARCH} \
+    && chmod +x /usr/local/bin/dumb-init
 
 # Docker download supports arm64 as aarch64 & amd64 as x86_64
 RUN set -vx; \
-  export ARCH=$(echo ${TARGETPLATFORM} | cut -d / -f2) \
-  && if [ "$ARCH" = "arm64" ]; then export ARCH=aarch64 ; fi \
-  && if [ "$ARCH" = "amd64" ]; then export ARCH=x86_64 ; fi \
-  && curl -L -o docker.tgz https://download.docker.com/linux/static/stable/${ARCH}/docker-${DOCKER_VERSION}.tgz \
-  && tar zxvf docker.tgz \
-  && install -o root -g root -m 755 docker/docker /usr/local/bin/docker \
-  && rm -rf docker docker.tgz \
-  && adduser --disabled-password --gecos "" --uid 1000 runner \
-  && groupadd docker \
-  && usermod -aG sudo runner \
-  && usermod -aG docker runner \
-  && echo "%sudo   ALL=(ALL:ALL) NOPASSWD:ALL" > /etc/sudoers
+    export ARCH=$(echo ${TARGETPLATFORM} | cut -d / -f2) \
+    && if [ "$ARCH" = "arm64" ]; then export ARCH=aarch64 ; fi \
+    && if [ "$ARCH" = "amd64" ]; then export ARCH=x86_64 ; fi \
+    && curl -L -o docker.tgz https://download.docker.com/linux/static/stable/${ARCH}/docker-${DOCKER_VERSION}.tgz \
+    && tar zxvf docker.tgz \
+    && install -o root -g root -m 755 docker/docker /usr/local/bin/docker \
+    && rm -rf docker docker.tgz \
+    && adduser --disabled-password --gecos "" --uid 1000 runner \
+    && groupadd docker \
+    && usermod -aG sudo runner \
+    && usermod -aG docker runner \
+    && echo "%sudo   ALL=(ALL:ALL) NOPASSWD:ALL" > /etc/sudoers
 
 ENV RUNNER_ASSETS_DIR=/runnertmp
+ENV HOME=/home/runner
 
 # Runner download supports amd64 as x64. Externalstmp is needed for making mount points work inside DinD.
 #
@@ -67,25 +70,28 @@ ENV RUNNER_ASSETS_DIR=/runnertmp
 # It is installed after installdependencies.sh and before removing /var/lib/apt/lists
 # to avoid rerunning apt-update on its own.
 RUN export ARCH=$(echo ${TARGETPLATFORM} | cut -d / -f2) \
-  && if [ "$ARCH" = "amd64" ]; then export ARCH=x64 ; fi \
-  && mkdir -p "$RUNNER_ASSETS_DIR" \
-  && cd "$RUNNER_ASSETS_DIR" \
-  && curl -L -o runner.tar.gz https://github.com/actions/runner/releases/download/v${RUNNER_VERSION}/actions-runner-linux-${ARCH}-${RUNNER_VERSION}.tar.gz \
-  && tar xzf ./runner.tar.gz \
-  && rm runner.tar.gz \
-  && ./bin/installdependencies.sh \
-  && mv ./externals ./externalstmp \
-  && apt-get install -y libyaml-dev \
-  && rm -rf /var/lib/apt/lists/*
+    && if [ "$ARCH" = "amd64" -o "$ARCH" = "x86_64" ]; then export ARCH=x64 ; fi \
+    && mkdir -p "$RUNNER_ASSETS_DIR" \
+    && cd "$RUNNER_ASSETS_DIR" \
+    && curl -L -o runner.tar.gz https://github.com/actions/runner/releases/download/v${RUNNER_VERSION}/actions-runner-linux-${ARCH}-${RUNNER_VERSION}.tar.gz \
+    && tar xzf ./runner.tar.gz \
+    && rm runner.tar.gz \
+    && ./bin/installdependencies.sh \
+    && mv ./externals ./externalstmp \
+    && apt-get install -y libyaml-dev \
+    && rm -rf /var/lib/apt/lists/*
 
 RUN echo AGENT_TOOLSDIRECTORY=/opt/hostedtoolcache > .env \
-  && mkdir /opt/hostedtoolcache \
-  && chgrp docker /opt/hostedtoolcache \
-  && chmod g+rwx /opt/hostedtoolcache
+    && mkdir /opt/hostedtoolcache \
+    && chgrp docker /opt/hostedtoolcache \
+    && chmod g+rwx /opt/hostedtoolcache
 
 COPY entrypoint.sh /
 COPY --chown=runner:docker patched $RUNNER_ASSETS_DIR/patched
 
+# Add the Python "User Script Directory" to the PATH
+ENV PATH="${PATH}:${HOME}/.local/bin"
+
 USER runner
 ENTRYPOINT ["/usr/local/bin/dumb-init", "--"]
 CMD ["/entrypoint.sh"]

runner/Dockerfile.dindrunner

@@ -1,11 +1,11 @@
 FROM ubuntu:20.04
 
 ENV DEBIAN_FRONTEND=noninteractive
-# Dev + DinD dependencies
-RUN apt update \
+RUN apt update -y \
     && apt install -y software-properties-common \
     && add-apt-repository -y ppa:git-core/ppa \
-    && apt install -y \
+    && apt update -y \
+    && apt install -y --no-install-recommends \
     build-essential \
     curl \
     ca-certificates \
@@ -13,19 +13,21 @@ RUN apt update \
     ftp \
     git \
     iproute2 \
-    iptables \
     iputils-ping \
+    iptables \
     jq \
     libunwind8 \
     locales \
     netcat \
+    net-tools \
     openssh-client \
     parallel \
-    python-is-python3 \
+    python3-pip \
     rsync \
     shellcheck \
-    sudo \
     supervisor \
+    software-properties-common \
+    sudo \
     telnet \
     time \
     tzdata \
@@ -34,7 +36,9 @@ RUN apt update \
     wget \
     zip \
     zstd \
-    && rm -rf /var/lib/apt/list/*
+    && ln -sf /usr/bin/python3 /usr/bin/python \
+    && ln -sf /usr/bin/pip3 /usr/bin/pip \
+    && rm -rf /var/lib/apt/lists/*
 
 # Runner user
 RUN adduser --disabled-password --gecos "" --uid 1000 runner \
@@ -44,7 +48,7 @@ RUN adduser --disabled-password --gecos "" --uid 1000 runner \
     && echo "%sudo   ALL=(ALL:ALL) NOPASSWD:ALL" > /etc/sudoers
 
 ARG TARGETPLATFORM
-ARG RUNNER_VERSION=2.274.1
+ARG RUNNER_VERSION=2.278.0
 ARG DOCKER_CHANNEL=stable
 ARG DOCKER_VERSION=19.03.13
 ARG DEBUG=false
@@ -70,6 +74,7 @@ RUN export ARCH=$(echo ${TARGETPLATFORM} | cut -d / -f2) \
 	docker --version
 
 ENV RUNNER_ASSETS_DIR=/runnertmp
+ENV HOME=/home/runner
 
 # Runner download supports amd64 as x64
 #
@@ -77,9 +82,9 @@ ENV RUNNER_ASSETS_DIR=/runnertmp
 # It is installed after installdependencies.sh and before removing /var/lib/apt/lists
 # to avoid rerunning apt-update on its own.
 RUN export ARCH=$(echo ${TARGETPLATFORM} | cut -d / -f2) \
-    && if [ "$ARCH" = "amd64" ]; then export ARCH=x64 ; fi \
+    && if [ "$ARCH" = "amd64" -o "$ARCH" = "x86_64" ]; then export ARCH=x64 ; fi \
     && mkdir -p "$RUNNER_ASSETS_DIR" \
-     && cd "$RUNNER_ASSETS_DIR" \
+    && cd "$RUNNER_ASSETS_DIR" \
     && curl -L -o runner.tar.gz https://github.com/actions/runner/releases/download/v${RUNNER_VERSION}/actions-runner-linux-${ARCH}-${RUNNER_VERSION}.tar.gz \
     && tar xzf ./runner.tar.gz \
     && rm runner.tar.gz \
@@ -88,9 +93,9 @@ RUN export ARCH=$(echo ${TARGETPLATFORM} | cut -d / -f2) \
     && rm -rf /var/lib/apt/lists/*
 
 RUN echo AGENT_TOOLSDIRECTORY=/opt/hostedtoolcache > /runner.env \
-  && mkdir /opt/hostedtoolcache \
-  && chgrp docker /opt/hostedtoolcache \
-  && chmod g+rwx /opt/hostedtoolcache
+    && mkdir /opt/hostedtoolcache \
+    && chgrp docker /opt/hostedtoolcache \
+    && chmod g+rwx /opt/hostedtoolcache
 
 COPY modprobe startup.sh /usr/local/bin/
 COPY supervisor/ /etc/supervisor/conf.d/
@@ -107,6 +112,9 @@ VOLUME /var/lib/docker
 
 COPY --chown=runner:docker patched $RUNNER_ASSETS_DIR/patched
 
+# Add the Python "User Script Directory" to the PATH
+ENV PATH="${PATH}:${HOME}/.local/bin"
+
 # No group definition, as that makes it harder to run docker.
 USER runner
 

runner/Dockerfile.ubuntu.1804

@@ -0,0 +1,97 @@
+FROM ubuntu:18.04
+
+ARG TARGETPLATFORM
+ARG RUNNER_VERSION=2.278.0
+ARG DOCKER_VERSION=19.03.12
+
+RUN test -n "$TARGETPLATFORM" || (echo "TARGETPLATFORM must be set" && false)
+
+ENV DEBIAN_FRONTEND=noninteractive
+RUN apt update -y \
+    && apt install -y software-properties-common \
+    && add-apt-repository -y ppa:git-core/ppa \
+    && apt update -y \
+    && apt install -y --no-install-recommends \
+    build-essential \
+    curl \
+    ca-certificates \
+    dnsutils \
+    ftp \
+    git \
+    iproute2 \
+    iputils-ping \
+    jq \
+    libunwind8 \
+    locales \
+    netcat \
+    openssh-client \
+    parallel \
+    python3-pip \
+    rsync \
+    shellcheck \
+    sudo \
+    telnet \
+    time \
+    tzdata \
+    unzip \
+    upx \
+    wget \
+    zip \
+    zstd \
+    && ln -sf /usr/bin/python3 /usr/bin/python \
+    && ln -sf /usr/bin/pip3 /usr/bin/pip \
+    && rm -rf /var/lib/apt/lists/*
+
+RUN export ARCH=$(echo ${TARGETPLATFORM} | cut -d / -f2) \
+    && curl -L -o /usr/local/bin/dumb-init https://github.com/Yelp/dumb-init/releases/download/v1.2.2/dumb-init_1.2.2_${ARCH} \
+    && chmod +x /usr/local/bin/dumb-init
+
+# Docker download supports arm64 as aarch64 & amd64 as x86_64
+RUN set -vx; \
+    export ARCH=$(echo ${TARGETPLATFORM} | cut -d / -f2) \
+    && if [ "$ARCH" = "arm64" ]; then export ARCH=aarch64 ; fi \
+    && if [ "$ARCH" = "amd64" ]; then export ARCH=x86_64 ; fi \
+    && curl -L -o docker.tgz https://download.docker.com/linux/static/stable/${ARCH}/docker-${DOCKER_VERSION}.tgz \
+    && tar zxvf docker.tgz \
+    && install -o root -g root -m 755 docker/docker /usr/local/bin/docker \
+    && rm -rf docker docker.tgz \
+    && adduser --disabled-password --gecos "" --uid 1000 runner \
+    && groupadd docker \
+    && usermod -aG sudo runner \
+    && usermod -aG docker runner \
+    && echo "%sudo   ALL=(ALL:ALL) NOPASSWD:ALL" > /etc/sudoers
+
+ENV RUNNER_ASSETS_DIR=/runnertmp
+ENV HOME=/home/runner
+
+# Runner download supports amd64 as x64. Externalstmp is needed for making mount points work inside DinD.
+#
+# libyaml-dev is required for ruby/setup-ruby action.
+# It is installed after installdependencies.sh and before removing /var/lib/apt/lists
+# to avoid rerunning apt-update on its own.
+RUN export ARCH=$(echo ${TARGETPLATFORM} | cut -d / -f2) \
+    && if [ "$ARCH" = "amd64" -o "$ARCH" = "x86_64" ]; then export ARCH=x64 ; fi \
+    && mkdir -p "$RUNNER_ASSETS_DIR" \
+    && cd "$RUNNER_ASSETS_DIR" \
+    && curl -L -o runner.tar.gz https://github.com/actions/runner/releases/download/v${RUNNER_VERSION}/actions-runner-linux-${ARCH}-${RUNNER_VERSION}.tar.gz \
+    && tar xzf ./runner.tar.gz \
+    && rm runner.tar.gz \
+    && ./bin/installdependencies.sh \
+    && mv ./externals ./externalstmp \
+    && apt-get install -y libyaml-dev \
+    && rm -rf /var/lib/apt/lists/*
+
+RUN echo AGENT_TOOLSDIRECTORY=/opt/hostedtoolcache > .env \
+    && mkdir /opt/hostedtoolcache \
+    && chgrp docker /opt/hostedtoolcache \
+    && chmod g+rwx /opt/hostedtoolcache
+
+COPY entrypoint.sh /
+COPY --chown=runner:docker patched $RUNNER_ASSETS_DIR/patched
+
+# Add the Python "User Script Directory" to the PATH
+ENV PATH="${PATH}:${HOME}/.local/bin"
+
+USER runner
+ENTRYPOINT ["/usr/local/bin/dumb-init", "--"]
+CMD ["/entrypoint.sh"]

runner/Makefile

@@ -2,7 +2,7 @@ NAME ?= summerwind/actions-runner
 DIND_RUNNER_NAME ?= ${NAME}-dind
 TAG ?= latest
 
-RUNNER_VERSION ?= 2.274.2
+RUNNER_VERSION ?= 2.278.0
 DOCKER_VERSION ?= 19.03.12
 
 # default list of platforms for which multiarch image is built
@@ -22,16 +22,15 @@ else
 	export PUSH_ARG="--push"
 endif
 
-docker-build:
+docker-build-ubuntu:
 	docker build --build-arg TARGETPLATFORM=amd64 --build-arg RUNNER_VERSION=${RUNNER_VERSION} --build-arg DOCKER_VERSION=${DOCKER_VERSION} -t ${NAME}:${TAG} .
-	docker build --build-arg TARGETPLATFORM=amd64 --build-arg RUNNER_VERSION=${RUNNER_VERSION} --build-arg DOCKER_VERSION=${DOCKER_VERSION} -t ${DIND_RUNNER_NAME}:${TAG} -f dindrunner.Dockerfile .
+	docker build --build-arg TARGETPLATFORM=amd64 --build-arg RUNNER_VERSION=${RUNNER_VERSION} --build-arg DOCKER_VERSION=${DOCKER_VERSION} -t ${DIND_RUNNER_NAME}:${TAG} -f Dockerfile.dindrunner .
 
-
-docker-push:
+docker-push-ubuntu:
 	docker push ${NAME}:${TAG}
 	docker push ${DIND_RUNNER_NAME}:${TAG}
 
-docker-buildx:
+docker-buildx-ubuntu:
 	export DOCKER_CLI_EXPERIMENTAL=enabled
 	@if ! docker buildx ls | grep -q container-builder; then\
 		docker buildx create --platform ${PLATFORMS} --name container-builder --use;\
@@ -46,5 +45,5 @@ docker-buildx:
 		--build-arg RUNNER_VERSION=${RUNNER_VERSION} \
 		--build-arg DOCKER_VERSION=${DOCKER_VERSION} \
 		-t "${DIND_RUNNER_NAME}:latest" \
-		-f dindrunner.Dockerfile \
+		-f Dockerfile.dindrunner \
 		. ${PUSH_ARG}

runner/entrypoint.sh

@@ -1,5 +1,10 @@
 #!/bin/bash
 
+if [ ! -z "${STARTUP_DELAY}" ]; then
+  echo "Delaying startup by ${STARTUP_DELAY} seconds" 1>&2
+  sleep ${STARTUP_DELAY}
+fi
+
 if [ -z "${GITHUB_URL}" ]; then
   echo "Working with public GitHub" 1>&2
   GITHUB_URL="https://github.com/"
@@ -29,21 +34,13 @@ else
   exit 1
 fi
 
-if [ -n "${RUNNER_WORKDIR}" ]; then
-  WORKDIR_ARG="--work ${RUNNER_WORKDIR}"
-fi
-
-if [ -n "${RUNNER_LABELS}" ]; then
-  LABEL_ARG="--labels ${RUNNER_LABELS}"
-fi
-
 if [ -z "${RUNNER_TOKEN}" ]; then
   echo "RUNNER_TOKEN must be set" 1>&2
   exit 1
 fi
 
 if [ -z "${RUNNER_REPO}" ] && [ -n "${RUNNER_GROUP}" ];then
-  RUNNER_GROUP_ARG="--runnergroup ${RUNNER_GROUP}"
+  RUNNER_GROUPS=${RUNNER_GROUP}
 fi
 
 # Hack due to https://github.com/summerwind/actions-runner-controller/issues/252#issuecomment-758338483
@@ -56,7 +53,45 @@ sudo chown -R runner:docker /runner
 mv /runnertmp/* /runner/
 
 cd /runner
-./config.sh --unattended --replace --name "${RUNNER_NAME}" --url "${GITHUB_URL}${ATTACH}" --token "${RUNNER_TOKEN}" ${RUNNER_GROUP_ARG} ${LABEL_ARG} ${WORKDIR_ARG}
+./config.sh --unattended --replace \
+  --name "${RUNNER_NAME}" \
+  --url "${GITHUB_URL}${ATTACH}" \
+  --token "${RUNNER_TOKEN}" \
+  --runnergroup "${RUNNER_GROUPS}" \
+  --labels "${RUNNER_LABELS}" \
+  --work "${RUNNER_WORKDIR}"
+
+if [ -f /runner/.runner ]; then
+  echo Runner has successfully been configured with the following data.
+  cat /runner/.runner
+  # Note: the `.runner` file's content should be something like the below:
+  #
+  # $ cat /runner/.runner
+  # {
+  # "agentId": 117, #=> corresponds to the ID of the runner
+  # "agentName": "THE_RUNNER_POD_NAME",
+  # "poolId": 1,
+  # "poolName": "Default",
+  # "serverUrl": "https://pipelines.actions.githubusercontent.com/SOME_RANDOM_ID",
+  # "gitHubUrl": "https://github.com/USER/REPO",
+  # "workFolder": "/some/work/dir" #=> corresponds to Runner.Spec.WorkDir
+  # }
+  #
+  # Especially `agentId` is important, as other than listing all the runners in the repo,
+  # this is the only change we could get the exact runnner ID which can be useful for further
+  # GitHub API call like the below. Note that 171 is the agentId seen above.
+  #   curl \
+  #     -H "Accept: application/vnd.github.v3+json" \
+  #     -H "Authorization: bearer ${GITHUB_TOKEN}"
+  #     https://api.github.com/repos/USER/REPO/actions/runners/171
+fi
+
+if [ -n "${RUNNER_REGISTRATION_ONLY}" ]; then
+  echo
+  echo "This runner is configured to be registration-only. Existing without starting the runner service..."
+  exit 0
+fi
+
 mkdir ./externals
 # Hack due to the DinD volumes
 mv ./externalstmp/* ./externals/
@@ -67,5 +102,10 @@ for f in runsvc.sh RunnerService.js; do
   sudo mv {patched,bin}/${f}
 done
 
+args=()
+if [ "${RUNNER_EPHEMERAL}" != "false" ]; then
+  args+=(--once)
+fi
+
 unset RUNNER_NAME RUNNER_REPO RUNNER_TOKEN
-exec ./bin/runsvc.sh --once
+exec ./bin/runsvc.sh "${args[@]}"

runner/startup.sh

@@ -17,6 +17,30 @@ function wait_for_process () {
     return 0
 }
 
+sudo /bin/bash <<SCRIPT
+mkdir -p /etc/docker
+
+echo "{}" > /etc/docker/daemon.json
+
+if [ -n "${MTU}" ]; then
+jq ".\"mtu\" = ${MTU}" /etc/docker/daemon.json > /tmp/.daemon.json && mv /tmp/.daemon.json /etc/docker/daemon.json
+# See https://docs.docker.com/engine/security/rootless/
+echo "environment=DOCKERD_ROOTLESS_ROOTLESSKIT_MTU=${MTU}" >> /etc/supervisor/conf.d/dockerd.conf
+fi
+
+if [ -n "${DOCKER_REGISTRY_MIRROR}" ]; then
+jq ".\"registry-mirrors\"[0] = \"${DOCKER_REGISTRY_MIRROR}\"" /etc/docker/daemon.json > /tmp/.daemon.json && mv /tmp/.daemon.json /etc/docker/daemon.json
+fi
+SCRIPT
+
+INFO "Using /etc/docker/daemon.json with the following content"
+
+cat /etc/docker/daemon.json
+
+INFO "Using /etc/supervisor/conf.d/dockerd.conf with the following content"
+
+cat /etc/supervisor/conf.d/dockerd.conf
+
 INFO "Starting supervisor"
 sudo /usr/bin/supervisord -n >> /dev/null 2>&1 &
 
@@ -27,6 +51,8 @@ for process in "${processes[@]}"; do
     wait_for_process "$process"
     if [ $? -ne 0 ]; then
         ERROR "$process is not running after max time"
+        ERROR "Dumping /var/log/dockerd.err.log to help investigation"
+        cat /var/log/dockerd.err.log
         exit 1
     else 
         INFO "$process is running"
@@ -34,7 +60,7 @@ for process in "${processes[@]}"; do
 done
 
 if [ -n "${MTU}" ]; then
-  ifconfig docker0 mtu ${MTU} up
+  sudo ifconfig docker0 mtu ${MTU} up
 fi
 
 # Wait processes to be running