feat: Workflow job based ephemeral runner scaling (#721)

This add support for two upcoming enhancements on the GitHub side of self-hosted runners, ephemeral runners, and `workflow_jow` events. You can't use these yet.

**These features are not yet generally available to all GitHub users**. Please take this pull request as a preparation to make it available to actions-runner-controller users as soon as possible after GitHub released the necessary features on their end.

**Ephemeral runners**:

The former, ephemeral runners, is basically the reliable alternative to `--once`, which we've been using when you enabled `ephemeral: true` (default in actions-runner-controller).

`--once` has been suffering from a race issue #466. `--ephemeral` fixes that.

To enable ephemeral runners with `actions/runner`, you give `--ephemeral` to `config.sh`. This updated version of `actions-runner-controller` does it for you, by using `--ephemeral` instead of `--once` when you set `RUNNER_FEATURE_FLAG_EPHEMERAL=true`.

Please read the section `Ephemeral Runners` in the updated version of our README for more information.

Note that ephemeral runners is not released on GitHub yet. And `RUNNER_FEATURE_FLAG_EPHEMERAL=true` won't work at all until the feature gets released on GitHub. Stay tuned for an announcement from GitHub!

**`workflow_job` events**:

`workflow_job` is the additional webhook event that corresponds to each GitHub Actions workflow job run. It provides `actions-runner-controller` a solid foundation to improve our webhook-based autoscale.

Formerly, we've been exploiting webhook events like `check_run` for autoscaling. However, as none of our supported events has included `labels`, you had to configure an HRA to only match relevant `check_run` events. It wasn't trivial.

In contrast, a `workflow_job` event payload contains `labels` of runners requested. `actions-runner-controller` is able to automatically decide which HRA to scale by filtering the corresponding RunnerDeployment by `labels` included in the webhook payload. So all you need to use webhook-based autoscale will be to enable `workflow_job` on GitHub and expose actions-runner-controller's webhook server to the internet.

Note that the current implementation of `workflow_job` support works in two ways, increment, and decrement. An increment happens when the webhook server receives` workflow_job` of `queued` status. A decrement happens when it receives `workflow_job` of `completed` status. The latter is used to make scaling-down faster so that you waste money less than before. You still don't suffer from flapping, as a scale-down is still subject to `scaleDownDelaySecondsAfterScaleOut `.

Please read the section `Example 3: Scale on each `workflow_job` event` in the updated version of our README for more information on its usage.

runner/Dockerfile

@@ -67,6 +67,24 @@ RUN set -vx; \
 ENV RUNNER_ASSETS_DIR=/runnertmp
 ENV HOME=/home/runner
 
+# Uncomment the below COPY to use your own custom build of actions-runner.
+#
+# To build a custom runner:
+# - Clone the actions/runner repo `git clone git@github.com:actions/runner.git $repo`
+# - Run `cd $repo/src`
+# - Run `./dev.sh layout Release linux-x64`
+# - Run `./dev.sh package Release linux-x64`
+# - Run cp ../_package/actions-runner-linux-x64-2.279.0.tar.gz ../../actions-runner-controller/runner/
+#   - Beware that `2.279.0` might change across versions
+#
+# See https://github.com/actions/runner/blob/main/.github/workflows/release.yml for more informatino on how you can use dev.sh
+#
+# If you're willing to uncomment the following line, you'd also need to comment-out the
+#   && curl -L -o runner.tar.gz https://github.com/actions/runner/releases/download/v${RUNNER_VERSION}/actions-runner-linux-${ARCH}-${RUNNER_VERSION}.tar.gz \
+# line in the next `RUN` command in this Dockerfile, to avoid overwiding this runner.tar.gz with a remote one.
+
+# COPY actions-runner-linux-x64-2.279.0.tar.gz /runnertmp/runner.tar.gz
+
 # Runner download supports amd64 as x64. Externalstmp is needed for making mount points work inside DinD.
 #
 # libyaml-dev is required for ruby/setup-ruby action.
@@ -76,6 +94,7 @@ RUN export ARCH=$(echo ${TARGETPLATFORM} | cut -d / -f2) \
     && if [ "$ARCH" = "amd64" ] || [ "$ARCH" = "x86_64" ] || [ "$ARCH" = "i386" ]; then export ARCH=x64 ; fi \
     && mkdir -p "$RUNNER_ASSETS_DIR" \
     && cd "$RUNNER_ASSETS_DIR" \
+    # Comment-out the below curl invocation when you use your own build of actions/runner
     && curl -L -o runner.tar.gz https://github.com/actions/runner/releases/download/v${RUNNER_VERSION}/actions-runner-linux-${ARCH}-${RUNNER_VERSION}.tar.gz \
     && tar xzf ./runner.tar.gz \
     && rm runner.tar.gz \

runner/entrypoint.sh

@@ -53,13 +53,20 @@ sudo chown -R runner:docker /runner
 mv /runnertmp/* /runner/
 
 cd /runner
+
+config_args=()
+if [ "${RUNNER_FEATURE_FLAG_EPHEMERAL:-}" == "true" -a "${RUNNER_EPHEMERAL}" != "false" ]; then
+  config_args+=(--ephemeral)
+  echo "Passing --ephemeral to config.sh to enable the ephemeral runner."
+fi
+
 ./config.sh --unattended --replace \
   --name "${RUNNER_NAME}" \
   --url "${GITHUB_URL}${ATTACH}" \
   --token "${RUNNER_TOKEN}" \
   --runnergroup "${RUNNER_GROUPS}" \
   --labels "${RUNNER_LABELS}" \
-  --work "${RUNNER_WORKDIR}"
+  --work "${RUNNER_WORKDIR}" "${config_args[@]}"
 
 if [ -f /runner/.runner ]; then
   echo Runner has successfully been configured with the following data.
@@ -103,8 +110,9 @@ for f in runsvc.sh RunnerService.js; do
 done
 
 args=()
-if [ "${RUNNER_EPHEMERAL}" != "false" ]; then
+if [ "${RUNNER_FEATURE_FLAG_EPHEMERAL:-}" != "true" -a "${RUNNER_EPHEMERAL}" != "false" ]; then
   args+=(--once)
+  echo "Passing --once to runsvc.sh to enable the legacy ephemeral runner."
 fi
 
 unset RUNNER_NAME RUNNER_REPO RUNNER_TOKEN