The primary goal of this change is to let the tester know about the config difference between the explicitly configured ephemeral work volume vs the automatically configured work volume with workVolumeClaimTemplate+containerMode=kubernetes.
* added troubleshooting solution
* added error example
* added entry to the pages index
* sorted
Co-authored-by: Mike Joseph <mike@Mikes-MacBook-Pro-5618.local>
Co-authored-by: Mike Joseph <mike@efrontier.com>
* added containerMode=kubernetes env variables to the runner
* removed unused logging
* restored configs and charts
* restored makefile cert version and acceptance/run
* added workVolumeClaimTemplate in pod definition, including logic
* added claim template name based on the runner
* Apply suggestions from code review
update errors
* added concurrent cleanup before runner pod is deleted
* update manifests
* added retry after 30s if pod cleanup contains err
* added admission webhook check, made workVolumeClaimTemplate mandatory for k8s
* style changes and added comments
* added izZero timestamp check for deleting runner-linked pods
* changed order of local variable to avoid copy if p is deleted
* removed docker from container mode k8s
* restored charts, config, makefile
* restored forked files back and not the ARC ones
* created PersistentVolume on containerMode k8s
* create pv only if storage class name is local-storage
* removed actions if storage class name is local-storage
* added service account validation if container mode kubernetes
* changed the coding style to match rest of the ARC
* added validation to the runnerdeployment webhook
* specified fields more precisely, added webhook validation to the replicaset as well
* remake manifests
* wraped delete runner-linked-pods in kube mode
* fixed empty line
* fixed import
* makefile changes for hooks
* added cleanup secrets
* create manifests
* docs
* update access modes
* update dockerfile
* nit changes
* fixed dockerfile
* rewrite allowing reuse for runners and runnersets
* deepcopy forgot to stage
* changed privileged
* make manifests
* partly moved to finalizer, still need to apply finalizer first
* finalizer added if env variable used in container mode exists
* bump runner version
* error message moved from Error to Info on cleanup pods/secrets
* removed useless dereferencing, added transformation tests of workVolumeClaimTemplate
* Apply suggestions from code review
* Update controllers/utils_test.go
Co-authored-by: Thomas Boop <52323235+thboop@users.noreply.github.com>
* Update controllers/utils_test.go
Co-authored-by: Thomas Boop <52323235+thboop@users.noreply.github.com>
* add hook version to cli, update to 0.1.2
* Apply suggestions from code review
* Update controllers/utils_test.go
* Update runner/Makefile
* Fix missing secret permission and the error handling
* Fix a runnerpod reconciler finalizer to not trigger unnecessary retry
Co-authored-by: Nikola Jokic <nikola-jokic@github.com>
Co-authored-by: Nikola Jokic <97525037+nikola-jokic@users.noreply.github.com>
Co-authored-by: Yusuke Kuoka <ykuoka@gmail.com>
* Make webhook-based scale operation asynchronous
This prevents race condition in the webhook-based autoscaler when it received another webhook event while processing another webhook event and both ended up
scaling up the same horizontal runner autoscaler.
Ref #1321
* Fix typos
* Update rather than Patch HRA to avoid race among webhook-based autoscaler servers
* Batch capacity reservation updates for efficient use of apiserver
* Fix potential never-ending HRA update conflicts in batch update
* Extract batchScaler out of webhook-based autoscaler for testability
* Fix log levels and batch scaler hang on start
* Correlate webhook event with scale trigger amount in logs
* Fix log message
This overhaul turns it into a shellcheck valid script with explicit error handling for all possible situations I could think of. This change takes https://github.com/actions-runner-controller/actions-runner-controller/pull/1409 into account and things can be merged in any order. There are a few important changes here to the logic:
- The wait logic for checking if docker comes up was fundamentally flawed because it checks for the PID. Docker will always come up and thus become visible in the process list, just to immediately die when it encounters an issue, after which supervisor starts it again. This means that our check so far is flaky due to the `sleep 1` it might encounter a PID, or it might not, and the existence of the PID does not mean anything. The `docker ps` check we have in the `entrypoint.sh` script does not suffer from this as it checks for a feature of docker and not a PID. I thus entirely removed the PID check, and instead I am handing things over to our `entrypoint.sh` script by setting the environment variables correctly.
- This change has an influence on the `docker0` interface MTU configuration, because the interface might or might not exist after we started docker. Hence, I changed this to a time boxed loop that tries for one minute to set up the interface's MTU. In case the command fails we log an error and continue with the run.
- I changed the entire MTU handling by validating its value before configuring it, logging an error and continuing without if it is set incorrectly. This ensures that we are not going to send our users on a bug hunt.
- The way we started supervisord did not make much sense to me. It sends itself into the background automatically, there is no need for us to do so with Bash.
The decision to not fail on errors but continue is a deliberate choice, because I believe that running a build is more important than having a perfectly configured system. However, this strategy might also hide issues for all users who are not properly checking their logs. It also makes testing harder. Hence, we could change all error conditions from graceful to panicking. We should then align the exit codes across `startup.sh` and `entrypoint.sh` to ensure that every possible error condition has its own unique error code for easy debugging.
* ci: align pipeline files and setups
* ci: more changes
* ci: various changes
* ci: fix setup-helm action ref
* ci: better pipeline name
* ci: more format aligning
* ci: more format aligning
* ci: better job name
* ci: supports multiple languages
* ci: better pipeline and job names
* ci: do a verb-noun thing for consistency
* ci: use 'arc' when talking holistically
* ci: add caching scope
* ci: put canary in a scope
* ci: fix syntax error
* ci: better pipeline and job names
* ci: better job name
Co-authored-by: toast-gear <toast-gear@users.noreply.github.com>
* Fix example manifests for webhook based scaling
I tried running these on my k8s cluster and I got some easy to fix errors, so I am committing them here.
* Fix example manifests for webhook autoscaling with workflow_jobs
* Fix the explation on how to setup webhooks on your cluster
* Replace unclear comment with actual code examples
There was a comment instructing users to add minReplicas and
maxReplicas to all the HRA yamls, so I just removed it and added
these attributes to the yamls themselves for clarity.
* Make clear that using the ingress example is just a suggestion
* Apply some text improvements suggested by @mumoshu
* Update examples so the webhook server is exposed on a NodePort
Co-authored-by: Yusuke Kuoka <ykuoka@gmail.com>
* Remove an unnecessary field from one the examples
Co-authored-by: Yusuke Kuoka <ykuoka@gmail.com>
* Apply suggestion from @mumoshu
Co-authored-by: Yusuke Kuoka <ykuoka@gmail.com>
* Remove namespace fields from webhook autoscaler examples
This change was suggested by @mumoshu
* Apply final suggestion from @mumoshu
Co-authored-by: Callum Tait <15716903+toast-gear@users.noreply.github.com>
Co-authored-by: Yusuke Kuoka <ykuoka@gmail.com>
A small improvement to our E2E test suite which allows you to set `ARC_E2E_NO_CLEANUP=whatever` to let it prevent the kind cluster cleanup on successful test run, so that you can rerun it without waiting for the new kind cluster to come up.
* doc: Use RunnerSet to retain various cache
In relation to #1286 and as a follow-up for #1340
* docs: clarify client vs daemon
* docs: better wording
* Separate RunnerSet examples for docker iimage layer caching
* Revert changes on testdata as it is going to be added via #1471 instead
* Update README.md
Co-authored-by: Callum Tait <15716903+toast-gear@users.noreply.github.com>
* fixup! Update README.md
* Remove the outdated RunnerSet limitation
Co-authored-by: Callum Tait <15716903+toast-gear@users.noreply.github.com>
This adds the test to verify the runner pod generation logic for the case that you use a generic ephemeral volume as "work".
It is almost an adaptation of the test cases writetn for RunnerSet in #1471, to RunnerDeployment and Runner.
