actions/actions-runner-controller
1
0
Fork 0
mirror of https://github.com/actions/actions-runner-controller.git synced 2025-12-10 19:50:30 +00:00
Code Issues Packages Projects Releases Wiki Activity

fix: force restartPolicy "Never" to prevent runner pods from stucking in Terminating when the container disappeared (#1395)

Browse Source 
Ref #1369
This commit is contained in:
Yusuke Kuoka
2022-05-14 17:07:17 +09:00
committed by GitHub
parent 3014e98681
commit 759349de11
4 changed files with 30 additions and 12 deletions
Download Patch File Download Diff File Expand all files Collapse all files

22
controllers/runner_pod.go Normal file
View File

@@ -0,0 +1,22 @@
package controllers
import corev1 "k8s.io/api/core/v1"
// Force the runner pod managed by either RunnerDeployment and RunnerSet to have restartPolicy=Never.
// See https://github.com/actions-runner-controller/actions-runner-controller/issues/1369 for more context.
//
// This is to prevent runner pods from stucking in Terminating when a K8s node disappeared along with the runnr pod and the runner container within it.
//
// Previously, we used restartPolicy of OnFailure, it turned wrong later, and therefore we now set Never.
//
// When the restartPolicy is OnFailure and the node disappeared, runner pods on the node seem to stuck in state.terminated==nil, state.waiting!=nil, and state.lastTerminationState!=nil,
// and will ever become Running.
// It's probably due to that the node onto which the pods have been scheduled will ever come back, hence the container restart attempt swill ever succeed,
// the pods stuck waiting for successful restarts forever.
//
// By forcing runner pods to never restart, we hope there will be no chances of pods being stuck waiting.
func forceRunnerPodRestartPolicyNever(pod *corev1.Pod) {
if pod.Spec.RestartPolicy != corev1.RestartPolicyNever {
pod.Spec.RestartPolicy = corev1.RestartPolicyNever
}
}