Runner container hooks Beta (#1853)

* Added ability to run Dockerfile.SUFFIX ContainerAction * Extracted IsDockerFile method * reformatted, moved from index to Last() * extracted IsDockerfile to DockerUtil with L0 * added check for IsDockerfile to account for docker:// * updated test to clearly show path/dockerfile:tag * fail if Data.Image is not Dockerfile or docker://[image] * Setup noops for JobPrepare and JobCleanup hooks * Add container jobstarted and jobcomplete hooks * Run 'index.js' instead of specific command hooks * Call jobprepare with command arg * Use right command name (hardcoded) Co-authored-by: Nikola Jokic <nikola-jokic@users.noreply.github.com> * Invoke hooks with arguments * Add PrepareJob hook to work with jobcontainers Co-authored-by: Nikola Jokic <nikola-jokic@users.noreply.github.com> * Rename methods * Use new hookcontainer to run prep and clean hooks * Get path from ENV * Use enums * Use IOUtils.cs * Move container files to folder * Move namespaces * Store "state" between hooks * Remove stdin stream in containerstephosts * Update Constants.cs * Throw if stdin fails * Cleanup obvious nullrefs and unused vars * Cleanup containerhook directory * Call step exec hook * Fix windows build * Remove hook from hookContainer * Rename file * More renamings * Add TODOs * Fix env name * Fix missing imports * Fix imports * Run script step on jobcontainer * Enable feature if env is set * Update ContainerHookManager.cs * Update ContainerHookManager.cs * Hooks allowed to work even when context isn't returned * Custom hooks enabled flag and additional null checks * New line at the end of the FeatureFlagManager.cs * Code refactoring * Supported just in time container building or pulling * Try mock-build for osx * Build all platforms * Run mock on self-hosted * Remove GITHUB prefix * Use ContainerHooksPath instead of CustomHooksPath * Null checks simplified * Code refactoring * Changing condition for image builing/pulling * Code refactoring * TODO comment removed Co-authored-by: Ferenc Hammerl <31069338+fhammerl@users.noreply.github.com> * Call container step if FF is on * Rename run script function * Use JToken instead of dynamic * Add TODO * Small refactoring + renames + TODOs * Throw on DetermineNodeRuntimeVersion * Fix formatting * Add run-container-step * Supported nodeJS in Alpine containers * Renamed Alpine to IsAlpine in HookResponse * Method for checking platform for alpine container * Added container hooks feature flag check * Update IsHookFeatureEnabled with new params * Rename featureflag method * Finish rename * Set collection null values to empty arrays when JSON serialising them * Disable FF until we merge * Update src/Runner.Worker/Container/ContainerHooks/HookContainer.cs * Fix method name * Change hookargs to superclass from interface * Using only Path.Combine in GenerateResponsePath * fix merge error * EntryPointArgs changed to list of args instead of one args string * Changed List to IEnumerable for EntryPointArgs and MountVolumes * Get ContainerRuntimePath for JobContainers from hooks * Read ContainerEnv from response file * Port mappings saved after creating services * Support case when responseFile doesn't exist * Check if response file exists * Logging in ExecuteHookScript * Save hook state after all 4 hooks * Code refactoring * Remove TODO Co-authored-by: Ferenc Hammerl <31069338+fhammerl@users.noreply.github.com> * Remove second TODO Co-authored-by: Ferenc Hammerl <31069338+fhammerl@users.noreply.github.com> * Removing container env changes * Removing containerEnv and dockerManager * Delete mock-build.yml * Update IOUtil.cs * Add comment about containerhooks * Fix merge mistake * Remove solved todo * Determine which shell to use for hooks scenario * Overload for method ExecuteHookScript with prependPath as arg * Adding HostContext to the GetDefaultShellForScript call * prependPath as a mandatory parameter * Improve logging for hooks * Small changes in logging * Allow null for ContainerEntryPointArgs * Changed log messages * Skip setting EntryPoint and EntryPointArgs if hooks are enabled * Throw if IsAlpine is null in PrepareJob * Code refactoring - added GetAndValidateResponse method * Code refactoring * Changes in exception message * Only save hookState if returned * Use FF from server * Empty line * Code refactoring Co-authored-by: Ferenc Hammerl <31069338+fhammerl@users.noreply.github.com> * Send null instead of string empty * Remove TODO * Code refactoring and some small changes * Allow Globals to be null to pass L0 * Fix setup in StepHostL0 * Throw exception earlier if response file doesn't exist and prepare_job hook is running * Refactoring GetResponse method * Changing exception message if response file is not found Co-authored-by: Ferenc Hammerl <31069338+fhammerl@users.noreply.github.com> * Chaning exception message if isAlpine is null for prepare_job hook * Rename hook folder * Fail if compatible hookfile not found * Use .Value instead of casting bool? to bool * Format spacing * Formatting * User user and system mvs * Use variables instead of entire context in featuremanager * Update stepTelemetry if step uses containerhooks * Restore import * Remove unneccessary field from HookContainer * Refactor response context and portmappings * Force allow hooks if FF is on * Code refactoring * Revert deleting usings * Better hookContainer defaults and use correct portmapping list * Make GetDefaultShellForScript a HostContext extension method * Generic hookresponse * Code refactoring, unnecessary properties removed - HookContainer moved to the HookInput.cs * Remove empty line * Code refactoring and better exception handling * code refactor, removing unnecessary props * Move hookstate to global ContainerHookState * Trace exception before we throw it for not losing information * Fix for null ref exception in GetResponse * Adding additional check for null response in prepareJob hook * Refactoring GetResponse with additional check * Update error messages * Ports in ResponseContainer changed from IList to IDictionary * Fix port format * Include dockerfile * Send null Registry obj if there's nothing in it * Minor formatting * Check if hookIndexPath exists relocated to the ContainerHookManager * Code refactoring - ValidateHookExecutable added to the ContainerHookManager * check if ContainerHooksPath when AllowRunnerContainerHooks is on * Submit JSON telemetry instead of boolean * Prefix step hooks with "run" * Rename FeatureManager * Fix flipped condition * Unify js shell path getter with ps1 and sh getter * Validate on run, not on instantiation of manager * Cleanup ExecuteAsync methods * Handle exception in executeHookScript * Better exception types * Remove comment * Simplify boolean * Allow jobs without jobContainer to run * Use JObject instead of JToken * Use correct Response type * Format class to move cleanupJobAsync to the end of public methods * Rename HookIndexPath to HookScriptPath * Refactor methods into expression bodies * Fix args class hierarchy * Fix argument order * Formatting * Fix nullref and don't swallow stacktrace * Whilelist HookArgs * Use FF in FeatureManager * Update src/Runner.Worker/ContainerOperationProvider.cs Co-authored-by: Tingluo Huang <tingluohuang@github.com> * Update src/Runner.Worker/ActionRunner.cs Co-authored-by: Thomas Boop <52323235+thboop@users.noreply.github.com> * Update src/Runner.Worker/ActionRunner.cs Co-authored-by: Thomas Boop <52323235+thboop@users.noreply.github.com> * Only mount well known dirs to job containers * Get trace from hostcontext * Use hook execution for setting telemetry Co-authored-by: Nikola Jokic <nikola.jokic@akvelon.com> Co-authored-by: Nikola Jokic <nikola-jokic@users.noreply.github.com> Co-authored-by: Nikola Jokic <97525037+nikola-jokic@users.noreply.github.com> Co-authored-by: Stefan Ruvceski <stefan.ruvceski@akvelon.com> Co-authored-by: ruvceskistefan <96768603+ruvceskistefan@users.noreply.github.com> Co-authored-by: Thomas Boop <thboop@github.com> Co-authored-by: stefanruvceski <ruvceskistefan@github.com> Co-authored-by: Tingluo Huang <tingluohuang@github.com> Co-authored-by: Thomas Boop <52323235+thboop@users.noreply.github.com>
2025-12-14 04:53:34 +00:00 · 2022-06-10 15:51:20 +02:00
parent ac7b34a071
commit 591f8c3510
24 changed files with 800 additions and 207 deletions
--- a/src/Runner.Worker/Handlers/StepHost.cs
+++ b/src/Runner.Worker/Handlers/StepHost.cs
@@ -1,5 +1,6 @@
 using System;
 using System.Collections.Generic;
+using GitHub.DistributedTask.Pipelines.ContextData;
 using System.Text;
 using System.Threading;
 using System.Threading.Tasks;
@@ -7,7 +8,9 @@ using GitHub.Runner.Worker.Container;
 using GitHub.Runner.Common;
 using GitHub.Runner.Sdk;
 using System.Linq;
-using GitHub.DistributedTask.Pipelines.ContextData;
+using GitHub.Runner.Worker.Container.ContainerHooks;
+using System.IO;
+using System.Threading.Channels;

 namespace GitHub.Runner.Worker.Handlers
 {
@@ -16,11 +19,12 @@ namespace GitHub.Runner.Worker.Handlers
        event EventHandler<ProcessDataReceivedEventArgs> OutputDataReceived;
        event EventHandler<ProcessDataReceivedEventArgs> ErrorDataReceived;

-        string ResolvePathForStepHost(string path);
+        string ResolvePathForStepHost(IExecutionContext executionContext, string path);

        Task<string> DetermineNodeRuntimeVersion(IExecutionContext executionContext, string preferredVersion);

-        Task<int> ExecuteAsync(string workingDirectory,
+        Task<int> ExecuteAsync(IExecutionContext context,
+                               string workingDirectory,
                               string fileName,
                               string arguments,
                               IDictionary<string, string> environment,
@@ -28,6 +32,7 @@ namespace GitHub.Runner.Worker.Handlers
                               Encoding outputEncoding,
                               bool killProcessOnCancel,
                               bool inheritConsoleHandler,
+                               string standardInInput,
                               CancellationToken cancellationToken);
    }

@@ -48,7 +53,7 @@ namespace GitHub.Runner.Worker.Handlers
        public event EventHandler<ProcessDataReceivedEventArgs> OutputDataReceived;
        public event EventHandler<ProcessDataReceivedEventArgs> ErrorDataReceived;

-        public string ResolvePathForStepHost(string path)
+        public string ResolvePathForStepHost(IExecutionContext executionContext, string path)
        {
            return path;
        }
@@ -58,7 +63,8 @@ namespace GitHub.Runner.Worker.Handlers
            return Task.FromResult<string>(preferredVersion);
        }

-        public async Task<int> ExecuteAsync(string workingDirectory,
+        public async Task<int> ExecuteAsync(IExecutionContext context,
+                                            string workingDirectory,
                                            string fileName,
                                            string arguments,
                                            IDictionary<string, string> environment,
@@ -66,10 +72,17 @@ namespace GitHub.Runner.Worker.Handlers
                                            Encoding outputEncoding,
                                            bool killProcessOnCancel,
                                            bool inheritConsoleHandler,
+                                            string standardInInput,
                                            CancellationToken cancellationToken)
        {
            using (var processInvoker = HostContext.CreateService<IProcessInvoker>())
            {
+                Channel<string> redirectStandardIn = null;
+                if (standardInInput != null)
+                {
+                    redirectStandardIn = Channel.CreateUnbounded<string>(new UnboundedChannelOptions() { SingleReader = true, SingleWriter = true });
+                    redirectStandardIn.Writer.TryWrite(standardInInput);
+                }
                processInvoker.OutputDataReceived += OutputDataReceived;
                processInvoker.ErrorDataReceived += ErrorDataReceived;

@@ -80,7 +93,7 @@ namespace GitHub.Runner.Worker.Handlers
                                                         requireExitCodeZero: requireExitCodeZero,
                                                         outputEncoding: outputEncoding,
                                                         killProcessOnCancel: killProcessOnCancel,
-                                                         redirectStandardIn: null,
+                                                         redirectStandardIn: redirectStandardIn,
                                                         inheritConsoleHandler: inheritConsoleHandler,
                                                         cancellationToken: cancellationToken);
            }
@@ -94,11 +107,15 @@ namespace GitHub.Runner.Worker.Handlers
        public event EventHandler<ProcessDataReceivedEventArgs> OutputDataReceived;
        public event EventHandler<ProcessDataReceivedEventArgs> ErrorDataReceived;

-        public string ResolvePathForStepHost(string path)
+        public string ResolvePathForStepHost(IExecutionContext executionContext, string path)
        {
            // make sure container exist.
            ArgUtil.NotNull(Container, nameof(Container));
-            ArgUtil.NotNullOrEmpty(Container.ContainerId, nameof(Container.ContainerId));
+            if (!FeatureManager.IsContainerHooksEnabled(executionContext.Global?.Variables))
+            {
+                // TODO: Remove nullcheck with executionContext.Global? by setting up ExecutionContext.Global at GitHub.Runner.Common.Tests.Worker.ExecutionContextL0.GetExpressionValues_ContainerStepHost
+                ArgUtil.NotNullOrEmpty(Container.ContainerId, nameof(Container.ContainerId));
+            }

            // remove double quotes around the path
            path = path.Trim('\"');
@@ -120,6 +137,19 @@ namespace GitHub.Runner.Worker.Handlers

        public async Task<string> DetermineNodeRuntimeVersion(IExecutionContext executionContext, string preferredVersion)
        {
+            // Optimistically use the default
+            string nodeExternal = preferredVersion;
+
+            if (FeatureManager.IsContainerHooksEnabled(executionContext.Global.Variables))
+            {
+                if (Container.IsAlpine)
+                {
+                    nodeExternal = CheckPlatformForAlpineContainer(executionContext, preferredVersion);
+                }
+                executionContext.Debug($"Running JavaScript Action with default external tool: {nodeExternal}");
+                return nodeExternal;
+            }
+
            // Best effort to determine a compatible node runtime
            // There may be more variation in which libraries are linked than just musl/glibc,
            // so determine based on known distribtutions instead
@@ -128,7 +158,6 @@ namespace GitHub.Runner.Worker.Handlers

            var output = new List<string>();
            var execExitCode = await dockerManager.DockerExec(executionContext, Container.ContainerId, string.Empty, osReleaseIdCmd, output);
-            string nodeExternal;
            if (execExitCode == 0)
            {
                foreach (var line in output)
@@ -136,26 +165,17 @@ namespace GitHub.Runner.Worker.Handlers
                    executionContext.Debug(line);
                    if (line.ToLower().Contains("alpine"))
                    {
-                        if (!Constants.Runner.PlatformArchitecture.Equals(Constants.Architecture.X64))
-                        {
-                            var os = Constants.Runner.Platform.ToString();
-                            var arch = Constants.Runner.PlatformArchitecture.ToString();
-                            var msg = $"JavaScript Actions in Alpine containers are only supported on x64 Linux runners. Detected {os} {arch}";
-                            throw new NotSupportedException(msg);
-                        }
-                        nodeExternal = $"{preferredVersion}_alpine";
-                        executionContext.Debug($"Container distribution is alpine. Running JavaScript Action with external tool: {nodeExternal}");
+                        nodeExternal = CheckPlatformForAlpineContainer(executionContext, preferredVersion);
                        return nodeExternal;
                    }
                }
            }
-            // Optimistically use the default
-            nodeExternal = preferredVersion;
            executionContext.Debug($"Running JavaScript Action with default external tool: {nodeExternal}");
            return nodeExternal;
        }

-        public async Task<int> ExecuteAsync(string workingDirectory,
+        public async Task<int> ExecuteAsync(IExecutionContext context,
+                                            string workingDirectory,
                                            string fileName,
                                            string arguments,
                                            IDictionary<string, string> environment,
@@ -163,12 +183,25 @@ namespace GitHub.Runner.Worker.Handlers
                                            Encoding outputEncoding,
                                            bool killProcessOnCancel,
                                            bool inheritConsoleHandler,
+                                            string standardInInput,
                                            CancellationToken cancellationToken)
        {
-            // make sure container exist.
            ArgUtil.NotNull(Container, nameof(Container));
-            ArgUtil.NotNullOrEmpty(Container.ContainerId, nameof(Container.ContainerId));
+            var containerHookManager = HostContext.GetService<IContainerHookManager>();
+            if (FeatureManager.IsContainerHooksEnabled(context.Global.Variables))
+            {
+                TranslateToContainerPath(environment);
+                await containerHookManager.RunScriptStepAsync(context,
+                                                                                   Container,
+                                                                                   workingDirectory,                                                                                   
+                                                                                   fileName,
+                                                                                   arguments,
+                                                                                   environment,
+                                                                                   PrependPath);
+                return (int)(context.Result ?? 0);
+            }

+            ArgUtil.NotNullOrEmpty(Container.ContainerId, nameof(Container.ContainerId));
            var dockerManager = HostContext.GetService<IDockerCommandManager>();
            string dockerClientPath = dockerManager.DockerPath;

@@ -202,12 +235,7 @@ namespace GitHub.Runner.Worker.Handlers
            dockerCommandArgs.Add(arguments);

            string dockerCommandArgstring = string.Join(" ", dockerCommandArgs);
-
-            // make sure all env are using container path
-            foreach (var envKey in environment.Keys.ToList())
-            {
-                environment[envKey] = this.Container.TranslateToContainerPath(environment[envKey]);
-            }
+            TranslateToContainerPath(environment);

            using (var processInvoker = HostContext.CreateService<IProcessInvoker>())
            {
@@ -221,7 +249,6 @@ namespace GitHub.Runner.Worker.Handlers
                // Let .NET choose the default.
                outputEncoding = null;
 #endif
-
                return await processInvoker.ExecuteAsync(workingDirectory: HostContext.GetDirectory(WellKnownDirectory.Work),
                                                         fileName: dockerClientPath,
                                                         arguments: dockerCommandArgstring,
@@ -234,5 +261,28 @@ namespace GitHub.Runner.Worker.Handlers
                                                         cancellationToken: cancellationToken);
            }
        }
+
+        private string CheckPlatformForAlpineContainer(IExecutionContext executionContext, string preferredVersion)
+        {
+            string nodeExternal = preferredVersion;
+            if (!Constants.Runner.PlatformArchitecture.Equals(Constants.Architecture.X64))
+            {
+                var os = Constants.Runner.Platform.ToString();
+                var arch = Constants.Runner.PlatformArchitecture.ToString();
+                var msg = $"JavaScript Actions in Alpine containers are only supported on x64 Linux runners. Detected {os} {arch}";
+                throw new NotSupportedException(msg);
+            }
+            nodeExternal = $"{preferredVersion}_alpine";
+            executionContext.Debug($"Container distribution is alpine. Running JavaScript Action with external tool: {nodeExternal}");
+            return nodeExternal;
+        }
+
+        private void TranslateToContainerPath(IDictionary<string, string> environment)
+        {
+            foreach (var envKey in environment.Keys.ToList())
+            {
+                environment[envKey] = this.Container.TranslateToContainerPath(environment[envKey]);
+            }
+        }
    }
 }