Compare commits

...

12 Commits

Author SHA1 Message Date
Thomas Boop
ed15c5389b Update releaseVersion 2021-08-03 11:30:11 -04:00
Thomas Boop
5af7b87074 Release notes for runner release 2.290.0 (#1237) 2021-08-03 11:12:43 -04:00
Tingluo Huang
110eb3a5de Add generateIdTokenUrl to env vars for actions. (#1234) 2021-08-02 14:47:50 -07:00
Tingluo Huang
bd1341e580 Print out resolved SHA for each action. (#1233) 2021-08-02 15:59:09 -04:00
Thomas Boop
85ce33b1d3 Composite Code Cleanup (#1232)
* composite polish

* Cleanup Condition Handling

* Refactor ConditionTraceWriter

* pr feedback

* cleanup
2021-08-02 14:57:25 -04:00
Thomas Boop
92ec3d0f29 Add better step telemetry and tracing for composite Actions (#1229)
* Add Step Telemetry

* better telemetry and tracing

* cleanup
2021-07-30 10:45:49 -04:00
Thomas Boop
4e95d0d6ad Support pre/post/container/composite actions within composite actions (#1222)
Support Composite Actions with uses: steps
2021-07-28 15:35:21 -04:00
Thomas Boop
5281434f3f Composite Actions Support ADR (#1144)
Composite Actions ADR
2021-07-27 09:29:50 -04:00
Ferenc Hammerl
e9a8bf29df Prefer higher libicu versions in installDependencies.sh (#1228)
* Update libicu dependencies

* Remove redundant comments (methodname is enough)
2021-07-27 14:30:53 +02:00
dependabot[bot]
a65331e887 Bump lodash in /src/Misc/expressionFunc/hashFiles (#1082)
Bumps [lodash](https://github.com/lodash/lodash) from 4.17.19 to 4.17.21.
- [Release notes](https://github.com/lodash/lodash/releases)
- [Commits](https://github.com/lodash/lodash/compare/4.17.19...4.17.21)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-07-26 23:34:52 -04:00
dependabot[bot]
908a082527 Bump hosted-git-info in /src/Misc/expressionFunc/hashFiles (#1087)
Bumps [hosted-git-info](https://github.com/npm/hosted-git-info) from 2.8.8 to 2.8.9.
- [Release notes](https://github.com/npm/hosted-git-info/releases)
- [Changelog](https://github.com/npm/hosted-git-info/blob/v2.8.9/CHANGELOG.md)
- [Commits](https://github.com/npm/hosted-git-info/compare/v2.8.8...v2.8.9)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-07-26 23:33:21 -04:00
dependabot[bot]
10ba74f59b Bump glob-parent in /src/Misc/expressionFunc/hashFiles (#1147)
Bumps [glob-parent](https://github.com/gulpjs/glob-parent) from 5.1.1 to 5.1.2.
- [Release notes](https://github.com/gulpjs/glob-parent/releases)
- [Changelog](https://github.com/gulpjs/glob-parent/blob/main/CHANGELOG.md)
- [Commits](https://github.com/gulpjs/glob-parent/compare/v5.1.1...v5.1.2)

---
updated-dependencies:
- dependency-name: glob-parent
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-07-26 22:55:26 -04:00
26 changed files with 741 additions and 160 deletions

View File

@@ -0,0 +1,92 @@
**Date**: 2021-06-10
**Status**: Accepted
## Context
We released [composite run steps](https://github.com/actions/runner/pull/554) last year which started our journey of reusing steps across different workflow files. To continue that journey, we want to expand composite run steps into composite actions.
We want to support the `uses` steps from workflows in composite actions, including:
- Container actions
- Javascript actions
- Other Composite actions (up to a limit of course!)
- The pre and post steps these actions can generate
## Guiding Principles
- Composite Actions should function as a single step or action, no matter how many steps it is composed of or how many levels of recursion it has
- In the future we may add a configurable option to make this no longer the case
- A workflow author should not need to understand the inner workings of a composite action in order to use it
- Composite actions should leverage inputs to get values they need, they will not have full access to the `context` objects. The secrets context will **not** be available to composite actions, users will need to pass these values in as an input.
- Other Actions should **just work** inside a composite action, without any code changes
## Decisions
### Composite Recursion Limit
- We will start with supporting a recursion limit of `10` composite actions deep
- We are free to bump this limit in the future, the code will be written to just require updating a variable. If the graph evaluates beyond the recursion limit, the job will fail in the pre-job phase (The `Set up job` step).
- A composite actions interface is its inputs and outputs, nothing else is carried over when invoking recursively.
### Pre/Post Steps in nested Actions
- We do not plan on adding the ability to configure a customizable pre or post step for composite actions at this time. However, we will execute the pre and post steps of any actions referenced in a composite action.
- Composite actions will generate a single pre-step and post-step for the entire composite action, even if there are multiple pre-steps and post-steps in the referenced actions.
- These steps will execute following the same ordering rules we have today, first to run has their pre step run first and their post step run last.
- For example, if you had a composite action with two pre steps and two posts steps:
```
- uses: action1
- uses: composite1
- uses: action2
```
The order of execution would be:
```
- prestep-action1
- prestep-composite1
- prestep-composite1-first-action-referenced
- prestep-composite1-second-action-referenced
- prestep-action2
- the job steps
- poststep-action2
- poststep-composite1
- poststep-composite1-the-second-action-referenced
- poststep-composite1-first-action-referenced
- poststep-action1
```
#### Set-state
- While the composite action has an individual combined pre/post action, the `set-state` command will not be shared.
- If the `set-state` command is used during a composite step, only the action that originally called `set-state` will have access to the env variable during the post run step.
- This prevents multiple actions that set the same state from interfering with the execution of another action's post step.
### Resolve Action Endpoint changes
- The resolve actions endpoint will now validate policy to ensure that the given workflow run has access to download that action.
- Older GHES/GHAE customers with newer runners will be locked out of composite uses steps until they upgrade their instance.
### Local actions
- Local actions will expand the tree, perform policy checks, and download actions Just in Time when the step is running.
- Like current local actions, we will not support presteps. If an action is running local, by the time we know that, the time to run presteps have already passed.
### If, continue-on-error, timeout-minutes - Not being considered at this time
- `if`, `continue-on-error`, `timeout-minutes` could be supported in composite run/uses steps. These values were not originally supported in our composite run steps implementation.
- Browsing the community forums and runner repo, there hasn't been a lot of noise asking for these features, so we will hold off on them.
- These values passed as input into the composite action will **not** be carried over as input into the individual steps the composite action runs.
### Defaults - Not being considered at this time
- In actions, we have the idea of [defaults](https://docs.github.com/en/actions/reference/workflow-syntax-for-github-actions#defaultsrun) , which allow you to specify a shell and working directory in one location, rather then on each step.
- However, `shell` is currently required in composite run steps
- In regular run steps, it is optional, and defaults to a different value based on the OS.
- We want to prioritize the right experience for the consumer, and make the action author continue to explicitly set these values. We can consider improving this experience in the future.
## Consequences
- Workflows are now more reusable across multiple workflow files
- Composite actions implement most of the existing workflow run steps, with room to expand these in the future
- Feature flags will control this rollout

View File

@@ -1,22 +1,17 @@
## Features
- Add Job Message size to both Worker and Listener logs for debugging (#1100)
- Add notice annotation level (in addition to error and warning) and support more annotation fields (#1175)
- Adds support for composite actions if the server supports it (#1222)
- Adds `generateIdTokenUri` to env variables for actions (#1234)
## Bugs
- Remove the `NODE_ICU_DATA` environment variable that may cause conflicts with node within the runner. (#1060)
- Handle cancelled jobs better to prevent orphaned processes (#1083)
- No longer fail to remove a `systemd` service with `svc.sh uninstall` if the script had previously been run from the wrong location (#1135)
- Send `SIGKILL` to the runner listener if it doesn't respond to `SIGINT` for 30 seconds
- Match runner group name when configuring even if there's only a single runner group
- Prefer higher `libicu` versions in `installDependencies.sh` (#1228)
## Misc
- Fix automation links in documentation (#1089)
- Improve developer and first contributor experience by improving tooling for VS Code (#1101, #1117, #1119, #1132)
- Fix bug where linux users are not able to run remove-svc.sh as root (#1127)
- Send step telemetry to server on JobCompletion (#1229)
- Print out the resolved SHA for each downloaded action (#1233)
## Windows x64
We recommend configuring the runner in a root folder of the Windows drive (e.g. "C:\actions-runner"). This will help avoid issues related to service identity folder permissions and long file path restrictions on Windows.

View File

@@ -1 +1 @@
<Update to ./src/runnerversion when creating release>
2.280.0

View File

@@ -1291,9 +1291,9 @@
}
},
"glob-parent": {
"version": "5.1.1",
"resolved": "https://registry.npmjs.org/glob-parent/-/glob-parent-5.1.1.tgz",
"integrity": "sha512-FnI+VGOpnlGHWZxthPGR+QhR78fuiK0sNLkHQv+bL9fQi57lNNdquIbna/WrfROrolq8GK5Ek6BiMwqL/voRYQ==",
"version": "5.1.2",
"resolved": "https://registry.npmjs.org/glob-parent/-/glob-parent-5.1.2.tgz",
"integrity": "sha512-AOIgSQCepiJYwP3ARnGx+5VnTu2HBYdzbGP45eLw1vr3zB3vZLeyed1sC9hnbcOc9/SrMyM5RPQrkGz4aS9Zow==",
"dev": true,
"requires": {
"is-glob": "^4.0.1"
@@ -1374,9 +1374,9 @@
"dev": true
},
"hosted-git-info": {
"version": "2.8.8",
"resolved": "https://registry.npmjs.org/hosted-git-info/-/hosted-git-info-2.8.8.tgz",
"integrity": "sha512-f/wzC2QaWBs7t9IYqB4T3sR1xviIViXJRJTWBlx2Gf3g0Xi5vI7Yy4koXQ1c9OYDGHN9sBy1DQ2AB8fqZBWhUg==",
"version": "2.8.9",
"resolved": "https://registry.npmjs.org/hosted-git-info/-/hosted-git-info-2.8.9.tgz",
"integrity": "sha512-mxIDAb9Lsm6DoOJ7xH+5+X4y1LU/4Hi50L9C5sIswK3JzULS4bwk1FvjdBgvYR4bzT4tuUQiC15FE2f5HbLvYw==",
"dev": true
},
"iconv-lite": {
@@ -1683,9 +1683,9 @@
}
},
"lodash": {
"version": "4.17.19",
"resolved": "https://registry.npmjs.org/lodash/-/lodash-4.17.19.tgz",
"integrity": "sha512-JNvd8XER9GQX0v2qJgsaN/mzFCNA5BRe/j8JN9d+tWyGLSodKQHKFicdwNYzWwI3wjRnaKPsGj1XkBjx/F96DQ==",
"version": "4.17.21",
"resolved": "https://registry.npmjs.org/lodash/-/lodash-4.17.21.tgz",
"integrity": "sha512-v2kDEe57lecTulaDIuNTPy3Ry4gLGJ6Z1O3vE1krgXZNrsQ+LFTGHVxVjcXPs17LhbZVGedAJv8XZ1tvj5FvSg==",
"dev": true
},
"lodash.unescape": {

View File

@@ -94,7 +94,6 @@ then
fi
}
# libssl version prefer: libssl1.1 -> libssl1.0.2 -> libssl1.0.0
apt_get_with_fallbacks libssl1.1$ libssl1.0.2$ libssl1.0.0$
if [ $? -ne 0 ]
then
@@ -103,8 +102,7 @@ then
exit 1
fi
# libicu version prefer: libicu66 -> libicu63 -> libicu60 -> libicu57 -> libicu55 -> libicu52
apt_get_with_fallbacks libicu66 libicu63 libicu60 libicu57 libicu55 libicu52
apt_get_with_fallbacks libicu72 libicu71 libicu70 libicu69 libicu68 libicu67 libicu66 libicu65 libicu63 libicu60 libicu57 libicu55 libicu52
if [ $? -ne 0 ]
then
echo "'$apt_get' failed with exit code '$?'"

View File

@@ -186,7 +186,7 @@ namespace GitHub.Runner.Listener.Configuration
}
else
{
Trace.Info("Found a self-hosted runner group with id {1} and name {2}", agentPool.Id, agentPool.Name);
Trace.Info($"Found a self-hosted runner group with id {agentPool.Id} and name {agentPool.Name}");
runnerSettings.PoolId = agentPool.Id;
runnerSettings.PoolName = agentPool.Name;
}

View File

@@ -295,8 +295,21 @@ namespace GitHub.Runner.Worker
{
throw new Exception("Required field 'name' is missing in ##[save-state] command.");
}
context.IntraActionState[stateName] = command.Data;
// Embedded steps (composite) keep track of the state at the root level
if (context.IsEmbedded)
{
var id = context.EmbeddedId;
if (!context.Root.EmbeddedIntraActionState.ContainsKey(id))
{
context.Root.EmbeddedIntraActionState[id] = new Dictionary<string, string>(StringComparer.OrdinalIgnoreCase);
}
context.Root.EmbeddedIntraActionState[id][stateName] = command.Data;
}
// Otherwise modify the ExecutionContext
else
{
context.IntraActionState[stateName] = command.Data;
}
context.Debug($"Save intra-action state {stateName} = {command.Data}");
}

View File

@@ -37,7 +37,10 @@ namespace GitHub.Runner.Worker
public interface IActionManager : IRunnerService
{
Dictionary<Guid, ContainerInfo> CachedActionContainers { get; }
Task<PrepareResult> PrepareActionsAsync(IExecutionContext executionContext, IEnumerable<Pipelines.JobStep> steps);
Dictionary<Guid, List<Pipelines.ActionStep>> CachedEmbeddedPreSteps { get; }
Dictionary<Guid, List<Guid>> CachedEmbeddedStepIds { get; }
Dictionary<Guid, Stack<Pipelines.ActionStep>> CachedEmbeddedPostSteps { get; }
Task<PrepareResult> PrepareActionsAsync(IExecutionContext executionContext, IEnumerable<Pipelines.JobStep> steps, Guid rootStepId = default(Guid));
Definition LoadAction(IExecutionContext executionContext, Pipelines.ActionStep action);
}
@@ -48,10 +51,20 @@ namespace GitHub.Runner.Worker
//81920 is the default used by System.IO.Stream.CopyTo and is under the large object heap threshold (85k).
private const int _defaultCopyBufferSize = 81920;
private const string _dotcomApiUrl = "https://api.github.com";
private readonly Dictionary<Guid, ContainerInfo> _cachedActionContainers = new Dictionary<Guid, ContainerInfo>();
private readonly Dictionary<Guid, ContainerInfo> _cachedActionContainers = new Dictionary<Guid, ContainerInfo>();
public Dictionary<Guid, ContainerInfo> CachedActionContainers => _cachedActionContainers;
public async Task<PrepareResult> PrepareActionsAsync(IExecutionContext executionContext, IEnumerable<Pipelines.JobStep> steps)
private readonly Dictionary<Guid, List<Pipelines.ActionStep>> _cachedEmbeddedPreSteps = new Dictionary<Guid, List<Pipelines.ActionStep>>();
public Dictionary<Guid, List<Pipelines.ActionStep>> CachedEmbeddedPreSteps => _cachedEmbeddedPreSteps;
private readonly Dictionary<Guid, List<Guid>> _cachedEmbeddedStepIds = new Dictionary<Guid, List<Guid>>();
public Dictionary<Guid, List<Guid>> CachedEmbeddedStepIds => _cachedEmbeddedStepIds;
private readonly Dictionary<Guid, Stack<Pipelines.ActionStep>> _cachedEmbeddedPostSteps = new Dictionary<Guid, Stack<Pipelines.ActionStep>>();
public Dictionary<Guid, Stack<Pipelines.ActionStep>> CachedEmbeddedPostSteps => _cachedEmbeddedPostSteps;
public async Task<PrepareResult> PrepareActionsAsync(IExecutionContext executionContext, IEnumerable<Pipelines.JobStep> steps, Guid rootStepId = default(Guid))
{
// Assert inputs
ArgUtil.NotNull(executionContext, nameof(executionContext));
@@ -64,10 +77,30 @@ namespace GitHub.Runner.Worker
PreStepTracker = new Dictionary<Guid, IActionRunner>()
};
var containerSetupSteps = new List<JobExtensionRunner>();
IOUtil.DeleteDirectory(HostContext.GetDirectory(WellKnownDirectory.Actions), executionContext.CancellationToken);
var depth = 0;
// We are running at the start of a job
if (rootStepId == default(Guid))
{
IOUtil.DeleteDirectory(HostContext.GetDirectory(WellKnownDirectory.Actions), executionContext.CancellationToken);
}
// We are running mid job due to a local composite action
else
{
if (!_cachedEmbeddedStepIds.ContainsKey(rootStepId))
{
_cachedEmbeddedStepIds[rootStepId] = new List<Guid>();
foreach (var compositeStep in steps)
{
var guid = Guid.NewGuid();
compositeStep.Id = guid;
_cachedEmbeddedStepIds[rootStepId].Add(guid);
}
}
depth = 1;
}
IEnumerable<Pipelines.ActionStep> actions = steps.OfType<Pipelines.ActionStep>();
executionContext.Output("Prepare all required actions");
var result = await PrepareActionsRecursiveAsync(executionContext, state, actions, 0);
var result = await PrepareActionsRecursiveAsync(executionContext, state, actions, depth, rootStepId);
if (state.ImagesToPull.Count > 0)
{
foreach (var imageToPull in result.ImagesToPull)
@@ -103,7 +136,7 @@ namespace GitHub.Runner.Worker
return new PrepareResult(containerSetupSteps, result.PreStepTracker);
}
private async Task<PrepareActionsState> PrepareActionsRecursiveAsync(IExecutionContext executionContext, PrepareActionsState state, IEnumerable<Pipelines.ActionStep> actions, Int32 depth = 0)
private async Task<PrepareActionsState> PrepareActionsRecursiveAsync(IExecutionContext executionContext, PrepareActionsState state, IEnumerable<Pipelines.ActionStep> actions, Int32 depth = 0, Guid parentStepId = default(Guid))
{
ArgUtil.NotNull(executionContext, nameof(executionContext));
if (depth > Constants.CompositeActionsMaxDepth)
@@ -187,24 +220,51 @@ namespace GitHub.Runner.Worker
state.ImagesToBuildInfo[setupInfo.Container.ActionRepository] = setupInfo.Container;
}
}
else if(setupInfo != null && setupInfo.Steps != null && setupInfo.Steps.Count > 0)
else if (setupInfo != null && setupInfo.Steps != null && setupInfo.Steps.Count > 0)
{
state = await PrepareActionsRecursiveAsync(executionContext, state, setupInfo.Steps, depth + 1);
state = await PrepareActionsRecursiveAsync(executionContext, state, setupInfo.Steps, depth + 1, action.Id);
}
var repoAction = action.Reference as Pipelines.RepositoryPathReference;
if (repoAction.RepositoryType != Pipelines.PipelineConstants.SelfAlias)
{
var definition = LoadAction(executionContext, action);
// TODO: Support pre's in composite actions
if (definition.Data.Execution.HasPre && depth < 1)
if (definition.Data.Execution.HasPre)
{
var actionRunner = HostContext.CreateService<IActionRunner>();
actionRunner.Action = action;
actionRunner.Stage = ActionRunStage.Pre;
actionRunner.Condition = definition.Data.Execution.InitCondition;
Trace.Info($"Add 'pre' execution for {action.Id}");
state.PreStepTracker[action.Id] = actionRunner;
// Root Step
if (depth < 1)
{
var actionRunner = HostContext.CreateService<IActionRunner>();
actionRunner.Action = action;
actionRunner.Stage = ActionRunStage.Pre;
actionRunner.Condition = definition.Data.Execution.InitCondition;
state.PreStepTracker[action.Id] = actionRunner;
}
// Embedded Step
else
{
if (!_cachedEmbeddedPreSteps.ContainsKey(parentStepId))
{
_cachedEmbeddedPreSteps[parentStepId] = new List<Pipelines.ActionStep>();
}
// Clone action so we can modify the condition without affecting the original
var clonedAction = action.Clone() as Pipelines.ActionStep;
clonedAction.Condition = definition.Data.Execution.InitCondition;
_cachedEmbeddedPreSteps[parentStepId].Add(clonedAction);
}
}
if (definition.Data.Execution.HasPost && depth > 0)
{
if (!_cachedEmbeddedPostSteps.ContainsKey(parentStepId))
{
// If we haven't done so already, add the parent to the post steps
_cachedEmbeddedPostSteps[parentStepId] = new Stack<Pipelines.ActionStep>();
}
// Clone action so we can modify the condition without affecting the original
var clonedAction = action.Clone() as Pipelines.ActionStep;
clonedAction.Condition = definition.Data.Execution.CleanupCondition;
_cachedEmbeddedPostSteps[parentStepId].Push(clonedAction);
}
}
}
@@ -355,6 +415,29 @@ namespace GitHub.Runner.Worker
Trace.Verbose($"Details: {StringUtil.ConvertToJson(compositeAction?.Steps)}");
Trace.Info($"Load: {compositeAction.Outputs?.Count ?? 0} number of outputs");
Trace.Info($"Details: {StringUtil.ConvertToJson(compositeAction?.Outputs)}");
if (CachedEmbeddedPreSteps.TryGetValue(action.Id, out var preSteps))
{
compositeAction.PreSteps = preSteps;
}
if (CachedEmbeddedPostSteps.TryGetValue(action.Id, out var postSteps))
{
compositeAction.PostSteps = postSteps;
}
if (_cachedEmbeddedStepIds.ContainsKey(action.Id))
{
for (var i = 0; i < compositeAction.Steps.Count; i++)
{
// Store Id's for later load actions
compositeAction.Steps[i].Id = _cachedEmbeddedStepIds[action.Id][i];
if (string.IsNullOrEmpty(executionContext.Global.Variables.Get("DistributedTask.EnableCompositeActions")) && compositeAction.Steps[i].Reference.Type != Pipelines.ActionSourceType.Script)
{
throw new Exception("`uses:` keyword is not currently supported.");
}
}
}
}
else
{
@@ -620,7 +703,7 @@ namespace GitHub.Runner.Worker
// make sure we get a clean folder ready to use.
IOUtil.DeleteDirectory(destDirectory, executionContext.CancellationToken);
Directory.CreateDirectory(destDirectory);
executionContext.Output($"Download action repository '{downloadInfo.NameWithOwner}@{downloadInfo.Ref}'");
executionContext.Output($"Download action repository '{downloadInfo.NameWithOwner}@{downloadInfo.Ref}' (SHA:{downloadInfo.ResolvedSha})");
}
await DownloadRepositoryActionAsync(executionContext, downloadInfo, destDirectory);
@@ -928,20 +1011,30 @@ namespace GitHub.Runner.Worker
}
else if (actionDefinitionData.Execution.ExecutionType == ActionExecutionType.Composite)
{
// TODO: we need to generate unique Id's for composite steps
Trace.Info($"Loading Composite steps");
var compositeAction = actionDefinitionData.Execution as CompositeActionExecutionData;
setupInfo.Steps = compositeAction.Steps;
// cache steps ids if not done so already
if (!_cachedEmbeddedStepIds.ContainsKey(repositoryAction.Id))
{
_cachedEmbeddedStepIds[repositoryAction.Id] = new List<Guid>();
foreach (var compositeStep in compositeAction.Steps)
{
var guid = Guid.NewGuid();
compositeStep.Id = guid;
_cachedEmbeddedStepIds[repositoryAction.Id].Add(guid);
}
}
// TODO: remove once we remove the DistributedTask.EnableCompositeActions FF
foreach (var step in compositeAction.Steps)
{
step.Id = Guid.NewGuid();
if (string.IsNullOrEmpty(executionContext.Global.Variables.Get("DistributedTask.EnableCompositeActions")) && step.Reference.Type != Pipelines.ActionSourceType.Script)
{
throw new Exception("`uses:` keyword is not currently supported.");
}
}
return setupInfo;
}
else
@@ -1102,9 +1195,11 @@ namespace GitHub.Runner.Worker
public sealed class CompositeActionExecutionData : ActionExecutionData
{
public override ActionExecutionType ExecutionType => ActionExecutionType.Composite;
public override bool HasPre => false;
public override bool HasPost => false;
public override bool HasPre => PreSteps.Count > 0;
public override bool HasPost => PostSteps.Count > 0;
public List<Pipelines.ActionStep> PreSteps { get; set; }
public List<Pipelines.ActionStep> Steps { get; set; }
public Stack<Pipelines.ActionStep> PostSteps { get; set; }
public MappingToken Outputs { get; set; }
}
@@ -1168,7 +1263,7 @@ namespace GitHub.Runner.Worker
public class ActionSetupInfo
{
public ActionContainer Container { get; set; }
public List<Pipelines.ActionStep> Steps {get; set;}
public List<Pipelines.ActionStep> Steps { get; set; }
}
public class PrepareActionsState

View File

@@ -480,6 +480,10 @@ namespace GitHub.Runner.Worker
return new CompositeActionExecutionData()
{
Steps = steps.Cast<Pipelines.ActionStep>().ToList(),
PreSteps = new List<Pipelines.ActionStep>(),
PostSteps = new Stack<Pipelines.ActionStep>(),
InitCondition = "always()",
CleanupCondition = "always()",
Outputs = outputs
};
}

View File

@@ -82,6 +82,28 @@ namespace GitHub.Runner.Worker
ActionExecutionData handlerData = definition.Data?.Execution;
ArgUtil.NotNull(handlerData, nameof(handlerData));
List<JobExtensionRunner> localActionContainerSetupSteps = null;
// Handle Composite Local Actions
// Need to download and expand the tree of referenced actions
if (handlerData.ExecutionType == ActionExecutionType.Composite &&
handlerData is CompositeActionExecutionData compositeHandlerData &&
Stage == ActionRunStage.Main &&
Action.Reference is Pipelines.RepositoryPathReference localAction &&
string.Equals(localAction.RepositoryType, Pipelines.PipelineConstants.SelfAlias, StringComparison.OrdinalIgnoreCase))
{
var actionManager = HostContext.GetService<IActionManager>();
var prepareResult = await actionManager.PrepareActionsAsync(ExecutionContext, compositeHandlerData.Steps, ExecutionContext.Id);
// Reload definition since post may exist now (from embedded steps that were JIT downloaded)
definition = taskManager.LoadAction(ExecutionContext, Action);
ArgUtil.NotNull(definition, nameof(definition));
handlerData = definition.Data?.Execution;
ArgUtil.NotNull(handlerData, nameof(handlerData));
// Save container setup steps so we can reference them later
localActionContainerSetupSteps = prepareResult.ContainerSetupSteps;
}
if (handlerData.HasPre &&
Action.Reference is Pipelines.RepositoryPathReference repoAction &&
string.Equals(repoAction.RepositoryType, Pipelines.PipelineConstants.SelfAlias, StringComparison.OrdinalIgnoreCase))
@@ -249,7 +271,8 @@ namespace GitHub.Runner.Worker
inputs,
environment,
ExecutionContext.Global.Variables,
actionDirectory: definition.Directory);
actionDirectory: definition.Directory,
localActionContainerSetupSteps: localActionContainerSetupSteps);
// Print out action details
handler.PrintActionDetails(Stage);

View File

@@ -0,0 +1,47 @@
using System;
using System.Text;
using GitHub.DistributedTask.Pipelines;
using GitHub.Runner.Common;
using GitHub.Runner.Sdk;
using ObjectTemplating = GitHub.DistributedTask.ObjectTemplating;
namespace GitHub.Runner.Worker
{
public sealed class ConditionTraceWriter : ObjectTemplating::ITraceWriter
{
private readonly IExecutionContext _executionContext;
private readonly Tracing _trace;
private readonly StringBuilder _traceBuilder = new StringBuilder();
public string Trace => _traceBuilder.ToString();
public ConditionTraceWriter(Tracing trace, IExecutionContext executionContext)
{
ArgUtil.NotNull(trace, nameof(trace));
_trace = trace;
_executionContext = executionContext;
}
public void Error(string format, params Object[] args)
{
var message = StringUtil.Format(format, args);
_trace.Error(message);
_executionContext?.Debug(message);
}
public void Info(string format, params Object[] args)
{
var message = StringUtil.Format(format, args);
_trace.Info(message);
_executionContext?.Debug(message);
_traceBuilder.AppendLine(message);
}
public void Verbose(string format, params Object[] args)
{
var message = StringUtil.Format(format, args);
_trace.Verbose(message);
_executionContext?.Debug(message);
}
}
}

View File

@@ -36,6 +36,7 @@ namespace GitHub.Runner.Worker
public interface IExecutionContext : IRunnerService
{
Guid Id { get; }
Guid EmbeddedId { get; }
string ScopeName { get; }
string ContextName { get; }
Task ForceCompleted { get; }
@@ -49,6 +50,7 @@ namespace GitHub.Runner.Worker
Dictionary<string, string> IntraActionState { get; }
Dictionary<string, VariableValue> JobOutputs { get; }
ActionsEnvironmentReference ActionsEnvironment { get; }
List<ActionsStepTelemetry> ActionsStepsTelemetry { get; }
DictionaryContextData ExpressionValues { get; }
IList<IFunctionInfo> ExpressionFunctions { get; }
JobContext JobContext { get; }
@@ -58,6 +60,10 @@ namespace GitHub.Runner.Worker
// Only job level ExecutionContext has PostJobSteps
Stack<IStep> PostJobSteps { get; }
HashSet<Guid> EmbeddedStepsWithPostRegistered{ get; }
// Keep track of embedded steps states
Dictionary<Guid, Dictionary<string, string>> EmbeddedIntraActionState { get; }
bool EchoOnActionCommand { get; set; }
@@ -68,8 +74,8 @@ namespace GitHub.Runner.Worker
// Initialize
void InitializeJob(Pipelines.AgentJobRequestMessage message, CancellationToken token);
void CancelToken();
IExecutionContext CreateChild(Guid recordId, string displayName, string refName, string scopeName, string contextName, Dictionary<string, string> intraActionState = null, int? recordOrder = null, IPagingLogger logger = null, bool isEmbedded = false, CancellationTokenSource cancellationTokenSource = null);
IExecutionContext CreateEmbeddedChild(string scopeName, string contextName);
IExecutionContext CreateChild(Guid recordId, string displayName, string refName, string scopeName, string contextName, Dictionary<string, string> intraActionState = null, int? recordOrder = null, IPagingLogger logger = null, bool isEmbedded = false, CancellationTokenSource cancellationTokenSource = null, Guid embeddedId = default(Guid));
IExecutionContext CreateEmbeddedChild(string scopeName, string contextName, Guid embeddedId, Dictionary<string, string> intraActionState = null);
// logging
long Write(string tag, string message);
@@ -132,6 +138,7 @@ namespace GitHub.Runner.Worker
private long _totalThrottlingDelayInMilliseconds = 0;
public Guid Id => _record.Id;
public Guid EmbeddedId { get; private set; }
public string ScopeName { get; private set; }
public string ContextName { get; private set; }
public Task ForceCompleted => _forceCompleted.Task;
@@ -140,6 +147,7 @@ namespace GitHub.Runner.Worker
public Dictionary<string, VariableValue> JobOutputs { get; private set; }
public ActionsEnvironmentReference ActionsEnvironment { get; private set; }
public List<ActionsStepTelemetry> ActionsStepsTelemetry { get; private set; }
public DictionaryContextData ExpressionValues { get; } = new DictionaryContextData();
public IList<IFunctionInfo> ExpressionFunctions { get; } = new List<IFunctionInfo>();
@@ -155,6 +163,11 @@ namespace GitHub.Runner.Worker
// Only job level ExecutionContext has StepsWithPostRegistered
public HashSet<Guid> StepsWithPostRegistered { get; private set; }
// Only job level ExecutionContext has EmbeddedStepsWithPostRegistered
public HashSet<Guid> EmbeddedStepsWithPostRegistered { get; private set; }
public Dictionary<Guid, Dictionary<string, string>> EmbeddedIntraActionState { get; private set; }
public bool EchoOnActionCommand { get; set; }
// An embedded execution context shares the same record ID, record name, and logger
@@ -245,13 +258,15 @@ namespace GitHub.Runner.Worker
public void RegisterPostJobStep(IStep step)
{
// TODO: Remove when we support composite post job steps
if (this.IsEmbedded)
{
throw new Exception("Composite actions do not currently support post steps");
if (step is IActionRunner actionRunner && !Root.EmbeddedStepsWithPostRegistered.Add(actionRunner.Action.Id))
{
Trace.Info($"'post' of '{actionRunner.DisplayName}' already push to child post step stack.");
}
return;
}
if (step is IActionRunner actionRunner && !Root.StepsWithPostRegistered.Add(actionRunner.Action.Id))
else if (step is IActionRunner actionRunner && !Root.StepsWithPostRegistered.Add(actionRunner.Action.Id))
{
Trace.Info($"'post' of '{actionRunner.DisplayName}' already push to post step stack.");
return;
@@ -261,7 +276,7 @@ namespace GitHub.Runner.Worker
Root.PostJobSteps.Push(step);
}
public IExecutionContext CreateChild(Guid recordId, string displayName, string refName, string scopeName, string contextName, Dictionary<string, string> intraActionState = null, int? recordOrder = null, IPagingLogger logger = null, bool isEmbedded = false, CancellationTokenSource cancellationTokenSource = null)
public IExecutionContext CreateChild(Guid recordId, string displayName, string refName, string scopeName, string contextName, Dictionary<string, string> intraActionState = null, int? recordOrder = null, IPagingLogger logger = null, bool isEmbedded = false, CancellationTokenSource cancellationTokenSource = null, Guid embeddedId = default(Guid))
{
Trace.Entering();
@@ -270,6 +285,7 @@ namespace GitHub.Runner.Worker
child.Global = Global;
child.ScopeName = scopeName;
child.ContextName = contextName;
child.EmbeddedId = embeddedId;
if (intraActionState == null)
{
child.IntraActionState = new Dictionary<string, string>(StringComparer.OrdinalIgnoreCase);
@@ -317,9 +333,9 @@ namespace GitHub.Runner.Worker
/// An embedded execution context shares the same record ID, record name, logger,
/// and a linked cancellation token.
/// </summary>
public IExecutionContext CreateEmbeddedChild(string scopeName, string contextName)
public IExecutionContext CreateEmbeddedChild(string scopeName, string contextName, Guid embeddedId, Dictionary<string, string> intraActionState = null)
{
return Root.CreateChild(_record.Id, _record.Name, _record.Id.ToString("N"), scopeName, contextName, logger: _logger, isEmbedded: true, cancellationTokenSource: CancellationTokenSource.CreateLinkedTokenSource(_cancellationTokenSource.Token));
return Root.CreateChild(_record.Id, _record.Name, _record.Id.ToString("N"), scopeName, contextName, logger: _logger, isEmbedded: true, cancellationTokenSource: CancellationTokenSource.CreateLinkedTokenSource(_cancellationTokenSource.Token), intraActionState: intraActionState, embeddedId: embeddedId);
}
public void Start(string currentOperation = null)
@@ -375,7 +391,7 @@ namespace GitHub.Runner.Worker
_logger.End();
// Skip if generated context name. Generated context names start with "__". After M271-ish the server will never send an empty context name.
// Skip if generated context name. Generated context names start with "__". After 3.2 the server will never send an empty context name.
if (!string.IsNullOrEmpty(ContextName) && !ContextName.StartsWith("__", StringComparison.Ordinal))
{
Global.StepsContext.SetOutcome(ScopeName, ContextName, (Outcome ?? Result ?? TaskResult.Succeeded).ToActionResult());
@@ -438,7 +454,7 @@ namespace GitHub.Runner.Worker
{
ArgUtil.NotNullOrEmpty(name, nameof(name));
// Skip if generated context name. Generated context names start with "__". After M271-ish the server will never send an empty context name.
// Skip if generated context name. Generated context names start with "__". After 3.2 the server will never send an empty context name.
if (string.IsNullOrEmpty(ContextName) || ContextName.StartsWith("__", StringComparison.Ordinal))
{
reference = null;
@@ -623,6 +639,9 @@ namespace GitHub.Runner.Worker
// Actions environment
ActionsEnvironment = message.ActionsEnvironment;
// ActionsStepTelemetry
ActionsStepsTelemetry = new List<ActionsStepTelemetry>();
// Service container info
Global.ServiceContainers = new List<ContainerInfo>();
@@ -683,6 +702,12 @@ namespace GitHub.Runner.Worker
// StepsWithPostRegistered for job ExecutionContext
StepsWithPostRegistered = new HashSet<Guid>();
// EmbeddedStepsWithPostRegistered for job ExecutionContext
EmbeddedStepsWithPostRegistered = new HashSet<Guid>();
// EmbeddedIntraActionState for job ExecutionContext
EmbeddedIntraActionState = new Dictionary<Guid, Dictionary<string,string>>();
// Job timeline record.
InitializeTimelineRecord(
timelineId: message.Timeline.Id,
@@ -947,6 +972,18 @@ namespace GitHub.Runner.Worker
context.Write(null, message);
}
public static void WriteDetails(this IExecutionContext context, string message)
{
if (context.IsEmbedded)
{
context.Debug(message);
}
else
{
context.Output(message);
}
}
// Do not add a format string overload. See comment on ExecutionContext.Write().
public static void Command(this IExecutionContext context, string message)
{

View File

@@ -11,6 +11,7 @@ using GitHub.DistributedTask.Pipelines.ContextData;
using GitHub.DistributedTask.Pipelines.ObjectTemplating;
using GitHub.DistributedTask.WebApi;
using GitHub.Runner.Common;
using GitHub.Runner.Common.Util;
using GitHub.Runner.Sdk;
using GitHub.Runner.Worker;
using GitHub.Runner.Worker.Expressions;
@@ -34,7 +35,66 @@ namespace GitHub.Runner.Worker.Handlers
Trace.Entering();
ArgUtil.NotNull(ExecutionContext, nameof(ExecutionContext));
ArgUtil.NotNull(Inputs, nameof(Inputs));
ArgUtil.NotNull(Data.Steps, nameof(Data.Steps));
List<Pipelines.ActionStep> steps;
if (stage == ActionRunStage.Pre)
{
ArgUtil.NotNull(Data.PreSteps, nameof(Data.PreSteps));
steps = Data.PreSteps;
}
else if (stage == ActionRunStage.Post)
{
ArgUtil.NotNull(Data.PostSteps, nameof(Data.PostSteps));
steps = new List<Pipelines.ActionStep>();
// Only register post steps for steps that actually ran
foreach (var step in Data.PostSteps.ToList())
{
if (ExecutionContext.Root.EmbeddedStepsWithPostRegistered.Contains(step.Id))
{
steps.Add(step);
}
else
{
Trace.Info($"Skipping executing post step id: {step.Id}, name: ${step.DisplayName}");
}
}
}
else
{
ArgUtil.NotNull(Data.Steps, nameof(Data.Steps));
steps = Data.Steps;
}
// Add Telemetry to JobContext to send with JobCompleteMessage
if (stage == ActionRunStage.Main)
{
var hasRunsStep = false;
var hasUsesStep = false;
foreach (var step in steps)
{
if (step.Reference.Type == Pipelines.ActionSourceType.Script)
{
hasRunsStep = true;
}
else
{
hasUsesStep = true;
}
}
var pathReference = Action as Pipelines.RepositoryPathReference;
var telemetry = new ActionsStepTelemetry {
Ref = GetActionRef(),
HasPreStep = Data.HasPre,
HasPostStep = Data.HasPost,
IsEmbedded = ExecutionContext.IsEmbedded,
Type = "composite",
HasRunsStep = hasRunsStep,
HasUsesStep = hasUsesStep,
StepCount = steps.Count
};
ExecutionContext.Root.ActionsStepsTelemetry.Add(telemetry);
}
try
{
@@ -45,7 +105,7 @@ namespace GitHub.Runner.Worker.Handlers
inputsData[i.Key] = new StringContextData(i.Value);
}
// Temporary hack until after M271-ish. After M271-ish the server will never send an empty
// Temporary hack until after 3.2. After 3.2 the server will never send an empty
// context name. Generated context names start with "__"
var childScopeName = ExecutionContext.GetFullyQualifiedContextName();
if (string.IsNullOrEmpty(childScopeName))
@@ -55,13 +115,28 @@ namespace GitHub.Runner.Worker.Handlers
// Create embedded steps
var embeddedSteps = new List<IStep>();
foreach (Pipelines.ActionStep stepData in Data.Steps)
// If we need to setup containers beforehand, do it
// only relevant for local composite actions that need to JIT download/setup containers
if (LocalActionContainerSetupSteps != null && LocalActionContainerSetupSteps.Count > 0)
{
foreach(var step in LocalActionContainerSetupSteps)
{
ArgUtil.NotNull(step, step.DisplayName);
var stepId = $"__{Guid.NewGuid()}";
step.ExecutionContext = ExecutionContext.CreateEmbeddedChild(childScopeName, stepId, Guid.NewGuid());
embeddedSteps.Add(step);
}
}
foreach (Pipelines.ActionStep stepData in steps)
{
var step = HostContext.CreateService<IActionRunner>();
step.Action = stepData;
step.Stage = stage;
step.Condition = stepData.Condition;
step.ExecutionContext = ExecutionContext.CreateEmbeddedChild(childScopeName, stepData.ContextName);
ExecutionContext.Root.EmbeddedIntraActionState.TryGetValue(step.Action.Id, out var intraActionState);
step.ExecutionContext = ExecutionContext.CreateEmbeddedChild(childScopeName, stepData.ContextName, step.Action.Id, intraActionState: intraActionState);
step.ExecutionContext.ExpressionValues["inputs"] = inputsData;
step.ExecutionContext.ExpressionValues["steps"] = ExecutionContext.Global.StepsContext.GetScope(childScopeName);
@@ -84,7 +159,6 @@ namespace GitHub.Runner.Worker.Handlers
ExecutionContext.ExpressionValues["inputs"] = inputsData;
ExecutionContext.ExpressionValues["steps"] = ExecutionContext.Global.StepsContext.GetScope(childScopeName);
ProcessOutputs();
ExecutionContext.Global.StepsContext.ClearScope(childScopeName);
}
catch (Exception ex)
{
@@ -148,6 +222,10 @@ namespace GitHub.Runner.Worker.Handlers
// Add Expression Functions
step.ExecutionContext.ExpressionFunctions.Add(new FunctionInfo<HashFilesFunction>(PipelineTemplateConstants.HashFiles, 1, byte.MaxValue));
step.ExecutionContext.ExpressionFunctions.Add(new FunctionInfo<AlwaysFunction>(PipelineTemplateConstants.Always, 0, 0));
step.ExecutionContext.ExpressionFunctions.Add(new FunctionInfo<CancelledFunction>(PipelineTemplateConstants.Cancelled, 0, 0));
step.ExecutionContext.ExpressionFunctions.Add(new FunctionInfo<FailureFunction>(PipelineTemplateConstants.Failure, 0, 0));
step.ExecutionContext.ExpressionFunctions.Add(new FunctionInfo<SuccessFunction>(PipelineTemplateConstants.Success, 0, 0));
// Initialize env context
Trace.Info("Initialize Env context for embedded step");
@@ -178,16 +256,17 @@ namespace GitHub.Runner.Worker.Handlers
}
}
var actionStep = step as IActionRunner;
try
{
// Evaluate and merge embedded-step env
var templateEvaluator = step.ExecutionContext.ToPipelineTemplateEvaluator();
var actionEnvironment = templateEvaluator.EvaluateStepEnvironment(actionStep.Action.Environment, step.ExecutionContext.ExpressionValues, step.ExecutionContext.ExpressionFunctions, Common.Util.VarUtil.EnvironmentVariableKeyComparer);
foreach (var env in actionEnvironment)
if (step is IActionRunner actionStep)
{
envContext[env.Key] = new StringContextData(env.Value ?? string.Empty);
// Evaluate and merge embedded-step env
var templateEvaluator = step.ExecutionContext.ToPipelineTemplateEvaluator();
var actionEnvironment = templateEvaluator.EvaluateStepEnvironment(actionStep.Action.Environment, step.ExecutionContext.ExpressionValues, step.ExecutionContext.ExpressionFunctions, Common.Util.VarUtil.EnvironmentVariableKeyComparer);
foreach (var env in actionEnvironment)
{
envContext[env.Key] = new StringContextData(env.Value ?? string.Empty);
}
}
}
catch (Exception ex)
@@ -198,13 +277,120 @@ namespace GitHub.Runner.Worker.Handlers
step.ExecutionContext.Complete(TaskResult.Failed);
}
await RunStepAsync(step);
// Register Callback
CancellationTokenRegistration? jobCancelRegister = null;
try
{
// Register job cancellation call back only if job cancellation token not been fire before each step run
if (!ExecutionContext.Root.CancellationToken.IsCancellationRequested)
{
// Test the condition again. The job was canceled after the condition was originally evaluated.
jobCancelRegister = ExecutionContext.Root.CancellationToken.Register(() =>
{
// Mark job as cancelled
ExecutionContext.Root.Result = TaskResult.Canceled;
ExecutionContext.Root.JobContext.Status = ExecutionContext.Root.Result?.ToActionResult();
step.ExecutionContext.Debug($"Re-evaluate condition on job cancellation for step: '{step.DisplayName}'.");
var conditionReTestTraceWriter = new ConditionTraceWriter(Trace, null); // host tracing only
var conditionReTestResult = false;
if (HostContext.RunnerShutdownToken.IsCancellationRequested)
{
step.ExecutionContext.Debug($"Skip Re-evaluate condition on runner shutdown.");
}
else
{
try
{
var templateEvaluator = step.ExecutionContext.ToPipelineTemplateEvaluator(conditionReTestTraceWriter);
var condition = new BasicExpressionToken(null, null, null, step.Condition);
conditionReTestResult = templateEvaluator.EvaluateStepIf(condition, step.ExecutionContext.ExpressionValues, step.ExecutionContext.ExpressionFunctions, step.ExecutionContext.ToExpressionState());
}
catch (Exception ex)
{
// Cancel the step since we get exception while re-evaluate step condition
Trace.Info("Caught exception from expression when re-test condition on job cancellation.");
step.ExecutionContext.Error(ex);
}
}
if (!conditionReTestResult)
{
// Cancel the step
Trace.Info("Cancel current running step.");
step.ExecutionContext.CancelToken();
}
});
}
else
{
if (ExecutionContext.Root.Result != TaskResult.Canceled)
{
// Mark job as cancelled
ExecutionContext.Root.Result = TaskResult.Canceled;
ExecutionContext.Root.JobContext.Status = ExecutionContext.Root.Result?.ToActionResult();
}
}
// Evaluate condition
step.ExecutionContext.Debug($"Evaluating condition for step: '{step.DisplayName}'");
var conditionTraceWriter = new ConditionTraceWriter(Trace, step.ExecutionContext);
var conditionResult = false;
var conditionEvaluateError = default(Exception);
if (HostContext.RunnerShutdownToken.IsCancellationRequested)
{
step.ExecutionContext.Debug($"Skip evaluate condition on runner shutdown.");
}
else
{
try
{
var templateEvaluator = step.ExecutionContext.ToPipelineTemplateEvaluator(conditionTraceWriter);
var condition = new BasicExpressionToken(null, null, null, step.Condition);
conditionResult = templateEvaluator.EvaluateStepIf(condition, step.ExecutionContext.ExpressionValues, step.ExecutionContext.ExpressionFunctions, step.ExecutionContext.ToExpressionState());
}
catch (Exception ex)
{
Trace.Info("Caught exception from expression.");
Trace.Error(ex);
conditionEvaluateError = ex;
}
}
if (!conditionResult && conditionEvaluateError == null)
{
// Condition is false
Trace.Info("Skipping step due to condition evaluation.");
step.ExecutionContext.Result = TaskResult.Skipped;
continue;
}
else if (conditionEvaluateError != null)
{
// Condition error
step.ExecutionContext.Error(conditionEvaluateError);
step.ExecutionContext.Result = TaskResult.Failed;
ExecutionContext.Result = TaskResult.Failed;
break;
}
else
{
await RunStepAsync(step);
}
}
finally
{
if (jobCancelRegister != null)
{
jobCancelRegister?.Dispose();
jobCancelRegister = null;
}
}
// Check failed or canceled
if (step.ExecutionContext.Result == TaskResult.Failed || step.ExecutionContext.Result == TaskResult.Canceled)
{
Trace.Info($"Update job result with current composite step result '{step.ExecutionContext.Result}'.");
ExecutionContext.Result = step.ExecutionContext.Result;
break;
ExecutionContext.Root.Result = TaskResultUtil.MergeTaskResults(ExecutionContext.Root.Result, step.ExecutionContext.Result.Value);
ExecutionContext.Root.JobContext.Status = ExecutionContext.Root.Result?.ToActionResult();
}
}
}

View File

@@ -1,4 +1,4 @@
using System.Collections.Generic;
using System.Collections.Generic;
using System.IO;
using System.Threading.Tasks;
using System;
@@ -50,8 +50,8 @@ namespace GitHub.Runner.Worker.Handlers
var dockerFile = Path.Combine(ActionDirectory, Data.Image);
ArgUtil.File(dockerFile, nameof(Data.Image));
ExecutionContext.Output($"##[group]Building docker image");
ExecutionContext.Output($"Dockerfile for action: '{dockerFile}'.");
ExecutionContext.WriteDetails(ExecutionContext.IsEmbedded ? "Building docker image" : $"##[group]Building docker image");
ExecutionContext.WriteDetails($"Dockerfile for action: '{dockerFile}'.");
var imageName = $"{dockerManager.DockerInstanceLabel}:{ExecutionContext.Id.ToString("N")}";
var buildExitCode = await dockerManager.DockerBuild(
ExecutionContext,
@@ -59,7 +59,7 @@ namespace GitHub.Runner.Worker.Handlers
dockerFile,
Directory.GetParent(dockerFile).FullName,
imageName);
ExecutionContext.Output("##[endgroup]");
ExecutionContext.WriteDetails(ExecutionContext.IsEmbedded ? "" : "##[endgroup]");
if (buildExitCode != 0)
{
@@ -69,6 +69,20 @@ namespace GitHub.Runner.Worker.Handlers
Data.Image = imageName;
}
string type = Action.Type == Pipelines.ActionSourceType.Repository ? "Dockerfile" : "DockerHub";
// Add Telemetry to JobContext to send with JobCompleteMessage
if (stage == ActionRunStage.Main)
{
var telemetry = new ActionsStepTelemetry {
Ref = GetActionRef(),
HasPreStep = Data.HasPre,
HasPostStep = Data.HasPost,
IsEmbedded = ExecutionContext.IsEmbedded,
Type = type
};
ExecutionContext.Root.ActionsStepsTelemetry.Add(telemetry);
}
// run container
var container = new ContainerInfo(HostContext)
{
@@ -200,6 +214,10 @@ namespace GitHub.Runner.Worker.Handlers
{
Environment["ACTIONS_CACHE_URL"] = cacheUrl;
}
if (systemConnection.Data.TryGetValue("GenerateIdTokenUrl", out var generateIdTokenUrl) && !string.IsNullOrEmpty(generateIdTokenUrl))
{
Environment["ACTIONS_ID_TOKEN_REQUEST_URL"] = generateIdTokenUrl;
}
foreach (var variable in this.Environment)
{

View File

@@ -20,6 +20,7 @@ namespace GitHub.Runner.Worker.Handlers
IStepHost StepHost { get; set; }
Dictionary<string, string> Inputs { get; set; }
string ActionDirectory { get; set; }
List<JobExtensionRunner> LocalActionContainerSetupSteps { get; set; }
Task RunAsync(ActionRunStage stage);
void PrintActionDetails(ActionRunStage stage);
}
@@ -41,12 +42,47 @@ namespace GitHub.Runner.Worker.Handlers
public IStepHost StepHost { get; set; }
public Dictionary<string, string> Inputs { get; set; }
public string ActionDirectory { get; set; }
public List<JobExtensionRunner> LocalActionContainerSetupSteps { get; set; }
public virtual string GetActionRef()
{
if (Action.Type == Pipelines.ActionSourceType.ContainerRegistry)
{
var registryAction = Action as Pipelines.ContainerRegistryReference;
return registryAction.Image;
}
else if (Action.Type == Pipelines.ActionSourceType.Repository)
{
var repoAction = Action as Pipelines.RepositoryPathReference;
if (string.Equals(repoAction.RepositoryType, Pipelines.PipelineConstants.SelfAlias, StringComparison.OrdinalIgnoreCase))
{
return repoAction.Path;
}
else
{
if (string.IsNullOrEmpty(repoAction.Path))
{
return $"{repoAction.Name}@{repoAction.Ref}";
}
else
{
return $"{repoAction.Name}/{repoAction.Path}@{repoAction.Ref}";
}
}
}
else
{
// this should never happen
Trace.Error($"Can't generate ref for {Action.Type.ToString()}");
}
return "";
}
public virtual void PrintActionDetails(ActionRunStage stage)
{
if (stage == ActionRunStage.Post)
{
ExecutionContext.Output($"Post job cleanup.");
ExecutionContext.WriteDetails($"Post job cleanup.");
return;
}
@@ -82,30 +118,30 @@ namespace GitHub.Runner.Worker.Handlers
groupName = "Action details";
}
ExecutionContext.Output($"##[group]{groupName}");
ExecutionContext.WriteDetails(ExecutionContext.IsEmbedded ? groupName : $"##[group]{groupName}");
if (this.Inputs?.Count > 0)
{
ExecutionContext.Output("with:");
ExecutionContext.WriteDetails("with:");
foreach (var input in this.Inputs)
{
if (!string.IsNullOrEmpty(input.Value))
{
ExecutionContext.Output($" {input.Key}: {input.Value}");
ExecutionContext.WriteDetails($" {input.Key}: {input.Value}");
}
}
}
if (this.Environment?.Count > 0)
{
ExecutionContext.Output("env:");
ExecutionContext.WriteDetails("env:");
foreach (var env in this.Environment)
{
ExecutionContext.Output($" {env.Key}: {env.Value}");
ExecutionContext.WriteDetails($" {env.Key}: {env.Value}");
}
}
ExecutionContext.Output("##[endgroup]");
ExecutionContext.WriteDetails(ExecutionContext.IsEmbedded ? "" : "##[endgroup]");
}
public override void Initialize(IHostContext hostContext)

View File

@@ -19,7 +19,8 @@ namespace GitHub.Runner.Worker.Handlers
Dictionary<string, string> inputs,
Dictionary<string, string> environment,
Variables runtimeVariables,
string actionDirectory);
string actionDirectory,
List<JobExtensionRunner> localActionContainerSetupSteps);
}
public sealed class HandlerFactory : RunnerService, IHandlerFactory
@@ -32,7 +33,8 @@ namespace GitHub.Runner.Worker.Handlers
Dictionary<string, string> inputs,
Dictionary<string, string> environment,
Variables runtimeVariables,
string actionDirectory)
string actionDirectory,
List<JobExtensionRunner> localActionContainerSetupSteps)
{
// Validate args.
Trace.Entering();
@@ -84,6 +86,7 @@ namespace GitHub.Runner.Worker.Handlers
handler.StepHost = stepHost;
handler.Inputs = inputs;
handler.ActionDirectory = actionDirectory;
handler.LocalActionContainerSetupSteps = localActionContainerSetupSteps;
return handler;
}
}

View File

@@ -53,6 +53,10 @@ namespace GitHub.Runner.Worker.Handlers
{
Environment["ACTIONS_CACHE_URL"] = cacheUrl;
}
if (systemConnection.Data.TryGetValue("GenerateIdTokenUrl", out var generateIdTokenUrl) && !string.IsNullOrEmpty(generateIdTokenUrl))
{
Environment["ACTIONS_ID_TOKEN_REQUEST_URL"] = generateIdTokenUrl;
}
// Resolve the target script.
string target = null;
@@ -69,6 +73,20 @@ namespace GitHub.Runner.Worker.Handlers
target = Data.Post;
}
// Add Telemetry to JobContext to send with JobCompleteMessage
if (stage == ActionRunStage.Main)
{
var telemetry = new ActionsStepTelemetry
{
Ref = GetActionRef(),
HasPreStep = Data.HasPre,
HasPostStep = Data.HasPost,
IsEmbedded = ExecutionContext.IsEmbedded,
Type = "node12"
};
ExecutionContext.Root.ActionsStepsTelemetry.Add(telemetry);
}
ArgUtil.NotNullOrEmpty(target, nameof(target));
target = Path.Combine(ActionDirectory, target);
ArgUtil.File(target, nameof(target));

View File

@@ -23,18 +23,6 @@ namespace GitHub.Runner.Worker.Handlers
public override void PrintActionDetails(ActionRunStage stage)
{
// We don't want to display the internal workings if composite (similar/equivalent information can be found in debug)
void writeDetails(string message)
{
if (ExecutionContext.IsEmbedded)
{
ExecutionContext.Debug(message);
}
else
{
ExecutionContext.Output(message);
}
}
if (stage == ActionRunStage.Post)
{
@@ -52,7 +40,7 @@ namespace GitHub.Runner.Worker.Handlers
firstLine = firstLine.Substring(0, firstNewLine);
}
writeDetails(ExecutionContext.IsEmbedded ? $"Run {firstLine}" : $"##[group]Run {firstLine}");
ExecutionContext.WriteDetails(ExecutionContext.IsEmbedded ? $"Run {firstLine}" : $"##[group]Run {firstLine}");
}
else
{
@@ -63,7 +51,7 @@ namespace GitHub.Runner.Worker.Handlers
foreach (var line in multiLines)
{
// Bright Cyan color
writeDetails($"\x1b[36;1m{line}\x1b[0m");
ExecutionContext.WriteDetails($"\x1b[36;1m{line}\x1b[0m");
}
string argFormat;
@@ -122,23 +110,23 @@ namespace GitHub.Runner.Worker.Handlers
if (!string.IsNullOrEmpty(shellCommandPath))
{
writeDetails($"shell: {shellCommandPath} {argFormat}");
ExecutionContext.WriteDetails($"shell: {shellCommandPath} {argFormat}");
}
else
{
writeDetails($"shell: {shellCommand} {argFormat}");
ExecutionContext.WriteDetails($"shell: {shellCommand} {argFormat}");
}
if (this.Environment?.Count > 0)
{
writeDetails("env:");
ExecutionContext.WriteDetails("env:");
foreach (var env in this.Environment)
{
writeDetails($" {env.Key}: {env.Value}");
ExecutionContext.WriteDetails($" {env.Key}: {env.Value}");
}
}
writeDetails(ExecutionContext.IsEmbedded ? "" : "##[endgroup]");
ExecutionContext.WriteDetails(ExecutionContext.IsEmbedded ? "" : "##[endgroup]");
}
public async Task RunAsync(ActionRunStage stage)
@@ -156,6 +144,16 @@ namespace GitHub.Runner.Worker.Handlers
var githubContext = ExecutionContext.ExpressionValues["github"] as GitHubContext;
ArgUtil.NotNull(githubContext, nameof(githubContext));
// Add Telemetry to JobContext to send with JobCompleteMessage
if (stage == ActionRunStage.Main)
{
var telemetry = new ActionsStepTelemetry {
IsEmbedded = ExecutionContext.IsEmbedded,
Type = "run",
};
ExecutionContext.Root.ActionsStepsTelemetry.Add(telemetry);
}
var tempDirectory = HostContext.GetDirectory(WellKnownDirectory.Temp);
Inputs.TryGetValue("script", out var contents);

View File

@@ -216,7 +216,7 @@ namespace GitHub.Runner.Worker
}
Trace.Info("Raising job completed event.");
var jobCompletedEvent = new JobCompletedEvent(message.RequestId, message.JobId, result, jobContext.JobOutputs, jobContext.ActionsEnvironment);
var jobCompletedEvent = new JobCompletedEvent(message.RequestId, message.JobId, result, jobContext.JobOutputs, jobContext.ActionsEnvironment, jobContext.ActionsStepsTelemetry);
var completeJobRetryLimit = 5;
var exceptions = new List<Exception>();

View File

@@ -354,43 +354,5 @@ namespace GitHub.Runner.Worker
executionContext.Complete(result, resultCode: resultCode);
}
private sealed class ConditionTraceWriter : ObjectTemplating::ITraceWriter
{
private readonly IExecutionContext _executionContext;
private readonly Tracing _trace;
private readonly StringBuilder _traceBuilder = new StringBuilder();
public string Trace => _traceBuilder.ToString();
public ConditionTraceWriter(Tracing trace, IExecutionContext executionContext)
{
ArgUtil.NotNull(trace, nameof(trace));
_trace = trace;
_executionContext = executionContext;
}
public void Error(string format, params Object[] args)
{
var message = StringUtil.Format(format, args);
_trace.Error(message);
_executionContext?.Debug(message);
}
public void Info(string format, params Object[] args)
{
var message = StringUtil.Format(format, args);
_trace.Info(message);
_executionContext?.Debug(message);
_traceBuilder.AppendLine(message);
}
public void Verbose(string format, params Object[] args)
{
var message = StringUtil.Format(format, args);
_trace.Verbose(message);
_executionContext?.Debug(message);
}
}
}
}

View File

@@ -0,0 +1,36 @@
using System.Runtime.Serialization;
namespace GitHub.DistributedTask.WebApi
{
/// <summary>
/// Information about a step run on the runner
/// </summary>
[DataContract]
public class ActionsStepTelemetry
{
[DataMember(EmitDefaultValue = false)]
public string Ref { get; set; }
[DataMember(EmitDefaultValue = false)]
public string Type { get; set; }
[DataMember(EmitDefaultValue = false)]
public bool? HasRunsStep { get; set; }
[DataMember(EmitDefaultValue = false)]
public bool? HasUsesStep { get; set; }
[DataMember(EmitDefaultValue = false)]
public bool IsEmbedded { get; set; }
[DataMember(EmitDefaultValue = false)]
public bool? HasPreStep { get; set; }
[DataMember(EmitDefaultValue = false)]
public bool? HasPostStep { get; set; }
[DataMember(EmitDefaultValue = false)]
public int? StepCount { get; set; }
}
}

View File

@@ -142,6 +142,19 @@ namespace GitHub.DistributedTask.WebApi
this.ActionsEnvironment = actionsEnvironment;
}
public JobCompletedEvent(
Int64 requestId,
Guid jobId,
TaskResult result,
Dictionary<String, VariableValue> outputs,
ActionsEnvironmentReference actionsEnvironment,
List<ActionsStepTelemetry> actionsStepsTelemetry)
: this(requestId, jobId, result, outputs)
{
this.ActionsEnvironment = actionsEnvironment;
this.ActionsStepsTelemetry = actionsStepsTelemetry;
}
[DataMember(EmitDefaultValue = false)]
public Int64 RequestId
{
@@ -169,6 +182,13 @@ namespace GitHub.DistributedTask.WebApi
get;
set;
}
[DataMember(EmitDefaultValue = false)]
public List<ActionsStepTelemetry> ActionsStepsTelemetry
{
get;
set;
}
}
[DataContract]

View File

@@ -877,6 +877,7 @@ namespace GitHub.Runner.Common.Tests.Worker
}
}
};
_hc.EnqueueInstance<IActionRunner>(new Mock<IActionRunner>().Object);
//Act
var steps = (await _actionManager.PrepareActionsAsync(_ec.Object, actions)).ContainerSetupSteps;
@@ -967,8 +968,7 @@ namespace GitHub.Runner.Common.Tests.Worker
var result = await _actionManager.PrepareActionsAsync(_ec.Object, actions);
//Assert
// TODO: Update this test
Assert.Equal(0, result.PreStepTracker.Count);
Assert.Equal(1, result.PreStepTracker.Count);
}
finally

View File

@@ -61,8 +61,8 @@ namespace GitHub.Runner.Common.Tests.Worker
_actionRunner.Action = action;
Dictionary<string, string> finialInputs = new Dictionary<string, string>();
_handlerFactory.Setup(x => x.Create(It.IsAny<IExecutionContext>(), It.IsAny<ActionStepDefinitionReference>(), It.IsAny<IStepHost>(), It.IsAny<ActionExecutionData>(), It.IsAny<Dictionary<string, string>>(), It.IsAny<Dictionary<string, string>>(), It.IsAny<Variables>(), It.IsAny<string>()))
.Callback((IExecutionContext executionContext, Pipelines.ActionStepDefinitionReference actionReference, IStepHost stepHost, ActionExecutionData data, Dictionary<string, string> inputs, Dictionary<string, string> environment, Variables runtimeVariables, string taskDirectory) =>
_handlerFactory.Setup(x => x.Create(It.IsAny<IExecutionContext>(), It.IsAny<ActionStepDefinitionReference>(), It.IsAny<IStepHost>(), It.IsAny<ActionExecutionData>(), It.IsAny<Dictionary<string, string>>(), It.IsAny<Dictionary<string, string>>(), It.IsAny<Variables>(), It.IsAny<string>(), It.IsAny<List<JobExtensionRunner>>()))
.Callback((IExecutionContext executionContext, Pipelines.ActionStepDefinitionReference actionReference, IStepHost stepHost, ActionExecutionData data, Dictionary<string, string> inputs, Dictionary<string, string> environment, Variables runtimeVariables, string taskDirectory, List<JobExtensionRunner> localActionContainerSetupSteps) =>
{
finialInputs = inputs;
})
@@ -107,8 +107,8 @@ namespace GitHub.Runner.Common.Tests.Worker
_actionRunner.Action = action;
Dictionary<string, string> finialInputs = new Dictionary<string, string>();
_handlerFactory.Setup(x => x.Create(It.IsAny<IExecutionContext>(), It.IsAny<ActionStepDefinitionReference>(), It.IsAny<IStepHost>(), It.IsAny<ActionExecutionData>(), It.IsAny<Dictionary<string, string>>(), It.IsAny<Dictionary<string, string>>(), It.IsAny<Variables>(), It.IsAny<string>()))
.Callback((IExecutionContext executionContext, Pipelines.ActionStepDefinitionReference actionReference, IStepHost stepHost, ActionExecutionData data, Dictionary<string, string> inputs, Dictionary<string, string> environment, Variables runtimeVariables, string taskDirectory) =>
_handlerFactory.Setup(x => x.Create(It.IsAny<IExecutionContext>(), It.IsAny<ActionStepDefinitionReference>(), It.IsAny<IStepHost>(), It.IsAny<ActionExecutionData>(), It.IsAny<Dictionary<string, string>>(), It.IsAny<Dictionary<string, string>>(), It.IsAny<Variables>(), It.IsAny<string>(), It.IsAny<List<JobExtensionRunner>>()))
.Callback((IExecutionContext executionContext, Pipelines.ActionStepDefinitionReference actionReference, IStepHost stepHost, ActionExecutionData data, Dictionary<string, string> inputs, Dictionary<string, string> environment, Variables runtimeVariables, string taskDirectory, List<JobExtensionRunner> localActionContainerSetupSteps) =>
{
finialInputs = inputs;
})
@@ -308,8 +308,8 @@ namespace GitHub.Runner.Common.Tests.Worker
_actionRunner.Action = action;
Dictionary<string, string> finialInputs = new Dictionary<string, string>();
_handlerFactory.Setup(x => x.Create(It.IsAny<IExecutionContext>(), It.IsAny<ActionStepDefinitionReference>(), It.IsAny<IStepHost>(), It.IsAny<ActionExecutionData>(), It.IsAny<Dictionary<string, string>>(), It.IsAny<Dictionary<string, string>>(), It.IsAny<Variables>(), It.IsAny<string>()))
.Callback((IExecutionContext executionContext, Pipelines.ActionStepDefinitionReference actionReference, IStepHost stepHost, ActionExecutionData data, Dictionary<string, string> inputs, Dictionary<string, string> environment, Variables runtimeVariables, string taskDirectory) =>
_handlerFactory.Setup(x => x.Create(It.IsAny<IExecutionContext>(), It.IsAny<ActionStepDefinitionReference>(), It.IsAny<IStepHost>(), It.IsAny<ActionExecutionData>(), It.IsAny<Dictionary<string, string>>(), It.IsAny<Dictionary<string, string>>(), It.IsAny<Variables>(), It.IsAny<string>(), It.IsAny<List<JobExtensionRunner>>()))
.Callback((IExecutionContext executionContext, Pipelines.ActionStepDefinitionReference actionReference, IStepHost stepHost, ActionExecutionData data, Dictionary<string, string> inputs, Dictionary<string, string> environment, Variables runtimeVariables, string taskDirectory, List<JobExtensionRunner> localActionContainerSetupSteps) =>
{
finialInputs = inputs;
})
@@ -359,8 +359,8 @@ namespace GitHub.Runner.Common.Tests.Worker
_actionRunner.Action = action;
Dictionary<string, string> finialInputs = new Dictionary<string, string>();
_handlerFactory.Setup(x => x.Create(It.IsAny<IExecutionContext>(), It.IsAny<ActionStepDefinitionReference>(), It.IsAny<IStepHost>(), It.IsAny<ActionExecutionData>(), It.IsAny<Dictionary<string, string>>(), It.IsAny<Dictionary<string, string>>(), It.IsAny<Variables>(), It.IsAny<string>()))
.Callback((IExecutionContext executionContext, Pipelines.ActionStepDefinitionReference actionReference, IStepHost stepHost, ActionExecutionData data, Dictionary<string, string> inputs, Dictionary<string, string> environment, Variables runtimeVariables, string taskDirectory) =>
_handlerFactory.Setup(x => x.Create(It.IsAny<IExecutionContext>(), It.IsAny<ActionStepDefinitionReference>(), It.IsAny<IStepHost>(), It.IsAny<ActionExecutionData>(), It.IsAny<Dictionary<string, string>>(), It.IsAny<Dictionary<string, string>>(), It.IsAny<Variables>(), It.IsAny<string>(), It.IsAny<List<JobExtensionRunner>>()))
.Callback((IExecutionContext executionContext, Pipelines.ActionStepDefinitionReference actionReference, IStepHost stepHost, ActionExecutionData data, Dictionary<string, string> inputs, Dictionary<string, string> environment, Variables runtimeVariables, string taskDirectory, List<JobExtensionRunner> localActionContainerSetupSteps) =>
{
finialInputs = inputs;
})

View File

@@ -141,7 +141,7 @@ namespace GitHub.Runner.Common.Tests.Worker
var jobExtension = new JobExtension();
jobExtension.Initialize(hc);
_actionManager.Setup(x => x.PrepareActionsAsync(It.IsAny<IExecutionContext>(), It.IsAny<IEnumerable<Pipelines.JobStep>>()))
_actionManager.Setup(x => x.PrepareActionsAsync(It.IsAny<IExecutionContext>(), It.IsAny<IEnumerable<Pipelines.JobStep>>(), It.IsAny<Guid>()))
.Returns(Task.FromResult(new PrepareResult(new List<JobExtensionRunner>(), new Dictionary<Guid, IActionRunner>())));
List<IStep> result = await jobExtension.InitializeJob(_jobEc, _message);
@@ -176,7 +176,7 @@ namespace GitHub.Runner.Common.Tests.Worker
var jobExtension = new JobExtension();
jobExtension.Initialize(hc);
_actionManager.Setup(x => x.PrepareActionsAsync(It.IsAny<IExecutionContext>(), It.IsAny<IEnumerable<Pipelines.JobStep>>()))
_actionManager.Setup(x => x.PrepareActionsAsync(It.IsAny<IExecutionContext>(), It.IsAny<IEnumerable<Pipelines.JobStep>>(), It.IsAny<Guid>()))
.Returns(Task.FromResult(new PrepareResult(new List<JobExtensionRunner>() { new JobExtensionRunner(null, "", "prepare1", null), new JobExtensionRunner(null, "", "prepare2", null) }, new Dictionary<Guid, IActionRunner>())));
List<IStep> result = await jobExtension.InitializeJob(_jobEc, _message);

View File

@@ -1 +1 @@
2.279.0
2.280.0