Compare commits

..

1 Commits

Author SHA1 Message Date
Thomas Boop
c9924d5b7c Check if file exists before reading 2020-01-03 15:02:47 -05:00
79 changed files with 920 additions and 7901 deletions

1
.gitignore vendored
View File

@@ -2,7 +2,6 @@
**/bin
**/obj
**/libs
**/lib
# editors
**/*.xproj

View File

@@ -1,61 +0,0 @@
# ADR 263: Self Hosted Runner Proxies
**Date**: 2019-11-13
**Status**: Accepted
## Context
- Proxy support is required for some enterprises and organizations to start using their own self hosted runners
- While there is not a standard convention, many applications support setting proxies via the environmental variables `http_proxy`, `https_proxy`, `no_proxy`, such as curl, wget, perl, python, docker, git, R, ect
- Some of these applications use `HTTPS_PROXY` versus `https_proxy`, but most understand or primarily support the lowercase variant
## Decision
We will update the Runner to use the conventional environment variables for proxies: `http_proxy`, `https_proxy` and `no_proxy` if they are set.
These are described in detail below:
- `https_proxy` a proxy URL for all https traffic. It may contain basic authentication credentials. For example:
- http://proxy.com
- http://127.0.0.1:8080
- http://user:password@proxy.com
- `http_proxy` a proxy URL for all http traffic. It may contain basic authentication credentials. For example:
- http://proxy.com
- http://127.0.0.1:8080
- http://user:password@proxy.com
- `no_proxy` a comma seperated list of hosts that should not use the proxy. An optional port may be specified
- `google.com`
- `yahoo.com:443`
- `google.com,bing.com`
We won't use `http_proxy` for https traffic when `https_proxy` is not set, this behavior lines up with any libcurl based tools (curl, git) and wget.
Otherwise action authors and workflow users need to adjust to differences between the runner proxy convention, and tools used by their actions and scripts.
Example:
Customer set `http_proxy=http://127.0.0.1:8888` and configure the runner against `https://github.com/owner/repo`, with the `https_proxy` -> `http_proxy` fallback, the runner will connect to server without any problem. However, if user runs `git push` to `https://github.com/owner/repo`, `git` won't use the proxy since it require `https_proxy` to be set for any https traffic.
> `golang`, `node.js` and other dev tools from the linux community use `http_proxy` for both http and https traffic base on my research.
A majority of our users are using Linux where these variables are commonly required to be set by various programs. By reading these values, we simplify the process for self hosted runners to set up proxy, and expose it in a way users are already familiar with.
A password provided for a proxy will be masked in the logs.
We will support the lowercase and uppercase variants, with lowercase taking priority if both are set.
### No Proxy Format
While exact implementations are different per application on handle `no_proxy` env, most applications accept a comma separated list of hosts. Some accept wildcard characters (*). We are going to do exact case-insentive matches, and not support wildcards at this time.
For example:
- example.com will match example.com, foo.example.com, foo.bar.example.com
- foo.example.com will match bar.foo.example.com and foo.example.com
We will not support IP addresses for `no_proxy`, only hostnames.
## Consequences
1. Enterprises and organizations needing proxy support will be able to embrace self hosted runners
2. Users will need to set these environmental variables before configuring the runner in order to use a proxy when configuring
3. The runner will read from the environmental variables during config and runtime and use the provided proxy if it exists
4. Users may need to pass these environmental variables into other applications if they do not natively take these variables
5. Action authors may need to update their workflows to react to the these environment variables
6. We will document the way of setting environmental variables for runners using the environmental variables and how the runner uses them
7. Like all other secrets, users will be able to relatively easily figure out proxy password if they can modify a workflow file running on a self hosted machine

View File

@@ -1,263 +0,0 @@
# ADR 0276: Problem Matchers
**Date** 2019-06-05
**Status** Accepted
## Context
Compilation failures during a CI build should surface good error messages.
For example, the actual compile errors from the typescript compiler should bubble as issues in the UI. And not simply "tsc exited with exit code 1".
VSCode has an extensible model for solving this type of problem. VSCode allows users to configure which problems matchers to use, when scanning output. For example, a user can apply the `tsc` problem matcher to receive a rich error output experience in VSCode, when compiling their typescript project.
The problem-matcher concept fits well with "setup" actions. For example, the `setup-nodejs` action will download node.js, add it to the PATH, and register the `tsc` problem matcher. For the duration of the job, the `tsc` problem matcher will be applied against the output.
## Decision
### Registration
#### Using `##` command
`##[add-matcher]path-to-problem-matcher-config.json`
Using a `##` command allows for flexibility:
- Ad hoc scripts can register problem matchers
- Allows problem matchers to be conditionally registered
Note, if a matcher with the same name is registered a second time, it will clobber the first instance.
#### Unregister using `##` command
A way out for rare cases where scoping is a problem.
`##[remove-matcher]owner`
For the this to be usable, the `owner` needs to be discoverable. Therefore, debug print the owner on registration.
### Single line matcher
Consider the output:
```
[...]
Build FAILED.
"C:\temp\problemmatcher\myproject\ConsoleApp1\ConsoleApp1.sln" (default target) (1) ->
"C:\temp\problemmatcher\myproject\ConsoleApp1\ConsoleApp1\ConsoleApp1.csproj" (default target) (2) ->
"C:\temp\problemmatcher\myproject\ConsoleApp1\ClassLibrary1\ClassLibrary1.csproj" (default target) (3) ->
(CoreCompile target) ->
Class1.cs(16,24): warning CS0612: 'ClassLibrary1.Helpers.MyHelper.Name' is obsolete [C:\temp\problemmatcher\myproject\ConsoleApp1\ClassLibrary1\ClassLibrary1.csproj]
"C:\temp\problemmatcher\myproject\ConsoleApp1\ConsoleApp1.sln" (default target) (1) ->
"C:\temp\problemmatcher\myproject\ConsoleApp1\ConsoleApp1\ConsoleApp1.csproj" (default target) (2) ->
"C:\temp\problemmatcher\myproject\ConsoleApp1\ClassLibrary1\ClassLibrary1.csproj" (default target) (3) ->
(CoreCompile target) ->
Helpers\MyHelper.cs(16,30): error CS1002: ; expected [C:\temp\problemmatcher\myproject\ConsoleApp1\ClassLibrary1\ClassLibrary1.csproj]
1 Warning(s)
1 Error(s)
```
The below match configuration uses a regular expression to discover problem lines. And the match groups are mapped into issue-properties.
```json
"owner": "msbuild",
"pattern": [
{
"regexp": "^\\s*([^:]+)\\((\\d+),(\\d+)\\): (error|warning) ([^:]+): (.*) \\[(.+)\\]$",
"file": 1,
"line": 2,
"column": 3,
"severity": 4,
"code": 5,
"message": 6,
"fromPath": 7
}
]
```
The above output and match configuration produces the following matches:
```
line: Class1.cs(16,24): warning CS0612: 'ClassLibrary1.Helpers.MyHelper.Name' is obsolete [C:\myrepo\myproject\ConsoleApp1\ClassLibrary1\ClassLibrary1.csproj]
file: Class1.cs
line: 16
column: 24
severity: warning
code: CS0612
message: 'ClassLibrary1.Helpers.MyHelper.Name' is obsolete
fromPath: C:\myrepo\myproject\ConsoleApp1\ClassLibrary1\ClassLibrary1.csproj
```
```
line: Helpers\MyHelper.cs(16,30): error CS1002: ; expected [C:\myrepo\myproject\ConsoleApp1\ClassLibrary1\ClassLibrary1.csproj]
file: Helpers\MyHelper.cs
line: 16
column: 30
severity: error
code: CS1002
message: ; expected
fromPath: C:\myrepo\myproject\ConsoleApp1\ClassLibrary1\ClassLibrary1.csproj
```
Additionally the line will appear red in the web UI (prefix with `##[error]`).
Note, an error does not imply task failure. Exit codes communicate failure.
Note, strip color codes when evaluating regular expressions.
### Multi-line matcher
Consider the below output from ESLint in stylish mode. The file name is printed once, yet multiple error lines are printed.
```
test.js
1:0 error Missing "use strict" statement strict
5:10 error 'addOne' is defined but never used no-unused-vars
✖ 2 problems (2 errors, 0 warnings)
```
The below match configuration uses multiple regular expressions, for the multiple lines.
And the last pattern of a multiline matcher can specify the `loop` property. This allows multiple errors to be discovered.
```json
"owner": "eslint-stylish",
"pattern": [
{
"regexp": "^([^\\s].*)$",
"file": 1
},
{
"regexp": "^\\s+(\\d+):(\\d+)\\s+(error|warning|info)\\s+(.*)\\s\\s+(.*)$",
"line": 1,
"column": 2,
"severity": 3,
"message": 4,
"code": 5,
"loop": true
}
]
```
The above output and match configuration produces two matches:
```
line: 1:0 error Missing "use strict" statement strict
file: test.js
line: 1
column: 0
severity: error
message: Missing "use strict" statement
code: strict
```
```
line: 5:10 error 'addOne' is defined but never used no-unused-vars
file: test.js
line: 5
column: 10
severity: error
message: 'addOne' is defined but never used
code: no-unused-vars
```
Note, in the above example only the error line will appear red in the web UI. The \"file\" line will not appear red.
### Other details
#### Configuration `owner`
Can be used to stomp over or remove.
#### Rooting the file
The goal of the file information is to provide a hyperlink in the UI.
Solving this problem means:
- Rooting the file when unrooted:
- Use the `fromPath` if specified (assume file path)
- Use the `github.workspace` (where the repo is cloned on disk)
- Match against a repository to determine the relative path within the repo
This is a place where we diverge from VSCode. VSCode task configuration are specific to the local workspace (workspace root is known or can be specified). We're solving a more generic problem, so we need more information - specifically the `fromPath` property - in order to accurately root the path.
In order to avoid creating inaccurate hyperlinks on the error issues, the agent will verify the file exists and is in the main repository. Otherwise omit the file property from the error issue and debug trace what happened.
#### Supported severity levels
Ordinal ignore case:
- `warning`
- `error`
Coalesce empty with \"error\". For any other values, omit logging an issue and debug trace what happened.
#### Default severity level
Problem matchers are unable to interpret severity strings other than `warning` and `error`. The `severity` match group expects `warning` or `error` (case insensitive).
However some tools indicate error/warning in different ways. For example `flake8` uses codes like `E100`, `W200`, and `F300` (error, warning, fatal, respectively).
Therefore, allow a property `severity`, sibling to `owner`, which identifies the default severity for the problem matcher. This allows two problem matchers are registered - one for warnings and one for errors.
For example, given the following `flake8` output:
```
./bootcamp/settings.py:156:80: E501 line too long (94 > 79 characters)
./bootcamp/settings.py:165:5: F403 'from local_settings import *' used; unable to detect undefined names
```
Two problem matchers can be used:
```json
{
"problemMatcher": [
{
"owner": "flake8",
"pattern": [
{
"regexp": "^(.+):(\\d+):(\\d+): ([EF]\\d+) (.+)$",
"file": 1,
"line": 2,
"column": 3,
"code": 4,
"message": 5
}
]
},
{
"owner": "flake8-warnings",
"severity": "warning",
"pattern": [
{
"regexp": "^(.+):(\\d+):(\\d+): (W\\d+) (.+)$",
"file": 1,
"line": 2,
"column": 3,
"code": 4,
"message": 5
}
]
}
]
}
```
#### Mitigate regular expression denial of service (ReDos)
If a matcher exceeds a 1 second timeout when processing a line, retry up to two three times total.
After three unsuccessful attempts, warn and eject the matcher. The matcher will not run again for the duration of the job.
### Where we diverge from VSCode
- We added the `fromPath` concept for rooting paths. This is done differently in VSCode, since a task is the scope (root path well known). For us, the job is the scope.
- VSCode allows additional activation info background tasks that are always running (recompile on files changed). They allow regular expressions to define when the matcher scope begins and ends. This is an interesting concept that we could leverage to help solve our scoping problem.
## Consequences
- Setup actions should register problem matchers

View File

@@ -1,93 +0,0 @@
# ADR 0277: Run action shell option
**Date** 2019-07-09
**Status** Accepted
## Context
run-actions run scripts using a platform specific shell:
`bash -eo pipefail` on non-windows, and `cmd.exe /c /d /s` on windows
The `shell` option overwrites this to allow different flags or completely different shells/interpreters
A small example is:
```yml
jobs:
bash-job:
actions:
- run: echo "Hello"
shell: bash
python-job:
actions:
- run: print("Hello")
shell: python {0}
```
## Decision
___
### Shell option
The keyword being used is `shell`
`shell` can be either:
1. Builtins / Explicitly supported keywords. It is useful to support at least `cmd`, and `powershell` on Windows. Because `cmd my_cmd_script` and `powershell my_ps1_script` are not valid the same way many Linux/cross-platform interpreters are, e.g. `bash myscript` or `python myscript`. Those tools (and potentially others) also require the correct file extension to run, or must be run in a particular way to get the exit codes consistently, so we must have first class knowledge about them. We provide default templates for these keywords as follows:
- `cmd`: Default is: `%ComSpec% /D /E:ON /V:OFF /S /C "CALL "{0}""` where the script name is automatically appended with `.cmd` and substituted for `{0}`
- Note this is equivalent to the default Windows behavior if no shell option is given
- `pwsh`: Default is: `pwsh -command "& '{0}'"` where the script is automatically appended with `.ps1`
- `powershell`: Default is: `powershell -command "& '{0}'"` where the script is automatically appended with `.ps1`
- `bash`: Uses `bash --noprofile --norc -eo pipefail {0}`
- The default behavior on non-Windows if no shell is given is to attempt this first
- `sh`: Uses `sh -e {0}`
- This is the default behavior on non-Windows if no shell is given, AND `bash` (see above) was not located on the PATH
- `python`: `python {0}`
- **NOTE**: The exact command ran may vary by machine. We only provide default arguments and command format for the listed shell. While the above behavior is expected on hosted machines, private runners may vary. For example, `sh` (or other commands) may actually be a link to `/bin/dash`, `/bin/bash`, or other
1. A template string: `command [...options] {0} [...more_options]`
- As above, the file name of the temporary script will be templated in. This gives users more control to have options at any location relative to the script path
- The first whitespace-delimited word of the string will be interpreted as the command
- e.g. `python {0} arg1 arg2` or similar can be used if passing args is needed. Some shells will require other options after the filename for various reasons
Note that (1) simply provides defaults that are executed with the same mechanism as (2). That is:
- A temporary script file is generated, and the path to that file is templated into the string at `{0}`
- The first word of the formatted string is assumed to be a command, and we attempt to locate its full path
- The fully qualified path to the command, plus the remaining arguments, is executed
- e.g. `shell: bash` expands to `/bin/bash --noprofile --norc -eo pipefail /runner/_layout/_work/_temp/f8d4fb2b-19d9-47e6-a786-4cc538d52761.sh` on my private runner
At this time, **THE LIST OF WELL-KNOWN SHELL OPTIONS IS**:
- cmd - Windows (hosted vs2017, vs2019) only
- powershell - Windows (hosted vs2017, vs2019) only
- sh - All hosted platforms
- pwsh - All hosted platforms
- bash - All hosted platforms
- python - All hosted platforms. Can use setup-python to configure which python will be used
___
### Containers
For container jobs, `shell` should just work the same as above, transparently. We will simply `exec` the command in the job container, passing the same arguments in
___
### Exit codes / Error action preference
For builtin shells, we provide defaults that make the most sense for CI, running within Actions, and being executed by our runner
bash/sh:
- Fail-fast behavior using `set -e o pipefail` is the default for `bash` and `shell` builtins, and by default when no option is given on non-Windows platforms
- Users can opt out of fail-fast and take full control easily by providing a template string to the shell options, eg: `bash {0}`.
- sh-like shells exit with the exit code of the last command executed in a script, and is our default behavior. Thus the runner reports the status of the step as fail/succeed based on this exit code
powershell/pwsh
- Fail-fast behavior when possible. For `pwsh` and `powershell` builtins, we will prepend `$ErrorActionPreference = 'stop'` to script contents
- We append `if ((Test-Path -LiteralPath variable:\LASTEXITCODE)) { exit $LASTEXITCODE }` to powershell scripts to get Action statuses to reflect the script's last exit code
- Users can always opt out by not using the builtins, and providing a shell option like: `pwsh -File {0}`, or `powershell -Command "& '{0}'"`, depending on need
cmd
- There doesnt seem to be a way to fully opt in to fail-fast behavior other than writing your script to check each error code and respond accordingly, so we cant actually provide that behavior by default, it will be completely up to the user to write this behavior into their script
- cmd.exe will exit (return the error code to the runner) with the errorlevel of the last program it executed. This is internally consistent with the previous default behavior (sh, pwsh) and is the cmd.exe default, so we keep that behavior
## Consequences
Valid `shell` options will depend on the hosted images. We will need to maintain tight image compat
First class support for a shell will require a major version schema change to modify. We cannot remove or modify the behavior of a well-known supported option, However, adding first class support for new shells is backwards compatible. For instance, we can add a well-known `python` option, because non-well-known options would have always needed to include `{0}`, e.g. `python {0}`

View File

@@ -1,60 +0,0 @@
# ADR 0278: Env Context
**Date**: 2019-09-30
**Status**: Accepted
## Context
User wants to reference workflow variables defined in workflow yaml file for action's input, displayName and condition.
## Decision
### Add `env` context in the runner
Runner will create and populate the `env` context for every job execution using following logic:
1. On job start, create `env` context with any environment variables in the job message, these are env defined in customer's YAML file's job/workflow level `env` section.
2. Update `env` context when customer use `::set-env::` to set env at the runner level.
3. Update `env` context with step's `env` block before each step runs.
The `env` context is only available in the runner, customer can't use the `env` context in any server evaluation part, just like the `runner` context
Example yaml:
```yaml
env:
env1: 10
env2: 20
env3: 30
jobs:
build:
env:
env1: 100
env2: 200
runs-on: ubuntu-latest
steps:
- run: |
echo ${{ env.env1 }} // 1000
echo $env1 // 1000
echo $env2 // 200
echo $env3 // 30
if: env.env2 == 200 // true
name: ${{ env.env1 }}_${{ env.env2 }} //1000_200
env:
env1: 1000
```
### Don't populate the `env` context with environment variables from runner machine.
With job container and container action, the `env` context may not have the right value customer want and will cause confusion.
Ex:
```yaml
build:
runs-on: ubuntu-latest <- $USER=runner in hosted machine
container: ubuntu:16.04 <- $USER=root in container
steps:
- run: echo ${{env.USER}} <- what should customer expect this output? runner/root
- uses: docker://ubuntu:18.04
with:
args: echo ${{env.USER}} <- what should customer expect this output? runner/root
```

View File

@@ -1,71 +0,0 @@
# ADR 0279: HashFiles Expression Function
**Date**: 2019-09-30
**Status**: Accepted
## Context
First party action `actions/cache` needs a input which is an explicit `key` used for restoring and saving the cache. For packages caching, the most comment `key` might be the hash result of contents from all `package-lock.json` under `node_modules` folder.
There are serval different ways to get the hash `key` input for `actions/cache` action.
1. Customer calculate the `key` themselves from a different action, customer won't like this since it needs extra step for using cache feature
```yaml
steps:
- run: |
hash=some_linux_hash_method(file1, file2, file3)
echo ::set-output name=hash::$hash
id: createHash
- uses: actions/cache@v1
with:
key: ${{ steps.createHash.outputs.hash }}
```
2. Make the `key` input of `actions/cache` follow certain convention to calculate hash, this limited the `key` input to a certain format customer may not want.
```yaml
steps:
- uses: actions/cache@v1
with:
key: ${{ runner.os }}|${{ github.workspace }}|**/package-lock.json
```
## Decision
### Add hashFiles() function to expression engine for calculate files' hash
`hashFiles()` will only allow on runner side since it needs to read files on disk, using `hashFiles()` on any server side evaluated expression will cause runtime errors.
`hashFiles()` will only support hashing files under the `$GITHUB_WORKSPACE` since the expression evaluated on the runner, if customer use job container or container action, the runner won't have access to file system inside the container.
`hashFiles()` will only take 1 parameters:
- `hashFiles('**/package-lock.json')` // Search files under $GITHUB_WORKSPACE and calculate a hash for them
**Question: Do we need to support more than one match patterns?**
Ex: `hashFiles('**/package-lock.json', '!toolkit/core/package-lock.json', '!toolkit/io/package-lock.json')`
Answer: Only support single match pattern for GA, we can always add later.
This will help customer has better experience with the `actions/cache` action's input.
```yaml
steps:
- uses: actions/cache@v1
with:
key: ${{hashFiles('**/package-lock.json')}}-${{github.ref}}-${{runner.os}}
```
For search pattern, we will use basic globbing (`*` `?` and `[]`) and globstar (`**`).
Additional pattern details:
- Root relative paths with `github.workspace` (the main repo)
- Make `*` match files that start with `.`
- Case insensitive on Windows
- Accept `\` or `/` path separators on Windows
Hashing logic:
1. Get all files under `$GITHUB_WORKSPACE`.
2. Use search pattern filter all files to get files that matches the search pattern. (search pattern only apply to file path not folder path)
3. Sort all matched files by full file path in alphabet order.
4. Use SHA256 algorithm to hash each matched file and store hash result.
5. Use SHA256 to hash all stored files' hash results to get the final 64 chars hash result.
**Question: Should we include the folder structure info into the hash?**
Answer: No

View File

@@ -1,30 +0,0 @@
# ADR 0280: Echoing of Command Input
**Date**: 2019-11-04
**Status**: Accepted
## Context
Command echoing as a default behavior tends to clutter the user logs, so we want to swap to a system where users have to opt in to see this information.
Command outputs will still be echoed in the case there are any errors processing such commands. This is so the end user can have more context on why the command failed and help with troubleshooting.
Echo output in the user logs can be explicitly controlled by the new commands `::echo::on` and `::echo::off`. By default, echoing is enabled if `ACTIONS_STEP_DEBUG` secret is enabled, otherwise echoing is disabled.
## Decision
- The only commands that currently echo output are
- `remove-matcher`
- `add-matcher`
- `add-path`
- These will no longer echo the command, if processed successfully
- All commands echo the input when any of these conditions is fulfilled:
1. When such commands fail with an error
2. When `::echo::on` is set
3. When the `ACTIONS_STEP_DEBUG` is set, and echoing hasn't been explicitly disabled with `::echo::off`
- There are a few commands that won't be echoed, even when echo is enabled. These are (as of 2019/11/04):
- `add-mask`
- `debug`
- `warning`
- `error`
- The three commands above will not echo, either because echoing the command would leak secrets (e.g. `add-mask`), or it would not add any additional troubleshooting information to the logs (e.g. `debug`). It's expected that future commands would follow these "echo-suppressing" guidelines as well. Echo-suppressed commands are still free to output other information to the logs, as deemed fit.

View File

@@ -1,48 +0,0 @@
# ADR 0297: Base64 Masking Trailing Characters
**Date** 2020-01-21
**Status** Proposed
## Context
The Runner registers a number of Value Encoders, which mask various encodings of a provided secret. Currently, we register a 3 base64 Encoders:
- The base64 encoded secret
- The secret with the first character removed then base64 encoded
- The secret with the first two characters removed then base64 encoded
This gives us good coverage across the board for secrets and secrets with a prefix (i.e. `base64($user:$pass)`).
However, we don't have great coverage for cases where the secret has a string appended to it before it is base64 encoded (i.e.: `base64($pass\n))`).
Most notably we've seen this as a result of user error where a user accidentially appends a newline or space character before encoding their secret in base64.
## Decision
### Trim end characters
We are going to modify all existing base64 encoders to trim information before registering as a secret.
We will trim:
- `=` from the end of all base64 strings. This is a padding character that contains no information.
- Based on the number of `=`'s at the end of a base64 string, a malicious user could predict the length of the original secret modulo 3.
- If a user saw `***==`, they would know the secret could be 1,4,7,10... characters.
- If a string contains `=` we will also trim the last non-padding character from the base64 secret.
- This character can change if a string is appended to the secret before the encoding.
### Register a fourth encoder
We will also add back in the original base64 encoded secret encoder for four total encoders:
- The base64 encoded secret
- The base64 encoded secret trimmed
- The secret with the first character removed then base64 encoded and trimmed
- The secret with the first two characters removed then base64 encoded and trimmed
This allows us to fully cover the most common scenario where a user base64 encodes their secret and expects the entire thing to be masked.
This will result in us only revealing length or bit information when a prefix or suffix is added to a secret before encoding.
## Consequences
- In the case where a secret has a prefix or suffix added before base64 encoding, we may now reveal up to 20 bits of information and the length of the original string modulo 3, rather then the original 16 bits and no length information
- Secrets with a suffix appended before encoding will now be masked across the board. Previously it was only masked if it was a multiple of 3 characters
- Performance will suffer in a neglible way

View File

@@ -1,61 +0,0 @@
# Runner Authentication and Authorization
## Goals
- Support runner installs in untrusted domains.
- The account that configures or runs the runner process is not relevant for accessing GitHub resources.
- Accessing GitHub resources is done with a per-job token which expires when job completes.
- The token is granted to trusted parts of the system including the runner, actions and script steps specified by the workflow author as trusted.
- All OAuth tokens that come from the Token Service that the runner uses to access Actions Service resources are the same. It's just the scope and expiration of the token that may vary.
## Configuration
Configuring a self-hosted runner is [covered here in the documentation](https://help.github.com/en/actions/hosting-your-own-runners/adding-self-hosted-runners).
Configuration is done with the user being authenticated via a time-limited, GitHub runner registration token.
*Your credentials are never used for registering the runner with the service.*
![Self-hosted runner config](../res/self-hosted-config.png)
During configuration, an RSA public/private key pair is created, the private key is stored in file on disk. On Windows, the content is protected with DPAPI (machine level encrypted - runner only valid on that machine) and on Linux/OSX with `chmod` permissions.
Using your credentials, the runner is registered with the service by sending the public key to the service which adds that runner to the pool and stores the public key, the Token Service will generate a `clientId` associated with the public key.
## Start and Listen
After configuring the runner, the runner can be started interactively (`./run.cmd` or `./run.sh`) or as a service.
![Self-hosted runner start](../res/self-hosted-start.png)
On start, the runner listener process loads the RSA private key (on Windows decrypting with machine key DPAPI), and asks the Token Service for an OAuth token which is signed with the RSA private key.
The server then responds with an OAuth token that grants permission to access the message queue (HTTP long poll), allowing the runner to acquire the messages it will eventually run.
## Run a workflow
When a workflow is run, its labels are evaluated, it is matched to a runner and a message is placed in a queue of messages for that runner.
The runner then starts listening for jobs via the message queue HTTP long poll.
The message is encrypted with the runner's public key, stored during runner configuration.
![Runner workflow run](../res/workflow-run.png)
A workflow is queued as a result of a triggered [event](https://help.github.com/en/actions/reference/events-that-trigger-workflows). Workflows can be scheduled to [run at specific UTC times](https://help.github.com/en/actions/reference/events-that-trigger-workflows#scheduled-events-schedule) using POSIX `cron` syntax.
An [OAuth token](http://self-issued.info/docs/draft-ietf-oauth-json-web-token.html) is generated, granting limited access to the host in Actions Service associated with the github.com repository/organization.
The lifetime of the OAuth token is the lifetime of the run or at most the [job timeout (default: 6 hours)](https://help.github.com/en/actions/reference/workflow-syntax-for-github-actions#jobsjob_idtimeout-minutes), plus 10 additional minutes.
## Accessing GitHub resources
The job message sent to the runner contains the OAuth token to talk back to the Actions Service.
The runner listener parent process will spawn a runner worker process for that job and send it the job message over IPC.
The token is never persisted.
Each action is run as a unique subprocess.
The encrypted access token will be provided as an environment variable in each action subprocess.
The token is registered with the runner as a secret and scrubbed from the logs as they are written.
Authentication in a workflow run to github.com can be accomplished by using the [`GITHUB_TOKEN`](https://help.github.com/en/actions/configuring-and-managing-workflows/authenticating-with-the-github_token#about-the-github_token-secret)) secret. This token expires after 60 minutes. Please note that this token is different from the OAuth token that the runner uses to talk to the Actions Service.
## Hosted runner authentication
Hosted runner authentication differs from self-hosted authentication in that runners do not undergo a registration process, but instead, the hosted runners get the OAuth token directly by reading the `.credentials` file. The scope of this particular token is limited for a given workflow job execution, and the token is revoked as soon as the job is finished.
![Hosted runner config and start](../res/hosted-config-start.png)

Binary file not shown.

Before

Width:  |  Height:  |  Size: 31 KiB

View File

@@ -1,52 +0,0 @@
# Markup used to generate the runner auth diagrams: https://websequencediagrams.com
title Runner Configuration (self-hosted only)
note left of Runner: GitHub repo URL as input
Runner->github.com: Retrieve Actions Service access using runner registration token
github.com->Runner: Access token for Actions Service
note left of Runner: Generate RSA key pair
note left of Runner: Store encrypted RSA private key on disk
Runner->Actions Service: Register runner using Actions Service access token
note right of Runner: Runner name, RSA public key sent
note right of Actions Service: Public key stored
Actions Service->Token Service: Register runner as an app along with the RSA public key
note right of Token Service: Public key stored
Token Service->Actions Service: Client Id for the runner application
Actions Service->Runner: Client Id and Token Endpoint URL
note left of Runner: Store runner configuration info into .runner file
note left of Runner: Store Token registration info into .credentials file
title Runner Start and Running (self-hosted only)
Runner.Listener->Runner.Listener: Start
note left of Runner.Listener: Load config info from .runner
note left of Runner.Listener: Load token registration from .credentials
Runner.Listener->Token Service: Exchange OAuth token (happens every 50 mins)
note right of Runner.Listener: Construct JWT token, use Client Id signed by RSA private key
note left of Actions Service: Find corresponding RSA public key, use Client Id\nVerify JWT token's signature
Token Service->Runner.Listener: OAuth token with limited permission and valid for 50 mins
Runner.Listener->Actions Service: Connect to Actions Service with OAuth token
Actions Service->Runner.Listener: Workflow job
title Running workflow
Runner.Listener->Service (Message Queue): Get message
note right of Runner.Listener: Authenticate with exchanged OAuth token
Event->Actions Service: Queue workflow
Actions Service->Actions Service: Generate OAuth token per job
Actions Service->Actions Service: Build job message with the OAuth token
Actions Service->Actions Service: Encrypt job message with the target runner's public key
Actions Service->Service (Message Queue): Send encrypted job message to runner
Service (Message Queue)->Runner.Listener: Send job
note right of Runner.Listener: Decrypt message with runner's private key
Runner.Listener->Runner.Worker: Create worker process per job and run the job
title Runner Configuration, Start and Running (hosted only)
Machine Management Service->Runner.Listener: Construct .runner configuration file, store token in .credentials
Runner.Listener->Runner.Listener: Start
note left of Runner.Listener: Load config info from .runner
note left of Runner.Listener: Load OAuth token from .credentials
Runner.Listener->Actions Service: Connect to Actions Service with OAuth token in .credentials
Actions Service->Runner.Listener: Workflow job

Binary file not shown.

Before

Width:  |  Height:  |  Size: 98 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 43 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 46 KiB

View File

@@ -1,30 +1,17 @@
## Features
- Expose whether debug is on/off via RUNNER_DEBUG. (#253)
- Upload log on runner when worker get killed due to cancellation timeout. (#255)
- Update config.sh/cmd --help documentation (#282)
- Set http_proxy and related env vars for job/service containers (#304)
- Set both http_proxy and HTTP_PROXY env for runner/worker processes. (#298)
- Remove runner flow: Change from PAT to "deletion token" in prompt (#225)
- Expose github.run_id and github.run_number to action runtime env. (#224)
## Bugs
- Verify runner Windows service hash started successfully after configuration (#236)
- Detect source file path in L0 without using env. (#257)
- Handle escaped '%' in commands data section (#200)
- Allow container to be null/empty during matrix expansion (#266)
- Translate problem matcher file to host path (#272)
- Change hashFiles() expression function to use @actions/glob. (#268)
- Default post-job action's condition to always(). (#293)
- Support action.yaml file as action's entry file (#288)
- Trace javascript action exit code to debug instead of user logs (#290)
- Change prompt message when removing a runner to lines up with GitHub.com UI (#303)
- Include step.env as part of env context. (#300)
- Update Base64 Encoders to deal with suffixes (#284)
- Clean up error messages for container scenarios (#221)
- Pick shell from prependpath (#231)
## Misc
- Move .sln file under ./src (#238)
- Treat warnings as errors during compile (#249)
- Runner code cleanup (#218 #227, #228, #229, #230)
- Consume dotnet core 3.1 in runner. (#213)
## Windows x64
We recommend configuring the runner in a root folder of the Windows drive (e.g. "C:\actions-runner"). This will help avoid issues related to service identity folder permissions and long file path restrictions on Windows
We recommend configuring the runner under "<DRIVE>:\actions-runner". This will help avoid issues related to service identity folder permissions and long file path restrictions on Windows
```
// Create a folder under the drive root
mkdir \actions-runner ; cd \actions-runner
@@ -32,7 +19,7 @@ mkdir \actions-runner ; cd \actions-runner
Invoke-WebRequest -Uri https://github.com/actions/runner/releases/download/v<RUNNER_VERSION>/actions-runner-win-x64-<RUNNER_VERSION>.zip -OutFile actions-runner-win-x64-<RUNNER_VERSION>.zip
// Extract the installer
Add-Type -AssemblyName System.IO.Compression.FileSystem ;
[System.IO.Compression.ZipFile]::ExtractToDirectory("$PWD\actions-runner-win-x64-<RUNNER_VERSION>.zip", "$PWD")
[System.IO.Compression.ZipFile]::ExtractToDirectory("$HOME\Downloads\actions-runner-win-x64-<RUNNER_VERSION>.zip", "$PWD")
```
## OSX
@@ -41,7 +28,7 @@ Add-Type -AssemblyName System.IO.Compression.FileSystem ;
// Create a folder
mkdir actions-runner && cd actions-runner
// Download the latest runner package
curl -O -L https://github.com/actions/runner/releases/download/v<RUNNER_VERSION>/actions-runner-osx-x64-<RUNNER_VERSION>.tar.gz
curl -O https://github.com/actions/runner/releases/download/v<RUNNER_VERSION>/actions-runner-osx-x64-<RUNNER_VERSION>.tar.gz
// Extract the installer
tar xzf ./actions-runner-osx-x64-<RUNNER_VERSION>.tar.gz
```
@@ -52,7 +39,7 @@ tar xzf ./actions-runner-osx-x64-<RUNNER_VERSION>.tar.gz
// Create a folder
mkdir actions-runner && cd actions-runner
// Download the latest runner package
curl -O -L https://github.com/actions/runner/releases/download/v<RUNNER_VERSION>/actions-runner-linux-x64-<RUNNER_VERSION>.tar.gz
curl -O https://github.com/actions/runner/releases/download/v<RUNNER_VERSION>/actions-runner-linux-x64-<RUNNER_VERSION>.tar.gz
// Extract the installer
tar xzf ./actions-runner-linux-x64-<RUNNER_VERSION>.tar.gz
```
@@ -63,7 +50,7 @@ tar xzf ./actions-runner-linux-x64-<RUNNER_VERSION>.tar.gz
// Create a folder
mkdir actions-runner && cd actions-runner
// Download the latest runner package
curl -O -L https://github.com/actions/runner/releases/download/v<RUNNER_VERSION>/actions-runner-linux-arm64-<RUNNER_VERSION>.tar.gz
curl -O https://github.com/actions/runner/releases/download/v<RUNNER_VERSION>/actions-runner-linux-arm64-<RUNNER_VERSION>.tar.gz
// Extract the installer
tar xzf ./actions-runner-linux-arm64-<RUNNER_VERSION>.tar.gz
```
@@ -74,7 +61,7 @@ tar xzf ./actions-runner-linux-arm64-<RUNNER_VERSION>.tar.gz
// Create a folder
mkdir actions-runner && cd actions-runner
// Download the latest runner package
curl -O -L https://github.com/actions/runner/releases/download/v<RUNNER_VERSION>/actions-runner-linux-arm-<RUNNER_VERSION>.tar.gz
curl -O https://github.com/actions/runner/releases/download/v<RUNNER_VERSION>/actions-runner-linux-arm-<RUNNER_VERSION>.tar.gz
// Extract the installer
tar xzf ./actions-runner-linux-arm-<RUNNER_VERSION>.tar.gz
```

View File

@@ -1,3 +0,0 @@
dist/
lib/
node_modules/

View File

@@ -1,59 +0,0 @@
{
"plugins": ["jest", "@typescript-eslint"],
"extends": ["plugin:github/es6"],
"parser": "@typescript-eslint/parser",
"parserOptions": {
"ecmaVersion": 9,
"sourceType": "module",
"project": "./tsconfig.json"
},
"rules": {
"eslint-comments/no-use": "off",
"import/no-namespace": "off",
"no-console": "off",
"no-unused-vars": "off",
"@typescript-eslint/no-unused-vars": "error",
"@typescript-eslint/explicit-member-accessibility": ["error", {"accessibility": "no-public"}],
"@typescript-eslint/no-require-imports": "error",
"@typescript-eslint/array-type": "error",
"@typescript-eslint/await-thenable": "error",
"@typescript-eslint/ban-ts-ignore": "error",
"camelcase": "off",
"@typescript-eslint/camelcase": "error",
"@typescript-eslint/class-name-casing": "error",
"@typescript-eslint/explicit-function-return-type": ["error", {"allowExpressions": true}],
"@typescript-eslint/func-call-spacing": ["error", "never"],
"@typescript-eslint/generic-type-naming": ["error", "^[A-Z][A-Za-z]*$"],
"@typescript-eslint/no-array-constructor": "error",
"@typescript-eslint/no-empty-interface": "error",
"@typescript-eslint/no-explicit-any": "error",
"@typescript-eslint/no-extraneous-class": "error",
"@typescript-eslint/no-for-in-array": "error",
"@typescript-eslint/no-inferrable-types": "error",
"@typescript-eslint/no-misused-new": "error",
"@typescript-eslint/no-namespace": "error",
"@typescript-eslint/no-non-null-assertion": "warn",
"@typescript-eslint/no-object-literal-type-assertion": "error",
"@typescript-eslint/no-unnecessary-qualifier": "error",
"@typescript-eslint/no-unnecessary-type-assertion": "error",
"@typescript-eslint/no-useless-constructor": "error",
"@typescript-eslint/no-var-requires": "error",
"@typescript-eslint/prefer-for-of": "warn",
"@typescript-eslint/prefer-function-type": "warn",
"@typescript-eslint/prefer-includes": "error",
"@typescript-eslint/prefer-interface": "error",
"@typescript-eslint/prefer-string-starts-ends-with": "error",
"@typescript-eslint/promise-function-async": "error",
"@typescript-eslint/require-array-sort-compare": "error",
"@typescript-eslint/restrict-plus-operands": "error",
"semi": "off",
"@typescript-eslint/semi": ["error", "never"],
"@typescript-eslint/type-annotation-spacing": "error",
"@typescript-eslint/unbound-method": "error"
},
"env": {
"node": true,
"es6": true,
"jest/globals": true
}
}

View File

@@ -1,3 +0,0 @@
dist/
lib/
node_modules/

View File

@@ -1,11 +0,0 @@
{
"printWidth": 80,
"tabWidth": 2,
"useTabs": false,
"semi": false,
"singleQuote": true,
"trailingComma": "none",
"bracketSpacing": false,
"arrowParens": "avoid",
"parser": "typescript"
}

View File

@@ -1 +0,0 @@
To update hashFiles under `Misc/layoutbin` run `npm install && npm run all`

File diff suppressed because it is too large Load Diff

View File

@@ -1,35 +0,0 @@
{
"name": "hashFiles",
"version": "1.0.0",
"description": "GitHub Actions HashFiles() expression function",
"main": "lib/hashFiles.js",
"scripts": {
"build": "tsc",
"format": "prettier --write **/*.ts",
"format-check": "prettier --check **/*.ts",
"lint": "eslint src/**/*.ts",
"pack": "ncc build -o ../../layoutbin/hashFiles",
"all": "npm run build && npm run format && npm run lint && npm run pack"
},
"repository": {
"type": "git",
"url": "git+https://github.com/actions/runner.git"
},
"keywords": [
"actions"
],
"author": "GitHub Actions",
"license": "MIT",
"dependencies": {
"@actions/glob": "^0.1.0"
},
"devDependencies": {
"@types/node": "^12.7.12",
"@typescript-eslint/parser": "^2.8.0",
"@zeit/ncc": "^0.20.5",
"eslint": "^5.16.0",
"eslint-plugin-github": "^2.0.0",
"prettier": "^1.19.1",
"typescript": "^3.6.4"
}
}

View File

@@ -1,55 +0,0 @@
import * as glob from '@actions/glob'
import * as crypto from 'crypto'
import * as fs from 'fs'
import * as stream from 'stream'
import * as util from 'util'
import * as path from 'path'
async function run(): Promise<void> {
// arg0 -> node
// arg1 -> hashFiles.js
// env[followSymbolicLinks] = true/null
// env[patterns] -> glob patterns
let followSymbolicLinks = false
const matchPatterns = process.env.patterns || ''
if (process.env.followSymbolicLinks === 'true') {
console.log('Follow symbolic links')
followSymbolicLinks = true
}
console.log(`Match Pattern: ${matchPatterns}`)
let hasMatch = false
const githubWorkspace = process.cwd()
const result = crypto.createHash('sha256')
let count = 0
const globber = await glob.create(matchPatterns, {followSymbolicLinks})
for await (const file of globber.globGenerator()) {
console.log(file)
if (!file.startsWith(`${githubWorkspace}${path.sep}`)) {
console.log(`Ignore '${file}' since it is not under GITHUB_WORKSPACE.`)
continue
}
if (fs.statSync(file).isDirectory()) {
console.log(`Skip directory '${file}'.`)
continue
}
const hash = crypto.createHash('sha256')
const pipeline = util.promisify(stream.pipeline)
await pipeline(fs.createReadStream(file), hash)
result.write(hash.digest())
count++
if (!hasMatch) {
hasMatch = true
}
}
result.end()
if (hasMatch) {
console.log(`Find ${count} files to hash.`)
console.error(`__OUTPUT__${result.digest('hex')}__OUTPUT__`)
} else {
console.error(`__OUTPUT____OUTPUT__`)
}
}
run()

View File

@@ -1,12 +0,0 @@
{
"compilerOptions": {
"target": "es6", /* Specify ECMAScript target version: 'ES3' (default), 'ES5', 'ES2015', 'ES2016', 'ES2017', 'ES2018', 'ES2019' or 'ESNEXT'. */
"module": "commonjs", /* Specify module code generation: 'none', 'commonjs', 'amd', 'system', 'umd', 'es2015', or 'ESNext'. */
"outDir": "./lib", /* Redirect output structure to the directory. */
"rootDir": "./src", /* Specify the root directory of input files. Use to control the output directory structure with --outDir. */
"strict": true, /* Enable all strict type-checking options. */
"noImplicitAny": true, /* Raise error on expressions and declarations with an implied 'any' type. */
"esModuleInterop": true /* Enables emit interoperability between CommonJS and ES Modules via creation of namespace objects for all imports. Implies 'allowSyntheticDefaultImports'. */
},
"exclude": ["node_modules", "**/*.test.ts"]
}

File diff suppressed because it is too large Load Diff

View File

@@ -3,6 +3,8 @@
<packageSources>
<!--To inherit the global NuGet package sources remove the <clear/> line below -->
<clear />
<add key="dotnet-core" value="https://www.myget.org/F/dotnet-core/api/v3/index.json" />
<add key="dotnet-buildtools" value="https://www.myget.org/F/dotnet-buildtools/api/v3/index.json" />
<add key="api.nuget.org" value="https://api.nuget.org/v3/index.json" />
</packageSources>
</configuration>

View File

@@ -9,27 +9,26 @@ namespace GitHub.Runner.Common
{
private static readonly EscapeMapping[] _escapeMappings = new[]
{
new EscapeMapping(token: "%", replacement: "%25"),
new EscapeMapping(token: ";", replacement: "%3B"),
new EscapeMapping(token: "\r", replacement: "%0D"),
new EscapeMapping(token: "\n", replacement: "%0A"),
new EscapeMapping(token: "]", replacement: "%5D"),
new EscapeMapping(token: "%", replacement: "%25"),
};
private static readonly EscapeMapping[] _escapeDataMappings = new[]
{
new EscapeMapping(token: "\r", replacement: "%0D"),
new EscapeMapping(token: "\n", replacement: "%0A"),
new EscapeMapping(token: "%", replacement: "%25"),
};
private static readonly EscapeMapping[] _escapePropertyMappings = new[]
{
new EscapeMapping(token: "%", replacement: "%25"),
new EscapeMapping(token: "\r", replacement: "%0D"),
new EscapeMapping(token: "\n", replacement: "%0A"),
new EscapeMapping(token: ":", replacement: "%3A"),
new EscapeMapping(token: ",", replacement: "%2C"),
new EscapeMapping(token: "%", replacement: "%25"),
};
private readonly Dictionary<string, string> _properties = new Dictionary<string, string>(StringComparer.OrdinalIgnoreCase);

View File

@@ -136,12 +136,6 @@ namespace GitHub.Runner.Common
}
}
public static class RunnerEvent
{
public static readonly string Register = "register";
public static readonly string Remove = "remove";
}
public static class Pipeline
{
public static class Path
@@ -168,8 +162,7 @@ namespace GitHub.Runner.Common
public static class Path
{
public static readonly string ActionsDirectory = "_actions";
public static readonly string ActionManifestYmlFile = "action.yml";
public static readonly string ActionManifestYamlFile = "action.yaml";
public static readonly string ActionManifestFile = "action.yml";
public static readonly string BinDirectory = "bin";
public static readonly string DiagDirectory = "_diag";
public static readonly string ExternalsDirectory = "externals";

View File

@@ -24,6 +24,7 @@ namespace GitHub.Runner.Common
private Guid _timelineId;
private Guid _timelineRecordId;
private string _pageId;
private FileStream _pageData;
private StreamWriter _pageWriter;
private int _byteCount;
@@ -39,6 +40,7 @@ namespace GitHub.Runner.Common
{
base.Initialize(hostContext);
_totalLines = 0;
_pageId = Guid.NewGuid().ToString();
_pagesFolder = Path.Combine(hostContext.GetDirectory(WellKnownDirectory.Diag), PagingFolder);
_jobServerQueue = HostContext.GetService<IJobServerQueue>();
Directory.CreateDirectory(_pagesFolder);
@@ -100,7 +102,7 @@ namespace GitHub.Runner.Common
{
EndPage();
_byteCount = 0;
_dataFileName = Path.Combine(_pagesFolder, $"{_timelineId}_{_timelineRecordId}_{++_pageCount}.log");
_dataFileName = Path.Combine(_pagesFolder, $"{_pageId}_{++_pageCount}.log");
_pageData = new FileStream(_dataFileName, FileMode.CreateNew);
_pageWriter = new StreamWriter(_pageData, System.Text.Encoding.UTF8);
}

View File

@@ -190,7 +190,7 @@ namespace GitHub.Runner.Listener
{
return GetArgOrPrompt(
name: Constants.Runner.CommandLine.Args.Token,
description: "Enter runner remove token:",
description: "Enter runner deletion token:",
defaultValue: string.Empty,
validator: Validators.NonEmptyValidator);
}
@@ -291,7 +291,7 @@ namespace GitHub.Runner.Listener
if (!string.IsNullOrEmpty(result))
{
// After read the arg from input commandline args, remove it from Arg dictionary,
// This will help if bad arg value passed through CommandLine arg, when ConfigurationManager ask CommandSetting the second time,
// This will help if bad arg value passed through CommandLine arg, when ConfigurationManager ask CommandSetting the second time,
// It will prompt for input instead of continue use the bad input.
_trace.Info($"Remove {name} from Arg dictionary.");
RemoveArg(name);

View File

@@ -1,18 +1,19 @@
using GitHub.DistributedTask.WebApi;
using GitHub.Runner.Common;
using GitHub.Runner.Common.Util;
using GitHub.Runner.Sdk;
using GitHub.Services.Common;
using GitHub.Services.OAuth;
using GitHub.Services.WebApi;
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Net.Http;
using System.Net.Http.Headers;
using System.Runtime.InteropServices;
using System.Security.Cryptography;
using System.Threading.Tasks;
using System.Runtime.InteropServices;
using GitHub.Runner.Common;
using GitHub.Runner.Sdk;
using System.Net.Http;
using System.Net.Http.Headers;
namespace GitHub.Runner.Listener.Configuration
{
@@ -108,7 +109,7 @@ namespace GitHub.Runner.Listener.Configuration
{
runnerSettings.GitHubUrl = inputUrl;
var githubToken = command.GetRunnerRegisterToken();
GitHubAuthResult authResult = await GetTenantCredential(inputUrl, githubToken, Constants.RunnerEvent.Register);
GitHubAuthResult authResult = await GetTenantCredential(inputUrl, githubToken);
runnerSettings.ServerUrl = authResult.TenantUrl;
creds = authResult.ToVssCredentials();
Trace.Info("cred retrieved via GitHub auth");
@@ -276,15 +277,12 @@ namespace GitHub.Runner.Listener.Configuration
throw new NotSupportedException("Message queue listen OAuth token.");
}
// Testing agent connection, detect any potential connection issue, like local clock skew that cause OAuth token expired.
// Testing agent connection, detect any protential connection issue, like local clock skew that cause OAuth token expired.
var credMgr = HostContext.GetService<ICredentialManager>();
VssCredentials credential = credMgr.LoadCredentials();
try
{
await _runnerServer.ConnectAsync(new Uri(runnerSettings.ServerUrl), credential);
// ConnectAsync() hits _apis/connectionData which is an anonymous endpoint
// Need to hit an authenticate endpoint to trigger OAuth token exchange.
await _runnerServer.GetAgentPoolsAsync();
_term.WriteSuccessMessage("Runner connection is good");
}
catch (VssOAuthTokenRequestException ex) when (ex.Message.Contains("Current server time is"))
@@ -375,7 +373,7 @@ namespace GitHub.Runner.Listener.Configuration
else
{
var githubToken = command.GetRunnerDeletionToken();
GitHubAuthResult authResult = await GetTenantCredential(settings.GitHubUrl, githubToken, Constants.RunnerEvent.Remove);
GitHubAuthResult authResult = await GetTenantCredential(settings.GitHubUrl, githubToken);
creds = authResult.ToVssCredentials();
Trace.Info("cred retrieved via GitHub auth");
}
@@ -519,23 +517,17 @@ namespace GitHub.Runner.Listener.Configuration
}
}
private async Task<GitHubAuthResult> GetTenantCredential(string githubUrl, string githubToken, string runnerEvent)
private async Task<GitHubAuthResult> GetTenantCredential(string githubUrl, string githubToken)
{
var gitHubUrlBuilder = new UriBuilder(githubUrl);
var githubApiUrl = $"{gitHubUrlBuilder.Scheme}://api.{gitHubUrlBuilder.Host}/actions/runner-registration";
var gitHubUrl = new UriBuilder(githubUrl);
var githubApiUrl = $"https://api.{gitHubUrl.Host}/repos/{gitHubUrl.Path.Trim('/')}/actions-runners/registration";
using (var httpClientHandler = HostContext.CreateHttpClientHandler())
using (var httpClient = new HttpClient(httpClientHandler))
{
httpClient.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("RemoteAuth", githubToken);
httpClient.DefaultRequestHeaders.UserAgent.Add(HostContext.UserAgent);
var bodyObject = new Dictionary<string, string>()
{
{"url", githubUrl},
{"runner_event", runnerEvent}
};
var response = await httpClient.PostAsync(githubApiUrl, new StringContent(StringUtil.ConvertToJson(bodyObject), null, "application/json"));
httpClient.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/vnd.github.shuri-preview+json"));
var response = await httpClient.PostAsync(githubApiUrl, new StringContent("", null, "application/json"));
if (response.IsSuccessStatusCode)
{

View File

@@ -568,10 +568,6 @@ namespace GitHub.Runner.Listener
{
Trace.Info("worker process has been killed.");
}
// When worker doesn't exit within cancel timeout, the runner will kill the worker process and worker won't finish upload job logs.
// The runner will try to upload these logs at this time.
await TryUploadUnfinishedLogs(message);
}
Trace.Info($"finish job request for job {message.JobId} with result: {resultOnAbandonOrCancel}");
@@ -716,121 +712,6 @@ namespace GitHub.Runner.Listener
}
}
// Best effort upload any logs for this job.
private async Task TryUploadUnfinishedLogs(Pipelines.AgentJobRequestMessage message)
{
Trace.Entering();
var logFolder = Path.Combine(HostContext.GetDirectory(WellKnownDirectory.Diag), PagingLogger.PagingFolder);
if (!Directory.Exists(logFolder))
{
return;
}
var logs = Directory.GetFiles(logFolder);
if (logs.Length == 0)
{
return;
}
try
{
var systemConnection = message.Resources.Endpoints.SingleOrDefault(x => string.Equals(x.Name, WellKnownServiceEndpointNames.SystemVssConnection));
ArgUtil.NotNull(systemConnection, nameof(systemConnection));
var jobServer = HostContext.GetService<IJobServer>();
VssCredentials jobServerCredential = VssUtil.GetVssCredential(systemConnection);
VssConnection jobConnection = VssUtil.CreateConnection(systemConnection.Url, jobServerCredential);
await jobServer.ConnectAsync(jobConnection);
var timeline = await jobServer.GetTimelineAsync(message.Plan.ScopeIdentifier, message.Plan.PlanType, message.Plan.PlanId, message.Timeline.Id, CancellationToken.None);
var updatedRecords = new List<TimelineRecord>();
var logPages = new Dictionary<Guid, Dictionary<int, string>>();
var logRecords = new Dictionary<Guid, TimelineRecord>();
foreach (var log in logs)
{
var logName = Path.GetFileNameWithoutExtension(log);
var logNameParts = logName.Split('_', StringSplitOptions.RemoveEmptyEntries);
if (logNameParts.Length != 3)
{
Trace.Warning($"log file '{log}' doesn't follow naming convension 'GUID_GUID_INT'.");
continue;
}
var logPageSeperator = logName.IndexOf('_');
var logRecordId = Guid.Empty;
var pageNumber = 0;
if (!Guid.TryParse(logNameParts[0], out Guid timelineId) || timelineId != timeline.Id)
{
Trace.Warning($"log file '{log}' is not belongs to current job");
continue;
}
if (!Guid.TryParse(logNameParts[1], out logRecordId))
{
Trace.Warning($"log file '{log}' doesn't follow naming convension 'GUID_GUID_INT'.");
continue;
}
if (!int.TryParse(logNameParts[2], out pageNumber))
{
Trace.Warning($"log file '{log}' doesn't follow naming convension 'GUID_GUID_INT'.");
continue;
}
var record = timeline.Records.FirstOrDefault(x => x.Id == logRecordId);
if (record != null)
{
if (!logPages.ContainsKey(record.Id))
{
logPages[record.Id] = new Dictionary<int, string>();
logRecords[record.Id] = record;
}
logPages[record.Id][pageNumber] = log;
}
}
foreach (var pages in logPages)
{
var record = logRecords[pages.Key];
if (record.Log == null)
{
// Create the log
record.Log = await jobServer.CreateLogAsync(message.Plan.ScopeIdentifier, message.Plan.PlanType, message.Plan.PlanId, new TaskLog(String.Format(@"logs\{0:D}", record.Id)), default(CancellationToken));
// Need to post timeline record updates to reflect the log creation
updatedRecords.Add(record.Clone());
}
for (var i = 1; i <= pages.Value.Count; i++)
{
var logFile = pages.Value[i];
// Upload the contents
using (FileStream fs = File.Open(logFile, FileMode.Open, FileAccess.Read, FileShare.ReadWrite))
{
var logUploaded = await jobServer.AppendLogContentAsync(message.Plan.ScopeIdentifier, message.Plan.PlanType, message.Plan.PlanId, record.Log.Id, fs, default(CancellationToken));
}
Trace.Info($"Uploaded unfinished log '{logFile}' for current job.");
IOUtil.DeleteFile(logFile);
}
}
if (updatedRecords.Count > 0)
{
await jobServer.UpdateTimelineRecordsAsync(message.Plan.ScopeIdentifier, message.Plan.PlanType, message.Plan.PlanId, message.Timeline.Id, updatedRecords, CancellationToken.None);
}
}
catch (Exception ex)
{
// Ignore any error during log upload since it's best effort
Trace.Error(ex);
}
}
// TODO: We need send detailInfo back to DT in order to add an issue for the job
private async Task CompleteJobRequestAsync(int poolId, Pipelines.AgentJobRequestMessage message, Guid lockToken, TaskResult result, string detailInfo = null)
{

View File

@@ -451,38 +451,16 @@ namespace GitHub.Runner.Listener
ext = "sh";
#endif
_term.WriteLine($@"
Commands:
.{separator}config.{ext} Configures the runner
.{separator}config.{ext} remove Unconfigures the runner
.{separator}run.{ext} Runs the runner interactively. Does not require any options.
Commands:,
.{separator}config.{ext} Configures the runner
.{separator}config.{ext} remove Unconfigures the runner
.{separator}run.{ext} Runs the runner interactively. Does not require any options.
Options:
--help Prints the help for each command
--version Prints the runner version
--commit Prints the runner commit
Config Options:
--unattended Disable interactive prompts for missing arguments. Defaults will be used for missing options
--url string Repository to add the runner to. Required if unattended
--token string Registration token. Required if unattended
--name string Name of the runner to configure (default {Environment.MachineName ?? "myrunner"})
--work string Relative runner work directory (default {Constants.Path.WorkDirectory})
--replace Replace any existing runner with the same name (default false)");
#if OS_WINDOWS
_term.WriteLine($@" --runasservice Run the runner as a service");
_term.WriteLine($@" --windowslogonaccount string Account to run the service as. Requires runasservice");
_term.WriteLine($@" --windowslogonpassword string Password for the service account. Requires runasservice");
#endif
_term.WriteLine($@"
Examples:
Configure a runner non-interactively:
.{separator}config.{ext} --unattended --url <url> --token <token>
Configure a runner non-interactively, replacing any existing runner with the same name:
.{separator}config.{ext} --unattended --url <url> --token <token> --replace [--name <name>]");
#if OS_WINDOWS
_term.WriteLine($@" Configure a runner to run as a service:");
_term.WriteLine($@" .{separator}config.{ext} --url <url> --token <token> --runasservice");
#endif
--help Prints the help for each command
");
}
}
}

View File

@@ -21,7 +21,6 @@ namespace GitHub.Runner.Sdk
private string _httpsProxyAddress;
private string _httpsProxyUsername;
private string _httpsProxyPassword;
private string _noProxyString;
private readonly List<ByPassInfo> _noProxyList = new List<ByPassInfo>();
private readonly HashSet<string> _noProxyUnique = new HashSet<string>(StringComparer.OrdinalIgnoreCase);
@@ -34,7 +33,6 @@ namespace GitHub.Runner.Sdk
public string HttpsProxyAddress => _httpsProxyAddress;
public string HttpsProxyUsername => _httpsProxyUsername;
public string HttpsProxyPassword => _httpsProxyPassword;
public string NoProxyString => _noProxyString;
public List<ByPassInfo> NoProxyList => _noProxyList;
@@ -73,10 +71,6 @@ namespace GitHub.Runner.Sdk
{
_httpProxyAddress = proxyHttpUri.AbsoluteUri;
// Set both environment variables since there are tools support both casing (curl, wget) and tools support only one casing (docker)
Environment.SetEnvironmentVariable("HTTP_PROXY", _httpProxyAddress);
Environment.SetEnvironmentVariable("http_proxy", _httpProxyAddress);
// the proxy url looks like http://[user:pass@]127.0.0.1:8888
var userInfo = Uri.UnescapeDataString(proxyHttpUri.UserInfo).Split(':', 2, StringSplitOptions.RemoveEmptyEntries);
if (userInfo.Length == 2)
@@ -103,10 +97,6 @@ namespace GitHub.Runner.Sdk
{
_httpsProxyAddress = proxyHttpsUri.AbsoluteUri;
// Set both environment variables since there are tools support both casing (curl, wget) and tools support only one casing (docker)
Environment.SetEnvironmentVariable("HTTPS_PROXY", _httpsProxyAddress);
Environment.SetEnvironmentVariable("https_proxy", _httpsProxyAddress);
// the proxy url looks like http://[user:pass@]127.0.0.1:8888
var userInfo = Uri.UnescapeDataString(proxyHttpsUri.UserInfo).Split(':', 2, StringSplitOptions.RemoveEmptyEntries);
if (userInfo.Length == 2)
@@ -131,12 +121,6 @@ namespace GitHub.Runner.Sdk
if (!string.IsNullOrEmpty(noProxyList))
{
_noProxyString = noProxyList;
// Set both environment variables since there are tools support both casing (curl, wget) and tools support only one casing (docker)
Environment.SetEnvironmentVariable("NO_PROXY", noProxyList);
Environment.SetEnvironmentVariable("no_proxy", noProxyList);
var noProxyListSplit = noProxyList.Split(',', StringSplitOptions.RemoveEmptyEntries);
foreach (string noProxy in noProxyListSplit)
{

View File

@@ -1,7 +1,6 @@
using GitHub.DistributedTask.Pipelines;
using GitHub.DistributedTask.WebApi;
using GitHub.Runner.Common.Util;
using GitHub.Runner.Worker.Container;
using System;
using System.Collections.Generic;
using System.IO;
@@ -16,14 +15,14 @@ namespace GitHub.Runner.Worker
{
void EnablePluginInternalCommand();
void DisablePluginInternalCommand();
bool TryProcessCommand(IExecutionContext context, string input, ContainerInfo container);
bool TryProcessCommand(IExecutionContext context, string input);
}
public sealed class ActionCommandManager : RunnerService, IActionCommandManager
{
private const string _stopCommand = "stop-commands";
private readonly Dictionary<string, IActionCommandExtension> _commandExtensions = new Dictionary<string, IActionCommandExtension>(StringComparer.OrdinalIgnoreCase);
private readonly HashSet<string> _registeredCommands = new HashSet<string>(StringComparer.OrdinalIgnoreCase);
private HashSet<string> _registeredCommands = new HashSet<string>(StringComparer.OrdinalIgnoreCase);
private readonly object _commandSerializeLock = new object();
private bool _stopProcessCommand = false;
private string _stopToken = null;
@@ -59,7 +58,7 @@ namespace GitHub.Runner.Worker
_registeredCommands.Remove("internal-set-repo-path");
}
public bool TryProcessCommand(IExecutionContext context, string input, ContainerInfo container)
public bool TryProcessCommand(IExecutionContext context, string input)
{
if (string.IsNullOrEmpty(input))
{
@@ -115,7 +114,7 @@ namespace GitHub.Runner.Worker
try
{
extension.ProcessCommand(context, input, actionCommand, container);
extension.ProcessCommand(context, input, actionCommand);
}
catch (Exception ex)
{
@@ -141,7 +140,7 @@ namespace GitHub.Runner.Worker
string Command { get; }
bool OmitEcho { get; }
void ProcessCommand(IExecutionContext context, string line, ActionCommand command, ContainerInfo container);
void ProcessCommand(IExecutionContext context, string line, ActionCommand command);
}
public sealed class InternalPluginSetRepoPathCommandExtension : RunnerService, IActionCommandExtension
@@ -151,7 +150,7 @@ namespace GitHub.Runner.Worker
public Type ExtensionType => typeof(IActionCommandExtension);
public void ProcessCommand(IExecutionContext context, string line, ActionCommand command, ContainerInfo container)
public void ProcessCommand(IExecutionContext context, string line, ActionCommand command)
{
if (!command.Properties.TryGetValue(SetRepoPathCommandProperties.repoFullName, out string repoFullName) || string.IsNullOrEmpty(repoFullName))
{
@@ -181,7 +180,7 @@ namespace GitHub.Runner.Worker
public Type ExtensionType => typeof(IActionCommandExtension);
public void ProcessCommand(IExecutionContext context, string line, ActionCommand command, ContainerInfo container)
public void ProcessCommand(IExecutionContext context, string line, ActionCommand command)
{
if (!command.Properties.TryGetValue(SetEnvCommandProperties.Name, out string envName) || string.IsNullOrEmpty(envName))
{
@@ -206,7 +205,7 @@ namespace GitHub.Runner.Worker
public Type ExtensionType => typeof(IActionCommandExtension);
public void ProcessCommand(IExecutionContext context, string line, ActionCommand command, ContainerInfo container)
public void ProcessCommand(IExecutionContext context, string line, ActionCommand command)
{
if (!command.Properties.TryGetValue(SetOutputCommandProperties.Name, out string outputName) || string.IsNullOrEmpty(outputName))
{
@@ -230,7 +229,7 @@ namespace GitHub.Runner.Worker
public Type ExtensionType => typeof(IActionCommandExtension);
public void ProcessCommand(IExecutionContext context, string line, ActionCommand command, ContainerInfo container)
public void ProcessCommand(IExecutionContext context, string line, ActionCommand command)
{
if (!command.Properties.TryGetValue(SaveStateCommandProperties.Name, out string stateName) || string.IsNullOrEmpty(stateName))
{
@@ -254,7 +253,7 @@ namespace GitHub.Runner.Worker
public Type ExtensionType => typeof(IActionCommandExtension);
public void ProcessCommand(IExecutionContext context, string line, ActionCommand command, ContainerInfo container)
public void ProcessCommand(IExecutionContext context, string line, ActionCommand command)
{
if (string.IsNullOrWhiteSpace(command.Data))
{
@@ -280,7 +279,7 @@ namespace GitHub.Runner.Worker
public Type ExtensionType => typeof(IActionCommandExtension);
public void ProcessCommand(IExecutionContext context, string line, ActionCommand command, ContainerInfo container)
public void ProcessCommand(IExecutionContext context, string line, ActionCommand command)
{
ArgUtil.NotNullOrEmpty(command.Data, "path");
context.PrependPath.RemoveAll(x => string.Equals(x, command.Data, StringComparison.CurrentCulture));
@@ -295,7 +294,7 @@ namespace GitHub.Runner.Worker
public Type ExtensionType => typeof(IActionCommandExtension);
public void ProcessCommand(IExecutionContext context, string line, ActionCommand command, ContainerInfo container)
public void ProcessCommand(IExecutionContext context, string line, ActionCommand command)
{
var file = command.Data;
@@ -307,9 +306,9 @@ namespace GitHub.Runner.Worker
}
// Translate file path back from container path
if (container != null)
if (context.Container != null)
{
file = container.TranslateToHostPath(file);
file = context.Container.TranslateToHostPath(file);
}
// Root the path
@@ -342,7 +341,7 @@ namespace GitHub.Runner.Worker
public Type ExtensionType => typeof(IActionCommandExtension);
public void ProcessCommand(IExecutionContext context, string line, ActionCommand command, ContainerInfo container)
public void ProcessCommand(IExecutionContext context, string line, ActionCommand command)
{
command.Properties.TryGetValue(RemoveMatcherCommandProperties.Owner, out string owner);
var file = command.Data;
@@ -370,9 +369,9 @@ namespace GitHub.Runner.Worker
else
{
// Translate file path back from container path
if (container != null)
if (context.Container != null)
{
file = container.TranslateToHostPath(file);
file = context.Container.TranslateToHostPath(file);
}
// Root the path
@@ -410,7 +409,7 @@ namespace GitHub.Runner.Worker
public Type ExtensionType => typeof(IActionCommandExtension);
public void ProcessCommand(IExecutionContext context, string inputLine, ActionCommand command, ContainerInfo container)
public void ProcessCommand(IExecutionContext context, string inputLine, ActionCommand command)
{
context.Debug(command.Data);
}
@@ -438,7 +437,7 @@ namespace GitHub.Runner.Worker
public Type ExtensionType => typeof(IActionCommandExtension);
public void ProcessCommand(IExecutionContext context, string inputLine, ActionCommand command, ContainerInfo container)
public void ProcessCommand(IExecutionContext context, string inputLine, ActionCommand command)
{
command.Properties.TryGetValue(IssueCommandProperties.File, out string file);
command.Properties.TryGetValue(IssueCommandProperties.Line, out string line);
@@ -455,10 +454,10 @@ namespace GitHub.Runner.Worker
{
issue.Category = "Code";
if (container != null)
if (context.Container != null)
{
// Translate file path back from container path
file = container.TranslateToHostPath(file);
file = context.Container.TranslateToHostPath(file);
command.Properties[IssueCommandProperties.File] = file;
}
@@ -518,7 +517,7 @@ namespace GitHub.Runner.Worker
public Type ExtensionType => typeof(IActionCommandExtension);
public void ProcessCommand(IExecutionContext context, string line, ActionCommand command, ContainerInfo container)
public void ProcessCommand(IExecutionContext context, string line, ActionCommand command)
{
var data = this is GroupCommandExtension ? command.Data : string.Empty;
context.Output($"##[{Command}]{data}");
@@ -532,7 +531,7 @@ namespace GitHub.Runner.Worker
public Type ExtensionType => typeof(IActionCommandExtension);
public void ProcessCommand(IExecutionContext context, string line, ActionCommand command, ContainerInfo container)
public void ProcessCommand(IExecutionContext context, string line, ActionCommand command)
{
ArgUtil.NotNullOrEmpty(command.Data, "value");

View File

@@ -33,7 +33,7 @@ namespace GitHub.Runner.Worker
{
private const int _defaultFileStreamBufferSize = 4096;
//81920 is the default used by System.IO.Stream.CopyTo and is under the large object heap threshold (85k).
//81920 is the default used by System.IO.Stream.CopyTo and is under the large object heap threshold (85k).
private const int _defaultCopyBufferSize = 81920;
private readonly Dictionary<Guid, ContainerInfo> _cachedActionContainers = new Dictionary<Guid, ContainerInfo>();
@@ -198,21 +198,14 @@ namespace GitHub.Runner.Worker
Trace.Info($"Load action that reference repository from '{actionDirectory}'");
definition.Directory = actionDirectory;
string manifestFile = Path.Combine(actionDirectory, Constants.Path.ActionManifestYmlFile);
string manifestFileYaml = Path.Combine(actionDirectory, Constants.Path.ActionManifestYamlFile);
string manifestFile = Path.Combine(actionDirectory, "action.yml");
string dockerFile = Path.Combine(actionDirectory, "Dockerfile");
string dockerFileLowerCase = Path.Combine(actionDirectory, "dockerfile");
if (File.Exists(manifestFile) || File.Exists(manifestFileYaml))
if (File.Exists(manifestFile))
{
var manifestManager = HostContext.GetService<IActionManifestManager>();
if (File.Exists(manifestFile))
{
definition.Data = manifestManager.Load(executionContext, manifestFile);
}
else
{
definition.Data = manifestManager.Load(executionContext, manifestFileYaml);
}
definition.Data = manifestManager.Load(executionContext, manifestFile);
Trace.Verbose($"Action friendly name: '{definition.Data.Name}'");
Trace.Verbose($"Action description: '{definition.Data.Description}'");
@@ -321,7 +314,7 @@ namespace GitHub.Runner.Worker
else
{
var fullPath = IOUtil.ResolvePath(actionDirectory, "."); // resolve full path without access filesystem.
throw new NotSupportedException($"Can't find 'action.yml', 'action.yaml' or 'Dockerfile' under '{fullPath}'. Did you forget to run actions/checkout before running your local action?");
throw new NotSupportedException($"Can't find 'action.yml' or 'Dockerfile' under '{fullPath}'. Did you forget to run actions/checkout before running your local action?");
}
}
else if (action.Reference.Type == Pipelines.ActionSourceType.Script)
@@ -484,7 +477,7 @@ namespace GitHub.Runner.Worker
int retryCount = 0;
// Allow up to 20 * 60s for any action to be downloaded from github graph.
// Allow up to 20 * 60s for any action to be downloaded from github graph.
int timeoutSeconds = 20 * 60;
while (retryCount < 3)
{
@@ -662,21 +655,12 @@ namespace GitHub.Runner.Worker
// find the docker file or action.yml file
var dockerFile = Path.Combine(actionEntryDirectory, "Dockerfile");
var dockerFileLowerCase = Path.Combine(actionEntryDirectory, "dockerfile");
var actionManifest = Path.Combine(actionEntryDirectory, Constants.Path.ActionManifestYmlFile);
var actionManifestYaml = Path.Combine(actionEntryDirectory, Constants.Path.ActionManifestYamlFile);
if (File.Exists(actionManifest) || File.Exists(actionManifestYaml))
var actionManifest = Path.Combine(actionEntryDirectory, "action.yml");
if (File.Exists(actionManifest))
{
executionContext.Debug($"action.yml for action: '{actionManifest}'.");
var manifestManager = HostContext.GetService<IActionManifestManager>();
ActionDefinitionData actionDefinitionData = null;
if (File.Exists(actionManifest))
{
actionDefinitionData = manifestManager.Load(executionContext, actionManifest);
}
else
{
actionDefinitionData = manifestManager.Load(executionContext, actionManifestYaml);
}
var actionDefinitionData = manifestManager.Load(executionContext, actionManifest);
if (actionDefinitionData.Execution.ExecutionType == ActionExecutionType.Container)
{
@@ -736,7 +720,7 @@ namespace GitHub.Runner.Worker
else
{
var fullPath = IOUtil.ResolvePath(actionEntryDirectory, "."); // resolve full path without access filesystem.
throw new InvalidOperationException($"Can't find 'action.yml', 'action.yaml' or 'Dockerfile' under '{fullPath}'. Did you forget to run actions/checkout before running your local action?");
throw new InvalidOperationException($"Can't find 'action.yml' or 'Dockerfile' under '{fullPath}'. Did you forget to run actions/checkout before running your local action?");
}
}
}

View File

@@ -32,7 +32,6 @@ namespace GitHub.Runner.Worker
public sealed class ActionManifestManager : RunnerService, IActionManifestManager
{
private TemplateSchema _actionManifestSchema;
private IReadOnlyList<String> _fileTable;
public override void Initialize(IHostContext hostContext)
{
@@ -62,9 +61,6 @@ namespace GitHub.Runner.Worker
// Get the file ID
var fileId = context.GetFileId(manifestFile);
_fileTable = context.GetFileTable();
// Read the file
var fileContent = File.ReadAllText(manifestFile);
using (var stringReader = new StringReader(fileContent))
{
@@ -120,7 +116,7 @@ namespace GitHub.Runner.Worker
if (actionDefinition.Execution == null)
{
executionContext.Debug($"Loaded action.yml file: {StringUtil.ConvertToJson(actionDefinition)}");
throw new ArgumentException($"Top level 'runs:' section is required for {manifestFile}");
throw new ArgumentException($"Top level 'run:' section is required for {manifestFile}");
}
else
{
@@ -269,15 +265,6 @@ namespace GitHub.Runner.Worker
}
}
// Add the file table
if (_fileTable?.Count > 0)
{
for (var i = 0 ; i < _fileTable.Count ; i++)
{
result.GetFileId(_fileTable[i]);
}
}
return result;
}
@@ -354,7 +341,7 @@ namespace GitHub.Runner.Worker
EntryPoint = entrypointToken?.Value,
Environment = envToken,
Cleanup = postEntrypointToken?.Value,
CleanupCondition = postIfToken?.Value ?? "always()"
CleanupCondition = postIfToken?.Value
};
}
}
@@ -370,7 +357,7 @@ namespace GitHub.Runner.Worker
{
Script = mainToken.Value,
Cleanup = postToken?.Value,
CleanupCondition = postIfToken?.Value ?? "always()"
CleanupCondition = postIfToken?.Value
};
}
}
@@ -428,5 +415,566 @@ namespace GitHub.Runner.Worker
}
}
}
/// <summary>
/// Converts a YAML file into a TemplateToken
/// </summary>
internal sealed class YamlObjectReader : IObjectReader
{
internal YamlObjectReader(
Int32? fileId,
TextReader input)
{
m_fileId = fileId;
m_parser = new Parser(input);
}
public Boolean AllowLiteral(out LiteralToken value)
{
if (EvaluateCurrent() is Scalar scalar)
{
// Tag specified
if (!string.IsNullOrEmpty(scalar.Tag))
{
// String tag
if (string.Equals(scalar.Tag, c_stringTag, StringComparison.Ordinal))
{
value = new StringToken(m_fileId, scalar.Start.Line, scalar.Start.Column, scalar.Value);
MoveNext();
return true;
}
// Not plain style
if (scalar.Style != ScalarStyle.Plain)
{
throw new NotSupportedException($"The scalar style '{scalar.Style}' on line {scalar.Start.Line} and column {scalar.Start.Column} is not valid with the tag '{scalar.Tag}'");
}
// Boolean, Float, Integer, or Null
switch (scalar.Tag)
{
case c_booleanTag:
value = ParseBoolean(scalar);
break;
case c_floatTag:
value = ParseFloat(scalar);
break;
case c_integerTag:
value = ParseInteger(scalar);
break;
case c_nullTag:
value = ParseNull(scalar);
break;
default:
throw new NotSupportedException($"Unexpected tag '{scalar.Tag}'");
}
MoveNext();
return true;
}
// Plain style, determine type using YAML 1.2 "core" schema https://yaml.org/spec/1.2/spec.html#id2804923
if (scalar.Style == ScalarStyle.Plain)
{
if (MatchNull(scalar, out var nullToken))
{
value = nullToken;
}
else if (MatchBoolean(scalar, out var booleanToken))
{
value = booleanToken;
}
else if (MatchInteger(scalar, out var numberToken) ||
MatchFloat(scalar, out numberToken))
{
value = numberToken;
}
else
{
value = new StringToken(m_fileId, scalar.Start.Line, scalar.Start.Column, scalar.Value);
}
MoveNext();
return true;
}
// Otherwise assume string
value = new StringToken(m_fileId, scalar.Start.Line, scalar.Start.Column, scalar.Value);
MoveNext();
return true;
}
value = default;
return false;
}
public Boolean AllowSequenceStart(out SequenceToken value)
{
if (EvaluateCurrent() is SequenceStart sequenceStart)
{
value = new SequenceToken(m_fileId, sequenceStart.Start.Line, sequenceStart.Start.Column);
MoveNext();
return true;
}
value = default;
return false;
}
public Boolean AllowSequenceEnd()
{
if (EvaluateCurrent() is SequenceEnd)
{
MoveNext();
return true;
}
return false;
}
public Boolean AllowMappingStart(out MappingToken value)
{
if (EvaluateCurrent() is MappingStart mappingStart)
{
value = new MappingToken(m_fileId, mappingStart.Start.Line, mappingStart.Start.Column);
MoveNext();
return true;
}
value = default;
return false;
}
public Boolean AllowMappingEnd()
{
if (EvaluateCurrent() is MappingEnd)
{
MoveNext();
return true;
}
return false;
}
/// <summary>
/// Consumes the last parsing events, which are expected to be DocumentEnd and StreamEnd.
/// </summary>
public void ValidateEnd()
{
if (EvaluateCurrent() is DocumentEnd)
{
MoveNext();
}
else
{
throw new InvalidOperationException("Expected document end parse event");
}
if (EvaluateCurrent() is StreamEnd)
{
MoveNext();
}
else
{
throw new InvalidOperationException("Expected stream end parse event");
}
if (MoveNext())
{
throw new InvalidOperationException("Expected end of parse events");
}
}
/// <summary>
/// Consumes the first parsing events, which are expected to be StreamStart and DocumentStart.
/// </summary>
public void ValidateStart()
{
if (EvaluateCurrent() != null)
{
throw new InvalidOperationException("Unexpected parser state");
}
if (!MoveNext())
{
throw new InvalidOperationException("Expected a parse event");
}
if (EvaluateCurrent() is StreamStart)
{
MoveNext();
}
else
{
throw new InvalidOperationException("Expected stream start parse event");
}
if (EvaluateCurrent() is DocumentStart)
{
MoveNext();
}
else
{
throw new InvalidOperationException("Expected document start parse event");
}
}
private ParsingEvent EvaluateCurrent()
{
if (m_current == null)
{
m_current = m_parser.Current;
if (m_current != null)
{
if (m_current is Scalar scalar)
{
// Verify not using achors
if (scalar.Anchor != null)
{
throw new InvalidOperationException($"Anchors are not currently supported. Remove the anchor '{scalar.Anchor}'");
}
}
else if (m_current is MappingStart mappingStart)
{
// Verify not using achors
if (mappingStart.Anchor != null)
{
throw new InvalidOperationException($"Anchors are not currently supported. Remove the anchor '{mappingStart.Anchor}'");
}
}
else if (m_current is SequenceStart sequenceStart)
{
// Verify not using achors
if (sequenceStart.Anchor != null)
{
throw new InvalidOperationException($"Anchors are not currently supported. Remove the anchor '{sequenceStart.Anchor}'");
}
}
else if (!(m_current is MappingEnd) &&
!(m_current is SequenceEnd) &&
!(m_current is DocumentStart) &&
!(m_current is DocumentEnd) &&
!(m_current is StreamStart) &&
!(m_current is StreamEnd))
{
throw new InvalidOperationException($"Unexpected parsing event type: {m_current.GetType().Name}");
}
}
}
return m_current;
}
private Boolean MoveNext()
{
m_current = null;
return m_parser.MoveNext();
}
private BooleanToken ParseBoolean(Scalar scalar)
{
if (MatchBoolean(scalar, out var token))
{
return token;
}
ThrowInvalidValue(scalar, c_booleanTag); // throws
return default;
}
private NumberToken ParseFloat(Scalar scalar)
{
if (MatchFloat(scalar, out var token))
{
return token;
}
ThrowInvalidValue(scalar, c_floatTag); // throws
return default;
}
private NumberToken ParseInteger(Scalar scalar)
{
if (MatchInteger(scalar, out var token))
{
return token;
}
ThrowInvalidValue(scalar, c_integerTag); // throws
return default;
}
private NullToken ParseNull(Scalar scalar)
{
if (MatchNull(scalar, out var token))
{
return token;
}
ThrowInvalidValue(scalar, c_nullTag); // throws
return default;
}
private Boolean MatchBoolean(
Scalar scalar,
out BooleanToken value)
{
// YAML 1.2 "core" schema https://yaml.org/spec/1.2/spec.html#id2804923
switch (scalar.Value ?? string.Empty)
{
case "true":
case "True":
case "TRUE":
value = new BooleanToken(m_fileId, scalar.Start.Line, scalar.Start.Column, true);
return true;
case "false":
case "False":
case "FALSE":
value = new BooleanToken(m_fileId, scalar.Start.Line, scalar.Start.Column, false);
return true;
}
value = default;
return false;
}
private Boolean MatchFloat(
Scalar scalar,
out NumberToken value)
{
// YAML 1.2 "core" schema https://yaml.org/spec/1.2/spec.html#id2804923
var str = scalar.Value;
if (!string.IsNullOrEmpty(str))
{
// Check for [-+]?(\.inf|\.Inf|\.INF)|\.nan|\.NaN|\.NAN
switch (str)
{
case ".inf":
case ".Inf":
case ".INF":
case "+.inf":
case "+.Inf":
case "+.INF":
value = new NumberToken(m_fileId, scalar.Start.Line, scalar.Start.Column, Double.PositiveInfinity);
return true;
case "-.inf":
case "-.Inf":
case "-.INF":
value = new NumberToken(m_fileId, scalar.Start.Line, scalar.Start.Column, Double.NegativeInfinity);
return true;
case ".nan":
case ".NaN":
case ".NAN":
value = new NumberToken(m_fileId, scalar.Start.Line, scalar.Start.Column, Double.NaN);
return true;
}
// Otherwise check [-+]?(\.[0-9]+|[0-9]+(\.[0-9]*)?)([eE][-+]?[0-9]+)?
// Skip leading sign
var index = str[0] == '-' || str[0] == '+' ? 1 : 0;
// Check for integer portion
var length = str.Length;
var hasInteger = false;
while (index < length && str[index] >= '0' && str[index] <= '9')
{
hasInteger = true;
index++;
}
// Check for decimal point
var hasDot = false;
if (index < length && str[index] == '.')
{
hasDot = true;
index++;
}
// Check for decimal portion
var hasDecimal = false;
while (index < length && str[index] >= '0' && str[index] <= '9')
{
hasDecimal = true;
index++;
}
// Check [-+]?(\.[0-9]+|[0-9]+(\.[0-9]*)?)
if ((hasDot && hasDecimal) || hasInteger)
{
// Check for end
if (index == length)
{
// Try parse
if (Double.TryParse(str, NumberStyles.AllowLeadingSign | NumberStyles.AllowDecimalPoint, CultureInfo.InvariantCulture, out var doubleValue))
{
value = new NumberToken(m_fileId, scalar.Start.Line, scalar.Start.Column, doubleValue);
return true;
}
// Otherwise exceeds range
else
{
ThrowInvalidValue(scalar, c_floatTag); // throws
}
}
// Check [eE][-+]?[0-9]
else if (index < length && (str[index] == 'e' || str[index] == 'E'))
{
index++;
// Skip sign
if (index < length && (str[index] == '-' || str[index] == '+'))
{
index++;
}
// Check for exponent
var hasExponent = false;
while (index < length && str[index] >= '0' && str[index] <= '9')
{
hasExponent = true;
index++;
}
// Check for end
if (hasExponent && index == length)
{
// Try parse
if (Double.TryParse(str, NumberStyles.AllowLeadingSign | NumberStyles.AllowDecimalPoint | NumberStyles.AllowExponent, CultureInfo.InvariantCulture, out var doubleValue))
{
value = new NumberToken(m_fileId, scalar.Start.Line, scalar.Start.Column, (Double)doubleValue);
return true;
}
// Otherwise exceeds range
else
{
ThrowInvalidValue(scalar, c_floatTag); // throws
}
}
}
}
}
value = default;
return false;
}
private Boolean MatchInteger(
Scalar scalar,
out NumberToken value)
{
// YAML 1.2 "core" schema https://yaml.org/spec/1.2/spec.html#id2804923
var str = scalar.Value;
if (!string.IsNullOrEmpty(str))
{
// Check for [0-9]+
var firstChar = str[0];
if (firstChar >= '0' && firstChar <= '9' &&
str.Skip(1).All(x => x >= '0' && x <= '9'))
{
// Try parse
if (Double.TryParse(str, NumberStyles.None, CultureInfo.InvariantCulture, out var doubleValue))
{
value = new NumberToken(m_fileId, scalar.Start.Line, scalar.Start.Column, doubleValue);
return true;
}
// Otherwise exceeds range
ThrowInvalidValue(scalar, c_integerTag); // throws
}
// Check for (-|+)[0-9]+
else if ((firstChar == '-' || firstChar == '+') &&
str.Length > 1 &&
str.Skip(1).All(x => x >= '0' && x <= '9'))
{
// Try parse
if (Double.TryParse(str, NumberStyles.AllowLeadingSign, CultureInfo.InvariantCulture, out var doubleValue))
{
value = new NumberToken(m_fileId, scalar.Start.Line, scalar.Start.Column, doubleValue);
return true;
}
// Otherwise exceeds range
ThrowInvalidValue(scalar, c_integerTag); // throws
}
// Check for 0x[0-9a-fA-F]+
else if (firstChar == '0' &&
str.Length > 2 &&
str[1] == 'x' &&
str.Skip(2).All(x => (x >= '0' && x <= '9') || (x >= 'a' && x <= 'f') || (x >= 'A' && x <= 'F')))
{
// Try parse
if (Int32.TryParse(str.Substring(2), NumberStyles.AllowHexSpecifier, CultureInfo.InvariantCulture, out var integerValue))
{
value = new NumberToken(m_fileId, scalar.Start.Line, scalar.Start.Column, integerValue);
return true;
}
// Otherwise exceeds range
ThrowInvalidValue(scalar, c_integerTag); // throws
}
// Check for 0o[0-9]+
else if (firstChar == '0' &&
str.Length > 2 &&
str[1] == 'o' &&
str.Skip(2).All(x => x >= '0' && x <= '7'))
{
// Try parse
var integerValue = default(Int32);
try
{
integerValue = Convert.ToInt32(str.Substring(2), 8);
}
// Otherwise exceeds range
catch (Exception)
{
ThrowInvalidValue(scalar, c_integerTag); // throws
}
value = new NumberToken(m_fileId, scalar.Start.Line, scalar.Start.Column, integerValue);
return true;
}
}
value = default;
return false;
}
private Boolean MatchNull(
Scalar scalar,
out NullToken value)
{
// YAML 1.2 "core" schema https://yaml.org/spec/1.2/spec.html#id2804923
switch (scalar.Value ?? string.Empty)
{
case "":
case "null":
case "Null":
case "NULL":
case "~":
value = new NullToken(m_fileId, scalar.Start.Line, scalar.Start.Column);
return true;
}
value = default;
return false;
}
private void ThrowInvalidValue(
Scalar scalar,
String tag)
{
throw new NotSupportedException($"The value '{scalar.Value}' on line {scalar.Start.Line} and column {scalar.Start.Column} is invalid for the type '{scalar.Tag}'");
}
private const String c_booleanTag = "tag:yaml.org,2002:bool";
private const String c_floatTag = "tag:yaml.org,2002:float";
private const String c_integerTag = "tag:yaml.org,2002:int";
private const String c_nullTag = "tag:yaml.org,2002:null";
private const String c_stringTag = "tag:yaml.org,2002:string";
private readonly Int32? m_fileId;
private readonly Parser m_parser;
private ParsingEvent m_current;
}
}

View File

@@ -141,7 +141,9 @@ namespace GitHub.Runner.Worker
// Load the inputs.
ExecutionContext.Debug("Loading inputs");
var templateEvaluator = ExecutionContext.ToPipelineTemplateEvaluator();
var templateTrace = ExecutionContext.ToTemplateTraceWriter();
var schema = new PipelineTemplateSchemaFactory().CreateSchema();
var templateEvaluator = new PipelineTemplateEvaluator(templateTrace, schema);
var inputs = templateEvaluator.EvaluateStepInputs(Action.Inputs, ExecutionContext.ExpressionValues);
foreach (KeyValuePair<string, string> input in inputs)
@@ -177,15 +179,17 @@ namespace GitHub.Runner.Worker
ExecutionContext.Debug("Loading env");
var environment = new Dictionary<String, String>(VarUtil.EnvironmentVariableKeyComparer);
#if OS_WINDOWS
var envContext = ExecutionContext.ExpressionValues["env"] as DictionaryContextData;
#else
var envContext = ExecutionContext.ExpressionValues["env"] as CaseSensitiveDictionaryContextData;
#endif
// Apply environment from env context, env context contains job level env and action's evn block
foreach (var env in envContext)
// Apply environment set using ##[set-env] first since these are job level env
foreach (var env in ExecutionContext.EnvironmentVariables)
{
environment[env.Key] = env.Value.ToString();
environment[env.Key] = env.Value ?? string.Empty;
}
// Apply action's env block later.
var actionEnvironment = templateEvaluator.EvaluateStepEnvironment(Action.Environment, ExecutionContext.ExpressionValues, VarUtil.EnvironmentVariableKeyComparer);
foreach (var env in actionEnvironment)
{
environment[env.Key] = env.Value ?? string.Empty;
}
// Apply action's intra-action state at last
@@ -293,7 +297,8 @@ namespace GitHub.Runner.Worker
return displayName;
}
// Try evaluating fully
var templateEvaluator = context.ToPipelineTemplateEvaluator();
var schema = new PipelineTemplateSchemaFactory().CreateSchema();
var templateEvaluator = new PipelineTemplateEvaluator(context.ToTemplateTraceWriter(), schema);
try
{
didFullyEvaluate = templateEvaluator.TryEvaluateStepDisplayName(tokenToParse, contextData, out displayName);

View File

@@ -2,9 +2,9 @@
using System.Collections.Generic;
using System.IO;
using GitHub.Runner.Common.Util;
using Pipelines = GitHub.DistributedTask.Pipelines;
using GitHub.Runner.Common;
using GitHub.Runner.Sdk;
using Pipelines = GitHub.DistributedTask.Pipelines;
namespace GitHub.Runner.Worker.Container
{
@@ -19,6 +19,7 @@ namespace GitHub.Runner.Worker.Container
public ContainerInfo()
{
}
public ContainerInfo(IHostContext hostContext, Pipelines.JobContainer container, bool isJobContainer = true, string networkAlias = null)
@@ -63,8 +64,6 @@ namespace GitHub.Runner.Worker.Container
UserMountVolumes[volume] = volume;
}
}
UpdateWebProxyEnv(hostContext.WebProxy);
}
public string ContainerId { get; set; }
@@ -224,26 +223,6 @@ namespace GitHub.Runner.Worker.Container
{
_pathMappings.Insert(0, new PathMapping(hostCommonPath, containerCommonPath));
}
private void UpdateWebProxyEnv(RunnerWebProxy webProxy)
{
// Set common forms of proxy variables if configured in Runner and not set directly by container.env
if (!String.IsNullOrEmpty(webProxy.HttpProxyAddress))
{
ContainerEnvironmentVariables.TryAdd("HTTP_PROXY", webProxy.HttpProxyAddress);
ContainerEnvironmentVariables.TryAdd("http_proxy", webProxy.HttpProxyAddress);
}
if (!String.IsNullOrEmpty(webProxy.HttpsProxyAddress))
{
ContainerEnvironmentVariables.TryAdd("HTTPS_PROXY", webProxy.HttpsProxyAddress);
ContainerEnvironmentVariables.TryAdd("https_proxy", webProxy.HttpsProxyAddress);
}
if (!String.IsNullOrEmpty(webProxy.NoProxyString))
{
ContainerEnvironmentVariables.TryAdd("NO_PROXY", webProxy.NoProxyString);
ContainerEnvironmentVariables.TryAdd("no_proxy", webProxy.NoProxyString);
}
}
}
public class MountVolume

View File

@@ -61,11 +61,16 @@ namespace GitHub.Runner.Worker
{
throw new NotSupportedException("Container feature is not supported when runner is already running inside container.");
}
#else
var initProcessCgroup = File.ReadLines("/proc/1/cgroup");
if (initProcessCgroup.Any(x => x.IndexOf(":/docker/", StringComparison.OrdinalIgnoreCase) >= 0))
#else
var path = "/proc/1/cgroup";
// OSX does not have this file, but you cannot run OSX as a base image for docker containers currently.
if (File.Exists(path))
{
throw new NotSupportedException("Container feature is not supported when runner is already running inside container.");
var initProcessCgroup = File.ReadLines(path);
if (initProcessCgroup.Any(x => x.IndexOf(":/docker/", StringComparison.OrdinalIgnoreCase) >= 0))
{
throw new NotSupportedException("Container feature is not supported when runner is already running inside container.");
}
}
#endif

View File

@@ -11,7 +11,6 @@ using GitHub.Runner.Worker.Container;
using GitHub.Services.WebApi;
using GitHub.DistributedTask.Pipelines;
using GitHub.DistributedTask.Pipelines.ContextData;
using GitHub.DistributedTask.Pipelines.ObjectTemplating;
using GitHub.DistributedTask.WebApi;
using GitHub.Runner.Common.Util;
using GitHub.Runner.Common;
@@ -21,7 +20,6 @@ using System.Text;
using System.Collections;
using ObjectTemplating = GitHub.DistributedTask.ObjectTemplating;
using Pipelines = GitHub.DistributedTask.Pipelines;
using GitHub.DistributedTask.Expressions2;
namespace GitHub.Runner.Worker
{
@@ -50,7 +48,6 @@ namespace GitHub.Runner.Worker
HashSet<string> OutputVariables { get; }
IDictionary<String, String> EnvironmentVariables { get; }
IDictionary<String, ContextScope> Scopes { get; }
IList<String> FileTable { get; }
StepsContext StepsContext { get; }
DictionaryContextData ExpressionValues { get; }
List<string> PrependPath { get; }
@@ -143,7 +140,6 @@ namespace GitHub.Runner.Worker
public HashSet<string> OutputVariables => _outputvariables;
public IDictionary<String, String> EnvironmentVariables { get; private set; }
public IDictionary<String, ContextScope> Scopes { get; private set; }
public IList<String> FileTable { get; private set; }
public StepsContext StepsContext { get; private set; }
public DictionaryContextData ExpressionValues { get; } = new DictionaryContextData();
public bool WriteDebug { get; private set; }
@@ -269,7 +265,6 @@ namespace GitHub.Runner.Worker
}
child.EnvironmentVariables = EnvironmentVariables;
child.Scopes = Scopes;
child.FileTable = FileTable;
child.StepsContext = StepsContext;
foreach (var pair in ExpressionValues)
{
@@ -573,15 +568,6 @@ namespace GitHub.Runner.Worker
}
}
// File table
FileTable = new List<String>(message.FileTable ?? new string[0]);
// Expression functions
if (Variables.GetBoolean("System.HashFilesV2") == true)
{
ExpressionConstants.UpdateFunction<Handlers.HashFiles>("hashFiles", 1, byte.MaxValue);
}
// Expression values
if (message.ContextData?.Count > 0)
{
@@ -893,13 +879,6 @@ namespace GitHub.Runner.Worker
}
}
public static PipelineTemplateEvaluator ToPipelineTemplateEvaluator(this IExecutionContext context)
{
var templateTrace = context.ToTemplateTraceWriter();
var schema = new PipelineTemplateSchemaFactory().CreateSchema();
return new PipelineTemplateEvaluator(templateTrace, schema, context.FileTable);
}
public static ObjectTemplating.ITraceWriter ToTemplateTraceWriter(this IExecutionContext context)
{
return new TemplateTraceWriter(context);

View File

@@ -1,126 +0,0 @@
using System;
using System.IO;
using GitHub.DistributedTask.Expressions2.Sdk;
using GitHub.DistributedTask.Pipelines.ContextData;
using GitHub.DistributedTask.Pipelines.ObjectTemplating;
using GitHub.Runner.Sdk;
using System.Reflection;
using System.Threading;
using System.Collections.Generic;
namespace GitHub.Runner.Worker.Handlers
{
public class FunctionTrace : ITraceWriter
{
private GitHub.DistributedTask.Expressions2.ITraceWriter _trace;
public FunctionTrace(GitHub.DistributedTask.Expressions2.ITraceWriter trace)
{
_trace = trace;
}
public void Info(string message)
{
_trace.Info(message);
}
public void Verbose(string message)
{
_trace.Info(message);
}
}
public sealed class HashFiles : Function
{
protected sealed override Object EvaluateCore(
EvaluationContext context,
out ResultMemory resultMemory)
{
resultMemory = null;
var templateContext = context.State as DistributedTask.ObjectTemplating.TemplateContext;
ArgUtil.NotNull(templateContext, nameof(templateContext));
templateContext.ExpressionValues.TryGetValue(PipelineTemplateConstants.GitHub, out var githubContextData);
ArgUtil.NotNull(githubContextData, nameof(githubContextData));
var githubContext = githubContextData as DictionaryContextData;
ArgUtil.NotNull(githubContext, nameof(githubContext));
githubContext.TryGetValue(PipelineTemplateConstants.Workspace, out var workspace);
var workspaceData = workspace as StringContextData;
ArgUtil.NotNull(workspaceData, nameof(workspaceData));
string githubWorkspace = workspaceData.Value;
bool followSymlink = false;
List<string> patterns = new List<string>();
var firstParameter = true;
foreach (var parameter in Parameters)
{
var parameterString = parameter.Evaluate(context).ConvertToString();
if (firstParameter)
{
firstParameter = false;
if (parameterString.StartsWith("--"))
{
if (string.Equals(parameterString, "--follow-symbolic-links", StringComparison.OrdinalIgnoreCase))
{
followSymlink = true;
continue;
}
else
{
throw new ArgumentOutOfRangeException($"Invalid glob option {parameterString}, avaliable option: '--follow-symbolic-links'.");
}
}
}
patterns.Add(parameterString);
}
context.Trace.Info($"Search root directory: '{githubWorkspace}'");
context.Trace.Info($"Search pattern: '{string.Join(", ", patterns)}'");
string binDir = Path.GetDirectoryName(Assembly.GetEntryAssembly().Location);
string runnerRoot = new DirectoryInfo(binDir).Parent.FullName;
string node = Path.Combine(runnerRoot, "externals", "node12", "bin", $"node{IOUtil.ExeExtension}");
string hashFilesScript = Path.Combine(binDir, "hashFiles");
var hashResult = string.Empty;
var p = new ProcessInvoker(new FunctionTrace(context.Trace));
p.ErrorDataReceived += ((_, data) =>
{
if (!string.IsNullOrEmpty(data.Data) && data.Data.StartsWith("__OUTPUT__") && data.Data.EndsWith("__OUTPUT__"))
{
hashResult = data.Data.Substring(10, data.Data.Length - 20);
context.Trace.Info($"Hash result: '{hashResult}'");
}
else
{
context.Trace.Info(data.Data);
}
});
p.OutputDataReceived += ((_, data) =>
{
context.Trace.Info(data.Data);
});
var env = new Dictionary<string, string>();
if (followSymlink)
{
env["followSymbolicLinks"] = "true";
}
env["patterns"] = string.Join(Environment.NewLine, patterns);
int exitCode = p.ExecuteAsync(workingDirectory: githubWorkspace,
fileName: node,
arguments: $"\"{hashFilesScript.Replace("\"", "\\\"")}\"",
environment: env,
requireExitCodeZero: false,
cancellationToken: new CancellationTokenSource(TimeSpan.FromSeconds(120)).Token).GetAwaiter().GetResult();
if (exitCode != 0)
{
throw new InvalidOperationException($"hashFiles('{ExpressionUtility.StringEscape(string.Join(", ", patterns))}') failed. Fail to hash files under directory '{githubWorkspace}'");
}
return hashResult;
}
}
}

View File

@@ -189,13 +189,13 @@ namespace GitHub.Runner.Worker.Handlers
container.ContainerEnvironmentVariables[variable.Key] = container.TranslateToContainerPath(variable.Value);
}
using (var stdoutManager = new OutputManager(ExecutionContext, ActionCommandManager, container))
using (var stderrManager = new OutputManager(ExecutionContext, ActionCommandManager, container))
using (var stdoutManager = new OutputManager(ExecutionContext, ActionCommandManager))
using (var stderrManager = new OutputManager(ExecutionContext, ActionCommandManager))
{
var runExitCode = await dockerManger.DockerRun(ExecutionContext, container, stdoutManager.OnDataReceived, stderrManager.OnDataReceived);
ExecutionContext.Debug($"Docker Action run completed with exit code {runExitCode}");
if (runExitCode != 0)
{
ExecutionContext.Error($"Docker run failed with exit code {runExitCode}");
ExecutionContext.Result = TaskResult.Failed;
}
}

View File

@@ -122,9 +122,9 @@ namespace GitHub.Runner.Worker.Handlers
else
{
var exitCode = await step;
ExecutionContext.Debug($"Node Action run completed with exit code {exitCode}");
if (exitCode != 0)
{
ExecutionContext.Error($"Node run failed with exit code {exitCode}");
ExecutionContext.Result = TaskResult.Failed;
}
}

View File

@@ -6,7 +6,6 @@ using System.Linq;
using System.Text.RegularExpressions;
using GitHub.Runner.Common;
using GitHub.Runner.Sdk;
using GitHub.Runner.Worker.Container;
using DTWebApi = GitHub.DistributedTask.WebApi;
namespace GitHub.Runner.Worker.Handlers
@@ -18,7 +17,6 @@ namespace GitHub.Runner.Worker.Handlers
private const string _timeoutKey = "GITHUB_ACTIONS_RUNNER_ISSUE_MATCHER_TIMEOUT";
private static readonly Regex _colorCodeRegex = new Regex(@"\x0033\[[0-9;]*m?", RegexOptions.Compiled | RegexOptions.CultureInvariant);
private readonly IActionCommandManager _commandManager;
private readonly ContainerInfo _container;
private readonly IExecutionContext _executionContext;
private readonly int _failsafe = 50;
private readonly object _matchersLock = new object();
@@ -27,11 +25,10 @@ namespace GitHub.Runner.Worker.Handlers
// Mapping that indicates whether a directory belongs to the workflow repository
private readonly Dictionary<string, string> _directoryMap = new Dictionary<string, string>();
public OutputManager(IExecutionContext executionContext, IActionCommandManager commandManager, ContainerInfo container = null)
public OutputManager(IExecutionContext executionContext, IActionCommandManager commandManager)
{
_executionContext = executionContext;
_commandManager = commandManager;
_container = container ?? executionContext.Container;
// Recursion failsafe (test override)
var failsafeString = Environment.GetEnvironmentVariable("RUNNER_TEST_GET_REPOSITORY_PATH_FAILSAFE");
@@ -84,7 +81,7 @@ namespace GitHub.Runner.Worker.Handlers
{
// This does not need to be inside of a critical section.
// The logging queues and command handlers are thread-safe.
if (_commandManager.TryProcessCommand(_executionContext, line, _container))
if (_commandManager.TryProcessCommand(_executionContext, line))
{
return;
}
@@ -260,7 +257,6 @@ namespace GitHub.Runner.Worker.Handlers
if (!string.IsNullOrWhiteSpace(match.File))
{
var file = match.File;
var translate = _container != null;
// Root using fromPath
if (!string.IsNullOrWhiteSpace(match.FromPath) && !Path.IsPathFullyQualified(file))
@@ -279,19 +275,11 @@ namespace GitHub.Runner.Worker.Handlers
ArgUtil.NotNullOrEmpty(workspace, "workspace");
file = Path.Combine(workspace, file);
translate = false;
}
// Remove relative pathing and normalize slashes
file = Path.GetFullPath(file);
// Translate to host
if (translate)
{
file = _container.TranslateToHostPath(file);
file = Path.GetFullPath(file);
}
// Check whether the file exists
if (File.Exists(file))
{

View File

@@ -78,7 +78,9 @@ namespace GitHub.Runner.Worker
// Evaluate the job-level environment variables
context.Debug("Evaluating job-level environment variables");
var templateEvaluator = context.ToPipelineTemplateEvaluator();
var templateTrace = context.ToTemplateTraceWriter();
var schema = new PipelineTemplateSchemaFactory().CreateSchema();
var templateEvaluator = new PipelineTemplateEvaluator(templateTrace, schema);
foreach (var token in message.EnvironmentVariables)
{
var environmentVariables = templateEvaluator.EvaluateStepEnvironment(token, jobContext.ExpressionValues, VarUtil.EnvironmentVariableKeyComparer);

View File

@@ -107,11 +107,6 @@ namespace GitHub.Runner.Worker
return await CompleteJobAsync(jobServer, jobContext, message, TaskResult.Failed);
}
if (jobContext.WriteDebug)
{
jobContext.SetRunnerContext("debug", "1");
}
jobContext.SetRunnerContext("os", VarUtil.OS);
string toolsDirectory = HostContext.GetDirectory(WellKnownDirectory.Tools);

View File

@@ -76,36 +76,15 @@ namespace GitHub.Runner.Worker
// Start
step.ExecutionContext.Start();
// Set GITHUB_ACTION
if (step is IActionRunner actionStep)
{
step.ExecutionContext.SetGitHubContext("action", actionStep.Action.Name);
}
// Initialize scope
if (InitializeScope(step, scopeInputs))
{
// Populate env context for each step
Trace.Info("Initialize Env context for step");
#if OS_WINDOWS
var envContext = new DictionaryContextData();
#else
var envContext = new CaseSensitiveDictionaryContextData();
#endif
step.ExecutionContext.ExpressionValues["env"] = envContext;
foreach (var pair in step.ExecutionContext.EnvironmentVariables)
{
envContext[pair.Key] = new StringContextData(pair.Value ?? string.Empty);
}
if (step is IActionRunner actionStep)
{
// Set GITHUB_ACTION
step.ExecutionContext.SetGitHubContext("action", actionStep.Action.Name);
// Evaluate and merge action's env block to env context
var templateEvaluator = step.ExecutionContext.ToPipelineTemplateEvaluator();
var actionEnvironment = templateEvaluator.EvaluateStepEnvironment(actionStep.Action.Environment, step.ExecutionContext.ExpressionValues, VarUtil.EnvironmentVariableKeyComparer);
foreach (var env in actionEnvironment)
{
envContext[env.Key] = new StringContextData(env.Value ?? string.Empty);
}
}
var expressionManager = HostContext.GetService<IExpressionManager>();
try
{
@@ -245,7 +224,7 @@ namespace GitHub.Runner.Worker
// Set the timeout
var timeoutMinutes = 0;
var templateEvaluator = step.ExecutionContext.ToPipelineTemplateEvaluator();
var templateEvaluator = CreateTemplateEvaluator(step.ExecutionContext);
try
{
timeoutMinutes = templateEvaluator.EvaluateStepTimeout(step.Timeout, step.ExecutionContext.ExpressionValues);
@@ -387,7 +366,7 @@ namespace GitHub.Runner.Worker
executionContext.Debug($"Initializing scope '{scope.Name}'");
executionContext.ExpressionValues["steps"] = stepsContext.GetScope(scope.ParentName);
executionContext.ExpressionValues["inputs"] = !String.IsNullOrEmpty(scope.ParentName) ? scopeInputs[scope.ParentName] : null;
var templateEvaluator = executionContext.ToPipelineTemplateEvaluator();
var templateEvaluator = CreateTemplateEvaluator(executionContext);
var inputs = default(DictionaryContextData);
try
{
@@ -443,7 +422,7 @@ namespace GitHub.Runner.Worker
executionContext.Debug($"Finalizing scope '{scope.Name}'");
executionContext.ExpressionValues["steps"] = stepsContext.GetScope(scope.Name);
executionContext.ExpressionValues["inputs"] = null;
var templateEvaluator = executionContext.ToPipelineTemplateEvaluator();
var templateEvaluator = CreateTemplateEvaluator(executionContext);
var outputs = default(DictionaryContextData);
try
{
@@ -475,5 +454,12 @@ namespace GitHub.Runner.Worker
executionContext.Complete(result, resultCode: resultCode);
}
private PipelineTemplateEvaluator CreateTemplateEvaluator(IExecutionContext executionContext)
{
var templateTrace = executionContext.ToTemplateTraceWriter();
var schema = new PipelineTemplateSchemaFactory().CreateSchema();
return new PipelineTemplateEvaluator(templateTrace, schema);
}
}
}

View File

@@ -5,7 +5,7 @@ using GitHub.DistributedTask.Expressions2.Sdk.Functions;
namespace GitHub.DistributedTask.Expressions2
{
public static class ExpressionConstants
internal static class ExpressionConstants
{
static ExpressionConstants()
{
@@ -24,12 +24,6 @@ namespace GitHub.DistributedTask.Expressions2
WellKnownFunctions.Add(name, new FunctionInfo<T>(name, minParameters, maxParameters));
}
public static void UpdateFunction<T>(String name, Int32 minParameters, Int32 maxParameters)
where T : Function, new()
{
WellKnownFunctions[name] = new FunctionInfo<T>(name, minParameters, maxParameters);
}
internal static readonly String False = "false";
internal static readonly String Infinity = "Infinity";
internal static readonly Int32 MaxDepth = 50;

View File

@@ -184,7 +184,6 @@ namespace GitHub.DistributedTask.ObjectTemplating
id = FileIds.Count + 1;
FileIds.Add(file, id);
FileNames.Add(file);
Memory.AddBytes(file);
}
return id;
@@ -192,12 +191,7 @@ namespace GitHub.DistributedTask.ObjectTemplating
internal String GetFileName(Int32 fileId)
{
return FileNames.Count >= fileId ? FileNames[fileId - 1] : null;
}
internal IReadOnlyList<String> GetFileTable()
{
return FileNames.AsReadOnly();
return FileNames[fileId - 1];
}
private String GetErrorPrefix(
@@ -205,9 +199,9 @@ namespace GitHub.DistributedTask.ObjectTemplating
Int32? line,
Int32? column)
{
var fileName = fileId.HasValue ? GetFileName(fileId.Value) : null;
if (!String.IsNullOrEmpty(fileName))
if (fileId != null)
{
var fileName = GetFileName(fileId.Value);
if (line != null && column != null)
{
return $"{fileName} {TemplateStrings.LineColumn(line, column)}:";

View File

@@ -30,14 +30,14 @@ namespace GitHub.DistributedTask.ObjectTemplating.Tokens
Column = column;
}
[DataMember(Name = "file", EmitDefaultValue = false)]
internal Int32? FileId { get; private set; }
[IgnoreDataMember]
internal Int32? FileId { get; set; }
[DataMember(Name = "line", EmitDefaultValue = false)]
internal Int32? Line { get; private set; }
internal Int32? Line { get; }
[DataMember(Name = "col", EmitDefaultValue = false)]
internal Int32? Column { get; private set; }
internal Int32? Column { get; }
[DataMember(Name = "type", EmitDefaultValue = false)]
internal Int32 Type { get; }

View File

@@ -115,12 +115,13 @@ namespace GitHub.DistributedTask.ObjectTemplating.Tokens
Object value,
JsonSerializer serializer)
{
base.WriteJson(writer, value, serializer);
if (value is TemplateToken token)
{
switch (token.Type)
{
case TokenType.Null:
if (token.FileId == null && token.Line == null && token.Column == null)
if (token.Line == null && token.Column == null)
{
writer.WriteNull();
}
@@ -129,17 +130,12 @@ namespace GitHub.DistributedTask.ObjectTemplating.Tokens
writer.WriteStartObject();
writer.WritePropertyName("type");
writer.WriteValue(token.Type);
if (token.FileId != null)
{
writer.WritePropertyName("file");
writer.WriteValue(token.FileId);
}
if (token.Line != null)
{
writer.WritePropertyName("line");
writer.WriteValue(token.Line);
}
if (token.Column != null)
if (token.Line != null)
{
writer.WritePropertyName("col");
writer.WriteValue(token.Column);
@@ -150,7 +146,7 @@ namespace GitHub.DistributedTask.ObjectTemplating.Tokens
case TokenType.Boolean:
var booleanToken = token as BooleanToken;
if (token.FileId == null && token.Line == null && token.Column == null)
if (token.Line == null && token.Column == null)
{
writer.WriteValue(booleanToken.Value);
}
@@ -159,17 +155,12 @@ namespace GitHub.DistributedTask.ObjectTemplating.Tokens
writer.WriteStartObject();
writer.WritePropertyName("type");
writer.WriteValue(token.Type);
if (token.FileId != null)
{
writer.WritePropertyName("file");
writer.WriteValue(token.FileId);
}
if (token.Line != null)
{
writer.WritePropertyName("line");
writer.WriteValue(token.Line);
}
if (token.Column != null)
if (token.Line != null)
{
writer.WritePropertyName("col");
writer.WriteValue(token.Column);
@@ -182,7 +173,7 @@ namespace GitHub.DistributedTask.ObjectTemplating.Tokens
case TokenType.Number:
var numberToken = token as NumberToken;
if (token.FileId == null && token.Line == null && token.Column == null)
if (token.Line == null && token.Column == null)
{
writer.WriteValue(numberToken.Value);
}
@@ -191,17 +182,12 @@ namespace GitHub.DistributedTask.ObjectTemplating.Tokens
writer.WriteStartObject();
writer.WritePropertyName("type");
writer.WriteValue(token.Type);
if (token.FileId != null)
{
writer.WritePropertyName("file");
writer.WriteValue(token.FileId);
}
if (token.Line != null)
{
writer.WritePropertyName("line");
writer.WriteValue(token.Line);
}
if (token.Column != null)
if (token.Line != null)
{
writer.WritePropertyName("col");
writer.WriteValue(token.Column);
@@ -214,7 +200,7 @@ namespace GitHub.DistributedTask.ObjectTemplating.Tokens
case TokenType.String:
var stringToken = token as StringToken;
if (token.FileId == null && token.Line == null && token.Column == null)
if (token.Line == null && token.Column == null)
{
writer.WriteValue(stringToken.Value);
}
@@ -223,17 +209,12 @@ namespace GitHub.DistributedTask.ObjectTemplating.Tokens
writer.WriteStartObject();
writer.WritePropertyName("type");
writer.WriteValue(token.Type);
if (token.FileId != null)
{
writer.WritePropertyName("file");
writer.WriteValue(token.FileId);
}
if (token.Line != null)
{
writer.WritePropertyName("line");
writer.WriteValue(token.Line);
}
if (token.Column != null)
if (token.Line != null)
{
writer.WritePropertyName("col");
writer.WriteValue(token.Column);
@@ -249,17 +230,12 @@ namespace GitHub.DistributedTask.ObjectTemplating.Tokens
writer.WriteStartObject();
writer.WritePropertyName("type");
writer.WriteValue(token.Type);
if (token.FileId != null)
{
writer.WritePropertyName("file");
writer.WriteValue(token.FileId);
}
if (token.Line != null)
{
writer.WritePropertyName("line");
writer.WriteValue(token.Line);
}
if (token.Column != null)
if (token.Line != null)
{
writer.WritePropertyName("col");
writer.WriteValue(token.Column);
@@ -277,17 +253,12 @@ namespace GitHub.DistributedTask.ObjectTemplating.Tokens
writer.WriteStartObject();
writer.WritePropertyName("type");
writer.WriteValue(token.Type);
if (token.FileId != null)
{
writer.WritePropertyName("file");
writer.WriteValue(token.FileId);
}
if (token.Line != null)
{
writer.WritePropertyName("line");
writer.WriteValue(token.Line);
}
if (token.Column != null)
if (token.Line != null)
{
writer.WritePropertyName("col");
writer.WriteValue(token.Column);
@@ -302,17 +273,12 @@ namespace GitHub.DistributedTask.ObjectTemplating.Tokens
writer.WriteStartObject();
writer.WritePropertyName("type");
writer.WriteValue(token.Type);
if (token.FileId != null)
{
writer.WritePropertyName("file");
writer.WriteValue(token.FileId);
}
if (token.Line != null)
{
writer.WritePropertyName("line");
writer.WriteValue(token.Line);
}
if (token.Column != null)
if (token.Line != null)
{
writer.WritePropertyName("col");
writer.WriteValue(token.Column);
@@ -335,17 +301,12 @@ namespace GitHub.DistributedTask.ObjectTemplating.Tokens
writer.WriteStartObject();
writer.WritePropertyName("type");
writer.WriteValue(token.Type);
if (token.FileId != null)
{
writer.WritePropertyName("file");
writer.WriteValue(token.FileId);
}
if (token.Line != null)
{
writer.WritePropertyName("line");
writer.WriteValue(token.Line);
}
if (token.Column != null)
if (token.Line != null)
{
writer.WritePropertyName("col");
writer.WriteValue(token.Column);

View File

@@ -39,8 +39,7 @@ namespace GitHub.DistributedTask.Pipelines
DictionaryContextData contextData,
WorkspaceOptions workspaceOptions,
IEnumerable<JobStep> steps,
IEnumerable<ContextScope> scopes,
IList<String> fileTable)
IEnumerable<ContextScope> scopes)
{
this.MessageType = JobRequestMessageTypes.PipelineAgentJobRequest;
this.Plan = plan;
@@ -75,11 +74,6 @@ namespace GitHub.DistributedTask.Pipelines
this.ContextData[pair.Key] = pair.Value;
}
}
if (fileTable?.Count > 0)
{
m_fileTable = new List<String>(fileTable);
}
}
[DataMember]
@@ -243,18 +237,6 @@ namespace GitHub.DistributedTask.Pipelines
}
}
public IList<String> FileTable
{
get
{
if (m_fileTable == null)
{
m_fileTable = new List<String>();
}
return m_fileTable;
}
}
// todo: remove after feature-flag DistributedTask.EvaluateContainerOnRunner is enabled everywhere
public void SetJobSidecarContainers(IDictionary<String, String> value)
{
@@ -363,11 +345,6 @@ namespace GitHub.DistributedTask.Pipelines
m_environmentVariables = null;
}
if (m_fileTable?.Count == 0)
{
m_fileTable = null;
}
if (m_maskHints?.Count == 0)
{
m_maskHints = null;
@@ -397,9 +374,6 @@ namespace GitHub.DistributedTask.Pipelines
[DataMember(Name = "EnvironmentVariables", EmitDefaultValue = false)]
private List<TemplateToken> m_environmentVariables;
[DataMember(Name = "FileTable", EmitDefaultValue = false)]
private List<String> m_fileTable;
[DataMember(Name = "Mask", EmitDefaultValue = false)]
private List<MaskHint> m_maskHints;

View File

@@ -35,19 +35,6 @@ namespace GitHub.DistributedTask.Pipelines.ContextData
throw new ArgumentException($"Unexpected type '{value?.GetType().Name}' encountered while reading '{objectDescription}'. The type '{nameof(DictionaryContextData)}' was expected.");
}
[EditorBrowsable(EditorBrowsableState.Never)]
public static CaseSensitiveDictionaryContextData AssertCaseSensitiveDictionary(
this PipelineContextData value,
String objectDescription)
{
if (value is CaseSensitiveDictionaryContextData dictionary)
{
return dictionary;
}
throw new ArgumentException($"Unexpected type '{value?.GetType().Name}' encountered while reading '{objectDescription}'. The type '{nameof(CaseSensitiveDictionaryContextData)}' was expected.");
}
[EditorBrowsable(EditorBrowsableState.Never)]
public static StringContextData AssertString(
this PipelineContextData value,

View File

@@ -154,11 +154,6 @@ namespace GitHub.DistributedTask.Pipelines.ObjectTemplating
if (value is StringToken containerLiteral)
{
if (String.IsNullOrEmpty(containerLiteral.Value))
{
return null;
}
result.Image = containerLiteral.Value;
}
else

View File

@@ -19,8 +19,7 @@ namespace GitHub.DistributedTask.Pipelines.ObjectTemplating
{
public PipelineTemplateEvaluator(
ITraceWriter trace,
TemplateSchema schema,
IList<String> fileTable)
TemplateSchema schema)
{
if (!String.Equals(schema.Version, PipelineTemplateConstants.Workflow_1_0, StringComparison.Ordinal))
{
@@ -29,7 +28,6 @@ namespace GitHub.DistributedTask.Pipelines.ObjectTemplating
m_trace = trace;
m_schema = schema;
m_fileTable = fileTable;
}
public Int32 MaxDepth => 50;
@@ -326,16 +324,6 @@ namespace GitHub.DistributedTask.Pipelines.ObjectTemplating
TraceWriter = m_trace,
};
// Add the file table
if (m_fileTable?.Count > 0)
{
foreach (var file in m_fileTable)
{
result.GetFileId(file);
}
}
// Add named context
if (contextData != null)
{
foreach (var pair in contextData)
@@ -358,7 +346,6 @@ namespace GitHub.DistributedTask.Pipelines.ObjectTemplating
private readonly ITraceWriter m_trace;
private readonly TemplateSchema m_schema;
private readonly IList<String> m_fileTable;
private readonly String[] s_contextNames = new[]
{
PipelineTemplateConstants.GitHub,

View File

@@ -1,572 +0,0 @@
using System;
using System.Globalization;
using System.IO;
using System.Linq;
using GitHub.DistributedTask.ObjectTemplating;
using GitHub.DistributedTask.ObjectTemplating.Tokens;
using YamlDotNet.Core;
using YamlDotNet.Core.Events;
namespace GitHub.DistributedTask.Pipelines.ObjectTemplating
{
/// <summary>
/// Converts a YAML file into a TemplateToken
/// </summary>
public sealed class YamlObjectReader : IObjectReader
{
internal YamlObjectReader(
Int32? fileId,
TextReader input)
{
m_fileId = fileId;
m_parser = new Parser(input);
}
public Boolean AllowLiteral(out LiteralToken value)
{
if (EvaluateCurrent() is Scalar scalar)
{
// Tag specified
if (!String.IsNullOrEmpty(scalar.Tag))
{
// String tag
if (String.Equals(scalar.Tag, c_stringTag, StringComparison.Ordinal))
{
value = new StringToken(m_fileId, scalar.Start.Line, scalar.Start.Column, scalar.Value);
MoveNext();
return true;
}
// Not plain style
if (scalar.Style != ScalarStyle.Plain)
{
throw new NotSupportedException($"The scalar style '{scalar.Style}' on line {scalar.Start.Line} and column {scalar.Start.Column} is not valid with the tag '{scalar.Tag}'");
}
// Boolean, Float, Integer, or Null
switch (scalar.Tag)
{
case c_booleanTag:
value = ParseBoolean(scalar);
break;
case c_floatTag:
value = ParseFloat(scalar);
break;
case c_integerTag:
value = ParseInteger(scalar);
break;
case c_nullTag:
value = ParseNull(scalar);
break;
default:
throw new NotSupportedException($"Unexpected tag '{scalar.Tag}'");
}
MoveNext();
return true;
}
// Plain style, determine type using YAML 1.2 "core" schema https://yaml.org/spec/1.2/spec.html#id2804923
if (scalar.Style == ScalarStyle.Plain)
{
if (MatchNull(scalar, out var nullToken))
{
value = nullToken;
}
else if (MatchBoolean(scalar, out var booleanToken))
{
value = booleanToken;
}
else if (MatchInteger(scalar, out var numberToken) ||
MatchFloat(scalar, out numberToken))
{
value = numberToken;
}
else
{
value = new StringToken(m_fileId, scalar.Start.Line, scalar.Start.Column, scalar.Value);
}
MoveNext();
return true;
}
// Otherwise assume string
value = new StringToken(m_fileId, scalar.Start.Line, scalar.Start.Column, scalar.Value);
MoveNext();
return true;
}
value = default;
return false;
}
public Boolean AllowSequenceStart(out SequenceToken value)
{
if (EvaluateCurrent() is SequenceStart sequenceStart)
{
value = new SequenceToken(m_fileId, sequenceStart.Start.Line, sequenceStart.Start.Column);
MoveNext();
return true;
}
value = default;
return false;
}
public Boolean AllowSequenceEnd()
{
if (EvaluateCurrent() is SequenceEnd)
{
MoveNext();
return true;
}
return false;
}
public Boolean AllowMappingStart(out MappingToken value)
{
if (EvaluateCurrent() is MappingStart mappingStart)
{
value = new MappingToken(m_fileId, mappingStart.Start.Line, mappingStart.Start.Column);
MoveNext();
return true;
}
value = default;
return false;
}
public Boolean AllowMappingEnd()
{
if (EvaluateCurrent() is MappingEnd)
{
MoveNext();
return true;
}
return false;
}
/// <summary>
/// Consumes the last parsing events, which are expected to be DocumentEnd and StreamEnd.
/// </summary>
public void ValidateEnd()
{
if (EvaluateCurrent() is DocumentEnd)
{
MoveNext();
}
else
{
throw new InvalidOperationException("Expected document end parse event");
}
if (EvaluateCurrent() is StreamEnd)
{
MoveNext();
}
else
{
throw new InvalidOperationException("Expected stream end parse event");
}
if (MoveNext())
{
throw new InvalidOperationException("Expected end of parse events");
}
}
/// <summary>
/// Consumes the first parsing events, which are expected to be StreamStart and DocumentStart.
/// </summary>
public void ValidateStart()
{
if (EvaluateCurrent() != null)
{
throw new InvalidOperationException("Unexpected parser state");
}
if (!MoveNext())
{
throw new InvalidOperationException("Expected a parse event");
}
if (EvaluateCurrent() is StreamStart)
{
MoveNext();
}
else
{
throw new InvalidOperationException("Expected stream start parse event");
}
if (EvaluateCurrent() is DocumentStart)
{
MoveNext();
}
else
{
throw new InvalidOperationException("Expected document start parse event");
}
}
private ParsingEvent EvaluateCurrent()
{
if (m_current == null)
{
m_current = m_parser.Current;
if (m_current != null)
{
if (m_current is Scalar scalar)
{
// Verify not using achors
if (scalar.Anchor != null)
{
throw new InvalidOperationException($"Anchors are not currently supported. Remove the anchor '{scalar.Anchor}'");
}
}
else if (m_current is MappingStart mappingStart)
{
// Verify not using achors
if (mappingStart.Anchor != null)
{
throw new InvalidOperationException($"Anchors are not currently supported. Remove the anchor '{mappingStart.Anchor}'");
}
}
else if (m_current is SequenceStart sequenceStart)
{
// Verify not using achors
if (sequenceStart.Anchor != null)
{
throw new InvalidOperationException($"Anchors are not currently supported. Remove the anchor '{sequenceStart.Anchor}'");
}
}
else if (!(m_current is MappingEnd) &&
!(m_current is SequenceEnd) &&
!(m_current is DocumentStart) &&
!(m_current is DocumentEnd) &&
!(m_current is StreamStart) &&
!(m_current is StreamEnd))
{
throw new InvalidOperationException($"Unexpected parsing event type: {m_current.GetType().Name}");
}
}
}
return m_current;
}
private Boolean MoveNext()
{
m_current = null;
return m_parser.MoveNext();
}
private BooleanToken ParseBoolean(Scalar scalar)
{
if (MatchBoolean(scalar, out var token))
{
return token;
}
ThrowInvalidValue(scalar, c_booleanTag); // throws
return default;
}
private NumberToken ParseFloat(Scalar scalar)
{
if (MatchFloat(scalar, out var token))
{
return token;
}
ThrowInvalidValue(scalar, c_floatTag); // throws
return default;
}
private NumberToken ParseInteger(Scalar scalar)
{
if (MatchInteger(scalar, out var token))
{
return token;
}
ThrowInvalidValue(scalar, c_integerTag); // throws
return default;
}
private NullToken ParseNull(Scalar scalar)
{
if (MatchNull(scalar, out var token))
{
return token;
}
ThrowInvalidValue(scalar, c_nullTag); // throws
return default;
}
private Boolean MatchBoolean(
Scalar scalar,
out BooleanToken value)
{
// YAML 1.2 "core" schema https://yaml.org/spec/1.2/spec.html#id2804923
switch (scalar.Value ?? String.Empty)
{
case "true":
case "True":
case "TRUE":
value = new BooleanToken(m_fileId, scalar.Start.Line, scalar.Start.Column, true);
return true;
case "false":
case "False":
case "FALSE":
value = new BooleanToken(m_fileId, scalar.Start.Line, scalar.Start.Column, false);
return true;
}
value = default;
return false;
}
private Boolean MatchFloat(
Scalar scalar,
out NumberToken value)
{
// YAML 1.2 "core" schema https://yaml.org/spec/1.2/spec.html#id2804923
var str = scalar.Value;
if (!String.IsNullOrEmpty(str))
{
// Check for [-+]?(\.inf|\.Inf|\.INF)|\.nan|\.NaN|\.NAN
switch (str)
{
case ".inf":
case ".Inf":
case ".INF":
case "+.inf":
case "+.Inf":
case "+.INF":
value = new NumberToken(m_fileId, scalar.Start.Line, scalar.Start.Column, Double.PositiveInfinity);
return true;
case "-.inf":
case "-.Inf":
case "-.INF":
value = new NumberToken(m_fileId, scalar.Start.Line, scalar.Start.Column, Double.NegativeInfinity);
return true;
case ".nan":
case ".NaN":
case ".NAN":
value = new NumberToken(m_fileId, scalar.Start.Line, scalar.Start.Column, Double.NaN);
return true;
}
// Otherwise check [-+]?(\.[0-9]+|[0-9]+(\.[0-9]*)?)([eE][-+]?[0-9]+)?
// Skip leading sign
var index = str[0] == '-' || str[0] == '+' ? 1 : 0;
// Check for integer portion
var length = str.Length;
var hasInteger = false;
while (index < length && str[index] >= '0' && str[index] <= '9')
{
hasInteger = true;
index++;
}
// Check for decimal point
var hasDot = false;
if (index < length && str[index] == '.')
{
hasDot = true;
index++;
}
// Check for decimal portion
var hasDecimal = false;
while (index < length && str[index] >= '0' && str[index] <= '9')
{
hasDecimal = true;
index++;
}
// Check [-+]?(\.[0-9]+|[0-9]+(\.[0-9]*)?)
if ((hasDot && hasDecimal) || hasInteger)
{
// Check for end
if (index == length)
{
// Try parse
if (Double.TryParse(str, NumberStyles.AllowLeadingSign | NumberStyles.AllowDecimalPoint, CultureInfo.InvariantCulture, out var doubleValue))
{
value = new NumberToken(m_fileId, scalar.Start.Line, scalar.Start.Column, doubleValue);
return true;
}
// Otherwise exceeds range
else
{
ThrowInvalidValue(scalar, c_floatTag); // throws
}
}
// Check [eE][-+]?[0-9]
else if (index < length && (str[index] == 'e' || str[index] == 'E'))
{
index++;
// Skip sign
if (index < length && (str[index] == '-' || str[index] == '+'))
{
index++;
}
// Check for exponent
var hasExponent = false;
while (index < length && str[index] >= '0' && str[index] <= '9')
{
hasExponent = true;
index++;
}
// Check for end
if (hasExponent && index == length)
{
// Try parse
if (Double.TryParse(str, NumberStyles.AllowLeadingSign | NumberStyles.AllowDecimalPoint | NumberStyles.AllowExponent, CultureInfo.InvariantCulture, out var doubleValue))
{
value = new NumberToken(m_fileId, scalar.Start.Line, scalar.Start.Column, (Double)doubleValue);
return true;
}
// Otherwise exceeds range
else
{
ThrowInvalidValue(scalar, c_floatTag); // throws
}
}
}
}
}
value = default;
return false;
}
private Boolean MatchInteger(
Scalar scalar,
out NumberToken value)
{
// YAML 1.2 "core" schema https://yaml.org/spec/1.2/spec.html#id2804923
var str = scalar.Value;
if (!String.IsNullOrEmpty(str))
{
// Check for [0-9]+
var firstChar = str[0];
if (firstChar >= '0' && firstChar <= '9' &&
str.Skip(1).All(x => x >= '0' && x <= '9'))
{
// Try parse
if (Double.TryParse(str, NumberStyles.None, CultureInfo.InvariantCulture, out var doubleValue))
{
value = new NumberToken(m_fileId, scalar.Start.Line, scalar.Start.Column, doubleValue);
return true;
}
// Otherwise exceeds range
ThrowInvalidValue(scalar, c_integerTag); // throws
}
// Check for (-|+)[0-9]+
else if ((firstChar == '-' || firstChar == '+') &&
str.Length > 1 &&
str.Skip(1).All(x => x >= '0' && x <= '9'))
{
// Try parse
if (Double.TryParse(str, NumberStyles.AllowLeadingSign, CultureInfo.InvariantCulture, out var doubleValue))
{
value = new NumberToken(m_fileId, scalar.Start.Line, scalar.Start.Column, doubleValue);
return true;
}
// Otherwise exceeds range
ThrowInvalidValue(scalar, c_integerTag); // throws
}
// Check for 0x[0-9a-fA-F]+
else if (firstChar == '0' &&
str.Length > 2 &&
str[1] == 'x' &&
str.Skip(2).All(x => (x >= '0' && x <= '9') || (x >= 'a' && x <= 'f') || (x >= 'A' && x <= 'F')))
{
// Try parse
if (Int32.TryParse(str.Substring(2), NumberStyles.AllowHexSpecifier, CultureInfo.InvariantCulture, out var integerValue))
{
value = new NumberToken(m_fileId, scalar.Start.Line, scalar.Start.Column, integerValue);
return true;
}
// Otherwise exceeds range
ThrowInvalidValue(scalar, c_integerTag); // throws
}
// Check for 0o[0-9]+
else if (firstChar == '0' &&
str.Length > 2 &&
str[1] == 'o' &&
str.Skip(2).All(x => x >= '0' && x <= '7'))
{
// Try parse
var integerValue = default(Int32);
try
{
integerValue = Convert.ToInt32(str.Substring(2), 8);
}
// Otherwise exceeds range
catch (Exception)
{
ThrowInvalidValue(scalar, c_integerTag); // throws
}
value = new NumberToken(m_fileId, scalar.Start.Line, scalar.Start.Column, integerValue);
return true;
}
}
value = default;
return false;
}
private Boolean MatchNull(
Scalar scalar,
out NullToken value)
{
// YAML 1.2 "core" schema https://yaml.org/spec/1.2/spec.html#id2804923
switch (scalar.Value ?? String.Empty)
{
case "":
case "null":
case "Null":
case "NULL":
case "~":
value = new NullToken(m_fileId, scalar.Start.Line, scalar.Start.Column);
return true;
}
value = default;
return false;
}
private void ThrowInvalidValue(
Scalar scalar,
String tag)
{
throw new NotSupportedException($"The value '{scalar.Value}' on line {scalar.Start.Line} and column {scalar.Start.Column} is invalid for the type '{scalar.Tag}'");
}
private const String c_booleanTag = "tag:yaml.org,2002:bool";
private const String c_floatTag = "tag:yaml.org,2002:float";
private const String c_integerTag = "tag:yaml.org,2002:int";
private const String c_nullTag = "tag:yaml.org,2002:null";
private const String c_stringTag = "tag:yaml.org,2002:string";
private readonly Int32? m_fileId;
private readonly Parser m_parser;
private ParsingEvent m_current;
}
}

View File

@@ -472,7 +472,7 @@
"matrix"
],
"one-of": [
"string",
"non-empty-string",
"container-mapping"
]
},
@@ -497,22 +497,10 @@
],
"mapping": {
"loose-key-type": "non-empty-string",
"loose-value-type": "services-container"
"loose-value-type": "container"
}
},
"services-container": {
"context": [
"github",
"strategy",
"matrix"
],
"one-of": [
"non-empty-string",
"container-mapping"
]
},
"container-env": {
"mapping": {
"loose-key-type": "non-empty-string",

View File

@@ -498,7 +498,7 @@ namespace GitHub.Runner.Common.Tests
_promptManager
.Setup(x => x.ReadValue(
Constants.Runner.CommandLine.Args.Token, // argName
"Enter runner remove token:", // description
"Enter runner deletion token:", // description
true, // secret
string.Empty, // defaultValue
Validators.NonEmptyValidator, // validator

View File

@@ -175,8 +175,8 @@ namespace GitHub.Runner.Common.Tests.Listener.Configuration
Assert.True(s.PoolId.Equals(_expectedPoolId));
Assert.True(s.WorkFolder.Equals(_expectedWorkFolder));
// validate GetAgentPoolsAsync gets called twice with automation pool type
_runnerServer.Verify(x => x.GetAgentPoolsAsync(It.IsAny<string>(), It.Is<TaskAgentPoolType>(p => p == TaskAgentPoolType.Automation)), Times.Exactly(2));
// validate GetAgentPoolsAsync gets called once with automation pool type
_runnerServer.Verify(x => x.GetAgentPoolsAsync(It.IsAny<string>(), It.Is<TaskAgentPoolType>(p => p == TaskAgentPoolType.Automation)), Times.Once);
_runnerServer.Verify(x => x.AddAgentAsync(It.IsAny<int>(), It.Is<TaskAgent>(a => a.Labels.Contains("self-hosted") && a.Labels.Contains(VarUtil.OS) && a.Labels.Contains(VarUtil.OSArchitecture))), Times.Once);
}

View File

@@ -33,7 +33,7 @@ namespace GitHub.Runner.Common.Tests.Listener
TaskOrchestrationPlanReference plan = new TaskOrchestrationPlanReference();
TimelineReference timeline = null;
Guid jobId = Guid.NewGuid();
var result = new Pipelines.AgentJobRequestMessage(plan, timeline, jobId, "someJob", "someJob", null, null, null, new Dictionary<string, VariableValue>(), new List<MaskHint>(), new Pipelines.JobResources(), new Pipelines.ContextData.DictionaryContextData(), new Pipelines.WorkspaceOptions(), new List<Pipelines.ActionStep>(), null, null);
var result = new Pipelines.AgentJobRequestMessage(plan, timeline, jobId, "someJob", "someJob", null, null, null, new Dictionary<string, VariableValue>(), new List<MaskHint>(), new Pipelines.JobResources(), new Pipelines.ContextData.DictionaryContextData(), new Pipelines.WorkspaceOptions(), new List<Pipelines.ActionStep>(), null);
result.ContextData["github"] = new Pipelines.ContextData.DictionaryContextData();
return result;
}

View File

@@ -43,7 +43,7 @@ namespace GitHub.Runner.Common.Tests.Listener
TaskOrchestrationPlanReference plan = new TaskOrchestrationPlanReference();
TimelineReference timeline = null;
Guid jobId = Guid.NewGuid();
return new Pipelines.AgentJobRequestMessage(plan, timeline, jobId, "test", "test", null, null, null, new Dictionary<string, VariableValue>(), new List<MaskHint>(), new Pipelines.JobResources(), new Pipelines.ContextData.DictionaryContextData(), new Pipelines.WorkspaceOptions(), new List<Pipelines.ActionStep>(), null, null);
return new Pipelines.AgentJobRequestMessage(plan, timeline, jobId, "test", "test", null, null, null, new Dictionary<string, VariableValue>(), new List<MaskHint>(), new Pipelines.JobResources(), new Pipelines.ContextData.DictionaryContextData(), new Pipelines.WorkspaceOptions(), new List<Pipelines.ActionStep>(), null);
}
private JobCancelMessage CreateJobCancelMessage()

View File

@@ -1,7 +1,8 @@
using System.IO;
using GitHub.Runner.Common.Util;
using System.IO;
using Xunit;
using System;
using GitHub.Runner.Sdk;
using System.Runtime.CompilerServices;
namespace GitHub.Runner.Common.Tests
{
@@ -20,16 +21,9 @@ namespace GitHub.Runner.Common.Tests
return projectDir;
}
public static string GetTestFilePath([CallerFilePath] string path = null)
{
return path;
}
public static string GetSrcPath()
{
string L0dir = Path.GetDirectoryName(GetTestFilePath());
string testDir = Path.GetDirectoryName(L0dir);
string srcDir = Path.GetDirectoryName(testDir);
string srcDir = Environment.GetEnvironmentVariable("GITHUB_RUNNER_SRC_DIR");
ArgUtil.Directory(srcDir, nameof(srcDir));
Assert.Equal(Src, Path.GetFileName(srcDir));
return srcDir;

View File

@@ -51,19 +51,6 @@ namespace GitHub.Runner.Common.Tests.Worker
Assert.True(ActionCommand.TryParse(message, commands, out verify));
Assert.True(IsEqualCommand(hc, test, verify));
message = "";
test = null;
verify = null;
//##[do-something k1=%253B=%250D=%250A=%255D;]%253B-%250D-%250A-%255D
message = "##[do-something k1=%253B=%250D=%250A=%255D;]%253B-%250D-%250A-%255D";
test = new ActionCommand("do-something")
{
Data = "%3B-%0D-%0A-%5D",
};
test.Properties.Add("k1", "%3B=%0D=%0A=%5D");
Assert.True(ActionCommand.TryParse(message, commands, out verify));
Assert.True(IsEqualCommand(hc, test, verify));
message = "";
test = null;
verify = null;
@@ -122,7 +109,7 @@ namespace GitHub.Runner.Common.Tests.Worker
message = "";
test = null;
verify = null;
//::do-something k1=;=%2C=%0D=%0A=]=%3A,::;-%0D-%0A-]-:-,
//::do-something k1=%3B=%0D=%0A=%5D;::%3B-%0D-%0A-%5D
message = "::do-something k1=;=%2C=%0D=%0A=]=%3A,::;-%0D-%0A-]-:-,";
test = new ActionCommand("do-something")
{
@@ -132,19 +119,6 @@ namespace GitHub.Runner.Common.Tests.Worker
Assert.True(ActionCommand.TryParseV2(message, commands, out verify));
Assert.True(IsEqualCommand(hc, test, verify));
message = "";
test = null;
verify = null;
//::do-something k1=;=%252C=%250D=%250A=]=%253A,::;-%250D-%250A-]-:-,
message = "::do-something k1=;=%252C=%250D=%250A=]=%253A,::;-%250D-%250A-]-:-,";
test = new ActionCommand("do-something")
{
Data = ";-%0D-%0A-]-:-,",
};
test.Properties.Add("k1", ";=%2C=%0D=%0A=]=%3A");
Assert.True(ActionCommand.TryParseV2(message, commands, out verify));
Assert.True(IsEqualCommand(hc, test, verify));
message = "";
test = null;
verify = null;

View File

@@ -1,10 +1,8 @@
using System;
using System.Collections.Generic;
using System.IO;
using System.Runtime.CompilerServices;
using GitHub.DistributedTask.WebApi;
using GitHub.Runner.Worker;
using GitHub.Runner.Worker.Container;
using Moq;
using Xunit;
using Pipelines = GitHub.DistributedTask.Pipelines;
@@ -13,35 +11,47 @@ namespace GitHub.Runner.Common.Tests.Worker
{
public sealed class ActionCommandManagerL0
{
private ActionCommandManager _commandManager;
private Mock<IExecutionContext> _ec;
private Mock<IExtensionManager> _extensionManager;
private Mock<IPipelineDirectoryManager> _pipelineDirectoryManager;
[Fact]
[Trait("Level", "L0")]
[Trait("Category", "Worker")]
public void EnablePluginInternalCommand()
{
using (TestHostContext hc = CreateTestContext())
using (TestHostContext _hc = new TestHostContext(this))
{
var extensionManger = new Mock<IExtensionManager>();
var directoryManager = new Mock<IPipelineDirectoryManager>();
var pluginCommand = new InternalPluginSetRepoPathCommandExtension();
pluginCommand.Initialize(_hc);
var envCommand = new SetEnvCommandExtension();
envCommand.Initialize(_hc);
extensionManger.Setup(x => x.GetExtensions<IActionCommandExtension>())
.Returns(new List<IActionCommandExtension>() { pluginCommand, envCommand });
_hc.SetSingleton<IExtensionManager>(extensionManger.Object);
_hc.SetSingleton<IPipelineDirectoryManager>(directoryManager.Object);
Mock<IExecutionContext> _ec = new Mock<IExecutionContext>();
_ec.Setup(x => x.Write(It.IsAny<string>(), It.IsAny<string>()))
.Returns((string tag, string line) =>
{
hc.GetTrace().Info($"{tag} {line}");
_hc.GetTrace().Info($"{tag} {line}");
return 1;
});
_ec.Setup(x => x.AddIssue(It.IsAny<Issue>(), It.IsAny<string>()))
.Callback((Issue issue, string message) =>
{
hc.GetTrace().Info($"{issue.Type} {issue.Message} {message ?? string.Empty}");
_hc.GetTrace().Info($"{issue.Type} {issue.Message} {message ?? string.Empty}");
});
ActionCommandManager commandManager = new ActionCommandManager();
commandManager.Initialize(_hc);
_commandManager.EnablePluginInternalCommand();
commandManager.EnablePluginInternalCommand();
Assert.True(_commandManager.TryProcessCommand(_ec.Object, "##[internal-set-repo-path repoFullName=actions/runner;workspaceRepo=true]somepath", null));
Assert.True(commandManager.TryProcessCommand(_ec.Object, "##[internal-set-repo-path repoFullName=actions/runner;workspaceRepo=true]somepath"));
_pipelineDirectoryManager.Verify(x => x.UpdateRepositoryDirectory(_ec.Object, "actions/runner", "somepath", true), Times.Once);
directoryManager.Verify(x => x.UpdateRepositoryDirectory(_ec.Object, "actions/runner", "somepath", true), Times.Once);
}
}
@@ -50,29 +60,47 @@ namespace GitHub.Runner.Common.Tests.Worker
[Trait("Category", "Worker")]
public void DisablePluginInternalCommand()
{
using (TestHostContext hc = CreateTestContext())
using (TestHostContext _hc = new TestHostContext(this))
{
var extensionManger = new Mock<IExtensionManager>();
var directoryManager = new Mock<IPipelineDirectoryManager>();
var pluginCommand = new InternalPluginSetRepoPathCommandExtension();
pluginCommand.Initialize(_hc);
var envCommand = new SetEnvCommandExtension();
envCommand.Initialize(_hc);
extensionManger.Setup(x => x.GetExtensions<IActionCommandExtension>())
.Returns(new List<IActionCommandExtension>() { pluginCommand, envCommand });
_hc.SetSingleton<IExtensionManager>(extensionManger.Object);
_hc.SetSingleton<IPipelineDirectoryManager>(directoryManager.Object);
Mock<IExecutionContext> _ec = new Mock<IExecutionContext>();
_ec.Setup(x => x.Write(It.IsAny<string>(), It.IsAny<string>()))
.Returns((string tag, string line) =>
{
hc.GetTrace().Info($"{tag} {line}");
_hc.GetTrace().Info($"{tag} {line}");
return 1;
});
_ec.Setup(x => x.AddIssue(It.IsAny<Issue>(), It.IsAny<string>()))
.Callback((Issue issue, string message) =>
{
hc.GetTrace().Info($"{issue.Type} {issue.Message} {message ?? string.Empty}");
_hc.GetTrace().Info($"{issue.Type} {issue.Message} {message ?? string.Empty}");
});
ActionCommandManager commandManager = new ActionCommandManager();
commandManager.Initialize(_hc);
_commandManager.EnablePluginInternalCommand();
commandManager.EnablePluginInternalCommand();
Assert.True(_commandManager.TryProcessCommand(_ec.Object, "##[internal-set-repo-path repoFullName=actions/runner;workspaceRepo=true]somepath", null));
Assert.True(commandManager.TryProcessCommand(_ec.Object, "##[internal-set-repo-path repoFullName=actions/runner;workspaceRepo=true]somepath"));
_commandManager.DisablePluginInternalCommand();
commandManager.DisablePluginInternalCommand();
Assert.False(_commandManager.TryProcessCommand(_ec.Object, "##[internal-set-repo-path repoFullName=actions/runner;workspaceRepo=true]somepath", null));
Assert.False(commandManager.TryProcessCommand(_ec.Object, "##[internal-set-repo-path repoFullName=actions/runner;workspaceRepo=true]somepath"));
_pipelineDirectoryManager.Verify(x => x.UpdateRepositoryDirectory(_ec.Object, "actions/runner", "somepath", true), Times.Once);
directoryManager.Verify(x => x.UpdateRepositoryDirectory(_ec.Object, "actions/runner", "somepath", true), Times.Once);
}
}
@@ -81,27 +109,42 @@ namespace GitHub.Runner.Common.Tests.Worker
[Trait("Category", "Worker")]
public void StopProcessCommand()
{
using (TestHostContext hc = CreateTestContext())
using (TestHostContext _hc = new TestHostContext(this))
{
var extensionManger = new Mock<IExtensionManager>();
var pluginCommand = new InternalPluginSetRepoPathCommandExtension();
pluginCommand.Initialize(_hc);
var envCommand = new SetEnvCommandExtension();
envCommand.Initialize(_hc);
extensionManger.Setup(x => x.GetExtensions<IActionCommandExtension>())
.Returns(new List<IActionCommandExtension>() { pluginCommand, envCommand });
_hc.SetSingleton<IExtensionManager>(extensionManger.Object);
Mock<IExecutionContext> _ec = new Mock<IExecutionContext>();
_ec.Setup(x => x.Write(It.IsAny<string>(), It.IsAny<string>()))
.Returns((string tag, string line) =>
{
hc.GetTrace().Info($"{tag} {line}");
_hc.GetTrace().Info($"{tag} {line}");
return 1;
});
_ec.Setup(x => x.AddIssue(It.IsAny<Issue>(), It.IsAny<string>()))
.Callback((Issue issue, string message) =>
{
hc.GetTrace().Info($"{issue.Type} {issue.Message} {message ?? string.Empty}");
_hc.GetTrace().Info($"{issue.Type} {issue.Message} {message ?? string.Empty}");
});
_ec.Setup(x => x.EnvironmentVariables).Returns(new Dictionary<string, string>());
Assert.True(_commandManager.TryProcessCommand(_ec.Object, "##[stop-commands]stopToken", null));
Assert.False(_commandManager.TryProcessCommand(_ec.Object, "##[set-env name=foo]bar", null));
Assert.True(_commandManager.TryProcessCommand(_ec.Object, "##[stopToken]", null));
Assert.True(_commandManager.TryProcessCommand(_ec.Object, "##[set-env name=foo]bar", null));
ActionCommandManager commandManager = new ActionCommandManager();
commandManager.Initialize(_hc);
Assert.True(commandManager.TryProcessCommand(_ec.Object, "##[stop-commands]stopToken"));
Assert.False(commandManager.TryProcessCommand(_ec.Object, "##[set-env name=foo]bar"));
Assert.True(commandManager.TryProcessCommand(_ec.Object, "##[stopToken]"));
Assert.True(commandManager.TryProcessCommand(_ec.Object, "##[set-env name=foo]bar"));
}
}
@@ -110,29 +153,41 @@ namespace GitHub.Runner.Common.Tests.Worker
[Trait("Category", "Worker")]
public void EchoProcessCommand()
{
using (TestHostContext hc = CreateTestContext())
using (TestHostContext _hc = new TestHostContext(this))
{
var extensionManager = new Mock<IExtensionManager>();
var echoCommand = new EchoCommandExtension();
echoCommand.Initialize(_hc);
extensionManager.Setup(x => x.GetExtensions<IActionCommandExtension>())
.Returns(new List<IActionCommandExtension>() { echoCommand });
_hc.SetSingleton<IExtensionManager>(extensionManager.Object);
Mock<IExecutionContext> _ec = new Mock<IExecutionContext>();
_ec.Setup(x => x.Write(It.IsAny<string>(), It.IsAny<string>()))
.Returns((string tag, string line) =>
{
hc.GetTrace().Info($"{tag} {line}");
_hc.GetTrace().Info($"{tag} {line}");
return 1;
});
_ec.SetupAllProperties();
ActionCommandManager commandManager = new ActionCommandManager();
commandManager.Initialize(_hc);
Assert.False(_ec.Object.EchoOnActionCommand);
Assert.True(_commandManager.TryProcessCommand(_ec.Object, "::echo::on", null));
Assert.True(commandManager.TryProcessCommand(_ec.Object, "::echo::on"));
Assert.True(_ec.Object.EchoOnActionCommand);
Assert.True(_commandManager.TryProcessCommand(_ec.Object, "::echo::off", null));
Assert.True(commandManager.TryProcessCommand(_ec.Object, "::echo::off"));
Assert.False(_ec.Object.EchoOnActionCommand);
Assert.True(_commandManager.TryProcessCommand(_ec.Object, "::echo::ON", null));
Assert.True(commandManager.TryProcessCommand(_ec.Object, "::echo::ON"));
Assert.True(_ec.Object.EchoOnActionCommand);
Assert.True(_commandManager.TryProcessCommand(_ec.Object, "::echo::Off ", null));
Assert.True(commandManager.TryProcessCommand(_ec.Object, "::echo::Off "));
Assert.False(_ec.Object.EchoOnActionCommand);
}
}
@@ -142,7 +197,7 @@ namespace GitHub.Runner.Common.Tests.Worker
[Trait("Category", "Worker")]
public void EchoProcessCommandDebugOn()
{
using (TestHostContext hc = CreateTestContext())
using (TestHostContext _hc = new TestHostContext(this))
{
// Set up a few things
// 1. Job request message (with ACTIONS_STEP_DEBUG = true)
@@ -150,7 +205,7 @@ namespace GitHub.Runner.Common.Tests.Worker
TimelineReference timeline = new TimelineReference();
Guid jobId = Guid.NewGuid();
string jobName = "some job name";
var jobRequest = new Pipelines.AgentJobRequestMessage(plan, timeline, jobId, jobName, jobName, null, null, null, new Dictionary<string, VariableValue>(), new List<MaskHint>(), new Pipelines.JobResources(), new Pipelines.ContextData.DictionaryContextData(), new Pipelines.WorkspaceOptions(), new List<Pipelines.ActionStep>(), null, null);
var jobRequest = new Pipelines.AgentJobRequestMessage(plan, timeline, jobId, jobName, jobName, null, null, null, new Dictionary<string, VariableValue>(), new List<MaskHint>(), new Pipelines.JobResources(), new Pipelines.ContextData.DictionaryContextData(), new Pipelines.WorkspaceOptions(), new List<Pipelines.ActionStep>(), null);
jobRequest.Resources.Repositories.Add(new Pipelines.RepositoryResource()
{
Alias = Pipelines.PipelineConstants.SelfAlias,
@@ -164,135 +219,84 @@ namespace GitHub.Runner.Common.Tests.Worker
var jobServerQueue = new Mock<IJobServerQueue>();
jobServerQueue.Setup(x => x.QueueTimelineRecordUpdate(It.IsAny<Guid>(), It.IsAny<TimelineRecord>()));
hc.SetSingleton(jobServerQueue.Object);
_hc.SetSingleton(jobServerQueue.Object);
var extensionManager = new Mock<IExtensionManager>();
var echoCommand = new EchoCommandExtension();
echoCommand.Initialize(_hc);
extensionManager.Setup(x => x.GetExtensions<IActionCommandExtension>())
.Returns(new List<IActionCommandExtension>() { echoCommand });
_hc.SetSingleton<IExtensionManager>(extensionManager.Object);
var configurationStore = new Mock<IConfigurationStore>();
configurationStore.Setup(x => x.GetSettings()).Returns(new RunnerSettings());
hc.SetSingleton(configurationStore.Object);
_hc.SetSingleton(configurationStore.Object);
var pagingLogger = new Mock<IPagingLogger>();
hc.EnqueueInstance(pagingLogger.Object);
_hc.EnqueueInstance(pagingLogger.Object);
ActionCommandManager commandManager = new ActionCommandManager();
commandManager.Initialize(_hc);
var _ec = new Runner.Worker.ExecutionContext();
_ec.Initialize(_hc);
// Initialize the job (to exercise logic that sets EchoOnActionCommand)
var ec = new Runner.Worker.ExecutionContext();
ec.Initialize(hc);
ec.InitializeJob(jobRequest, System.Threading.CancellationToken.None);
_ec.InitializeJob(jobRequest, System.Threading.CancellationToken.None);
ec.Complete();
_ec.Complete();
Assert.True(ec.EchoOnActionCommand);
Assert.True(_ec.EchoOnActionCommand);
Assert.True(_commandManager.TryProcessCommand(ec, "::echo::off", null));
Assert.False(ec.EchoOnActionCommand);
Assert.True(commandManager.TryProcessCommand(_ec, "::echo::off"));
Assert.False(_ec.EchoOnActionCommand);
Assert.True(_commandManager.TryProcessCommand(ec, "::echo::on", null));
Assert.True(ec.EchoOnActionCommand);
Assert.True(commandManager.TryProcessCommand(_ec, "::echo::on"));
Assert.True(_ec.EchoOnActionCommand);
}
}
[Fact]
[Trait("Level", "L0")]
[Trait("Category", "Worker")]
public void EchoProcessCommandInvalid()
{
using (TestHostContext hc = CreateTestContext())
using (TestHostContext _hc = new TestHostContext(this))
{
var extensionManager = new Mock<IExtensionManager>();
var echoCommand = new EchoCommandExtension();
echoCommand.Initialize(_hc);
extensionManager.Setup(x => x.GetExtensions<IActionCommandExtension>())
.Returns(new List<IActionCommandExtension>() { echoCommand });
_hc.SetSingleton<IExtensionManager>(extensionManager.Object);
Mock<IExecutionContext> _ec = new Mock<IExecutionContext>();
_ec.Setup(x => x.Write(It.IsAny<string>(), It.IsAny<string>()))
.Returns((string tag, string line) =>
{
hc.GetTrace().Info($"{tag} {line}");
_hc.GetTrace().Info($"{tag} {line}");
return 1;
});
_ec.SetupAllProperties();
ActionCommandManager commandManager = new ActionCommandManager();
commandManager.Initialize(_hc);
// Echo commands below are considered "processed", but are invalid
// 1. Invalid echo value
Assert.True(_commandManager.TryProcessCommand(_ec.Object, "::echo::invalid", null));
Assert.True(commandManager.TryProcessCommand(_ec.Object, "::echo::invalid"));
Assert.Equal(TaskResult.Failed, _ec.Object.CommandResult);
Assert.False(_ec.Object.EchoOnActionCommand);
// 2. No value
Assert.True(_commandManager.TryProcessCommand(_ec.Object, "::echo::", null));
Assert.True(commandManager.TryProcessCommand(_ec.Object, "::echo::"));
Assert.Equal(TaskResult.Failed, _ec.Object.CommandResult);
Assert.False(_ec.Object.EchoOnActionCommand);
}
}
[Fact]
[Trait("Level", "L0")]
[Trait("Category", "Worker")]
public void AddMatcherTranslatesFilePath()
{
using (TestHostContext hc = CreateTestContext())
{
// Create a problem matcher config file
var hostDirectory = hc.GetDirectory(WellKnownDirectory.Temp);
var hostFile = Path.Combine(hostDirectory, "my-matcher.json");
Directory.CreateDirectory(hostDirectory);
var content = @"
{
""problemMatcher"": [
{
""owner"": ""my-matcher"",
""pattern"": [
{
""regexp"": ""^ERROR: (.+)$"",
""message"": 1
}
]
}
]
}";
File.WriteAllText(hostFile, content);
// Setup translation info
var container = new ContainerInfo();
var containerDirectory = "/some-container-directory";
var containerFile = Path.Combine(containerDirectory, "my-matcher.json");
container.AddPathTranslateMapping(hostDirectory, containerDirectory);
// Act
_commandManager.TryProcessCommand(_ec.Object, $"::add-matcher::{containerFile}", container);
// Assert
_ec.Verify(x => x.AddMatchers(It.IsAny<IssueMatchersConfig>()), Times.Once);
}
}
private TestHostContext CreateTestContext([CallerMemberName] string testName = "")
{
var hostContext = new TestHostContext(this, testName);
// Mock extension manager
_extensionManager = new Mock<IExtensionManager>();
var commands = new IActionCommandExtension[]
{
new AddMatcherCommandExtension(),
new EchoCommandExtension(),
new InternalPluginSetRepoPathCommandExtension(),
new SetEnvCommandExtension(),
};
foreach (var command in commands)
{
command.Initialize(hostContext);
}
_extensionManager.Setup(x => x.GetExtensions<IActionCommandExtension>())
.Returns(new List<IActionCommandExtension>(commands));
hostContext.SetSingleton<IExtensionManager>(_extensionManager.Object);
// Mock pipeline directory manager
_pipelineDirectoryManager = new Mock<IPipelineDirectoryManager>();
hostContext.SetSingleton<IPipelineDirectoryManager>(_pipelineDirectoryManager.Object);
// Execution context
_ec = new Mock<IExecutionContext>();
// Command manager
_commandManager = new ActionCommandManager();
_commandManager.Initialize(hostContext);
return hostContext;
}
}
}

View File

@@ -125,7 +125,7 @@ namespace GitHub.Runner.Common.Tests.Worker
Directory.CreateDirectory(Path.GetDirectoryName(watermarkFile));
File.WriteAllText(watermarkFile, DateTime.UtcNow.ToString());
Directory.CreateDirectory(Path.Combine(Path.GetDirectoryName(watermarkFile), "notexist"));
File.Copy(Path.Combine(TestUtil.GetSrcPath(), "Test", TestDataFolderName, "dockerfileaction.yml"), Path.Combine(Path.GetDirectoryName(watermarkFile), "notexist", "action.yml"));
File.Copy(Path.Combine(Environment.GetEnvironmentVariable("GITHUB_RUNNER_SRC_DIR"), "Test", TestDataFolderName, "dockerfileaction.yml"), Path.Combine(Path.GetDirectoryName(watermarkFile), "notexist", "action.yml"));
//Act
await _actionManager.PrepareActionsAsync(_ec.Object, actions);
@@ -373,45 +373,6 @@ namespace GitHub.Runner.Common.Tests.Worker
}
}
[Fact]
[Trait("Level", "L0")]
[Trait("Category", "Worker")]
public async void PrepareActions_RepositoryActionWithActionYamlFile_DockerHubImage()
{
try
{
//Arrange
Setup();
var actionId = Guid.NewGuid();
var actions = new List<Pipelines.ActionStep>
{
new Pipelines.ActionStep()
{
Name = "action",
Id = actionId,
Reference = new Pipelines.RepositoryPathReference()
{
Name = "TingluoHuang/runner_L0",
Ref = "RepositoryActionWithActionYamlFile_DockerHubImage",
RepositoryType = "GitHub"
}
}
};
var actionDir = Path.Combine(_hc.GetDirectory(WellKnownDirectory.Actions), "TingluoHuang", "runner_L0", "RepositoryActionWithActionYamlFile_DockerHubImage");
//Act
var steps = await _actionManager.PrepareActionsAsync(_ec.Object, actions);
Assert.Equal((steps[0].Data as ContainerSetupInfo).StepIds[0], actionId);
Assert.Equal("ubuntu:18.04", (steps[0].Data as ContainerSetupInfo).Container.Image);
}
finally
{
Teardown();
}
}
[Fact]
[Trait("Level", "L0")]
[Trait("Category", "Worker")]
@@ -711,7 +672,7 @@ namespace GitHub.Runner.Common.Tests.Worker
name: 'Hello World'
description: 'Greet the world and record the time'
author: 'GitHub'
inputs:
inputs:
greeting: # id of input
description: 'The greeting we choose - will print ""{greeting}, World!"" on stdout'
required: true
@@ -811,7 +772,7 @@ runs:
name: 'Hello World'
description: 'Greet the world and record the time'
author: 'GitHub'
inputs:
inputs:
greeting: # id of input
description: 'The greeting we choose - will print ""{greeting}, World!"" on stdout'
required: true
@@ -910,7 +871,7 @@ runs:
name: 'Hello World'
description: 'Greet the world and record the time'
author: 'GitHub'
inputs:
inputs:
greeting: # id of input
description: 'The greeting we choose - will print ""{greeting}, World!"" on stdout'
required: true
@@ -964,87 +925,6 @@ runs:
}
}
[Fact]
[Trait("Level", "L0")]
[Trait("Category", "Worker")]
public void LoadsNodeActionDefinitionYaml()
{
try
{
// Arrange.
Setup();
const string Content = @"
# Container action
name: 'Hello World'
description: 'Greet the world and record the time'
author: 'GitHub'
inputs:
greeting: # id of input
description: 'The greeting we choose - will print ""{greeting}, World!"" on stdout'
required: true
default: 'Hello'
entryPoint: # id of input
description: 'optional docker entrypoint overwrite.'
required: false
outputs:
time: # id of output
description: 'The time we did the greeting'
icon: 'hello.svg' # vector art to display in the GitHub Marketplace
color: 'green' # optional, decorates the entry in the GitHub Marketplace
runs:
using: 'node12'
main: 'task.js'
";
Pipelines.ActionStep instance;
string directory;
directory = Path.Combine(_workFolder, Constants.Path.ActionsDirectory, "GitHub/actions".Replace(Path.AltDirectorySeparatorChar, Path.DirectorySeparatorChar), "master");
string file = Path.Combine(directory, Constants.Path.ActionManifestYamlFile);
Directory.CreateDirectory(Path.GetDirectoryName(file));
File.WriteAllText(file, Content);
instance = new Pipelines.ActionStep()
{
Id = Guid.NewGuid(),
Reference = new Pipelines.RepositoryPathReference()
{
Name = "GitHub/actions",
Ref = "master",
RepositoryType = Pipelines.RepositoryTypes.GitHub
}
};
// Act.
Definition definition = _actionManager.LoadAction(_ec.Object, instance);
// Assert.
Assert.NotNull(definition);
Assert.Equal(directory, definition.Directory);
Assert.NotNull(definition.Data);
Assert.NotNull(definition.Data.Inputs); // inputs
Dictionary<string, string> inputDefaults = new Dictionary<string, string>(StringComparer.OrdinalIgnoreCase);
foreach (var input in definition.Data.Inputs)
{
var name = input.Key.AssertString("key").Value;
var value = input.Value.AssertScalar("value").ToString();
_hc.GetTrace().Info($"Default: {name} = {value}");
inputDefaults[name] = value;
}
Assert.Equal(2, inputDefaults.Count);
Assert.True(inputDefaults.ContainsKey("greeting"));
Assert.Equal("Hello", inputDefaults["greeting"]);
Assert.True(string.IsNullOrEmpty(inputDefaults["entryPoint"]));
Assert.NotNull(definition.Data.Execution); // execution
Assert.NotNull((definition.Data.Execution as NodeJSActionExecutionData));
Assert.Equal("task.js", (definition.Data.Execution as NodeJSActionExecutionData).Script);
}
finally
{
Teardown();
}
}
[Fact]
[Trait("Level", "L0")]
[Trait("Category", "Worker")]
@@ -1060,7 +940,7 @@ runs:
name: 'Hello World'
description: 'Greet the world and record the time'
author: 'GitHub'
inputs:
inputs:
greeting: # id of input
description: 'The greeting we choose - will print ""{greeting}, World!"" on stdout'
required: true
@@ -1159,7 +1039,7 @@ runs:
name: 'Hello World'
description: 'Greet the world and record the time'
author: 'GitHub'
inputs:
inputs:
greeting: # id of input
description: 'The greeting we choose - will print ""{greeting}, World!"" on stdout'
required: true
@@ -1257,7 +1137,7 @@ runs:
name: 'Hello World'
description: 'Greet the world and record the time'
author: 'GitHub'
inputs:
inputs:
greeting: # id of input
description: 'The greeting we choose - will print ""{greeting}, World!"" on stdout'
required: true
@@ -1325,7 +1205,7 @@ runs:
name: 'Hello World'
description: 'Greet the world and record the time'
author: 'GitHub'
inputs:
inputs:
greeting: # id of input
description: 'The greeting we choose - will print ""{greeting}, World!"" on stdout'
required: true
@@ -1396,7 +1276,7 @@ runs:
name: 'Hello World'
description: 'Greet the world and record the time'
author: 'GitHub'
inputs:
inputs:
greeting: # id of input
description: 'The greeting we choose - will print ""{greeting}, World!"" on stdout'
required: true
@@ -1496,7 +1376,7 @@ runs:
name: 'Hello World'
description: 'Greet the world and record the time'
author: 'Test Corporation'
inputs:
inputs:
greeting: # id of input
description: 'The greeting we choose - will print ""{greeting}, World!"" on stdout'
required: true
@@ -1553,7 +1433,7 @@ runs:
private void CreateAction(string yamlContent, out Pipelines.ActionStep instance, out string directory)
{
directory = Path.Combine(_workFolder, Constants.Path.ActionsDirectory, "GitHub/actions".Replace(Path.AltDirectorySeparatorChar, Path.DirectorySeparatorChar), "master");
string file = Path.Combine(directory, Constants.Path.ActionManifestYmlFile);
string file = Path.Combine(directory, Constants.Path.ActionManifestFile);
Directory.CreateDirectory(Path.GetDirectoryName(file));
File.WriteAllText(file, yamlContent);
instance = new Pipelines.ActionStep()
@@ -1571,7 +1451,7 @@ runs:
private void CreateSelfRepoAction(string yamlContent, out Pipelines.ActionStep instance, out string directory)
{
directory = Path.Combine(_workFolder, "actions", "actions");
string file = Path.Combine(directory, Constants.Path.ActionManifestYmlFile);
string file = Path.Combine(directory, Constants.Path.ActionManifestFile);
Directory.CreateDirectory(Path.GetDirectoryName(file));
File.WriteAllText(file, yamlContent);
instance = new Pipelines.ActionStep()

View File

@@ -109,52 +109,6 @@ namespace GitHub.Runner.Common.Tests.Worker
}
}
[Fact]
[Trait("Level", "L0")]
[Trait("Category", "Worker")]
public void Load_ContainerAction_Dockerfile_Post_DefaultCondition()
{
try
{
//Arrange
Setup();
var actionManifest = new ActionManifestManager();
actionManifest.Initialize(_hc);
//Act
var result = actionManifest.Load(_ec.Object, Path.Combine(TestUtil.GetTestDataPath(), "dockerfileaction_cleanup_default.yml"));
//Assert
Assert.Equal("Hello World", result.Name);
Assert.Equal("Greet the world and record the time", result.Description);
Assert.Equal(2, result.Inputs.Count);
Assert.Equal("greeting", result.Inputs[0].Key.AssertString("key").Value);
Assert.Equal("Hello", result.Inputs[0].Value.AssertString("value").Value);
Assert.Equal("entryPoint", result.Inputs[1].Key.AssertString("key").Value);
Assert.Equal("", result.Inputs[1].Value.AssertString("value").Value);
Assert.Equal(ActionExecutionType.Container, result.Execution.ExecutionType);
var containerAction = result.Execution as ContainerActionExecutionData;
Assert.Equal("Dockerfile", containerAction.Image);
Assert.Equal("main.sh", containerAction.EntryPoint);
Assert.Equal("cleanup.sh", containerAction.Cleanup);
Assert.Equal("always()", containerAction.CleanupCondition);
Assert.Equal("bzz", containerAction.Arguments[0].ToString());
Assert.Equal("Token", containerAction.Environment[0].Key.ToString());
Assert.Equal("foo", containerAction.Environment[0].Value.ToString());
Assert.Equal("Url", containerAction.Environment[1].Key.ToString());
Assert.Equal("bar", containerAction.Environment[1].Value.ToString());
}
finally
{
Teardown();
}
}
[Fact]
[Trait("Level", "L0")]
[Trait("Category", "Worker")]
@@ -365,50 +319,6 @@ namespace GitHub.Runner.Common.Tests.Worker
}
}
[Fact]
[Trait("Level", "L0")]
[Trait("Category", "Worker")]
public void Load_NodeAction_Cleanup_DefaultCondition()
{
try
{
//Arrange
Setup();
var actionManifest = new ActionManifestManager();
actionManifest.Initialize(_hc);
//Act
var result = actionManifest.Load(_ec.Object, Path.Combine(TestUtil.GetTestDataPath(), "nodeaction_cleanup_default.yml"));
//Assert
Assert.Equal("Hello World", result.Name);
Assert.Equal("Greet the world and record the time", result.Description);
Assert.Equal(2, result.Inputs.Count);
Assert.Equal("greeting", result.Inputs[0].Key.AssertString("key").Value);
Assert.Equal("Hello", result.Inputs[0].Value.AssertString("value").Value);
Assert.Equal("entryPoint", result.Inputs[1].Key.AssertString("key").Value);
Assert.Equal("", result.Inputs[1].Value.AssertString("value").Value);
Assert.Equal(1, result.Deprecated.Count);
Assert.True(result.Deprecated.ContainsKey("greeting"));
result.Deprecated.TryGetValue("greeting", out string value);
Assert.Equal("This property has been deprecated", value);
Assert.Equal(ActionExecutionType.NodeJS, result.Execution.ExecutionType);
var nodeAction = result.Execution as NodeJSActionExecutionData;
Assert.Equal("main.js", nodeAction.Script);
Assert.Equal("cleanup.js", nodeAction.Cleanup);
Assert.Equal("always()", nodeAction.CleanupCondition);
}
finally
{
Teardown();
}
}
[Fact]
[Trait("Level", "L0")]
[Trait("Category", "Worker")]

View File

@@ -314,12 +314,6 @@ namespace GitHub.Runner.Common.Tests.Worker
githubContext.Add("event", JToken.Parse("{\"foo\":\"bar\"}").ToPipelineContextData());
_context.Add("github", githubContext);
#if OS_WINDOWS
_context["env"] = new DictionaryContextData();
#else
_context["env"] = new CaseSensitiveDictionaryContextData();
#endif
_ec = new Mock<IExecutionContext>();
_ec.Setup(x => x.ExpressionValues).Returns(_context);
_ec.Setup(x => x.IntraActionState).Returns(new Dictionary<string, string>());

View File

@@ -25,7 +25,7 @@ namespace GitHub.Runner.Common.Tests.Worker
TimelineReference timeline = new TimelineReference();
Guid jobId = Guid.NewGuid();
string jobName = "some job name";
var jobRequest = new Pipelines.AgentJobRequestMessage(plan, timeline, jobId, jobName, jobName, null, null, null, new Dictionary<string, VariableValue>(), new List<MaskHint>(), new Pipelines.JobResources(), new Pipelines.ContextData.DictionaryContextData(), new Pipelines.WorkspaceOptions(), new List<Pipelines.ActionStep>(), null, null);
var jobRequest = new Pipelines.AgentJobRequestMessage(plan, timeline, jobId, jobName, jobName, null, null, null, new Dictionary<string, VariableValue>(), new List<MaskHint>(), new Pipelines.JobResources(), new Pipelines.ContextData.DictionaryContextData(), new Pipelines.WorkspaceOptions(), new List<Pipelines.ActionStep>(), null);
jobRequest.Resources.Repositories.Add(new Pipelines.RepositoryResource()
{
Alias = Pipelines.PipelineConstants.SelfAlias,
@@ -101,7 +101,7 @@ namespace GitHub.Runner.Common.Tests.Worker
TimelineReference timeline = new TimelineReference();
Guid jobId = Guid.NewGuid();
string jobName = "some job name";
var jobRequest = new Pipelines.AgentJobRequestMessage(plan, timeline, jobId, jobName, jobName, null, null, null, new Dictionary<string, VariableValue>(), new List<MaskHint>(), new Pipelines.JobResources(), new Pipelines.ContextData.DictionaryContextData(), new Pipelines.WorkspaceOptions(), new List<Pipelines.ActionStep>(), null, null);
var jobRequest = new Pipelines.AgentJobRequestMessage(plan, timeline, jobId, jobName, jobName, null, null, null, new Dictionary<string, VariableValue>(), new List<MaskHint>(), new Pipelines.JobResources(), new Pipelines.ContextData.DictionaryContextData(), new Pipelines.WorkspaceOptions(), new List<Pipelines.ActionStep>(), null);
jobRequest.Resources.Repositories.Add(new Pipelines.RepositoryResource()
{
Alias = Pipelines.PipelineConstants.SelfAlias,
@@ -152,7 +152,7 @@ namespace GitHub.Runner.Common.Tests.Worker
TimelineReference timeline = new TimelineReference();
Guid jobId = Guid.NewGuid();
string jobName = "some job name";
var jobRequest = new Pipelines.AgentJobRequestMessage(plan, timeline, jobId, jobName, jobName, null, null, null, new Dictionary<string, VariableValue>(), new List<MaskHint>(), new Pipelines.JobResources(), new Pipelines.ContextData.DictionaryContextData(), new Pipelines.WorkspaceOptions(), new List<Pipelines.ActionStep>(), null, null);
var jobRequest = new Pipelines.AgentJobRequestMessage(plan, timeline, jobId, jobName, jobName, null, null, null, new Dictionary<string, VariableValue>(), new List<MaskHint>(), new Pipelines.JobResources(), new Pipelines.ContextData.DictionaryContextData(), new Pipelines.WorkspaceOptions(), new List<Pipelines.ActionStep>(), null);
jobRequest.Resources.Repositories.Add(new Pipelines.RepositoryResource()
{
Alias = Pipelines.PipelineConstants.SelfAlias,

View File

@@ -100,7 +100,7 @@ namespace GitHub.Runner.Common.Tests.Worker
};
Guid jobId = Guid.NewGuid();
_message = new Pipelines.AgentJobRequestMessage(plan, timeline, jobId, "test", "test", null, null, null, new Dictionary<string, VariableValue>(), new List<MaskHint>(), new Pipelines.JobResources(), new Pipelines.ContextData.DictionaryContextData(), new Pipelines.WorkspaceOptions(), steps, null, null);
_message = new Pipelines.AgentJobRequestMessage(plan, timeline, jobId, "test", "test", null, null, null, new Dictionary<string, VariableValue>(), new List<MaskHint>(), new Pipelines.JobResources(), new Pipelines.ContextData.DictionaryContextData(), new Pipelines.WorkspaceOptions(), steps, null);
GitHubContext github = new GitHubContext();
github["repository"] = new Pipelines.ContextData.StringContextData("actions/runner");
_message.ContextData.Add("github", github);

View File

@@ -63,7 +63,7 @@ namespace GitHub.Runner.Common.Tests.Worker
TaskOrchestrationPlanReference plan = new TaskOrchestrationPlanReference();
TimelineReference timeline = new Timeline(Guid.NewGuid());
Guid jobId = Guid.NewGuid();
_message = new Pipelines.AgentJobRequestMessage(plan, timeline, jobId, testName, testName, null, null, null, new Dictionary<string, VariableValue>(), new List<MaskHint>(), new Pipelines.JobResources(), new Pipelines.ContextData.DictionaryContextData(), new Pipelines.WorkspaceOptions(), new List<Pipelines.ActionStep>(), null, null);
_message = new Pipelines.AgentJobRequestMessage(plan, timeline, jobId, testName, testName, null, null, null, new Dictionary<string, VariableValue>(), new List<MaskHint>(), new Pipelines.JobResources(), new Pipelines.ContextData.DictionaryContextData(), new Pipelines.WorkspaceOptions(), new List<Pipelines.ActionStep>(), null);
_message.Variables[Constants.Variables.System.Culture] = "en-US";
_message.Resources.Endpoints.Add(new ServiceEndpoint()
{

View File

@@ -8,7 +8,6 @@ using System.Threading.Tasks;
using System.Runtime.CompilerServices;
using GitHub.Runner.Sdk;
using GitHub.Runner.Worker;
using GitHub.Runner.Worker.Container;
using GitHub.Runner.Worker.Handlers;
using Moq;
using Xunit;
@@ -749,130 +748,6 @@ namespace GitHub.Runner.Common.Tests.Worker
Environment.SetEnvironmentVariable("RUNNER_TEST_GET_REPOSITORY_PATH_FAILSAFE", "");
}
#if OS_LINUX
[Fact]
[Trait("Level", "L0")]
[Trait("Category", "Worker")]
public async void MatcherFile_JobContainer()
{
var matchers = new IssueMatchersConfig
{
Matchers =
{
new IssueMatcherConfig
{
Owner = "my-matcher-1",
Patterns = new[]
{
new IssuePatternConfig
{
Pattern = @"(.+): (.+)",
File = 1,
Message = 2,
},
},
},
},
};
var container = new ContainerInfo();
using (var hostContext = Setup(matchers: matchers, jobContainer: container))
using (_outputManager)
{
// Setup github.workspace, github.repository
var workDirectory = hostContext.GetDirectory(WellKnownDirectory.Work);
ArgUtil.NotNullOrEmpty(workDirectory, nameof(workDirectory));
Directory.CreateDirectory(workDirectory);
var workspaceDirectory = Path.Combine(workDirectory, "workspace");
Directory.CreateDirectory(workspaceDirectory);
_executionContext.Setup(x => x.GetGitHubContext("workspace")).Returns(workspaceDirectory);
_executionContext.Setup(x => x.GetGitHubContext("repository")).Returns("my-org/workflow-repo");
// Setup a git repository
await CreateRepository(hostContext, workspaceDirectory, "https://github.com/my-org/workflow-repo");
// Create test files
var file = Path.Combine(workspaceDirectory, "some-file.txt");
File.WriteAllText(file, "");
// Add translation path
container.AddPathTranslateMapping(workspaceDirectory, "/container/path/to/workspace");
// Process
Process($"/container/path/to/workspace/some-file.txt: some error 1");
Process($"some-file.txt: some error 2");
Assert.Equal(2, _issues.Count);
Assert.Equal("some error 1", _issues[0].Item1.Message);
Assert.Equal("some-file.txt", _issues[0].Item1.Data["file"]);
Assert.Equal("some error 2", _issues[1].Item1.Message);
Assert.Equal("some-file.txt", _issues[1].Item1.Data["file"]);
}
}
[Fact]
[Trait("Level", "L0")]
[Trait("Category", "Worker")]
public async void MatcherFile_StepContainer()
{
var matchers = new IssueMatchersConfig
{
Matchers =
{
new IssueMatcherConfig
{
Owner = "my-matcher-1",
Patterns = new[]
{
new IssuePatternConfig
{
Pattern = @"(.+): (.+)",
File = 1,
Message = 2,
},
},
},
},
};
var container = new ContainerInfo();
using (var hostContext = Setup(matchers: matchers, stepContainer: container))
using (_outputManager)
{
// Setup github.workspace, github.repository
var workDirectory = hostContext.GetDirectory(WellKnownDirectory.Work);
ArgUtil.NotNullOrEmpty(workDirectory, nameof(workDirectory));
Directory.CreateDirectory(workDirectory);
var workspaceDirectory = Path.Combine(workDirectory, "workspace");
Directory.CreateDirectory(workspaceDirectory);
_executionContext.Setup(x => x.GetGitHubContext("workspace")).Returns(workspaceDirectory);
_executionContext.Setup(x => x.GetGitHubContext("repository")).Returns("my-org/workflow-repo");
// Setup a git repository
await CreateRepository(hostContext, workspaceDirectory, "https://github.com/my-org/workflow-repo");
// Create test files
var file = Path.Combine(workspaceDirectory, "some-file.txt");
File.WriteAllText(file, "");
// Add translation path
container.AddPathTranslateMapping(workspaceDirectory, "/container/path/to/workspace");
// Process
Process($"/container/path/to/workspace/some-file.txt: some error 1");
Process($"some-file.txt: some error 2");
Assert.Equal(2, _issues.Count);
Assert.Equal("some error 1", _issues[0].Item1.Message);
Assert.Equal("some-file.txt", _issues[0].Item1.Data["file"]);
Assert.Equal("some error 2", _issues[1].Item1.Message);
Assert.Equal("some-file.txt", _issues[1].Item1.Data["file"]);
}
}
#endif
[Fact]
[Trait("Level", "L0")]
[Trait("Category", "Worker")]
@@ -931,9 +806,7 @@ namespace GitHub.Runner.Common.Tests.Worker
private TestHostContext Setup(
[CallerMemberName] string name = "",
IssueMatchersConfig matchers = null,
ContainerInfo jobContainer = null,
ContainerInfo stepContainer = null)
IssueMatchersConfig matchers = null)
{
matchers?.Validate();
@@ -951,8 +824,6 @@ namespace GitHub.Runner.Common.Tests.Worker
.Returns(true);
_executionContext.Setup(x => x.Variables)
.Returns(_variables);
_executionContext.Setup(x => x.Container)
.Returns(jobContainer);
_executionContext.Setup(x => x.GetMatchers())
.Returns(matchers?.Matchers ?? new List<IssueMatcherConfig>());
_executionContext.Setup(x => x.Add(It.IsAny<OnMatcherChanged>()))
@@ -973,8 +844,8 @@ namespace GitHub.Runner.Common.Tests.Worker
});
_commandManager = new Mock<IActionCommandManager>();
_commandManager.Setup(x => x.TryProcessCommand(It.IsAny<IExecutionContext>(), It.IsAny<string>(), It.IsAny<ContainerInfo>()))
.Returns((IExecutionContext executionContext, string line, ContainerInfo container) =>
_commandManager.Setup(x => x.TryProcessCommand(It.IsAny<IExecutionContext>(), It.IsAny<string>()))
.Returns((IExecutionContext executionContext, string line) =>
{
if (line.IndexOf("##[some-command]") >= 0)
{
@@ -985,7 +856,7 @@ namespace GitHub.Runner.Common.Tests.Worker
return false;
});
_outputManager = new OutputManager(_executionContext.Object, _commandManager.Object, stepContainer);
_outputManager = new OutputManager(_executionContext.Object, _commandManager.Object);
return hostContext;
}

View File

@@ -19,7 +19,6 @@ namespace GitHub.Runner.Common.Tests.Worker
private Mock<IExecutionContext> _ec;
private StepsRunner _stepsRunner;
private Variables _variables;
private Dictionary<string, string> _env;
private DictionaryContextData _contexts;
private JobContext _jobContext;
private StepsContext _stepContext;
@@ -33,11 +32,6 @@ namespace GitHub.Runner.Common.Tests.Worker
_variables = new Variables(
hostContext: hc,
copy: variablesToCopy);
_env = new Dictionary<string, string>()
{
{"env1", "1"},
{"test", "github_actions"}
};
_ec = new Mock<IExecutionContext>();
_ec.SetupAllProperties();
_ec.Setup(x => x.Variables).Returns(_variables);
@@ -70,9 +64,9 @@ namespace GitHub.Runner.Common.Tests.Worker
// Arrange.
var variableSets = new[]
{
new[] { CreateStep(hc, TaskResult.Succeeded, "success()"), CreateStep(hc, TaskResult.Succeeded, "success()") },
new[] { CreateStep(hc, TaskResult.Succeeded, "success()"), CreateStep(hc, TaskResult.Succeeded, "success() || failure()") },
new[] { CreateStep(hc, TaskResult.Succeeded, "success()"), CreateStep(hc, TaskResult.Succeeded, "always()") }
new[] { CreateStep(TaskResult.Succeeded, "success()"), CreateStep(TaskResult.Succeeded, "success()") },
new[] { CreateStep(TaskResult.Succeeded, "success()"), CreateStep(TaskResult.Succeeded, "success() || failure()") },
new[] { CreateStep(TaskResult.Succeeded, "success()"), CreateStep(TaskResult.Succeeded, "always()") }
};
foreach (var variableSet in variableSets)
{
@@ -102,12 +96,12 @@ namespace GitHub.Runner.Common.Tests.Worker
// Arrange.
var variableSets = new[]
{
new[] { CreateStep(hc, TaskResult.Failed, "success()", true), CreateStep(hc, TaskResult.Succeeded, "success()") },
new[] { CreateStep(hc, TaskResult.Failed, "success()", true), CreateStep(hc, TaskResult.Succeeded, "success() || failure()") },
new[] { CreateStep(hc, TaskResult.Failed, "success()", true), CreateStep(hc, TaskResult.Succeeded, "always()") },
new[] { CreateStep(hc, TaskResult.Failed, "success()", true), CreateStep(hc, TaskResult.Failed, "success()", true) },
new[] { CreateStep(hc, TaskResult.Failed, "success()", true), CreateStep(hc, TaskResult.Failed, "success() || failure()", true) },
new[] { CreateStep(hc, TaskResult.Failed, "success()", true), CreateStep(hc, TaskResult.Failed, "always()", true) }
new[] { CreateStep(TaskResult.Failed, "success()", true), CreateStep(TaskResult.Succeeded, "success()") },
new[] { CreateStep(TaskResult.Failed, "success()", true), CreateStep(TaskResult.Succeeded, "success() || failure()") },
new[] { CreateStep(TaskResult.Failed, "success()", true), CreateStep(TaskResult.Succeeded, "always()") },
new[] { CreateStep(TaskResult.Failed, "success()", true), CreateStep(TaskResult.Failed, "success()", true) },
new[] { CreateStep(TaskResult.Failed, "success()", true), CreateStep(TaskResult.Failed, "success() || failure()", true) },
new[] { CreateStep(TaskResult.Failed, "success()", true), CreateStep(TaskResult.Failed, "always()", true) }
};
foreach (var variableSet in variableSets)
{
@@ -139,12 +133,12 @@ namespace GitHub.Runner.Common.Tests.Worker
{
new
{
Steps = new[] { CreateStep(hc, TaskResult.Failed, "success()"), CreateStep(hc, TaskResult.Succeeded, "success()") },
Steps = new[] { CreateStep(TaskResult.Failed, "success()"), CreateStep(TaskResult.Succeeded, "success()") },
Expected = false,
},
new
{
Steps = new[] { CreateStep(hc, TaskResult.Failed, "success()"), CreateStep(hc, TaskResult.Succeeded, "success() || failure()") },
Steps = new[] { CreateStep(TaskResult.Failed, "success()"), CreateStep(TaskResult.Succeeded, "success() || failure()") },
Expected = true,
},
};
@@ -178,27 +172,27 @@ namespace GitHub.Runner.Common.Tests.Worker
{
new
{
Steps = new[] { CreateStep(hc, TaskResult.Succeeded, "success()"), CreateStep(hc, TaskResult.Succeeded, "always()") },
Steps = new[] { CreateStep(TaskResult.Succeeded, "success()"), CreateStep(TaskResult.Succeeded, "always()") },
Expected = TaskResult.Succeeded,
},
new
{
Steps = new[] { CreateStep(hc, TaskResult.Failed, "success()"), CreateStep(hc, TaskResult.Succeeded, "always()") },
Steps = new[] { CreateStep(TaskResult.Failed, "success()"), CreateStep(TaskResult.Succeeded, "always()") },
Expected = TaskResult.Failed,
},
new
{
Steps = new[] { CreateStep(hc, TaskResult.Failed, "success()"), CreateStep(hc, TaskResult.Succeeded, "always()") },
Steps = new[] { CreateStep(TaskResult.Failed, "success()"), CreateStep(TaskResult.Succeeded, "always()") },
Expected = TaskResult.Failed,
},
new
{
Steps = new[] { CreateStep(hc, TaskResult.Succeeded, "success()"), CreateStep(hc, TaskResult.Failed, "always()") },
Steps = new[] { CreateStep(TaskResult.Succeeded, "success()"), CreateStep(TaskResult.Failed, "always()") },
Expected = TaskResult.Failed,
},
new
{
Steps = new[] { CreateStep(hc, TaskResult.Succeeded, "success()"), CreateStep(hc, TaskResult.Failed, "always()", true) },
Steps = new[] { CreateStep(TaskResult.Succeeded, "success()"), CreateStep(TaskResult.Failed, "always()", true) },
Expected = TaskResult.Succeeded,
},
};
@@ -232,47 +226,47 @@ namespace GitHub.Runner.Common.Tests.Worker
{
new
{
Steps = new[] { CreateStep(hc, TaskResult.Failed, "success()"), CreateStep(hc, TaskResult.Succeeded, "success()") },
Steps = new[] { CreateStep(TaskResult.Failed, "success()"), CreateStep(TaskResult.Succeeded, "success()") },
Expected = TaskResult.Failed
},
new
{
Steps = new[] { CreateStep(hc, TaskResult.Failed, "success()"), CreateStep(hc, TaskResult.Succeeded, "success() || failure()") },
Steps = new[] { CreateStep(TaskResult.Failed, "success()"), CreateStep(TaskResult.Succeeded, "success() || failure()") },
Expected = TaskResult.Failed
},
new
{
Steps = new[] { CreateStep(hc, TaskResult.Failed, "success()"), CreateStep(hc, TaskResult.Succeeded, "always()") },
Steps = new[] { CreateStep(TaskResult.Failed, "success()"), CreateStep(TaskResult.Succeeded, "always()") },
Expected = TaskResult.Failed
},
new
{
Steps = new[] { CreateStep(hc, TaskResult.Failed, "success()", continueOnError: true), CreateStep(hc, TaskResult.Failed, "success()") },
Steps = new[] { CreateStep(TaskResult.Failed, "success()", continueOnError: true), CreateStep(TaskResult.Failed, "success()") },
Expected = TaskResult.Failed
},
new
{
Steps = new[] { CreateStep(hc, TaskResult.Failed, "success()", continueOnError: true), CreateStep(hc, TaskResult.Succeeded, "success()") },
Steps = new[] { CreateStep(TaskResult.Failed, "success()", continueOnError: true), CreateStep(TaskResult.Succeeded, "success()") },
Expected = TaskResult.Succeeded
},
new
{
Steps = new[] { CreateStep(hc, TaskResult.Failed, "success()", continueOnError: true), CreateStep(hc, TaskResult.Failed, "success()", continueOnError: true) },
Steps = new[] { CreateStep(TaskResult.Failed, "success()", continueOnError: true), CreateStep(TaskResult.Failed, "success()", continueOnError: true) },
Expected = TaskResult.Succeeded
},
new
{
Steps = new[] { CreateStep(hc, TaskResult.Succeeded, "success() || failure()") },
Steps = new[] { CreateStep(TaskResult.Succeeded, "success() || failure()") },
Expected = TaskResult.Succeeded
},
new
{
Steps = new[] { CreateStep(hc, TaskResult.Succeeded, "success()"), CreateStep(hc, TaskResult.Failed, "success()") },
Steps = new[] { CreateStep(TaskResult.Succeeded, "success()"), CreateStep(TaskResult.Failed, "success()") },
Expected = TaskResult.Failed
},
new
{
Steps = new[] { CreateStep(hc, TaskResult.Succeeded, "success()"), CreateStep(hc, TaskResult.Succeeded, "success()") },
Steps = new[] { CreateStep(TaskResult.Succeeded, "success()"), CreateStep(TaskResult.Succeeded, "success()") },
Expected = TaskResult.Succeeded
},
// Abandoned
@@ -310,17 +304,17 @@ namespace GitHub.Runner.Common.Tests.Worker
{
new
{
Step = new[] { CreateStep(hc, TaskResult.Failed, "success()"), CreateStep(hc, TaskResult.Succeeded, "success()") },
Step = new[] { CreateStep(TaskResult.Failed, "success()"), CreateStep(TaskResult.Succeeded, "success()") },
Expected = false
},
new
{
Step = new[] { CreateStep(hc, TaskResult.Failed, "success()"), CreateStep(hc, TaskResult.Succeeded, "success() || failure()") },
Step = new[] { CreateStep(TaskResult.Failed, "success()"), CreateStep(TaskResult.Succeeded, "success() || failure()") },
Expected = true
},
new
{
Step = new[] { CreateStep(hc, TaskResult.Failed, "success()"), CreateStep(hc, TaskResult.Succeeded, "always()") },
Step = new[] { CreateStep(TaskResult.Failed, "success()"), CreateStep(TaskResult.Succeeded, "always()") },
Expected = true
}
};
@@ -351,9 +345,9 @@ namespace GitHub.Runner.Common.Tests.Worker
// Arrange.
var variableSets = new[]
{
new[] { CreateStep(hc, TaskResult.Succeeded, "success()"), CreateStep(hc, TaskResult.Succeeded, "always()") },
new[] { CreateStep(hc, TaskResult.Failed, "success()"), CreateStep(hc, TaskResult.Succeeded, "always()") },
new[] { CreateStep(hc, TaskResult.Canceled, "success()"), CreateStep(hc, TaskResult.Succeeded, "always()") }
new[] { CreateStep(TaskResult.Succeeded, "success()"), CreateStep(TaskResult.Succeeded, "always()") },
new[] { CreateStep(TaskResult.Failed, "success()"), CreateStep(TaskResult.Succeeded, "always()") },
new[] { CreateStep(TaskResult.Canceled, "success()"), CreateStep(TaskResult.Succeeded, "always()") }
};
foreach (var variableSet in variableSets)
{
@@ -387,8 +381,8 @@ namespace GitHub.Runner.Common.Tests.Worker
// Arrange.
var variableSets = new[]
{
new[] { CreateStep(hc, TaskResult.Succeeded, "success()") },
new[] { CreateStep(hc, TaskResult.Succeeded, "success()") },
new[] { CreateStep(TaskResult.Succeeded, "success()") },
new[] { CreateStep(TaskResult.Succeeded, "success()") },
};
foreach (var variableSet in variableSets)
{
@@ -405,134 +399,18 @@ namespace GitHub.Runner.Common.Tests.Worker
}
}
[Fact]
[Trait("Level", "L0")]
[Trait("Category", "Worker")]
public async Task StepEnvOverrideJobEnvContext()
{
using (TestHostContext hc = CreateTestContext())
{
// Arrange.
var env1 = new MappingToken(null, null, null);
env1.Add(new StringToken(null, null, null, "env1"), new StringToken(null, null, null, "100"));
env1.Add(new StringToken(null, null, null, "env2"), new BasicExpressionToken(null, null, null, "env.test"));
var step1 = CreateStep(hc, TaskResult.Succeeded, "success()", env: env1);
_ec.Object.Result = null;
_ec.Setup(x => x.JobSteps).Returns(new Queue<IStep>(new[] { step1.Object }));
// Act.
await _stepsRunner.RunAsync(jobContext: _ec.Object);
// Assert.
Assert.Equal(TaskResult.Succeeded, _ec.Object.Result ?? TaskResult.Succeeded);
#if OS_WINDOWS
Assert.Equal("100", _ec.Object.ExpressionValues["env"].AssertDictionary("env")["env1"].AssertString("100"));
Assert.Equal("github_actions", _ec.Object.ExpressionValues["env"].AssertDictionary("env")["env2"].AssertString("github_actions"));
#else
Assert.Equal("100", _ec.Object.ExpressionValues["env"].AssertCaseSensitiveDictionary("env")["env1"].AssertString("100"));
Assert.Equal("github_actions", _ec.Object.ExpressionValues["env"].AssertCaseSensitiveDictionary("env")["env2"].AssertString("github_actions"));
#endif
}
}
[Fact]
[Trait("Level", "L0")]
[Trait("Category", "Worker")]
public async Task PopulateEnvContextForEachStep()
{
using (TestHostContext hc = CreateTestContext())
{
// Arrange.
var env1 = new MappingToken(null, null, null);
env1.Add(new StringToken(null, null, null, "env1"), new StringToken(null, null, null, "100"));
env1.Add(new StringToken(null, null, null, "env2"), new BasicExpressionToken(null, null, null, "env.test"));
var step1 = CreateStep(hc, TaskResult.Succeeded, "success()", env: env1);
var env2 = new MappingToken(null, null, null);
env2.Add(new StringToken(null, null, null, "env1"), new StringToken(null, null, null, "1000"));
env2.Add(new StringToken(null, null, null, "env3"), new BasicExpressionToken(null, null, null, "env.test"));
var step2 = CreateStep(hc, TaskResult.Succeeded, "success()", env: env2);
_ec.Object.Result = null;
_ec.Setup(x => x.JobSteps).Returns(new Queue<IStep>(new[] { step1.Object, step2.Object }));
// Act.
await _stepsRunner.RunAsync(jobContext: _ec.Object);
// Assert.
Assert.Equal(TaskResult.Succeeded, _ec.Object.Result ?? TaskResult.Succeeded);
#if OS_WINDOWS
Assert.Equal("1000", _ec.Object.ExpressionValues["env"].AssertDictionary("env")["env1"].AssertString("1000"));
Assert.Equal("github_actions", _ec.Object.ExpressionValues["env"].AssertDictionary("env")["env3"].AssertString("github_actions"));
Assert.False(_ec.Object.ExpressionValues["env"].AssertDictionary("env").ContainsKey("env2"));
#else
Assert.Equal("1000", _ec.Object.ExpressionValues["env"].AssertCaseSensitiveDictionary("env")["env1"].AssertString("1000"));
Assert.Equal("github_actions", _ec.Object.ExpressionValues["env"].AssertCaseSensitiveDictionary("env")["env3"].AssertString("github_actions"));
Assert.False(_ec.Object.ExpressionValues["env"].AssertCaseSensitiveDictionary("env").ContainsKey("env2"));
#endif
}
}
[Fact]
[Trait("Level", "L0")]
[Trait("Category", "Worker")]
public async Task PopulateEnvContextAfterSetupStepsContext()
{
using (TestHostContext hc = CreateTestContext())
{
// Arrange.
var env1 = new MappingToken(null, null, null);
env1.Add(new StringToken(null, null, null, "env1"), new StringToken(null, null, null, "100"));
var step1 = CreateStep(hc, TaskResult.Succeeded, "success()", env: env1, name: "foo", setOutput: true);
var env2 = new MappingToken(null, null, null);
env2.Add(new StringToken(null, null, null, "env1"), new StringToken(null, null, null, "1000"));
env2.Add(new StringToken(null, null, null, "env2"), new BasicExpressionToken(null, null, null, "steps.foo.outputs.test"));
var step2 = CreateStep(hc, TaskResult.Succeeded, "success()", env: env2);
_ec.Object.Result = null;
_ec.Setup(x => x.JobSteps).Returns(new Queue<IStep>(new[] { step1.Object, step2.Object }));
// Act.
await _stepsRunner.RunAsync(jobContext: _ec.Object);
// Assert.
Assert.Equal(TaskResult.Succeeded, _ec.Object.Result ?? TaskResult.Succeeded);
#if OS_WINDOWS
Assert.Equal("1000", _ec.Object.ExpressionValues["env"].AssertDictionary("env")["env1"].AssertString("1000"));
Assert.Equal("something", _ec.Object.ExpressionValues["env"].AssertDictionary("env")["env2"].AssertString("something"));
#else
Assert.Equal("1000", _ec.Object.ExpressionValues["env"].AssertCaseSensitiveDictionary("env")["env1"].AssertString("1000"));
Assert.Equal("something", _ec.Object.ExpressionValues["env"].AssertCaseSensitiveDictionary("env")["env2"].AssertString("something"));
#endif
}
}
private Mock<IActionRunner> CreateStep(TestHostContext hc, TaskResult result, string condition, Boolean continueOnError = false, MappingToken env = null, string name = "Test", bool setOutput = false)
private Mock<IStep> CreateStep(TaskResult result, string condition, Boolean continueOnError = false)
{
// Setup the step.
var step = new Mock<IActionRunner>();
var step = new Mock<IStep>();
step.Setup(x => x.Condition).Returns(condition);
step.Setup(x => x.ContinueOnError).Returns(new BooleanToken(null, null, null, continueOnError));
step.Setup(x => x.Action)
.Returns(new DistributedTask.Pipelines.ActionStep()
{
Name = name,
Id = Guid.NewGuid(),
Environment = env
});
step.Setup(x => x.RunAsync()).Returns(Task.CompletedTask);
// Setup the step execution context.
var stepContext = new Mock<IExecutionContext>();
stepContext.SetupAllProperties();
stepContext.Setup(x => x.WriteDebug).Returns(true);
stepContext.Setup(x => x.Variables).Returns(_variables);
stepContext.Setup(x => x.EnvironmentVariables).Returns(_env);
stepContext.Setup(x => x.ExpressionValues).Returns(_contexts);
stepContext.Setup(x => x.JobContext).Returns(_jobContext);
stepContext.Setup(x => x.StepsContext).Returns(_stepContext);
@@ -544,24 +422,13 @@ namespace GitHub.Runner.Common.Tests.Worker
stepContext.Object.Result = r;
}
});
var trace = hc.GetTrace();
stepContext.Setup(x => x.Write(It.IsAny<string>(), It.IsAny<string>())).Callback((string tag, string message) => { trace.Info($"[{tag}]{message}"); });
stepContext.Object.Result = result;
step.Setup(x => x.ExecutionContext).Returns(stepContext.Object);
if (setOutput)
{
step.Setup(x => x.RunAsync()).Callback(() => { _stepContext.SetOutput(null, name, "test", "something", out string reference); }).Returns(Task.CompletedTask);
}
else
{
step.Setup(x => x.RunAsync()).Returns(Task.CompletedTask);
}
return step;
}
private string FormatSteps(IEnumerable<Mock<IActionRunner>> steps)
private string FormatSteps(IEnumerable<Mock<IStep>> steps)
{
return String.Join(
" ; ",

View File

@@ -67,7 +67,7 @@ namespace GitHub.Runner.Common.Tests.Worker
new Pipelines.ContextData.DictionaryContextData()
},
};
var jobRequest = new Pipelines.AgentJobRequestMessage(plan, timeline, JobId, jobName, jobName, new StringToken(null, null, null, "ubuntu"), sidecarContainers, null, variables, new List<MaskHint>(), resources, context, null, actions, null, null);
var jobRequest = new Pipelines.AgentJobRequestMessage(plan, timeline, JobId, jobName, jobName, new StringToken(null, null, null, "ubuntu"), sidecarContainers, null, variables, new List<MaskHint>(), resources, context, null, actions, null);
return jobRequest;
}

View File

@@ -1,26 +0,0 @@
name: 'Hello World'
description: 'Greet the world and record the time'
author: 'Test Corporation'
inputs:
greeting: # id of input
description: 'The greeting we choose - will print ""{greeting}, World!"" on stdout'
required: true
default: 'Hello'
entryPoint: # id of input
description: 'optional docker entrypoint overwrite.'
required: false
outputs:
time: # id of output
description: 'The time we did the greeting'
icon: 'hello.svg' # vector art to display in the GitHub Marketplace
color: 'green' # optional, decorates the entry in the GitHub Marketplace
runs:
using: 'docker'
image: 'Dockerfile'
args:
- 'bzz'
entrypoint: 'main.sh'
env:
Token: foo
Url: bar
post-entrypoint: 'cleanup.sh'

View File

@@ -1,21 +0,0 @@
name: 'Hello World'
description: 'Greet the world and record the time'
author: 'Test Corporation'
inputs:
greeting: # id of input
description: 'The greeting we choose - will print ""{greeting}, World!"" on stdout'
required: true
default: 'Hello'
deprecationMessage: 'This property has been deprecated'
entryPoint: # id of input
description: 'optional docker entrypoint overwrite.'
required: false
outputs:
time: # id of output
description: 'The time we did the greeting'
icon: 'hello.svg' # vector art to display in the GitHub Marketplace
color: 'green' # optional, decorates the entry in the GitHub Marketplace
runs:
using: 'node12'
main: 'main.js'
post: 'cleanup.js'

View File

@@ -1,16 +1,5 @@
@setlocal
@echo off
rem add expected utils to path
IF EXIST C:\Program Files\Git\usr\bin (
SET PATH=C:\Program Files\Git\usr\bin;%PATH%
)
IF EXIST C:\Program Files\Git\mingw64\bin (
SET PATH=C:\Program Files\Git\mingw64\bin;%PATH%
)
IF EXIST C:\Program Files\Git\bin (
SET PATH=C:\Program Files\Git\bin;%PATH%
)
rem Check if SH_PATH is defined.
if defined SH_PATH (
goto run

View File

@@ -137,6 +137,8 @@ function runtest ()
ulimit -n 1024
fi
export GITHUB_RUNNER_SRC_DIR=${SCRIPT_DIR}
dotnet msbuild -t:test -p:PackageRuntime="${RUNTIME_ID}" -p:BUILDCONFIG="${BUILD_CONFIG}" -p:RunnerVersion="${RUNNER_VERSION}" ./dir.proj || failed "failed tests"
}

View File

@@ -1 +1 @@
2.165.2
2.164.0