Use different url format for accessing API

docs/adrs/0361-wrapper-action.md

@@ -1,75 +0,0 @@
-# ADR 361: Wrapper Action
-
-**Date**: 2020-03-06
-
-**Status**: Pending
-
-## Context
-
-In addition to action's regular execution, action author may wants their action has a chance to participate in:
-- Job initialize  
-  My Action will collect machine resource usage (CPU/RAM/Disk) during a workflow job execution, we need to start perf recorder at the begin of the job.
-- Job cleanup  
-  My Action will dirty local workspace or machine environment during execution, we need to cleanup these changes at the end of the job.  
-  Ex: `actions/checkout@v2` will write `github.token` into local `.git/config` during execution, it has post job cleanup defined to undo the changes.
-
-## Decision
-
-### Add `pre` and `post` execution to action
-
-Node Action Example:
-
-```yaml
- name: 'My action with pre'
- description: 'My action with pre'
- runs:
-   using: 'node12'
-   pre: 'setup.js'
-   pre-if: 'success()' // Optional
-   main: 'index.js'
-   post: 'cleanup.js'
-   post-if: 'success()' // Optional
-```
-
-Container Action Example:
-
-```yaml
- name: 'My action with pre'
- description: 'My action with pre'
- runs:
-   using: 'docker'
-   image: 'mycontainer:latest'
-   pre-entrypoint: 'setup.sh'
-   pre-if: 'success()' // Optional
-   entrypoint: 'entrypoint.sh'
-   post-entrypoint: 'cleanup.sh'
-   post-if: 'success()' // Optional
-```
-
-Both `pre` and `post` will has default `pre-if/post-if` sets to `always()`.  
-Setting `pre` to `always()` will make sure no matter what condition evaluate result the `main` gets at runtime, the `pre` has always run already.   
-`pre` executes in order of how the steps are defined.  
-`pre` will always be added to job steps list during job setup.  
-> Action referenced from local repository (`./my-action`) won't get `pre` setup correctly since the repository haven't checkout during job initialize.  
-> We can't use GitHub api to download the repository since there is a about 3 mins delay between `git push` and the new commit available to download using GitHub api.
-
-`post` will be pushed into a `poststeps` stack lazily when the action's `pre` or `main` execution passed `if` condition check and about to run, you can't have an action that only contains a `post`, we will pop and run each `post` after all `pre` and `main` finished.
-> Currently `post` works for both repository action (`org/repo@v1`) and local action (`./my-action`)
-
-Valid action:
-- only has `main`
-- has `pre` and `main`
-- has `main` and `post`
-- has `pre`, `main` and `post`
-
-Invalid action:
-- only has `pre`
-- only has `post`
-- has `pre` and `post`
-
-Potential downside of introducing `pre`:
-
-- Extra magic wrt step order. Users should control the step order. Especially when we introduce templates.  
-- Eliminates the possibility to lazily download the action tarball, since `pre` always run by default, we have to download the tarball to check whether action defined a `pre`  
-- `pre` doesn't work with local action, we suggested customer use local action for testing their action changes, ex CI for their action, to avoid delay between `git push` and GitHub repo tarball download api.  
-- Condition on the `pre` can't be controlled using dynamic step outputs. `pre` executes too early.

docs/adrs/0397-runner-registration-labels.md

@@ -1,56 +0,0 @@
-# ADR 0397: Support adding custom labels during runner config
-**Date**: 2020-03-30
-
-**Status**: Approved
-
-## Context
-
-Since configuring self-hosted runners is commonly automated via scripts, the labels need to be able to be created during configuration.  The runner currently registers the built-in labels (os, arch) during registration but does not accept labels via command line args to extend the set registered.
-
-See Issue: https://github.com/actions/runner/issues/262
-
-This is another version of [ADR275](https://github.com/actions/runner/pull/275)
-
-## Decision
-
-This ADR proposes that we add a `--labels` option to `config`, which could be used to add custom additional labels to the configured runner.
-
-For example, to add a single extra label the operator could run:
-```bash
-./config.sh --labels mylabel
-```
-> Note: the current runner command line parsing and envvar override algorithm only supports a single argument (key).
-
-This would add the label `mylabel` to the runner, and enable users to select the runner in their workflow using this label:
-```yaml
-runs-on: [self-hosted, mylabel]
-```
-
-To add multiple labels the operator could run:
-```bash
-./config.sh --labels mylabel,anotherlabel
-```
-> Note: the current runner command line parsing and envvar override algorithm only supports a single argument (key).
-
-This would add the label `mylabel` and `anotherlabel` to the runner, and enable users to select the runner in their workflow using this label:
-```yaml
-runs-on: [self-hosted, mylabel, anotherlabel]
-```
-
-It would not be possible to remove labels from an existing runner using `config.sh`, instead labels would have to be removed using the GitHub UI.
-
-The labels argument will split on commas, trim and discard empty strings.  That effectively means don't use commans in unattended config label names.  Alternatively we could choose to escape commans but it's a nice to have.
-
-## Replace
-
-If an existing runner exists and the option to replace is chosen (interactively of via unattend as in this scenario), then the labels will be replaced / overwritten (not merged).
-
-## Overriding built-in labels
-
-Note that it is possible to register "built-in" hosted labels like `ubuntu-latest` and is not considered an error.  This is an effective way for the org / runner admin to dictate by policy through registration that this set of runners will be used without having to edit all the workflow files now and in the future.
-
-We will also not make other restrictions such as limiting explicitly adding os / arch labels and validating.  We will assume that explicit labels were added for a reason and not restricting offers the most flexibility and future proofing / compat.
-
-## Consequences
-
-The ability to add custom labels to a self-hosted runner would enable most scenarios where job runner selection based on runner capabilities or characteristics are required.

releaseNote.md

@@ -1,11 +1,24 @@
 ## Features
-  - Runner support for GHES Alpha (#381 #386 #390 #393 $401) 
-  - Allow secrets context in Container.env (#388)
+  - Update Runner Register GitHub API URL to Support Org-level Runner (#339 #345 #352) 
+  - Preserve workflow file/line/column for better error messages (#356)
+  - Switch to use token service instead of SPS for exchanging oauth token. (#325)
+  - Load and print machine setup info from .setup_info (#364)
+  - Expose job name as $GITHUB_JOB (#366)
+  - Add support for job outputs. (#365) 
+  - Set CI=true when launch process in actions runner. (#374)
+  - Set steps.<id>.outcome and steps.<id>.conclusion. (#372)
+  - Add support for workflow/job defaults. (#369)
+  - Expose GITHUB_REPOSITORY_OWNER and ${{github.repository_owner}}. (#378)
+
 ## Bugs
-  - Raise warning when volume mount root. (#413)
-  - Fix typo (#394)
+  - Use authenticate endpoint for testing runner connection. (#311) 
+  - Commands translate file path from container action (#331)
+  - Change problem matchers output to debug (#363)
+  - Switch hashFiles to extension function (#362)
+  - Add expanded volumes strings to container mounts (#384)
+
 ## Misc
-  - N/A
+  - Add runner auth documentation (#357) 
 
 ## Windows x64
 We recommend configuring the runner in a root folder of the Windows drive (e.g. "C:\actions-runner"). This will help avoid issues related to service identity folder permissions and long file path restrictions on Windows

src/Misc/dotnet-install.sh

@@ -172,7 +172,7 @@ get_current_os_name() {
         return 0
     elif [ "$uname" = "FreeBSD" ]; then
         echo "freebsd"
-        return 0
+        return 0        
     elif [ "$uname" = "Linux" ]; then
         local linux_platform_name
         linux_platform_name="$(get_linux_platform_name)" || { echo "linux" && return 0 ; }
@@ -728,12 +728,11 @@ downloadcurl() {
     # Append feed_credential as late as possible before calling curl to avoid logging feed_credential
     remote_path="${remote_path}${feed_credential}"
 
-    local curl_options="--retry 20 --retry-delay 2 --connect-timeout 15 -sSL -f --create-dirs "
     local failed=false
     if [ -z "$out_path" ]; then
-        curl $curl_options "$remote_path" || failed=true
+        curl --retry 10 -sSL -f --create-dirs "$remote_path" || failed=true
     else
-        curl $curl_options -o "$out_path" "$remote_path" || failed=true
+        curl --retry 10 -sSL -f --create-dirs -o "$out_path" "$remote_path" || failed=true
     fi
     if [ "$failed" = true ]; then
         say_verbose "Curl download failed"
@@ -749,12 +748,12 @@ downloadwget() {
 
     # Append feed_credential as late as possible before calling wget to avoid logging feed_credential
     remote_path="${remote_path}${feed_credential}"
-    local wget_options="--tries 20 --waitretry 2 --connect-timeout 15 "
+
     local failed=false
     if [ -z "$out_path" ]; then
-        wget -q $wget_options -O - "$remote_path" || failed=true
+        wget -q --tries 10 -O - "$remote_path" || failed=true
     else
-        wget $wget_options -O "$out_path" "$remote_path" || failed=true
+        wget --tries 10 -O "$out_path" "$remote_path" || failed=true
     fi
     if [ "$failed" = true ]; then
         say_verbose "Wget download failed"

src/Misc/expressionFunc/hashFiles/package.json

@@ -27,7 +27,7 @@
     "@types/node": "^12.7.12",
     "@typescript-eslint/parser": "^2.8.0",
     "@zeit/ncc": "^0.20.5",
-    "eslint": "^6.8.0",
+    "eslint": "^5.16.0",
     "eslint-plugin-github": "^2.0.0",
     "prettier": "^1.19.1",
     "typescript": "^3.6.4"

src/Runner.Listener/Configuration/ConfigurationManager.cs

@@ -93,8 +93,11 @@ namespace GitHub.Runner.Listener.Configuration
             while (true)
             {
                 // Get the URL
+                var isGitHub = StringUtil.ConvertToBoolean(Environment.GetEnvironmentVariable("_IS_GITHUB"));
                 var inputUrl = command.GetUrl();
-                if (inputUrl.Contains("codedev.ms", StringComparison.OrdinalIgnoreCase))
+                if (!inputUrl.Contains("github.com", StringComparison.OrdinalIgnoreCase) &&
+                    !inputUrl.Contains("github.localhost", StringComparison.OrdinalIgnoreCase) &&
+                    !isGitHub)
                 {
                     runnerSettings.ServerUrl = inputUrl;
                     // Get the credentials
@@ -115,7 +118,7 @@ namespace GitHub.Runner.Listener.Configuration
                 try
                 {
                     // Determine the service deployment type based on connection data. (Hosted/OnPremises)
-                    runnerSettings.IsHostedServer = runnerSettings.GitHubUrl == null || IsHostedServer(new UriBuilder(runnerSettings.GitHubUrl));
+                    runnerSettings.IsHostedServer = await IsHostedServer(runnerSettings.ServerUrl, creds);
 
                     // Validate can connect.
                     await _runnerServer.ConnectAsync(new Uri(runnerSettings.ServerUrl), creds);
@@ -246,6 +249,14 @@ namespace GitHub.Runner.Listener.Configuration
             {
                 UriBuilder configServerUrl = new UriBuilder(runnerSettings.ServerUrl);
                 UriBuilder oauthEndpointUrlBuilder = new UriBuilder(agent.Authorization.AuthorizationUrl);
+                if (!runnerSettings.IsHostedServer && Uri.Compare(configServerUrl.Uri, oauthEndpointUrlBuilder.Uri, UriComponents.SchemeAndServer, UriFormat.Unescaped, StringComparison.OrdinalIgnoreCase) != 0)
+                {
+                    oauthEndpointUrlBuilder.Scheme = configServerUrl.Scheme;
+                    oauthEndpointUrlBuilder.Host = configServerUrl.Host;
+                    oauthEndpointUrlBuilder.Port = configServerUrl.Port;
+                    Trace.Info($"Set oauth endpoint url's scheme://host:port component to match runner configure url's scheme://host:port: '{oauthEndpointUrlBuilder.Uri.AbsoluteUri}'.");
+                }
+
                 var credentialData = new CredentialData
                 {
                     Scheme = Constants.Configuration.OAuth,
@@ -487,26 +498,31 @@ namespace GitHub.Runner.Listener.Configuration
             return agent;
         }
 
-        private bool IsHostedServer(UriBuilder gitHubUrl)
+        private async Task<bool> IsHostedServer(string serverUrl, VssCredentials credentials)
         {
-            return string.Equals(gitHubUrl.Host, "github.com", StringComparison.OrdinalIgnoreCase) ||
-                string.Equals(gitHubUrl.Host, "www.github.com", StringComparison.OrdinalIgnoreCase) ||
-                string.Equals(gitHubUrl.Host, "github.localhost", StringComparison.OrdinalIgnoreCase);
+            // Determine the service deployment type based on connection data. (Hosted/OnPremises)
+            var locationServer = HostContext.GetService<ILocationServer>();
+            VssConnection connection = VssUtil.CreateConnection(new Uri(serverUrl), credentials);
+            await locationServer.ConnectAsync(connection);
+            try
+            {
+                var connectionData = await locationServer.GetConnectionDataAsync();
+                Trace.Info($"Server deployment type: {connectionData.DeploymentType}");
+                return connectionData.DeploymentType.HasFlag(DeploymentFlags.Hosted);
+            }
+            catch (Exception ex)
+            {
+                // Since the DeploymentType is Enum, deserialization exception means there is a new Enum member been added.
+                // It's more likely to be Hosted since OnPremises is always behind and customer can update their agent if are on-prem
+                Trace.Error(ex);
+                return true;
+            }
         }
 
         private async Task<GitHubAuthResult> GetTenantCredential(string githubUrl, string githubToken, string runnerEvent)
         {
-            var githubApiUrl = "";
             var gitHubUrlBuilder = new UriBuilder(githubUrl);
-            if (IsHostedServer(gitHubUrlBuilder))
-            {
-                githubApiUrl = $"{gitHubUrlBuilder.Scheme}://api.{gitHubUrlBuilder.Host}/actions/runner-registration";
-            }
-            else
-            {
-                githubApiUrl = $"{gitHubUrlBuilder.Scheme}://{gitHubUrlBuilder.Host}/api/v3/actions/runner-registration";
-            }
-
+            var githubApiUrl = $"{gitHubUrlBuilder.Scheme}://{gitHubUrlBuilder.Host}/api/v3/actions/runner-registration";
             using (var httpClientHandler = HostContext.CreateHttpClientHandler())
             using (var httpClient = new HttpClient(httpClientHandler))
             {

src/Runner.Worker/ActionManager.cs

@@ -51,11 +51,11 @@ namespace GitHub.Runner.Worker
             List<JobExtensionRunner> containerSetupSteps = new List<JobExtensionRunner>();
             IEnumerable<Pipelines.ActionStep> actions = steps.OfType<Pipelines.ActionStep>();
 
-            // TODO: Deprecate the PREVIEW_ACTION_TOKEN
+            // TODO: Depreciate the PREVIEW_ACTION_TOKEN
             // Log even if we aren't using it to ensure users know.
             if (!string.IsNullOrEmpty(executionContext.Variables.Get("PREVIEW_ACTION_TOKEN")))
             {
-                executionContext.Warning("The 'PREVIEW_ACTION_TOKEN' secret is deprecated. Please remove it from the repository's secrets");
+                executionContext.Warning("The 'PREVIEW_ACTION_TOKEN' secret is depreciated. Please remove it from the repository's secrets");
             }
 
             // Clear the cache (for self-hosted runners)
@@ -512,7 +512,7 @@ namespace GitHub.Runner.Worker
                                     var authToken = Environment.GetEnvironmentVariable("_GITHUB_ACTION_TOKEN");
                                     if (string.IsNullOrEmpty(authToken))
                                     {
-                                        // TODO: Deprecate the PREVIEW_ACTION_TOKEN
+                                        // TODO: Depreciate the PREVIEW_ACTION_TOKEN
                                         authToken = executionContext.Variables.Get("PREVIEW_ACTION_TOKEN");
                                     }
 

src/Runner.Worker/ContainerOperationProvider.cs

@@ -47,7 +47,7 @@ namespace GitHub.Runner.Worker
                                                 condition: $"{PipelineTemplateConstants.Always}()",
                                                 displayName: "Stop containers",
                                                 data: data);
-
+            
             executionContext.Debug($"Register post job cleanup for stopping/deleting containers.");
             executionContext.RegisterPostJobStep(nameof(StopContainersAsync), postJobStep);
 
@@ -180,11 +180,6 @@ namespace GitHub.Runner.Worker
             foreach (var volume in container.UserMountVolumes)
             {
                 Trace.Info($"User provided volume: {volume.Value}");
-                var mount = new MountVolume(volume.Value);
-                if (string.Equals(mount.SourceVolumePath, "/", StringComparison.OrdinalIgnoreCase))
-                {
-                    executionContext.Warning($"Volume mount {volume.Value} is going to mount '/' into the container which may cause file ownership change in the entire file system and cause Actions Runner to lose permission to access the disk.");
-                }
             }
 
             // Pull down docker image with retry up to 3 times

src/Runner.Worker/JobExtension.cs

@@ -136,7 +136,7 @@ namespace GitHub.Runner.Worker
                         var url = new Uri(runnerSettings.GitHubUrl);
                         var portInfo = url.IsDefaultPort ? string.Empty : $":{url.Port.ToString(CultureInfo.InvariantCulture)}";
                         context.SetGitHubContext("url", $"{url.Scheme}://{url.Host}{portInfo}");
-                        context.SetGitHubContext("api_url", $"{url.Scheme}://{url.Host}{portInfo}/api/v3");
+                        context.SetGitHubContext("api_url", $"{url.Scheme}://api.{url.Host}{portInfo}");
                     }
 
                     // Evaluate the job-level environment variables

src/Sdk/DTPipelines/Pipelines/ObjectTemplating/PipelineTemplateConstants.cs

@@ -8,7 +8,6 @@ namespace GitHub.DistributedTask.Pipelines.ObjectTemplating
     {
         public const String Always = "always";
         public const String BooleanStepsContext = "boolean-steps-context";
-        public const String BooleanStrategyContext = "boolean-strategy-context";
         public const String CancelTimeoutMinutes = "cancel-timeout-minutes";
         public const String Cancelled = "cancelled";
         public const String Checkout = "checkout";

src/Sdk/DTPipelines/Pipelines/ObjectTemplating/YamlObjectReader.cs

@@ -1,4 +1,4 @@
-using System;
+using System;
 using System.Globalization;
 using System.IO;
 using System.Linq;
@@ -12,7 +12,7 @@ namespace GitHub.DistributedTask.Pipelines.ObjectTemplating
     /// <summary>
     /// Converts a YAML file into a TemplateToken
     /// </summary>
-    internal sealed class YamlObjectReader : IObjectReader
+    public sealed class YamlObjectReader : IObjectReader
     {
         internal YamlObjectReader(
             Int32? fileId,

src/Sdk/DTPipelines/workflow-v1.0.json

@@ -739,7 +739,7 @@
     "container-env": {
       "mapping": {
         "loose-key-type": "non-empty-string",
-        "loose-value-type": "string-runner-context"
+        "loose-value-type": "string"
       }
     },
 

src/Test/L0/Listener/Configuration/ConfigurationManagerL0.cs

@@ -37,7 +37,7 @@ namespace GitHub.Runner.Common.Tests.Listener.Configuration
 
         private Mock<IRSAKeyManager> _rsaKeyManager;
         private string _expectedToken = "expectedToken";
-        private string _expectedServerUrl = "https://codedev.ms";
+        private string _expectedServerUrl = "https://localhost";
         private string _expectedAgentName = "expectedAgentName";
         private string _expectedPoolName = "poolName";
         private string _expectedAuthType = "pat";

src/runnerversion

@@ -1 +1 @@
-2.169.0
+2.168.0