.

* Bump xunit from 2.4.1 to 2.7.1 in /src

Bumps [xunit](https://github.com/xunit/xunit) from 2.4.1 to 2.7.1.
- [Commits](https://github.com/xunit/xunit/compare/2.4.1...2.7.1)

---
updated-dependencies:
- dependency-name: xunit
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Appease xunit warnings after upgrading to v2.7.1

* Appease the whitespace linter

* Appease the whitespace linter

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Patrick Ellis <319655+pje@users.noreply.github.com>

images/Dockerfile

@@ -5,8 +5,8 @@ ARG TARGETOS
 ARG TARGETARCH
 ARG RUNNER_VERSION
 ARG RUNNER_CONTAINER_HOOKS_VERSION=0.6.0
-ARG DOCKER_VERSION=25.0.4
-ARG BUILDX_VERSION=0.13.1
+ARG DOCKER_VERSION=25.0.5
+ARG BUILDX_VERSION=0.13.2
 
 RUN apt update -y && apt install curl unzip -y
 
@@ -39,12 +39,15 @@ ENV RUNNER_MANUALLY_TRAP_SIG=1
 ENV ACTIONS_RUNNER_PRINT_LOG_TO_STDOUT=1
 ENV ImageOS=ubuntu22
 
-RUN apt-get update -y \
-    && apt-get install -y --no-install-recommends \
-    sudo \
-    lsb-release \
+# 'gpg-agent' and 'software-properties-common' are needed for the 'add-apt-repository' command that follows
+RUN apt update -y \
+    && apt install -y --no-install-recommends sudo lsb-release gpg-agent software-properties-common \
     && rm -rf /var/lib/apt/lists/*
 
+# Configure git-core/ppa based on guidance here:  https://git-scm.com/download/linux
+RUN add-apt-repository ppa:git-core/ppa \
+    && apt update -y
+
 RUN adduser --disabled-password --gecos "" --uid 1001 runner \
     && groupadd docker --gid 123 \
     && usermod -aG sudo runner \

releaseNote.md

@@ -1,28 +1,24 @@
 ## What's Changed
-* Load '_runnerSettings' in the early point of JobRunner.cs by @TingluoHuang in https://github.com/actions/runner/pull/3218
-* Add new SessionConflict return code by @eeSquared in https://github.com/actions/runner/pull/3215
-* backoff if we retried polling for more than 50 times in less than 30minutes by @aiqiaoy in https://github.com/actions/runner/pull/3232
-* Update dotnet sdk to latest version @6.0.421 by @github-actions in https://github.com/actions/runner/pull/3244
-* Cleanup enabled feature flags. by @TingluoHuang in https://github.com/actions/runner/pull/3246
-* Relax the condition to stop uploading to Results by @yacaovsnc in https://github.com/actions/runner/pull/3230
-* Cleanup enabled feature flags. by @TingluoHuang in https://github.com/actions/runner/pull/3248
-* Replace invalid file name chars in diag log name by @ericsciple in https://github.com/actions/runner/pull/3249
 
-## New Contributors
-* @eeSquared made their first contribution in https://github.com/actions/runner/pull/3215
-* @aiqiaoy made their first contribution in https://github.com/actions/runner/pull/3232
+- Do not give up when uploading steps metadata by @yacaovsnc in https://github.com/actions/runner/pull/3280
+- Upgrade node20 to 20.13.1 by @pje in https://github.com/actions/runner/pull/3284
+- Delete all the contentHash files by @pje in https://github.com/actions/runner/pull/3285
+- Make it easy to install `git` on an Action Runner Image by @jww3 in https://github.com/actions/runner/pull/3273
+- Install `gpg-agent` during actions/runner container image build by @jww3 in https://github.com/actions/runner/pull/3294
 
-**Full Changelog**: https://github.com/actions/runner/compare/v2.315.0...v2.316.0
+**Full Changelog**: https://github.com/actions/runner/compare/v2.316.1...v2.317.0
 
 _Note: Actions Runner follows a progressive release policy, so the latest release might not be available to your enterprise, organization, or repository yet.
 To confirm which version of the Actions Runner you should expect, please view the download instructions for your enterprise, organization, or repository.
 See https://docs.github.com/en/enterprise-cloud@latest/actions/hosting-your-own-runners/adding-self-hosted-runners_
 
 ## Windows x64
+
 We recommend configuring the runner in a root folder of the Windows drive (e.g. "C:\actions-runner"). This will help avoid issues related to service identity folder permissions and long file path restrictions on Windows.
 
 The following snipped needs to be run on `powershell`:
-``` powershell
+
+```powershell
 # Create a folder under the drive root
 mkdir \actions-runner ; cd \actions-runner
 # Download the latest runner package
@@ -33,12 +29,14 @@ Add-Type -AssemblyName System.IO.Compression.FileSystem ;
 ```
 
 ## [Pre-release] Windows arm64
+
 **Warning:** Windows arm64 runners are currently in preview status and use [unofficial versions of nodejs](https://unofficial-builds.nodejs.org/). They are not intended for production workflows.
 
 We recommend configuring the runner in a root folder of the Windows drive (e.g. "C:\actions-runner"). This will help avoid issues related to service identity folder permissions and long file path restrictions on Windows.
 
 The following snipped needs to be run on `powershell`:
-``` powershell
+
+```powershell
 # Create a folder under the drive root
 mkdir \actions-runner ; cd \actions-runner
 # Download the latest runner package
@@ -50,7 +48,7 @@ Add-Type -AssemblyName System.IO.Compression.FileSystem ;
 
 ## OSX x64
 
-``` bash
+```bash
 # Create a folder
 mkdir actions-runner && cd actions-runner
 # Download the latest runner package
@@ -61,7 +59,7 @@ tar xzf ./actions-runner-osx-x64-<RUNNER_VERSION>.tar.gz
 
 ## OSX arm64 (Apple silicon)
 
-``` bash
+```bash
 # Create a folder
 mkdir actions-runner && cd actions-runner
 # Download the latest runner package
@@ -72,7 +70,7 @@ tar xzf ./actions-runner-osx-arm64-<RUNNER_VERSION>.tar.gz
 
 ## Linux x64
 
-``` bash
+```bash
 # Create a folder
 mkdir actions-runner && cd actions-runner
 # Download the latest runner package
@@ -83,7 +81,7 @@ tar xzf ./actions-runner-linux-x64-<RUNNER_VERSION>.tar.gz
 
 ## Linux arm64
 
-``` bash
+```bash
 # Create a folder
 mkdir actions-runner && cd actions-runner
 # Download the latest runner package
@@ -94,7 +92,7 @@ tar xzf ./actions-runner-linux-arm64-<RUNNER_VERSION>.tar.gz
 
 ## Linux arm
 
-``` bash
+```bash
 # Create a folder
 mkdir actions-runner && cd actions-runner
 # Download the latest runner package
@@ -104,6 +102,7 @@ tar xzf ./actions-runner-linux-arm-<RUNNER_VERSION>.tar.gz
 ```
 
 ## Using your self hosted runner
+
 For additional details about configuring, running, or shutting down the runner please check out our [product docs.](https://help.github.com/en/actions/automating-your-workflow-with-github-actions/adding-self-hosted-runners)
 
 ## SHA-256 Checksums

src/Misc/contentHash/dotnetRuntime/linux-arm

@@ -1 +0,0 @@
-54d95a44d118dba852395991224a6b9c1abe916858c87138656f80c619e85331

src/Misc/contentHash/dotnetRuntime/linux-arm64

@@ -1 +0,0 @@
-68015af17f06a824fa478e62ae7393766ce627fd5599ab916432a14656a19a52

src/Misc/contentHash/dotnetRuntime/linux-x64

@@ -1 +0,0 @@
-a2628119ca419cb54e279103ffae7986cdbd0814d57c73ff0dc74c38be08b9ae

src/Misc/contentHash/dotnetRuntime/osx-arm64

@@ -1 +0,0 @@
-de71ca09ead807e1a2ce9df0a5b23eb7690cb71fff51169a77e4c3992be53dda

src/Misc/contentHash/dotnetRuntime/osx-x64

@@ -1 +0,0 @@
-d009e05e6b26d614d65be736a15d1bd151932121c16a9ff1b986deadecc982b9

src/Misc/contentHash/dotnetRuntime/win-arm64

@@ -1 +0,0 @@
-f730db39c2305800b4653795360ba9c10c68f384a46b85d808f1f9f0ed3c42e4

src/Misc/contentHash/dotnetRuntime/win-x64

@@ -1 +0,0 @@
-a35b5722375490e9473cdcccb5e18b41eba3dbf4344fe31abc9821e21f18ea5a

src/Misc/contentHash/externals/linux-arm

@@ -1 +0,0 @@
-4bf3e1af0d482af1b2eaf9f08250248a8c1aea8ec20a3c5be116d58cdd930009

src/Misc/contentHash/externals/linux-arm64

@@ -1 +0,0 @@
-ec1719a8cb4d8687328aa64f4aa7c4e3498a715d8939117874782e3e6e63a14b

src/Misc/contentHash/externals/linux-x64

@@ -1 +0,0 @@
-50538de29f173bb73f708c4ed2c8328a62b8795829b97b2a6cb57197e2305287

src/Misc/contentHash/externals/osx-arm64

@@ -1 +0,0 @@
-a0a96cbb7593643b69e669bf14d7b29b7f27800b3a00bb3305aebe041456c701

src/Misc/contentHash/externals/osx-x64

@@ -1 +0,0 @@
-6255b22692779467047ecebd60ad46984866d75cdfe10421d593a7b51d620b09

src/Misc/contentHash/externals/win-arm64

@@ -1 +0,0 @@
-6ff1abd055dc35bfbf06f75c2f08908f660346f66ad1d8f81c910068e9ba029d

src/Misc/contentHash/externals/win-x64

@@ -1 +0,0 @@
-433a6d748742d12abd20dc2a79b62ac3d9718ae47ef26f8e84dc8c180eea3659

src/Misc/externals.sh

@@ -5,10 +5,11 @@ PRECACHE=$2
 NODE_URL=https://nodejs.org/dist
 UNOFFICIAL_NODE_URL=https://unofficial-builds.nodejs.org/download/release
 NODE_ALPINE_URL=https://github.com/actions/alpine_nodejs/releases/download
+# When you update Node versions you must also create a new release of alpine_nodejs at that updated version.
+# Follow the instructions here: https://github.com/actions/alpine_nodejs?tab=readme-ov-file#getting-started
 NODE16_VERSION="16.20.2"
-NODE20_VERSION="20.8.1"
-# used only for win-arm64, remove node16 unofficial version when official version is available
-NODE16_UNOFFICIAL_VERSION="16.20.0"
+NODE20_VERSION="20.13.1"
+NODE16_UNOFFICIAL_VERSION="16.20.0" # used only for win-arm64, remove node16 unofficial version when official version is available
 
 get_abs_path() {
   # exploits the fact that pwd will print abs path when no args

src/Runner.Common/ActionsRunServer.cs

@@ -20,12 +20,12 @@ namespace GitHub.Runner.Common
     {
         private bool _hasConnection;
         private VssConnection _connection;
-        private TaskAgentHttpClient _taskAgentClient;
+        private ActionsRunServerHttpClient _actionsRunServerClient;
 
         public async Task ConnectAsync(Uri serverUrl, VssCredentials credentials)
         {
             _connection = await EstablishVssConnection(serverUrl, credentials, TimeSpan.FromSeconds(100));
-            _taskAgentClient = _connection.GetClient<TaskAgentHttpClient>();
+            _actionsRunServerClient = _connection.GetClient<ActionsRunServerHttpClient>();
             _hasConnection = true;
         }
 
@@ -42,7 +42,7 @@ namespace GitHub.Runner.Common
             CheckConnection();
             var jobMessage = RetryRequest<AgentJobRequestMessage>(async () =>
                                                     {
-                                                        return await _taskAgentClient.GetJobMessageAsync(id, cancellationToken);
+                                                        return await _actionsRunServerClient.GetJobMessageAsync(id, cancellationToken);
                                                     }, cancellationToken);
 
             return jobMessage;

src/Runner.Common/BrokerServer.cs

@@ -22,6 +22,8 @@ namespace GitHub.Runner.Common
 
         Task<TaskAgentMessage> GetRunnerMessageAsync(Guid? sessionId, TaskAgentStatus status, string version, string os, string architecture, bool disableUpdate, CancellationToken token);
 
+        Task DeleteRunnerMessageAsync(Guid sessionId, string jobMessageKey, CancellationToken cancellationToken);
+
         Task UpdateConnectionIfNeeded(Uri serverUri, VssCredentials credentials);
 
         Task ForceRefreshConnection(VssCredentials credentials);
@@ -75,6 +77,12 @@ namespace GitHub.Runner.Common
             await _brokerHttpClient.DeleteSessionAsync(cancellationToken);
         }
 
+        public async Task DeleteRunnerMessageAsync(Guid sessionId, string jobMessageKey, CancellationToken cancellationToken)
+        {
+            CheckConnection();
+            await _brokerHttpClient.DeleteRunnerMessageAsync(sessionId, jobMessageKey, cancellationToken);
+        }
+
         public Task UpdateConnectionIfNeeded(Uri serverUri, VssCredentials credentials)
         {
             if (_brokerUri != serverUri || !_hasConnection)

src/Runner.Common/JobServerQueue.cs

@@ -613,7 +613,7 @@ namespace GitHub.Runner.Common
 
         private void SendResultsTelemetry(Exception ex)
         {
-            var issue = new Issue() { Type = IssueType.Warning, Message = $"Caught exception with results. {ex.Message}" };
+            var issue = new Issue() { Type = IssueType.Warning, Message = $"Caught exception with results. {HostContext.SecretMasker.MaskSecrets(ex.Message)}" };
             issue.Data[Constants.Runner.InternalTelemetryIssueDataKey] = Constants.Runner.ResultsUploadFailure;
 
             var telemetryRecord = new TimelineRecord()
@@ -709,7 +709,9 @@ namespace GitHub.Runner.Common
                             {
                                 Trace.Info("Catch exception during update steps, skip update Results.");
                                 Trace.Error(e);
-                                if (!_resultsServiceOnly)
+                                _resultsServiceExceptionsCount++;
+                                // If we hit any exceptions uploading to Results, let's skip any additional uploads to Results unless Results is serving logs
+                                if (!_resultsServiceOnly && _resultsServiceExceptionsCount > 3)
                                 {
                                     _resultsClientInitiated = false;
                                     SendResultsTelemetry(e);

src/Runner.Common/RunServer.cs

@@ -62,7 +62,9 @@ namespace GitHub.Runner.Common
             CheckConnection();
             return RetryRequest<AgentJobRequestMessage>(
                 async () => await _runServiceHttpClient.GetJobMessageAsync(requestUri, id, VarUtil.OS, cancellationToken), cancellationToken,
-                shouldRetry: ex => ex is not TaskOrchestrationJobAlreadyAcquiredException);
+                shouldRetry: ex => ex is not TaskOrchestrationJobNotFoundException &&
+                    ex is not TaskOrchestrationJobAlreadyAcquiredException &&
+                    ex is not TaskOrchestrationJobUnprocessableException);
         }
 
         public Task CompleteJobAsync(

src/Runner.Listener/BrokerMessageListener.cs

@@ -314,7 +314,30 @@ namespace GitHub.Runner.Listener
 
         public async Task DeleteMessageAsync(TaskAgentMessage message)
         {
-            await Task.CompletedTask;
+            Trace.Entering();
+            ArgUtil.NotNull(_session, nameof(_session));
+
+            if (message == null || _session.SessionId == Guid.Empty)
+            {
+                return;
+            }
+
+            var jobMessageKey = "";
+            if (MessageUtil.IsRunServiceJob(message.MessageType))
+            {
+                var messageRef = StringUtil.ConvertFromJson<RunnerJobRequestRef>(message.Body);
+                jobMessageKey = messageRef.RunnerRequestId;
+            }
+            else
+            {
+                // Broker currently doesn't support delete for other message types
+                return;
+            }
+
+            using (var cs = new CancellationTokenSource(TimeSpan.FromSeconds(30)))
+            {
+                await _brokerServer.DeleteRunnerMessageAsync(_session.SessionId, jobMessageKey, cs.Token);
+            }
         }
 
         private bool IsGetNextMessageExceptionRetriable(Exception ex)

src/Runner.Listener/MessageListener.cs

@@ -398,7 +398,15 @@ namespace GitHub.Runner.Listener
             {
                 using (var cs = new CancellationTokenSource(TimeSpan.FromSeconds(30)))
                 {
-                    await _runnerServer.DeleteAgentMessageAsync(_settings.PoolId, message.MessageId, _session.SessionId, cs.Token);
+                    if (MessageUtil.IsRunServiceJob(message.MessageType))
+                    {
+                        var messageRef = StringUtil.ConvertFromJson<RunnerJobRequestRef>(message.Body);
+                        await _brokerServer.DeleteRunnerMessageAsync(_session.SessionId, messageRef.RunnerRequestId, cs.Token);
+                    }
+                    else
+                    {
+                        await _runnerServer.DeleteAgentMessageAsync(_settings.PoolId, message.MessageId, _session.SessionId, cs.Token);
+                    }
                 }
             }
         }

src/Runner.Listener/Runner.cs

@@ -544,6 +544,8 @@ namespace GitHub.Runner.Listener
                                 }
                                 else
                                 {
+                                    skipMessageDeletion = true;
+
                                     var messageRef = StringUtil.ConvertFromJson<RunnerJobRequestRef>(message.Body);
                                     Pipelines.AgentJobRequestMessage jobRequestMessage = null;
 
@@ -563,13 +565,15 @@ namespace GitHub.Runner.Listener
                                         await runServer.ConnectAsync(new Uri(messageRef.RunServiceUrl), creds);
                                         try
                                         {
-                                            jobRequestMessage =
-                                            await runServer.GetJobMessageAsync(messageRef.RunnerRequestId,
-                                            messageQueueLoopTokenSource.Token);
+                                            jobRequestMessage = await runServer.GetJobMessageAsync(messageRef.RunnerRequestId, messageQueueLoopTokenSource.Token);
                                         }
-                                        catch (TaskOrchestrationJobAlreadyAcquiredException)
+                                        catch (Exception ex) when (
+                                            ex is TaskOrchestrationJobNotFoundException ||
+                                            ex is TaskOrchestrationJobAlreadyAcquiredException ||
+                                            ex is TaskOrchestrationJobUnprocessableException)
                                         {
-                                            Trace.Info("Job is already acquired, skip this message.");
+                                            Trace.Error($"Skipping job: {ex.Message}");
+                                            skipMessageDeletion = false;
                                             continue;
                                         }
                                         catch (Exception ex)

src/Sdk/DTWebApi/WebApi/ActionsRunServerHttpClient.cs

@@ -0,0 +1,95 @@
+using System;
+using System.Collections.Generic;
+using System.Globalization;
+using System.IO;
+using System.IO.Compression;
+using System.Linq;
+using System.Net;
+using System.Net.Http;
+using System.Net.Http.Headers;
+using System.Threading;
+using System.Threading.Tasks;
+using GitHub.Services.Common;
+using GitHub.Services.Common.Diagnostics;
+using GitHub.Services.WebApi;
+using Newtonsoft.Json;
+
+namespace GitHub.DistributedTask.WebApi
+{
+    [ResourceArea(TaskResourceIds.AreaId)]
+    public class ActionsRunServerHttpClient : TaskAgentHttpClient
+    {
+        private static readonly JsonSerializerSettings s_serializerSettings;
+
+        static ActionsRunServerHttpClient()
+        {
+            s_serializerSettings = new VssJsonMediaTypeFormatter().SerializerSettings;
+            s_serializerSettings.DateParseHandling = DateParseHandling.None;
+            s_serializerSettings.FloatParseHandling = FloatParseHandling.Double;
+        }
+
+        public ActionsRunServerHttpClient(
+            Uri baseUrl,
+            VssCredentials credentials)
+            : base(baseUrl, credentials)
+        {
+        }
+
+        public ActionsRunServerHttpClient(
+            Uri baseUrl,
+            VssCredentials credentials,
+            VssHttpRequestSettings settings)
+            : base(baseUrl, credentials, settings)
+        {
+        }
+
+        public ActionsRunServerHttpClient(
+            Uri baseUrl,
+            VssCredentials credentials,
+            params DelegatingHandler[] handlers)
+            : base(baseUrl, credentials, handlers)
+        {
+        }
+
+        public ActionsRunServerHttpClient(
+            Uri baseUrl,
+            VssCredentials credentials,
+            VssHttpRequestSettings settings,
+            params DelegatingHandler[] handlers)
+            : base(baseUrl, credentials, settings, handlers)
+        {
+        }
+
+        public ActionsRunServerHttpClient(
+            Uri baseUrl,
+            HttpMessageHandler pipeline,
+            Boolean disposeHandler)
+            : base(baseUrl, pipeline, disposeHandler)
+        {
+        }
+
+        public Task<Pipelines.AgentJobRequestMessage> GetJobMessageAsync(
+            string messageId,
+            object userState = null,
+            CancellationToken cancellationToken = default)
+        {
+            HttpMethod httpMethod = new HttpMethod("GET");
+            Guid locationId = new Guid("25adab70-1379-4186-be8e-b643061ebe3a");
+            object routeValues = new { messageId = messageId };
+
+            return SendAsync<Pipelines.AgentJobRequestMessage>(
+                httpMethod,
+                locationId,
+                routeValues: routeValues,
+                version: new ApiResourceVersion(6.0, 1),
+                userState: userState,
+                cancellationToken: cancellationToken);
+        }
+
+        protected override async Task<T> ReadJsonContentAsync<T>(HttpResponseMessage response, CancellationToken cancellationToken = default(CancellationToken))
+        {
+            var json = await response.Content.ReadAsStringAsync(cancellationToken).ConfigureAwait(false);
+            return JsonConvert.DeserializeObject<T>(json, s_serializerSettings);
+        }
+    }
+}

src/Sdk/DTWebApi/WebApi/Exceptions.cs

@@ -1539,6 +1539,26 @@ namespace GitHub.DistributedTask.WebApi
         }
     }
 
+    [Serializable]
+    [ExceptionMapping("0.0", "3.0", "TaskOrchestrationJobUnprocessableException", "GitHub.DistributedTask.WebApi.TaskOrchestrationJobUnprocessableException, GitHub.DistributedTask.WebApi, Version=14.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a")]
+    public sealed class TaskOrchestrationJobUnprocessableException : DistributedTaskException
+    {
+        public TaskOrchestrationJobUnprocessableException(String message)
+            : base(message)
+        {
+        }
+
+        public TaskOrchestrationJobUnprocessableException(String message, Exception innerException)
+            : base(message, innerException)
+        {
+        }
+
+        private TaskOrchestrationJobUnprocessableException(SerializationInfo info, StreamingContext context)
+            : base(info, context)
+        {
+        }
+    }
+
     [Serializable]
     [ExceptionMapping("0.0", "3.0", "TaskOrchestrationPlanSecurityException", "GitHub.DistributedTask.WebApi.TaskOrchestrationPlanSecurityException, GitHub.DistributedTask.WebApi, Version=14.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a")]
     public sealed class TaskOrchestrationPlanSecurityException : DistributedTaskException

src/Sdk/DTWebApi/WebApi/TaskAgentHttpClient.cs

@@ -141,24 +141,6 @@ namespace GitHub.DistributedTask.WebApi
             return ReplaceAgentAsync(poolId, agent.Id, agent, userState, cancellationToken);
         }
 
-        public Task<Pipelines.AgentJobRequestMessage> GetJobMessageAsync(
-            string messageId,
-            object userState = null,
-            CancellationToken cancellationToken = default)
-        {
-            HttpMethod httpMethod = new HttpMethod("GET");
-            Guid locationId = new Guid("25adab70-1379-4186-be8e-b643061ebe3a");
-            object routeValues = new { messageId = messageId };
-
-            return SendAsync<Pipelines.AgentJobRequestMessage>(
-                httpMethod,
-                locationId,
-                routeValues: routeValues,
-                version: new ApiResourceVersion(6.0, 1),
-                userState: userState,
-                cancellationToken: cancellationToken);
-        }
-
         protected Task<T> SendAsync<T>(
             HttpMethod method,
             Guid locationId,

src/Sdk/RSWebApi/Contracts/RunServiceError.cs

@@ -0,0 +1,17 @@
+using System.Runtime.Serialization;
+
+namespace GitHub.Actions.RunService.WebApi
+{
+    [DataContract]
+    public class RunServiceError
+    {
+        [DataMember(Name = "source", EmitDefaultValue = false)]
+        public string Source { get; set; }
+
+        [DataMember(Name = "statusCode", EmitDefaultValue = false)]
+        public int StatusCode { get; set; }
+
+        [DataMember(Name = "errorMessage", EmitDefaultValue = false)]
+        public string ErrorMessage { get; set; }
+    }
+}

src/Sdk/RSWebApi/RunServiceHttpClient.cs

@@ -9,6 +9,7 @@ using GitHub.DistributedTask.WebApi;
 using GitHub.Services.Common;
 using GitHub.Services.OAuth;
 using GitHub.Services.WebApi;
+using Newtonsoft.Json;
 using Sdk.RSWebApi.Contracts;
 using Sdk.WebApi.WebApi;
 
@@ -16,6 +17,15 @@ namespace GitHub.Actions.RunService.WebApi
 {
     public class RunServiceHttpClient : RawHttpClientBase
     {
+        private static readonly JsonSerializerSettings s_serializerSettings;
+
+        static RunServiceHttpClient()
+        {
+            s_serializerSettings = new VssJsonMediaTypeFormatter().SerializerSettings;
+            s_serializerSettings.DateParseHandling = DateParseHandling.None;
+            s_serializerSettings.FloatParseHandling = FloatParseHandling.Double;
+        }
+
         public RunServiceHttpClient(
             Uri baseUrl,
             VssOAuthCredential credentials)
@@ -76,6 +86,7 @@ namespace GitHub.Actions.RunService.WebApi
                 httpMethod,
                 requestUri: requestUri,
                 content: requestContent,
+                readErrorContent: true,
                 cancellationToken: cancellationToken);
 
             if (result.IsSuccess)
@@ -83,14 +94,35 @@ namespace GitHub.Actions.RunService.WebApi
                 return result.Value;
             }
 
+            if (TryParseErrorContent(result.ErrorContent, out RunServiceError error))
+            {
+                switch ((HttpStatusCode)error.StatusCode)
+                {
+                    case HttpStatusCode.NotFound:
+                        throw new TaskOrchestrationJobNotFoundException($"Job message not found '{messageId}'. {error.ErrorMessage}");
+                    case HttpStatusCode.Conflict:
+                        throw new TaskOrchestrationJobAlreadyAcquiredException($"Job message already acquired '{messageId}'. {error.ErrorMessage}");
+                    case HttpStatusCode.UnprocessableEntity:
+                        throw new TaskOrchestrationJobUnprocessableException($"Unprocessable job '{messageId}'. {error.ErrorMessage}");
+                }
+            }
+
+            // Temporary back compat
             switch (result.StatusCode)
             {
                 case HttpStatusCode.NotFound:
                     throw new TaskOrchestrationJobNotFoundException($"Job message not found: {messageId}");
                 case HttpStatusCode.Conflict:
                     throw new TaskOrchestrationJobAlreadyAcquiredException($"Job message already acquired: {messageId}");
-                default:
-                    throw new Exception($"Failed to get job message: {result.Error}");
+            }
+
+            if (!string.IsNullOrEmpty(result.ErrorContent))
+            {
+                throw new Exception($"Failed to get job message: {result.Error}. {result.ErrorContent}");
+            }
+            else
+            {
+                throw new Exception($"Failed to get job message: {result.Error}");
             }
         }
 
@@ -174,5 +206,32 @@ namespace GitHub.Actions.RunService.WebApi
                     throw new Exception($"Failed to renew job: {result.Error}");
             }
         }
+
+        protected override async Task<T> ReadJsonContentAsync<T>(HttpResponseMessage response, CancellationToken cancellationToken = default(CancellationToken))
+        {
+            var json = await response.Content.ReadAsStringAsync(cancellationToken).ConfigureAwait(false);
+            return JsonConvert.DeserializeObject<T>(json, s_serializerSettings);
+        }
+
+        private static bool TryParseErrorContent(string errorContent, out RunServiceError error)
+        {
+            if (!string.IsNullOrEmpty(errorContent))
+            {
+                try
+                {
+                    error = JsonUtility.FromString<RunServiceError>(errorContent);
+                    if (error?.Source == "actions-run-service")
+                    {
+                        return true;
+                    }
+                }
+                catch (Exception)
+                {
+                }
+            }
+
+            error = null;
+            return false;
+        }
     }
 }

src/Sdk/WebApi/WebApi/BrokerHttpClient.cs

@@ -63,8 +63,7 @@ namespace GitHub.Actions.RunService.WebApi
             string os = null,
             string architecture = null,
             bool? disableUpdate = null,
-            CancellationToken cancellationToken = default
-        )
+            CancellationToken cancellationToken = default)
         {
             var requestUri = new Uri(Client.BaseAddress, "message");
 
@@ -123,8 +122,42 @@ namespace GitHub.Actions.RunService.WebApi
             throw new Exception($"Failed to get job message: {result.Error}");
         }
 
-        public async Task<TaskAgentSession> CreateSessionAsync(
+        public async Task DeleteRunnerMessageAsync(
+            Guid? sessionId,
+            string jobMessageKey,
+            CancellationToken cancellationToken = default)
+        {
+            var requestUri = new Uri(Client.BaseAddress, "message");
 
+            List<KeyValuePair<string, string>> queryParams = new List<KeyValuePair<string, string>>();
+
+            if (sessionId != null)
+            {
+                queryParams.Add("sessionId", sessionId.Value.ToString());
+            }
+
+            if (!string.IsNullOrEmpty(jobMessageKey))
+            {
+                queryParams.Add("jobMessageKey", jobMessageKey);
+            }
+
+            queryParams.Add("status", TaskAgentStatus.Online.ToString());
+
+            var result = await SendAsync<object>(
+                new HttpMethod("DELETE"),
+                requestUri: requestUri,
+                queryParameters: queryParams,
+                cancellationToken: cancellationToken);
+
+            if (result.IsSuccess)
+            {
+                return;
+            }
+
+            throw new Exception($"Failed to get job message: StatusCode={result.StatusCode} Error={result.Error}");
+        }
+
+        public async Task<TaskAgentSession> CreateSessionAsync(
            TaskAgentSession session,
            CancellationToken cancellationToken = default)
         {

src/Sdk/WebApi/WebApi/RawHttpClientBase.cs

@@ -106,10 +106,11 @@ namespace Sdk.WebApi.WebApi
             Uri requestUri,
             HttpContent content = null,
             IEnumerable<KeyValuePair<String, String>> queryParameters = null,
+            Boolean readErrorContent = false,
             Object userState = null,
             CancellationToken cancellationToken = default(CancellationToken))
         {
-            return SendAsync<T>(method, null, requestUri, content, queryParameters, userState, cancellationToken);
+            return SendAsync<T>(method, null, requestUri, content, queryParameters, readErrorContent, userState, cancellationToken);
         }
 
         protected async Task<RawHttpClientResult<T>> SendAsync<T>(
@@ -118,18 +119,20 @@ namespace Sdk.WebApi.WebApi
             Uri requestUri,
             HttpContent content = null,
             IEnumerable<KeyValuePair<String, String>> queryParameters = null,
+            Boolean readErrorContent = false,
             Object userState = null,
             CancellationToken cancellationToken = default(CancellationToken))
         {
             using (VssTraceActivity.GetOrCreate().EnterCorrelationScope())
             using (HttpRequestMessage requestMessage = CreateRequestMessage(method, additionalHeaders, requestUri, content, queryParameters))
             {
-                return await SendAsync<T>(requestMessage, userState, cancellationToken).ConfigureAwait(false);
+                return await SendAsync<T>(requestMessage, readErrorContent, userState, cancellationToken).ConfigureAwait(false);
             }
         }
 
         protected async Task<RawHttpClientResult<T>> SendAsync<T>(
             HttpRequestMessage message,
+            Boolean readErrorContent = false,
             Object userState = null,
             CancellationToken cancellationToken = default(CancellationToken))
         {
@@ -145,8 +148,14 @@ namespace Sdk.WebApi.WebApi
                 }
                 else
                 {
+                    var errorContent = default(string);
+                    if (readErrorContent)
+                    {
+                        errorContent = await response.Content.ReadAsStringAsync(cancellationToken).ConfigureAwait(false);
+                    }
+
                     string errorMessage = $"Error: {response.ReasonPhrase}";
-                    return RawHttpClientResult<T>.Fail(errorMessage, response.StatusCode);
+                    return RawHttpClientResult<T>.Fail(errorMessage, response.StatusCode, errorContent);
                 }
             }
         }

src/Sdk/WebApi/WebApi/RawHttpClientResult.cs

@@ -5,15 +5,27 @@ namespace Sdk.WebApi.WebApi
     public class RawHttpClientResult
     {
         public bool IsSuccess { get; protected set; }
+
+        /// <summary>
+        /// A description of the HTTP status code, like "Error: Unprocessable Entity"
+        /// </summary>
         public string Error { get; protected set; }
+
+        /// <summary>
+        /// The raw of the HTTP response, for unsuccessful HTTP status codes
+        /// </summary>
+        public string ErrorContent { get; protected set; }
+
         public HttpStatusCode StatusCode { get; protected set; }
+
         public bool IsFailure => !IsSuccess;
 
-        protected RawHttpClientResult(bool isSuccess, string error, HttpStatusCode statusCode)
+        protected RawHttpClientResult(bool isSuccess, string error, HttpStatusCode statusCode, string errorContent = null)
         {
             IsSuccess = isSuccess;
             Error = error;
             StatusCode = statusCode;
+            ErrorContent = errorContent;
         }
     }
 
@@ -21,13 +33,13 @@ namespace Sdk.WebApi.WebApi
     {
         public T Value { get; private set; }
 
-        protected internal RawHttpClientResult(T value, bool isSuccess, string error, HttpStatusCode statusCode)
-            : base(isSuccess, error, statusCode)
+        protected internal RawHttpClientResult(T value, bool isSuccess, string error, HttpStatusCode statusCode, string errorContent)
+            : base(isSuccess, error, statusCode, errorContent)
         {
             Value = value;
         }
 
-        public static RawHttpClientResult<T> Fail(string message, HttpStatusCode statusCode) => new RawHttpClientResult<T>(default(T), false, message, statusCode);
-        public static RawHttpClientResult<T> Ok(T value) => new RawHttpClientResult<T>(value, true, string.Empty, HttpStatusCode.OK);
+        public static RawHttpClientResult<T> Fail(string message, HttpStatusCode statusCode, string errorContent) => new RawHttpClientResult<T>(default(T), false, message, statusCode, errorContent);
+        public static RawHttpClientResult<T> Ok(T value) => new RawHttpClientResult<T>(value, true, string.Empty, HttpStatusCode.OK, null);
     }
 }

src/Test/L0/CommandLineParserL0.cs

@@ -68,7 +68,7 @@ namespace GitHub.Runner.Common.Tests
                 trace.Info("Parsed");
 
                 trace.Info("Commands: {0}", clp.Commands.Count);
-                Assert.True(clp.Commands.Count == 2);
+                Assert.Equal(2, clp.Commands.Count);
             }
         }
 
@@ -88,7 +88,7 @@ namespace GitHub.Runner.Common.Tests
                 trace.Info("Parsed");
 
                 trace.Info("Args: {0}", clp.Args.Count);
-                Assert.True(clp.Args.Count == 2);
+                Assert.Equal(2, clp.Args.Count);
                 Assert.True(clp.Args.ContainsKey("arg1"));
                 Assert.Equal("arg1val", clp.Args["arg1"]);
                 Assert.True(clp.Args.ContainsKey("arg2"));
@@ -112,7 +112,7 @@ namespace GitHub.Runner.Common.Tests
                 trace.Info("Parsed");
 
                 trace.Info("Args: {0}", clp.Flags.Count);
-                Assert.True(clp.Flags.Count == 2);
+                Assert.Equal(2, clp.Flags.Count);
                 Assert.Contains("flag1", clp.Flags);
                 Assert.Contains("flag2", clp.Flags);
             }

src/Test/L0/ConstantGenerationL0.cs

@@ -24,7 +24,7 @@ namespace GitHub.Runner.Common.Tests
                 "osx-arm64"
             };
 
-            Assert.True(BuildConstants.Source.CommitHash.Length == 40, $"CommitHash should be SHA-1 hash {BuildConstants.Source.CommitHash}");
+            Assert.Equal(40, BuildConstants.Source.CommitHash.Length);
             Assert.True(validPackageNames.Contains(BuildConstants.RunnerPackage.PackageName), $"PackageName should be one of the following '{string.Join(", ", validPackageNames)}', current PackageName is '{BuildConstants.RunnerPackage.PackageName}'");
         }
     }

src/Test/L0/Listener/CommandSettingsL0.cs

@@ -806,7 +806,7 @@ namespace GitHub.Runner.Common.Tests
                         "test runner" });
 
                 // Assert.
-                Assert.True(command.Validate().Count == 0);
+                Assert.Equal(0, command.Validate().Count);
             }
         }
 
@@ -844,7 +844,7 @@ namespace GitHub.Runner.Common.Tests
                 var command = new CommandSettings(hc, args: new string[] { validCommand, $"--{flag}" });
 
                 // Assert.
-                Assert.True(command.Validate().Count == 0);
+                Assert.Equal(0, command.Validate().Count);
             }
         }
 
@@ -874,7 +874,7 @@ namespace GitHub.Runner.Common.Tests
                 var command = new CommandSettings(hc, args: new string[] { validCommand, $"--{arg}", argValue });
 
                 // Assert.
-                Assert.True(command.Validate().Count == 0);
+                Assert.Equal(0, command.Validate().Count);
             }
         }
 

src/Test/L0/Listener/Configuration/ConfigurationManagerL0.cs

@@ -190,11 +190,11 @@ namespace GitHub.Runner.Common.Tests.Listener.Configuration
                 trace.Info("Configured, verifying all the parameter value");
                 var s = configManager.LoadSettings();
                 Assert.NotNull(s);
-                Assert.True(s.ServerUrl.Equals(_expectedServerUrl));
-                Assert.True(s.AgentName.Equals(_expectedAgentName));
-                Assert.True(s.PoolId.Equals(_secondRunnerGroupId));
-                Assert.True(s.WorkFolder.Equals(_expectedWorkFolder));
-                Assert.True(s.Ephemeral.Equals(true));
+                Assert.Equal(_expectedServerUrl, s.ServerUrl);
+                Assert.Equal(_expectedAgentName, s.AgentName);
+                Assert.Equal(_secondRunnerGroupId, s.PoolId);
+                Assert.Equal(_expectedWorkFolder, s.WorkFolder);
+                Assert.True(s.Ephemeral);
 
                 // validate GetAgentPoolsAsync gets called twice with automation pool type
                 _runnerServer.Verify(x => x.GetAgentPoolsAsync(It.IsAny<string>(), It.Is<TaskAgentPoolType>(p => p == TaskAgentPoolType.Automation)), Times.Exactly(2));
@@ -292,11 +292,11 @@ namespace GitHub.Runner.Common.Tests.Listener.Configuration
                 trace.Info("Configured, verifying all the parameter value");
                 var s = configManager.LoadSettings();
                 Assert.NotNull(s);
-                Assert.True(s.ServerUrl.Equals(_expectedServerUrl));
-                Assert.True(s.AgentName.Equals(_expectedAgentName));
-                Assert.True(s.PoolId.Equals(_secondRunnerGroupId));
-                Assert.True(s.WorkFolder.Equals(_expectedWorkFolder));
-                Assert.True(s.Ephemeral.Equals(true));
+                Assert.Equal(_expectedServerUrl, s.ServerUrl);
+                Assert.Equal(_expectedAgentName, s.AgentName);
+                Assert.Equal(_secondRunnerGroupId, s.PoolId);
+                Assert.Equal(_expectedWorkFolder, s.WorkFolder);
+                Assert.True(s.Ephemeral);
 
                 // validate GetAgentPoolsAsync gets called twice with automation pool type
                 _runnerServer.Verify(x => x.GetAgentPoolsAsync(It.IsAny<string>(), It.Is<TaskAgentPoolType>(p => p == TaskAgentPoolType.Automation)), Times.Exactly(2));

src/Test/L0/Listener/JobDispatcherL0.cs

@@ -734,7 +734,10 @@ namespace GitHub.Runner.Common.Tests.Listener
                 await jobDispatcher.WaitAsync(CancellationToken.None);
 
                 Assert.True(jobDispatcher.RunOnceJobCompleted.Task.IsCompleted, "JobDispatcher should set task complete token for one time agent.");
-                Assert.True(jobDispatcher.RunOnceJobCompleted.Task.Result, "JobDispatcher should set task complete token to 'TRUE' for one time agent.");
+                if (jobDispatcher.RunOnceJobCompleted.Task.IsCompleted)
+                {
+                    Assert.True(await jobDispatcher.RunOnceJobCompleted.Task, "JobDispatcher should set task complete token to 'TRUE' for one time agent.");
+                }
             }
         }
 

src/Test/L0/Listener/RunnerL0.cs

@@ -126,7 +126,7 @@ namespace GitHub.Runner.Common.Tests.Listener
                 //wait for the runner to run one job
                 if (!await signalWorkerComplete.WaitAsync(2000))
                 {
-                    Assert.True(false, $"{nameof(_messageListener.Object.GetNextMessageAsync)} was not invoked.");
+                    Assert.Fail($"{nameof(_messageListener.Object.GetNextMessageAsync)} was not invoked.");
                 }
                 else
                 {
@@ -305,8 +305,11 @@ namespace GitHub.Runner.Common.Tests.Listener
                 await Task.WhenAny(runnerTask, Task.Delay(30000));
 
                 Assert.True(runnerTask.IsCompleted, $"{nameof(runner.ExecuteCommand)} timed out.");
-                Assert.True(!runnerTask.IsFaulted, runnerTask.Exception?.ToString());
-                Assert.True(runnerTask.Result == Constants.Runner.ReturnCode.Success);
+                Assert.False(runnerTask.IsFaulted, runnerTask.Exception?.ToString());
+                if (runnerTask.IsCompleted)
+                {
+                    Assert.Equal(Constants.Runner.ReturnCode.Success, await runnerTask);
+                }
 
                 _jobDispatcher.Verify(x => x.Run(It.IsAny<Pipelines.AgentJobRequestMessage>(), true), Times.Once(),
                      $"{nameof(_jobDispatcher.Object.Run)} was not invoked.");
@@ -406,7 +409,10 @@ namespace GitHub.Runner.Common.Tests.Listener
 
                 Assert.True(runnerTask.IsCompleted, $"{nameof(runner.ExecuteCommand)} timed out.");
                 Assert.True(!runnerTask.IsFaulted, runnerTask.Exception?.ToString());
-                Assert.True(runnerTask.Result == Constants.Runner.ReturnCode.Success);
+                if (runnerTask.IsCompleted)
+                {
+                    Assert.Equal(Constants.Runner.ReturnCode.Success, await runnerTask);
+                }
 
                 _jobDispatcher.Verify(x => x.Run(It.IsAny<Pipelines.AgentJobRequestMessage>(), true), Times.Once(),
                      $"{nameof(_jobDispatcher.Object.Run)} was not invoked.");
@@ -492,7 +498,10 @@ namespace GitHub.Runner.Common.Tests.Listener
 
                 Assert.True(runnerTask.IsCompleted, $"{nameof(runner.ExecuteCommand)} timed out.");
                 Assert.True(!runnerTask.IsFaulted, runnerTask.Exception?.ToString());
-                Assert.True(runnerTask.Result == Constants.Runner.ReturnCode.RunOnceRunnerUpdating);
+                if (runnerTask.IsCompleted)
+                {
+                    Assert.Equal(Constants.Runner.ReturnCode.RunOnceRunnerUpdating, await runnerTask);
+                }
 
                 _updater.Verify(x => x.SelfUpdate(It.IsAny<AgentRefreshMessage>(), It.IsAny<IJobDispatcher>(), false, It.IsAny<CancellationToken>()), Times.Once);
                 _jobDispatcher.Verify(x => x.Run(It.IsAny<Pipelines.AgentJobRequestMessage>(), true), Times.Never());

src/Test/L0/ProcessExtensionL0.cs

@@ -58,7 +58,7 @@ namespace GitHub.Runner.Common.Tests
                         trace.Error(ex);
                     }
 
-                    Assert.True(false, "Fail to retrive process environment variable.");
+                    Assert.Fail("Failed to retrieve process environment variable.");
                 }
                 finally
                 {

src/Test/L0/RunnerWebProxyL0.cs

@@ -65,7 +65,14 @@ namespace GitHub.Runner.Common.Tests
                 }
             }
 
-            Assert.True(badCode.Count == 0, $"The following code is using Raw HttpClientHandler() which will not follow the proxy setting agent have. Please use HostContext.CreateHttpClientHandler() instead.\n {string.Join("\n", badCode)}");
+            if (badCode.Count > 0)
+            {
+                Assert.Fail($"The following code is using Raw HttpClientHandler() which will not follow the proxy setting agent have. Please use HostContext.CreateHttpClientHandler() instead.\n {string.Join("\n", badCode)}");
+            }
+            else
+            {
+                Assert.True(true);
+            }
         }
 
         [Fact]
@@ -112,7 +119,14 @@ namespace GitHub.Runner.Common.Tests
                 }
             }
 
-            Assert.True(badCode.Count == 0, $"The following code is using Raw HttpClient() which will not follow the proxy setting agent have. Please use New HttpClient(HostContext.CreateHttpClientHandler()) instead.\n {string.Join("\n", badCode)}");
+            if (badCode.Count > 0)
+            {
+                Assert.Fail($"The following code is using Raw HttpClient() which will not follow the proxy setting agent have. Please use New HttpClient(HostContext.CreateHttpClientHandler()) instead.\n {string.Join("\n", badCode)}");
+            }
+            else
+            {
+                Assert.True(true);
+            }
         }
 
         [Fact]

src/Test/L0/Worker/ActionManagerL0.cs

@@ -460,7 +460,7 @@ runs:
                 //Act
                 var steps = (await _actionManager.PrepareActionsAsync(_ec.Object, actions)).ContainerSetupSteps;
 
-                Assert.True(steps.Count == 0);
+                Assert.Equal(0, steps.Count);
             }
             finally
             {
@@ -915,7 +915,7 @@ runs:
                 var steps = (await _actionManager.PrepareActionsAsync(_ec.Object, actions)).ContainerSetupSteps;
 
                 // node.js based action doesn't need any extra steps to build/pull containers.
-                Assert.True(steps.Count == 0);
+                Assert.Equal(0, steps.Count);
             }
             finally
             {
@@ -1051,7 +1051,7 @@ runs:
                 var steps = (await _actionManager.PrepareActionsAsync(_ec.Object, actions)).ContainerSetupSteps;
 
                 // node.js based action doesn't need any extra steps to build/pull containers.
-                Assert.True(steps.Count == 0);
+                Assert.Equal(0, steps.Count);
                 var watermarkFile = Path.Combine(_hc.GetDirectory(WellKnownDirectory.Actions), "TingluoHuang/runner_L0", "CompositeBasic.completed");
                 Assert.True(File.Exists(watermarkFile));
                 // Comes from the composite action
@@ -1245,7 +1245,7 @@ runs:
                 // Assert.
                 Assert.NotNull(definition);
                 Assert.NotNull(definition.Data);
-                Assert.True(definition.Data.Execution.ExecutionType == ActionExecutionType.Script);
+                Assert.Equal(ActionExecutionType.Script, definition.Data.Execution.ExecutionType);
             }
             finally
             {

src/Test/Test.csproj

@@ -16,7 +16,7 @@
 
     <ItemGroup>
         <PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.2.0" />
-        <PackageReference Include="xunit" Version="2.4.1" />
+        <PackageReference Include="xunit" Version="2.7.1" />
         <PackageReference Include="xunit.runner.visualstudio" Version="2.4.1" />
         <PackageReference Include="System.Buffers" Version="4.5.1" />
         <PackageReference Include="System.Reflection.TypeExtensions" Version="4.4.0" />

src/runnerversion

@@ -1 +1 @@
-2.316.0
+2.317.0