* Added ability to run Dockerfile.SUFFIX ContainerAction
* Extracted IsDockerFile method
* reformatted, moved from index to Last()
* extracted IsDockerfile to DockerUtil with L0
* added check for IsDockerfile to account for docker://
* updated test to clearly show path/dockerfile:tag
* fail if Data.Image is not Dockerfile or docker://[image]
* Setup noops for JobPrepare and JobCleanup hooks
* Add container jobstarted and jobcomplete hooks
* Run 'index.js' instead of specific command hooks
* Call jobprepare with command arg
* Use right command name (hardcoded)
Co-authored-by: Nikola Jokic <nikola-jokic@users.noreply.github.com>
* Invoke hooks with arguments
* Add PrepareJob hook to work with jobcontainers
Co-authored-by: Nikola Jokic <nikola-jokic@users.noreply.github.com>
* Rename methods
* Use new hookcontainer to run prep and clean hooks
* Get path from ENV
* Use enums
* Use IOUtils.cs
* Move container files to folder
* Move namespaces
* Store "state" between hooks
* Remove stdin stream in containerstephosts
* Update Constants.cs
* Throw if stdin fails
* Cleanup obvious nullrefs and unused vars
* Cleanup containerhook directory
* Call step exec hook
* Fix windows build
* Remove hook from hookContainer
* Rename file
* More renamings
* Add TODOs
* Fix env name
* Fix missing imports
* Fix imports
* Run script step on jobcontainer
* Enable feature if env is set
* Update ContainerHookManager.cs
* Update ContainerHookManager.cs
* Hooks allowed to work even when context isn't returned
* Custom hooks enabled flag and additional null checks
* New line at the end of the FeatureFlagManager.cs
* Code refactoring
* Supported just in time container building or pulling
* Try mock-build for osx
* Build all platforms
* Run mock on self-hosted
* Remove GITHUB prefix
* Use ContainerHooksPath instead of CustomHooksPath
* Null checks simplified
* Code refactoring
* Changing condition for image builing/pulling
* Code refactoring
* TODO comment removed
Co-authored-by: Ferenc Hammerl <31069338+fhammerl@users.noreply.github.com>
* Call container step if FF is on
* Rename run script function
* Use JToken instead of dynamic
* Add TODO
* Small refactoring + renames + TODOs
* Throw on DetermineNodeRuntimeVersion
* Fix formatting
* Add run-container-step
* Supported nodeJS in Alpine containers
* Renamed Alpine to IsAlpine in HookResponse
* Method for checking platform for alpine container
* Added container hooks feature flag check
* Update IsHookFeatureEnabled with new params
* Rename featureflag method
* Finish rename
* Set collection null values to empty arrays when JSON serialising them
* Disable FF until we merge
* Update src/Runner.Worker/Container/ContainerHooks/HookContainer.cs
* Fix method name
* Change hookargs to superclass from interface
* Using only Path.Combine in GenerateResponsePath
* fix merge error
* EntryPointArgs changed to list of args instead of one args string
* Changed List to IEnumerable for EntryPointArgs and MountVolumes
* Get ContainerRuntimePath for JobContainers from hooks
* Read ContainerEnv from response file
* Port mappings saved after creating services
* Support case when responseFile doesn't exist
* Check if response file exists
* Logging in ExecuteHookScript
* Save hook state after all 4 hooks
* Code refactoring
* Remove TODO
Co-authored-by: Ferenc Hammerl <31069338+fhammerl@users.noreply.github.com>
* Remove second TODO
Co-authored-by: Ferenc Hammerl <31069338+fhammerl@users.noreply.github.com>
* Removing container env changes
* Removing containerEnv and dockerManager
* Delete mock-build.yml
* Update IOUtil.cs
* Add comment about containerhooks
* Fix merge mistake
* Remove solved todo
* Determine which shell to use for hooks scenario
* Overload for method ExecuteHookScript with prependPath as arg
* Adding HostContext to the GetDefaultShellForScript call
* prependPath as a mandatory parameter
* Improve logging for hooks
* Small changes in logging
* Allow null for ContainerEntryPointArgs
* Changed log messages
* Skip setting EntryPoint and EntryPointArgs if hooks are enabled
* Throw if IsAlpine is null in PrepareJob
* Code refactoring - added GetAndValidateResponse method
* Code refactoring
* Changes in exception message
* Only save hookState if returned
* Use FF from server
* Empty line
* Code refactoring
Co-authored-by: Ferenc Hammerl <31069338+fhammerl@users.noreply.github.com>
* Send null instead of string empty
* Remove TODO
* Code refactoring and some small changes
* Allow Globals to be null to pass L0
* Fix setup in StepHostL0
* Throw exception earlier if response file doesn't exist and prepare_job hook is running
* Refactoring GetResponse method
* Changing exception message if response file is not found
Co-authored-by: Ferenc Hammerl <31069338+fhammerl@users.noreply.github.com>
* Chaning exception message if isAlpine is null for prepare_job hook
* Rename hook folder
* Fail if compatible hookfile not found
* Use .Value instead of casting bool? to bool
* Format spacing
* Formatting
* User user and system mvs
* Use variables instead of entire context in featuremanager
* Update stepTelemetry if step uses containerhooks
* Restore import
* Remove unneccessary field from HookContainer
* Refactor response context and portmappings
* Force allow hooks if FF is on
* Code refactoring
* Revert deleting usings
* Better hookContainer defaults and use correct portmapping list
* Make GetDefaultShellForScript a HostContext extension method
* Generic hookresponse
* Code refactoring, unnecessary properties removed - HookContainer moved to the HookInput.cs
* Remove empty line
* Code refactoring and better exception handling
* code refactor, removing unnecessary props
* Move hookstate to global ContainerHookState
* Trace exception before we throw it for not losing information
* Fix for null ref exception in GetResponse
* Adding additional check for null response in prepareJob hook
* Refactoring GetResponse with additional check
* Update error messages
* Ports in ResponseContainer changed from IList to IDictionary
* Fix port format
* Include dockerfile
* Send null Registry obj if there's nothing in it
* Minor formatting
* Check if hookIndexPath exists relocated to the ContainerHookManager
* Code refactoring - ValidateHookExecutable added to the ContainerHookManager
* check if ContainerHooksPath when AllowRunnerContainerHooks is on
* Submit JSON telemetry instead of boolean
* Prefix step hooks with "run"
* Rename FeatureManager
* Fix flipped condition
* Unify js shell path getter with ps1 and sh getter
* Validate on run, not on instantiation of manager
* Cleanup ExecuteAsync methods
* Handle exception in executeHookScript
* Better exception types
* Remove comment
* Simplify boolean
* Allow jobs without jobContainer to run
* Use JObject instead of JToken
* Use correct Response type
* Format class to move cleanupJobAsync to the end of public methods
* Rename HookIndexPath to HookScriptPath
* Refactor methods into expression bodies
* Fix args class hierarchy
* Fix argument order
* Formatting
* Fix nullref and don't swallow stacktrace
* Whilelist HookArgs
* Use FF in FeatureManager
* Update src/Runner.Worker/ContainerOperationProvider.cs
Co-authored-by: Tingluo Huang <tingluohuang@github.com>
* Update src/Runner.Worker/ActionRunner.cs
Co-authored-by: Thomas Boop <52323235+thboop@users.noreply.github.com>
* Update src/Runner.Worker/ActionRunner.cs
Co-authored-by: Thomas Boop <52323235+thboop@users.noreply.github.com>
* Only mount well known dirs to job containers
* Get trace from hostcontext
* Use hook execution for setting telemetry
Co-authored-by: Nikola Jokic <nikola.jokic@akvelon.com>
Co-authored-by: Nikola Jokic <nikola-jokic@users.noreply.github.com>
Co-authored-by: Nikola Jokic <97525037+nikola-jokic@users.noreply.github.com>
Co-authored-by: Stefan Ruvceski <stefan.ruvceski@akvelon.com>
Co-authored-by: ruvceskistefan <96768603+ruvceskistefan@users.noreply.github.com>
Co-authored-by: Thomas Boop <thboop@github.com>
Co-authored-by: stefanruvceski <ruvceskistefan@github.com>
Co-authored-by: Tingluo Huang <tingluohuang@github.com>
Co-authored-by: Thomas Boop <52323235+thboop@users.noreply.github.com>
* composite action github.action_path set based on the StepHost
* in progress on updating github context for input template
* Fixed updating the context data for evaluation
* refactored logic so it is a little cleaner
* removed resolving the action_path in CompositeActionHandler
* removed added DeepClone
* added feature flag and modified the dict in place
* refactored step host to change context data. Added L0
* repaired spaces
* moved logic from step host to execution context, added recursive translation
* removed empty lines
* moved to extension methods
* Update src/Test/L0/Worker/StepHostL0.cs
Co-authored-by: Ferenc Hammerl <31069338+fhammerl@users.noreply.github.com>
* Added continue on error to composite action
* changed from boolean-strategy-context -> boolean-steps-context for action_yaml
* refactored composite handler to always set outcome
* retrigger checks
* fixed typo in ??= operator
* boolean-steps-context accepts the same context as string-steps-context
* setting the outcome only on continue-on-error
* moved continue on error logic to the execution context
* Added L0 table tests for continue-on-error ExecutionContext
* Added missing mocks on StepsRunnerL0 for this update
* removed empty line and added one line separating the call
* Removed empty line
Co-authored-by: Ferenc Hammerl <31069338+fhammerl@users.noreply.github.com>
Co-authored-by: Ferenc Hammerl <31069338+fhammerl@users.noreply.github.com>
* Use 16 to run RunnerService.js
* Execute hashfiles using node16
* Run downloadCert.js using node16
* Run makeWebRequest.js using node16
* Run macos-run-invoker.js using node16
* Run hashFiles.js using node16
* Update tests to use node16
* Update documentation to recommend node16
* Duplicate macos service js fix for 16
* Add PR link
* Revert ADR node change
* Merge node12/16 retainment IFs
* Try both node12 and node16
* Close if
* Revert "Update tests to use node16"
This reverts commit bbca7b9f1c.
* Fix condition
* Unfurl if condition
* Allow user to force a node version
* Format update template
* Comment env var
* Rename vars
* Fix naming
* Fix rename
* Set node ver override if job message has it
* Format executionContext
* Can only receive 'forceNode12' or nothing from FF
No specific node version from server
Co-authored-by: Ferenc Hammerl <hammerl.ferenc@gmail.com>
* conditional support for composite actions
* Fix Conditional function evaluation
* Push launch.json temporarily
* Revert "Push launch.json temporarily"
* rename context
* Cleanup comments
* fix success/failure functions to run based on pre/main steps
* idea of step_status
* change to use steps context, WIP
* add inputs to possible if condition expressions
* use action_status
* pr cleanup
* Added right stages
* Test on stage in conditional functions
* Fix naming and formatting
* Fix tests
* Add success and failure L0s
* Remove comment
* Remove whitespace
* Undo formatting
* Add L0 for step-if parsing
* Add ADR
Co-authored-by: Thomas Boop <thboop@github.com>
* Always register post steps for local actions
* Register post steps along with their conditions
* remove debug code
Co-authored-by: Ferenc Hammerl <fhammerl@github.com>
* Add generateIdTokenUrl as an env var
* Add generateIdTokenUrl to env vars
* Add basic telemetry class and submit it on jobcompleted
* Use constructor overload
* Rename telemetry to jobTelemetry
* Rename telemetry file
* Make JobTelemetryType a string
* Collect telemetry
* Remove debugger
* Update src/Runner.Worker/ActionCommandManager.cs
Co-authored-by: Thomas Boop <52323235+thboop@users.noreply.github.com>
* Use same JobTelemetry for all contexts
* Mask telemetry data
* Mask in JobRunner instead
* Empty line
* Change method signature
Returning with a List suggests we clone it and that the
original doesn't change..
* Update launch.json
Co-authored-by: Thomas Boop <52323235+thboop@users.noreply.github.com>
* Add Notice Command
* Add Feature Flag For Enhanced Annotations
* Fix tests
* Add validation for columns and lines
* Fix order to match service
* Remove console.write
* Make Validation Better
* Cleanup
* Handle empty/whitespace strings
* Add more validation for line/column ranges
* Make Validation Debug, Not Throw
* Change casing to 🐫 from 🐍
* Give notice a well known tag
* Cleanup
* Sanitize invalid commands rather than fail
During job run we may fail to resolve actions download info, and this
stack is fully controlled by GitHub actions so it should be counted as
infrastructure failure instead of user failure.
* Remove redundant code (display name is already evaluated in ActionRunner beforehand for each step)
* remove
* Remove nesting information for composite steps.
* put messages in debug logs if composite. if not, put these messages as outputs
* Fix group issue
* Fix end group issue
* Exploring child Linked Cancellation Tokens
* Preliminary Timeout-minutes fix
* Final Solution for resolving cancellation token's timeout vs. cancellation
* Clean up + Fix error handling
* Use linked tokens instead
* Clean up
* one liner
* Remove JobExecutionContext => Replace with public Root accessor
* Move CreateLinkedTokenSource in the CreateCompositeStep Function
This PR changes GITHUB_ACTION to use the step ContextName, instead of refname. The behavior is behind a feature flag. Refname is an otherwise deprecated property.
Primary motivation: For composite actions, we need a distinct GITHUB_ACTION for each nested step. This PR adds code to generate a default context name for nested steps.
For nested steps, GITHUB_ACTION will be set to "{ScopeName}.{ContextName}" to ensure no collisions.
A corresponding change will be made on the server so context name is never empty. Generated context names will start with "__".
A follow-up PR is required to avoid tracking "step" context values (outputs/conclusion/result) for generated context names. Waiting on telemetry from the server to confirm it's safe to assume leading "__" is a generate context name.
* Add basic framework for baby steps runner
* Basic logic for adding steps / invoking composite action steps
* Composite Steps Runner MVP
* Fix null object reference error
* intialize composiute
* Comment out code that is handled by stepsrunner
* Add composite clean up step
* Remove previous 'workarounds' from StepsRunner. Clean Up PR
* Remove todo
* Remove todo
* Fix using unitialized object yikes
* Remove time delay
* Format handler
* Move output handler into action handler
* Add try to evaluate display name
* Remove while loop yikes
* Abstract away the windows encoding check during step running
* Github context set to {ScopeName}.{ContextName} or {ContextName} if ScopeName is null
* Remove setting result to sucess since result defaults to sucess
* Fix windows error
* Fix windows
* revert:
* Windows fix
* Fix Windows Error in Abstraction
* Remove Composite Steps Runner => consolidate into Composite Steps Runner
* Remove unn. attribute in ExecutionContext
* Change protection levels, plus change function name to more clear meaning
* Remove location param
* location pt.2 fix
* Remove outputs step
* Remove temp directory
* new line
* Add arguitl not null
* better comment
* Change encoding name
* Check count > 0 for composite steps, import System.Threading
* Change function header encodingutil
* Add TODO
* Add await
* Handle Failed Step
* Move over SetAllCompositeOutputs to the handler
* Remove timeout-minutes setting in steps-level
* Use only ExecutionContext
* Move using to the top
* Remove redundant check
* Change function name
* Remove testing code
* Consolidate error code
* Consolidate code
* Change HandleOutput => ProcessCompositeActionOutputs
* Remove set the timeout comment
* Add Cancelling functionality + Remove unn. parameter
* Composite Action Run Steps
* Env Flow => Able to get env variables and overwrite current env variables => but it doesn't 'stick'
* clean up
* Clean up trace messages + add Trace debug in ActionManager
* Add debugging message
* Optimize runtime of code
* Change String to string
* Add comma to Composite
* Change JobSteps to a List, Change Register Step function name
* Add TODO, remove unn. content
* Remove unnecessary code
* Fix unit tests
* Fix env format
* Remove comment
* Remove TODO message for context
* Add verbose trace logs which are only viewable by devs
* Initial Start for FileTable stuff
* Progress towards passing FileTable or FileID or FileName
* Sort usings in Composite Action Handler
* Change 0 to location
* Update context variables in composite action yaml
* Add helpful error message for null steps
* Pass fileID to all children token of root action token
* Change confusing term context => templateContext, Eliminate _fileTable and only use ExecutionContext.FileTable + update this table when need be
* Remove unnessary FileID attribute from CompositeActionExecutionData
* Clean up file path for error message
* Remove todo
* Initial start/framework for output handling
* Outline different class vs Handler approach
* Remove InitializeScope
* Remove InitializeScope
* Fix Workflow Step Env overiding Parent Env
* First Approach for Attaching ID + Group ID to each Composite Action Step
* Add GroupID to the ActionDefinitionData
* starting foundation for handling clean up outputs step
* Pass outputs data to each composite action step to enable set-output functionality
* Create ScopeName for whole composite action.
This will enable us to add to the StepsContext[ScopeName] for the composite action which will allow us to use all these outputs in the cleanup step
* Hook up composite output step to handler => tmmrw implement composite output handler
* Add post composite action step to cleanup outputs => triggers composite output cleanup handler
* Fix Outputs Token handling start. Add individual step scope names.
* Set up Scope Name and Context Name naming system{
* Figured out how to pass Parent Execution Context to clean up step
* Figured out how to pass Parent Execution Context and scope names to
clean up step
* Add GetOutput function for StepsContext
* Generate child scope name correctly if parent scope name is null
* Simplify InitializeScope()
* Outputs are set correctly and able to get all final outputs in handler
* Parse through Action Outputs
* Fix null ScopeName + ContextName in CompositeOutputHandler
* Shift over handling of Action Outputs to output handler
* First attempt to fix null retrievals for output variables
* Basic Support for Outputs Done.
* Clean up pt.1
* Refactor outputs to avoid using Action Reference
* Clean up code
* Clean up part 2
* Add clarifying comments for the output handler
* Remove TODO
* Remove env in composite action scope
* Clean up
* Revert back
* revert back
* add back envToken
* Remove unnecessary code
* Add file length check
* Clean up
* Fix logging issue
* Figure out how to handle set-env edge cases
* formatting
* fix unit tests
* Fix windows unit test syntax error
* Fix period
* Sanity check for fileTable add + remove unn. code
* revert back
* Add back line break
* Fix null errors
* Address situation if FileTable is null + add sanity check for adding file to fileTable
* add line
* Revert
* Fix unit tests to instantiate a FileTable
* Fix logic for trimming manifestfile path
* Add null check
* Revert
* Revert
* revert
* spacing
* Add filetable to testing file, remove ? since we know filetable should never be non null
* Fix Throw logic
* Clarify template outputs token
* Add another type support for outputs to avoid container unit tests errors
* Add mapping for parity
* Build support for new outputs format
* Build support for new outputs format
* Refactor to avoid duplication of action yaml for workflow yaml
* revert
* revert
* revert
* spacing
* Composite Action Run Steps
* Env Flow => Able to get env variables and overwrite current env variables => but it doesn't 'stick'
* clean up
* Clean up trace messages + add Trace debug in ActionManager
* Add debugging message
* Optimize runtime of code
* Change String to string
* Add comma to Composite
* Change JobSteps to a List, Change Register Step function name
* Add TODO, remove unn. content
* Remove unnecessary code
* Fix unit tests
* Fix env format
* Remove comment
* Remove TODO message for context
* Add verbose trace logs which are only viewable by devs
* Initial Start for FileTable stuff
* Progress towards passing FileTable or FileID or FileName
* Sort usings in Composite Action Handler
* Change 0 to location
* Update context variables in composite action yaml
* Add helpful error message for null steps
* Pass fileID to all children token of root action token
* Change confusing term context => templateContext, Eliminate _fileTable and only use ExecutionContext.FileTable + update this table when need be
* Remove unnessary FileID attribute from CompositeActionExecutionData
* Clean up file path for error message
* Remove todo
* Initial start/framework for output handling
* Outline different class vs Handler approach
* Remove InitializeScope
* Remove InitializeScope
* Fix Workflow Step Env overiding Parent Env
* First Approach for Attaching ID + Group ID to each Composite Action Step
* Add GroupID to the ActionDefinitionData
* starting foundation for handling clean up outputs step
* Pass outputs data to each composite action step to enable set-output functionality
* Create ScopeName for whole composite action.
This will enable us to add to the StepsContext[ScopeName] for the composite action which will allow us to use all these outputs in the cleanup step
* Hook up composite output step to handler => tmmrw implement composite output handler
* Add post composite action step to cleanup outputs => triggers composite output cleanup handler
* Fix Outputs Token handling start. Add individual step scope names.
* Set up Scope Name and Context Name naming system{
* Figured out how to pass Parent Execution Context to clean up step
* Figured out how to pass Parent Execution Context and scope names to
clean up step
* Add GetOutput function for StepsContext
* Generate child scope name correctly if parent scope name is null
* Simplify InitializeScope()
* Outputs are set correctly and able to get all final outputs in handler
* Parse through Action Outputs
* Fix null ScopeName + ContextName in CompositeOutputHandler
* Shift over handling of Action Outputs to output handler
* First attempt to fix null retrievals for output variables
* Basic Support for Outputs Done.
* Clean up pt.1
* Refactor outputs to avoid using Action Reference
* Clean up code
* Clean up part 2
* Add clarifying comments for the output handler
* Remove TODO
* Remove env in composite action scope
* Clean up
* Revert back
* revert back
* add back envToken
* Remove unnecessary code
* Add file length check
* Clean up
* Figure out how to handle set-env edge cases
* formatting
* fix unit tests
* Fix windows unit test syntax error
* Fix period
* Sanity check for fileTable add + remove unn. code
* revert back
* Add back line break
* Fix null errors
* Address situation if FileTable is null + add sanity check for adding file to fileTable
* add line
* Revert
* Fix unit tests to instantiate a FileTable
* Fix logic for trimming manifestfile path
* Add null check
* Revert
* Revert
* revert
* spacing
* Add filetable to testing file, remove ? since we know filetable should never be non null
* Fix Throw logic
* Clarify template outputs token
* Add another type support for outputs to avoid container unit tests errors
* Add mapping for parity
* Build support for new outputs format
* Refactor to avoid duplication of action yaml for workflow yaml
* Move SDK work in ActionManifestManager, Condense Code
* Defer runs evaluation till after for loop to ensure order doesn't matter
* Fix logic error in setting scope and context names
* Add Regex + Add Child Context name null resolution
* move private function to bottom of class
* Composite Action Run Steps
* Env Flow => Able to get env variables and overwrite current env variables => but it doesn't 'stick'
* clean up
* Clean up trace messages + add Trace debug in ActionManager
* Add debugging message
* Optimize runtime of code
* Change String to string
* Add comma to Composite
* Change JobSteps to a List, Change Register Step function name
* Add TODO, remove unn. content
* Remove unnecessary code
* Fix unit tests
* Fix env format
* Remove comment
* Remove TODO message for context
* Add verbose trace logs which are only viewable by devs
* Sort usings in Composite Action Handler
* Change 0 to location
* Update context variables in composite action yaml
* Add helpful error message for null steps
* Fix Workflow Step Env overiding Parent Env
* Remove env in composite action scope
* Clean up
* Revert back
* revert back
* add back envToken
* Remove unnecessary code
* Figure out how to handle set-env edge cases
* formatting
* fix unit tests
* Fix windows unit test syntax error
* Composite Action Run Steps
* Clean up trace messages + add Trace debug in ActionManager
* Change String to string
* Add comma to Composite
* Change JobSteps to a List, Change Register Step function name
* Add TODO, remove unn. content
* Remove unnecessary code
* Fix unit tests
* Add verbose trace logs which are only viewable by devs
* Sort usings in Composite Action Handler
* Change 0 to location
* Update context variables in composite action yaml
* Add helpful error message for null steps
