[Windows] Implement new directories hierarchy (#8616)

.github/workflows/windows2019.yml

@@ -5,7 +5,7 @@ on:
   pull_request_target:
     types: labeled
     paths:
-    - 'images/win/**'
+    - 'images/windows/**'
 
 defaults:
   run:

.github/workflows/windows2022.yml

@@ -5,7 +5,7 @@ on:
   pull_request_target:
     types: labeled
     paths:
-    - 'images/win/**'
+    - 'images/windows/**'
 
 defaults:
   run:

CONTRIBUTING.md

@@ -34,18 +34,18 @@ Here are a few things you can do that will increase the likelihood of your pull
 
 - For every new tool add validation scripts and update software report script to make sure that it is included to documentation
 - If the tool is available in other platforms (macOS, Windows, Linux), make sure you include it in as many as possible.
-- If installing a few versions of the tool, consider putting the list of versions in the corresponding `toolset.json` file. It will help other customers to configure their builds flexibly. See [toolset-windows-2019.json](images/win/toolsets/toolset-2019.json) as example.
+- If installing a few versions of the tool, consider putting the list of versions in the corresponding `toolset.json` file. It will help other customers to configure their builds flexibly. See [toolset-windows-2019.json](images/windows/toolsets/toolset-2019.json) as example.
 - Use consistent naming across all files
 - Validation scripts should be simple and shouldn't change image content
 
 ### Windows
 
-- Add a script that will install the tool and put the script in the `scripts/Installers` folder.  
+- Add a script that will install the tool and put the script in the `scripts/build` folder.  
-There are a bunch of helper functions that could simplify your code: `Choco-Install`, `Install-Binary`, `Install-VsixExtension`, `Start-DownloadWithRetry`, `Test-IsWin19`, `Test-IsWin22` (find the full list of helpers in [ImageHelpers.psm1](images/win/scripts/ImageHelpers/ImageHelpers.psm1)).
+There are a bunch of helper functions that could simplify your code: `Choco-Install`, `Install-Binary`, `Install-VsixExtension`, `Start-DownloadWithRetry`, `Test-IsWin19`, `Test-IsWin22` (find the full list of helpers in [ImageHelpers.psm1](images/windows/scripts/helpers/ImageHelpers.psm1)).
-- Add a script that will validate the tool installation and put the script in the `scripts/Tests` folder.  
+- Add a script that will validate the tool installation and put the script in the `scripts/tests` folder.  
 We use [Pester v5](https://github.com/pester/pester) for validation scripts. If the tests for the tool are complex enough, create a separate `*.Tests.ps1`. Otherwise, use `Tools.Tests.ps1` for simple tests.  
 Add `Invoke-PesterTests -TestFile <testFileName> [-TestName <describeName>]` at the end of the installation script to make sure that your tests will be run.
-- Add changes to the software report generator `images/win/scripts/SoftwareReport/SoftwareReport.Generator.ps1`. The software report generator is used to generate an image's README file, e.g. [Windows2019-Readme.md](images/win/Windows2019-Readme.md) and uses [MarkdownPS](https://github.com/Sarafian/MarkdownPS).
+- Add changes to the software report generator `images/windows/scripts/docs-gen/SoftwareReport.Generator.ps1`. The software report generator is used to generate an image's README file, e.g. [Windows2019-Readme.md](images/windows/Windows2019-Readme.md) and uses [MarkdownPS](https://github.com/Sarafian/MarkdownPS).
 
 ### Ubuntu
 

README.md

@@ -36,12 +36,11 @@ To build a VM machine from this repo's source, see the [instructions](docs/creat
 
 [ubuntu-22.04]: https://github.com/actions/runner-images/blob/main/images/linux/Ubuntu2204-Readme.md
 [ubuntu-20.04]: https://github.com/actions/runner-images/blob/main/images/linux/Ubuntu2004-Readme.md
-[windows-2022]: https://github.com/actions/runner-images/blob/main/images/win/Windows2022-Readme.md
+[windows-2022]: https://github.com/actions/runner-images/blob/main/images/windows/Windows2022-Readme.md
-[windows-2019]: https://github.com/actions/runner-images/blob/main/images/win/Windows2019-Readme.md
+[windows-2019]: https://github.com/actions/runner-images/blob/main/images/windows/Windows2019-Readme.md
 [macOS-11]: https://github.com/actions/runner-images/blob/main/images/macos/macos-11-Readme.md
 [macOS-12]: https://github.com/actions/runner-images/blob/main/images/macos/macos-12-Readme.md
 [macOS-13]: https://github.com/actions/runner-images/blob/main/images/macos/macos-13-Readme.md
-[macOS-10.15]: https://github.com/actions/runner-images/blob/main/images/macos/macos-10.15-Readme.md
 [self-hosted runners]: https://help.github.com/en/actions/hosting-your-own-runners
 
 ## Announcements

docs/create-image-and-azure-resources.md

@@ -216,7 +216,7 @@ Where:
 - `InstallPassword` - password for the user used to install software (Windows only)
 - `Location` - location where resources will be created (e.g. "East US")
 - `ImageName` and `ImageResourceGroupName` - name of the resource group where managed image will be stored
-- `TemplatePath` - path to the Packer template file (e.g. "images/win/windows2022.json")
+- `TemplatePath` - path to the Packer template file (e.g. "images/windows/templates/windows-2022.json")
 
 ### Required variables
 
@@ -269,9 +269,9 @@ Generated tool versions and details can be found in related projects:
 
 > :warning: These scripts are intended to run on a VM deployed in Azure
 
-The user, created during the image generation, does not exist in the result image hence some configuration files related to the user's home directory need to be changed as well as the file permissions for some directories. Scripts for that are located in the `post-generation` folder in the repository:
+The user, created during the image generation, does not exist in the result image hence some configuration files related to the user's home directory need to be changed as well as the file permissions for some directories. Scripts for that are located in the `post-gen` folder in the repository:
 
-- Windows: <https://github.com/actions/runner-images/tree/main/images/win/post-generation>
+- Windows: <https://github.com/actions/runner-images/tree/main/images/windows/assets/post-gen>
 - Linux: <https://github.com/actions/runner-images/tree/main/images/linux/post-generation>
 
 **Note:** The default user for Linux should have `sudo privileges`.

helpers/GenerateResourcesAndImage.ps1

@@ -18,10 +18,10 @@ Function Get-PackerTemplatePath {
 
     switch ($ImageType) {
         ([ImageType]::Windows2019) {
-            $relativeTemplatePath = Join-Path "win" "windows2019.json"
+            $relativeTemplatePath = Join-Path "windows" "templates" "windows-2019.json"
         }
         ([ImageType]::Windows2022) {
-            $relativeTemplatePath = Join-Path "win" "windows2022.json"
+            $relativeTemplatePath = Join-Path "windows" "templates" "windows-2022.json"
         }
         ([ImageType]::Ubuntu2004) {
             $relativeTemplatePath = Join-Path "linux" "ubuntu2004.json"

images.CI/linux-and-win/azure-pipelines/image-generation.yml

@@ -12,6 +12,9 @@ parameters:
   - name: image_type
     type: string
 
+  - name: image_template_name
+    type: string
+
   - name: image_readme_name
     type: string
 
@@ -62,11 +65,17 @@ jobs:
       targetType: 'inline'
       script: |
         $ImageType = "${{ parameters.image_type }}"
-        $TemplateDirectoryName = if ($ImageType.StartsWith("ubuntu")) { "linux" } else { "win" }
+        $TemplateDirectoryName = if ($ImageType.StartsWith("ubuntu")) { "linux" } else { "windows/templates" }
         $TemplateDirectoryPath = Join-Path "images" $TemplateDirectoryName | Resolve-Path
-        $TemplatePath = Join-Path $TemplateDirectoryPath "$ImageType.pkr.hcl"
+        
-        if ( -not (Test-Path $TemplatePath) ) {
+        $TemplateFileName = "${{ parameters.image_template_name }}"
-            $TemplatePath = Join-Path $TemplateDirectoryPath "$ImageType.json"
+        if ($TemplateFileName) {
+          $TemplatePath = Join-Path $TemplateDirectoryPath $TemplateFileName
+        } else {
+          $TemplatePath = Join-Path $TemplateDirectoryPath "$ImageType.pkr.hcl"
+          if ( -not (Test-Path $TemplatePath) ) {
+              $TemplatePath = Join-Path $TemplateDirectoryPath "$ImageType.json"
+          }
         }
         Write-Host "##vso[task.setvariable variable=TemplateDirectoryPath;]$TemplateDirectoryPath"
         Write-Host "##vso[task.setvariable variable=TemplatePath;]$TemplatePath"
@@ -104,8 +113,12 @@ jobs:
     inputs:
       targetType: 'inline'
       script: |
-        $readmePath = Join-Path "$(TemplateDirectoryPath)" "${{ parameters.image_readme_name }}"
+        $ImageType = "${{ parameters.image_type }}"
-        $softwareReportPath = Join-Path "$(TemplateDirectoryPath)" "software-report.json"
+        $rootDirectoryName = if ($ImageType.StartsWith("ubuntu")) { "linux" } else { "windows" }
+        $rootDirectoryPath = Join-Path "images" $rootDirectoryName | Resolve-Path
+        
+        $readmePath = Join-Path $rootDirectoryPath "${{ parameters.image_readme_name }}"
+        $softwareReportPath = Join-Path $rootDirectoryPath "software-report.json"
 
         Copy-Item -Path $readmePath -Destination "$(Build.ArtifactStagingDirectory)/"
         if (Test-Path $softwareReportPath) {

images.CI/linux-and-win/azure-pipelines/windows2019.yml

@@ -17,4 +17,5 @@ jobs:
 - template: image-generation.yml
   parameters:
     image_type: windows2019
-    image_readme_name: Windows2019-Readme.md
+    image_readme_name: Windows2019-Readme.md
+    image_template_name: windows-2019.json

images.CI/linux-and-win/azure-pipelines/windows2022.yml

@@ -17,4 +17,5 @@ jobs:
 - template: image-generation.yml
   parameters:
     image_type: windows2022
-    image_readme_name: Windows2022-Readme.md
+    image_readme_name: Windows2022-Readme.md
+    image_template_name: windows-2022.json

images/win/scripts/README.md

@@ -1 +0,0 @@
-Common scripts for all Windows images regardless of Visual Studio or OS version

images/windows/assets/post-gen/VSConfiguration.ps1

@@ -1,14 +1,14 @@
-$vsInstallRoot = (Get-VisualStudioInstance).InstallationPath
+$vsInstallRoot = (Get-VisualStudioInstance).InstallationPath
-$devEnvPath = "$vsInstallRoot\Common7\IDE\devenv.exe"
+$devEnvPath = "$vsInstallRoot\Common7\IDE\devenv.exe"
-
+
-# Initialize Visual Studio Experimental Instance
+# Initialize Visual Studio Experimental Instance
-# The Out-Null cmdlet is required to ensure PowerShell waits until the '/ResetSettings' command fully completes.
+# The Out-Null cmdlet is required to ensure PowerShell waits until the '/ResetSettings' command fully completes.
-& "$devEnvPath" /RootSuffix Exp /ResetSettings General.vssettings /Command File.Exit | Out-Null
+& "$devEnvPath" /RootSuffix Exp /ResetSettings General.vssettings /Command File.Exit | Out-Null
-
+
-cmd.exe /c "`"$devEnvPath`" /updateconfiguration"
+cmd.exe /c "`"$devEnvPath`" /updateconfiguration"
-
+
-#
+#
-# https://github.com/actions/runner-images/issues/5301
+# https://github.com/actions/runner-images/issues/5301
-#
+#
-$warmup_vdproj = $(Join-Path $PSScriptRoot 'warmup.vdproj')
+$warmup_vdproj = $(Join-Path $PSScriptRoot 'warmup.vdproj')
-& "$devEnvPath" $warmup_vdproj /build Release | Out-Null
+& "$devEnvPath" $warmup_vdproj /build Release | Out-Null

images/windows/scripts/build/Configure-DynamicPort.ps1

@@ -1,14 +1,14 @@
-# https://support.microsoft.com/en-us/help/929851/the-default-dynamic-port-range-for-tcp-ip-has-changed-in-windows-vista
+# https://support.microsoft.com/en-us/help/929851/the-default-dynamic-port-range-for-tcp-ip-has-changed-in-windows-vista
-# The new default start port is 49152, and the new default end port is 65535.
+# The new default start port is 49152, and the new default end port is 65535.
-# Default port configuration was changed during image generation by Visual Studio Enterprise Installer to:
+# Default port configuration was changed during image generation by Visual Studio Enterprise Installer to:
-#   Protocol tcp Dynamic Port Range
+#   Protocol tcp Dynamic Port Range
-#   ---------------------------------
+#   ---------------------------------
-#   Start Port      : 1024
+#   Start Port      : 1024
-#   Number of Ports : 64511
+#   Number of Ports : 64511
-Write-Host "Set the dynamic port range to start at port 49152 and to end at the 65536 (16384 ports)"
+Write-Host "Set the dynamic port range to start at port 49152 and to end at the 65536 (16384 ports)"
-$null = netsh int ipv4 set dynamicport tcp start=49152 num=16384
+$null = netsh int ipv4 set dynamicport tcp start=49152 num=16384
-$null = netsh int ipv4 set dynamicport udp start=49152 num=16384
+$null = netsh int ipv4 set dynamicport udp start=49152 num=16384
-$null = netsh int ipv6 set dynamicport tcp start=49152 num=16384
+$null = netsh int ipv6 set dynamicport tcp start=49152 num=16384
-$null = netsh int ipv6 set dynamicport udp start=49152 num=16384
+$null = netsh int ipv6 set dynamicport udp start=49152 num=16384
-
+
 Invoke-PesterTests -TestFile "WindowsFeatures" -TestName "DynamicPorts"

images/windows/scripts/build/Disable-JITDebugger.ps1

@@ -1,16 +1,16 @@
-Write-Host "Disable Just-In-Time Debugger"
+Write-Host "Disable Just-In-Time Debugger"
-
+
-# Turn off Application Error Debugger
+# Turn off Application Error Debugger
-New-ItemProperty -Path "HKLM:\SOFTWARE\Microsoft\Windows NT\CurrentVersion\AeDebug" -Name Debugger -Value "-" -Type String -Force
+New-ItemProperty -Path "HKLM:\SOFTWARE\Microsoft\Windows NT\CurrentVersion\AeDebug" -Name Debugger -Value "-" -Type String -Force
-New-ItemProperty -Path "HKLM:\SOFTWARE\WOW6432Node\Microsoft\Windows NT\CurrentVersion\AeDebug" -Name Debugger -Value "-" -Type String -Force
+New-ItemProperty -Path "HKLM:\SOFTWARE\WOW6432Node\Microsoft\Windows NT\CurrentVersion\AeDebug" -Name Debugger -Value "-" -Type String -Force
-
+
-# Turn off the Debug dialog
+# Turn off the Debug dialog
-New-ItemProperty -Path "HKLM:\SOFTWARE\Microsoft\.NETFramework" -Name DbgManagedDebugger -Value "-" -Type String -Force
+New-ItemProperty -Path "HKLM:\SOFTWARE\Microsoft\.NETFramework" -Name DbgManagedDebugger -Value "-" -Type String -Force
-New-ItemProperty -Path "HKLM:\SOFTWARE\WOW6432Node\Microsoft\.NETFramework" -Name DbgManagedDebugger -Value "-" -Type String -Force
+New-ItemProperty -Path "HKLM:\SOFTWARE\WOW6432Node\Microsoft\.NETFramework" -Name DbgManagedDebugger -Value "-" -Type String -Force
-
+
-# Disable the WER UI
+# Disable the WER UI
-New-ItemProperty -Path "HKLM:\SOFTWARE\Microsoft\Windows\Windows Error Reporting" -Name DontShowUI -Value 1 -Type DWORD -Force
+New-ItemProperty -Path "HKLM:\SOFTWARE\Microsoft\Windows\Windows Error Reporting" -Name DontShowUI -Value 1 -Type DWORD -Force
-# Send all reports to the user's queue
+# Send all reports to the user's queue
-New-ItemProperty -Path "HKLM:\SOFTWARE\Microsoft\Windows\Windows Error Reporting" -Name ForceQueue -Value 1 -Type DWORD -Force
+New-ItemProperty -Path "HKLM:\SOFTWARE\Microsoft\Windows\Windows Error Reporting" -Name ForceQueue -Value 1 -Type DWORD -Force
-# Default consent choice 1 - Always ask (default)
+# Default consent choice 1 - Always ask (default)
 New-ItemProperty -Path "HKLM:\SOFTWARE\Microsoft\Windows\Windows Error Reporting\Consent" -Name DefaultConsent -Value 1 -Type DWORD -Force

images/windows/scripts/build/Install-AliyunCli.ps1

@@ -1,29 +1,29 @@
-################################################################################
+################################################################################
-##  File:  Install-AliyunCli.ps1
+##  File:  Install-AliyunCli.ps1
-##  Desc:  Install Alibaba Cloud CLI
+##  Desc:  Install Alibaba Cloud CLI
-##  Supply chain security: Alibaba Cloud CLI - checksum validation
+##  Supply chain security: Alibaba Cloud CLI - checksum validation
-################################################################################
+################################################################################
-
+
-Write-Host "Download Latest aliyun-cli archive"
+Write-Host "Download Latest aliyun-cli archive"
-$repoUrl = "https://api.github.com/repos/aliyun/aliyun-cli/releases/latest"
+$repoUrl = "https://api.github.com/repos/aliyun/aliyun-cli/releases/latest"
-$installerFileName = "aliyun-cli-windows"
+$installerFileName = "aliyun-cli-windows"
-$assets = (Invoke-RestMethod -Uri $repoUrl).assets
+$assets = (Invoke-RestMethod -Uri $repoUrl).assets
-$downloadUrl = ($assets.browser_download_url -ilike "*aliyun-cli-windows-*-amd64.zip*") | Select-Object -First 1
+$downloadUrl = ($assets.browser_download_url -ilike "*aliyun-cli-windows-*-amd64.zip*") | Select-Object -First 1
-$packagePath = Start-DownloadWithRetry -Url $downloadUrl -Name "$installerFileName.zip"
+$packagePath = Start-DownloadWithRetry -Url $downloadUrl -Name "$installerFileName.zip"
-
+
-#region Supply chain security - Alibaba Cloud CLI
+#region Supply chain security - Alibaba Cloud CLI
-$fileHash = (Get-FileHash -Path $packagePath -Algorithm SHA256).Hash
+$fileHash = (Get-FileHash -Path $packagePath -Algorithm SHA256).Hash
-$hashUrl = ($assets.browser_download_url -ilike "*SHASUMS256.txt*") | Select-Object -First 1
+$hashUrl = ($assets.browser_download_url -ilike "*SHASUMS256.txt*") | Select-Object -First 1
-$externalHash = (Invoke-RestMethod -Uri $hashURL).ToString().Split("`n").Where({ $_ -ilike "*$installerFileName*" }).Split(' ')[0]
+$externalHash = (Invoke-RestMethod -Uri $hashURL).ToString().Split("`n").Where({ $_ -ilike "*$installerFileName*" }).Split(' ')[0]
-Use-ChecksumComparison $fileHash $externalHash
+Use-ChecksumComparison $fileHash $externalHash
-#endregion
+#endregion
-
+
-Write-Host "Expand aliyun-cli archive"
+Write-Host "Expand aliyun-cli archive"
-$aliyunPath = "C:\aliyun-cli"
+$aliyunPath = "C:\aliyun-cli"
-New-Item -Path $aliyunPath -ItemType Directory -Force
+New-Item -Path $aliyunPath -ItemType Directory -Force
-Extract-7Zip -Path $packagePath -DestinationPath $aliyunPath
+Extract-7Zip -Path $packagePath -DestinationPath $aliyunPath
-
+
-# Add aliyun-cli to path
+# Add aliyun-cli to path
-Add-MachinePathItem $aliyunPath
+Add-MachinePathItem $aliyunPath
-
+
-Invoke-PesterTests -TestFile "CLI.Tools" -TestName "Aliyun CLI"
+Invoke-PesterTests -TestFile "CLI.Tools" -TestName "Aliyun CLI"

images/windows/scripts/build/Install-Bazel.ps1

@@ -1,10 +1,10 @@
-################################################################################
+################################################################################
-##  File:  Install-Bazel.ps1
+##  File:  Install-Bazel.ps1
-##  Desc:  Install Bazel and Bazelisk (A user-friendly launcher for Bazel)
+##  Desc:  Install Bazel and Bazelisk (A user-friendly launcher for Bazel)
-################################################################################
+################################################################################
-
+
-Choco-Install -PackageName bazel
+Choco-Install -PackageName bazel
-
+
-npm install -g @bazel/bazelisk
+npm install -g @bazel/bazelisk
-
+
 Invoke-PesterTests -TestFile "Tools" -TestName "Bazel"

images/windows/scripts/build/Install-Pipx.ps1

@@ -1,14 +1,14 @@
-#   Set custom directorys for pipx
+#   Set custom directorys for pipx
-$env:PIPX_BIN_DIR = "${env:ProgramFiles(x86)}\pipx_bin"
+$env:PIPX_BIN_DIR = "${env:ProgramFiles(x86)}\pipx_bin"
-$env:PIPX_HOME = "${env:ProgramFiles(x86)}\pipx"
+$env:PIPX_HOME = "${env:ProgramFiles(x86)}\pipx"
-
+
-#   Install pipx
+#   Install pipx
-pip install pipx
+pip install pipx
-
+
-#   Set environment variables
+#   Set environment variables
-Add-MachinePathItem "${env:PIPX_BIN_DIR}"
+Add-MachinePathItem "${env:PIPX_BIN_DIR}"
-Set-SystemVariable -SystemVariable PIPX_BIN_DIR -Value $env:PIPX_BIN_DIR
+Set-SystemVariable -SystemVariable PIPX_BIN_DIR -Value $env:PIPX_BIN_DIR
-Set-SystemVariable -SystemVariable PIPX_HOME -Value $env:PIPX_HOME
+Set-SystemVariable -SystemVariable PIPX_HOME -Value $env:PIPX_HOME
-
+
-#   Test pipx
+#   Test pipx
 Invoke-PesterTests -TestFile "Tools" -TestName "Pipx"

images/windows/scripts/build/Install-PipxPackages.ps1

@@ -1,12 +1,12 @@
-$pipxToolset = (Get-ToolsetContent).pipx
+$pipxToolset = (Get-ToolsetContent).pipx
-foreach($tool in $pipxToolset) {
+foreach($tool in $pipxToolset) {
-    if ($tool.python) {
+    if ($tool.python) {
-        $pythonPath = (Get-Item  -Path "${env:AGENT_TOOLSDIRECTORY}\Python\${tool.python}.*\x64\python-${tool.python}*").FullName
+        $pythonPath = (Get-Item  -Path "${env:AGENT_TOOLSDIRECTORY}\Python\${tool.python}.*\x64\python-${tool.python}*").FullName
-        Write-Host "Install ${tool.package} into python ${tool.python}"
+        Write-Host "Install ${tool.package} into python ${tool.python}"
-        pipx install $tool.package --python $pythonPath
+        pipx install $tool.package --python $pythonPath
-    } else {
+    } else {
-        Write-Host "Install ${tool.package} into default python"
+        Write-Host "Install ${tool.package} into default python"
-        pipx install $tool.package
+        pipx install $tool.package
-    }
+    }
-}
+}
 Invoke-PesterTests -TestFile "PipxPackages"

images/windows/scripts/build/Install-PowerShellModules.ps1

@@ -1,26 +1,26 @@
-# Set TLS1.2
+# Set TLS1.2
-[Net.ServicePointManager]::SecurityProtocol = [Net.ServicePointManager]::SecurityProtocol -bor "Tls12"
+[Net.ServicePointManager]::SecurityProtocol = [Net.ServicePointManager]::SecurityProtocol -bor "Tls12"
-
+
-# Install PowerShell modules
+# Install PowerShell modules
-$modules = (Get-ToolsetContent).powershellModules
+$modules = (Get-ToolsetContent).powershellModules
-
+
-foreach($module in $modules)
+foreach($module in $modules)
-{
+{
-    $moduleName = $module.name
+    $moduleName = $module.name
-    Write-Host "Installing ${moduleName} module"
+    Write-Host "Installing ${moduleName} module"
-
+
-    if ($module.versions)
+    if ($module.versions)
-    {
+    {
-        foreach ($version in $module.versions)
+        foreach ($version in $module.versions)
-        {
+        {
-            Write-Host " - $version"
+            Write-Host " - $version"
-            Install-Module -Name $moduleName -RequiredVersion $version -Scope AllUsers -SkipPublisherCheck -Force
+            Install-Module -Name $moduleName -RequiredVersion $version -Scope AllUsers -SkipPublisherCheck -Force
-        }
+        }
-        continue
+        continue
-    }
+    }
-
+
-    Install-Module -Name $moduleName -Scope AllUsers -SkipPublisherCheck -Force
+    Install-Module -Name $moduleName -Scope AllUsers -SkipPublisherCheck -Force
-}
+}
-
+
-Import-Module Pester
+Import-Module Pester
-Invoke-PesterTests -TestFile "PowerShellModules" -TestName "PowerShellModules"
+Invoke-PesterTests -TestFile "PowerShellModules" -TestName "PowerShellModules"

images/windows/scripts/build/Install-PyPy.ps1

@@ -1,124 +1,124 @@
-################################################################################
+################################################################################
-##  File:  Install-PyPy.ps1
+##  File:  Install-PyPy.ps1
-##  Team:  CI-Build
+##  Team:  CI-Build
-##  Desc:  Install PyPy
+##  Desc:  Install PyPy
-##  Supply chain security: checksum validation
+##  Supply chain security: checksum validation
-################################################################################
+################################################################################
-function Install-PyPy
+function Install-PyPy
-{
+{
-    param(
+    param(
-        [String]$PackagePath,
+        [String]$PackagePath,
-        [String]$Architecture
+        [String]$Architecture
-    )
+    )
-
+
-    # Create PyPy toolcache folder
+    # Create PyPy toolcache folder
-    $pypyToolcachePath = Join-Path -Path $env:AGENT_TOOLSDIRECTORY -ChildPath "PyPy"
+    $pypyToolcachePath = Join-Path -Path $env:AGENT_TOOLSDIRECTORY -ChildPath "PyPy"
-    if (-not (Test-Path $pypyToolcachePath)) {
+    if (-not (Test-Path $pypyToolcachePath)) {
-        Write-Host "Create PyPy toolcache folder"
+        Write-Host "Create PyPy toolcache folder"
-        New-Item -ItemType Directory -Path $pypyToolcachePath | Out-Null
+        New-Item -ItemType Directory -Path $pypyToolcachePath | Out-Null
-    }
+    }
-
+
-    # Expand archive with binaries
+    # Expand archive with binaries
-    $packageName = [IO.Path]::GetFileNameWithoutExtension((Split-Path -Path $packagePath -Leaf))
+    $packageName = [IO.Path]::GetFileNameWithoutExtension((Split-Path -Path $packagePath -Leaf))
-    $tempFolder = Join-Path -Path $pypyToolcachePath -ChildPath $packageName
+    $tempFolder = Join-Path -Path $pypyToolcachePath -ChildPath $packageName
-    Extract-7Zip -Path $packagePath -DestinationPath $pypyToolcachePath
+    Extract-7Zip -Path $packagePath -DestinationPath $pypyToolcachePath
-
+
-    # Get Python version from binaries
+    # Get Python version from binaries
-    $pypyApp = Get-ChildItem -Path "$tempFolder\pypy*.exe" | Where-Object Name -match "pypy(\d+)?.exe" | Select-Object -First 1
+    $pypyApp = Get-ChildItem -Path "$tempFolder\pypy*.exe" | Where-Object Name -match "pypy(\d+)?.exe" | Select-Object -First 1
-    $pythonVersion = & $pypyApp -c "import sys;print('{}.{}.{}'.format(sys.version_info[0],sys.version_info[1],sys.version_info[2]))"
+    $pythonVersion = & $pypyApp -c "import sys;print('{}.{}.{}'.format(sys.version_info[0],sys.version_info[1],sys.version_info[2]))"
-
+
-    $pypyFullVersion = & $pypyApp -c "import sys;print('{}.{}.{}'.format(*sys.pypy_version_info[0:3]))"
+    $pypyFullVersion = & $pypyApp -c "import sys;print('{}.{}.{}'.format(*sys.pypy_version_info[0:3]))"
-    Write-Host "Put '$pypyFullVersion' to PYPY_VERSION file"
+    Write-Host "Put '$pypyFullVersion' to PYPY_VERSION file"
-    New-Item -Path "$tempFolder\PYPY_VERSION" -Value $pypyFullVersion | Out-Null
+    New-Item -Path "$tempFolder\PYPY_VERSION" -Value $pypyFullVersion | Out-Null
-
+
-    if ($pythonVersion)
+    if ($pythonVersion)
-    {
+    {
-        Write-Host "Installing PyPy $pythonVersion"
+        Write-Host "Installing PyPy $pythonVersion"
-        $pypyVersionPath = Join-Path -Path $pypyToolcachePath -ChildPath $pythonVersion
+        $pypyVersionPath = Join-Path -Path $pypyToolcachePath -ChildPath $pythonVersion
-        $pypyArchPath = Join-Path -Path $pypyVersionPath -ChildPath $architecture
+        $pypyArchPath = Join-Path -Path $pypyVersionPath -ChildPath $architecture
-
+
-        Write-Host "Create PyPy '${pythonVersion}' folder in '${pypyVersionPath}'"
+        Write-Host "Create PyPy '${pythonVersion}' folder in '${pypyVersionPath}'"
-        New-Item -ItemType Directory -Path $pypyVersionPath -Force | Out-Null
+        New-Item -ItemType Directory -Path $pypyVersionPath -Force | Out-Null
-
+
-        Write-Host "Move PyPy '${pythonVersion}' files to '${pypyArchPath}'"
+        Write-Host "Move PyPy '${pythonVersion}' files to '${pypyArchPath}'"
-        Invoke-SBWithRetry -Command {
+        Invoke-SBWithRetry -Command {
-            Move-Item -Path $tempFolder -Destination $pypyArchPath -ErrorAction Stop | Out-Null
+            Move-Item -Path $tempFolder -Destination $pypyArchPath -ErrorAction Stop | Out-Null
-        }
+        }
-
+
-        Write-Host "Install PyPy '${pythonVersion}' in '${pypyArchPath}'"
+        Write-Host "Install PyPy '${pythonVersion}' in '${pypyArchPath}'"
-        if (Test-Path "$pypyArchPath\python.exe") {
+        if (Test-Path "$pypyArchPath\python.exe") {
-            cmd.exe /c "cd /d $pypyArchPath && python.exe -m ensurepip && python.exe -m pip install --upgrade pip"
+            cmd.exe /c "cd /d $pypyArchPath && python.exe -m ensurepip && python.exe -m pip install --upgrade pip"
-        } else {
+        } else {
-            $pypyName = $pypyApp.Name
+            $pypyName = $pypyApp.Name
-            cmd.exe /c "cd /d $pypyArchPath && mklink python.exe $pypyName && python.exe -m ensurepip && python.exe -m pip install --upgrade pip"
+            cmd.exe /c "cd /d $pypyArchPath && mklink python.exe $pypyName && python.exe -m ensurepip && python.exe -m pip install --upgrade pip"
-        }
+        }
-
+
-        # Create pip.exe if missing
+        # Create pip.exe if missing
-        $pipPath = Join-Path -Path $pypyArchPath -ChildPath "Scripts/pip.exe"
+        $pipPath = Join-Path -Path $pypyArchPath -ChildPath "Scripts/pip.exe"
-        if (-not (Test-Path $pipPath))
+        if (-not (Test-Path $pipPath))
-        {
+        {
-            $pip3Path = Join-Path -Path $pypyArchPath -ChildPath "Scripts/pip3.exe"
+            $pip3Path = Join-Path -Path $pypyArchPath -ChildPath "Scripts/pip3.exe"
-            Copy-Item -Path $pip3Path -Destination $pipPath 
+            Copy-Item -Path $pip3Path -Destination $pipPath 
-        }
+        }
-
+
-        if ($LASTEXITCODE -ne 0)
+        if ($LASTEXITCODE -ne 0)
-        {
+        {
-            Throw "Error happened during PyPy installation"
+            Throw "Error happened during PyPy installation"
-            exit 1
+            exit 1
-        }
+        }
-
+
-        Write-Host "Create complete file"
+        Write-Host "Create complete file"
-        New-Item -ItemType File -Path $pypyVersionPath -Name "$architecture.complete" | Out-Null
+        New-Item -ItemType File -Path $pypyVersionPath -Name "$architecture.complete" | Out-Null
-    }
+    }
-    else
+    else
-    {
+    {
-        Write-Host "PyPy application is not found. Failed to expand '$packagePath' archive"
+        Write-Host "PyPy application is not found. Failed to expand '$packagePath' archive"
-        exit 1
+        exit 1
-    }
+    }
-}
+}
-
+
-# Get PyPy content from toolset
+# Get PyPy content from toolset
-$toolsetVersions = Get-ToolsetContent | Select-Object -ExpandProperty toolcache | Where-Object Name -eq "PyPy"
+$toolsetVersions = Get-ToolsetContent | Select-Object -ExpandProperty toolcache | Where-Object Name -eq "PyPy"
-
+
-# Get PyPy releases
+# Get PyPy releases
-$pypyVersions = Invoke-RestMethod https://downloads.python.org/pypy/versions.json
+$pypyVersions = Invoke-RestMethod https://downloads.python.org/pypy/versions.json
-
+
-# required for html parsing
+# required for html parsing
-$checksums = (Invoke-RestMethod -Uri 'https://www.pypy.org/checksums.html' | ConvertFrom-HTML).SelectNodes('//*[@id="content"]/article/div/pre')
+$checksums = (Invoke-RestMethod -Uri 'https://www.pypy.org/checksums.html' | ConvertFrom-HTML).SelectNodes('//*[@id="content"]/article/div/pre')
-
+
-Write-Host "Starting installation PyPy..."
+Write-Host "Starting installation PyPy..."
-foreach($toolsetVersion in $toolsetVersions.versions)
+foreach($toolsetVersion in $toolsetVersions.versions)
-{
+{
-    # Query latest PyPy version
+    # Query latest PyPy version
-    $latestMajorPyPyVersion = $pypyVersions |
+    $latestMajorPyPyVersion = $pypyVersions |
-        Where-Object {$_.python_version.StartsWith("$toolsetVersion") -and $_.stable -eq $true} |
+        Where-Object {$_.python_version.StartsWith("$toolsetVersion") -and $_.stable -eq $true} |
-        Select-Object -ExpandProperty files -First 1 |
+        Select-Object -ExpandProperty files -First 1 |
-        Where-Object platform -like "win*"
+        Where-Object platform -like "win*"
-    
+    
-    if ($latestMajorPyPyVersion)
+    if ($latestMajorPyPyVersion)
-    {
+    {
-        $filename = $latestMajorPyPyVersion.filename
+        $filename = $latestMajorPyPyVersion.filename
-        Write-Host "Found PyPy '$filename' package"
+        Write-Host "Found PyPy '$filename' package"
-        $tempPyPyPackagePath = Start-DownloadWithRetry -Url $latestMajorPyPyVersion.download_url -Name $filename
+        $tempPyPyPackagePath = Start-DownloadWithRetry -Url $latestMajorPyPyVersion.download_url -Name $filename
-
+
-        #region Supply chain security
+        #region Supply chain security
-        $localFileHash = (Get-FileHash -Path $tempPyPyPackagePath -Algorithm SHA256).Hash
+        $localFileHash = (Get-FileHash -Path $tempPyPyPackagePath -Algorithm SHA256).Hash
-        $distributorFileHash = $null
+        $distributorFileHash = $null
-
+
-        ForEach($node in $checksums) {
+        ForEach($node in $checksums) {
-            if($node.InnerText -ilike "*${filename}*") {
+            if($node.InnerText -ilike "*${filename}*") {
-                $distributorFileHash = $node.InnerText.ToString().Split("`n").Where({ $_ -ilike "*${filename}*" }).Split(' ')[0]
+                $distributorFileHash = $node.InnerText.ToString().Split("`n").Where({ $_ -ilike "*${filename}*" }).Split(' ')[0]
-            }
+            }
-        }
+        }
-        
+        
-        Use-ChecksumComparison -LocalFileHash $localFileHash -DistributorFileHash $distributorFileHash
+        Use-ChecksumComparison -LocalFileHash $localFileHash -DistributorFileHash $distributorFileHash
-        #endregion
+        #endregion
-
+
-        Install-PyPy -PackagePath $tempPyPyPackagePath -Architecture $toolsetVersions.arch
+        Install-PyPy -PackagePath $tempPyPyPackagePath -Architecture $toolsetVersions.arch
-    }
+    }
-    else
+    else
-    {
+    {
-        Write-Host "Failed to query PyPy version '$toolsetVersion'"
+        Write-Host "Failed to query PyPy version '$toolsetVersion'"
-        exit 1
+        exit 1
-    }
+    }
-}
+}

images/windows/scripts/build/Install-Toolset.ps1

@@ -1,58 +1,58 @@
-################################################################################
+################################################################################
-##  File:  Install-Toolset.ps1
+##  File:  Install-Toolset.ps1
-##  Team:  CI-Build
+##  Team:  CI-Build
-##  Desc:  Install toolset
+##  Desc:  Install toolset
-################################################################################
+################################################################################
-
+
-[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12 -bor [Net.SecurityProtocolType]::Tls13
+[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12 -bor [Net.SecurityProtocolType]::Tls13
-
+
-Function Install-Asset {
+Function Install-Asset {
-    param(
+    param(
-        [Parameter(Mandatory=$true)]
+        [Parameter(Mandatory=$true)]
-        [object] $ReleaseAsset
+        [object] $ReleaseAsset
-    )
+    )
-
+
-    $releaseAssetName = [System.IO.Path]::GetFileNameWithoutExtension($ReleaseAsset.filename)
+    $releaseAssetName = [System.IO.Path]::GetFileNameWithoutExtension($ReleaseAsset.filename)
-    $assetFolderPath = Join-Path $env:TEMP $releaseAssetName
+    $assetFolderPath = Join-Path $env:TEMP $releaseAssetName
-    $assetArchivePath = Start-DownloadWithRetry -Url $ReleaseAsset.download_url -Name $ReleaseAsset.filename
+    $assetArchivePath = Start-DownloadWithRetry -Url $ReleaseAsset.download_url -Name $ReleaseAsset.filename
-
+
-    Write-Host "Extract $($ReleaseAsset.filename) content..."
+    Write-Host "Extract $($ReleaseAsset.filename) content..."
-    if ($assetArchivePath.EndsWith(".tar.gz")) {
+    if ($assetArchivePath.EndsWith(".tar.gz")) {
-        $assetTarPath = $assetArchivePath.TrimEnd(".tar.gz")
+        $assetTarPath = $assetArchivePath.TrimEnd(".tar.gz")
-        Extract-7Zip -Path $assetArchivePath -DestinationPath $assetTarPath
+        Extract-7Zip -Path $assetArchivePath -DestinationPath $assetTarPath
-        Extract-7Zip -Path $assetTarPath -DestinationPath $assetFolderPath
+        Extract-7Zip -Path $assetTarPath -DestinationPath $assetFolderPath
-    } else {
+    } else {
-        Extract-7Zip -Path $assetArchivePath -DestinationPath $assetFolderPath
+        Extract-7Zip -Path $assetArchivePath -DestinationPath $assetFolderPath
-    }
+    }
-
+
-    Write-Host "Invoke installation script..."
+    Write-Host "Invoke installation script..."
-    Push-Location -Path $assetFolderPath
+    Push-Location -Path $assetFolderPath
-    Invoke-Expression .\setup.ps1
+    Invoke-Expression .\setup.ps1
-    Pop-Location
+    Pop-Location
-}
+}
-
+
-# Get toolcache content from toolset
+# Get toolcache content from toolset
-$ToolsToInstall = @("Python", "Node", "Go")
+$ToolsToInstall = @("Python", "Node", "Go")
-
+
-$tools = Get-ToolsetContent | Select-Object -ExpandProperty toolcache | Where-Object { $ToolsToInstall -contains $_.Name }
+$tools = Get-ToolsetContent | Select-Object -ExpandProperty toolcache | Where-Object { $ToolsToInstall -contains $_.Name }
-
+
-foreach ($tool in $tools) {
+foreach ($tool in $tools) {
-    # Get versions manifest for current tool
+    # Get versions manifest for current tool
-    $assets = Invoke-SBWithRetry -Command { Invoke-RestMethod $tool.url }
+    $assets = Invoke-SBWithRetry -Command { Invoke-RestMethod $tool.url }
-
+
-    # Get github release asset for each version
+    # Get github release asset for each version
-    foreach ($toolVersion in $tool.versions) {
+    foreach ($toolVersion in $tool.versions) {
-        $asset = $assets | Where-Object version -like $toolVersion `
+        $asset = $assets | Where-Object version -like $toolVersion `
-                         | Select-Object -ExpandProperty files `
+                         | Select-Object -ExpandProperty files `
-                         | Where-Object { ($_.platform -eq $tool.platform) -and ($_.arch -eq $tool.arch) -and ($_.toolset -eq $tool.toolset) } `
+                         | Where-Object { ($_.platform -eq $tool.platform) -and ($_.arch -eq $tool.arch) -and ($_.toolset -eq $tool.toolset) } `
-                         | Select-Object -First 1
+                         | Select-Object -First 1
-
+
-        Write-Host "Installing $($tool.name) $toolVersion $($tool.arch)..."
+        Write-Host "Installing $($tool.name) $toolVersion $($tool.arch)..."
-        if ($null -ne $asset) {
+        if ($null -ne $asset) {
-            Install-Asset -ReleaseAsset $asset
+            Install-Asset -ReleaseAsset $asset
-        } else {
+        } else {
-            Write-Host "Asset was not found in versions manifest"
+            Write-Host "Asset was not found in versions manifest"
-            exit 1
+            exit 1
-        }
+        }
-    }
+    }
 }

images/windows/scripts/build/Install-Vsix.ps1

@@ -1,24 +1,24 @@
-###################################################################################
+###################################################################################
-##  File:  Install-Vsix.ps1
+##  File:  Install-Vsix.ps1
-##  Desc:  Install the Visual Studio Extensions from toolset.json
+##  Desc:  Install the Visual Studio Extensions from toolset.json
-###################################################################################
+###################################################################################
-
+
-$toolset = Get-ToolsetContent
+$toolset = Get-ToolsetContent
-$vsixPackagesList = $toolset.visualStudio.vsix
+$vsixPackagesList = $toolset.visualStudio.vsix
-if (-not $vsixPackagesList) {
+if (-not $vsixPackagesList) {
-    Write-Host "No extensions to install"
+    Write-Host "No extensions to install"
-    exit 0
+    exit 0
-}
+}
-
+
-$vsixPackagesList | ForEach-Object {
+$vsixPackagesList | ForEach-Object {
-    # Retrieve cdn endpoint to avoid HTTP error 429 https://github.com/actions/runner-images/issues/3074
+    # Retrieve cdn endpoint to avoid HTTP error 429 https://github.com/actions/runner-images/issues/3074
-    $vsixPackage = Get-VsixExtenstionFromMarketplace -ExtensionMarketPlaceName $_
+    $vsixPackage = Get-VsixExtenstionFromMarketplace -ExtensionMarketPlaceName $_
-    if ($vsixPackage.FileName.EndsWith(".vsix")) {
+    if ($vsixPackage.FileName.EndsWith(".vsix")) {
-        Install-VsixExtension -Url $vsixPackage.DownloadUri -Name $vsixPackage.FileName
+        Install-VsixExtension -Url $vsixPackage.DownloadUri -Name $vsixPackage.FileName
-    } else {
+    } else {
-        $argumentList = ('/install', '/quiet', '/norestart')
+        $argumentList = ('/install', '/quiet', '/norestart')
-        Install-Binary -Url $vsixPackage.DownloadUri -Name $vsixPackage.FileName -ArgumentList $argumentList
+        Install-Binary -Url $vsixPackage.DownloadUri -Name $vsixPackage.FileName -ArgumentList $argumentList
-    }
+    }
-}
+}
-
+
 Invoke-PesterTests -TestFile "Vsix"