Add template for minimal image and update tutorial (#7461)

docs/create-image-and-azure-resources.md

@@ -1,104 +1,173 @@
 # GitHub Actions Runner Images
-The runner-images project uses [Packer](https://www.packer.io/) to generate disk images for the following platforms: Windows 2019/2022, Ubuntu 20.04/22.04. 
-Each image is configured through a JSON template that Packer understands and which specifies where to build the image (Azure in this case), and what scripts to run to install software and prepare the disk.
-The Packer process initializes a connection to Azure subscription via Azure CLI, and automatically creates the temporary Azure resources required to build the source VM(temporary resource group, network interfaces, and VM from the "clean" image specified in the template). 
-If the VM deployment succeeds, the build agent connects to the VM and starts to execute installation steps from the JSON template.
-If any step in the JSON template fails, image generation will be aborted and the temporary VM will be terminated. Packer will also attempt to cleanup all the temporary resources it created (unless otherwise told).
-After successful image generation, a snapshot of the temporary VM will be converted to VHD image and then uploaded to the specified Azure Storage Account.
 
-## Prerequisites and Image-generation
+The runner-images project uses [Packer](https://www.packer.io/) to generate disk images for Windows 2019/2022 and Ubuntu 20.04/22.04.
-### Build Agent requirements
-- `OS` - Windows/Linux
-- `packer 1.8.2 or higher` - Can be downloaded from https://www.packer.io/downloads
-- `PowerShell 5.0 or higher` or `PSCore` for linux distributes.
-- `Azure CLI ` - https://docs.microsoft.com/en-us/cli/azure/install-azure-cli
-- `Azure Az Powershell module` - https://docs.microsoft.com/en-us/powershell/azure/install-az-ps
-- `Git for Windows` - https://gitforwindows.org/
 
-> To connect to a temporary VM packer uses WinRM or SSH connections on public IP interfaces.
+Each image is configured by a JSON or HCL2 Packer template that specifies where to build the image (Azure in this case)
-If you use a build agent located in an Azure subscription, please make sure that HTTPS/SSH ports are allowed for incoming/outgoing connections.
+and what steps to run to install software and prepare the disk.
-In case of firewall restrictions, prohibiting connections from public addresses, private virtual network resources can be deployed and passed as arguments to the packer. This approach allows virtual machines to use private connections inside VLAN.
 
-### Service principal
+The Packer process initializes a connection to Azure subscription using Azure CLI and creates temporary resources
-Packer uses Service Principal to authorize in Azure infrastructure. To setup image-generation CI or use packer manually — SP with full read-write permissions for selected Azure subscription needed.
+required for the build process: resource group, network interfaces and virtual machine from the "clean" image specified in the template.
-Detailed instruction can be found in [Azure documentation](https://docs.microsoft.com/en-us/azure/active-directory/develop/howto-create-service-principal-portal)
 
-### Prepare environment and image deployment
+If the VM deployment succeeds, Packer connects it using ssh or WinRM and begins executing installation steps from the template one-by-one.
-#### How to prepare Windows build agent
+If any step fails, image generation is aborted and the temporary VM is terminated.
-Local machine or [Azure VM](https://docs.microsoft.com/en-us/azure/virtual-machines/windows/quick-create-cli) can be used as a build agent.
+Packer also attempts to cleanup all the temporary resources it created (unless otherwise configured).
 
-Download & install `packer` from https://www.packer.io/downloads, or install it via [Chocolatey](https://chocolatey.org/):
+After successful completion of all installation steps Packer converts snapshot of the temporary VM to VHD image
-```
+and uploads it to the specified Azure Storage Account.
-choco install packer
-```
 
-Download & install `git` from https://github.com/git-for-windows/git/releases, or install it via [Chocolatey](https://chocolatey.org/):
+## Build agent preparation
-```
-choco install git -params '"/GitAndUnixToolsOnPath"'
-```
 
-Install the Azure Az PowerShell module - https://docs.microsoft.com/en-us/powershell/azure/install-az-ps.
+Build agent is a machine where Packer process will be started.
-```
+You can use any physical or virtual machine running OS Windows or Linux.
-Install-Module -Name Az -Repository PSGallery -Force
+Of course you may also use [Azure VM](https://docs.microsoft.com/en-us/azure/virtual-machines/windows/quick-create-cli).
-```
+In any case you will need these software installed:
 
-Install Azure CLI - https://docs.microsoft.com/en-us/cli/azure/install-azure-cli-windows?view=azure-cli-latest&tabs=azure-cli.
+- Packer 1.8.2 or higher.
-```
-Invoke-WebRequest -Uri https://aka.ms/installazurecliwindows -OutFile .\AzureCLI.msi; Start-Process msiexec.exe -Wait -ArgumentList '/I AzureCLI.msi /quiet'; rm .\AzureCLI.msi
-```
 
-Download Runner Images repository.
+  Download and install it manually from [here](https://www.packer.io/downloads) or use [Chocolatey](https://chocolatey.org/):
-```
+
-Set-Location c:\
+  ```powershell
+  choco install packer
+  ```
+
+- Git.
+
+  For Linux - install the latest version from your distro's package repo.
+
+  For Windows - download and install it from [here](https://gitforwindows.org/) of use [Chocolatey](https://chocolatey.org/):
+
+  ```powershell
+  choco install git -params '"/GitAndUnixToolsOnPath"'
+  ```
+
+- Powershell 5.0 or higher.
+
+  In Windows you already have it.
+
+  For Linux follow instructions [here](https://learn.microsoft.com/en-us/windows-server/administration/linux-package-repository-for-microsoft-software)
+  to add Microsoft's Linux Software Repository and then install package `powershell`.
+- Azure CLI.
+
+  Follow instructions [here](https://docs.microsoft.com/en-us/cli/azure/install-azure-cli).
+  Or if you use Windows you may run this command in Powershell instead:
+
+  ```powershell
+  Invoke-WebRequest -Uri https://aka.ms/installazurecliwindows -OutFile .\AzureCLI.msi
+  Start-Process msiexec.exe -Wait -ArgumentList '/I AzureCLI.msi /quiet'; rm .\AzureCLI.msi
+  ```
+
+- [Az Powershell module](https://docs.microsoft.com/en-us/powershell/azure/install-az-ps).
+  
+  Run this command in Powershell:
+
+  ```powershell
+  Install-Module -Name Az -Repository PSGallery -Force
+  ```
+
+## Automated image generation
+
+This repo bundles script that automates image generation process.
+You only need a build agent configured as described above and active Azure subsctiption.
+We suggest to start with UbuntuMinimal image because it includes only a minimal set of required software and builds in less then half an hour.
+
+All steps here are supposed to run in Powershell.
+
+First, clone runner-images repository and change directory:
+
+```powershell
 git clone https://github.com/actions/runner-images.git
+Set-Location runner-images
 ```
 
-Import [GenerateResourcesAndImage](../helpers/GenerateResourcesAndImage.ps1) script from `/helpers` folder, and run `GenerateResourcesAndImage` function via Powershell.
+Then import [GenerateResourcesAndImage](../helpers/GenerateResourcesAndImage.ps1) script from `helpers` subdirectory:
-
-```
-Set-Location C:\runner-images
 
+```powershell
 Import-Module .\helpers\GenerateResourcesAndImage.ps1
-
-GenerateResourcesAndImage -SubscriptionId {YourSubscriptionId} -ResourceGroupName "myTestResourceGroup" -ImageGenerationRepositoryRoot "$pwd" -ImageType Ubuntu2004 -AzureLocation "East US"
-```
-Where:
-- `SubscriptionId` - The Azure subscription Id where resources will be created.
-- `ResourceGroupName` - The Azure resource group name where the Azure resources will be created.
-- `ImageGenerationRepositoryRoot` - The root path of the image generation repository source.
-- `ImageType` - The type of the image being generated. Valid options are: "Windows2019", "Windows2022", "Ubuntu2004", "Ubuntu2204".
-- `AzureLocation` - The location of the resources being created in Azure. For example "East US".
-
-The function automatically creates all required Azure resources and kicks off packer image generation for the selected image type.
-
-For optional authentication via service principal make sure to provide the following params — `AzureClientId`, `AzureClientSecret`, `AzureTenantId`, so the whole command will be:
-
-```
-GenerateResourcesAndImage -SubscriptionId {YourSubscriptionId} -ResourceGroupName "myTestResourceGroup" -ImageGenerationRepositoryRoot "$pwd" -ImageType Ubuntu2004 -AzureLocation "East US" -AzureClientId {AADApplicationID} -AzureClientSecret {AADApplicationSecret} -AzureTenantId {AADTenantID}
 ```
 
-As extra options, you can add more params for permit to add Azure Tags on resources and enable https for Storage Account. It could be helpful on some tenants with hardenning policy. Params are — `EnableHttpsTrafficOnly` (Boolean) and `tags` (HashTable), so the whole command will be:
+Finally, run `GenerateResourcesAndImage` function setting mandatory arguments: image type and where to create resources:
 
-```
+- `SubscriptionId` - your Azure Subscription ID
-GenerateResourcesAndImage -SubscriptionId {YourSubscriptionId} -ResourceGroupName "myTestResourceGroup" -ImageGenerationRepositoryRoot "$pwd" -ImageType Ubuntu2004 -AzureLocation "East US" -EnableHttpsTrafficOnly $true -tags @{dept="devops";env="prod"}
+- `ResourceGroupName` - name of the resource group that will be created within your subscription (e.g. "imagegen-test")
+- `AzureLocation` - location where resources will be created (e.g. "East US")
+- `ImageType` - what image to build (we suggest choosing "UbuntuMinimal" here, other valid options are "Windows2019", "Windows2022", "Ubuntu2004", "Ubuntu2204")
+
+This function automatically creates all required Azure resources and kicks off packer image generation for the selected image type.
+
+When image is ready you may proceed to [deployment](#generated-machine-deployment)
+
+## Image generation customization
+
+Function `GenerateResourcesAndImage` accepts a bunch of arguments that may help you generating image in your specific environment.
+
+For example, you may want that all the resources involved in image generation process are tagged.
+In this case pass a HashTable of tags as a value for `Tags` parameter.
+
+If you don't want function to authenticate interactively, you should create Service Principal and invoke the function with parameters `AzureClientId`, `AzureClientSecret` and `AzureTenantId`.
+You can find more details [in corresponding section below](#azure-subscription-authentication).
+
+Use `get-help GenerateResourcesAndImage -Detailed` for the complete list of parameters available.
+
+### Network security
+
+To connect to a temporary virtual machine Packer uses WinRM or SSH.
+
+If your build agent is located outside of the Azure subscription where temporary VM is created, the public network interface and public IP address is used.
+Make sure that firewalls are configured properly and WinRM (tcp port 5986) and ssh (tcp port 22) connections are allowed both outgoing for build agent and incoming for temporary VM.
+Also if you don't want temporary VM to be accessible from everywhere, set `RestrictToAgentIpAddress` parameter value to `$true`
+to setup firewall rules allowing access only from your build agent public IP address.
+
+If your build agent and temporary VM are in the same subscription you can configure Packer to connect using private virtual network.
+To achieve that set proper values for environment variables `VNET_RESOURCE_GROUP`, `VNET_NAME` and `VNET_SUBNET`.
+
+### Azure subscription authentication
+
+Packer uses Service Principal to authenticate in Azure infrastructure.
+For more information about Service Principals refer to
+[Azure documentation](https://docs.microsoft.com/en-us/azure/active-directory/develop/howto-create-service-principal-portal).
+
+Function `GenerateResourcesAndImage` is able to create Service Principle to be used by Packer.
+It uses Connect-AzAccount cmdlet that invokes interactive authentication process by default.
+If you don't want to use interactive authentication you should create Service Principal with full read-write permissions for selected Azure subscription on your own
+and provide proper values for parameters `AzureClientId`, `AzureClientSecret` and `AzureTenantId`.
+
+Here is an example of how to create Service Principle using Az Powershell module:
+
+```powershell
+$credentials = [Microsoft.Azure.PowerShell.Cmdlets.Resources.MSGraph.Models.ApiV10.MicrosoftGraphPasswordCredential]@{
+  StartDateTime = Get-Date
+  EndDateTime = (Get-Date).AddDays(7)
+}
+
+$sp = New-AzADServicePrincipal -DisplayName "imagegen-app"
+$appCred = New-AzADAppCredential -ApplicationId $sp.AppId -PasswordCredentials $credentials
+
+Start-Sleep -Seconds 30
+New-AzRoleAssignment -RoleDefinitionName "Contributor" -PrincipalId $sp.Id
+Start-Sleep -Seconds 30
+
+@{
+  ClientId = $sp.AppId
+  ClientSecret = $appCred.SecretText
+  TenantId = (Get-AzSubscription -SubscriptionId $SubscriptionId).TenantId
+}
 ```
 
-*Please, check synopsis of `GenerateResourcesAndImage` for details about non-mandatory parameters.*
+## Generated machine deployment
 
-#### Generated VM Deployment
 After the successful image generation, Virtual Machine can be created from the generated VHD using [CreateAzureVMFromPackerTemplate](../helpers/CreateAzureVMFromPackerTemplate.ps1) script.
 
-```
+```powershell
-Set-Location C:\runner-images
-
 Import-Module .\helpers\CreateAzureVMFromPackerTemplate.ps1
 
 CreateAzureVMFromPackerTemplate -SubscriptionId {YourSubscriptionId}  -ResourceGroupName {ResourceGroupName} -TemplateFile "C:\BuildVmImages\temporaryTemplate.json" -VirtualMachineName "testvm1" -AdminUsername "shady1" -AdminPassword "SomeSecurePassword1" -AzureLocation "eastus"
 ```
+
 Where:
+
 - `SubscriptionId` - The Azure subscription Id where resources will be created.
 - `ResourceGroupName` - The Azure resource group name where the Azure virtual machine will be created.
-- `TemplateFilePath` - The path to the json ARM-template generated by packer during image generation locally.* 
+- `TemplateFilePath` - The path to the json ARM-template generated by packer during image generation locally.*
 - `VirtualMachineName` - The name of the virtual machine to be generated.
 - `AdminUserName` - The administrator username for the virtual machine to be created.
 - `AdminPassword` - The administrator password for the virtual machine to be created.
@@ -108,20 +177,27 @@ Where:
 
 The function creates an Azure VM from a template and generates network resources in Azure to make the VM accessible.
 
-## Additional
+## Manual image generation
-### User variables
+
-The Packer template includes `variables` section containing user variables used in image generation. Each variable is defined as a key/value strings. User variables can be passed to packer via predefined environment variables, or as direct arguments, in case if packer started manually.
+If you want more control over image generation process you may run Packer directly. This section describes variables defined in Packer template. Some of them may be set using environment variabes.
+
+### Required variables
+
+| Template var | Env var | Description
+| ------------ | ------- | -----------
+| `subscription_id` | `ARM_SUBSCRIPTION_ID` | Subscription under which the build will be performed.
+| `client_id` | `ARM_CLIENT_ID` | The Active Directory service principal associated with your builder.
+| `client_secret` | `ARM_CLIENT_SECRET` | The password or secret for your service principal; may be omitted if `client_cert_path` is set.
+| `client_cert_path` | `ARM_CLIENT_CERT_PATH` | The location of a PEM file containing a certificate and private key for service principal; may be omitted if `client_secret` is set.
+| `location` | `ARM_RESOURCE_LOCATION` | Azure datacenter in which your VM will build.
+| `resource_group` | `ARM_RESOURCE_GROUP` | Resource group under which the final artifact will be stored.
+| `storage_account` | `ARM_STORAGE_ACCOUNT` | Storage account under which the final artifact will be stored.
+
+### Optional variables
 
 - `build_resource_group_name` - Specify an existing resource group to run the build in it. By default, a temporary resource group will be created and destroyed as part of the build. If you do not have permission to do so, use build_resource_group_name to specify an existing resource group to run the build in it.
-- `client_id` - The application ID of the AAD Service Principal. Requires `client_secret`.
 - `object_id` - The object ID for the AAD SP. Will be derived from the oAuth token if empty.
-- `client_secret` - The password or secret for your service principal.
-- `client_cert_path` - The location of a PEM file containing a certificate and private key for service principal.
-- `subscription_id` - The subscription to use.
 - `tenant_id` - The Active Directory tenant identifier with which your `client_id` and `subscription_id` are associated. If not specified, `tenant_id` will be looked up using `subscription_id`.
-- `resource_group` - Resource group under which the final artifact will be stored.
-- `storage_account` - Storage account under which the final artifact will be stored.
-- `location` - Azure datacenter in which your VM will be built.
 - `temp_resource_group_name` - Name assigned to the temporary resource group created during the build. If this value is not set, a random value will be assigned. This resource group is deleted at the end of the build.
 - `private_virtual_network_with_public_ip` - This value allows you to set a `virtual_network_name` and obtain a public IP. If this value is not set and `virtual_network_name` is defined Packer is only allowed to be executed from a host on the same subnet / virtual network.
 - `virtual_network_name` - Use a pre-existing virtual network for the VM. This option enables private communication with the VM, no public IP address is used or provisioned (unless you set `private_virtual_network_with_public_ip`).
@@ -130,6 +206,7 @@ The Packer template includes `variables` section containing user variables used
 - `capture_name_prefix` - VHD prefix. The final artifacts will be named PREFIX-osDisk.UUID and PREFIX-vmTemplate.UUID.
 
 ### Builder variables
+
 The `builders` section contains variables for the `azure-arm` builder used in the project. Most of the builder variables are inherited from the `user variables` section, however, the variables can be overwritten to adjust image-generation performance.
 
 - `vm_size` - Size of the VM used for building. This can be changed when you deploy a VM from your VHD.
@@ -139,9 +216,11 @@ The `builders` section contains variables for the `azure-arm` builder used in th
 **Detailed Azure builders documentation can be found in [packer documentation](https://www.packer.io/docs/builders/azure).**
 
 ### Toolset
+
 Configuration for some installed software is located in `toolset.json` files. These files define the list of Ruby, Python, Go versions, the list of PowerShell modules and VS components that will be installed to image. They can be changed if these tools are not required to reduce image generation time or image size.
 
 Generated tool versions and details can be found in related projects:
+
 - [Python](https://github.com/actions/python-versions/)
 - [Go](https://github.com/actions/go-versions)
 - [Node](https://github.com/actions/node-versions)
@@ -151,24 +230,30 @@ Generated tool versions and details can be found in related projects:
 > :warning: These scripts are intended to run on a VM deployed in Azure
 
 The user, created during the image generation, does not exist in the result VHD hence some configuration files related to the user's home directory need to be changed as well as the file permissions for some directories. Scripts for that are located in the `post-generation` folder in the repository:
-- Windows: https://github.com/actions/runner-images/tree/main/images/win/post-generation
+
-- Linux: https://github.com/actions/runner-images/tree/main/images/linux/post-generation
+- Windows: <https://github.com/actions/runner-images/tree/main/images/win/post-generation>
+- Linux: <https://github.com/actions/runner-images/tree/main/images/linux/post-generation>
 
 **Note:** The default user for Linux should have `sudo privileges`.
 
 The scripts are copied to the VHD during the image generation process to the following paths:
+
 - Windows: `C:\post-generation`
-- Linux:  `/opt/post-generation` 
+- Linux:  `/opt/post-generation`
 
 #### Running scripts
 
-##### Ubuntu
+- Ubuntu
 
-        sudo su -c "find /opt/post-generation -mindepth 1 -maxdepth 1 -type f -name '*.sh' -exec bash {} \;"
+  ```bash
+  sudo su -c "find /opt/post-generation -mindepth 1 -maxdepth 1 -type f -name '*.sh' -exec bash {} \;"
+  ```
 
-##### Windows
+- Windows
 
-        Get-ChildItem C:\post-generation -Filter *.ps1 | ForEach-Object { & $_.FullName }
+  ```powershell
+  Get-ChildItem C:\post-generation -Filter *.ps1 | ForEach-Object { & $_.FullName }
+  ```
 
 #### Script details
 

helpers/GenerateResourcesAndImage.ps1

@@ -5,6 +5,7 @@ enum ImageType {
     Windows2022 = 2
     Ubuntu2004 = 3
     Ubuntu2204 = 4
+    UbuntuMinimal = 5
 }
 
 Function Get-PackerTemplatePath {
@@ -28,6 +29,9 @@ Function Get-PackerTemplatePath {
         ([ImageType]::Ubuntu2204) {
             $relativeTemplatePath = Join-Path "linux" "ubuntu2204.pkr.hcl"
         }
+        ([ImageType]::UbuntuMinimal) {
+            $relativeTemplatePath = Join-Path "linux" "ubuntuminimal.pkr.hcl"
+        }
         default { throw "Unknown type of image" }
     }
 
@@ -63,7 +67,7 @@ Function GenerateResourcesAndImage {
         .PARAMETER ImageGenerationRepositoryRoot
             The root path of the image generation repository source.
         .PARAMETER ImageType
-            The type of the image being generated. Valid options are: {"Windows2019", "Windows2022", "Ubuntu2004", "Ubuntu2204"}.
+            The type of the image being generated. Valid options are: {"Windows2019", "Windows2022", "Ubuntu2004", "Ubuntu2204", "UbuntuMinimal"}.
         .PARAMETER AzureLocation
             The location of the resources being created in Azure. For example "East US".
         .PARAMETER Force
@@ -77,9 +81,10 @@ Function GenerateResourcesAndImage {
         .PARAMETER RestrictToAgentIpAddress
             If set, access to the VM used by packer to generate the image is restricted to the public IP address this script is run from. 
             This parameter cannot be used in combination with the virtual_network_name packer parameter.
-        
         .PARAMETER AllowBlobPublicAccess
             The Azure storage account will be created with this option.
+        .PARAMETER EnableHttpsTrafficOnly
+            The Azure storage account will be created with this option.
         .PARAMETER OnError
             Specify how packer handles an error during image creation.
         .EXAMPLE

images/linux/scripts/installers/apt-common.sh

@@ -1,6 +1,6 @@
 #!/bin/bash -e
 ################################################################################
-##  File:  basic.sh
+##  File:  apt-common.sh
 ##  Desc:  Installs basic command line utilities and dev packages
 ################################################################################
 source $HELPER_SCRIPTS/install.sh

images/linux/scripts/installers/apt-vital.sh

@@ -0,0 +1,9 @@
+#!/bin/bash -e
+################################################################################
+##  File:  apt-vital.sh
+##  Desc:  Installs vital command line utilities
+################################################################################
+source $HELPER_SCRIPTS/install.sh
+
+vital_packages=$(get_toolset_value .apt.vital_packages[])
+apt-get install -y --no-install-recommends $vital_packages

images/linux/scripts/installers/docker-moby.sh

@@ -29,23 +29,27 @@ systemctl is-enabled --quiet docker.service || systemctl enable docker.service
 sleep 10
 docker info
 
-# If credentials are provided, attempt to log into Docker Hub
+if [ "${DOCKERHUB_PULL_IMAGES:-yes}" -eq "yes" ]; then
-# with a paid account to avoid Docker Hub's rate limit.
+    # If credentials are provided, attempt to log into Docker Hub
-if [ "${DOCKERHUB_LOGIN}" ] && [ "${DOCKERHUB_PASSWORD}" ]; then
+    # with a paid account to avoid Docker Hub's rate limit.
-  docker login --username "${DOCKERHUB_LOGIN}" --password "${DOCKERHUB_PASSWORD}"
+    if [ "${DOCKERHUB_LOGIN}" ] && [ "${DOCKERHUB_PASSWORD}" ]; then
+    docker login --username "${DOCKERHUB_LOGIN}" --password "${DOCKERHUB_PASSWORD}"
+    fi
+
+    # Pull images
+    images=$(get_toolset_value '.docker.images[]')
+    for image in $images; do
+        docker pull "$image"
+    done
+
+    # Always attempt to logout so we do not leave our credentials on the built
+    # image. Logout _should_ return a zero exit code even if no credentials were
+    # stored from earlier.
+    docker logout
+else
+    echo "Skipping docker images pulling"
 fi
 
-# Pull images
-images=$(get_toolset_value '.docker.images[]')
-for image in $images; do
-    docker pull "$image"
-done
-
-# Always attempt to logout so we do not leave our credentials on the built
-# image. Logout _should_ return a zero exit code even if no credentials were
-# stored from earlier.
-docker logout
-
 # Install amazon-ecr-credential-helper
 aws_latest_release_url="https://api.github.com/repos/awslabs/amazon-ecr-credential-helper/releases/latest"
 aws_helper_url=$(curl "${authString[@]}" -sL $aws_latest_release_url | jq -r '.body' | awk -F'[()]' '/linux-amd64/ {print $2}')
@@ -53,3 +57,6 @@ download_with_retries "$aws_helper_url" "/usr/bin" docker-credential-ecr-login
 chmod +x /usr/bin/docker-credential-ecr-login
 
 invoke_tests "Tools" "Docker"
+if [ "${DOCKERHUB_PULL_IMAGES:-yes}" -eq "yes" ]; then
+    invoke_tests "Tools" "Docker images"
+fi

images/linux/scripts/installers/post-deployment.sh

@@ -24,5 +24,11 @@ echo "PATH=$ENVPATH" | sudo tee -a /etc/environment
 echo "Updated /etc/environment: $(cat /etc/environment)"
 
 # Clean yarn and npm cache
-yarn cache clean
+if yarn --version > /dev/null
-npm cache clean --force
+then
+  yarn cache clean
+fi
+if npm --version
+then
+  npm cache clean --force
+fi

images/linux/scripts/tests/Apt.Tests.ps1

@@ -1,6 +1,6 @@
 Import-Module "$PSScriptRoot/../helpers/Common.Helpers.psm1"
 
-$cmd = (Get-ToolsetContent).apt.cmd_packages
+$cmd = (Get-ToolsetContent).apt.cmd_packages + (Get-ToolsetContent).apt.vital_packages
 
 Describe "Apt" {
 

images/linux/scripts/tests/Tools.Tests.ps1

@@ -79,13 +79,13 @@ Describe "Docker" {
     It "docker-credential-ecr-login" {
         "docker-credential-ecr-login -v" | Should -ReturnZeroExitCode
     }
+}
 
-    Context "docker images" {
+Describe "Docker images" {
-        $testCases = (Get-ToolsetContent).docker.images | ForEach-Object { @{ ImageName = $_ } }
+    $testCases = (Get-ToolsetContent).docker.images | ForEach-Object { @{ ImageName = $_ } }
 
-        It "<ImageName>" -TestCases $testCases {
+    It "<ImageName>" -TestCases $testCases {
-           sudo docker images "$ImageName" --format "{{.Repository}}" | Should -Not -BeNullOrEmpty
+       sudo docker images "$ImageName" --format "{{.Repository}}" | Should -Not -BeNullOrEmpty
-        }
     }
 }
 

images/linux/toolsets/toolset-2004.json

@@ -133,13 +133,24 @@
         }
     ],
     "apt": {
+        "vital_packages": [
+            "bzip2",
+            "curl",
+            "g++",
+            "gcc",
+            "make",
+            "jq",
+            "tar",
+            "unzip",
+            "wget"
+        ],
         "common_packages": [
             "autoconf",
             "automake",
-            "build-essential",
             "dbus",
             "dnsutils",
             "dpkg",
+            "dpkg-dev",
             "fakeroot",
             "fonts-noto-color-emoji",
             "gnupg2",
@@ -149,6 +160,7 @@
             "lib32z1",
             "libc++abi-dev",
             "libc++-dev",
+            "libc6-dev",
             "libcurl4",
             "libgbm-dev",
             "libgconf-2-4",
@@ -171,7 +183,6 @@
             "pkg-config",
             "python-is-python3",
             "rpm",
-            "tar",
             "texinfo",
             "tk",
             "tzdata",
@@ -187,14 +198,11 @@
             "binutils",
             "bison",
             "brotli",
-            "bzip2",
             "coreutils",
-            "curl",
             "file",
             "flex",
             "ftp",
             "haveged",
-            "jq",
             "m4",
             "mediainfo",
             "netcat",
@@ -215,8 +223,6 @@
             "swig",
             "telnet",
             "time",
-            "unzip",
-            "wget",
             "zip"
         ]
     },

images/linux/toolsets/toolset-2204.json

@@ -116,13 +116,24 @@
         }
     ],
     "apt": {
+        "vital_packages": [
+            "bzip2",
+            "curl",
+            "g++",
+            "gcc",
+            "make",
+            "jq",
+            "tar",
+            "unzip",
+            "wget"
+        ],
         "common_packages": [
             "autoconf",
             "automake",
-            "build-essential",
             "dbus",
             "dnsutils",
             "dpkg",
+            "dpkg-dev",
             "fakeroot",
             "fonts-noto-color-emoji",
             "gnupg2",
@@ -132,6 +143,7 @@
             "lib32z1",
             "libc++abi-dev",
             "libc++-dev",
+            "libc6-dev",
             "libcurl4",
             "libgbm-dev",
             "libgconf-2-4",
@@ -155,7 +167,6 @@
             "pkg-config",
             "python-is-python3",
             "rpm",
-            "tar",
             "texinfo",
             "tk",
             "tzdata",
@@ -171,14 +182,11 @@
             "binutils",
             "bison",
             "brotli",
-            "bzip2",
             "coreutils",
-            "curl",
             "file",
             "flex",
             "ftp",
             "haveged",
-            "jq",
             "lz4",
             "m4",
             "mediainfo",
@@ -200,8 +208,6 @@
             "swig",
             "telnet",
             "time",
-            "unzip",
-            "wget",
             "zip"
         ]
     },

images/linux/ubuntu2004.json

@@ -153,6 +153,18 @@
             ],
             "execute_command": "sudo sh -c '{{ .Vars }} {{ .Path }}'"
         },
+        {
+            "type": "shell",
+            "scripts": [
+                "{{template_dir}}/scripts/installers/apt-vital.sh"
+            ],
+            "environment_vars": [
+                "HELPER_SCRIPTS={{user `helper_script_folder`}}",
+                "INSTALLER_SCRIPT_FOLDER={{user `installer_script_folder`}}",
+                "DEBIAN_FRONTEND=noninteractive"
+            ],
+            "execute_command": "sudo sh -c '{{ .Vars }} {{ .Path }}'"
+        },
         {
             "type": "shell",
             "scripts": [
@@ -193,10 +205,10 @@
         {
             "type": "shell",
             "scripts": [
+                "{{template_dir}}/scripts/installers/apt-common.sh",
                 "{{template_dir}}/scripts/installers/azcopy.sh",
                 "{{template_dir}}/scripts/installers/azure-cli.sh",
                 "{{template_dir}}/scripts/installers/azure-devops-cli.sh",
-                "{{template_dir}}/scripts/installers/basic.sh",
                 "{{template_dir}}/scripts/installers/bicep.sh",
                 "{{template_dir}}/scripts/installers/aliyun-cli.sh",
                 "{{template_dir}}/scripts/installers/apache.sh",

images/linux/ubuntu2204.pkr.hcl

@@ -256,6 +256,12 @@ build {
     scripts          = ["${path.root}/scripts/installers/configure-environment.sh"]
   }
 
+  provisioner "shell" {
+    environment_vars = ["DEBIAN_FRONTEND=noninteractive", "HELPER_SCRIPTS=${local.helper_script_folder}", "INSTALLER_SCRIPT_FOLDER=${local.installer_script_folder}"]
+    execute_command  = "sudo sh -c '{{ .Vars }} {{ .Path }}'"
+    scripts          = ["${path.root}/scripts/installers/apt-vital.sh"]
+  }
+
   provisioner "shell" {
     environment_vars = ["HELPER_SCRIPTS=${var.helper_script_folder}"]
     execute_command  = "sudo sh -c '{{ .Vars }} {{ .Path }}'"
@@ -278,10 +284,10 @@ build {
     environment_vars = ["HELPER_SCRIPTS=${var.helper_script_folder}", "INSTALLER_SCRIPT_FOLDER=${var.installer_script_folder}", "DEBIAN_FRONTEND=noninteractive"]
     execute_command  = "sudo sh -c '{{ .Vars }} {{ .Path }}'"
     scripts          = [
+                        "${path.root}/scripts/installers/apt-common.sh",
                         "${path.root}/scripts/installers/azcopy.sh",
                         "${path.root}/scripts/installers/azure-cli.sh",
                         "${path.root}/scripts/installers/azure-devops-cli.sh",
-                        "${path.root}/scripts/installers/basic.sh",
                         "${path.root}/scripts/installers/bicep.sh",
                         "${path.root}/scripts/installers/aliyun-cli.sh",
                         "${path.root}/scripts/installers/apache.sh",

images/linux/ubuntuminimal.pkr.hcl

@@ -0,0 +1,304 @@
+locals {
+  image_os = "ubuntu22"
+
+  toolset_file_name = "toolset-2204.json"
+
+  image_folder            = "/imagegeneration"
+  helper_script_folder    = "/imagegeneration/helpers"
+  installer_script_folder = "/imagegeneration/installers"
+  imagedata_file          = "/imagegeneration/imagedata.json"
+}
+
+variable "allowed_inbound_ip_addresses" {
+  type    = list(string)
+  default = []
+}
+
+variable "azure_tags" {
+  type    = map(string)
+  default = {}
+}
+
+variable "build_resource_group_name" {
+  type    = string
+  default = "${env("BUILD_RESOURCE_GROUP_NAME")}"
+}
+
+variable "capture_name_prefix" {
+  type    = string
+  default = "packer"
+}
+
+variable "client_id" {
+  type    = string
+  default = "${env("ARM_CLIENT_ID")}"
+}
+
+variable "client_secret" {
+  type      = string
+  default   = "${env("ARM_CLIENT_SECRET")}"
+  sensitive = true
+}
+
+variable "client_cert_path" {
+  type      = string
+  default   = "${env("ARM_CLIENT_CERT_PATH")}"
+}
+
+variable "commit_url" {
+  type      = string
+  default   = ""
+}
+
+variable "image_version" {
+  type    = string
+  default = "dev"
+}
+
+variable "install_password" {
+  type  = string
+  default = ""
+}
+
+variable "location" {
+  type    = string
+  default = "${env("ARM_RESOURCE_LOCATION")}"
+}
+
+variable "private_virtual_network_with_public_ip" {
+  type    = bool
+  default = false
+}
+
+variable "resource_group" {
+  type    = string
+  default = "${env("ARM_RESOURCE_GROUP")}"
+}
+
+variable "run_validation_diskspace" {
+  type    = bool
+  default = false
+}
+
+variable "storage_account" {
+  type    = string
+  default = "${env("ARM_STORAGE_ACCOUNT")}"
+}
+
+variable "subscription_id" {
+  type    = string
+  default = "${env("ARM_SUBSCRIPTION_ID")}"
+}
+
+variable "temp_resource_group_name" {
+  type    = string
+  default = "${env("TEMP_RESOURCE_GROUP_NAME")}"
+}
+
+variable "tenant_id" {
+  type    = string
+  default = "${env("ARM_TENANT_ID")}"
+}
+
+variable "virtual_network_name" {
+  type    = string
+  default = "${env("VNET_NAME")}"
+}
+
+variable "virtual_network_resource_group_name" {
+  type    = string
+  default = "${env("VNET_RESOURCE_GROUP")}"
+}
+
+variable "virtual_network_subnet_name" {
+  type    = string
+  default = "${env("VNET_SUBNET")}"
+}
+
+variable "vm_size" {
+  type    = string
+  default = "Standard_D4s_v4"
+}
+
+source "azure-arm" "build_vhd" {
+  location = "${var.location}"
+
+  // Auth
+  tenant_id        = "${var.tenant_id}"
+  subscription_id  = "${var.subscription_id}"
+  client_id        = "${var.client_id}"
+  client_secret    = "${var.client_secret}"
+  client_cert_path = "${var.client_cert_path}"
+
+  // Base image
+  image_offer     = "0001-com-ubuntu-server-jammy"
+  image_publisher = "canonical"
+  image_sku       = "22_04-lts"
+
+  // Target location
+  storage_account        = "${var.storage_account}"
+  resource_group_name    = "${var.resource_group}"
+  capture_container_name = "images"
+  capture_name_prefix    = "${var.capture_name_prefix}"
+
+  // Resource group for VM
+  build_resource_group_name = "${var.build_resource_group_name}"
+  temp_resource_group_name  = "${var.temp_resource_group_name}"
+
+  // Networking for VM
+  private_virtual_network_with_public_ip = "${var.private_virtual_network_with_public_ip}"
+  virtual_network_resource_group_name    = "${var.virtual_network_resource_group_name}"
+  virtual_network_name                   = "${var.virtual_network_name}"
+  virtual_network_subnet_name            = "${var.virtual_network_subnet_name}"
+  allowed_inbound_ip_addresses           = "${var.allowed_inbound_ip_addresses}"
+  
+  // VM Configuration
+  vm_size         = "${var.vm_size}"
+  os_disk_size_gb = "86"
+  os_type         = "Linux"
+  
+  dynamic "azure_tag" {
+    for_each = var.azure_tags
+    content {
+      name = azure_tag.key
+      value = azure_tag.value
+    }
+  }
+}
+
+build {
+  sources = ["source.azure-arm.build_vhd"]
+
+  // Create folder to store temporary data
+  provisioner "shell" {
+    execute_command = "sudo sh -c '{{ .Vars }} {{ .Path }}'"
+    inline          = ["mkdir ${local.image_folder}",
+                       "chmod 777 ${local.image_folder}"]
+  }
+
+  // Add apt wrapper to implement retries
+  provisioner "shell" {
+    execute_command = "sudo sh -c '{{ .Vars }} {{ .Path }}'"
+    script          = "${path.root}/scripts/base/apt-mock.sh"
+  }
+
+  // Install MS package repos
+  provisioner "shell" {
+    environment_vars = ["DEBIAN_FRONTEND=noninteractive"]
+    execute_command  = "sudo sh -c '{{ .Vars }} {{ .Path }}'"
+    scripts          = ["${path.root}/scripts/base/repos.sh"]
+  }
+
+  // Configure apt
+  provisioner "shell" {
+    environment_vars = ["DEBIAN_FRONTEND=noninteractive"]
+    execute_command  = "sudo sh -c '{{ .Vars }} {{ .Path }}'"
+    script           = "${path.root}/scripts/base/apt.sh"
+  }
+
+  // Configure limits
+  provisioner "shell" {
+    execute_command = "sudo sh -c '{{ .Vars }} {{ .Path }}'"
+    script          = "${path.root}/scripts/base/limits.sh"
+  }
+
+  provisioner "file" {
+    destination = "${local.helper_script_folder}"
+    source      = "${path.root}/scripts/helpers"
+  }
+
+  provisioner "file" {
+    destination = "${local.installer_script_folder}"
+    source      = "${path.root}/scripts/installers"
+  }
+
+  provisioner "file" {
+    destination = "${local.image_folder}"
+    sources     = [
+      "${path.root}/post-generation",
+      "${path.root}/scripts/tests"
+    ]
+  }
+
+  provisioner "file" {
+    destination = "${local.installer_script_folder}/toolset.json"
+    source      = "${path.root}/toolsets/${local.toolset_file_name}"
+  }
+
+  // Generate image data file
+  provisioner "shell" {
+    environment_vars = ["IMAGE_VERSION=${var.image_version}", "IMAGEDATA_FILE=${local.imagedata_file}"]
+    execute_command  = "sudo sh -c '{{ .Vars }} {{ .Path }}'"
+    scripts          = ["${path.root}/scripts/installers/preimagedata.sh"]
+  }
+
+  // Create /etc/environment, configure waagent etc.
+  provisioner "shell" {
+    environment_vars = ["IMAGE_VERSION=${var.image_version}", "IMAGE_OS=${local.image_os}", "HELPER_SCRIPTS=${local.helper_script_folder}"]
+    execute_command  = "sudo sh -c '{{ .Vars }} {{ .Path }}'"
+    scripts          = ["${path.root}/scripts/installers/configure-environment.sh"]
+  }
+
+  provisioner "shell" {
+    environment_vars = ["DEBIAN_FRONTEND=noninteractive", "HELPER_SCRIPTS=${local.helper_script_folder}", "INSTALLER_SCRIPT_FOLDER=${local.installer_script_folder}"]
+    execute_command  = "sudo sh -c '{{ .Vars }} {{ .Path }}'"
+    scripts          = ["${path.root}/scripts/installers/apt-vital.sh"]
+  }
+
+  provisioner "shell" {
+    environment_vars = ["DEBIAN_FRONTEND=noninteractive", "HELPER_SCRIPTS=${local.helper_script_folder}"]
+    execute_command  = "sudo sh -c '{{ .Vars }} {{ .Path }}'"
+    scripts          = ["${path.root}/scripts/installers/powershellcore.sh"]
+  }
+
+  provisioner "shell" {
+    environment_vars = ["HELPER_SCRIPTS=${local.helper_script_folder}", "INSTALLER_SCRIPT_FOLDER=${local.installer_script_folder}"]
+    execute_command  = "sudo sh -c '{{ .Vars }} pwsh -f {{ .Path }}'"
+    scripts          = ["${path.root}/scripts/installers/Install-PowerShellModules.ps1"]
+  }
+
+  provisioner "shell" {
+    environment_vars = ["DEBIAN_FRONTEND=noninteractive", "HELPER_SCRIPTS=${local.helper_script_folder}", "INSTALLER_SCRIPT_FOLDER=${local.installer_script_folder}"]
+    execute_command  = "sudo sh -c '{{ .Vars }} {{ .Path }}'"
+    scripts          = [
+                        "${path.root}/scripts/installers/git.sh",
+                        "${path.root}/scripts/installers/github-cli.sh",
+                        "${path.root}/scripts/installers/zstd.sh"
+    ]
+  }
+
+  provisioner "shell" {
+    execute_command   = "/bin/sh -c '{{ .Vars }} {{ .Path }}'"
+    expect_disconnect = true
+    scripts           = ["${path.root}/scripts/base/reboot.sh"]
+  }
+
+  provisioner "shell" {
+    execute_command     = "sudo sh -c '{{ .Vars }} {{ .Path }}'"
+    pause_before        = "1m0s"
+    scripts             = ["${path.root}/scripts/installers/cleanup.sh"]
+    start_retry_timeout = "10m"
+  }
+
+  provisioner "shell" {
+    execute_command = "sudo sh -c '{{ .Vars }} {{ .Path }}'"
+    script          = "${path.root}/scripts/base/apt-mock-remove.sh"
+  }
+
+  provisioner "shell" {
+    environment_vars = ["HELPER_SCRIPT_FOLDER=${local.helper_script_folder}", "INSTALLER_SCRIPT_FOLDER=${local.installer_script_folder}", "IMAGE_FOLDER=${local.image_folder}"]
+    execute_command  = "sudo sh -c '{{ .Vars }} {{ .Path }}'"
+    scripts          = ["${path.root}/scripts/installers/post-deployment.sh"]
+  }
+
+  provisioner "shell" {
+    environment_vars = ["RUN_VALIDATION=${var.run_validation_diskspace}"]
+    scripts          = ["${path.root}/scripts/installers/validate-disk-space.sh"]
+  }
+
+  provisioner "shell" {
+    execute_command = "sudo sh -c '{{ .Vars }} {{ .Path }}'"
+    inline          = ["sleep 30", "/usr/sbin/waagent -force -deprovision+user && export HISTSIZE=0 && sync"]
+  }
+
+}