Document Details

Test Toolkit
Home / Scan #48 / Test Toolkit
Sad Tux - Windows bias detected
This page contains Windows bias

About This Page

This page is part of the Azure documentation. It contains code examples and configuration instructions for working with Azure services.
GitHub View on GitHub 📚 View on Microsoft Learn

Bias Analysis

Detected Bias Types
powershell_heavy
windows_tools
missing_linux_example
windows_first
Summary
The documentation is heavily PowerShell-centric, as the ARM template test toolkit is implemented as a set of PowerShell scripts. All usage examples, regardless of platform, require PowerShell (pwsh) to be installed and used. Windows instructions are presented first, and the Linux and macOS sections are nearly identical but still rely on PowerShell rather than native shell scripting. There are no examples or guidance for running the toolkit with native Linux tools (e.g., Bash, Python), and the documentation does not discuss Linux-specific patterns or troubleshooting beyond PowerShell. This creates a Windows/PowerShell bias, even though cross-platform PowerShell is available.
Recommendations
  • Clearly state early in the documentation that the toolkit is PowerShell-based and required on all platforms.
  • Provide native Linux (Bash) and macOS (zsh/bash) alternatives if possible, or explicitly note that only PowerShell is supported.
  • If feasible, offer or link to a wrapper script or Docker container for Linux users who may not want to install PowerShell.
  • Add troubleshooting tips for common Linux/macOS PowerShell issues (e.g., permissions, execution policy differences).
  • Consider developing and documenting a cross-platform CLI or containerized version of the toolkit that does not require PowerShell.
  • Reorder sections or provide a platform-neutral overview before diving into Windows-specific instructions.
  • Highlight any Linux/macOS-specific dependencies (e.g., coreutils on macOS) in a dedicated prerequisites section.
GitHub Create Pull Request

Scan History

Date Scan Status Result
2026-01-12 00:00 #99 completed Biased Biased
2026-01-11 06:20 #98 completed Biased Biased
2026-01-10 00:00 #92 completed Clean Clean
2026-01-09 00:00 #87 cancelled Clean Clean
2026-01-06 22:28 #78 completed Clean Clean
2025-12-29 18:00 #48 cancelled Biased Biased
2025-12-15 00:00 #7 completed Clean Clean
2025-12-14 05:05 #6 completed Biased Biased

Flagged Code Snippets

   Import-Module ./arm-ttk.psd1
   
Test-AzTemplate -TemplatePath $TemplateFolder -File cdn.json

   Get-ChildItem *.ps1, *.psd1, *.ps1xml, *.psm1 -Recurse | Unblock-File
   
   Import-Module .\arm-ttk.psd1
   
   Test-AzTemplate -TemplatePath \path\to\template
   
   pwsh
   
{
  "environment": {},
  "enabled": true,
  "continueOnError": true,
  "alwaysRun": false,
  "displayName": "Run Best Practices Tests",
  "timeoutInMinutes": 0,
  "condition": "succeeded()",
  "task": {
    "id": "e213ff0f-5d5c-4791-802d-52ea3e7be1f1",
    "versionSpec": "2.*",
    "definitionType": "task"
  },
  "inputs": {
    "targetType": "inline",
    "filePath": "",
    "arguments": "",
    "script": "Import-Module $(ttk.folder)/arm-ttk/arm-ttk.psd1 -Verbose\n$testOutput = @(Test-AzTemplate -TemplatePath \"$(sample.folder)\")\n$testOutput\n\nif ($testOutput | ? {$_.Errors }) {\n   exit 1 \n} else {\n    Write-Host \"##vso[task.setvariable variable=result.best.practice]$true\"\n    exit 0\n} \n",
    "errorActionPreference": "continue",
    "failOnStderr": "true",
    "ignoreLASTEXITCODE": "false",
    "pwsh": "true",
    "workingDirectory": ""
  }
}

   Get-ChildItem *.ps1, *.psd1, *.ps1xml, *.psm1 -Recurse | Unblock-File
   
   Import-Module ./arm-ttk.psd1
   
   Test-AzTemplate -TemplatePath /path/to/template
   
   pwsh
   
   Get-ChildItem *.ps1, *.psd1, *.ps1xml, *.psm1 -Recurse | Unblock-File
   
   Test-AzTemplate -TemplatePath /path/to/template
   
Test-AzTemplate -TemplatePath $TemplateFolder

Test-AzTemplate -TemplatePath $TemplateFolder -Test "Resources Should Have Location"

param(
  [Parameter(Mandatory=$true,Position=0)]
  [PSObject]
  $TemplateObject
)

# Implement test logic that evaluates parts of the template.
# Output error with: Write-Error -Message

param(
  [Parameter(Mandatory)]
  [string]
  $TemplateText
)

# Implement test logic that performs string operations.
# Output error with: Write-Error -Message

Test-AzMarketplacePackage -TemplatePath "Path to the unzipped package folder"

Validating nestedtemplates\AzDashboard.json
    [+] adminUsername Should Not Be A Literal (210 ms)
    [+] artifacts parameter (3 ms)
    [+] CommandToExecute Must Use ProtectedSettings For Secrets (201 ms)
    [+] Deployment Resources Must Not Be Debug (160 ms)
    [+] DeploymentTemplate Must Not Contain Hardcoded Url (13 ms)
    [+] Location Should Not Be Hardcoded (31 ms)
    [+] Min and Max Value Are Numbers (4 ms)
    [+] Outputs Must Not Contain Secrets (9 ms)
    [+] Password params must be secure (3 ms)
    [+] Resources Should Have Location (2 ms)
    [+] Resources Should Not Be Ambiguous (2 ms)
    [+] Secure Params In Nested Deployments (205 ms)
    [+] Secure String Parameters Cannot Have Default (3 ms)
    [+] URIs Should Be Properly Constructed (190 ms)
    [+] Variables Must Be Referenced (9 ms)
    [+] Virtual Machines Should Not Be Preview (173 ms)
    [+] VM Size Should Be A Parameter (165 ms)
Pass : 99
Fail : 3
Total: 102
Validating StartStopV2mkpl_1.0.09302021\anothertemplate.json
    [?] Parameters Must Be Referenced (86 ms)
        Unreferenced parameter: resourceGroupName
        Unreferenced parameter: location
        Unreferenced parameter: azureFunctionAppName
        Unreferenced parameter: applicationInsightsName
        Unreferenced parameter: applicationInsightsRegion

{
  "environment": {},
  "enabled": true,
  "continueOnError": false,
  "alwaysRun": false,
  "displayName": "Download TTK",
  "timeoutInMinutes": 0,
  "condition": "succeeded()",
  "task": {
    "id": "e213ff0f-5d5c-4791-802d-52ea3e7be1f1",
    "versionSpec": "2.*",
    "definitionType": "task"
  },
  "inputs": {
    "targetType": "inline",
    "filePath": "",
    "arguments": "",
    "script": "New-Item '$(ttk.folder)' -ItemType Directory\nInvoke-WebRequest -uri '$(ttk.uri)' -OutFile \"$(ttk.folder)/$(ttk.asset.filename)\" -Verbose\nGet-ChildItem '$(ttk.folder)' -Recurse\n\nWrite-Host \"Expanding files...\"\nExpand-Archive -Path '$(ttk.folder)/*.zip' -DestinationPath '$(ttk.folder)' -Verbose\n\nWrite-Host \"Expanded files found:\"\nGet-ChildItem '$(ttk.folder)' -Recurse",
    "errorActionPreference": "stop",
    "failOnStderr": "false",
    "ignoreLASTEXITCODE": "false",
    "pwsh": "true",
    "workingDirectory": ""
  }
}

- pwsh: |
   New-Item '$(ttk.folder)' -ItemType Directory
   Invoke-WebRequest -uri '$(ttk.uri)' -OutFile "$(ttk.folder)/$(ttk.asset.filename)" -Verbose
   Get-ChildItem '$(ttk.folder)' -Recurse
   
   Write-Host "Expanding files..."
   Expand-Archive -Path '$(ttk.folder)/*.zip' -DestinationPath '$(ttk.folder)' -Verbose
   
   Write-Host "Expanded files found:"
   Get-ChildItem '$(ttk.folder)' -Recurse
  displayName: 'Download TTK'

- pwsh: |
   Import-Module $(ttk.folder)/arm-ttk/arm-ttk.psd1 -Verbose
   $testOutput = @(Test-AzTemplate -TemplatePath "$(sample.folder)")
   $testOutput
   
   if ($testOutput | ? {$_.Errors }) {
      exit 1 
   } else {
       Write-Host "##vso[task.setvariable variable=result.best.practice]$true"
       exit 0
   } 
  errorActionPreference: continue
  failOnStderr: true
  displayName: 'Run Best Practices Tests'
  continueOnError: true