Proposed Pull Request Change

📄 Document Links

View on GitHub View on Microsoft Learn

--- title: VM watch Collectors Suite description: Learn more information about VM watch Collectors Suite. author: ofemifowode ms.author: ofemifowode ms.service: azure-virtual-machines ms.topic: concept-article ms.date: 02/05/2025 ms.subservice: monitoring # Customer intent: As a systems administrator, I want to implement VM watch collectors to monitor VM health metrics and logs, so that I can proactively identify issues, optimize performance, and ensure the reliability of the virtual machine environment. --- # VM watch Collectors Suite VM watch collectors are designed to gather VM health data on various resources like disk and network, by running health checks within the VM. This suite of collectors aid in identifying issues, monitoring performance trends, and optimizing resources to enhance the overall user experience. This article provides a summary of all available collectors in VM watch, along with the corresponding checks, metrics, logs, and parameter configurations. For detailed descriptions of each check, metric, and log, refer to the [VM watch overview](/azure/virtual-machines/azure-vm-watch) page. ### Prerequisites This article assumes that you're familiar with: - [VM watch checks, metrics, and logs](/azure/virtual-machines/azure-vm-watch) - [Install VM watch to virtual machines and scale sets](/azure/virtual-machines/install-vm-watch?tabs=ARM-template-1%2Ccli-2) > [!NOTE] > | **Name** | **Description** | > |---|---| > | **Collector** | Logical grouping of similar tests where you can collect checks, metrics, and logs to determine the health of a particular resource | > | **Signals** | What is emitted to reflect the health status of VMs. The three types of signals emitted are checks, metrics, and logs | > | **Group** | Indicates whether the collectors are part of the core or optional group. Core group collectors are enabled by default, while optional group collectors can be enabled or disabled based on your requirements | > | **Tags** | Used to categorize and filter checks, metrics, and logs | > | **Eligibility** | Determines whether a collector is eligible to be executed based on the environment attributes you specify | > | **Default Behavior** | Standard setting and action that would be followed if no custom configurations are provided. | > | **Overwritable Parameters** | Associated parameters that can be customized to override the default configuration | ### Groups, tags and corresponding checks, metrics, and event logs | Collector Name | Group | Tags | Checks | Metrics | Event Logs | |---|---|---|---|---|---| | outbound_connectivity| Core|Network| <ul><li>outbound_connectivity</li></ul>||| | dns| Core|Network| <ul><li>dns</li> </ul>||| | tcp_stats | Core|Network| | <ul><li>SegmentsRetransmitted</li><li>TCPSynRetransmits (Linux only)</li> <li>NormalizedSegmentsRetransmitted</li> <li>ConnectionResets</li> <li>NormalizedConnectionResets</li> <li>FailedConnectionAttempts</li> <li> NormalizedFailedConnectionAttempts </li><li> ActiveConnectionOpenings </li><li> PassiveConnectionOpenings </li><li> CurrentConnections </li><li> SegmentsReceived </li><li> SegmentsSent </li> </ul>|| | clock_skew | Core|Clock| <ul><li>clockskew</li> </ul>||| | disk_io | Core|Disk|<ul><li>disk_io</li> </ul>| <ul><li>UsedSpaceInBytes</li><li>FreeSpaceInBytes</li><li>CapacityInBytes</li><li>UsedPercent</li> </ul>|| | disk_iops | Core|Disk||<ul> <li>WriteOps</li> <li>ReadOps</li> </ul>|| | imds | Core|IMDS| <ul> <li>imds</li> </ul>||| | process | Core|Process| <ul> <li>process</li> </ul>||| | process_memory | Core|Process| |<ul> <li>ProcessRSSPercent</li> <li>ProcessPageFaults</li> <li>MachineMemoryTotalInBytes</li><li>MachineMemoryUsedPercent</li><li>TotalPageFaults</li></ul>|| | process_cpu | Core|Process| |<ul> <li>ProcessCPUCoreUsage</li> <li>ProcessCPUMachineUsage</li> <li>MachineTotalCpuUsage</li></ul>|| | process_monitor | Optional|Process|<ul> <li>process_monitor</li> </ul>|<ul> <li>UpTime</li> </ul>|| | system_error | Core|OS| |<ul><li>SystemErrors</li> </ul>|| | az_storage_blob | Optional|AzBlob| <ul> <li>az_storage_blob</li> </ul>||| | hardware_health_monitor | Optional|Hardware| | | <ul> <li>hardware_health_monitor</li> </ul>| | hardware_health_nvidia_smi | Optional|Hardware| | | <ul> <li>hardware_health_nvidia_smi</li> </ul>| ### Eligibility, default behavior, and overwritable parameters | Collector Name | Eligibility | Default Behavior | Overwritable Parameters | |---|---|---|---| | outbound_connectivity| Eligible if EnvironmentAttribute "OutboundConnectivityDisabled" isn't set or set to "false" |This collector is executed every 60s. In each execution, it sends an http GET request to `http://www.msftconnecttest.com/connecttest.txt` with a time-out of 5s. If the request fails, it retries at most two more times with and interval of 10s. The verification is marked as "Failed" if all the retries fail. | <ul> <li>OUTBOUND_CONNECTIVITY_INTERVAL: the execution interval of the Collector. Default: 60s</li> <li>OUTBOUND_CONNECTIVITY_URLS: the URLs that this Collector sends http GET requests to. URLs are provided as a string using `,` as separator. Default: `http://www.msftconnecttest.com/connecttest.txt`</li> <li>OUTBOUND_CONNECTIVITY_TIMEOUT_IN_MILLISECONDS: the http GET request time-out in milliseconds. Default: 5000</li> <li>OUTBOUND_CONNECTIVITY_TOTAL_ATTEMPTS: the total number of attempts to send http request if the previous one fails. Default: 3</li> <li>OUTBOUND_CONNECTIVITY_RETRY_INTERVAL_IN_SECONDS: the retry interval in seconds if the previous http request fails. Default: 10</li> </ul> | | dns| Eligible if EnvironmentAttribute "OutboundConnectivityDisabled" isn't set or set to "false" |This Collector is executed every 180s. In each execution, it tries to resolve the DNS name `www.msftconnecttest.com` . The verification is marked as "Failed" if the DNS name can't be resolved. | <ul> <li>DNS_INTERVAL: the execution interval of the Collector. Default: 180s</li> <li>DNS_NAMES: the domain names to be resolved separated by `,`. Default: `www.msftconnecttest.com`</li> </ul>| | tcp_stats| Always eligible |This collector is executed every 180s. In each execution, it collects the TCP statistics of the last 180s. | <ul> <li>TCP_STATS_INTERVAL: the execution interval of the Collector. Default: 180s</li>  </ul> | | clock_skew| Eligible if EnvironmentAttribute "OutboundConnectivityDisabled" isn't set or set to "false"|This collector is executed every 180s. In each execution, it retrieves the clock offset between the remote NTP server `time.windows.com` and the VM. The verification is marked as "Failed" if the clock skew is larger than 5.0 seconds. In Windows VM, if connecting to remote NTP server fails, it fallbacks to check Windows Time Service with w32tm command. The verification is marked as "Failed" if the w32tm command returns "Leap Indicator: 3(not synchronized)". | <ul> <li>CLOCK_SKEW_INTERVAL: the execution interval of the Collector. Default: 180s</li> <li>CLOCK_SKEW_NTP_SERVER: the remote NTP server used to calculate clock skew. Default: time.windows.com</li> <li>CLOCK_SKEW_TIME_SKEW_THRESHOLD_IN_SECONDS: the threshold in seconds of clock offset to mark the verification as "Failed". Default: 5.0</li> </ul> | | disk_io| Always eligible if mount points aren't specified. If mount points are explicitly specified, only eligible when data disks are attached to the VM |This collector is executed every 180s. In each execution, it verifies the disk io availability in each available mount point by creating a folder, creating a file, writing bytes to it, deleting it and delete the folder. Then it collects the disk usage info including used space, free space, total capacity and used percentage from each mount point. | <ul> <li>DISK_IO_INTERVAL: the execution interval of the Collector. Default: 180s</li> <li>DISK_IO_MOUNT_POINTS: the mount points separated by `,`. No default value</li> <li>DISK_IO_IGNORE_FS_LIST: the file system list that should be ignored separated by `,`. Default: tmpfs,devtmpfs,devfs,iso9660,overlay,aufs,squashfs,autofs</li> <li>DISK_IO_FILENAME: the name of the file used to verify the file read/write. Default: vmwatch-{timestamp}.txt </li> </ul> | | disk_iops| Always eligible |This collector is executed every 180s. In each execution, it collects the disk read and write operations per second metrics from each available disk device. | <ul> <li>DISK_IOPS_INTERVAL: the execution interval of the Collector. Default: 180s</li> <li>DISK_IOPS_DEVICES: the device names separated by `,`. No default value</li> <li>DISK_IOPS_IGNORE_DEVICE_REGEX: the regex of the device name that should be ignored. Default: loop</li> </ul> | | imds| Always eligible|This collector is executed every 180s. In each execution, it queries the IMDS endpoint `http://169.254.169.254/metadata/instance/compute` and verifies the response body contains the information (SubscriptionId, ResourceGroup, VMId, ResourceId) of the VM. The query time-out is 10s. If the query fails, it retries at most another three more times with an interval of 15s, 30s, and 45s. | <ul> <li>IMDS_INTERVAL: the execution interval of the Collector. Default: 180s</li> <li>IMDS_ENDPOINT: the URL of the IMDS endpoint. Default:`http://169.254.169.254/metadata/instance/compute`</li> <li>IMDS_TIMEOUT_IN_SECONDS: the time-out in seconds of each query. Default: 10</li> <li>IMDS_QUERY_TOTAL_ATTEMPTS: the total number of attempts to send http request if the previous one fails. Default: 4</li> <li>IMDS_RETRY_INTERVAL_IN_SEONDS: the retry interval in seconds if the previous http request fails. Default: 15, 30, 45</li> </ul> | | process| Always eligible|This collector is executed every 180s. In each execution, it creates and executes command `${SYTEM_DIR}\system32\cmd.exe /c echo hello` in Windows machine and `/bin/sh -c echo hello` in Linux machine. The time-out of process execution is 10s. | <ul> <li>PROCESS_INTERVAL: the execution interval of the Collector. Default: 180s</li> <li>PROCESS_TIMEOUT: the time-out of process execution. Default: 10s</li> </ul>| | process_memory| Always eligible|This collector is executed every 180s. In each execution, it selects the top three processes with the most memory usage and reports the ProcessRSSPercent, ProcessPageFaults, MachineMemoryTotalInBytes, MachineMemoryUsedPercent, and TotalPageFaults. | <ul> <li>PROCESS_MEMORY_INTERVAL: the execution interval of the Collector. Default: 180s</li>  </ul>| | process_cpu| Always eligible|This collector is executed every 180s. In each execution, it selects the top three processes with the most CPU usage and reports the ProcessCoreUsage, ProcessMachineUsage, and MachineTotalCpuUsage. | <ul> <li>PROCESS_CPU_INTERVAL: the execution interval of the Collector. Default: 180s</li>  </ul>| | process_monitor| Always eligible|Not executed. If explicitly enabled by the user, this collector verifies if the selected process is running and collect its running time in seconds. | <ul> <li>PROCESS_MONITOR_INTERVAL: the execution interval of the Collector. Default: 180s</li> <li>PROCESS_MONITOR_PROCESS_NAMES: the Regular Expression of process names to be monitored separated by `,`. No default value</li> </ul> | | system_error| Eligible in Windows machine|The Collector is executed every three mins. In each execution, it subscribes to the "System" channel of Windows EventLog and queries events with level defined in SystemData <=2 (including LOG_ALWAYS, Critical, Error). The measurementTarget is defined as Source_EventId of the EventLog using default Windows locale. A cap of no more than 10 different measurementTargets is applied in each collection. | <ul> <li>SYSTEM_ERROR_MEASUREMENT_TARGET_CAP: the cap of total different measurementTargets in each collection. Default: 10</li> </ul> | | az_storage_blob| Eligible if EnvironmentAttribute "OutboundConnectivityDisabled" isn't set or set to "false" |Not executed. If explicitly enabled by the user, this collector verifies if the VM can have access to the selected Azure Storage Blob by using either Managed Identity or SAS token. | <ul> <li>AZ_STORAGE_BLOB_INTERVAL: the execution interval of the Collector. Default: 180s</li> <li>AZ_STORAGE_ACCOUNT_NAME: the Azure Storage account name. No default value</li> <li>AZ_STORAGE_CONTAINER_NAME: the Azure Storage Container name. No default value</li> <li>AZ_STORAGE_BLOB_NAME: the Azure Storage Blob name. No default value</li> <li>AZ_STORAGE_BLOB_DOMAIN_NAME: the Azure Storage domain name. No default value</li> <li>AZ_STORAGE_SAS_TOKEN_BASE64: the Base64 encoded Azure Storage SAS token. No default value</li> <li>AZ_STORAGE_USE_MANAGED_IDENTITY: if the managed identity will be used for authentication. Default: false</li> <li>AZ_STORAGE_MANAGED_IDENTITY_CLIENT_ID: the managed identity client ID for authentication. No default value</li> </ul>| | hardware_health_monitor| Eligible in Windows machine|Not executed. If explicitly enabled by the user, this collector collects hardware health info from Windows event log, currently only disk related critical events are collected, including events with ID 7, 500, 504, 505, 512 and 549. | <ul> <li>HARDWARE_HEALTH_MONITOR_INTERVAL: the execution interval of the Collector. Default: 180s</li>  </ul> | | hardware_health_nvidia_smi | Eligible in Linux Ubuntu machine|Not executed. If explicitly enabled by the user, this collector collects hardware health info from Windows event log, currently only disk related critical events are collected, including events with ID 7, 500, 504, 505, 512 and 549. | <ul> <li>HARDWARE_HEALTH_NVIDIA_SMI_INTERVAL: the execution interval of the Collector. Default: 60s</li><li>HARDWARE_HEALTH_NVIDIA_SMI_INTERVAL: the time-out of running /usr/bin/nvidia-smi command. Default: 10s</li> </ul> | ### Next steps - [Configure VM watch](/azure/virtual-machines/configure-vm-watch) - [Configure Event Hubs for VM watch](/azure/virtual-machines/configure-eventhub-vm-watch) - [Install VM watch](/azure/virtual-machines/install-vm-watch?tabs=ARM-template-1%2Ccli-2) - [VM watch overview](/azure/virtual-machines/azure-vm-watch)