Proposed Pull Request Change

📄 Document Links

View on GitHub View on Microsoft Learn

--- title: How to view Azure Service Fabric entities' aggregated health description: Describes how to query, view, and evaluate Azure Service Fabric entities' aggregated health, through health queries and general queries. ms.topic: how-to ms.author: tomcassidy author: tomvcassidy ms.service: azure-service-fabric services: service-fabric ms.date: 03/22/2026 # Customer intent: As a cloud administrator, I want to retrieve and evaluate the aggregated health of Service Fabric entities, so that I can quickly identify and troubleshoot any issues within my applications and services in the cluster. --- # View Service Fabric health reports Azure Service Fabric introduces a [health model](service-fabric-health-introduction.md) with health entities on which system components and watchdogs can report local conditions that they are monitoring. The [health store](service-fabric-health-introduction.md#health-store) aggregates all health data to determine whether entities are healthy. The cluster is automatically populated with health reports sent by the system components. Read more at [Use system health reports to troubleshoot](service-fabric-understand-and-troubleshoot-with-system-health-reports.md). Service Fabric provides multiple ways to get the aggregated health of the entities: * [Service Fabric Explorer](service-fabric-visualizing-your-cluster.md) or other visualization tools * Health queries (through PowerShell, API, or REST) * General queries that return a list of entities that have health as one of the properties (through PowerShell, API, or REST) To demonstrate these options, let's use a local cluster with five nodes and the [fabric:/WordCount application](https://github.com/Azure-Samples/service-fabric-wordcount/raw/master/WordCountV1.sfpkg). The **fabric:/WordCount** application contains two default services, a stateful service of type `WordCountServiceType`, and a stateless service of type `WordCountWebServiceType`. I changed the `ApplicationManifest.xml` to require seven target replicas for the stateful service and one partition. Because there are only five nodes in the cluster, the system components report a warning on the service partition because it is below the target count. ```xml <Service Name="WordCountService"> <StatefulService ServiceTypeName="WordCountServiceType" TargetReplicaSetSize="7" MinReplicaSetSize="2"> <UniformInt64Partition PartitionCount="[WordCountService_PartitionCount]" LowKey="1" HighKey="26" /> </StatefulService> </Service> ``` ## Health in Service Fabric Explorer Service Fabric Explorer provides a visual view of the cluster. In the image below, you can see that: * The application **fabric:/WordCount** is red (in error) because it has an error event reported by **MyWatchdog** for the property **Availability**. * One of its services, **fabric:/WordCount/WordCountService** is yellow (in warning). The service is configured with seven replicas and the cluster has five nodes, so two replicas can't be placed. Although it's not shown here, the service partition is yellow because of a system report from `System.FM` saying that `Partition is below target replica or instance count`. The yellow partition triggers the yellow service. * The cluster is red because of the red application. The evaluation uses default policies from the cluster manifest and application manifest. They are strict policies and do not tolerate any failure. View of the cluster with Service Fabric Explorer: ![View of the cluster with Service Fabric Explorer.][1] [1]: ./media/service-fabric-view-entities-aggregated-health/servicefabric-explorer-cluster-health.png > [!NOTE] > Read more about [Service Fabric Explorer](service-fabric-visualizing-your-cluster.md). > > ## Health queries Service Fabric exposes health queries for each of the supported [entity types](service-fabric-health-introduction.md#health-entities-and-hierarchy). They can be accessed through the API, using methods on [FabricClient.HealthManager](/dotnet/api/system.fabric.fabricclient.healthmanager), PowerShell cmdlets, and REST. These queries return complete health information about the entity: the aggregated health state, entity health events, child health states (when applicable), unhealthy evaluations (when the entity is not healthy), and children health statistics (when applicable). > [!NOTE] > A health entity is returned when it is fully populated in the health store. The entity must be active (not deleted) and have a system report. Its parent entities on the hierarchy chain must also have system reports. If any of these conditions are not satisfied, the health queries return a [FabricException](/dotnet/api/system.fabric.fabricexception) with [FabricErrorCode](/dotnet/api/system.fabric.fabricerrorcode) `FabricHealthEntityNotFound` that shows why the entity is not returned. > > The health queries must pass in the entity identifier, which depends on the entity type. The queries accept optional health policy parameters. If no health policies are specified, the [health policies](service-fabric-health-introduction.md#health-policies) from the cluster or application manifest are used for evaluation. If the manifests don't contain a definition for health policies, the default health policies are used for evaluation. The default health policies do not tolerate any failures. The queries also accept filters for returning only partial children or events--the ones that respect the specified filters. Another filter allows excluding the children statistics. > [!NOTE] > The output filters are applied on the server side, so the message reply size is reduced. We recommended that you use the output filters to limit the data returned, rather than apply filters on the client side. > > An entity's health contains: * The aggregated health state of the entity. Computed by the health store based on entity health reports, child health states (when applicable), and health policies. Read more about [entity health evaluation](service-fabric-health-introduction.md#health-evaluation). * The health events on the entity. * The collection of health states of all children for the entities that can have children. The health states contain entity identifiers and the aggregated health state. To get complete health for a child, call the query health for the child entity type and pass in the child identifier. * The unhealthy evaluations that point to the report that triggered the state of the entity, if the entity is not healthy. The evaluations are recursive, containing the children health evaluations that triggered current health state. For example, a watchdog reported an error against a replica. The application health shows an unhealthy evaluation due to an unhealthy service; the service is unhealthy due to a partition in error; the partition is unhealthy due to a replica in error; the replica is unhealthy due to the watchdog error health report. * The health statistics for all children types of the entities that have children. For example, cluster health shows the total number of applications, services, partitions, replicas, and deployed entities in the cluster. Service health shows the total number of partitions and replicas under the specified service. ## Get cluster health Returns the health of the cluster entity and contains the health states of applications and nodes (children of the cluster). Input: * [Optional] The cluster health policy used to evaluate the nodes and the cluster events. * [Optional] The application health policy map, with the health policies used to override the application manifest policies. * [Optional] Filters for events, nodes, and applications that specify which entries are of interest and should be returned in the result (for example, errors only, or both warnings and errors). All events, nodes, and applications are used to evaluate the entity aggregated health, regardless of the filter. * [Optional] Filter to exclude health statistics. * [Optional] Filter to include fabric:/System health statistics in the health statistics. Only applicable when the health statistics are not excluded. By default, the health statistics include only statistics for user applications and not the System application. ### API To get cluster health, create a `FabricClient` and call the [GetClusterHealthAsync](/dotnet/api/system.fabric.fabricclient.healthclient.getclusterhealthasync) method on its **HealthManager**. The following call gets the cluster health: ```csharp ClusterHealth clusterHealth = await fabricClient.HealthManager.GetClusterHealthAsync(); ``` The following code gets the cluster health by using a custom cluster health policy and filters for nodes and applications. It specifies that the health statistics include the fabric:/System statistics. It creates [ClusterHealthQueryDescription](/dotnet/api/system.fabric.description.clusterhealthquerydescription), which contains the input information. ```csharp var policy = new ClusterHealthPolicy() { MaxPercentUnhealthyNodes = 20 }; var nodesFilter = new NodeHealthStatesFilter() { HealthStateFilterValue = HealthStateFilter.Error | HealthStateFilter.Warning }; var applicationsFilter = new ApplicationHealthStatesFilter() { HealthStateFilterValue = HealthStateFilter.Error }; var healthStatisticsFilter = new ClusterHealthStatisticsFilter() { ExcludeHealthStatistics = false, IncludeSystemApplicationHealthStatistics = true }; var queryDescription = new ClusterHealthQueryDescription() { HealthPolicy = policy, ApplicationsFilter = applicationsFilter, NodesFilter = nodesFilter, HealthStatisticsFilter = healthStatisticsFilter }; ClusterHealth clusterHealth = await fabricClient.HealthManager.GetClusterHealthAsync(queryDescription); ``` ### PowerShell The cmdlet to get the cluster health is [Get-ServiceFabricClusterHealth](/powershell/module/servicefabric/get-servicefabricclusterhealth). First, connect to the cluster by using the [Connect-ServiceFabricCluster](/powershell/module/servicefabric/connect-servicefabriccluster) cmdlet. The state of the cluster is five nodes, the system application, and fabric:/WordCount configured as described. The following cmdlet gets cluster health by using default health policies. The aggregated health state is warning, because the fabric:/WordCount application is in warning. Note how the unhealthy evaluations provide details on the conditions that triggered the aggregated health. ```xml PS D:\ServiceFabric> Get-ServiceFabricClusterHealth AggregatedHealthState : Warning UnhealthyEvaluations : Unhealthy applications: 100% (1/1), MaxPercentUnhealthyApplications=0%. Unhealthy application: ApplicationName='fabric:/WordCount', AggregatedHealthState='Warning'. Unhealthy services: 100% (1/1), ServiceType='WordCountServiceType', MaxPercentUnhealthyServices=0%. Unhealthy service: ServiceName='fabric:/WordCount/WordCountService', AggregatedHealthState='Warning'. Unhealthy partitions: 100% (1/1), MaxPercentUnhealthyPartitionsPerService=0%. Unhealthy partition: PartitionId='af2e3e44-a8f8-45ac-9f31-4093eb897600', AggregatedHealthState='Warning'. Unhealthy event: SourceId='System.FM', Property='State', HealthState='Warning', ConsiderWarningAsError=false. NodeHealthStates : NodeName : _Node_4 AggregatedHealthState : Ok NodeName : _Node_3 AggregatedHealthState : Ok NodeName : _Node_2 AggregatedHealthState : Ok NodeName : _Node_1 AggregatedHealthState : Ok NodeName : _Node_0 AggregatedHealthState : Ok ApplicationHealthStates : ApplicationName : fabric:/System AggregatedHealthState : Ok ApplicationName : fabric:/WordCount AggregatedHealthState : Warning HealthEvents : None HealthStatistics : Node : 5 Ok, 0 Warning, 0 Error Replica : 6 Ok, 0 Warning, 0 Error Partition : 1 Ok, 1 Warning, 0 Error Service : 1 Ok, 1 Warning, 0 Error DeployedServicePackage : 6 Ok, 0 Warning, 0 Error DeployedApplication : 5 Ok, 0 Warning, 0 Error Application : 0 Ok, 1 Warning, 0 Error ``` The following PowerShell cmdlet gets the health of the cluster by using a custom application policy. It filters results to get only applications and nodes in error or warning. As a result, no nodes are returned, as they are all healthy. Only the fabric:/WordCount application respects the applications filter. Because the custom policy specifies to consider warnings as errors for the fabric:/WordCount application, the application is evaluated as in error, and so is the cluster. ```powershell PS D:\ServiceFabric> $appHealthPolicy = New-Object -TypeName System.Fabric.Health.ApplicationHealthPolicy $appHealthPolicy.ConsiderWarningAsError = $true $appHealthPolicyMap = New-Object -TypeName System.Fabric.Health.ApplicationHealthPolicyMap $appUri1 = New-Object -TypeName System.Uri -ArgumentList "fabric:/WordCount" $appHealthPolicyMap.Add($appUri1, $appHealthPolicy) Get-ServiceFabricClusterHealth -ApplicationHealthPolicyMap $appHealthPolicyMap -ApplicationsFilter "Warning,Error" -NodesFilter "Warning,Error" -ExcludeHealthStatistics AggregatedHealthState : Error UnhealthyEvaluations : Unhealthy applications: 100% (1/1), MaxPercentUnhealthyApplications=0%. Unhealthy application: ApplicationName='fabric:/WordCount', AggregatedHealthState='Error'. Unhealthy services: 100% (1/1), ServiceType='WordCountServiceType', MaxPercentUnhealthyServices=0%. Unhealthy service: ServiceName='fabric:/WordCount/WordCountService', AggregatedHealthState='Error'. Unhealthy partitions: 100% (1/1), MaxPercentUnhealthyPartitionsPerService=0%. Unhealthy partition: PartitionId='af2e3e44-a8f8-45ac-9f31-4093eb897600', AggregatedHealthState='Error'. Unhealthy event: SourceId='System.FM', Property='State', HealthState='Warning', ConsiderWarningAsError=true. NodeHealthStates : None ApplicationHealthStates : ApplicationName : fabric:/WordCount AggregatedHealthState : Error HealthEvents : None ``` ### REST You can get cluster health with a [GET request](/rest/api/servicefabric/get-the-health-of-a-cluster) or a [POST request](/rest/api/servicefabric/get-the-health-of-a-cluster-by-using-a-health-policy) that includes health policies described in the body. ## Get node health Returns the health of a node entity and contains the health events reported on the node. Input: * [Required] The node name that identifies the node. * [Optional] The cluster health policy settings used to evaluate health. * [Optional] Filters for events that specify which entries are of interest and should be returned in the result (for example, errors only, or both warnings and errors). All events are used to evaluate the entity aggregated health, regardless of the filter. ### API To get node health through the API, create a `FabricClient` and call the [GetNodeHealthAsync](/dotnet/api/system.fabric.fabricclient.healthclient.getnodehealthasync) method on its HealthManager. The following code gets the node health for the specified node name: ```csharp NodeHealth nodeHealth = await fabricClient.HealthManager.GetNodeHealthAsync(nodeName); ``` The following code gets the node health for the specified node name and passes in events filter and custom policy through [NodeHealthQueryDescription](/dotnet/api/system.fabric.description.nodehealthquerydescription): ```csharp var queryDescription = new NodeHealthQueryDescription(nodeName) { HealthPolicy = new ClusterHealthPolicy() { ConsiderWarningAsError = true }, EventsFilter = new HealthEventsFilter() { HealthStateFilterValue = HealthStateFilter.Warning }, }; NodeHealth nodeHealth = await fabricClient.HealthManager.GetNodeHealthAsync(queryDescription); ``` ### PowerShell The cmdlet to get the node health is [Get-ServiceFabricNodeHealth](/powershell/module/servicefabric/get-servicefabricnodehealth). First, connect to the cluster by using the [Connect-ServiceFabricCluster](/powershell/module/servicefabric/connect-servicefabriccluster) cmdlet. The following cmdlet gets the node health by using default health policies: ```powershell PS D:\ServiceFabric> Get-ServiceFabricNodeHealth _Node_1 NodeName : _Node_1 AggregatedHealthState : Ok HealthEvents : SourceId : System.FM Property : State HealthState : Ok SequenceNumber : 3 SentAt : 7/13/2017 4:39:23 PM ReceivedAt : 7/13/2017 4:40:47 PM TTL : Infinite Description : Fabric node is up. RemoveWhenExpired : False IsExpired : False Transitions : Error->Ok = 7/13/2017 4:40:47 PM, LastWarning = 1/1/0001 12:00:00 AM ``` The following cmdlet gets the health of all nodes in the cluster: ```powershell PS D:\ServiceFabric> Get-ServiceFabricNode | Get-ServiceFabricNodeHealth | select NodeName, AggregatedHealthState | ft -AutoSize NodeName AggregatedHealthState -------- --------------------- _Node_4 Ok _Node_3 Ok _Node_2 Ok _Node_1 Ok _Node_0 Ok ``` ### REST You can get node health with a [GET request](/rest/api/servicefabric/get-the-health-of-a-node) or a [POST request](/rest/api/servicefabric/get-the-health-of-a-node-by-using-a-health-policy) that includes health policies described in the body. ## Get application health Returns the health of an application entity. It contains the health states of the deployed application and service children. Input: * [Required] The application name (URI) that identifies the application. * [Optional] The application health policy used to override the application manifest policies. * [Optional] Filters for events, services, and deployed applications that specify which entries are of interest and should be returned in the result (for example, errors only, or both warnings and errors). All events, services, and deployed applications are used to evaluate the entity aggregated health, regardless of the filter. * [Optional] Filter to exclude the health statistics. If not specified, the health statistics include the ok, warning, and error count for all application children: services, partitions, replicas, deployed applications, and deployed service packages. ### API To get application health, create a `FabricClient` and call the [GetApplicationHealthAsync](/dotnet/api/system.fabric.fabricclient.healthclient.getapplicationhealthasync) method on its HealthManager. The following code gets the application health for the specified application name (URI): ```csharp ApplicationHealth applicationHealth = await fabricClient.HealthManager.GetApplicationHealthAsync(applicationName); ``` The following code gets the application health for the specified application name (URI), with filters and custom policies specified via [ApplicationHealthQueryDescription](/dotnet/api/system.fabric.description.applicationhealthquerydescription). ```csharp HealthStateFilter warningAndErrors = HealthStateFilter.Error | HealthStateFilter.Warning; var serviceTypePolicy = new ServiceTypeHealthPolicy() { MaxPercentUnhealthyPartitionsPerService = 0, MaxPercentUnhealthyReplicasPerPartition = 5, MaxPercentUnhealthyServices = 0, }; var policy = new ApplicationHealthPolicy() { ConsiderWarningAsError = false, DefaultServiceTypeHealthPolicy = serviceTypePolicy, MaxPercentUnhealthyDeployedApplications = 0, }; var queryDescription = new ApplicationHealthQueryDescription(applicationName) { HealthPolicy = policy, EventsFilter = new HealthEventsFilter() { HealthStateFilterValue = warningAndErrors }, ServicesFilter = new ServiceHealthStatesFilter() { HealthStateFilterValue = warningAndErrors }, DeployedApplicationsFilter = new DeployedApplicationHealthStatesFilter() { HealthStateFilterValue = warningAndErrors }, }; ApplicationHealth applicationHealth = await fabricClient.HealthManager.GetApplicationHealthAsync(queryDescription); ``` ### PowerShell The cmdlet to get the application health is [Get-ServiceFabricApplicationHealth](/powershell/module/servicefabric/get-servicefabricapplicationhealth). First, connect to the cluster by using the [Connect-ServiceFabricCluster](/powershell/module/servicefabric/connect-servicefabriccluster) cmdlet. The following cmdlet returns the health of the **fabric:/WordCount** application: ```powershell PS D:\ServiceFabric> Get-ServiceFabricApplicationHealth fabric:/WordCount ApplicationName : fabric:/WordCount AggregatedHealthState : Warning UnhealthyEvaluations : Unhealthy services: 100% (1/1), ServiceType='WordCountServiceType', MaxPercentUnhealthyServices=0%. Unhealthy service: ServiceName='fabric:/WordCount/WordCountService', AggregatedHealthState='Warning'. Unhealthy partitions: 100% (1/1), MaxPercentUnhealthyPartitionsPerService=0%. Unhealthy partition: PartitionId='af2e3e44-a8f8-45ac-9f31-4093eb897600', AggregatedHealthState='Warning'. Unhealthy event: SourceId='System.FM', Property='State', HealthState='Warning', ConsiderWarningAsError=false. ServiceHealthStates : ServiceName : fabric:/WordCount/WordCountWebService AggregatedHealthState : Ok ServiceName : fabric:/WordCount/WordCountService AggregatedHealthState : Warning DeployedApplicationHealthStates : ApplicationName : fabric:/WordCount NodeName : _Node_4 AggregatedHealthState : Ok ApplicationName : fabric:/WordCount NodeName : _Node_3 AggregatedHealthState : Ok ApplicationName : fabric:/WordCount NodeName : _Node_0 AggregatedHealthState : Ok ApplicationName : fabric:/WordCount NodeName : _Node_2 AggregatedHealthState : Ok ApplicationName : fabric:/WordCount NodeName : _Node_1 AggregatedHealthState : Ok HealthEvents : SourceId : System.CM Property : State HealthState : Ok SequenceNumber : 282 SentAt : 7/13/2017 5:57:05 PM ReceivedAt : 7/13/2017 5:57:05 PM TTL : Infinite Description : Application has been created. RemoveWhenExpired : False IsExpired : False Transitions : Error->Ok = 7/13/2017 5:57:05 PM, LastWarning = 1/1/0001 12:00:00 AM HealthStatistics : Replica : 6 Ok, 0 Warning, 0 Error Partition : 1 Ok, 1 Warning, 0 Error Service : 1 Ok, 1 Warning, 0 Error DeployedServicePackage : 6 Ok, 0 Warning, 0 Error DeployedApplication : 5 Ok, 0 Warning, 0 Error ``` The following PowerShell cmdlet passes in custom policies. It also filters children and events. ```powershell PS D:\ServiceFabric> Get-ServiceFabricApplicationHealth -ApplicationName fabric:/WordCount -ConsiderWarningAsError $true -ServicesFilter Error -EventsFilter Error -DeployedApplicationsFilter Error -ExcludeHealthStatistics ApplicationName : fabric:/WordCount AggregatedHealthState : Error UnhealthyEvaluations : Unhealthy services: 100% (1/1), ServiceType='WordCountServiceType', MaxPercentUnhealthyServices=0%. Unhealthy service: ServiceName='fabric:/WordCount/WordCountService', AggregatedHealthState='Error'. Unhealthy partitions: 100% (1/1), MaxPercentUnhealthyPartitionsPerService=0%. Unhealthy partition: PartitionId='af2e3e44-a8f8-45ac-9f31-4093eb897600', AggregatedHealthState='Error'. Unhealthy event: SourceId='System.FM', Property='State', HealthState='Warning', ConsiderWarningAsError=true. ServiceHealthStates : ServiceName : fabric:/WordCount/WordCountService AggregatedHealthState : Error DeployedApplicationHealthStates : None HealthEvents : None ``` ### REST You can get application health with a [GET request](/rest/api/servicefabric/get-the-health-of-an-application) or a [POST request](/rest/api/servicefabric/get-the-health-of-an-application-by-using-an-application-health-policy) that includes health policies described in the body. ## Get service health Returns the health of a service entity. It contains the partition health states. Input: * [Required] The service name (URI) that identifies the service. * [Optional] The application health policy used to override the application manifest policy. * [Optional] Filters for events and partitions that specify which entries are of interest and should be returned in the result (for example, errors only, or both warnings and errors). All events and partitions are used to evaluate the entity aggregated health, regardless of the filter. * [Optional] Filter to exclude health statistics. If not specified, the health statistics show the ok, warning, and error count for all partitions and replicas of the service. ### API To get service health through the API, create a `FabricClient` and call the [GetServiceHealthAsync](/dotnet/api/system.fabric.fabricclient.healthclient.getservicehealthasync) method on its HealthManager. The following example gets the health of a service with specified service name (URI): ```csharp ServiceHealth serviceHealth = await fabricClient.HealthManager.GetServiceHealthAsync(serviceName); ``` The following code gets the service health for the specified service name (URI), specifying filters and custom policy via [ServiceHealthQueryDescription](/dotnet/api/system.fabric.description.servicehealthquerydescription): ```csharp var queryDescription = new ServiceHealthQueryDescription(serviceName) { EventsFilter = new HealthEventsFilter() { HealthStateFilterValue = HealthStateFilter.All }, PartitionsFilter = new PartitionHealthStatesFilter() { HealthStateFilterValue = HealthStateFilter.Error }, }; ServiceHealth serviceHealth = await fabricClient.HealthManager.GetServiceHealthAsync(queryDescription); ``` ### PowerShell The cmdlet to get the service health is [Get-ServiceFabricServiceHealth](/powershell/module/servicefabric/get-servicefabricservicehealth). First, connect to the cluster by using the [Connect-ServiceFabricCluster](/powershell/module/servicefabric/connect-servicefabriccluster) cmdlet. The following cmdlet gets the service health by using default health policies: ```powershell PS D:\ServiceFabric> Get-ServiceFabricServiceHealth -ServiceName fabric:/WordCount/WordCountService ServiceName : fabric:/WordCount/WordCountService AggregatedHealthState : Warning UnhealthyEvaluations : Unhealthy partitions: 100% (1/1), MaxPercentUnhealthyPartitionsPerService=0%. Unhealthy partition: PartitionId='af2e3e44-a8f8-45ac-9f31-4093eb897600', AggregatedHealthState='Warning'. Unhealthy event: SourceId='System.FM', Property='State', HealthState='Warning', ConsiderWarningAsError=false. PartitionHealthStates : PartitionId : af2e3e44-a8f8-45ac-9f31-4093eb897600 AggregatedHealthState : Warning HealthEvents : SourceId : System.FM Property : State HealthState : Ok SequenceNumber : 15 SentAt : 7/13/2017 5:57:05 PM ReceivedAt : 7/13/2017 5:57:18 PM TTL : Infinite Description : Service has been created. RemoveWhenExpired : False IsExpired : False Transitions : Error->Ok = 7/13/2017 5:57:18 PM, LastWarning = 1/1/0001 12:00:00 AM HealthStatistics : Replica : 5 Ok, 0 Warning, 0 Error Partition : 0 Ok, 1 Warning, 0 Error ``` ### REST You can get service health with a [GET request](/rest/api/servicefabric/get-the-health-of-a-service) or a [POST request](/rest/api/servicefabric/get-the-health-of-a-service-by-using-a-health-policy) that includes health policies described in the body. ## Get partition health Returns the health of a partition entity. It contains the replica health states. Input: * [Required] The partition ID (GUID) that identifies the partition. * [Optional] The application health policy used to override the application manifest policy. * [Optional] Filters for events and replicas that specify which entries are of interest and should be returned in the result (for example, errors only, or both warnings and errors). All events and replicas are used to evaluate the entity aggregated health, regardless of the filter. * [Optional] Filter to exclude health statistics. If not specified, the health statistics show how many replicas are in ok, warning, and error states. ### API To get partition health through the API, create a `FabricClient` and call the [GetPartitionHealthAsync](/dotnet/api/system.fabric.fabricclient.healthclient.getpartitionhealthasync) method on its HealthManager. To specify optional parameters, create [PartitionHealthQueryDescription](/dotnet/api/system.fabric.description.partitionhealthquerydescription). ```csharp PartitionHealth partitionHealth = await fabricClient.HealthManager.GetPartitionHealthAsync(partitionId); ``` ### PowerShell The cmdlet to get the partition health is [Get-ServiceFabricPartitionHealth](/powershell/module/servicefabric/get-servicefabricpartitionhealth). First, connect to the cluster by using the [Connect-ServiceFabricCluster](/powershell/module/servicefabric/connect-servicefabriccluster) cmdlet. The following cmdlet gets the health for all partitions of the **fabric:/WordCount/WordCountService** service and filters out replica health states: ```powershell PS D:\ServiceFabric> Get-ServiceFabricPartition fabric:/WordCount/WordCountService | Get-ServiceFabricPartitionHealth -ReplicasFilter None PartitionId : af2e3e44-a8f8-45ac-9f31-4093eb897600 AggregatedHealthState : Warning UnhealthyEvaluations : Unhealthy event: SourceId='System.FM', Property='State', HealthState='Warning', ConsiderWarningAsError=false. ReplicaHealthStates : None HealthEvents : SourceId : System.FM Property : State HealthState : Warning SequenceNumber : 72 SentAt : 7/13/2017 5:57:29 PM ReceivedAt : 7/13/2017 5:57:48 PM TTL : Infinite Description : Partition is below target replica or instance count. fabric:/WordCount/WordCountService 7 2 af2e3e44-a8f8-45ac-9f31-4093eb897600 N/P RD _Node_2 Up 131444422260002646 N/S RD _Node_4 Up 131444422293113678 N/S RD _Node_3 Up 131444422293113679 N/S RD _Node_1 Up 131444422293118720 N/S RD _Node_0 Up 131444422293118721 (Showing 5 out of 5 replicas. Total available replicas: 5.) RemoveWhenExpired : False IsExpired : False Transitions : Ok->Warning = 7/13/2017 5:57:48 PM, LastError = 1/1/0001 12:00:00 AM SourceId : System.PLB Property : ServiceReplicaUnplacedHealth_Secondary_af2e3e44-a8f8-45ac-9f31-4093eb897600 HealthState : Warning SequenceNumber : 131444445174851664 SentAt : 7/13/2017 6:35:17 PM ReceivedAt : 7/13/2017 6:35:18 PM TTL : 00:01:05 Description : The Load Balancer was unable to find a placement for one or more of the Service's Replicas: Secondary replica could not be placed due to the following constraints and properties: TargetReplicaSetSize: 7 Placement Constraint: N/A Parent Service: N/A Constraint Elimination Sequence: Existing Secondary Replicas eliminated 4 possible node(s) for placement -- 1/5 node(s) remain. Existing Primary Replica eliminated 1 possible node(s) for placement -- 0/5 node(s) remain. Nodes Eliminated By Constraints: Existing Secondary Replicas -- Nodes with Partition's Existing Secondary Replicas/Instances: -- FaultDomain:fd:/4 NodeName:_Node_4 NodeType:NodeType4 UpgradeDomain:4 UpgradeDomain: ud:/4 Deactivation Intent/Status: None/None FaultDomain:fd:/3 NodeName:_Node_3 NodeType:NodeType3 UpgradeDomain:3 UpgradeDomain: ud:/3 Deactivation Intent/Status: None/None FaultDomain:fd:/1 NodeName:_Node_1 NodeType:NodeType1 UpgradeDomain:1 UpgradeDomain: ud:/1 Deactivation Intent/Status: None/None FaultDomain:fd:/0 NodeName:_Node_0 NodeType:NodeType0 UpgradeDomain:0 UpgradeDomain: ud:/0 Deactivation Intent/Status: None/None Existing Primary Replica -- Nodes with Partition's Existing Primary Replica or Secondary Replicas: -- FaultDomain:fd:/2 NodeName:_Node_2 NodeType:NodeType2 UpgradeDomain:2 UpgradeDomain: ud:/2 Deactivation Intent/Status: None/None RemoveWhenExpired : True IsExpired : False Transitions : Error->Warning = 7/13/2017 5:57:48 PM, LastOk = 1/1/0001 12:00:00 AM HealthStatistics : Replica : 5 Ok, 0 Warning, 0 Error ``` ### REST You can get partition health with a [GET request](/rest/api/servicefabric/sfclient-api-getpartitionhealth) or a [POST request](/rest/api/servicefabric/get-the-health-of-a-partition-by-using-a-health-policy) that includes health policies described in the body. ## Get replica health Returns the health of a stateful service replica or a stateless service instance. Input: * [Required] The partition ID (GUID) and replica ID that identifies the replica. * [Optional] The application health policy parameters used to override the application manifest policies. * [Optional] Filters for events that specify which entries are of interest and should be returned in the result (for example, errors only, or both warnings and errors). All events are used to evaluate the entity aggregated health, regardless of the filter. ### API To get the replica health through the API, create a `FabricClient` and call the [GetReplicaHealthAsync](/dotnet/api/system.fabric.fabricclient.healthclient.getreplicahealthasync) method on its HealthManager. To specify advanced parameters, use [ReplicaHealthQueryDescription](/dotnet/api/system.fabric.description.replicahealthquerydescription). ```csharp ReplicaHealth replicaHealth = await fabricClient.HealthManager.GetReplicaHealthAsync(partitionId, replicaId); ``` ### PowerShell The cmdlet to get the replica health is [Get-ServiceFabricReplicaHealth](/powershell/module/servicefabric/get-servicefabricreplicahealth). First, connect to the cluster by using the [Connect-ServiceFabricCluster](/powershell/module/servicefabric/connect-servicefabriccluster) cmdlet. The following cmdlet gets the health of the primary replica for all partitions of the service: ```powershell PS D:\ServiceFabric> Get-ServiceFabricPartition fabric:/WordCount/WordCountService | Get-ServiceFabricReplica | where {$_.ReplicaRole -eq "Primary"} | Get-ServiceFabricReplicaHealth PartitionId : af2e3e44-a8f8-45ac-9f31-4093eb897600 ReplicaId : 131444422260002646 AggregatedHealthState : Ok HealthEvents : SourceId : System.RA Property : State HealthState : Ok SequenceNumber : 131444422263668344 SentAt : 7/13/2017 5:57:06 PM ReceivedAt : 7/13/2017 5:57:18 PM TTL : Infinite Description : Replica has been created._Node_2 RemoveWhenExpired : False IsExpired : False Transitions : Error->Ok = 7/13/2017 5:57:18 PM, LastWarning = 1/1/0001 12:00:00 AM ``` ### REST You can get replica health with a [GET request](/rest/api/servicefabric/get-the-health-of-a-replica) or a [POST request](/rest/api/servicefabric/get-the-health-of-a-replica-by-using-a-health-policy) that includes health policies described in the body. ## Get deployed application health Returns the health of an application deployed on a node entity. It contains the deployed service package health states. Input: * [Required] The application name (URI) and node name (string) that identify the deployed application. * [Optional] The application health policy used to override the application manifest policies. * [Optional] Filters for events and deployed service packages that specify which entries are of interest and should be returned in the result (for example, errors only, or both warnings and errors). All events and deployed service packages are used to evaluate the entity aggregated health, regardless of the filter. * [Optional] Filter to exclude health statistics. If not specified, the health statistics show the number of deployed service packages in ok, warning, and error health states. ### API To get the health of an application deployed on a node through the API, create a `FabricClient` and call the [GetDeployedApplicationHealthAsync](/dotnet/api/system.fabric.fabricclient.healthclient.getdeployedapplicationhealthasync) method on its HealthManager. To specify optional parameters, use [DeployedApplicationHealthQueryDescription](/dotnet/api/system.fabric.description.deployedapplicationhealthquerydescription). ```csharp DeployedApplicationHealth health = await fabricClient.HealthManager.GetDeployedApplicationHealthAsync( new DeployedApplicationHealthQueryDescription(applicationName, nodeName)); ``` ### PowerShell The cmdlet to get the deployed application health is [Get-ServiceFabricDeployedApplicationHealth](/powershell/module/servicefabric/get-servicefabricdeployedapplicationhealth). First, connect to the cluster by using the [Connect-ServiceFabricCluster](/powershell/module/servicefabric/connect-servicefabriccluster) cmdlet. To find out where an application is deployed, run [Get-ServiceFabricApplicationHealth](/powershell/module/servicefabric/get-servicefabricapplicationhealth) and look at the deployed application children. The following cmdlet gets the health of the **fabric:/WordCount** application deployed on **_Node_2**. ```powershell PS D:\ServiceFabric> Get-ServiceFabricDeployedApplicationHealth -ApplicationName fabric:/WordCount -NodeName _Node_0 ApplicationName : fabric:/WordCount NodeName : _Node_0 AggregatedHealthState : Ok DeployedServicePackageHealthStates : ServiceManifestName : WordCountServicePkg ServicePackageActivationId : NodeName : _Node_0 AggregatedHealthState : Ok ServiceManifestName : WordCountWebServicePkg ServicePackageActivationId : NodeName : _Node_0 AggregatedHealthState : Ok HealthEvents : SourceId : System.Hosting Property : Activation HealthState : Ok SequenceNumber : 131444422261848308 SentAt : 7/13/2017 5:57:06 PM ReceivedAt : 7/13/2017 5:57:17 PM TTL : Infinite Description : The application was activated successfully. RemoveWhenExpired : False IsExpired : False Transitions : Error->Ok = 7/13/2017 5:57:17 PM, LastWarning = 1/1/0001 12:00:00 AM HealthStatistics : DeployedServicePackage : 2 Ok, 0 Warning, 0 Error ``` ### REST You can get deployed application health with a [GET request](/rest/api/servicefabric/get-the-health-of-a-deployed-application) or a [POST request](/rest/api/servicefabric/get-the-health-of-a-deployed-application-by-using-a-health-policy) that includes health policies described in the body. ## Get deployed service package health Returns the health of a deployed service package entity. Input: * [Required] The application name (URI), node name (string), and service manifest name (string) that identify the deployed service package. * [Optional] The application health policy used to override the application manifest policy. * [Optional] Filters for events that specify which entries are of interest and should be returned in the result (for example, errors only, or both warnings and errors). All events are used to evaluate the entity aggregated health, regardless of the filter. ### API To get the health of a deployed service package through the API, create a `FabricClient` and call the [GetDeployedServicePackageHealthAsync](/dotnet/api/system.fabric.fabricclient.healthclient.getdeployedservicepackagehealthasync) method on its HealthManager. To specify optional parameters, use [DeployedServicePackageHealthQueryDescription](/dotnet/api/system.fabric.description.deployedservicepackagehealthquerydescription). ```csharp DeployedServicePackageHealth health = await fabricClient.HealthManager.GetDeployedServicePackageHealthAsync( new DeployedServicePackageHealthQueryDescription(applicationName, nodeName, serviceManifestName)); ``` ### PowerShell The cmdlet to get the deployed service package health is [Get-ServiceFabricDeployedServicePackageHealth](/powershell/module/servicefabric/get-servicefabricdeployedservicepackagehealth). First, connect to the cluster by using the [Connect-ServiceFabricCluster](/powershell/module/servicefabric/connect-servicefabriccluster) cmdlet. To see where an application is deployed, run [Get-ServiceFabricApplicationHealth](/powershell/module/servicefabric/get-servicefabricapplicationhealth) and look at the deployed applications. To see which service packages are in an application, look at the deployed service package children in the [Get-ServiceFabricDeployedApplicationHealth](/powershell/module/servicefabric/get-servicefabricdeployedapplicationhealth) output. The following cmdlet gets the health of the **WordCountServicePkg** service package of the **fabric:/WordCount** application deployed on **_Node_2**. The entity has **System.Hosting** reports for successful service-package and entry-point activation, and successful service-type registration. ```powershell PS D:\ServiceFabric> Get-ServiceFabricDeployedApplication -ApplicationName fabric:/WordCount -NodeName _Node_2 | Get-ServiceFabricDeployedServicePackageHealth -ServiceManifestName WordCountServicePkg ApplicationName : fabric:/WordCount ServiceManifestName : WordCountServicePkg ServicePackageActivationId : NodeName : _Node_2 AggregatedHealthState : Ok HealthEvents : SourceId : System.Hosting Property : Activation HealthState : Ok SequenceNumber : 131444422267693359 SentAt : 7/13/2017 5:57:06 PM ReceivedAt : 7/13/2017 5:57:18 PM TTL : Infinite Description : The ServicePackage was activated successfully. RemoveWhenExpired : False IsExpired : False Transitions : Error->Ok = 7/13/2017 5:57:18 PM, LastWarning = 1/1/0001 12:00:00 AM SourceId : System.Hosting Property : CodePackageActivation:Code:EntryPoint HealthState : Ok SequenceNumber : 131444422267903345 SentAt : 7/13/2017 5:57:06 PM ReceivedAt : 7/13/2017 5:57:18 PM TTL : Infinite Description : The CodePackage was activated successfully. RemoveWhenExpired : False IsExpired : False Transitions : Error->Ok = 7/13/2017 5:57:18 PM, LastWarning = 1/1/0001 12:00:00 AM SourceId : System.Hosting Property : ServiceTypeRegistration:WordCountServiceType HealthState : Ok SequenceNumber : 131444422272458374 SentAt : 7/13/2017 5:57:07 PM ReceivedAt : 7/13/2017 5:57:18 PM TTL : Infinite Description : The ServiceType was registered successfully. RemoveWhenExpired : False IsExpired : False Transitions : Error->Ok = 7/13/2017 5:57:18 PM, LastWarning = 1/1/0001 12:00:00 AM ``` ### REST You can get deployed service package health with a [GET request](/rest/api/servicefabric/get-the-health-of-a-service-package) or a [POST request](/rest/api/servicefabric/get-the-health-of-a-service-package-by-using-a-health-policy) that includes health policies described in the body. ## Health chunk queries The health chunk queries can return multi-level cluster children (recursively), per input filters. It supports advanced filters that allow a lot of flexibility in choosing the children to be returned. The filters can specify children by the unique identifier or by other group identifiers and/or health states. By default, no children are included, as opposed to health commands that always include first-level children. The [health queries](service-fabric-view-entities-aggregated-health.md#health-queries) return only first-level children of the specified entity per required filters. To get the children of the children, you must call additional health APIs for each entity of interest. Similarly, to get the health of specific entities, you must call one health API for each desired entity. The chunk query advanced filtering allows you to request multiple items of interest in one query, minimizing the message size and the number of messages. The value of the chunk query is that you can get health state for more cluster entities (potentially all cluster entities starting at required root) in one call. You can express complex health query such as: * Return only applications in error, and for those applications include all services in warning or error. For returned services, include all partitions. * Return only the health of four applications, specified by their names. * Return only the health of applications of a desired application type. * Return all deployed entities on a node. Returns all applications, all deployed applications on the specified node and all the deployed service packages on that node. * Return all replicas in error. Returns all applications, services, partitions, and only replicas in error. * Return all applications. For a specified service, include all partitions. Currently, the health chunk query is exposed only for the cluster entity. It returns a cluster health chunk, which contains: * The cluster aggregated health state. * The health state chunk list of nodes that respect input filters. * The health state chunk list of applications that respect input filters. Each application health state chunk contains a chunk list with all services that respect input filters and a chunk list with all deployed applications that respect the filters. Same for the children of services and deployed applications. This way, all entities in the cluster can be potentially returned if requested, in a hierarchical fashion. ### Cluster health chunk query Returns the health of the cluster entity and contains the hierarchical health state chunks of required children. Input: * [Optional] The cluster health policy used to evaluate the nodes and the cluster events. * [Optional] The application health policy map, with the health policies used to override the application manifest policies. * [Optional] Filters for nodes and applications that specify which entries are of interest and should be returned in the result. The filters are specific to an entity/group of entities or are applicable to all entities at that level. The list of filters can contain one general filter and/or filters for specific identifiers to fine-grain entities returned by the query. If empty, the children are not returned by default. Read more about the filters at [NodeHealthStateFilter](/dotnet/api/system.fabric.health.nodehealthstatefilter) and [ApplicationHealthStateFilter](/dotnet/api/system.fabric.health.applicationhealthstatefilter). The application filters can recursively specify advanced filters for children. The chunk result includes the children that respect the filters. Currently, the chunk query does not return unhealthy evaluations or entity events. That extra information can be obtained using the existing cluster health query. ### API To get cluster health chunk, create a `FabricClient` and call the [GetClusterHealthChunkAsync](/dotnet/api/system.fabric.fabricclient.healthclient.getclusterhealthchunkasync) method on its **HealthManager**. You can pass in [ClusterHealthQueryDescription](/dotnet/api/system.fabric.description.clusterhealthchunkquerydescription) to describe health policies and advanced filters. The following code gets cluster health chunk with advanced filters. ```csharp var queryDescription = new ClusterHealthChunkQueryDescription(); queryDescription.ApplicationFilters.Add(new ApplicationHealthStateFilter() { // Return applications only if they are in error HealthStateFilter = HealthStateFilter.Error }); // Return all replicas var wordCountServiceReplicaFilter = new ReplicaHealthStateFilter() { HealthStateFilter = HealthStateFilter.All }; // Return all replicas and all partitions var wordCountServicePartitionFilter = new PartitionHealthStateFilter() { HealthStateFilter = HealthStateFilter.All }; wordCountServicePartitionFilter.ReplicaFilters.Add(wordCountServiceReplicaFilter); // For specific service, return all partitions and all replicas var wordCountServiceFilter = new ServiceHealthStateFilter() { ServiceNameFilter = new Uri("fabric:/WordCount/WordCountService"), }; wordCountServiceFilter.PartitionFilters.Add(wordCountServicePartitionFilter); // Application filter: for specific application, return no services except the ones of interest var wordCountApplicationFilter = new ApplicationHealthStateFilter() { // Always return fabric:/WordCount application ApplicationNameFilter = new Uri("fabric:/WordCount"), }; wordCountApplicationFilter.ServiceFilters.Add(wordCountServiceFilter); queryDescription.ApplicationFilters.Add(wordCountApplicationFilter); var result = await fabricClient.HealthManager.GetClusterHealthChunkAsync(queryDescription); ``` ### PowerShell The cmdlet to get the cluster health is [Get-ServiceFabricClusterChunkHealth](/powershell/module/servicefabric/get-servicefabricclusterhealthchunk). First, connect to the cluster by using the [Connect-ServiceFabricCluster](/powershell/module/servicefabric/connect-servicefabriccluster) cmdlet. The following code gets nodes only if they are in Error except for a specific node, which should always be returned. ```xml PS D:\ServiceFabric> $errorFilter = [System.Fabric.Health.HealthStateFilter]::Error; $allFilter = [System.Fabric.Health.HealthStateFilter]::All; $nodeFilter1 = New-Object System.Fabric.Health.NodeHealthStateFilter -Property @{HealthStateFilter=$errorFilter} $nodeFilter2 = New-Object System.Fabric.Health.NodeHealthStateFilter -Property @{NodeNameFilter="_Node_1";HealthStateFilter=$allFilter} # Create node filter list that will be passed in the cmdlet $nodeFilters = New-Object System.Collections.Generic.List[System.Fabric.Health.NodeHealthStateFilter] $nodeFilters.Add($nodeFilter1) $nodeFilters.Add($nodeFilter2) Get-ServiceFabricClusterHealthChunk -NodeFilters $nodeFilters HealthState : Warning NodeHealthStateChunks : TotalCount : 1 NodeName : _Node_1 HealthState : Ok ApplicationHealthStateChunks : None ``` The following cmdlet gets cluster chunk with application filters. ```xml PS D:\ServiceFabric> $errorFilter = [System.Fabric.Health.HealthStateFilter]::Error; $allFilter = [System.Fabric.Health.HealthStateFilter]::All; # All replicas $replicaFilter = New-Object System.Fabric.Health.ReplicaHealthStateFilter -Property @{HealthStateFilter=$allFilter} # All partitions $partitionFilter = New-Object System.Fabric.Health.PartitionHealthStateFilter -Property @{HealthStateFilter=$allFilter} $partitionFilter.ReplicaFilters.Add($replicaFilter) # For WordCountService, return all partitions and all replicas $svcFilter1 = New-Object System.Fabric.Health.ServiceHealthStateFilter -Property @{ServiceNameFilter="fabric:/WordCount/WordCountService"} $svcFilter1.PartitionFilters.Add($partitionFilter) $svcFilter2 = New-Object System.Fabric.Health.ServiceHealthStateFilter -Property @{HealthStateFilter=$errorFilter} $appFilter = New-Object System.Fabric.Health.ApplicationHealthStateFilter -Property @{ApplicationNameFilter="fabric:/WordCount"} $appFilter.ServiceFilters.Add($svcFilter1) $appFilter.ServiceFilters.Add($svcFilter2) $appFilters = New-Object System.Collections.Generic.List[System.Fabric.Health.ApplicationHealthStateFilter] $appFilters.Add($appFilter) Get-ServiceFabricClusterHealthChunk -ApplicationFilters $appFilters HealthState : Error NodeHealthStateChunks : None ApplicationHealthStateChunks : TotalCount : 1 ApplicationName : fabric:/WordCount ApplicationTypeName : WordCount HealthState : Error ServiceHealthStateChunks : TotalCount : 1 ServiceName : fabric:/WordCount/WordCountService HealthState : Error PartitionHealthStateChunks : TotalCount : 1 PartitionId : af2e3e44-a8f8-45ac-9f31-4093eb897600 HealthState : Error ReplicaHealthStateChunks : TotalCount : 5 ReplicaOrInstanceId : 131444422293118720 HealthState : Ok ReplicaOrInstanceId : 131444422293118721 HealthState : Ok ReplicaOrInstanceId : 131444422293113678 HealthState : Ok ReplicaOrInstanceId : 131444422293113679 HealthState : Ok ReplicaOrInstanceId : 131444422260002646 HealthState : Error ``` The following cmdlet returns all deployed entities on a node. ```xml PS D:\ServiceFabric> $errorFilter = [System.Fabric.Health.HealthStateFilter]::Error; $allFilter = [System.Fabric.Health.HealthStateFilter]::All; $dspFilter = New-Object System.Fabric.Health.DeployedServicePackageHealthStateFilter -Property @{HealthStateFilter=$allFilter} $daFilter = New-Object System.Fabric.Health.DeployedApplicationHealthStateFilter -Property @{HealthStateFilter=$allFilter;NodeNameFilter="_Node_2"} $daFilter.DeployedServicePackageFilters.Add($dspFilter) $appFilter = New-Object System.Fabric.Health.ApplicationHealthStateFilter -Property @{HealthStateFilter=$allFilter} $appFilter.DeployedApplicationFilters.Add($daFilter) $appFilters = New-Object System.Collections.Generic.List[System.Fabric.Health.ApplicationHealthStateFilter] $appFilters.Add($appFilter) Get-ServiceFabricClusterHealthChunk -ApplicationFilters $appFilters HealthState : Error NodeHealthStateChunks : None ApplicationHealthStateChunks : TotalCount : 2 ApplicationName : fabric:/System HealthState : Ok DeployedApplicationHealthStateChunks : TotalCount : 1 NodeName : _Node_2 HealthState : Ok DeployedServicePackageHealthStateChunks : TotalCount : 1 ServiceManifestName : FAS ServicePackageActivationId : HealthState : Ok ApplicationName : fabric:/WordCount ApplicationTypeName : WordCount HealthState : Error DeployedApplicationHealthStateChunks : TotalCount : 1 NodeName : _Node_2 HealthState : Ok DeployedServicePackageHealthStateChunks : TotalCount : 1 ServiceManifestName : WordCountServicePkg ServicePackageActivationId : HealthState : Ok ``` ### REST You can get cluster health chunk with a [GET request](/rest/api/servicefabric/get-the-health-of-a-cluster-using-health-chunks) or a [POST request](/rest/api/servicefabric/health-of-cluster) that includes health policies and advanced filters described in the body. ## General queries General queries return a list of Service Fabric entities of a specified type. They are exposed through the API (via the methods on **FabricClient.QueryManager**), PowerShell cmdlets, and REST. These queries aggregate subqueries from multiple components. One of them is the [health store](service-fabric-health-introduction.md#health-store), which populates the aggregated health state for each query result. > [!NOTE] > General queries return the aggregated health state of the entity and do not contain rich health data. If an entity is not healthy, you can follow up with health queries to get all its health information, including events, child health states, and unhealthy evaluations. > > If general queries return an unknown health state for an entity, it's possible that the health store doesn't have complete data about the entity. It's also possible that a subquery to the health store wasn't successful (for example, there was a communication error, or the health store was throttled). Follow up with a health query for the entity. If the subquery encountered transient errors, such as network issues, this follow-up query may succeed. It may also give you more details from the health store about why the entity is not exposed. The queries that contain **HealthState** for entities are: * Node list: Returns the list nodes in the cluster (paged). * API: [FabricClient.QueryClient.GetNodeListAsync](/dotnet/api/system.fabric.fabricclient.queryclient.getnodelistasync) * PowerShell: Get-ServiceFabricNode * Application list: Returns the list of applications in the cluster (paged). * API: [FabricClient.QueryClient.GetApplicationListAsync](/dotnet/api/system.fabric.fabricclient.queryclient.getapplicationlistasync) * PowerShell: Get-ServiceFabricApplication * Service list: Returns the list of services in an application (paged). * API: [FabricClient.QueryClient.GetServiceListAsync](/dotnet/api/system.fabric.fabricclient.queryclient.getservicelistasync) * PowerShell: Get-ServiceFabricService * Partition list: Returns the list of partitions in a service (paged). * API: [FabricClient.QueryClient.GetPartitionListAsync](/dotnet/api/system.fabric.fabricclient.queryclient.getpartitionlistasync) * PowerShell: Get-ServiceFabricPartition * Replica list: Returns the list of replicas in a partition (paged). * API: [FabricClient.QueryClient.GetReplicaListAsync](/dotnet/api/system.fabric.fabricclient.queryclient.getreplicalistasync) * PowerShell: Get-ServiceFabricReplica * Deployed application list: Returns the list of deployed applications on a node. * API: [FabricClient.QueryClient.GetDeployedApplicationListAsync](/dotnet/api/system.fabric.fabricclient.queryclient.getdeployedapplicationlistasync) * PowerShell: Get-ServiceFabricDeployedApplication * Deployed service package list: Returns the list of service packages in a deployed application. * API: [FabricClient.QueryClient.GetDeployedServicePackageListAsync](/dotnet/api/system.fabric.fabricclient.queryclient.getdeployedservicepackagelistasync) * PowerShell: Get-ServiceFabricDeployedApplication > [!NOTE] > Some of the queries return paged results. The return of these queries is a list derived from [PagedList\<T>](/dotnet/api/system.fabric.query.pagedlist-1). If the results do not fit a message, only a page is returned and a ContinuationToken that tracks where enumeration stopped. Continue to call the same query and pass in the continuation token from the previous query to get next results. ### Examples The following code gets the unhealthy applications in the cluster: ```csharp var applications = fabricClient.QueryManager.GetApplicationListAsync().Result.Where( app => app.HealthState == HealthState.Error); ``` The following cmdlet gets the application details for the fabric:/WordCount application. Notice that health state is at warning. ```powershell PS C:\> Get-ServiceFabricApplication -ApplicationName fabric:/WordCount ApplicationName : fabric:/WordCount ApplicationTypeName : WordCount ApplicationTypeVersion : 1.0.0 ApplicationStatus : Ready HealthState : Warning ApplicationParameters : { "WordCountWebService_InstanceCount" = "1"; "_WFDebugParams_" = "[{"ServiceManifestName":"WordCountWebServicePkg","CodePackageName":"Code","EntryPointType":"Main","Debug ExePath":"C:\\Program Files (x86)\\Microsoft Visual Studio 14.0\\Common7\\Packages\\Debugger\\VsDebugLaunchNotify.exe","DebugArguments":" {74f7e5d5-71a9-47e2-a8cd-1878ec4734f1} -p [ProcessId] -tid [ThreadId]","EnvironmentBlock":"_NO_DEBUG_HEAP=1\u0000"},{"ServiceManifestName":"WordCountServicePkg","CodeP ackageName":"Code","EntryPointType":"Main","DebugExePath":"C:\\Program Files (x86)\\Microsoft Visual Studio 14.0\\Common7\\Packages\\Debugger\\VsDebugLaunchNotify.exe","DebugArguments":" {2ab462e6-e0d1-4fda-a844-972f561fe751} -p [ProcessId] -tid [ThreadId]","EnvironmentBlock":"_NO_DEBUG_HEAP=1\u0000"}]" } ``` The following cmdlet gets the services with a health state of error: ```powershell PS D:\ServiceFabric> Get-ServiceFabricApplication | Get-ServiceFabricService | where {$_.HealthState -eq "Error"} ServiceName : fabric:/WordCount/WordCountService ServiceKind : Stateful ServiceTypeName : WordCountServiceType IsServiceGroup : False ServiceManifestVersion : 1.0.0 HasPersistedState : True ServiceStatus : Active HealthState : Error ``` ## Cluster and application upgrades During a monitored upgrade of the cluster and application, Service Fabric checks health to ensure that everything remains healthy. If an entity is unhealthy as evaluated by using configured health policies, the upgrade applies upgrade-specific policies to determine the next action. The upgrade may be paused to allow user interaction (such as fixing error conditions or changing policies), or it may automatically roll back to the previous good version. During a *cluster* upgrade, you can get the cluster upgrade status. The upgrade status includes unhealthy evaluations, which point to what is unhealthy in the cluster. If the upgrade is rolled back due to health issues, the upgrade status remembers the last unhealthy reasons. This information can help administrators investigate what went wrong after the upgrade rolled back or stopped. Similarly, during an *application* upgrade, any unhealthy evaluations are contained in the application upgrade status. The following shows the application upgrade status for a modified fabric:/WordCount application. A watchdog reported an error on one of its replicas. The upgrade is rolling back because the health checks are not respected. ```powershell PS C:\> Get-ServiceFabricApplicationUpgrade fabric:/WordCount ApplicationName : fabric:/WordCount ApplicationTypeName : WordCount TargetApplicationTypeVersion : 1.0.0.0 ApplicationParameters : {} StartTimestampUtc : 4/21/2017 5:23:26 PM FailureTimestampUtc : 4/21/2017 5:23:37 PM FailureReason : HealthCheck UpgradeState : RollingBackInProgress UpgradeDuration : 00:00:23 CurrentUpgradeDomainDuration : 00:00:00 CurrentUpgradeDomainProgress : UD1 NodeName : _Node_1 UpgradePhase : Upgrading NodeName : _Node_2 UpgradePhase : Upgrading NodeName : _Node_3 UpgradePhase : PreUpgradeSafetyCheck PendingSafetyChecks : EnsurePartitionQuorum - PartitionId: 30db5be6-4e20-4698-8185-4bd7ca744020 NextUpgradeDomain : UD2 UpgradeDomainsStatus : { "UD1" = "Completed"; "UD2" = "Pending"; "UD3" = "Pending"; "UD4" = "Pending" } UnhealthyEvaluations : Unhealthy services: 100% (1/1), ServiceType='WordCountServiceType', MaxPercentUnhealthyServices=0%. Unhealthy service: ServiceName='fabric:/WordCount/WordCountService', AggregatedHealthState='Error'. Unhealthy partitions: 100% (1/1), MaxPercentUnhealthyPartitionsPerService=0%. Unhealthy partition: PartitionId='a1f83a35-d6bf-4d39-b90d-28d15f39599b', AggregatedHealthState='Error'. Unhealthy replicas: 20% (1/5), MaxPercentUnhealthyReplicasPerPartition=0%. Unhealthy replica: PartitionId='a1f83a35-d6bf-4d39-b90d-28d15f39599b', ReplicaOrInstanceId='131031502346844058', AggregatedHealthState='Error'. Error event: SourceId='DiskWatcher', Property='Disk'. UpgradeKind : Rolling RollingUpgradeMode : UnmonitoredAuto ForceRestart : False UpgradeReplicaSetCheckTimeout : 00:15:00 ``` Read more about the [Service Fabric application upgrade](service-fabric-application-upgrade.md). ## Use health evaluations to troubleshoot Whenever there is an issue with the cluster or an application, look at the cluster or application health to pinpoint what is wrong. The unhealthy evaluations provide details about what triggered the current unhealthy state. If you need to, you can drill down into unhealthy child entities to identify the root cause. For example, consider an application unhealthy because there is an error report on one of its replicas. The following PowerShell cmdlet shows the unhealthy evaluations: ```powershell PS D:\ServiceFabric> Get-ServiceFabricApplicationHealth fabric:/WordCount -EventsFilter None -ServicesFilter None -DeployedApplicationsFilter None -ExcludeHealthStatistics ApplicationName : fabric:/WordCount AggregatedHealthState : Error UnhealthyEvaluations : Unhealthy services: 100% (1/1), ServiceType='WordCountServiceType', MaxPercentUnhealthyServices=0%. Unhealthy service: ServiceName='fabric:/WordCount/WordCountService', AggregatedHealthState='Error'. Unhealthy partitions: 100% (1/1), MaxPercentUnhealthyPartitionsPerService=0%. Unhealthy partition: PartitionId='af2e3e44-a8f8-45ac-9f31-4093eb897600', AggregatedHealthState='Error'. Unhealthy replicas: 20% (1/5), MaxPercentUnhealthyReplicasPerPartition=0%. Unhealthy replica: PartitionId='af2e3e44-a8f8-45ac-9f31-4093eb897600', ReplicaOrInstanceId='131444422260002646', AggregatedHealthState='Error'. Error event: SourceId='MyWatchdog', Property='Memory'. ServiceHealthStates : None DeployedApplicationHealthStates : None HealthEvents : None ``` You can look at the replica to get more information: ```powershell PS D:\ServiceFabric> Get-ServiceFabricReplicaHealth -ReplicaOrInstanceId 131444422260002646 -PartitionId af2e3e44-a8f8-45ac-9f31-4093eb897600 PartitionId : af2e3e44-a8f8-45ac-9f31-4093eb897600 ReplicaId : 131444422260002646 AggregatedHealthState : Error UnhealthyEvaluations : Error event: SourceId='MyWatchdog', Property='Memory'. HealthEvents : SourceId : System.RA Property : State HealthState : Ok SequenceNumber : 131444422263668344 SentAt : 7/13/2017 5:57:06 PM ReceivedAt : 7/13/2017 5:57:18 PM TTL : Infinite Description : Replica has been created._Node_2 RemoveWhenExpired : False IsExpired : False Transitions : Error->Ok = 7/13/2017 5:57:18 PM, LastWarning = 1/1/0001 12:00:00 AM SourceId : MyWatchdog Property : Memory HealthState : Error SequenceNumber : 131444451657749403 SentAt : 7/13/2017 6:46:05 PM ReceivedAt : 7/13/2017 6:46:05 PM TTL : Infinite Description : RemoveWhenExpired : False IsExpired : False Transitions : Warning->Error = 7/13/2017 6:46:05 PM, LastOk = 1/1/0001 12:00:00 AM ``` > [!NOTE] > The unhealthy evaluations show the first reason the entity is evaluated to current health state. There may be multiple other events that trigger this state, but they are not be reflected in the evaluations. To get more information, drill down into the health entities to figure out all the unhealthy reports in the cluster. > > ## Next steps [Use system health reports to troubleshoot](service-fabric-understand-and-troubleshoot-with-system-health-reports.md) [Add custom Service Fabric health reports](service-fabric-report-health.md) [How to report and check service health](service-fabric-diagnostics-how-to-report-and-check-service-health.md) [Monitor and diagnose services locally](service-fabric-diagnostics-how-to-monitor-and-diagnose-services-locally.md) [Service Fabric application upgrade](service-fabric-application-upgrade.md)