APD/PDL in vSphere 6 with VMAX and VPLEX

Recently I have seen a number of internal emails here at EMC asking about our recommendations around All Paths Dead (APD) and Permanent Device Loss (PDL) in vSphere 6 on our storage arrays – mostly VPLEX to be specific.  There have been significant changes in APD/PDL in vSphere 6 which is why the questions are being posed.  It just so happened that I was updating my VMAX TechBook on this topic but as I won’t publish until June I thought I could preview it here.

Let’s start with a quick primer here on the terms.

All paths down or APD, occurs on an ESXi host when a storage device is removed in an uncontrolled manner from the host (or the device fails), and the VMkernel core storage stack does not know how long the loss of device access will last. VMware, however, assumes the condition is temporary. A typical way of getting into APD would be if zoning was removed.

Permanent device loss or PDL, is similar to APD (and hence why initially VMware could not distinguish between the two) except it represents an unrecoverable loss of access to the storage. VMware assumes the storage is never coming back. Removing the device from the storage group could cause this.

VMware’s approach to APD/PDL has evolved over the vSphere versions from not being able to distinguish between the two in vSphere 4, until today in vSphere 6 where a new capability called VM Component Protection is introduced.  As the blog is for vSphere 6 I’m going to go straight to that though the VMAX TechBook has more info on the previous vSphere versions (and more in June) if you want to review (http://www.emc.com/collateral/hardware/solution-overview/h2529-vmware-esx-svr-w-symmetrix-wp-ldv.pdf).

vSphere 6 offers some new capabilities around APD and PDL for the HA cluster which allow automated recovery of VMs. The capabilities are enabled through a new feature in vSphere 6 called VM Component Protection or VMCP. When VMCP is enabled, vSphere can detect datastore accessibility failures, APD or PDL, and then recover affected virtual machines. VMCP allows the user to determine the response that vSphere HA will make, ranging from the creation of event alarms to virtual machine restarts on other hosts.

VMCP is enabled in the vSphere HA edit screen of the vSphere Web Client. Note that like all new vSphere 6 features, this functionality is not available in the thick client. Navigate to the cluster, then the Manage tab and Settings sub-tab. Highlight the vSphere HA under Services and select Edit.  Check the box for “Protect against Storage Connectivity Loss”.  Here is how that looks:

Click to enlarge in new window

Once VMCP is enabled, storage protection levels and virtual machine remediations can be chosen for APD and PDL conditions as shown below:

Click to enlarge in new window

The PDL settings are the simpler of the two failure conditions to configure. This is because there are only two choices: vSphere can issue events, or it can initiate power off of the VMs and restart them on the surviving host(s). As the purpose of HA is to keep the VMs running, the default choice should always be to power off and restart. Once either option is selected, the table at the top of the edit settings is updated to reflect that choice.

Click to enlarge in new window

As APD events are by nature transient, and not a permanent condition like PDL, VMware provides a more nuanced ability to control the behavior within VMCP. Essentially, however, there are still two options to choose from: vSphere can issue events, or it can initiate power off of the VMs and restart them on the surviving host(s) (aggressively or conservatively). The output is similar to PDL.

Click to enlarge in new window

If issue events is selected, vSphere will do nothing more than notify the user through events when an APD event occurs. As such no further configuration is necessary. If, however, either aggressive or conservative restart of the VMs is chosen, additional options may be selected to further define how vSphere is to behave. The formerly grayed-out option “Delay for VM failover for APD” is now available and a minute value can be selected after which the restart of the VMs would proceed. Note that this delay is in addition to the default 140 second APD timeout. The difference in approaches to restarting the VMs is straightforward. If the outcome of the VM failover is unknown, say in the situation of a network partition, then the conservative approach would not terminate the VM, while the aggressive approach would. Note that if the cluster does not have sufficient resources, neither approach will terminate the VM.

In addition to setting the delay for the restart of the VMs, the user can choose whether vSphere should take action if the APD condition resolves before the user-configured delay period is reached. If the setting “Response for APD recovery after APD timeout” is set to “Reset VMs”, and APD recovers before the delay is complete, the affected VMs will be reset which will recover the applications that were impacted by the IO failures. This setting does not have any impact if vSphere is only configured to issue events in the case of APD. VMware and EMC recommend leaving this set to disabled so as to not unnecessarily disrupt the VMs. The additional settings are noted below:

Click to enlarge in new window

As previously stated, since the purpose of HA is to maintain the availability of VMs, VMware and EMC recommend setting APD to power off and restart the VMs conservatively. Depending on the business requirements, the 3 minute default delay can be adjusted higher or lower.

If you are looking to see whether APD/PDL has been detected, VMware has a new view under the monitor tab of the cluster and sub-tab vSphere HA.  If either condition occurs, an entry will appear like so:

Click to enlarge in new window

Note that this is a “current” view and does not record historical occurrences.  I’ve noticed that sometimes PDL is tough to catch.  If you open the vmkernel.log file though and search for “PDL” you’ll see an entry like this:

Click to enlarge in new window

I hope that was helpful to those running an HA cluster with vSphere 6.  Remember the recommendations hold true for both VMAX and VPLEX.  I will note one specific setting that VMware has gone back and forth on now over this past year. Originally VMware’s best practice for the host parameter Disk.AutoremoveOnPDL, was to disable it for vMSC.  VMware has now reversed course and recommends leaving it as default (as we do on VMAX). The KB is here, though it makes no mention of VMware’s previous position:  PDL AutoRemove. In any case since as it is now default, nothing needs to be done.