ALUA and Mobility ID on VMAX

A recent interaction with a customer led to this blog post concerning the Mobility ID that we support for VMAX devices and the multipathing policy of ALUA (Asymmetrical Logical Unit Access). Mobility devices are the only ones that support ALUA on the VMAX.

ALUA multipathing is typically associated with non-VMAX arrays like the former Clariion (CX) and also the current VNX. So the easiest way to explain it (to myself I suppose) is that on these types of arrays there is a concept that a LUN is “owned” by a processor. When I go to access that LUN from my host, I want to do so on a path to the processor that owns the LUN. This is known as the optimized path. Now on these multi-processor arrays each processor owns some LUNs but they are accessible from any of the processors. There is obviously communication between the processors so I could certainly access a LUN on  a processor that doesn’t own it, however that would be using what is called an unoptimized path. The second processor would need to communicate with the first to get me access. In the grand scheme of things it’s not like waiting in line for coffee, but every bit of time counts so I want to use that optimized path. And this is what ALUA does for you.

Moving on, VMAX doesn’t have LUN ownership, save for perhaps thinking in terms of SRDF/Metro (ALUA has some interesting implications there but as it isn’t supported yet I digress), but we do support ALUA with the Mobility ID. So what’s that?

We released supported for the Mobility Safe ID back in May of 2016. It is a new format of device identifier that provides a universally unique volume identifier.  It also increases the number of devices that can be supported (future) and makes migration easier across an environment.  Right now, the only devices in the VMware space that will use this ID by default are the Protocol Endpoint (PE) – a component of VVols – and VVols themselves (though they aren’t presented to ESXi hosts like the PE). The funny thing about the PE, however, is that it does not support ALUA. An exception to every rule as they say. While the Mobility ID can be enabled when you create a device, currently we do not see a benefit in doing so in a VMware environment.  Below is an example of a traditional WWN format and the new Mobility Safe ID (Identifier row).

As an aside, one of the things people notice about the Mobility ID format is that it not longer contain the VMAX SID.

Back to our customer case. As I noted right now I do not see general business cases for using the Mobility ID and therefore ALUA. This particular customer, however, needed all their device WWN formats to be unique and hence decided upon the Mobility ID. When you present devices to ESXi that use the Mobility ID, they are still claimed by VMware using the VMAX custom, default SATP – VMW_SATP_SYMM – unless PP/VE is being used. (I should quickly say that PP/VE avoids this problem I am illuminating here because it will automatically pick up Mobility ID as ALUA. This issue is a default NMP one. Just another reason to always use PP/VE if you can 🙂 ). Why doesn’t ALUA claim them? Well by default VMware claim rules are not setup that way. Remember Mobility ID is the only format on the VMAX that can use ALUA, but it does not have to. Experience has shown that when using datastores, claiming Mobility ID devices with the default SATP is perfectly fine. The issue appears to be, however, when using RDMs with particular operating systems. When a Mobility ID device is mapped to, in this case, a Red Hat OS, Red Hat recognizes the device as supporting ALUA and wants to use it. Unfortunately, because ESXi did not claim the device as ALUA, but Red Hat thinks it should be, Red Hat can’t use the device. To resolve this, we need to tell VMware to claim all Mobility ID devices as ALUA and to do that we need a custom claim rule.

Now there is no custom SATP for ALUA for VMAX, therefore we want to use VMware’s default one, VMW_SATP_ALUA. Here is the list of all the NMP SATPs on my box – notice CX does have a custom ALUA since that is its default SATP:

[root@dsib1143:~] esxcli storage nmp satp list
Name Default PSP Description
——————- ————- ———————————————
VMW_SATP_SYMM VMW_PSP_RR Supports EMC Symmetrix
VMW_SATP_MSA VMW_PSP_MRU Placeholder (plugin not loaded)
VMW_SATP_ALUA VMW_PSP_MRU Placeholder (plugin not loaded)
VMW_SATP_DEFAULT_AP VMW_PSP_MRU Placeholder (plugin not loaded)
VMW_SATP_SVC VMW_PSP_FIXED Placeholder (plugin not loaded)
VMW_SATP_EQL VMW_PSP_FIXED Placeholder (plugin not loaded)
VMW_SATP_INV VMW_PSP_FIXED Placeholder (plugin not loaded)
VMW_SATP_EVA VMW_PSP_FIXED Placeholder (plugin not loaded)
VMW_SATP_ALUA_CX VMW_PSP_RR Placeholder (plugin not loaded)
VMW_SATP_CX VMW_PSP_MRU Placeholder (plugin not loaded)
VMW_SATP_LSI VMW_PSP_MRU Placeholder (plugin not loaded)
VMW_SATP_DEFAULT_AA VMW_PSP_FIXED Supports non-specific active/active arrays
VMW_SATP_LOCAL VMW_PSP_FIXED Supports direct attached devices

To add a new rule, issue the following on each ESXi host that will have Mobility ID devices. Notice here we want to use the RoundRobin PSP, not MRU, and we want to change the iops from 1000 to 1 as is recommended by us and VMware. Feel free to adjust the description (-e).

esxcli storage nmp satp rule add -V EMC -M SYMMETRIX -s VMW_SATP_ALUA -c tpgs_on -P VMW_PSP_RR -O iops=1 -e “VMAX ALUA rule for Mobility IDs”

Now check the rule is added:

[root@dsib1143:~] esxcli storage nmp satp rule list
Name                 Device  Vendor   Model             Driver  Transport  Options                     Rule Group  Claim Options                        Default PSP  PSP Options  Description
——————-  ——  ——-  —————-  ——  ———  ————————–  ———-  ———————————–  ———–  ———–  ————————————-
VMW_SATP_ALUA                EMC      SYMMETRIX                                                        user        tpgs_on                              VMW_PSP_RR   iops=1       VMAX ALUA rule for Mobility IDs

 

Once the rule is added we have some options. If there are already devices presented, you can reboot and they will receive the new SATP when the box comes up. You can also unclaim the devices that are presented, and then rescan for storage and they will pick up the new SATP. If you have lots of devices or are actively using them, this option is less desirable. If you haven’t presented any devices yet, then you don’t need to reboot. Since ESXi will query the devices when they are presented, only those with Mobility ID will get this SATP. As an example let’s start with a pre-rule look at two devices. In red is a Mobility ID device and in blue a standard. Currently they both are using the standard SATP VMW_SATP_SYMM.

Now, post reboot the Mobility ID device has been assigned ALUA.

So even if you aren’t using Mobility ID for all devices, you can be sure each device will get the right SATP. Our recommendation, therefore, is that if you are going to use the Mobility ID for any devices, even if you don’t ever plan on using RDMs, add the ALUA rule. And I should mention for SRDF/Metro you will need ALUA with Mobility ID as we don’t support other pathing options like VMW_SATP_SYMM NMP/RR. I strongly recommend using PP/VE in these cases.