Just a short (for me anyway) post here on a vSphere 7 feature I get pinged about every so often: Clustered VMDK. VMware introduced this feature in vSphere 7. It is an attribute of a VMFS6 datastore which can be enabled on a new datastore – post-creation as the option is not available in the create datastore wizard – or on an existing datastore. When a VMFS has clustered VMDK enabled, it means a vmdk can be shared with multiple VMs. Now those of you who use a database and feature called Oracle RAC and run it on VMFS, might scratch your head and say you already do this without the clustered VMDK feature. Indeed, I have a number of whitepapers that cover the configuration. So why do we need this feature? Well, in the case of Oracle, we use a setting called multi-writer flag to share our vmdk with multiple VMs. That works perfectly because Oracle doesn’t need the one thing clustered VMDK provides: SCSI-3 Persistent Reservations (PR). SCSI3-PR is used for I/O fencing, which basically means it controls the access of multiple nodes to the same device. The Oracle software handles the fencing so it doesn’t require SCSI3-PR. So what does need SCSI3-PR? Windows Server Failover Cluster (WSFC). Before clustered VMDK, you had to use physical RDMs with WSFC, and, well, no one likes those for a host of reasons. If you want more detail on setting up WSFC with clustered VMDK, VMware has lots here. Note that during validation of the WSFC you will get a warning [Failure issuing call to Persistent Reservation REGISTER. RESERVATION KEY 0xa SERVICE ACTION RESERVATION KEY 0xb for Test Disk 0 from node dsib2031: Incorrect function] about the Microsoft feature called “Storage Spaces” which is not something you use on the SAN. We do not support the feature on the PowerMax so you can safely ignore the warning. The first test for SCSI-3 Persistent Reservations is the one that matters.
Clustered VMDK comes with a bunch of restrictions/limitations and a few requirements, two of which are pertinent to the array:
- The physical disk that virtual disks are stored on must support ATS SCSI commands.
- The physical disk must support SCSI-3 Persistent Reservations, specifically Write Exclusive All Registrants (WEAR) type reservations.
The first we’ve supported on VMAX/PowerMax since VMware introduced VAAI back in vSphere 4 so no problem there. The reason VMware needs it is they don’t support SCSI-2 with clustered. The second one is where I get all the questions because in many discussions around clustered VMDK, the SCSI-3 portion of the statement is left out and only the WEAR type reservations is included. Now if you don’t know that WEAR is a type of reservation under the SCSI3-PR umbrella, it is confusing. In fact SCSI3-PR has a number of different types besides WEAR:
- Write Exclusive (WE)
- Exclusive Access (EA)
- Write Exclusive Registrants Only (WERO)
- Exclusive Access Registrants Only (EARO)
- Write Exclusive All Registrants (WEAR)
- Exclusive Access All Registrants (EAAR)
The PowerMax, supports all these types, but for clustered VMDK only the WEAR one matters. Note also that if you are using SRDF it sits on top of the devices so it has no bearing on supportability of SCSI-3.
Finally, how do I know if my array supports it and how do you enable the feature? Well, VMware has that covered. When you go to create a datastore in vSphere 7, there is a new column in the create datastore wizard called Clustered VMDK Supported, which will tell you whether the array device, in this case a PowerMax, supports the feature (it does).
But I should note, there is no checkbox to enable clustered VMDK during datastore creation. So after you create the datastore, you can see the option is disabled.
Select the enable button to receive the following notice where you will need to select ENABLE again to complete the process.
Now the datastore shows support for Clustered VMDK and you are ready.
As I mentioned, lots of different limitations and recommendations to read about for this feature. Some will be the same as Oracle shared vmdks, some different. If you plan on using WSFC with clustered VMDK, definitely read up on all of them before you start.
Though this post was just about FC/VMFS and WSFC, I want to remind users we do not support SCSI3-PR with VMware Virtual Volumes (vVols). I point that out simply because VMware does support WSFC with vVols but we cannot yet.