One area of VMware/PowerMax integration that has frustrated customers since inception is the inability to get VAAI command information from the array. The VAAI paper mentions that there is a column available called “Extent Based Clone” which will display source or target depending on how the device is being used; but that has very limited application covering only one primitive, XCOPY. Customers have needed to rely on VMware’s reporting through esxtop.
Now in the latest PowerMaxOS release, 5978.444.444, we’ve added those VAAI statistics. We cover three of the major primitives, UNMAP, Write Same, and XCOPY. These primitives are the ones the might have the most impact to the environment.
Prior to PowerMaxOS 5978.444.444 there is no user interface to view VAAI primitive activity on the array. For Full Copy, however, if the Solutions Enabler command symdev show is run against the source or target device involved in a hardware-accelerated operation, the parameter “Extent Based Clone” will either display “Source” or “Target”.
Beginning with PowerMaxOS 5978.444.444, Unisphere for PowerMax includes the following VAAI statistics at the thin device and storage group level:
- VAAI Total Command Count
- VAAI Total Time
- VAAI Unmap Command Count
- VAAI Unmap KB
- VAAI Unmap MB
- VAAI Write Same Command Count
- VAAI Write Same KB
- VAAI Write Same MB
- VAAI XCopy Command Count
- VAAI XCopy KB
- VAAI XCopy MB
* Note that these statistics are not available for vVols.
In the Performance tab of Unisphere, you can find the VAAI metrics under “All”. This is displayed below. The only VAAI metric that is a KPI is VAAI Total Command Count.
The statistics available in Unisphere do not directly correlate with what VMware records for commands in esxtop so it is not beneficial to compare them; however the Unisphere stats provide information not available in esxtop such as the amount of storage that was reclaimed during a recent auto UNMAP process.
In this example, two VMs were removed from a datastore with auto UNMAP enabled. VMware issued 3024 UNMAP commands, however the array only required 10 commands to unmap the space on the array as seen in figure below. Those 10 commands reclaimed about 215 MB of space.
In the following example, a virtual machine was cloned within the same datastore which will engage XCOPY. Two clones were executed, the first using the default transfer size of 4 MB, and the second using the maximum transfer size (without a claim rule) of 16 MB. Note the values in the red and green boxes.
The next graph is a screenshot of the results of the two clones. Note the first run in the left of each graph where there were 12 XCOPY commands issued for a total of ~48 MB. When the transfer size was changed to 16 MB and the clone re-run, notice that the command count was reduced to 3 because the copy size had been quadrupled. The size of the actual copied data, however, does not change in the graph on the right.
Reminder: the max transfer size is just that, a max, and that depending on the data being copied there may not always be perfect ratio between the original copy count and the one done after changing the transfer size.
The new statistics are obviously most useful to tell when a particular device or devices in a storage group are involved in VAAI activity. Knowing this information can perhaps help diagnose performance issues or in determining if even VAAI commands are being issued or if the ESXi host is doing the work.
For issues with VAAI performance, in particular with SDRS, please see https://www.dell.com/support/kbdoc/en-us/533648.