I recently updated my Virtual Volume (VVol) whitepaper to include some answers to customer-prompted questions, along with an important discussion around VVol limitations. For those familiar with the paper I’ll outline the changes here so you don’t need to go searching for them.
Let’s start with the questions. Some of these are new, some are answered in the document already, but it never hurts to review.
Q – How do I add a new storage resource to a storage container in Unisphere for PowerMax?
A – The new HTML 5 interface of Unisphere for PowerMax does take some time to get used to because some screens require a single click, some a double-click. Storage container management is one of those areas where you’ll use both. I’ve stepped through how to add a new storage resource to a container below. I’m starting from Storage -> VVol Dashboard and then I’ve clicked on the Storage Containers link in the Summary box. From there just follow these steps:
Q – How many storage containers are supported and why would you want to use more than one?
A – We support 16 storage containers on an array. Each storage container can have multiple storage resources, each of which needs to be associated with a different service level. So let’s take an example. I create Storage Container A and in that container I place 2 storage resources, Storage Resource A with a service level of Diamond, and Storage Resource B with a service level of Gold. If I then try to add Storage Resource C with a service level of Diamond to this container, it will fail because I already have a storage resource with Diamond. But I could add another resource with a different service level I have not used yet like Silver. In fact I could fill the container will all the service levels. OK, well if I can have all my service levels in a single container do I really need another container? Perhaps not. Some customers, however, like the idea of having a different storage container, and thus VVol datastore, for different functions, e.g. Test and Development versus Production. Or maybe you want to give each business unit a different container so you can keep track of total storage for charge back. Again, it isn’t required, but it might be useful. There is another important reason large enterprise customers may need another container and that is part of our next question.
Q – How many VVols are supported on an array, storage container, or storage resource?
A – Our arrays support 64,000 VVols. Since a VVol is just another device, that 64k represents the total supported devices on an array. If you run both VVols and VMFS, or perhaps physical environments, it would be VVols + regular devices <= 64k. Within the array, as I’ve noted, a storage container is comprised of storage resources. Therefore we aren’t concerned with the limits of the container, rather the limits of the storage resource. In our current implementation a storage resource can support 4096 VVols. OK, so I definitely hear some brains humming with math concerning service levels and our 64k max VVols so let’s address that. What if I only want Diamond VVols does that mean I can only have 4k even though you support 64k? Well, no, that’s where the multiple containers come into play. Since you can have a storage resource of Diamond in every container, 16 x 4k = 64k. OK that’s reassuring, though it would be nice if I didn’t have to use so many containers. I agree. That’s why you can get an E-Pack (special patch you ask for through support) to increase the 4k per storage resource to 8k. By the way, what happens if I breach the 4k or 8k limit? Your VVol creation would fail and you’d have to choose a different storage resource or even container. Well then, it would be good to be able to keep track just in case and avoid the failure. Indeed, read on.
Q – How do I see how many VVols are in my storage container/storage resource?
A – This certainly could be important if you are one of those customers approaching the limits of a storage resource. Currently we don’t offer a way to do this in Unisphere, unfortunately, only CLI with Solutions Enabler. The key to pulling the number of VVols is to use the output switch with the XML format when running the detail of a storage container. The command is:
symcfg show -sc -sc_name VVol_Multi_SL_Container -sid 357 -detail -out xml
The image below shows this output. The first red box in the figure highlights the name of the container while the second red box details the number of non-snapshot VVols, 15, and snapshot VVols, 4. This storage container therefore has 19 VVols. As the max VVol limit does not apply to the storage container, each storage resource needs to be examined. The blue and green boxes in the figure contain the VVol (non-snapshot and snapshot) counts for each resource, 5 and 14 respectively. Thus adding those together equals the 19 VVols in the container.
Q – Is there an individual VVol size limit?
A – Yes 16 TB. Now remember this is a single VVol, or VMDK. Yes, VVols are VMDKs. So don’t confuse this with the VVol datastore. VMFS datastores are limited to 64 TB but a VVol datastore is not. In fact a storage container, and thus VVol datastore, max size is 2^64 or 2 to the power of 64 – a ridiculous number. However, if you regularly need VMDKs in your VMFS environment larger than 16 TB, and you are thinking about VVols, please leave me a comment and I can pass it along to our product management for VVols.
So other than the questions, I added a section in the paper on determining the WWN and device ID of a VVol. I failed to put that in the first release but it is useful information, particularly when checking performance of a VM in Unisphere.
Identifying VVol WWN in Unisphere
Although vCenter provides no means to map a VVol to the underlying array device, Unisphere for PowerMax does offer this capability. In order to take advantage of the feature, the vCenter involved with VVols needs to be added to Unisphere. This can be done through the VMWARE -> vCenters and ESXi menu in Unisphere seen here (note this is also covered extensively in my TechBook and another blog post):
Once the vCenter is added, start in the figure below by selecting an ESXi host in that vCenter and double-clicking.
In steps 2-4 below, begin by selecting the Virtual Machines tab, double-click a VVol VM, and in the side panel on the right that will appear, select the hyperlink next to the Virtual Disks row.
In step 5 below, double-click on one of the Hard disks (vmdk) and in the right-hand panel that appears, all the information about the VVol is displayed. The blue box highlighted in the figure contains the VVol WWN .
To determine the device ID, it is necessary to use Solutions Enabler as the ID is not included in the Unisphere output. The command to list the device IDs with the WWN is:
symdev list -vvol -wwn -sid 357
Using this command, one can see in the figure below that the device ID for the VVol previously identified is 00043.
This device ID can then be used to monitor performance of the VM in Unisphere.
If you have another VVol question that eludes you, please leave a comment and I’d be happy to answer it and perhaps even add it to our list here.
Dell PowerFlex and PowerMax with VMware 
Very useful information.
Can you please help me for my below queries.. 🙂
Powermax2000 and ESXI 6.7
Can we check queuing length at PE level?
is there any option to limit on storage end since there is no option to limit for host IO ?
Thanks. Through esxtop you can view both the current value and the active value of the PE queue. Unfortunately, as I note in the white paper we do not support Host IO Limits on the array, however VMware does support SIOC so through the storage policy you can add a limitation.
Hello Drew! I am trying to delete HA info and it seems I cannot do this from the VMware side either through the web client or when ssh’d into the host. The exercise I am trying to undertake is to totally wipe out all VVol’s and the Storage container from within Unisphere. I ran the command you mentioned above. Here is the output:
C:\Windows\system32>symdev list -vvol -wwn -sid 94
Symmetrix ID: 000197802394
Device Name Device
—————————- ————————————————–
Sym Physical Config Attr WWN
—————————- ————————————————–
001E8 Not Visible VVOL 600009700BCA399A2E01007300001A03
002C0 Not Visible VVOL 600009700BCA399A2E01007300001A15
My question is this. Is there a way to delete these using a symdev command?
Thanks for your time!
-Larry
Unfortunately there is no way to remove vVols via Unisphere or SE. When you say you cannot delete the files from within vCenter, I presume you first disable HA, then navigate the datastore files and attempt to delete the folder and it simply does nothing or is it that you cannot see the folder/files?
That is correct I have disabled HA in a Test cluster I have setup with 2 hosts. When the VVOL was originally created it was part of a cluster where HA was enabled. I am able to see the files/folders but I cannot delete them from either the vSphere web client or when ssh’d in to the host. We just had EMC engineering involved with several cases where we have been battling some very odd behavior in our environment when using VVol’s on our PowerMax array. VM’s shutting down shortly after the first snapshot is taken from our CommVault backup, vm’s getting corrupted meaning they will not boot back up and often times the first 2 hard drives of the VM show as OMB. For now until the issue is resolved we have gone back to using traditional TDEV’s and everything seems to work fine. EMC engineering was able to dial in to the array and apply some sort of bin file change and a script to clean up what they were calling orphaned VVol’s. The clean-up was completed this past Monday and we were then able to delete the storage resource then the storage container. We are now in a mode of trying to set everything back up to see if the problem can be recreated. We are using VMware ESX 6.7 latest updates as of a few days ago. I know I gave you more info than you probably wanted but this has been a very perplexing issue that we are trying to get our arms around. Any input, advice, or comments that you have are greatly appreciated.
If the vCenter is essentially ignoring your request to delete the folder/files, support will have to dial back into the box and remove the vVols from the array. As you are testing, however, it might be more efficient to unmount the datastore and just create a new container on the array and use that to re-test. Support can remove the vVols at a later time when you complete the re-creation. The HAs files consume next to nothing in the SRP so leaving them there for the duration of your testing will not impact resources.
Thanks for your quick responses Drew. I was able to unmount the vVol then I created a new vVol (new name) and mounted it to one host that had HA disabled. At that point I was then able to delete the files and go back in to Unisphere and remove the storage resource and subsequently delete the Storage Container. So phase I of our testing is passed. I may hit you back up once we get back to the CommVault (CV) backup/restore testing. One thing that I failed to mention earlier is that the CV snapshot / vm Power-off / vm corruption I mentioned seems to be more predominant on Windows vm’s where SQL server is running. We have also had a couple of our Linux vm’s take a hit but they seemed to recover better than the Windows vm’s. Also, based on some reading I have done from DELL/EMC docs (probably papers that you have authored?) I came across something where it was suggested if you were seeing time-out type issues then something to try is to increase the GateKeeper’s (GK’s) from 5 to 15 so we have incorporated that in to our go forward plan. One final question for now. Is there any benefit to having more than one PE per host? We have a total of 25 hosts in 2 clusters on for our Linux vm’s and the other for Windows. The plan was to simply use 2 Storage Containers and grow them as needed. Does this all seem feasible and practical in a simplistic design? Also, the PowerMax array is ALL flash so are there any other considerations that you can comment on? Thanks again, I really appreciate your time!
Multiple PEs per host is unsupported. There must be only 1 unique PE per host. They cannot be shared either. The best practices are in the vVol doc here on the blog but 1 or 2 containers is a fairly typical setup. Some customers choose to use 1 and simply separate workloads based on service levels, while others will use a different container for prod and dev/test. Service providers may use more for the purposes of separation.