Tuesday, September 6, 2016

VMware ESXi Claim Rules unleashed

This blog article explains VMware ESXi claim rules in general and the differences between core and SATP (Storage Array Type Plugin) claim rules. In support we get a lot of questions about claim rules for several reasons. One could be performance tuning while another one could be the usage of a 3rd party multipathing plugin. With a claim rule you can pretty much define how a device of a certain vendor should behave if the the default setting won’t be sufficient. But let’s start from the beginning. There are two kind of claim rules:

Core Claim Rules
SATP Claim Rules

They differentiate in the way where they integrate in the PSA. The next figure visualises this:

Figure 1: VMware Claim Rules

If you want to learn more about the I/O flow in ESXi please follow my blog article: VMware ESXi Storage I/O Path.

Core Claim Rules

Now that we know that there are two kinds of claim rules we start with core claim rules. The PSA (Pluggable Storage Architecture) uses claim rules to determine which multipathing module should be used. The only complete MPP (Multipathing Plugin) today next to the NMP (Native Multipathing Plugin) of VMware is PowerPath from DELL/EMC.

There are existing default rules to claim devices for example for local devices. With local devices it doesn’t matter which other 3rd party MPP is installed as these devices won’t get claimed. Claim rules changes cannot be archived in ESXi on the fly. Instead there is the need to add, remove and then load them from a file. Changing an existing claim rule or add a new one has to be done with the following commands:

Run the esxcli storage core claimrule modification command (add, remove or move).

Run the esxcli storage core claimrule load command to replace the currently rule in VMkernel from the configuration file.

There are different options to create core claim rules. By adding a claim rules manually you can mask a path using the MASK_PATH claim rule to hide a LUN from ESXi. To find out what your installed multipathing plugins are use the following command:

~ # esxcli storage core plugin list

Plugin name Plugin class

----------- ------------

NMP MP

As you see on my system there is no 3rd party MPP installed. If you want to see what core claim rules are in place please use the following command:

~ # esxcli storage core claimrule list

Rule Class Rule Class Type Plugin Matches

---------- ----- ------- --------- --------- ---------------------------------

MP 0 runtime transport NMP transport=usb

MP 1 runtime transport NMP transport=sata

MP 2 runtime transport NMP transport=ide

MP 3 runtime transport NMP transport=block

MP 4 runtime transport NMP transport=unknown

MP 101 runtime vendor MASK_PATH vendor=DELL model=Universal Xport

MP 101 file vendor MASK_PATH vendor=DELL model=Universal Xport

MP 65535 runtime vendor NMP vendor=* model=*

Rule Class Rule XCOPY Use Array Reported Values XCOPY Use Multiple Segments XCOPY Max Transfer Size

---------- ----- ------------------------------- --------------------------- -----------------------

MP 0 false false 0

MP 1 false false 0

MP 2 false false 0

MP 3 false false 0

MP 4 false false 0

MP 101 false false 0

MP 65535 false false 0

You see that there are quite a few already exist. The ones from 0 - 4 as I mentioned earlier are for local devices and one rule is for using the MASK_PATH for specific DELL devices. To find out more about MASK_PATH please follow VMware KB 1009449. There are certain reasons to MASK a LUN from ESXi and KB 1009449 explains this very well.

The numbers of the rules itself are defined by VMware:

Rules 0 - 100 are reserved for internal use by VMware
Rules 101 - 65435 are available for general use. Any 3rd party MPP installed use claim rules in this range. By default the PSA claim rule 101 masks Dell array pseudo devices. DO NOT remove this rule, unless you want to unmask these devices.
Rules 65436 - 65535 are reserved for internal use by VMware

A very good explanation you find in the following “vSphere Command-Line Interface Concepts and Examples” starting on Page 93.

Unclaim Devices from DELL/EMC PowerPath

If you are using the DELL/EMC PowerPath MPP the previous command to show all core claim rules looks different as you see:

~ # esxcli storage core claimrule list

Rule Class Rule Class Type Plugin Matches

---------- ---- ------- --------- --------- ---------------------------------

MP 0 runtime transport NMP transport=usb

MP 1 runtime transport NMP transport=sata

MP 2 runtime transport NMP transport=ide

MP 3 runtime transport NMP transport=block

MP 4 runtime transport NMP transport=unknown

MP 101 runtime vendor MASK_PATH vendor=DELL model=Universal Xport

MP 101 file vendor MASK_PATH vendor=DELL model=Universal Xport

MP 250 runtime vendor PowerPath vendor=DGC model=*

MP 250 file vendor PowerPath vendor=DGC model=*

MP 260 runtime vendor PowerPath vendor=EMC model=SYMMETRIX

MP 260 file vendor PowerPath vendor=EMC model=SYMMETRIX

MP 270 runtime vendor PowerPath vendor=EMC model=Invista

MP 270 file vendor PowerPath vendor=EMC model=Invista

MP 280 runtime vendor PowerPath vendor=HITACHI model=*

MP 280 file vendor PowerPath vendor=HITACHI model=*

MP 290 file vendor PowerPath vendor=HP model=*

MP 300 runtime vendor PowerPath vendor=COMPAQ model=HSV111 (C)COMPAQ

MP 300 file vendor PowerPath vendor=COMPAQ model=HSV111 (C)COMPAQ

MP 310 runtime vendor PowerPath vendor=EMC model=Celerra

MP 310 file vendor PowerPath vendor=EMC model=Celerra

MP 320 runtime vendor PowerPath vendor=IBM model=2107900

MP 320 file vendor PowerPath vendor=IBM model=2107900

MP 330 runtime vendor PowerPath vendor=IBM model=2810XIV

MP 330 file vendor PowerPath vendor=IBM model=2810XIV

MP 340 runtime vendor PowerPath vendor=XtremIO model=XtremApp

MP 340 file vendor PowerPath vendor=XtremIO model=XtremApp

MP 350 file vendor PowerPath vendor=NETAPP model=*

MP 65535 runtime vendor NMP vendor=* model=*

What does that exactly mean now? You see that the Plugin for the highlighted ones is now PowerPath. Everything which does not have the transport mode:

USB
SATA
IDE
BLOCK
UNKNOWN

or is from the following particular vendors:

DGC (EMC Clariion and EMC VNX)
EMC Symmetrix
EMC Invista
All Hitachi models (HDS - Hitachi Data Systems)
Compaq (HP EVA)
EMC Celerra
Several IBM models
EMC XtremIO
All Netapp Models

will now get claimed by PowerPath. Everything outside this will use the NMP. The problem is that if you eventually reach out to your vendor or choice to work on an issue, let’s say this is Netapp and they are eventually recommend to use the NMP for their devices. Don’t get me wrong the MPP of DELL/EMC is fully supported but the support comes from DELL/EMC and if you ask me I would recommend to use the NMP for all non DELL/EMC devices which should make troubleshooting easier not getting another vendor involved. If you are using the DELL/EMC MPP you are obviously also not able to list your non PowerPath claimed devices with commands like esxcli storage nmp device list or esxcli storage core device list. To unclaim Netapp devices you have to create a core claim rule which adds in before the PowerPath claim rule Netapp (<350). You are able to create a claim rule with the ID 349 but I would suggest to take something which adds in right at the beginning of rules which are available for general usage. This could be something between 102 - 249. I’m a big fan to start with a 1 at the end in a new hundred range so I decided to use the ID 201. To create a core claim rule you have to first create and then load the new claimrule.

~ # esxcli storage core claimrule add --rule 201 --type vendor --vendor=NETAPP --plugin NMP

~ # esxcli storage core claimrule load

After this please reboot your system so the new system core claim rules get into effect. If you now trigger the claimrule list command again you should see your new rule.

~ # esxcli storage core claimrule list

Rule Class Rule Class Type Plugin Matches

---------- ---- ------- ------ ------- ---------------------

MP 201 runtime vendor NMP vendor=NETAPP model=*

MP 201 file vendor NMP vendor=NETAPP model=*

Unclaim specific devices from DELL/EMC PowerPath

Unclaiming all of your Netapp devices in the above example was easy but how about you are using DELL/EMC PowerPath in combination with software like PernixData FVP or Architect which only works in combination with VMware’s NMP? This software is using an own PSP (Path Selection Policy). Another reason why you would unclaim specific devices from PowerPath is because you eventually plan to use PernixData FVP or Architect only for just a handful of devices while others stay with DELL/EMC PowerPath. This use case could be interesting if one or more LUN’s have the need to get accelerated while others should stay with the DELL/EMC MPP. The only way to get around this is to create core claim rule based on devices.

You need to find the list of LUN ID’s which you are intend to mask (I removed the last column “Partition” to make the output easier to read):

~ # esxcli storage vmfs extent list

Volume Name VMFS UUID Extent Number Device Name

---------------------------- ----------------------------------- ------------- ------------------------------------

am1ifst002-lun0253-utility 56b212c4-be05b2fb-ade6-001b21857010 0 naa.6589cfc0000000a211a7722cb83865b3

am1ifvh004_local 57ac6114-a43a4a55-49e1-001b2182fc60 0 naa.5000c5007f9ac90b

am1ifst002-lun0000-cluster02 56b32c74-42a00190-37c1-001b21857010 0 naa.6589cfc0000005a9b84630fe075f41b2

am1ifst002-lun0254-hb01 56b212a5-f4d11213-ee96-001b21857010 0 naa.6589cfc000000fb4cf8a83d1ffd9ec0c

am1ifst001-lun0001-cluster02 55883a14-21a51000-d5e9-001b21857010 0 naa.60014058161a44edb63cd3ca1d94a0d4

am1ifst002-lun0255-hb02 56b21293-995fc851-c15f-001b21857010 0 naa.6589cfc0000003bb0e6b545b02f58352

am1ifst001-lun0000-cluster02 55883a01-dd413d6a-ccee-001b21857010 0 naa.6001405ee3d0593d61f4d3873da453d5

The next step is to isolate the information for the claiming of a single device. In this case we take the first device as an example:

Volume Name: am1ifst002-lun0253-utility
Device Name: naa.6589cfc0000000a211a7722cb83865b3

With the following command you find out the HBA:Controller:Target:LUN_ID:

~ # esxcli storage nmp path list -d naa.6589cfc0000000a211a7722cb83865b3 | grep -i vmhba

Runtime Name: vmhba33:C0:T0:L253

Runtime Name: vmhba33:C1:T0:L253

With the above commands you now have all information to create the device claim rule. There is one claim rule needed for every runtime path of the device. As we use two runtime paths we need to create two claim rules. We remember that we are allowed to create claim rules between 102 - 249 before the PowerPath claim rules start. Why not start very low with the number 111.

esxcli storage core claimrule add --rule 111 -t location -A vmhba33 -C 0 -T 0 -L 253 -P NMP

esxcli storage core claimrule add --rule 112 -t location -A vmhba33 -C 1 -T 0 -L 253 -P NMP

Reboot the host so the claim rules get into affect.

I am sure you see that my HBA is the ESXi iSCSI Software HBA. Using FC or FCoE as your transport protocol or physical iSCSI controllers you would usually see at least two HBA’s e.g. vmhba1 and vmhba2. Please keep in mind that if you add additional paths to a device you then have to unclaim them. This is not always easy to remember so please keep that in mind! Also if you plan an extension of paths for the future you might want to leave some numbers free and start with rule 121 for the next device. When you now finally have claimed this device and and in case you are using PernixData FVP you will see a third path with another LUN ID and another vmhba, target etc. This one you obviously don’t need to claim.

SATP Claim Rules

After we discussed core claim rules we now get into the second group of claim rule which are SATP based. Let me first explain why we even need them and why they are named SATP claim rules. The SATP is the control plane within the NMP itself. It knows which kind of storage the ESXi server is connected to. It also defines which PSP the device of that particular storage system is currently using. Let’s take our example again:

Volume Name: am1ifst002-lun0253-utility
Device Name: naa.6589cfc0000000a211a7722cb83865b3

~ # esxcli storage nmp device list -d naa.6589cfc0000000a211a7722cb83865b3

naa.6589cfc0000000a211a7722cb83865b3

Device Display Name: FreeBSD Local SCSI/iSCSI Disk (naa.6589cfc0000000a211a7722cb83865b3)

Storage Array Type: VMW_SATP_ALUA

Storage Array Type Device Config: {implicit_support=on;explicit_support=off; explicit_allow=on;alua_followover=on; action_OnRetryErrors=off; {TPG_id=1,TPG_state=AO}}

Path Selection Policy: VMW_PSP_FIXED

Path Selection Policy Device Config: {preferred=vmhba33:C0:T0:L253;current=vmhba33:C0:T0:L253}

Path Selection Policy Device Custom Config: type=iops,iops=1000

Working Paths: vmhba33:C0:T0:L253

Is USB: false

It looks like this device is using VMware’s VMW_PSP_FIXED and concludes that the device always only uses one path due to the fixed PSP and the load would not get distributed through the two paths we have. This could be all well and good if your storage vendor recommends to use the fixed PSP. To find out what the default SATP / NMP configuration looks like the following command gives you an overview:

~ # esxcli storage nmp satp list

Name Default PSP Description

------------------- ------------- -------------------------------------------------------

VMW_SATP_ALUA VMW_PSP_FIXED Supports non-specific arrays that use the ALUA protocol

VMW_SATP_MSA VMW_PSP_MRU Placeholder (plugin not loaded)

VMW_SATP_DEFAULT_AP VMW_PSP_MRU Placeholder (plugin not loaded)

VMW_SATP_SVC VMW_PSP_FIXED Placeholder (plugin not loaded)

VMW_SATP_EQL VMW_PSP_FIXED Placeholder (plugin not loaded)

VMW_SATP_INV VMW_PSP_FIXED Placeholder (plugin not loaded)

VMW_SATP_EVA VMW_PSP_FIXED Placeholder (plugin not loaded)

VMW_SATP_ALUA_CX VMW_PSP_RR Placeholder (plugin not loaded)

VMW_SATP_SYMM VMW_PSP_RR Placeholder (plugin not loaded)

VMW_SATP_CX VMW_PSP_MRU Placeholder (plugin not loaded)

VMW_SATP_LSI VMW_PSP_MRU Placeholder (plugin not loaded)

VMW_SATP_DEFAULT_AA VMW_PSP_FIXED Supports non-specific active/active arrays

VMW_SATP_LOCAL VMW_PSP_FIXED Supports direct attached devices

There are quite a lot of default SATP / PSP options available for a good number of different storage systems in the field. The one you will see most of the time though is the VMW_SATP_ALUA which is in use for all ALUA (Asymmetric Logical Unit Access) based storage systems. There are number of blog articles or KB articles available just search for ALUA but basically it is the behaviour how I/O flows with a ALUA based storage system and how we avoid I/O going through the interconnect which could be inefficient as well as how to act if failures happen. One of the best articles I have read about the PSP's work in combination with ALUA and what can be done to improve performance with All Flash implementations you find here. The article is about three years old but I really like the idea of using SIOC (Storage I/O Control) similar mechanism to improve write I/O performance in case there is a saturation of the active controller service optimised paths to the ESXi server. I have no idea if this feature ever made it any further but I assume not otherwise we would have heard about it.

SATP / PSP Configuration

Didn’t someone tell me that Round Robin is much better for me? That again depends on what the suggested PSP for your storage vendor is but most vendors today support and suggest Round Robin for their devices. To change the default PSP for our SATP VMW_SATP_ALUA you simply set this SATP to the VMW_PSP_RR with the following command:

~ # esxcli storage nmp satp set --default-psp=VMW_PSP_RR --satp=VMW_SATP_ALUA

Default PSP for VMW_SATP_ALUA is now VMW_PSP_RR

Very important this is 100% non-disruptive as it just gets into effect for new devices. Running the esxcli storage nmp satp list again shows that the Default PSP for the SATP VMW_SATP_ALUA has now changed as I highlighted below:

~ # esxcli storage nmp satp list

Name Default PSP Description

------------------- ------------- -------------------------------------------------------

VMW_SATP_ALUA VMW_PSP_RR Supports non-specific arrays that

Please never change the default PSP to a 3rd party PSP and only to the NMP PSP as otherwise you will run into issues when uninstalling a software. The settings for a 3rd party PSP you do with customization or a custom SATP claim rule!

From now all new LUNs with the SATP: VMW_SATP_ALUA will get the default PSP: VMW_PSP_RR. The bad news is that all existing LUNs will stay as they are. The good news is that there are two ways how to get this fixed but even if this will work in 99% of all cases just fine I would recommend to do these changes in maintenance mode specially when you are using a 3rd party PSP like PernixData FVP it is even not supported to do this while VMs are running.

Set back all devices to the default PSP with the following loop (local storage is always excluded) for all devices with the SATP VMW_SATP_ALUA:

~ # for i in `esxcli storage nmp device list | grep -B 1 VMW_SATP_ALUA | grep -A 1 'Device Display Name:' | sed -n '/'VMW_SATP_LOCAL'/{s/.*//;x;d;};x;p;${x;p;}' | grep 'naa\|eui' | awk '{print $NF}' | sed 's/(//g' | sed 's/)//g'` ; do esxcli storage nmp device set --device $i --default ; done

The second way of doing this is change all devices using the SATP: VMW_SATP_ALUA to the the now default PSP VMW_PSP_RR:

Please keep in mind if you are using a 3rd party PSP like PernixData FVP you have to change the PSP to the one you are using. The same is obviously also true for other 3rd party PSP. The statement then would look like this:

~ # for i in `esxcli storage nmp device list | grep -A 1 'Device Display Name:' | sed -n '/'VMW_SATP_LOCAL'/{s/.*//;x;d;};x;p;${x;p;}' | grep 'naa\|eui' | awk '{print $NF}' | sed 's/(//g' | sed 's/)//g'` ; do esxcli storage nmp device set -d $i --psp=PRNX_PSP_RR ; done

In case you want to set back all your devices back to default and not only the devices with the SATP: VMW_SATP_ALUA just remove: | grep -B 1 VMW_SATP_ALUA. To finish up, with the command to show the device status we see that the example device now uses the PSP VMW_PSP_RR:

~ # esxcli storage nmp device list -d naa.6589cfc0000000a211a7722cb83865b3

naa.6589cfc0000000a211a7722cb83865b3

Device Display Name: FreeBSD Local SCSI/iSCSI Disk (naa.6589cfc0000000a211a7722cb83865b3)

Storage Array Type: VMW_SATP_ALUA

Storage Array Type Device Config: {implicit_support=on;explicit_support=off; explicit_allow=on;alua_followover=on; action_OnRetryErrors=off; {TPG_id=1,TPG_state=AO}}

Path Selection Policy: VMW_PSP_RR

Path Selection Policy Device Config: {policy=rr,iops=1000,bytes=10485760,useANO=0; lastPathIndex=1: NumIOsPending=0,numBytesPending=0}

Path Selection Policy Device Custom Config: type=iops,iops=1000

Working Paths: vmhba33:C0:T0:L253, vmhba33:C1:T0:L253

Is USB: false

PSP Tuning

In the next section we get into the tuning of the devices configured by an individual SATP and PSP. As you see in the example below the device config is set to these values:

Path Selection Policy Device Config: {policy=rr,iops=1000,bytes=10485760,useANO=0; lastPathIndex=1: NumIOsPending=0,numBytesPending=0}

Before explaining these two values and the way how to tune them let’s understand first how Round Robin works. Round Robin uses an algorithm to balance the load across all active storage paths. A path is selected and used until a specific amount of data (in IOPS or bytes) has been transferred. When this quantity is reached the PSP selects the next path in the list. The quantity which has been configured is the limit of a single path. There are two types of limits: IOPS and bytes.

IOPS: The Round Robin PSP is set to a default limit of 1000 IOPS. When 1000 IOPS have passed this path the PSP switches over to the next path.

Bytes: The Round Robin PSP is set to a default limit of 10485760 Byte (10 MiB). When 10 MiB have been passed through the actual path the PSP switches to the next path.

When you search for the PSP tuning you always only find the IOPS tuning but I recently had a case where the bytes setting was the issue. The bytes setting only applies in very rare cases but it could happen when the block size is very big. There are two ways how to tune the PSP. I am a big fan of keeping things clean so my favourite is a SATP claim rule instead of a the below custom config but the great benefit is that this works non disruptively without a reboot.

Custom PSP LUN config

Configure a custom config (on the fly no reboot needed). The following loop sets all existing LUNs to the IOPS setting of 1. Please contact your storage vendor what they think is the best setting. And yes 1 IOPS is not always needed and the best but a lot of vendors tell you it is the holy grail to tune this.

Again if you want to just change your LUN’s with a certain SATP take the example from above which only runs this loop for the SATP VMW_SATP_ALUA. Please keep in mind if you are using a 3rd party PSP like PernixData FVP you have to run this using the generic PSP config versus the round robin device config as well as the IOPS setting works a little bit different. The statement then would look like this:

Create a custom claim rule which applies then also for new LUN’s (reboot required) but not by default to existing LUN’s. How are we getting the existing LUN’s now to the intended setting as well as future new LUN’s? First have a look how many system SATP claim rules are already in place with (the list is not complete):

~ # esxcli storage nmp satp rule list

Name Device Vendor Model Driver Transport Options

------------------- ------ ------- ---------------- ------ --------- --------------------------

VMW_SATP_ALUA LSI INF-01-00 reset_on_attempted_reserve

VMW_SATP_ALUA NETAPP reset_on_attempted_reserve

VMW_SATP_ALUA IBM 2810XIV

VMW_SATP_ALUA IBM 2107900 reset_on_attempted_reserve

VMW_SATP_ALUA IBM 2145

VMW_SATP_ALUA

VMW_SATP_MSA MSA1000 VOLUME

VMW_SATP_DEFAULT_AP DEC HSG80

VMW_SATP_DEFAULT_AP HSVX700

VMW_SATP_DEFAULT_AP HSV100

VMW_SATP_DEFAULT_AP HSV110

VMW_SATP_EQL EQLOGIC

VMW_SATP_INV EMC Invista

VMW_SATP_INV EMC LUNZ

VMW_SATP_EVA HSV200

VMW_SATP_EVA HSV210

VMW_SATP_EVA HSVX740

VMW_SATP_EVA HSV101

VMW_SATP_EVA HSV111

VMW_SATP_EVA HSV300

VMW_SATP_EVA HSV400

VMW_SATP_EVA HSV450

VMW_SATP_ALUA_CX DGC

VMW_SATP_SYMM EMC SYMMETRIX

Rule Group Claim Options Default PSP PSP Options Description
---------- ------------- ----------- ----------- --------------------------------------------------------------------------
system      tpgs_on       VMW_PSP_MRU               NetApp E-Series arrays with ALUA support
system      tpgs_on       VMW_PSP_RR                NetApp arrays with ALUA support
system      tpgs_on       VMW_PSP_RR                IBM 2810XIV arrays with ALUA support
system                    VMW_PSP_RR
system                    VMW_PSP_RR
system      tpgs_on                                 Any array with ALUA support
system                                              MSA 1000/1500 [Legacy product, Not supported in this release]
system
system      tpgs_off                                active/passive HP StorageWorks SVSP
system                                              active/passive EVA 3000 GL [Legacy product, Not supported in this release]
system                                              active/passive EVA 5000 GL [Legacy product, Not supported in this release]
system                                              All EqualLogic Arrays
system
system                                              Invista LUNZ
system      tpgs_off                                active/active EVA 4000/6000 XL
system      tpgs_off                                active/active EVA 8000/8100 XL
system      tpgs_off                                active/active HP StorageWorks SVSP
system      tpgs_off                                active/active EVA 3000 GL [Legacy product, Not supported in this release]
system      tpgs_off                                active/active EVA 5000 GL [Legacy product, Not supported in this release]
system      tpgs_off                                active/active EVA 4400
system      tpgs_off                                active/active EVA 6400
system      tpgs_off                                active/active EVA 8400
system      tpgs_on                                 CLARiiON array in ALUA mode
system                                              EMC Symmetrix

They have all certain settings like using the default PSP of VMW_PSP_RR for all Netapp devices but they don’t have a tuned setting configured. To create a new SATP claim rule you need to know some information like:

Vendor: your vendor
Model: your vendor model
Options: specific options how to react on LUN resets
Claim Options: specific things for e.g. ALUA
Default PSP: the wanted default PSP
PSP Options: the IOPS setting

to then create the the PSP with a certain IOPS or byte setting. There are two ways how to get information about your devices. The following command gives you information about the specific devices. Let’s take our example again:

~ # esxcli storage core device list -d naa.6589cfc0000000a211a7722cb83865b3 | grep 'Vendor\|Model'

Vendor: FreeBSD
Model: iSCSI Disk

The next esxcli command give you the possibility to find specific settings about the device:

~ # esxcli storage nmp device list -d naa.6589cfc0000000a211a7722cb83865b3 | grep 'Storage Array Type Device Config'

Storage Array Type Device Config: {implicit_support=on;explicit_support=off; explicit_allow=on;alua_followover=on; action_OnRetryErrors=off; {TPG_id=1,TPG_state=AO}}

Another other to find out specifics about the device is the following:

Perform a rescan of your host: ~ # esxcli storage core adapter rescan --all

Grep for the scanned information in your vmkernel.log: ~ # grep ScsiScan /var/log/vmkernel.log

The grep should give you your vendor and model number of your device.

After you gathered all the information you can create a custom claim rule:

~ # esxcli storage nmp satp rule add -s "Your_SATP" -V "Your_Vendor" -M "Your_Model" -P "VMW_PSP_RR" -O "iops=1"

Staying with the example of having another 3rd party PSP in use the SATP has to look like this. In recent support cases we found out that it would also work to stay with the Native PSP as in case of the PernixData PSP the setting will get adapted in other words if the SATP Claim Rule is set to VMW_PSP_RR the LUN then should have PRNX_PSP_RR.

~ # esxcli storage nmp satp rule add -s "Your_SATP" -V "Your_Vendor" -M "Your_Model" -P "PRNX_PSP_RR" -O "type=iops,iops=1"

In case you are using PernixData FVP it is important to stop the daemon prnxd (/etc/init.d/prnxd stop) before you run the following command as otherwise the claiming logic will add the custom setting of 1000 IOPS again.

To remove all custom config and set your devices to default please use the following command (works the same way for the native PSP or any 3rd party PSP. Some would think the generic deviceconfig only works for 3rd party PSPs but it works for the Native PSP as well as the 3rdp party PSP):

Now reboot and all of your devices with a custom SATP rule should have the intended setting without any custom config: esxcli storage nmp device list -d naa.6589cfc0000000a211a7722cb83865b3

naa.6589cfc0000000a211a7722cb83865b3

Device Display Name: FreeBSD Local SCSI/iSCSI Disk (naa.6589cfc0000000a211a7722cb83865b3)

Storage Array Type: VMW_SATP_ALUA

Storage Array Type Device Config: {implicit_support=on;explicit_support=off; explicit_allow=on;alua_followover=on; action_OnRetryErrors=off; {TPG_id=1,TPG_state=AO}}

Path Selection Policy: VMW_PSP_RR

Path Selection Policy Device Config: {policy=rr,iops=1,bytes=10485760,useANO=0; lastPathIndex=0: NumIOsPending=0,numBytesPending=0}

Path Selection Policy Device Custom Config:

Working Paths: vmhba32:C1:T0:L253, vmhba32:C0:T0:L253

Is USB: false

What is very important to understand is the fact that you have to create the SATP claim rules with the exact same naming so please honour upper- and lower case. If there are any empty spaces after the “vendor” or “model” please NOT use them. I tested quite a lot and I found an article that you should honour any existing spaces but that is not true. The correct creating in our case is the following without any free space even we found them using the grep command after the rescan: -V "FreeBSD" -M "iSCSI Disk". There are easy ways to check if all the LUNs have the correct setting with the following two commands:

Check if all devices are using RR now

~ # esxcli storage nmp device list | grep 'Path Selection Policy:' | egrep -v 'Storage Array Type: VMW_SATP_LOCAL|VMW_PSP_FIXED'

Check if all IOPS settings are set correctly:

~ # esxcli storage nmp device list | grep -i iops

Troubleshooting

What happens if the claim rule does not work? All information about this is in /etc/vmware/esx.conf and the vmkernel.log. Below you see that there are two SATP claim rules (Rule 0000 and 0001) for the SATP: VMW_SATP_ALUA as well as the default PSP setting for the VMW_SATP_ALUA we switched earlier to VMW_PSP_RR.

Default PSP for VMW_SATP_ALUA:

/storage/plugin/NMP/config[VMW_SATP_ALUA]/defaultpsp = "VMW_PSP_RR"

SATP Claim Rule 0000 vor Vendor: “FreeBSD” and Model: “iSCSI Disk”:

/storage/plugin/NMP/config[VMW_SATP_ALUA]/rules[0000]/vendor = "FreeBSD"

/storage/plugin/NMP/config[VMW_SATP_ALUA]/rules[0000]/claimOptions = "tpgs_on"

/storage/plugin/NMP/config[VMW_SATP_ALUA]/rules[0000]/psp = "VMW_PSP_RR"

/storage/plugin/NMP/config[VMW_SATP_ALUA]/rules[0000]/model = "iSCSI Disk"

/storage/plugin/NMP/config[VMW_SATP_ALUA]/rules[0000]/pspOptions = "iops=1"

/storage/plugin/NMP/config[VMW_SATP_ALUA]/rules[0000]/ruleGroup = "user"

/storage/plugin/NMP/config[VMW_SATP_ALUA]/rules[0000]/force = "true"

SATP Claim Rule 0000 vor Vendor: “FreeBSD” and Model: “iSCSI Disk”:

/storage/plugin/NMP/config[VMW_SATP_ALUA]/rules[0001]/vendor = "SYNOLOGY"

/storage/plugin/NMP/config[VMW_SATP_ALUA]/rules[0001]/claimOptions = "tpgs_on"

/storage/plugin/NMP/config[VMW_SATP_ALUA]/rules[0001]/psp = "VMW_PSP_RR"

/storage/plugin/NMP/config[VMW_SATP_ALUA]/rules[0001]/model = "iSCSI Storage"

/storage/plugin/NMP/config[VMW_SATP_ALUA]/rules[0001]/pspOptions = "iops=1"

/storage/plugin/NMP/config[VMW_SATP_ALUA]/rules[0001]/ruleGroup = "user"

/storage/plugin/NMP/config[VMW_SATP_ALUA]/rules[0001]/force = "true"

Please be aware that in case you want to remove a SATP claim rule it has to be done in the exact same way you have created it. This is very simple by just replacing the “add” with “remove” and remove the --force at the end. But in case you just miss something or wrote something wrong you won’t be able to remove this SATP claim rule but at least you get an error message about it. For the one in our example above it would be (again no spaces!):

~ # esxcli storage nmp satp rule remove -s "VMW_SATP_ALUA" -V "FreeBSD" -M "iSCSI Disk" -P "VMW_PSP_RR" -O "iops=1" -c "tpgs_on"

The setting for each individual claimed LUN through a user defined SATP Claim Rule:

/storage/lun[naa.6589cfc000000fb4cf8a83d1ffd9ec0c]/fromUser = "true"

/storage/lun[naa.6589cfc000000fb4cf8a83d1ffd9ec0c]/displayName = "FreeBSD Local SCSI/iSCSI Disk (naa.6589cfc000000fb4cf8a83d1ffd9ec0c)"

/storage/lun[naa.6589cfc0000000a211a7722cb83865b3]/fromUser = "true"

/storage/lun[naa.6589cfc0000000a211a7722cb83865b3]/displayName = "FreeBSD Local SCSI/iSCSI Disk (naa.6589cfc0000000a211a7722cb83865b3)"

/storage/lun[naa.60014058161a44edb63cd3ca1d94a0d4]/fromUser = "true"

/storage/lun[naa.60014058161a44edb63cd3ca1d94a0d4]/displayName = "SYNOLOGY iSCSI Disk (naa.60014058161a44edb63cd3ca1d94a0d4)"

/storage/lun[naa.6001405ee3d0593d61f4d3873da453d5]/fromUser = "true"

/storage/lun[naa.6001405ee3d0593d61f4d3873da453d5]/displayName = "SYNOLOGY iSCSI Disk (naa.6001405ee3d0593d61f4d3873da453d5)"

/storage/lun[naa.6589cfc0000003bb0e6b545b02f58352]/fromUser = "true"

/storage/lun[naa.6589cfc0000003bb0e6b545b02f58352]/displayName = "FreeBSD Local SCSI/iSCSI Disk (naa.6589cfc0000003bb0e6b545b02f58352)"

/storage/lun[naa.6589cfc0000005a9b84630fe075f41b2]/fromUser = "true"

/storage/lun[naa.6589cfc0000005a9b84630fe075f41b2]/displayName = "FreeBSD Local SCSI/iSCSI Disk (naa.6589cfc0000005a9b84630fe075f41b2)"

You also find the by default created Dell Core Claim rules:

/storage/PSA/MP/claimrule[0101]/type = "vendor"

/storage/PSA/MP/claimrule[0101]/plugin = "MASK_PATH"

/storage/PSA/MP/claimrule[0101]/match/vendor = "DELL"

/storage/PSA/MP/claimrule[0101]/match/model = "Universal Xport"

Please also have a look in the vmkernel.log if the claiming worked properly. You can search for this term and then make your way down (e.g. less /var/run/log/vmkernel.log → /claimrules):

2016-10-11T13:06:30.027Z cpu1:33438)ScsiClaimrule: 2397: Enabling claimrules for MP plugins.

SATP Vendor Recommendations

The following list includes vendors with their recommendations and as much as possible KB articles I could find around them.

esxcli storage nmp satp rule add -s "VMW_SATP_EQL" -V "EQLOGIC" -M "100E-00" -P "VMW_PSP_RR" -O "iops=3" -e "EQL Optimized; EQL TR1091" --force

esxcli storage nmp satp rule add -s "VMW_SATP_DEFAULT_AA" -V "3PARdata" -M "VV" -P "VMW_PSP_RR" -O "iops=1" -c tpgs_off -e "3Par A/A Optimized; HP QL226-97872" --force

esxcli storage nmp satp rule add -s "VMW_SATP_ALUA" -V "3PARdata" -M "VV" -P "VMW_PSP_RR" -O "iops=1" -c tpgs_on -e "3Par ALUA Optimized; HP QL226-97872" --force

esxcli storage nmp satp rule add -s "VMW_SATP_ALUA" -V "Nimble" -M "Server" -P "VMW_PSP_RR" -O "iops=1" -e "Nimble ALUA Optimized; Nimble KB-000059" --force

esxcli storage nmp satp rule add -s "VMW_SATP_SYMM" -V "EMC" -M "SYMMETRIX" -P "VMW_PSP_RR" -O "iops=1" -e "VMAX ALUA Optimized; EMC H2529.13" --force

esxcli storage nmp satp rule add -s "VMW_SATP_EVA" -V "HP" -M "HSV*" -P "VMW_PSP_RR" -O "iops=1" -c tpgs_off -e "EVA A/A Optimized; HP 4AA1-2185ENW" --force

esxcli storage nmp satp rule add -s "VMW_SATP_ALUA" -V "HP" -M "HSV*" -P "VMW_PSP_RR" -O "iops=1" -c tpgs_on -e "EVA ALUA Optimized; HP 4AA1-2185ENW" --force

esxcli storage nmp satp rule add -s "VMW_SATP_DEFAULT_AA" -V "XtremIO" -M "XtremApp" -P "VMW_PSP_RR" -O "iops=1" -c tpgs_off -e "XtremIO A/A RR Optimized; EMC 302-001-288" --force

esxcli storage nmp satp rule add -s "VMW_SATP_ALUA" -V "PURE" -M "FlashArray" -P "VMW_PSP_RR" -O "iops=1" -e "Pure ALUA Optimized; Pure vSphere BPG" --force

esxcli storage nmp satp rule add -s "VMW_SATP_ALUA" -V "Skyera" -M "Skyhawk" -P "VMW_PSP_RR" -O "iops=1" -e "Skyera Skyhawk ALUA RR Optimized" --force

esxcli storage nmp satp rule add -s "VMW_SATP_ALUA" -V "TEGILE" -M "ZEBI-FC" -P "VMW_PSP_RR" -c tpgs_on -e "Tegile FC ALUA; Tegile TCS" --force

esxcli storage nmp satp rule add -s "VMW_SATP_DEFAULT_AA" -V "COMPELNT" -M "Compellent Vol" -P "VMW_PSP_RR" -O "iops=3" -e "Compellent Optimized; Dell 680-041-020" --force

There are quite a few KB articles available which also describe to change the settings with Microsoft PowerShell but without getting too much into detail please take this VMware KB: 2069356 as an example as it provides a small script.

Conclusion

Now we come to the conclusion of this article. I know the article became very long, but I think what was missing was a complete article explaining both core and SATP claim rules. I didn’t want to go into the depth of ALUA or specific filter plugins for VAAI as this would have been then much longer. From experience I can tell you that a lot of people struggle with these setting and I believe that this article is able to at least change this a bit. I think I said this earlier in the article but tuning does not really always help to solve a performance issue but sometimes it definitely does. But for certain storage systems it tuning won’t make any difference. Enjoy and let me know if you have any questions, concerns or suggestions.