iSCSI Multipath for Equallogic round 3

I posted a rather cryptic set of commands way back when, followed it with a script from Dell that would setup multipath on vSphere 4.x for you, then followed that with setting up MPIO in the GUI.

Now I’m back with a twist. Seems Equallogic published a white paper back in November about MPIO with vSphere5 (note that 4.x has the same issue which VMware mentions but the NetApp doc doesn’t). VMware followed that with a KB article so here’s the 30-second version.

Setting up iSCSI MPIO with two VMkernel ports each with a dedicated NIC can have problems. Equallogic (unlike other iSCSI vendors which typically use the iSCSI nop commands) uses ICMP for iSCSI management such as login/logout. By default VMware uses the loweset-numbered vmkernel port on a particular network to respond to ICMP.

So in your Equallogic iSCSI setup, you’ll have vmk1 and vmk2 each configured with dedicated NICs. If the NIC for vmk1 goes down – no more ICMP and the Equallogic will have login and path-determination issues.

The fix is to set the first VMKernel port on the switch to use both NICs (ie like a normal VMkernel port), then set up two additional VMkernel ports like you would for iSCSI MPIO. Note that you will not do any further configuration with the lowest numbered port – it cannot be used for iSCSI binding and should not be included on the “access” list on the Equallogic.

VMkernel will respond to any pings to the second two VMkernel ports (vmk2 and 3 in the example) from the first one (vmk1), which will have redundant network connectivity. Note that since it is all on the same switch, losing both NICs will kill all iSCSI and ICMP returns.

Since the second two ports will have “iSCSI Port Binding” enabled iSCSI traffic will be dedicated to those two ports only. You will never see iSCSI traffic on the lowest numbered VMkernel port.

This entry was posted in Equallogic, Storage, Virtualization, VMware. Bookmark the permalink.

11 Responses to iSCSI Multipath for Equallogic round 3

  1. Ken says:

    When you mention “NIC goes down” is this a hardware failure of the NIC port or just any transient condition where the NIC port will return to service?
    Reading the papers at the end of this link http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=20078 has me confused.

    Do we create a new VMKERNEL port and somehow unbind the others so this new “Highly Available Storage Heartbeat port is the lowest VMKERNEL port? Per the doc it has to be the lowest VMkernel port on the vSwitch and is not bound to the software initiator. We currently have four 1-1 mapped VMkernel ports for iSCSI

    We have recurring “Failed on Physical Path” errors in our esxi (4.1 U2) host logs that trigger LSI_SAS (129)errors on the Windows 2008R2 SP1 servers. I’ve landed on this trying to determine if these symptoms are related to this heartbeat issue. Any insight would be greatly appreciated.

    VMware and EQ have yet to find anything wrong, but they have not mentioned this yet. We are running the latest ESXi 4.1 U2 builds and the latest EQ firmware. We are also running the latest EQ MEM.

    • JAndrews says:

      Anything that would prevent the lowest numbered vmk from accessing the network, since it is used to respond to ICMP requests on that network and EQ used ICMP for multipathing administration.

      You would want the lowest numbered VMkernel port on that network to not have an iSCSI bind.

      Assuming you have four VMkernel ports on one vSwitch with four physical nics and a one-to-one iSCSI binding:
      locate the lowest numbered VMkernel port (vmkx where “x” is the lowest number on that network)
      make a note of the VMK name
      make a note of the IP address
      make a note of the physical nic it was bound to
      unbind the vmk
      change the IP address of this port to a new address on that network.
      You may as well enable management on this VMK and rename it appropriately
      Create a new vmkernel port and give it the IP address and name you noted earlier
      Bind the new vmk to the pnic noted earlier

      If your problem is that there are network issues associated with the pnic bound to the lowest numbered vmk then this gives that VMK three additional paths to respond to ICMP requests.
      Do you have a managed switch between the host and the Equallogic that can report on traffic issues?

      • Ken says:

        Thanks for the information. Two Cisco 4948’s 1G are between. I’ve not found anything obvious from that side (drops etc.). Everything is setup to my knowledge with the best practice guides in mind.

  2. JAndrews says:

    Are the errors constant or sporadic? After making sure the lowest numbered VMK isn’t bound can you eliminate some physical paths? drop all but one on the EQ and all but one on the host, then add back until the problem resurfaces?

    • Ken says:

      Sporadic, but occurs across all VM’s which reside on two ESXi hosts.
      Right now 2 ports go to one of the 4948’s and the other two to the other 4948.
      The four ports are sourced from two different Physical NIC’s on the host, and staggered, for redundancy.

      Here is a snip from the messages log when it happens. We just recently put in the Dell MEM, so those messages are new, we still had the path failures before doing that in an attempt to resolve. Once this hits, the Server 2008R2 VM’s show the LSI_SAS(129) in the event viewer (reset to device).

      messages log

      Jul 26 17:14:56 vmkernel: 5:20:47:47.959 cpu2:6174)WARNING: dell_psp_eql_routed: PspEqlCheckScsiStat: Bad SCSI status: device stat 2, host stat 0, plugin stat 0
      Jul 26 17:14:56 vmkernel: 5:20:47:47.959 cpu2:6174)WARNING: dell_psp_eql_routed: PspEqlSetMpioId: Error setting Mpio Session ID on path 19
      Jul 26 17:15:45 vmkernel: 5:20:48:37.238 cpu19:19227)VSCSI: 2252: handle 8542(vscsi0:0):Reset request on FSS handle 3741097 (1 outstanding commands)
      Jul 26 17:15:45 vmkernel: 5:20:48:37.238 cpu2:4247)VSCSI: 2526: handle 8542(vscsi0:0):Reset [Retries: 0/0]
      Jul 26 17:15:45 vmkernel: 5:20:48:37.239 cpu4:4860)WARNING: iscsi_vmk: iscsivmk_TaskMgmtAbortCommands: vmhba38:CH:1 T:0 L:0 : Abort task response indicates task with itt=0xf2a19 has been completed on the target but the task response has not arrived
      Jul 26 17:15:45 vmkernel: 5:20:48:37.239 cpu2:4247)NMP: nmp_CompleteCommandForPath: Command 0x28 (0x4102be9d0240) to NMP device “naa.6090a02860e2e031f1f85431a6b6522c” failed on physical path “vmhba38:C1:T0:L0” H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
      Jul 26 17:15:45 vmkernel: 5:20:48:37.239 cpu2:4247)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe: NMP device “naa.6090a02860e2e031f1f85431a6b6522c” state in doubt; requested fast path state update…
      Jul 26 17:15:45 vmkernel: 5:20:48:37.239 cpu2:4247)ScsiDeviceIO: 1688: Command 0x28 to device “naa.6090a02860e2e031f1f85431a6b6522c” failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
      Jul 26 17:15:45 vmkernel: 5:20:48:37.239 cpu2:4247)VSCSI: 2326: handle 8542(vscsi0:0):Completing reset (0 outstanding commands)

      I give your suggestions a go. Thanks

  3. Phil says:

    The latest MEM 1.1 appears to do this automatically with a “Storage Heartbeat” vmk added via the script….can you confirm?

  4. Matt says:

    Hey, I just want to say thanks!

    This “Equallogic (unlike other iSCSI vendors which typically use the iSCSI nop commands) uses ICMP for iSCSI management such as login/logout. By default VMware uses the loweset-numbered vmkernel port on a particular network to respond to ICMP.” solved a problem where we were not able to vmkping our equallogic ips, dynamic discovery was not working, and we were getting a tonne of these errors:

    Oct 28 13:19:29 ATT1-VH30 vmkernel: 7:21:35:14.267 cpu4:4303)WARNING: dell_psp_eql_routed: PspEqlCheckScsiStat: Bad SCSI status: device stat 0, host stat 5, plugin stat 0
    Oct 28 13:19:29 ATT1-VH30 vmkernel: 7:21:35:14.267 cpu4:4303)WARNING: dell_psp_eql_routed: PspEqlIpcXceive: Error writing request: 0xbad0001
    Oct 28 13:19:29 ATT1-VH30 vmkernel: 7:21:35:14.267 cpu4:4303)WARNING: dell_psp_eql_routed: PspEqlIoctlReconfigRequest: Error occurred in IPC, not copying response
    Oct 28 13:19:39 ATT1-VH30 vmkernel: 7:21:35:24.677 cpu2:4303)WARNING: dell_psp_eql_routed: PspEqlCheckScsiStat: Bad SCSI status: device stat 0, host stat 5, plugin stat 0
    Oct 28 13:19:39 ATT1-VH30 vmkernel: 7:21:35:24.677 cpu2:4303)WARNING: dell_psp_eql_routed: PspEqlIpcXceive: Error writing request: 0xbad0001
    Oct 28 13:19:39 ATT1-VH30 vmkernel: 7:21:35:24.677 cpu2:4303)WARNING: dell_psp_eql_routed: PspEqlIoctlReconfigRequest: Error occurred in IPC, not copying response
    Oct 28 13:19:39 ATT1-VH30 vmkernel: 7:21:35:24.677 cpu2:4303)WARNING: dell_psp_eql_routed: PspEqlIoctlReconfigRequest: Returning error status 0xbad0001
    Oct 28 13:22:06 ATT1-VH30 vmkernel: 7:21:37:51.445 cpu4:4305)WARNING: dell_psp_eql_routed: PspEqlCheckScsiStat: Bad SCSI status: device stat 0, host stat 5, plugin stat 0
    Oct 28 13:22:06 ATT1-VH30 vmkernel: 7:21:37:51.445 cpu4:4305)WARNING: dell_psp_eql_routed: PspEqlIpcXceive: Error writing request: 0xbad0001
    Oct 28 13:22:06 ATT1-VH30 vmkernel: 7:21:37:51.445 cpu4:4305)WARNING: dell_psp_eql_routed: PspEqlIoctlReconfigRequest: Error occurred in IPC, not copying response

    vmk0 was not connected, once I unbound it and deleted it, everything worked perfectly!

  5. Don Williams says:

    FYI: This issue is resolved in ESXi v5.1.

  6. Don Williams says:

    MEM can create the Storage Heartbeat VMKernel port on an empty iSCSI configuration. However, if it’s setup already, in ESXi v5.0 it’s not hard to unbind the lowest VMK port and re-use it for the SHB VMK port. That VMK port, not being bound to the iSCSI adapter will always be able to respond to the PINGs. Since ESX has a different stack for TCP/IP and iSCSI. That VMK port will have access to all the ports being used for iSCSI. So as long as one is up and responding, iSCSI logins will function. Since when a NIC (or switch fails) those sessions will die. The initiator will then try to restore them via the surviving links.

    Re: PING. The reason EQL uses PING during login it to make sure that the port on the array it wants to use for the connection is able to reach the requesting initiator port. Since the logins actually occur to a physical port IP address, not the Group IP address.

Leave a Reply