iSCSI Network Showing Heavy Latency - Clears up When Disable NIC on Backup Server
I have a puzzleing issue at hand that I am looking for some help on. I am running Backup Exec 2012 fully patched up (live update shows nothing more). It is a physical server with one NIC on the server vlan and one NIC on the iSCSI network. Using the iSCSI initiator in Windows 2008 R2 Standard, I have connections to each of my LUNs in read only mode and then an active full connection to one of our SAN LUNs to store the deduplication data on.
Randomly through the day over the last 3 days, we have received complaints from our users that email and all other applications are slow. We hop into vcenter and we see that nearly all of our servers are reporting very high disk latency. These servers are spread out on each of our storage devices which I will break out later. The last two times, we went to the backup server (with no evidence it was at fault) and disabled the nic for the iSCSI network. Immediately all latency stopped and things restored to normal. The times of the day do not line up at all and even when it is not occuring, we see latency issues on the ESXi hosts.
We have three types of iscsi attached storage. Left Hand P4500 SAN storage, Dell Equalogic storage, and some cheap Neatgear NAS storage. We have not see these devices show any abnormal I/O during these high latency times. Servers hosted on each the LeftHand and the Equalogic have shown to be affected. We are running HP ProCurve switches for this Iscsi network with Jumo frames enabled throughout (as far as we can see)
The backup server does not have a connection to a LUN on the cheaper left hand devices, and i do not believe that servers running from those devices were affected. I will double check on that though.
This has occured three times that were noticeable. THe first and second time I had jobs running. The third time I am absolutely positive that there were no jobs running on the backup server but disabling that iscsi nic resolved the issue immediately.
We have been trying to get SAN transport mode to work and therefore have the LUNS setup on the backup server in a read only mode. To do this i followed the article to disable auto mount and then i went into diskpart and set them as read only and then onlined them.
What could be causing this. We are completely mistified and need some assistance.