Video Screencast Help
Symantec to Separate Into Two Focused, Industry-Leading Technology Companies. Learn more.

iSCSI Network Showing Heavy Latency - Clears up When Disable NIC on Backup Server

Created: 25 Oct 2012 | 3 comments

I have a puzzleing issue at hand that I am looking for some help on.  I am running Backup Exec 2012 fully patched up (live update shows nothing more).  It is a physical server with one NIC on the server vlan and one NIC on the iSCSI network.  Using the iSCSI initiator in Windows 2008 R2 Standard, I have connections to each of my LUNs in read only mode and then an active full connection to one of our SAN LUNs to store the deduplication data on. 

Randomly through the day over the last 3 days, we have received complaints from our users that email and all other applications are slow.  We hop into vcenter and we see that nearly all of our servers are reporting very high disk latency.  These servers are spread out on each of our storage devices which I will break out later.  The last two times, we went to the backup server (with no evidence it was at fault) and disabled the nic for the iSCSI network.  Immediately all latency stopped and things restored to normal.  The times of the day do not line up at all and even when it is not occuring, we see latency issues on the ESXi hosts.  

We have three types of iscsi attached storage.  Left Hand P4500 SAN storage, Dell Equalogic storage, and some cheap Neatgear NAS storage.  We have not see these devices show any abnormal I/O during these high latency times.  Servers hosted on each the LeftHand and the Equalogic have shown to be affected.  We are running HP ProCurve switches for this Iscsi network with Jumo frames enabled throughout (as far as we can see)

The backup server does not have a connection to a LUN on the cheaper left hand devices, and i do not believe that servers running from those devices were affected.  I will double check on that though.  

This has occured three times that were noticeable.  THe first and second time I had jobs running.  The third time I am absolutely positive that there were no jobs running on the backup server but disabling that iscsi nic resolved the issue immediately.

We have been trying to get SAN transport mode to work and therefore have the LUNS setup on the backup server in a read only mode.  To do this i followed the article to disable auto mount and then i went into diskpart and set them as read only and then onlined them. 

What could be causing this.  We are completely mistified and need some assistance.  

Comments 3 CommentsJump to latest comment

CraigV's picture

Hi,

Enabling jumbo frames can sometimes flood a network, but it should be giving you a performance increase, not a drop...

Have you checked the VMware TN below on this:

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1007654

Thanks!

Alternative ways to access Backup Exec Technical Support:

https://www-secure.symantec.com/connect/blogs/alte...

Neil Natic's picture

Unfortunately the issue still exists.  as soon as i enable that NIC it floods the network.  Any other ideas?

I have a ticket open on it with a level 2 priority which is a 24 call back.  I got an email within 4 that said when can i reach you.  I responded immediately with I am available now and got an auto reply from the technician stating that he was out until monday.  No call monday.  got a voicemail from him Tuesday saying he would call back later today and not a word since.  

teiva-boy's picture

You will have much better support if you contact either Equallogic, VMware, or perhaps LeftHand.  Their networking teams are well versed and can assist you in your networkin issues.  I dodn't believe this is a BackupExec issue, but rather a networking/switch issue most likely.

Symantec has little networking expertise and staff to address iSCSI.  In fact, as a majority, Symantec opposes iSCSI and prefers FC and SCSI over iSCSI.  I've had way too many arguments with product management in the NBU and Storage Foundation teams to count.  

There is an online portal, save yourself the long hold times. Create ticket online, then call in with ticket # in hand :-) http://mysupport.symantec.com "We backup data to restore, we don't backup data just to back it up."