Video Screencast Help
Symantec Appoints Michael A. Brown CEO. Learn more.

BackupExec 2010 w/ Win2k8 x64 and IBM TS3100 running slow

Created: 11 Dec 2010 | 13 comments

Hello,

 

I recently installed BackupExec 2010 R2 in Windows 2008 x64 server.

I'm using TS3100 over SCSI/LVD connect (directly to the server, not storage)

My local storage is netapp fas2020, both server and storage ethernet is bonded to 2gbit.

HBA is HP(LSI) 2000 series SCSI320.

 

During product evaluation i tried the following configuration:

 

1. Backup from disk to tape

2. Backup from NDMP (over ethernet) to tape

 

Both tests shows the following results:

1. NDMP to tape never passed 1000mb/m

2. Disk to tape never passed 1500mb/m

 

I'm using Symantec drivers for the tape library, and Microsoft drivers for HBA.

block size is set to 512KB, Buffer is 1MB, buffer count is 32 and high water count is 30.

 

Backups through NDMP starts very slow, after a while it reaches a maximum speed of 1000mb/m

Comments 13 CommentsJump to latest comment

ZeRoC00L's picture

What kind of drive(s) are located in the IBM TS3100 ?

If this response answers your concern, please mark it as a "solution"

Tal Hazan's picture

I was following BackpExec KB to remove vendor drivers and replace with unknown tape library drivers.

Tal Hazan's picture

Unknown Media Changer - Microsoft - ID5 LUN1

IBM ULTRIUM III 3580 TAPE DRIVE - Symantec - ID5 LUN0

LSI Adapter - Ultra320 SCSI 2000 Series w/1020/2030

teiva-boy's picture

First off, you need to understand that NDMP performance has nothing to do with BackupExec in 90+% of cases.  NDMP is a set of commands executed remotely by the backup server, telling the NDMP device what to do.

In the case of a NetApp filer, BE tells the FAS2020 to backup a certain LUN, in which the controller head does a UFS_DUMP, and streams that dump over the network or via FC to your target.  Thus, in many cases, you are limited by the speed of your disk, bandwidth, and the CPU power of the filer itself.  BackupExec only receives the stream as fast as your filer can create and spit it out.

There are commands to verify the throughput of the filer that you can run via CLI within the ONTAP OS.  I dont recall them off-hand, but if I remember later, I have it in some notes somewhere, I'll post an update.  This gives you a good baseline for what your device *could* theoretically put out via NDMP.

FYI, the FAS2020, is one of the lower powered models, with I believe a single Intel Mobile Celeron processor and just a single GB of memory in its base config (my 2yr old laptop is more powerful than your filer).  The 2050 too is equally a lower model, even though it has a higher model #.  The best in the FAS2000 lineup is the FAS2040, with a Xeon processor, and more RAM.

If you need to test your tape throughput, the baseline you can do is to copy a few hundred GB's of mixed data (both large files, and little ones) to your local HDD's on your media server, and back that up, to get a better idea of the throughput of your tape.

There was also a posting in the BLOG section I believe in the forums for tuning LTO, that would be handy for you to look at as well to verify your tape settings are the most optimum for your particular setup. https://www-secure.symantec.com/connect/articles/tuning-my-lto4-tape-drive

Once you've identified how fast your tape can write data, and then figured out how fast the FAS2020 can read and dump data; more than likely you may need to play with aggregates within the filer to get more performance.  

Note, there are a number of posts on the NetApp forums about FAS2020 performance issues. Best to check some of those as well, to see what was done to reduce the CPU load on this particular filer.

There is an online portal, save yourself the long hold times. Create ticket online, then call in with ticket # in hand :-) http://mysupport.symantec.com "We backup data to restore, we don't backup data just to back it up."

Tal Hazan's picture

Teiva-boy, thank you for the detailed repond.

 

I also tried backing up C locally to tape, never passed 1200MB/m.

 

I also updated the library and LTO3 drive firmware to latest. I will see if there's any imporovement.

 

I'm suspecting several things that might cause the performance hit:

 

1. Low quality cable

2. AntiVirus?

3. Old Tapes

CraigV's picture

Hi Tal,

 

I take it that RSM is also disabled on your media server? Generally that service causes issues with BE.

Alternative ways to access Backup Exec Technical Support:

https://www-secure.symantec.com/connect/blogs/alte...

teiva-boy's picture

The command from the ONTAP is to perform an "ndmpcopy."  Just read-up on the correct syntax to use in the help files of via netapp support.

AV should most def be excluded on the entire BE directory where it's installed, the running beremote.exe process, any B2D folders, and the BE SQL instance.

Tapes can be long formatted, but I dont think it'll make a difference IMO.

Since you are on LTO3, you may want to play with those tape block and buffer sizes.  In my link, I posted, those numbers dont match to yours.  Not that I expect them to, as each vendor is different, but following the methodology in the article, you can see if you can get more performance writing from C:\ to tape.

There is an online portal, save yourself the long hold times. Create ticket online, then call in with ticket # in hand :-) http://mysupport.symantec.com "We backup data to restore, we don't backup data just to back it up."

Tal Hazan's picture

It seems that NDMP is not the issue.

 

I was able to identify that there's a bottleneck with Server to Tape communication following those steps:

 

1. Changed registry value to allow 254 I/O's on scsi card.

2. Downgraded SCSI drivers from Microsoft's to LSI.

 

Performance maxed to 1300MB/s quite fast. but never passed 1450MB/s.

 

I'm going to check the card configuration today (BIOS) and try another cable.

AV is disabled.

teiva-boy's picture

Great news!  It's nice to see you are getting quantitative gains with some simple troubleshooting.

There is an online portal, save yourself the long hold times. Create ticket online, then call in with ticket # in hand :-) http://mysupport.symantec.com "We backup data to restore, we don't backup data just to back it up."

Tal Hazan's picture

Yep, Thank you !

 

Any chance you know if there are compatability issues with HP SCSI HBA (SC11XE) and IBM's tape libraries?

 

I was trying to get into the card BIOS' but was getting an error about undentified tape drives.

 

Means that i cannot configure the cards bios...

teiva-boy's picture

The firmware for that card is pretty dated, and hasn't changed since 2006 it looks like.  I'm surprised you were able to get the LSI drivers working on an HP card, it's been mixed in my experience.  But more often than not, the actual hardware drivers are updated more often than the OEM reseller or MS's default ones.

As for getting into the BIOS I'm not sure, I have no direct experience with that model card.

There is an online portal, save yourself the long hold times. Create ticket online, then call in with ticket # in hand :-) http://mysupport.symantec.com "We backup data to restore, we don't backup data just to back it up."

Tal Hazan's picture

Yes, My guess is that this card is a bit problematic.

 

Now, after applying some windows 2008 network tuning, reboot both the server and the tape library (after firmware upgrade) I'm getting 2.7GB/m - NDMP via ethernet to the tape drive.

 

Better, but not there yet.