performance on Netbackup 6.5.4
Gentlemen,
I have problem to backup my data within the specified period.In fact ,I have about 6TB full data to run netbackup job every week. But according to our calcuation, it will take more 33 hours which is more than 24 hours ( 1 day ) . We are to backup via the dedicated 1Gbps ethernet link huge amount of data everyday.
In order to maximize the perfomance of netbackup ,which of the following ways is the best option to do ? ( Which one is the bottle neck of the performace ?)
( 1 ) Increase the size of Memory of the backup server ? ( Provided that memory slots are available )
( 2 ) Increase the CPU speed of the backup server?
( 3 ) Backup over the FC ( fibre channel ) link instead of gibabit ethernet ? ( in that case , which would be running the speed of the gibabit ethernet , 1Gbps , or only 80 % of 1 Gbps ?)
( 4 ) Increase the number of backup tape drives in the tape library?
( 5 ) Segregate the amount of data to backup and run the backup job concurrently?
Is there any way to compute the backup window time frame ?
Thx.
Comments
Where have you noted the bottleneck?
If the bottleneck is your network, then increasing hardware (adding drives, cpu, or memory) will not have much of an affect, nor will segregating the data. If the botteneck is network, then I would suggest utilizing FC.
Hi, Thx for your kind reply.
Hi,
Thx for your kind reply. The problem is how do I know the bottle neck is in network or in server?
More information needed
There are several specifics to a configuration you have not mentioned which affect performance:
Basically, you have to specify details of your configuration in order allow hypothesis on your bandwitdth restrictions.
There are some tweaks in the
There are some tweaks in the NBU tuning guide found on the support pages for download. Mainly for network buffers...
You could trunk your Gb links on your media server for more bandwidth to receive more data in parallel.
Are you using jumbo frames on a dedicated backup network? Perhaps think about it, since most servers have a 2nd NIC not in use in 75% of the customers I run into.
There is an online portal, save yourself the long hold times. Create ticket online, then call in with ticket # in hand :-) http://mysupport.symantec.com "We backup data to restore, we don't backup data just to back it up."
Hi Teiva-boy, Thx for your
Hi Teiva-boy,
Thx for your reply. I have dowloaded the NBU tunning guide for 6.5.4 but still cannot find out how to trunk Gb links on my media server? Right Now, I'm using the Sun Fire X4150 which has 4 NIC ports with 1Gbps enabled. But those are configured with different VLAN subnets ( etc, one port is MGMT VLAN and another port for Operation VLAN ). Could it be possible for trunking of those different VLAN NIC ports.
Pls.advise. Thx.
Could it be possible for
Could it be possible for trunking of those different VLAN NIC ports
No it's not possible. Create several trunks each in it's own VLAN.
Hi Gavrilov, Thx. As far as I
Hi Gavrilov,
Thx. As far as I know there is no specific trunking software for Redhat linux . I understood SUN has sun trunking software only for their Solaris 8,9 and 10 eventhough I want to crate trunking on one VLAN environment.Pls.correct me if i'm wrong.
Thx.
On every OS you can create
On every OS you can create trunk over only one VLAN. On Solaris you can use Sun Trunk or aggregate links. On RHEL AFAIK you also have any tools for load balance of incoming traffic.
You wrote Sun Fire X4150
You wrote Sun Fire X4150 which has 4 NIC ports with 1Gbps enabled. But those are configured with different VLAN subnets ( etc, one port is MGMT VLAN)
But this server has a dadicated network managmet port and you shouldn't loose one NIC for managment
Hi Gavrilov, Thx. In fact 3
Hi Gavrilov,
Thx. In fact 3 NICs has 3 VLANs. ( MGT VLAN, Operation VLAN and Backup VLAN , etc ).
Hi, Thx and appreciate your
Hi,
Thx and appreciate your advise. Since I have atleast 3 VLANs connected to 3 diff NICs ( altogether is 4 NICs ). I don't think i can trunk different vlans to one trunk ports. Thanks.
Yes , 3 NICs +3 VLANS=No
Yes , 3 NICs +3 VLANS=No trunks :)
How many files?
That is a goodly amount of data, how many files is it roughly? Netbackup doesn't like millions of files at all.
Have you considered going with Synthetic backups?
Hi Nathan, Thx for kind
Hi Nathan,
Thx for kind reply.Sorry for my late reply. I'm trying simulate the real environment.. Hve setup the 2 SVr , one for NBU Master and one NBU client . Each has 1 TB of Disk space mounted to one ISCSI SAN storage. When I try to backup the files ( 3 GB x 300 ) in the client to the SVR NBU , the speed is quite disappointing. ( About 6 Hrs ). But the production, about 5 TB of FULL backup has to be done. within 24 HRs.
SAN is connected to both svrs with 1 Gbps ethernet link.
Hi Nathan, Thx. In fact
Hi Nathan,
Thx. In fact binary files need to be backup. Is there limitation that the number of files that netbackup support? According to the Synthetic backup, need storage space. But i m wondering how to backup the full amount of 5 TB data every week within 24 hours. This is my concern.
Schedules to include in a policy for synthetic backups
A policy for synthetic backups must contain at least three types of schedules:
■
At least one traditional, full backup must be run successfully to create a full image. The synthetic backup jobs fails if there is not at least one previous full image
Yes netbackup does like more files based on your policy type
Flash backup can do magic in backing up files more in count
check the admin guide to know about it
Thanks, Karthikeyan Sundaram.
It's a separate solutions but
It's a separate solutions but you can performe all of them together:
1. Create a trunk on Media server no increase LAN speed.
2. Use SAN-clients for backup (it's a better solution)
3. Use DSSU to perform a bakcup during weekend and after that you can relocate this images to tape.
4. If you using LTO3 upgrade up to LTO4.
6. If you using 1G insure that you enable jumbo-frames.
Jambo-frame
Hi Gavrilov,
Sorry for late reply. My reponse is below.
1 . I have the limitation on switch ports. But haven't tried yet. Pls.advise any of solution to configure the trunk port on my Redhat Ent 5.
http://www.cyberciti.biz/tips/howto-configure-linux-virtual-local-area-network-vlan.html
http://www.linuxjournal.com/article/7268
2 .According to the approved design, the backup to be done only on ethernet interface due to ORACLE ARMAN .
3. In order to use the DSSU , we may need the some storage space but we have tight space on the SAN to store the image.
4. We'are planning to use the LTO4 drive.
5. I have setup the jambo-frames but the performance is that significant yet.
Right now, I am tesing the netbackup performance in the development environment . Previously , I used the linksys switch to connect to the SAN interface of the server. ( I'm using ISCSI in my development , eth0 is connected to backup LAN and eth2 is connected to the SAN on each master NBU server and netbackup client server. ) I have recently used the cisco switch and set the jambo-frames in both NIC in the server and Cisco switch . But the performance on 100 GB data is not that significant without/ with using jambo-frames as well as Linksys switch and Cisco Switch. But the data is 3 GB x 33 nos of binary files , But I noticed that netbackup transformed those 33x 3GB binary files to about 100 GB tar file. I believe it take some time to transform these files to one single 100 GB file but there is no bottle neck in the network. Should i upgrade RAM?
Make a simple test -
Make a simple test - transfer 1G file via ftp/sftp, NFS. check practical speed. I never see 1G LAN speed more then 65 MB/sec.
According to my observation
According to my observation Gigabit Ether net trunking is not an option for backup. We tentatively installed this configuration on one of our Oracle DB clients, but that didn't helped to boost the backup performance.
I think the better solution is to :
a) convert your client to SAN media server
b) use 3-4 LTO3 or, better, LTO4 drives to backup this SAN media server.
Please don't forget to tweak NetBackup buffers on that SAN media server.
I presume you are already familiar with NetBackup Tuning Guide.
Please tell me how you tried
Please tell me how you tried to do it (OS, trunk version, iopolicy) and why you decide that this doesn't work.
jambo-frame and performance tunning
Hi,
Thanks for your kind advise. According to the approved design, we are limited to use the ethernet backup only and definitely we will use the LTO4drive. but right now I m testing the performance of the NBU on D2D using the jambo-frames but the performance is not that significantly improved yet.
I still need to go through all steps of performance tunning guide.
Thx.
What speed you expect to get
What speed you expect to get between client and media server? And what speed do you have now?
I agree with Garilov... 6
I agree with Garilov...
6 TB of data is quite a lot for any one client, no matter how large our NetBackup environments are.
If had to backup 6 TB every night, then I'd definitely be looking at adding some HBAs to the client, getting some NetBackup SSO licences and sharing with the normal media servers.
1 Gb/s
1000 Mb/s
125 MB/s
60% efficiency
75 MB/s max practical
6 TB
6,144 GB
6,291,456 MB
83,886 seconds
23 hours
Based on these numbers you'll be lucky to get the job completed in 35 hours, let alone 23 - as the above assumes that the disks + the client + the LAN + the media server + the SAN + the tape drives are all not busy doing other things - and it is highly unlikely that even just two of these components are idle at any given time.
Also consider, if it's 6 TB now, what will it be next year, and the year after that.
IMHO - a bigger box for the client, two HBA ports per fabric (e.g. two dual port cards), 4 x LTO4 (i.e. 2 x LTO4 per fabric), and very carefully balance the file systems and disk groups with your NetBackup policy path selections and/or database elections - you'll need to work very closely with your DBAs.
Pay careful attention to the "Backup Planning Guide" - and consider how busy the client hardware will be supporting all this additional work load - i.e. how much CPU, RAM and disk space you need just for NetBackup.
HTH,
Dave.
I would aim to complete the
I would aim to complete the backup in under six hours, to allow for maintenance windows, down time, day to day problems, backup re-runs, restores, other works on either client, media server, LAN, SAN, library, tape drives, O/S upgrades, S/W patches, firmware updates, etc etc - i.e. someone else will always be delaying your backups!
90 MB/s per LTO4 practical
4 drives
360 MB/s
2880 Mb/s
2.81 Gb/s
Therefore 2 x 4 Gb/s FC paths are required for redundancy
6 TB
6,144 GB
6,291,456 MB
360 MB/s
17,476 seconds
5 hours
Also, reading above, it still wasn't clear to me how many files are involved. Someone else picked up on this too. If this is one huge database server then it's probably not an issue, but if this is a file server with millions or even tens of millions of small files, then you will highly likely need to use the FlashBackup capabilities of the Snapshot Client option (which is within the "NetBackup Enterprise Client" licence).
HTH.
LTO-3 is probably fast enough
"90 MB/s per LTO4 practical"
There's really no need to go to LTO-4 if LTO-3 will do for speed. LTO-3 speed is 60MB/s native & 120MB/s compressed.
If there is a problem with data coming into the media server then you should NOT go to a faster tape drive, you will only end up starving it for data even more than a slower drive.
The best thing would be to backup to disk and from there to tape. But, it sounds like there is not much budget for that possibility based on the comments by the OP.
If your media server can only ingest 150MB/s, then it can optimally only drive 2 LTO-3 drives at best.
Remember.....Google is your friend!
One final point - NetBackup
One final point - NetBackup does scale! It just takes time to work it all out.
You will have to do your math for each stage of data producer (clients) and data consumers (media servers) - or in your case both in the same box (i.e. the client itself) - and think about the performance profile, capabilities and limititations of each single component/stage/hop along the entire data path end-to-end, e.g.:
disks - RAID - shelves/controllers/heads - storage HBAs - SAN path - switch - SAN path - client HBAs - CPU/memory - NetBackup buffers - client HBAs - SAN path - switch - SAN path - tape HBAs - tape drives - tapes.
A word of advice - do not be tempted to zone both your FC disk and your FC tape through the same HBAs. I believe that the standard recommendations still are not to mix disk and tape traffic. So, I recommend additional separate HBAs in the client specifically for backup traffic to tape.
An example of scaling, we have four media servers - each has:
NICs - Eight - in a 2 trunks of 4 Gb/s in IPMP failover config.
HBAs - Seven dual port HBAs - i.e. 14 FC connections per media server - thus 7 on each fabric.
Zoning - Carefully patched and zoned so that all backup SAN traffic always stays within the ASICs of the SAN switches - i.e. the backup FC traffic never enters the backplane of any SAN switch nor does the backup FC traffic traverse switch to switch. We have disk storage arrays, VTLs and tape libraries connected to our media servers hence the number of FC connections in each. There are four media servers as we determined that site resilience, site H/A and site DR were all important factors.
Your business requirements may well be different.
6 TB of data is not inconsequential.
Good luck.
Ok. Look, you have 3 X 1G
Ok.
Look, you have 3 X 1G NICs (3 VLANs). What speed do you get for this NICs? Something about 3X50(MB)=150 MB/sec. Speed between Media-server and TapeLibrary is 360 MB/s. As you understand speed between clients and mediaserver isn't enough. You have to increase a LAN speed for example by adding new NICs (Adding Quad Ethernet card is a cheapest way) or use SAN-client/SAN-media server (it's more expensive).
P.S. Have you check your Pramble size?
The simples way to increase
The simples way to increase your perforance is sit down, draw your topology, mark every link (LAN/SAN) and write it's theoretically speed. Most bottlenecks will be filtered on this stage. Next step - begin test every link to understand is it new bottlenecks? What is the way to make it better?
If you want us to do it draw the picture of your environment, mark the connection speeds and we'll look at it.
Now we are just cheating and do nothing :)
NBU Development Environment
Hi ,
Kindly refer the attached drawing.The performance is as follow.
RCP = remote file copy command in Red Hat Linux ENT5.
Advise
Hi Gavrilov,
I'm sorry for late reply. Kindly advise base on the following info I've provided.
with LTO-4 it should be an
with LTO-4 it should be an achievable goal, you'll need to maximize I/O
we run netbackup on a Proliant DL380 G5 server. it's i/o is around 3gbps. g6 servers are double that. a cheapo G6 server will run $2500
we use 2 gigabit NIC's teamed together
for the clients teaming Gbps NIC's will also help a lot
we're an HP shop but the tech is pretty much all the same no matter who you buy from. G5 servers have much faster I/O than G4 or earlier generation. upgrading your clients to G6 servers which is what is out now will speed up your backup times.
Hi sql-noob, Thx for your
Hi sql-noob,
Thx for your kind advise. I'm currently using the SUN FireX4150 SVR with quad core memory and 8GB or RAM. According to the Design the server spec has been approvd and i can't change unless i upgrade the memory . But I noticed that during the backup/archive time , the CPU and memory utilization is not that high. I' m not very sure that my server is G5 or G6 . May I know what is the difference ?Will try the "bonding "on the NIC again. It's that significantly improved performance ? I ve tried the jambo frame but not that significantly improved .
Would you like to reply?
Login or Register to post your comment.