Video Screencast Help
Search Video Help Close Back
to help
New in the Rewards Catalog: Vouchers for "Symantec Technical Specialist" and "Symantec Certified Specialist" exams.

Netbackup 6.5.4 + Sun Unified Storage 7410: Slow backup performance with large volumes and many files

Updated: 21 May 2010 | 3 comments
AdNovum's picture
0 0 Votes
Login to vote

Hi,

We have a Sun Unified Storage 7410 system with 16T disk capacity that we want to backup via NDMP over 1Gb LAN. Our Netbackup 6.5.4 backup server runs on Solaris 10 on a Fujitsu Primepower 450 and has a fibre attached Quantum PX 506 LTO tape library. Two drives are used for this policy.

We have a backup policy vor each volume. Example:

/export/adnorbck    0.6 T   (mostly large files, 19784 inodes)
/export/archive        1.1 T   (many small files, 12672510 inodes)
/export/backup        0.9 T  (mixed small and large files, 896027 inodes)

The volume /export/adnorbck is backed up fine with a mostly constant speed of about 60 MB/s. But the volumes /export/archive and /export/backup have very slow write performance (ca 6 MB/s). We tried to back up the volume /export/backup twice but every time it seems to stop at the same position (330361290 kB and 12000 files written). The backup status says:

07/16/2009 08:24:00 - requesting resource adnbackup-hcart3-robot-tld-0
07/16/2009 08:24:00 - requesting resource adnbackup.NBU_CLIENT.MAXJOBS.adnnfs05
07/16/2009 08:24:00 - requesting resource adnbackup.NBU_POLICY.MAXJOBS.adnnfs05
07/16/2009 08:24:00 - granted resource adnbackup.NBU_CLIENT.MAXJOBS.adnnfs05
07/16/2009 08:24:00 - granted resource adnbackup.NBU_POLICY.MAXJOBS.adnnfs05
07/16/2009 08:24:00 - granted resource KK3315
07/16/2009 08:24:00 - granted resource HP.ULTRIUM3-SCSI.001
07/16/2009 08:24:00 - granted resource adnbackup-hcart3-robot-tld-0
07/16/2009 08:24:03 - estimated 0 kbytes needed
07/16/2009 08:24:04 - started process bpbrm (pid=21462)
07/16/2009 08:24:04 - connecting
07/16/2009 08:24:04 - connected; connect time: 0:00:00
07/16/2009 08:24:06 - mounting KK3315
07/16/2009 08:25:19 - mounted KK3315; mount time: 0:01:13
07/16/2009 08:25:19 - positioning KK3315 to file 2
07/16/2009 08:26:27 - positioned KK3315; position time: 0:01:08
07/16/2009 08:26:27 - begin writing
07/17/2009 05:12:23 - Error ndmpagent (pid=21467) aborting operation - no mover progress
07/17/2009 05:12:23 - Error ndmpagent (pid=21467) NDMP backup failed, path = /export/backup
07/17/2009 05:17:17 - Error bpbrm (pid=21462) socket read failed: errno = 62 - Timer expired
07/17/2009 05:22:17 - Error bpbrm (pid=21462) socket read failed: errno = 62 - Timer expired
07/17/2009 05:22:17 - Error bptm (pid=21468) media manager terminated by parent process
07/17/2009 07:01:18 - end writing; write time: 22:34:51
termination requested by administrator (150)

We cancelled the backup at 07:01 because nothing is written.

Does somebody have a clue what is going wrong here?

Thanks in advance for help.

Regards,
Bernd

discussion Filed Under:

Comments

schmaustech's picture
17
Jul
2009
0 Votes 0
Login to vote

Stalling Backup

Does the Sun Unified Storage 7410 have verbose logging you could possible turn on so you can see at what state the job dies?  Specifically I would be looking to see if it is stopping at the same file/directory every time.

If the system is constantly stopping in the same position everytime, I would think there is a specific file/directory that is causing the hang up.  If you could determine or approximate what file/directory, then exclude it and attempt your backup, it could confirm what I suspect.

I have seen this behavior when trying to backup up files related to a Hyperion Database.  They would cause the job to stall and eventually timeout out.  Excluding the databases, allowed the backup to continue without issues.  

Regards,

Benjamin Schmaus

Yasuhisa Ishikawa's picture
20
Jul
2009
0 Votes 0
Login to vote

I've heard about similar issue

I've heard about similar issue caused by heavy load on Sun NAS.
Can you configure your backups so that backup targets are devided into some smaller pieces, and run them in series - not simultaneously?

Authorized Symantec Consultant(ASC) Data Protection in Tokyo, Japan

AdNovum's picture
02
Aug
2009
0 Votes 0
Login to vote

Hi, Splitting the volume into

Hi,

Splitting the volume into smaller pieces helps, but the backup still hangs at the same file and byte count. Seems to be a bad file. I'll look if I can enable debugging.

About the slow throughput issue: This is the answer from a Sun supporter for the Sun storage J7410

File Size -  Single Thread Throughput (GigE)
=============================
8k - 4.93MB/sec
64k - 22.3 MB/sec
128k - 33.8 MB/sec
1mb - 55.7 MB/sec
1gb - 58.4 MB/sec

Best regards,
Bernd