Video Screencast Help
Symantec to Separate Into Two Focused, Industry-Leading Technology Companies. Learn more.

Help: Very slow backup

Created: 22 Apr 2013 • Updated: 30 Apr 2013 | 13 comments
This issue has been solved. See solution.

Have a Windows VM Client (Windows 2003 x64) that has about 30GB of data in it, but it takes forever to backup. It renders a very low througput. Can you guys point me to the steps to troobleshoot this kind of situation?

 

My Master is a SLES 10 SP3 running NBU 7.1.0.4, no disk staging, backups go directly to LTO4 tape drives using fibre.

Operating Systems:

Comments 13 CommentsJump to latest comment

mph999's picture

Sure, first thing, for a backup that competes successfully for this client (although slow) please copy/ paste the details from the Details tab for the job in activity monitor.

I would like to see the lines that look like ;

Waited xx times for full buffer, dalayed yy times

Waited xx times for empty buffer, dalayed yy times

I presume, there is only one client that is slow, and that you are backing this up with NBU client software installed on it (so backed up as a regular client).

Also, has this backup every 'worked', if so, are you aware of anything that changed.

Martin

 

Regards,  Martin
 
Setting Logs in NetBackup:
http://www.symantec.com/docs/TECH75805
 
luisvidal's picture

mph999,

 

Sadly, I only have on record one 'Completed' backup (see below):

============================================================================

04/22/2013 20:40:13 - Info bpbrm (pid=21148) NBU-CLIENT is the host to backup data from
04/22/2013 20:40:13 - Info nbjm (pid=8839) starting backup job (jobid=129359) for client NBU-CLIENT, policy NBU-POLICY, schedule INCR-D
04/22/2013 20:40:13 - estimated 590886 kbytes needed
04/22/2013 20:40:13 - Info nbjm (pid=8839) started backup job for client NBU-CLIENT, policy NBU-POLICY, schedule INCR-D on storage unit NBU-MASTER-hcart-robot-tld-0
04/22/2013 20:40:14 - Info bpbrm (pid=21148) telling media manager to start backup on client
04/22/2013 20:40:14 - Info bptm (pid=21156) using 65536 data buffer size
04/22/2013 20:40:14 - Info bptm (pid=21156) using 12 data buffers
04/22/2013 20:40:14 - Info bpbrm (pid=21148) spawning a brm child process
04/22/2013 20:40:14 - Info bpbrm (pid=21148) child pid: 25731
04/22/2013 20:40:16 - Info bpbrm (pid=21148) sending bpsched msg: CONNECTING TO CLIENT FOR NBU-CLIENT_1366677613
04/22/2013 20:40:16 - connecting
04/22/2013 20:40:17 - Info bpbrm (pid=21148) start bpbkar on client
04/22/2013 20:40:17 - connected; connect time: 0:00:00
04/22/2013 20:40:17 - begin writing
04/22/2013 20:40:19 - Info bpbkar (pid=44916) Backup started
04/22/2013 20:40:19 - Info bpbrm (pid=21148) Sending the file list to the client
04/22/2013 20:49:37 - Warning bpbrm (pid=25731) from client NBU-CLIENT: WRN - Removable Storage Management: unable to export database (WIN32 21: The device is not ready. )
04/22/2013 20:49:43 - Info bpbrm (pid=21148) media manager for backup id NBU-CLIENT_1366677613 exited with status 0: the requested operation was successfully completed
04/22/2013 20:49:43 - end writing; write time: 0:09:26
the requested operation was successfully completed  (0)

=============================================================================

This one is still going as I write this post:

=============================================================================

04/22/2013 20:40:13 - Info nbjm (pid=8839) starting backup job (jobid=129360) for client NBU-CLIENT, policy NBU-POLICY, schedule INCR-D
04/22/2013 20:40:13 - Info nbjm (pid=8839) requesting STANDARD_RESOURCE resources from RB for backup job (jobid=129360, request id:{5C04D450-ABAE-11E2-8732-963E8E4EA941})
04/22/2013 20:40:13 - requesting resource NBU-MASTER-hcart-robot-tld-0
04/22/2013 20:40:13 - requesting resource NBU-NBU-MASTER.NBU_CLIENT.MAXJOBS.NBU-CLIENT
04/22/2013 20:40:13 - requesting resource NBU-NBU-MASTER.NBU_POLICY.MAXJOBS.NBU-POLICY
04/22/2013 20:40:13 - awaiting resource NBU-MASTER-hcart-robot-tld-0. No drives are available.
04/22/2013 20:40:39 - granted resource  NBU-NBU-MASTER.NBU_CLIENT.MAXJOBS.NBU-CLIENT
04/22/2013 20:40:39 - granted resource  NBU-NBU-MASTER.NBU_POLICY.MAXJOBS.NBU-POLICY
04/22/2013 20:40:39 - granted resource  TAPE1
04/22/2013 20:40:39 - granted resource  NBU-MASTER-TLD-000-hcart-2
04/22/2013 20:40:39 - granted resource  NBU-MASTER-hcart-robot-tld-0
04/22/2013 20:40:39 - estimated 2119502 kbytes needed
04/22/2013 20:40:39 - Info nbjm (pid=8839) started backup job for client NBU-CLIENT, policy NBU-POLICY, schedule INCR-D on storage unit NBU-MASTER-hcart-robot-tld-0
04/22/2013 20:40:40 - Info bpbrm (pid=21148) NBU-CLIENT is the host to backup data from
04/22/2013 20:40:41 - Info bpbrm (pid=21148) telling media manager to start backup on client
04/22/2013 20:40:41 - Info bptm (pid=21156) using 65536 data buffer size
04/22/2013 20:40:41 - Info bptm (pid=21156) using 12 data buffers
04/22/2013 20:40:41 - Info bpbrm (pid=21148) spawning a brm child process
04/22/2013 20:40:41 - Info bpbrm (pid=21148) child pid: 26294
04/22/2013 20:40:42 - Info bpbrm (pid=21148) sending bpsched msg: CONNECTING TO CLIENT FOR NBU-CLIENT_1366677639
04/22/2013 20:40:42 - connecting
04/22/2013 20:40:46 - Info bpbrm (pid=21148) start bpbkar on client
04/22/2013 20:40:46 - Info bpbkar (pid=44800) Backup started
04/22/2013 20:40:46 - Info bpbrm (pid=21148) Sending the file list to the client
04/22/2013 20:40:46 - connected; connect time: 0:00:00
04/22/2013 20:40:46 - begin writing
04/23/2013 02:27:51 - current media TAPE1 complete, requesting next media Any
04/23/2013 02:27:51 - current media -- complete, awaiting next media Any. Waiting for resources.
          Reason: Drives are in use, Media server: NBU-NBU-MASTER,
          Robot Type(Number): TLD(0), Media ID: N/A, Drive Name: N/A,
          Volume Pool: ENCR_FLO0_Offsite_1, Storage Unit: NBU-MASTER-hcart-robot-tld-0, Drive Scan Host: N/A,
          Disk Pool: N/A, Disk Volume: N/A
04/23/2013 02:29:13 - granted resource  TAPE2
04/23/2013 02:29:13 - granted resource  NBU-MASTER-TLD-000-hcart-2
04/23/2013 02:29:13 - granted resource  NBU-MASTER-hcart-robot-tld-0
04/23/2013 02:29:13 - mounting TAPE2
04/23/2013 02:29:57 - mounted TAPE2; mount time: 0:00:44
04/23/2013 02:29:57 - positioning TAPE2 to file 1
04/23/2013 02:30:04 - positioned TAPE2; position time: 0:00:07
04/23/2013 02:30:04 - begin writing

============================================================================

 

The backup was working fine until the servers were virtualized. These are 2 domain controllers that were P2V about 2 weeks ago. And, yes, only these 2 clients have this issue. Both are identical.

 

NetBackup v7.5.0.6

SuSE Linux Enterprise Server 10 SP 3

StefanosM's picture

also tell us if you are using VADP (vmware API) backup or the client over the network backup. Also give us the total number of the files the client has.

luisvidal's picture

Stefanos,

No, they are been backed up with a normal WINDOWS policy. Each client has about 49K files.

NetBackup v7.5.0.6

SuSE Linux Enterprise Server 10 SP 3

jim dalton's picture

Backup the same data but instead of writing to the device/net write it to nul. Needs to be done on the cmdline.

Search for it on the forum : its been done many times. That way you can tell if its the server + client s/w or the rest of it.

It took 9mins to write, but its an INC: is that slow? Seems ok to me...what were you expecting?

I would say your tape buffer size is a touch low for lto4 but this may not improve depending on the bottleneck

A bit more clarity needed I think eg start with a full lets see whats what.

Jim

luisvidal's picture

I will try that Jim. Yes, the first backup log is for an INC, but that is the only one that have completed in 2 weeks---just one stream, the other one started at the same time and has been going for 13+ hours. 

NetBackup v7.5.0.6

SuSE Linux Enterprise Server 10 SP 3

mph999's picture

Hmm, odd, I expected to see the lines I suggested.

Only way to get those then, is to look in the bptm log (media ) and bbpbkar log (client).

Martin

 

Regards,  Martin
 
Setting Logs in NetBackup:
http://www.symantec.com/docs/TECH75805
 
RonCaplinger's picture

04/22/2013 20:40:41 - Info bptm (pid=21156) using 65536 data buffer size
04/22/2013 20:40:41 - Info bptm (pid=21156) using 12 data buffers

 

Those buffer settings are fairly low for writing directly to LTO4 tape drives.  Like Martin said, we would need to see the "Waited xx times for full buffer, dalayed yy times" and "Waited xx times for empty buffer, dalayed yy times" lines from the bptm and bpbkar logs.

mph999's picture

Increase the size to 262144

Increase the number to 64

If that performs better, try the number at 128, test again then try 256

At some point it will make no difference, or will decrease, in which case move back to the previous value.

 

Martin

 

Regards,  Martin
 
Setting Logs in NetBackup:
http://www.symantec.com/docs/TECH75805
 
luisvidal's picture

Will do, Martin. Thanks, and will keep you guys posted.

NetBackup v7.5.0.6

SuSE Linux Enterprise Server 10 SP 3

Jean-Pierre Bailly's picture

You say backup was working fine until servers were virtualised.

For me this points to bandwidth issues getting your data from the client server to the media server.

Or your virtual server doesn't have enough ressources to do whatever it has to do and also do the backup.

mph999's picture

I think Jean--Pierre you will be correct. 

 

Regards,  Martin
 
Setting Logs in NetBackup:
http://www.symantec.com/docs/TECH75805
 
luisvidal's picture

Hi guys, resolved the issue by using the W2KOption.exe and forcing the use of the system provider for VSS

 

W2KOption.exe -backup -snapshotprovidertype 1

 

http://www.symantec.com/docs/TECH156732

 

 

Thanks to everyone for their help.

NetBackup v7.5.0.6

SuSE Linux Enterprise Server 10 SP 3

SOLUTION