Video Screencast Help

Slow duplication to tape from disk, help find the bottleneck?

Created: 23 Jan 2013 • Updated: 25 Mar 2013 | 4 comments
This issue has been solved. See solution.

Hi All,

 

We are running NBU 7.5.0.3 and I have noticed that recently my duplication rate from disk (ExaGrid) to tape (quantum tape library)  has been quite slow. Historically I would que up 4-6 images at a time to duplicate to tape and que 3 of those jobs at a time (leaving a 4th drive free incase I need a restore during the day). I would generally be able to que up 6-9 jobs of those per day, and could make quick work of a month-end duplication.  Now it appears my duplications are taking much much longer, and im getting 3 or less per day.

 

What sort of logging can I turn on, or what sorts of performance monitoring can I do to try and find the bottleneck along the chain of devices?

 

Basic device overview:

Single NBU server - backs up to:

4 ExaGrid deduplication arrays - then duplicated to:

1 Quantum Tape library with

4  fibre attached LTO-5 drives

Comments 4 CommentsJump to latest comment

Marianne's picture

Does Exagrid have direct tape copy like the Quantum DXi?

If not, I guess you are duplicating via network? If so, this this is where you need to start - get your networking team involved.

If Exagrid is a dedupe appliance, it is possible that the appliance is battling to re-hydrate data before sending to tape. Get your Exagrid Support team to assist with troubleshooting.

From NBU point of view bptm log on master/media server will tell you if tape was waiting for full or empty buffers.
Waiting for full buffers either means slow disk read or slow network.

 

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

Mark_Solutions's picture

Has anything changed recently in your setup?

I am wondering if you have started to do any GRT backups (exchange etc.) as this is where your holdup could be

The GRT part allocates a tape drive even though it is doing its cataloging (which could take an hour or so) before it even thinks about writing to the tape.

If that is the case you could look at using a proxy host for the GRT part, decide not to duplicate the GRT data or use a smaller fragment size for the Disk Area which i have seen greatly increases GRT work

Hope this helps

Authorised Symantec Consultant

Don't forget to "Mark as Solution" if someones advice has solved your issue - and please bring back the Thumbs Up!!.

Nate.D's picture

Marianne: thanks for the response, I am checking on the direct tape copy. We are duplicating via network, and put PRTG monitors on the exagrid ports and saw pretty slow throughput while netbackup was in 'reading' state in the job. That makes me lean towards the re-hydration issue. I am trying to duplicate off of the oldest set of backups so that would make sense. I forwarded the issue off to our ExaGrid people to see if they can help look into it.

 

If I can get through the last bit of the backup for that months set done i should be able to catch up on the rest with less hydration and I may be able to get out of this. I will keep this thread updated but it will take a few days.

 

Thanks all

If I was helpful in solving your issue please mark my post with a thumbs up or a solution!  Have a great day :)

Nate.D's picture

Resolution for this was; we were trying to duplicate backups that were several months old on our deduplication device, these backups had been heavily deduplicated by that point so the rehydration was taking an excessive time. I had to abandon several large backups that were taking too long to duplicate. Eventually I caught up to current and the issue is gone.

If I was helpful in solving your issue please mark my post with a thumbs up or a solution!  Have a great day :)

SOLUTION