Video Screencast Help

Virtual media server NBUhangs when used for backups

Created: 11 Apr 2013 | 12 comments
hidetoshi_hishinuma@dell.com's picture

Hi guys,

I have an environment with:

  • Phisycal Master Server running Win2K8 R2 SP1 (lets name it MasterServer01);
  • Virtual Media Server running Win2K8 R2 SP1 on a ESX Cluster running Vmware 5.0 U1 (lets name it MediaServer01);
  • Netbackup master and media server Version is 7.5.0.5;
  • Dell DR4000 OST appliance;
  • OST plugin is installed on both master and media server;
  • Storage Server, disk pool and storage unit (STU01) are properly configured;
  • No tape library configured on this solution;
  • Vcenter credentials are configured and backup hosts set.
  • Using vmware policy for backups.

Problem:

I´ve set STU01 (Open Storage) to use MediaServer01 as the media server. Backup operation appeared to start and run as it should be using vmware agent (could see a great throughput of 85MB/s on job details). However the vm MediaServer01 (entire vm) shuts down at the end of the backup.

An interesting point is that the virtual disk becomes unavailable after shutdown. Have to reattach it to the vm MediaServer01 and turn it on again.

Tried both, hotadd and nbd transports, with the same results.

Note: When setting the STU01 to use MasterServer01 as the media server, backups of vms runs succesfully.

Does anybody here have any suggestion to solve this strange problem?

Regards,

Hide

Operating Systems:

Comments 12 CommentsJump to latest comment

Yasuhisa Ishikawa's picture

It is curious issue as NetBackup does not touch VM configuration while backup.

Do you deploy any VM aware tool in your environment that monitor VM load and try to move vmdk from one VM to another?

Authorized Symantec Consultant(ASC) Data Protection in Tokyo, Japan

hidetoshi_hishinuma@dell.com's picture

Hi Ishikawa-san,

Thanks for your prompt reply.

Agree with you that it´s a curious issue. :(

In this environment, no monitoring tool is available.

MediaServer01 vm has 8GB of RAM, more than enough to handle the amount of jobs (one at a time).

I will increase the RAM to 16GB, rerun a backup and see if problem persists.

If you have any other tip, please let me know. :)

Regards,
Hide

drm163's picture

Hi

I have recently been investigating the hotadd transport option and found this - not sure if it is relevant, but the symtpoms do match. This is from the VMWare VDDK release notes:

http://www.vmware.com/support/developer/vddk/VDDK-1.2.1-Relnotes.html

Under known issues and workarounds:

Do not use HotAdd mode to back up the backup proxy

After using HotAdd transport mode to back up the virtual machine that serves as the backup proxy, the backup completes successfully, but during cleanup, regular virtual disk is mistakenly removed along with HotAdded disk. This will be fixed in the next release. If you mistakenly use HotAdd mode, you will have to run the vSphere Client, select the proxy VM, edit settings, and re-add the “Hard Disk” VMDK file from storage.

Cheers

hidetoshi_hishinuma@dell.com's picture

Hi David,
Sorry for this late reponse.
It really seems to be a solution.

The vm is in fact the backup proxy.
I will uncheck the option for hotadd and run a backup.

I will have a feedback only in December.

But I´ll let you know.

Thank you for your tip.

Regards,
Hidetoshi

Jeff Foglietta's picture

For future consideration: Virtual media servers are never a good idea

Jeff Foglietta

Practice Lead - Data Protection and High Availability

Ron Cohn's picture

Why not virtual?  Nothing I have seen or read states otherwise...

Ron Cohn
"I maybe lost, but I am making good time..."

Environment: NBU 7.6.0.3 for Windows
Write to EMC DD4200 -> EMC DD2500 -> ADIC i500

chack22's picture

Disagree, and for a small environment, it may be the best option.

We've been migrating our large NBU environment from physical to virtual where able (master and medias) The only place we haven't been able to do it is where we still have tape in our main facility. We have close to 30 media servers, in 10 different data centers.

We have yet to run into an issue where the VMware folks were down our throats due to excessive I/O, memory bursting, etc. and we've had virtual media's for at least the past 3 years, and with our latest cutover to the SQL agent, have close to 40000 jobs running daily.

It works, and works well as long as you tune properly.

Jeff Foglietta's picture

You seem to be the exception to the rule. Your ESXi farm is obviously quite large and resource heavy. That's a perfect world. You're quite fortunate. However this isn't the case for most environments. The key word here is "Small" environment. I/O tends to be an issue in larger environments. I would also like to point out that there have been issues regarding deduplication using a Virtual Media Server in any size environment with CPU ultilization and I/O waits degrading performance. I've spoken to many of my contacts within Symantec and the concensus has been Virtual Media Servers are not recommended. Sure it will work. After all a VM is just another computer, right? So why not? The sharing of resources however, tends to restrict its ability to perform in most environments. And depending on how you have your resource allocation dynamics configured in VMware you could lose CPU and Memory resources to a higher priority VM in an instant making things even worse. Not likley a virtual media server is configured with dynamic cpu/mem but its certainly not impossible to think there are some environments out there configured that way.

Jeff Foglietta

Practice Lead - Data Protection and High Availability

chack22's picture

I wish Symantec would get their stories straight. I passed along your note to 3 engineers and 3 product managers in Roseville and heard two different answers (they probably compared notes) one along your line, and another towards mine. The only real consensus between the two was that backups are generally being run 'off-hours' and during non-business hour windows.

Yes, we are quite fortunate in our main environment in that we have the resources to do what we need (as well as a 10 gig network to leverage in most places) but in talking to folks at Vision and at the Customer Forum, we've found that a number of folks are doing the same. Again, proper tuning can alleviate many of the concerns, and using Resource Limits in NetBackup help reduce the strain on a VMware environment.

Without completely hijacking this thread, I'd be interested in any documentation surrounding dedupe with a VM Media Server. Do you have something you can share? We use both MSDP and OST.

Kevin Good's picture

While it is technically supportable to run the media server in a virtual machine environment, the paradigm doesn't really fit... VM's are great for a number of guests which have bursts of activity, or do not generate a substantial amount of I/O... Media Servers have sustained periods (their whole backup window) of intense activity, generating lots of I/O...

Backups are IT-101 (Do Backups) Doing Backups well is an art form. Nobody cares about getting their data backed up... They only care that you can restore it for them!

Ron Cohn's picture

It would be nice to have a Symantec Tech hop in, but when we were having issues with one of our media servers, Symantec ran benchmarks using both physical and virtual media servers, and the virtual media server sustained a higher I/O rate than the physical.

We have all of our SQL servers as virtual and with our change rate (which is extreme), never had a performance issue.

Ron Cohn
"I maybe lost, but I am making good time..."

Environment: NBU 7.6.0.3 for Windows
Write to EMC DD4200 -> EMC DD2500 -> ADIC i500

Mark_Solutions's picture

And a virtual media server (or at least a client) is the only way to do hot add - don't see a waya round that!

So based on the thread so far it looks like it just need the media server (the VMware backup host) excluding from the policy and maybe that will sort it

Authorised Symantec Consultant

Don't forget to "Mark as Solution" if someones advice has solved your issue - and please bring back the Thumbs Up!!.