Video Screencast Help
Symantec to Separate Into Two Focused, Industry-Leading Technology Companies. Learn more.
Netting Out NetBackup

Nuts and bolts in NetBackup for VMware: Discovery job in VMware Intelligent Policy

Created: 26 Aug 2011 • Updated: 22 Jan 2013 • 18 comments
AbdulRasheed's picture
+11 11 Votes
Login to vote

There was a request from Rizwan to explain how NetBackup for VMware works.  Several votes and comments later, I came to know about it from a technical support engineer who had pointed me to that post. We hear you loud and clear, we are looking into getting more details in NetBackup for VMware System Administrator’s Guide. While we wait for a formal documentation update, I thought I better write a set of blogs on what was requested by the community. So here it goes.

  Let me not repeat what is already there in the manual. If you are new to NetBackup for VMware, I would strongly recommend going through NetBackup for VMware System Administrator's Guide for an introduction.

   For this discussion let us assume that the NetBackup master server, media server and VMware backup host are three different systems. Also let us assume that the VMware backup host and VMware discovery host are the same system. The request was to provide a flow of communications for backups and restores. In order to get there, first we need to discuss virtual machine discovery as well.

  NetBackup for VMware features two kinds of virtual machine discovery.  The newer VMware Intelligent Policy (VIP) discovery and original browse and select discovery. The former creates a discovery job in activity monitor; hence let us discuss its process flow before getting into backup.

VMware Intelligent Policy Discovery: This game changing discovery method is part of Symantec’s V-Ray vision; a set of technologies to provide unique visibility into virtual machines. Your virtual machine protection strategy enters auto pilot mode with VIP. In the policy you are simply specifying selection criteria for what needs to be protected (e.g. all the virtual machines in data center X, all virtual machines in cluster Y, all virtual machines with RedHat Linux etc.). NetBackup automatically discovers the VMs matching the criteria and runs backups. How cool is that! Visibility matters! Let us go through a high level overview of the process flow for VIP discovery job in this blog.

  1. When backup window opens, NetBackup Policy Execution Manager (nbpem) on master server looks to see if there is a VM list xml file in its policy database (we will explain this later) to see if it can use the pre-existing  list of virtual machines. If the xml file is present, it examines its age to see if a new one needs to be generated. Let us assume that the xml file is not present or it is older than the time interval specified in the policy attribute named “Reuse VM selection query results for”. Then a discovery job shows up in Activity Monitor.
  2. nbpem on master contacts NetBackup Client Service (nbcs) on media server*.
  3. nbcs on media server starts NetBackup Client Service (nbcs) on VMware backup host*.
  4. nbcs has a plug-in for VMware which gets loaded. VMware APIs for Data Protection (vADP) calls are implemented in this plug-in. NetBackup has credentials to access the vSphere host (vCenter or ESX/ESXi), it logs into the vSphere host and runs VM discovery based on the selection query given in the policy. The vSphere host returns an xml file with the results. This file may or may not be processed further by the plug-in based on the kind of query.  
  5. nbcs returns the VM list xml file to master server through nbcs on media server*.
  6. nbpem on master server uses the xml file to generate the client list. The discovery job in Activity monitor moves to Done state.
  7. If the backup is due it is started by nbpem.  The xml file persists in NetBackup database until its age exceeds the time specified in the policy attribute named “Reuse VM selection query results for”

*We assumed here that media server and VMware backup host are separate hosts. As NetBackup is an enterprise platform, there could be a firewall between NetBackup master server and VMware backup/discovery host. The nbcs on media server starts nbcs on VMware backup host on behalf of master server in this case. If the VMware backup host and media server are the same, NetBackup is smart enough to use a singe nbcs process.

VMware Intellgent Policy is availble in NetBackup 7.1 and NetBackup 5200 series appliances with software version 2.0.

Nuts and Bolts in NetBackup for VMware series continues here:

Back to Nuts and Bolts in NetBackup for VMware series

Comments 18 CommentsJump to latest comment

AbdulRasheed's picture

As I mentioned in the blog, I am writing this series per community request. Please do give your feedback so that I can update this blog as well as the other ones coming in this series. Once we know what you guys are looking for, we can also easily accommodate those in our documentation.  Thank you for your time and feedback!

Warm regards,

Abdul "Rasheed" Rasheed

Tweet me @AbdulRasheed127

+1
Login to vote
rizwan84tx's picture

Hi Adbul,

The reason i requested for VMware backup backup & restore flowchart is to equip ourself in troubleshooting any Vmware related backup or restore issues. We would all appreciate if symantec can provide us details process flow, like the what process are involved during the backp/restore operation and on which port they call query for.

Recently we had restore problem for VMware, where the cause was unknown after checking the VxMS (dont know what VxMS does) logs that Backup host needs access on port 902 in ESX host. There are many such things which has to be explained.

I really appreciate your effort in helping us with this request.

Best Regards,

Rizwan

-4
Login to vote
AbdulRasheed's picture

Hi Rizwan,

I did write a rather long blog on vStorage backup process flow. I didn't go in depth in ports there as the needs are different for different transport methods. I shall write about it when I am back in office.

Restore process flow coming soon.

I appreciate the feedback.

 

 

Warm regards,

Abdul "Rasheed" Rasheed

Tweet me @AbdulRasheed127

-2
Login to vote
Marianne's picture

Maybe publish a separate TN for port usage with VMware backups? Or update the existing ones? e.g. http://www.symantec.com/docs/TECH136090

The ONLY reference to port(s) in the NBU fo VMware guide is the following:

Connect using port number
If the default port number has not been changed on the VMware server, no
port specification is required. In that case, make sure that the Connect using
port number box is not checked.
If the VMware server has been configured to use a different port, click the
Connect using port number box and specify that port number.

 

This is hardly helpful in a secure environment with multiple DMZ's.............

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

+2
Login to vote
AbdulRasheed's picture

I shall work on a blog explaining the transport methods which is the right place for me to explain the port requirements.

Warm regards,

Abdul "Rasheed" Rasheed

Tweet me @AbdulRasheed127

-3
Login to vote
dvs001's picture

Abdul,

Is there a way to invoke nbcs from the cli on the hotadd proxy client to remove the master/media server connections as a troubleshooting mechanism?  I have often wanted to run the VIP query straight from the hotadd.

 

Thanks for your time.

Darrel

+2
Login to vote
vksingh's picture

Hi Abdul,

Can my backup host be a VM?

thanks

-4
Login to vote
Marianne's picture

Please have a look at this recommendation right in the beginning of the blog:

"If you are new to NetBackup for VMware, I would strongly recommend going through NetBackup for VMware System Administrator's Guide for an introduction."

See NBU for VMware Admin Guide  www.symantec.com/docs/DOC3663

 

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

0
Login to vote
AbdulRasheed's picture

Hi vksing,

  As Marianne had mentioned, the details are in the admin guide. A VM can indeed be a backup, recovery and discovery host. You will be able to use hotadd and NBD transports. More on transports and ports coming soon.

Warm regards,

Abdul "Rasheed" Rasheed

Tweet me @AbdulRasheed127

+8
Login to vote
e-security's picture

Hello,

thank you for the useful blog post.

We like very much the VMWare Intelligent Policy discovery, the automatic query mechanism and the other related features.

In our environment we often need to manually start a backup of a single VM, but this is impossibile when using the VIP: trying to launch a manual backup on a VIP policy will result in the backup of every VM matching the query.

We also opened an idea/feature request on Symantec Connect suggesting a solution: https://www-secure.symantec.com/connect/ideas/automatic-vm-selection-through-query-manual-backup-selection-problem

Do you know if there is a way to launch a manual backup of a single VM from a automatic query policy?

+2
Login to vote
AbdulRasheed's picture

 

Hi e-security,

  You brought up something we had been looking to implement. Thank you very much for posting the idea. The more votes we get, its priority will go up in the implementation list. 

   You are right, if you need a onetime backup you can run it from a separate policy at this time. I have seen two customers who had created a policy dedicated for manual backups while using VIP. VIP automates protection and manual policy (normally in the deactivated state) is used for one time backups while preparing for upgrades, patching and such.

Warm regards,

Abdul "Rasheed" Rasheed

Tweet me @AbdulRasheed127

+3
Login to vote
e-security's picture

That's exactly how we're dealing with the problem right now (so that's 3 clients now ;).

The sooner this is implemented, the better: the re-validation time when adding to the manual backup even a single VM on a some-hundreds-VMs-environment is taking forever!

Regards

-3
Login to vote
puga's picture

In our environment "Test query" button in automatic selection of clients (in policy) takes from 5 to 20 minutes. Any chance we could lower this waiting time? As far as I understand, NBU could fetch vm machine list only once, and then "apply" filter rule(s), instead of every time making a connection to vCenter / ESX to get a long list of machines and then "apply/test" filter rule(s).

We're using 7.1.0.4 on master/media servers and backup/proxy hosts.

Thanks.

+2
Login to vote
rizwan84tx's picture

In your backup host, open the registry and add a new key "BACKUP" under "HKLM\SOFTWARE\Veritas\NetBackup\CurrentVersion\Config"

Click the key BACKUP and in the right hand pane, right click and add a new DWORD key with name "disableIPResolution" and value 0.

We have 3 vcenter domain, after doing the above change, the VM query only take 2 -5 minutes to fetch the inventory. Hope this is helpful.

Best Regards,

Rizwan

+6
Login to vote
AbdulRasheed's picture

Hi puga,

  As Rizwaan mentioned, disableIPresolution will mitigate the problem if you have a large environment where the proxy system cannot resolve all the VM hostnames. 

   Also, it is possible to minimize the number of times NetBackup contacts vCenter server by using the 'reuse query every xx' hours field. Becuase of this setting, the fetch does not occur for each job. 

  If this does not resolve the problem, please do work with technical support. Perhaps you have an environmental issue that they can help to isolate it for you. 

 

Warm regards,

Abdul "Rasheed" Rasheed

Tweet me @AbdulRasheed127

0
Login to vote
ajay-csc's picture

Hi all,

 

My vm clients on vmx 7, no errors in vcb side sanpshot is happening fine on vcb end .Netbackup version is 7.1.0.4 .and my backup off host and media server is same .Still full backups fail with 156 and incrementals with 13.any suggestions?

nick80's picture

Hi Abdul,

Vey useful blog but Im trying to understand something slightly different in ragards to Netbackup VMware comms process.

In short whilst monitoring the Master server NIC (through TCP Dump) we are seeing attempted communication to a huge number of virtual machine IP addresses  over port 1556 ad 13724.

14:44:30.526337 IP <<master server >>>.39312 > xxx.xx.250.23.veritas_pbx: S 1942355283:1942355283(0) win 5840 <mss 1460,sackOK,timestamp 1566508170 0 ,nop,wscale 7>

14:44:40.526429 IP <<master server >>>..53055 > xxx.xx.250.23.vnetd: S 4152706775:4152706775(0) win 5840 <mss 1460,sackOK,timestamp 1566518170 0,nop,wscale 7>

14:45:10.550537 IP <<master server >>>..48410 > xxx.xxx.145.13.veritas_pbx: S 3037287126:3037287126(0) win 5840 <mss 1460,sackOK,timestamp 15665481940,nop,wscale 7>

14:45:11.550586 IP <<master server >>>..39265 > xxx.xxx.145.13.vnetd: S 3698696433:3698696433(0) win 5840 <mss 1460,sackOK,timestamp 1566549194 0,nop,wscale 7>

I know this is not part of the backup itself as these are being done through the Vstorage API and seperate backup host rater than Netbackup clients. I expected to see comms between the master and Vcenter servers over 443 but nothing between the individual VM's. Because of the work we are currently undertaking I need to explain what this attempted comms is and why its happening.

I hope you can help ?

Regards

Nick

+5
Login to vote
AbdulRasheed's picture

Hi Nick,

  As you already know, NetBackup for VMware backups are agentless. No agent is required in the VM to stream backups. 

  The ports 1556 (PBX) and 13724 (vnetd) are ports used by NetBackup agents. My guess here is that master is trying to contact the VM itself. It can happen for a few reasons that I can think of are...

1. You have application discovery turned on in the policy. In such cases, attempts will be made to contact the VM to discover application topology (Exchange, SQL Server, SharePoint) etc. 

2. You may have also configured some VMs in a Standard/Windows policy in addition to VMware/FlashBackup policy. 

I recommend working with our Technical Support team to find out the actual reason behind this. We can identify this from the logs in the master server that is attempting connections to VM IP addresses. 

 

 

Warm regards,

Abdul "Rasheed" Rasheed

Tweet me @AbdulRasheed127

0
Login to vote