Video Screencast Help
Symantec to Separate Into Two Focused, Industry-Leading Technology Companies. Learn more.
Netting Out NetBackup

Nuts and bolts in NetBackup for VMware: Understanding V-Ray vision through backup process flow

Created: 27 Aug 2011 • Updated: 22 Jan 2013 • 16 comments
AbdulRasheed's picture
+11 11 Votes
Login to vote

Now that we know how VM discovery job works for a VMware Intelligent Policy (VIP), let us move on to the actual backup job. The process flow described here is the same for both VIP and browse-and-select type policies unless specified otherwise.

As we did in the previous blog, let us assume that the NetBackup master server, media server and VMware backup host are three different systems.

  1. NetBackup Policy Execution Manager (nbpem) triggers a backup job once the policy, schedule, virtual machine combination is due and backup window is open. It works in conjunction with NetBackup Job Manager (nbjm), NetBackup Resource Broker (nbrb) and NetBackup Enterprise Media Manager (nbemm) to identify the resources (media server, storage unit etc.) for backup. This resource identification process is well detailed in NetBackup Documentation, hence let is not spend time on this. I will focus on things specific to vStorage backups.
  2. If this is VIP, the VMware resource limits specified by the backup administrator is also taken into account at this time. NetBackup for VMware (NetBackup 7.1 and NetBackup 5200 series appliances with software version 2.0) provides the flexibility to throttle VMware resources used in vSphere environment. For example, you can specify that you don’t want more than 4 concurrent backup jobs running from a vSphere datastore. This level of control let the administrators to tune the number of backups such that user and application experience for vSphere platform is minimally affected on account of running backup jobs.
  3. Once the backup resources are identified and VMware resource limits are set, nbjm contacts the selected media server and starts NetBackup Backup and Restore Manager (bpbrm*) on it. You will see a snapshot job (commonly referred to as parent job) going Active in the Activity monitor at this time.
  4. The bpbrm process on media server contacts the VMware backup host and starts NetBackup Frozen Image Service (bpfis) process.
  5. The bpfis process is in charge of all things snapshot in NetBackup. This process is capable of creating snapshots by loading respective shared libraries (UNIX/Linux) or dynamic link libraries (Windows) coded to handle various kinds of snapshots. In our example, it loads the code related to creating vSphere snapshots. This dynamic link library implements vStorage APIs for Data Protection (vADP) calls.
  6. bpfis armed with vADP contacts the vSphere host (vCenter or ESX/ESXi host for which credentials are stored in NetBackup database) and initiates the snapshot for the VM in question. If there are multiple VMs, a bpfis is started for each VM so that the snapshot operations occur in parallel. As I mentioned in step 2, you can control the number of concurrent snapshots by setting VMware Resource Limits in NetBackup when using VIP. The way bpfis contacts vSphere host is via standard SSL port (default is 443).
  7. Once the snapshot is created, bpfis who initiated the snapshot reports back to bpbrm on media server. The snapshot job (parent job) is moved to Done state in Activity Monitor.
  8. bpbrm now initiates two things in parallel. It contacts VMware backup host again and starts NetBackup Backup and Archive (bpbkar) process. At the same time it also starts NetBackup Backup Tape Manager (bptm)** to prepare the storage (tape or disk) in order to receive backup stream from VMware Backup Host. This time the backup job (commonly known as the child job) shows up in Activity Monitor.
  9. Let us spend some time with bpbkar for now. It is imperative to understand what happens here because this is an area where none of the competitors can match NetBackup’s patent pending granular recovery technology. This is part of Symantec’s V-Ray vision to provide unique visibility into virtual machines. Many vendors can claim that they can restore full virtual machine or individual files from the same backup. If you ask them how it is implemented, you are likely to see their weakness quickly.

The bpbkar process loads VERITAS Mapping Services (VxMS) which among many other things, also implement VMware Disk Development Kit (VDDK) APIs. The VDDK APIs are used for reading from vSphere datastore. The real beauty of V-Ray comes from VxMS which has patent pending technology to map the stream during runtime and identify what is actually inside the VMDK file being read.

Here are some of the salient aspects of Symantec V-Ray within VxMS:

9.a) You are generating the catalog of all the files inside the VMDK (supported for VMDK files from both Windows and Linux VMs). Remember that this is not a two-pass process, the operation occurs while backup data is being streamed. This catalog information is send to master server via bpbrm process on media server.

9.b) As you are looking at file system inode level, you also get to identify unused and deleted blocks. This is quite powerful vision into the VM data. If the application on VM had allocated 1TB of space for a file, of which just 100GB is currently used, the backup stream would include just that 100GB. Similarly if you deleted a 1TB file that was fully allocated in the past, VxMS knows to skip the deleted blocks (unless the blocks are now allocated for a new file) from the backup stream. This optimization not only speeds up the backup stream, but it reduces needed storage even when deduplication is not enabled.

9.c) If source side deduplication feature is enabled, the VMware backup host does the deduplication. The NetBackup deduplication plug-in is not seeing just a giant VMDK file, it understands the mapping info generated by VxMS and hence it sees (with the V-Ray vision!) the actual files in the file system within the VMDK. This V-Ray vision is established by NetBackup deduplication plug-in loading a dedicated stream handler that understands the mapping info generated by VxMS.

9.d) Remember that all these operations occur at the VMware backup host. Your ESX resources (and hence the resources from VMs) are not used in these cases. This is true offhost backup with no burden on your production vSphere. Compare this with other vendors running index engines on VMs or helper VMs and including unallocated blocks and deleted files in all backups. Even the source side deduplication is happening in an offhost system in case of NetBackup for VMware.

  1. Now let us go back to media server, we mentioned that bpbrm had started bptm. (Technically speaking, bptm forks another bptm as well. One is in charge of preparing the storage and writing backup streams to it and another one receives backup stream from VMware backup host.  As this part is covered well in NetBackup Documentation, I am not distinguishing various bptm processes here.) The bptm process prepares the backup storage to receive data. If the storage unit allocated is tape based (known as Media Manager Storage Unit), it works with Media Manager daemons to get appropriate tape mounted into a drive. If the storage is BasicDisk, AdvancedDisk or third party OpenStorage; appropriate plug-ins are loaded by bptm. The backup stream sent by bpbkar (see step 9) is directed to the storage by bptm. In order to understand another aspect of V-Ray, we need to talk about a special case here when the storage chosen is a NetBackup Deduplication Pool.

For those new to NetBackup Deduplication, I would recommend reading this blog series. If you chose not to do deduplication at VMware Backup host (see 9.c), you are not losing V-Ray vision. The deduplication plug-in for media server deduplication pool, NetBackup 5200 series appliances and NetBackup 5000 series appliances also have V-Ray vision by making use of the dedupe stream handler which sees the actual files within a VMDK in backup streams. In fact, this V-Ray vision is one of the key differentiators for NetBackup deduplication when compared with third party target-only deduplication devices.

  1. Once the backup stream is complete, bpbkar on VMware Backup host notifies bpbrm on media server and exits.
  2. bpbrm on media server contacts VMware backup host again and starts a bpfis process to release the snapshot.
  3. bpfis on VMware backup hosts loads the dynamic link library implementing vADP. It contacts the vSphere host and initiates snapshot deletion.
  4. Once the snapshot deletion is over, bpfis on VMware backup host notifies bpbrm on media server and exits.
  5. bpbrm waits for bptm to finish the final housekeeping tasks. Once bptm processes exit, bpbrm notifies nbjm on master server.
  6. nbjm works in conjunction with nbrb and nbemm to release the resources used for backups. In parallel, it also notifies nbpem about the status of the job.
  7. nbpem calculates when the backup for the VM will become due again and sets it timer accordingly. The backup job now moves to Done state in Activity Monitor.

*For those new to NetBackup, the ‘bp’ in bpbrm, bptm, bpbkar etc. stands for ‘backupplus’, the old name of NetBackup.

**Although ‘t’ in bptm stands for tape, bptm has undergone significant overhaul during NetBackup 6.x. This process now handles tape as well as disk destinations.

Back to Nuts and Bolts in NetBackup for VMware series

Comments 16 CommentsJump to latest comment

AbdulRasheed's picture

This blog was an effort to educate on NetBackup for VMware process flow per community request. Normally I do not write a blog this long. This time I had to break the rule to provide the full technical details. Let me know if there are specific areas here that needs to be elaborated. We can do this through comments for the time being.

My responses may be delayed this week as I will be at VMWorld. Are you going too? If yes, hope to see you at Symantec booth #339. Symantec is a platinum sponsor and we have a great show for you!

Warm regards,

Abdul "Rasheed" Rasheed

Tweet me @AbdulRasheed127

+2
Login to vote
rizwan84tx's picture

Very Interesting post; thank you very much for producing this information to us. It is very helpful.

Best Regards,

Rizwan

-1
Login to vote
Kev_D's picture

Thank you very much for taking the time to both produce and share this information.

Such a useful insight into how the "miracle" of Granular Restore is achieved.

Keep up the good work, and I look forward to reading some of your earlier blogs, as well as your future ones.

Kevin

-1
Login to vote
kirans's picture

Thanks for sharing the process of VMware backup & restore.......

+1
Login to vote
Mike 3G's picture

If the vmdk is on a SAN and the NBU backup host has access, then I understand that it can acquire the files from the SAN and write to tape. However, if the datastore is local storage on the ESXi server, how are the vmdks accessed for backup after the snapshot is created?

BTW, our master server is NBU 7.0.1 running on RHEL4 and our backup host is virtualized.

Thanks, Mike

-1
Login to vote
AbdulRasheed's picture

 

Hi Mike,

  If your datastore is on directly attached storage (DAS, the disk storage is internal or dedicated to the system), NetBackup for VMware can protect the virtual machines using NBD or hot-add transport. See my blog on transport methods here: https://www-secure.symantec.com/connect/blogs/nuts-and-bolts-netbackup-vmware-transport-methods-and-tcp-ports

  No matter what transport method you choose, all the V-Ray capabilities are available to you.

 

 

Warm regards,

Abdul "Rasheed" Rasheed

Tweet me @AbdulRasheed127

+1
Login to vote
Scott G's picture

This is a great article (I love knowing how things work), but could you elaborate a little on the file mapping process?

We are having a huge problem with our larger file servers (4-8 TB).  Since VMware has a 2 TB limit, we ended up using dynamic volumes in Windows to acheive the size (I argued for splitting it up into 2 TB shares using basic disks, but was trumped).

The VM backs up sucessfully, but when browsing for file restores on any of the servers, we just see the boot volume.  The data volume is not there, though the backup image is the correct size.  I have not yet been able to do a test restore of the full VM (still trying to reserve enough space to do it).

We are running netbackup 7.5.0.4 and VMware 4.1 (at a fairly recent patch level; I don't have the specific patch level with me).  

We opened a ticket with Symantec, but they can't figure out what is happening.  Any insight would be a real help at this point.

-1
Login to vote
Baljinder Sandhu's picture

Hi Abdul,

I have question on free disk space required for snapshot on SAN.

I need to backup Windows file server with VMhost server it is a VM on vSphere 4, this file server has 3 LUNs (2 TB each) presented from SAN, data drive on Windows server is spanned to all 3 LUNS. All datastores on SAN have only 100GB free capacity. 

How much free space I need on Datastore for snapshot by bpfis?

  

Thanks,

Baljinder

-1
Login to vote
Hollycapri's picture

Hi Abdul,

Congrats for the useful article.

However, all the time you mention "contact vSphere host (vCenter or ESX/ESXi host)".

What did you mean with VCenter or ESX? When is VCenter and when is ESX? How Netbackup knows or work contacting both?

I am asking, because I realized that our network is overloaded, a high MPLS traffic, when backing up VMware machines. In my case, VCenter is in another site, that is being acessed via MPLS, and ESX and Datastore is local. So, what kind of traffic should be sent to VCenter when running backups?

Just as clarification:

- VCenter is on remote site under MPLS

- ESX, Datastore and Netbackup Master Server working at the same local network

Thank you in advance!

+1
Login to vote
AbdulRasheed's picture

Hi Holicapri, 

   The reason I say "vCenter or ESX/ESXi" is because NetBackup V-Ray works whether you have given NetBackup access to vCenter server or to ESX/ESXi servers. If you provide credentials to vCenter server (recommended), NetBackup can protect any virtual machine managed by the vCenter server. Some customers may have a business need to limit access to specific ESX/ESXi hosts. In that case, you can provide NetBackup the credentials to access ESX/ESXi host directly. Of course, that limits protection to only those VMs hosted by the ESX/ESXi host. 

   If your vCenter server is remote (and you have provided vCenter credentials in NetBackup), that system will be contacted for all VADP operations (instructions to quiesce VMs, create snapshots etc.). The only exception is the actual data transport from the datastore which will be done from ESXi VMkernel network directly to backup host. 

   Is your backup host not indended to backup vSphere environments at the remote site? If yes, you may want to configure local ESXi host credentials in NetBackup rather than using vCenter credentials. 

 

 

  

Warm regards,

Abdul "Rasheed" Rasheed

Tweet me @AbdulRasheed127

+1
Login to vote
daza's picture

Hi Abdul,

we have some issues with Restores of the VMware machines that were backed up using VIP. Could you please describe in more details how the tar process is working on the client and can we log it on the client?

thanks

+2
Login to vote
AbdulRasheed's picture

Hi Daza, 

  Hope you are working with our tech support team on your current issue. The restore process flow is something I had been getting requests to blog. I shall write about it sometime in June. Thank you for the comment. 

Warm regards,

Abdul "Rasheed" Rasheed

Tweet me @AbdulRasheed127

+1
Login to vote
Feroz Shaikh's picture

Hello Abdul,

Great Post ! really appreciate all your efforts and time spent to write this blog. Amazing info here and well explained.

Keep up the good work and hope to read all your interesting blogs till now and future posts.

Cheers,

Feroz Shaikh.

-----

Feroz Shaikh

-1
Login to vote
jayeshv's picture

Hi Abdul,

i would like to know how much space is required for creating a snapshot by bpfis on Datastore. An approx percentage which i need to keep a free space on datastore at the time of backup.

 

0
Login to vote
shashi0621's picture

Thanks.

 

But i have some doubts on this:

 

1..vray techonology is only for vm backups?

2..what special vray does on client side as compare to normal backups?
 

0
Login to vote