Video Screencast Help
Scheduled Maintenance: Symantec Connect is scheduled to be down Saturday, April 19 from 10am to 2pm Pacific Standard Time (GMT: 5pm to 9pm) for server migration and upgrades.
Please accept our apologies in advance for any inconvenience this might cause.

Nuts and bolts in NetBackup for VMware: Transport methods and TCP ports

Created: 14 Oct 2011 • Updated: 22 Jan 2013 • 40 comments
AbdulRasheed's picture
+19 19 Votes
Login to vote

It had been a while since my last blog on NetBackup for VMware. The next in line in that series was restore process flow. However, since there had been many questions on transport methods and TCP ports, let us talk about those for now. I will get to restore process flow soon!

Thus this blog is just an addendum to Understanding V-Ray vision through backup process flow.  As that blog was too long already, I didn’t spend much time explaining various transport types and TCP ports but I received so many questions on those. Sorry for the delay.

Let us examine the transport types and ports usage through the view from VMware backup host.

There are three unique ways a backup host can stream data from VM data store. They are SAN, hot-add and NBD transports.

SAN Transport

  The VMware backup host must be a physical system in order to use SAN transport.  The data store LUNs are zoned to the backup host. The backup host can directly read VMDK objects from the datastore using vADP. The data stores can be Fibre Channel or iSCSI connected.

Pros:

  True offhost backups, zero resources tapped from ESX/ESXi hosts

Cons:

  Works only for SAN attached (Fibre Channel or iSCSI) data stores

Hot-add Transport

  You can think of hot-add transport as a special case of SAN transport where the backup host is also a virtual machine. Plus there is a bonus, it works for non-SAN data stores as well. The backup host VM accesses the VMDK objects of other VMs directly from the data store (similar to SAN transport). It can protect all VMs on datastores to which it has access.

Pros:

  No need for a physical backup host

  Does provide offhost backups for those VMs not collocated with VM backup host

  The most efficient backup method for NFS based data stores (e.g. NetApp or FileStore providing VM storage)

Cons:

  It does use ESX server resources where the VM backup host is deployed

  You cannot protect VMDK files larger than 1TB (a vADP limitation with hot-add)

NBD Transport

   NBD stands for network block device. In this method, VMware Network File Copy (NFC) protocol is used such that the VMDK object looks like a block device to the backup host which can be read through network. You need one NFC connection for each VMDK file being backed up. Thus backup is streamed from ESX/ESXi system’s VMkernel port to VMware backup host. If your ESX/ESXi hosts have VMKernel ports (known as the management ports) with dedicated uplinks, your virtual machine traffic links are not affected.

Pros:

  Works for all kinds of data stores, even the ones directly attached (DAS) to ESX hosts

  The simplest one to setup from infrastructure perspective works with both physical and virtual backup hosts.

Cons:

  Not an offhost solution.  ESX resources are used for backups.

Which one is right for you? Well, it depends on your business needs. For a large enterprise data center where you already have invested heavily on Fibre Channel SAN for your vSphere infrastructure, SAN transport is ideal. Hot-add can also be used especially if you like to spread backup hosts onto to multiple multi-node ESX clusters. With the increasing popularity of 10Gb Ethernet, NBD is not inferior either.  Good news is that NetBackup has the ability to automatically try various transports during backups.

TCP ports to vSphere infrastructure

  What ports are needed from NetBackup to vSphere infrastructure? The answer is quite simple; you need access to TCP port 443 and 902.

Ability to connect to TCP port 443 on vCenter server is mandatory. This is the port at which NetBackup connects for all things needed from vCenter server. VM discovery requests, snapshot creation, snapshot deletion etc.

The backup host may also need the ability to connect to TCP port 902 on ESX/ESXi hosts. This is needed for specific use cases.  If these are not applicable to your environment, no need for this port to be opened in firewall.

   1. You are using NBD/NBDSSL transport for backups and/or restores.

   2. You are doing restores through Restore ESX server bypassing vCenter

SAN and NBD Transports using a physical VMware backup host

Hot-add and NBD Transports using a physical VMware backup host

Back to Nuts and Bolts in NetBackup for VMware series

Comments 40 CommentsJump to latest comment

captain jack sparrow's picture

Thanx Abdul for this information... This one was more informative unlike previous series of yours....

 

 Cheers !!!

CJS

 

0
Login to vote
AbdulRasheed's picture

It is only through the feedback that we would know if things like these are beneficial for the esteemed community and whether more like this need to be produced. I appreciate your time and feedback.

Warm regards,

Abdul "Rasheed" Rasheed

Tweet me @AbdulRasheed127

-4
Login to vote
margoe's picture

 Thank you for this information, but which ports Master server and vCenter (or  ESX) server use for communication?

 

margo

+4
Login to vote
AbdulRasheed's picture

Hi Margo, 

 

Technically speaking, the master server does not talk to vCenter server directly for backups or restores. Those are always managed via VMware backup host.

The exception to this rule is when you are browsing the virtual infrastructure (e.g. you are testing the query string for VIP) from an administration console connected to the master server, in this case it tries to communicate to vCenter/ESX server at port 443.

Hence the short answer to your question is…..   NetBackup master uses port 443 to talk to virtual machine server (vCenter or ESX depending on what you had configured) if the specific configuration task requires a connection.

If you have a security requirement to limit connections to vCenter, you can house Windows Administration Console role on VMware backup host. In this case, the master never needs to connect to vCenter directly.

Warm regards,

Abdul "Rasheed" Rasheed

Tweet me @AbdulRasheed127

0
Login to vote
craigsale's picture

Firstly great post.

I need some clarification if using SAN transport type on Windows Server 2008 for Vmware backup host.

NBU 7.1

Do the LUNs need to be just visible and Offline or do they need to show as Online.

This is for a backups.

I am aware of the Vmware best practices but I have seen some users stating the backups have been working with LUNs showing as offline but I contradict this from previous experience.

http://kb.vmware.com/selfservice/microsites/search...

NBD works and SAN fails with No Transport Method available for the given type.

 

 

0
Login to vote
AbdulRasheed's picture

Hi Craig,

  You bring up an interesting point. This is an area that is controlled by VMware API and we have seen mixed results with different versions of API.

   Strictly for backups, the SAN policy may be set to OfflineShared. I have seen this configuration working for many customers. However for restores, you would need to switch the SAN policy or online the LUN in question explicitly.

Warm regards,

Abdul "Rasheed" Rasheed

Tweet me @AbdulRasheed127

+2
Login to vote
Alize's picture

May I know if the Hot-add mode is appliable to 5220 appliance? I was told that in order to utilize SAN transport, we must use a physical Backup host instead of a VM backup host which will only connect via LAN.

0
Login to vote
AbdulRasheed's picture

Hi Alize,

  You can use hot-add mode with 5220 appliance. In fact, this is a popular method with NetBackup 5220. Large environments use multiple hot-add hosts (the VMware backup hosts on VMs) which act as the V-Ray lenses for NetBackup 5220 appliance. This way you can scale-out streaming of backups from 1000+ VMs to a single appliance! You get to deduplicate the backup stream closer to the source or you could send the whole stream to appliance to do the deduplication. In NetBackup for VMware design, these hot-add hosts can send multiple streams concurrently to the appliance.

  SAN transport is different from hot-add transport. SAN transport requires a physical backup host that is zoned to see the vSphere data stores. In those cases, NetBackup 5220 currently requires a seperate physical backup host because some of the key APIs (from VADP) are not ported to Linux (appliance OS uses Linux kernel) yet. Symantec (as usual) is taking the leadership role and working with VMware to make the port of required APIs to Linux possible.

Warm regards,

Abdul "Rasheed" Rasheed

Tweet me @AbdulRasheed127

+2
Login to vote
StefanosM's picture

Recently I install a netbackup VMware solution to a customer using the hotadd method.

The backup systems ware in a different network subnet form ESXs and vcenter. As there is NO document from Symantec that described the firewall ports need to be opened, I folow your article and sent to the customer an email to open the ports 443 and 902.

Guess what, it did not work. I open a case to Symantec asking for ports and the answer was the following ports

SSL            443
SSH            21 & 22
http             80
vnetd          13724
veritas_pbx 1556
bpcd           13782
vopied         13783  (????)

No mentioning at all the 902 port, which I thing is needed. And he added to the email:
"Please not that we offer the VMware ports as advised but unconfirmed information and that you should always confirm the VMware ports with VMware themselves"  

After that we add a second virtual network card bypassing the firewall. It is not good to do test to the client’s environment
I really do not know the ports needed, but definitely are not only 443 and 902. 

Reading the veem manual, it has a page with a big range of ports need to be open for VMware backups. But every backup software, is different

stefanos.

-4
Login to vote
AbdulRasheed's picture

 

Hi smtp,

 

I see your confusion here. Note that the blog above is talking about the ports needed between VMware backup host and vSphere infrastructure. I didn’t spend anytime going through ports needed between VMware backup host (which is essentially a NetBackup client or NetBackup media server) and rest of the NetBackup environment. This part is well covered in NetBackup system administrator’s guide.

 

In your specific example, you are using hot-add which is a VMware backup host on a VM. The above blog only deals with communication requirement between this VM and your VMware infrastructure. As your media server(s) and master server are on different network, you also need to follow the instructions in NetBackup System Administrator’s guide on how to enable communications between master/media servers and NetBackup client.

 

Hope this helps. If you have difficulty in locating documentation for server to client communication for NetBackup, send me a note and I shall find it for you. 

Warm regards,

Abdul "Rasheed" Rasheed

Tweet me @AbdulRasheed127

-2
Login to vote
StefanosM's picture

Probably you do not understand what I’m writing. It maybe my fault as English is not my native language.

My problem is the communication from netbackup backup host and the VMware infrastructure.  And My bigger problem is the answer from support…

-2
Login to vote
bcblake's picture

Thanks for the write up. I've been trying to find something similar produced by VMware but most of their documentation is related to the VDDK and VADP APIs.

So in your statement:

"Thus backup is streamed from ESX/ESXi system’s VMkernel port to VMware backup host. If your ESX/ESXi hosts have VMKernel ports (known as the management ports) with dedicated uplinks, your virtual machine traffic links are not affected."

Any idea which VMkernel port is used? If you have multiple VMkernel ports defined (for example, one dedicated for vMotion and another on a separate VLAN or dedicated 10GbE link for NFS datastores or dedicated for mangement traffic), what is the logic to decide which VMkernel port to use?

I know with ESXi this changes slightly as there no longer a service console port, and when configuring a new network port you only have connection types for Virtual Machine (VM port group) or VMkernel. But when adding a VMkernel port, you can specifically select the port properties to include: vMotion, Fault Tolerance logging and Management Traffic. I guess the question there is: do any of these port properties HAVE to be selected in order for the NFC traffic to use the port, or will it use any VMkernel port... and if so, which VMkernel port (hopefully it will be smart enough to try to use one on the same subnet if it exists).

Thanks

+2
Login to vote
AbdulRasheed's picture

 

Hi bcblake,

Unfortunately, VDDK documentation does not state specific requirements as NBD inherits its root from Linux style kernel module. Technically, NBD can function so long as there is a path from backup host to any of the VMkernel port types. Having said that I would recommend not mixing vMotion traffic with backups as the former might be responding to DRS rules. As a best practice, I would suggest creating a dedicated network segment that backup host and an ESXi management port can use. You can create multiple management ports on an ESXi host, allocate one port to production segment for vCenter to manage the host, allocate another port to a completely different non-routable segment for backup traffic. This also allows to securely route backup traffic without going for processor intensive NBDSSL transport. 

Hope this helps. 

Warm regards,

Abdul "Rasheed" Rasheed

Tweet me @AbdulRasheed127

0
Login to vote
bcblake's picture

Thanks Abdul.

I figured this may be the case, and I am working with an environment where a new NIC will be added to a group of ESX servers and they will be placed onto a dedicated 10GbE network that will be non-routable. So as long as the VMware Access Host is on that same network, then I'm thinking normal routing rules should apply and the ESX server will decide to use the 10GbE NIC to route NBD traffic to the VMware Access Host. I just wanted to make sure that when we add the new NIC to make sure it is added to an appropriate vSwitch and what type of connection type should be used (VMkernel), which is what you have confirmed.

Brian

+2
Login to vote
captain jack sparrow's picture

Hi Abdul

 

you Nuts & bolts series had helped alot me in many cases to design  and provide robust solution for VMware

 

however V-Ray is tighten with both virtualization market leader. Looking forward the series for Hyper-V too.

As far i know presently there are limitation from Hyper-V end , but need to know What NetBackup can benefit to this customers

 Cheers !!!

CJS

 

0
Login to vote
AbdulRasheed's picture

Appreciate the feedback. 

Warm regards,

Abdul "Rasheed" Rasheed

Tweet me @AbdulRasheed127

+2
Login to vote
captain jack sparrow's picture

Awaiting .....

 Cheers !!!

CJS

 

+1
Login to vote
The Director's picture

Thanks for the information. I wonder if there is any documentation that expands on the best practices or recommendations for NFS backups of VMware? In your example above you show the VMware Backup Host as a VMware guest. I cannot find any references to doing this in the NetBackup 7.1 Admin Guide for VMware.

+2
Login to vote
AbdulRasheed's picture

 

Hi tD,

 

   Good catch. I understand that we do not call that out explicitly in out documentation. I shall work with our writers to add a section on NFS data stores.

 

  If data stores are on NFS, you can use any transport method other than SAN transport. That gives us hot-add, NBD and NBDSSL. The choice between hot-add and NBD may be determined as follows.

 

1.       If you do not want ESXi network stack to be used for backups at all, hot-add is recommended. Place the hot-add systems on strategic ESXi hosts such that it has access to all the data stores in your environment. If you have VMware HA, let it be monitored by it so that it is restarted on a different ESX host in case the one currently hosting fails. I would also recommend setting the resource pools such that you have enough reservations on CPU and memory for the hot-add host. With multiple hot-add hosts, you can scale-out backup processing.

2.       If you are using an enterprise grade NFS storage with dedicated network where VMkernel port is attached, you can simply use NBD or NBDSSL. This works great and also eliminates the operational complexity in managing hot-add VMs. The downside with NBD/NBDSSL is that it is not a 100% offhost solution as you do use the network stack from ESXi hosts. With modern day hardware, it may not be a concern.

 

 

 

Warm regards,

Abdul "Rasheed" Rasheed

Tweet me @AbdulRasheed127

+2
Login to vote
captain jack sparrow's picture

Hi Abdul,

 

Can you also share some light on Hyper-V integration too

 Cheers !!!

CJS

 

0
Login to vote
AbdulRasheed's picture

I am travelling a lot next month. Long flights are my faviourite time to write technical articles!  I shall keep this in mind. Thank you for the suggestion. 

Warm regards,

Abdul "Rasheed" Rasheed

Tweet me @AbdulRasheed127

0
Login to vote
AICT-IT's picture

Hello AbdulRasheed, 

we have upgraded from 7.1 -->7.5--> 7.5.0.1 however we always face a slow performance issue when backup vmware vShpere 5 (7 x HP BL460c) max to 1 or 2 MB/s when using SAN or NBD, the backuphost is the master server too, Windows Server 2008 R2, and device is HP MSL 6060 LTO 3 all are FC 4Gbps,

Tape Buffers: 128 x 256K

Regards,

 

Hany Hammad

+2
Login to vote
AbdulRasheed's picture

Hi Hany,

   Are you saying that the performance went down after the upgrade? I would highly recommend working with support on this one. We can isolate the problem using a NULL storage unit to see if the issue is on write side or read side of the job. 

 

Warm regards,

Abdul "Rasheed" Rasheed

Tweet me @AbdulRasheed127

0
Login to vote
AICT-IT's picture

Hello AbdulRasheed,

No, we had this problem from the beginig , we've worked with support and no progress even when we used null stu, I think it is related to the reading side because I've tried the SKIP_DISK_WRITES, however nothing changed,

 

Thanks,

 

Hany Hammad

-2
Login to vote
Rami Nasser's picture

Zone all data stores to that Vmware Backup Host Server with Read/Write rights so that you can see storage in the Disk Managment Tab Under Storage

0
Login to vote
captain jack sparrow's picture

Regarding your query for assigning those Data Stores to Backup host, we strictly rely on zoning , we do not want those LUNs to be initialized , or mounted

Below are the steps that we need to follow

Run Following syntax from Backup Host . Run CMD with administrative privileges

·         Diskpart automount disabled

o    Disabling automount would prevent Windows from automatically mounting any new basic volumes that are added to the system.

·         Diskpart automount scrub

o    Scrub would Removes volume mount point directories and registry settings for volumes that are no longer in the system. This prevents volumes that were previously in the system from being automatically mounted and given their former volume mount point(s) when they are added back to the system.

Above syntax must be executed prior to zoning is done. Also make note that above service are by default enabled in Windows 2k8 family, and they are restored back to default state after patch is applied. So we need to disabled them manually

·         Disable Virtual Disk Service

o    After zoning is done and LUNs are visible in Disk management as Basic Online with Healthy status , Disable Virtual Disk Service from Services console. This would prevent this LUNs being visible to others accessing Backup host

Hope above clears your doubts

 Cheers !!!

CJS

 

+1
Login to vote
Rami Nasser's picture

thanks for your direct explaination for zoning

+2
Login to vote
Robert-J's picture

For backups : Present the LUNs as readonly,

For restores : Use NBD which goes over the IP network (better if its 10 Gib/s)

The real gotcha is when you patch or apply windows maintenance packs I have observed that automount is set to the default which is enabled after the reboot

If the LUNs are automounted expected them to be initialized and all the data that is your job to protect is now usless.  

 

0
Login to vote
AbdulRasheed's picture

 

Hi Robert, 

   What you propose is certainly a great safety net. On Windows 2008 or Windows 2008 R2, you can set SAN policy to onlineAll and make SAN disks as read-only. 

   Fortunately, now you also have the option to use NetBackup 5220 appliance or a RedHat Linux system as a backup host! For large production environments, these provide much safer method to do SAN transport backups as you don't have to worry about the notorious behavior in Windows systems.

Warm regards,

Abdul "Rasheed" Rasheed

Tweet me @AbdulRasheed127

+4
Login to vote
I like 3.14's picture

I'm a bit late to this string, but had a question directly related to this post (Abdul), and don't like to start new strings unless needed.. 

 

There are a few hoops to jump through to use a Windows host for SAN backups for VMWare, as previously mentioned (then you have to worry about updates resetting your "safety net", also mentioned previously).

 

I'm using a 5220 appliance, and am getting ready to set up SAN backups for VMWare.  The above post seems to suggest that there's no special configuration needed, as there is with Windows.  Is this correct?  Can I just zone the appliance to the data stores and set up the policy, or is there something else I need to do.

 

I have looked through the documentation, but to be honest, the directions for setting up SAN-based VMWare backups, as provided in a different string from Abdul, is much better and easier to follow.  The manual for VMWare backups is not very straightfoward, IMO.  Thus my post here.  ;)   And don't get me started on the SAN Client admin guide... (my next project).

+2
Login to vote
AbdulRasheed's picture

Hi, 

  You are right, no special safety net configuration is required if you are using NetBackup 5220. Here is what you would be doing. 

  1. Identify the World Wide Names of the FC adapters you are going to use from NetBackup 5220. This can be done from CLISH interface under Fibre Channel menu. 

  2. Provide those to your SAN administrator so that he/she can add those to the zone where datastore LUNs are visible

  3. (optionally) Verify the NetBackup 5220 can see the data store LUNs. Run scan sub-command, followed by show sub-command to verify this. 

   That is all you need to do. 

Appreciate the feedback. If there are specific sections you would like us to rewrite (I shall also go through it), I will be happy to notify the writers. Let me know if I can help. I am travelling a bit until December 14. My responses may be delayed. 

 

Warm regards,

Abdul "Rasheed" Rasheed

Tweet me @AbdulRasheed127

-2
Login to vote
Suny's picture

Hi Abdul,

One of my customer was using NetBackup access credentials for the ESX servers to authenticate and backup VMs but recently they setup a vCenter so removed the ESX servers and added vCenter.

Now they are facing Network choke issue when running backups and we are using transport types as"Try all".

Do i need to Zone the LUN to the vCenter also?

http://www.symantec.com/connect/articles/netbackup-vmware-backup-over-san

Regards,

Suny

0
Login to vote
AbdulRasheed's picture

Hi Suny,

  No, vCenter server need to see the datastore LUNs. There is no need to do zoning for vCenter server. 

  I hope that SAN is listed higher in the priority than NBD. Can you validate the SAN transport is indeed functional by unchecking 'Try all' and specifically selecting SAN Transport? 

Warm regards,

Abdul "Rasheed" Rasheed

Tweet me @AbdulRasheed127

-2
Login to vote
captain jack sparrow's picture

repeating my comment from same post

You do require to zone datastore LUN to backup host in order to leverage SAN transport benefits.

 

Regarding your query for

Regarding your query for assigning those Data Stores to Backup host, we strictly rely on zoning , we do not want those LUNs to be initialized , or mounted

Below are the steps that we need to follow

Run Following syntax from Backup Host . Run CMD with administrative privileges

·         Diskpart automount disabled

o    Disabling automount would prevent Windows from automatically mounting any new basic volumes that are added to the system.

·         Diskpart automount scrub

o    Scrub would Removes volume mount point directories and registry settings for volumes that are no longer in the system. This prevents volumes that were previously in the system from being automatically mounted and given their former volume mount point(s) when they are added back to the system.

Above syntax must be executed prior to zoning is done. Also make note that above service are by default enabled in Windows 2k8 family, and they are restored back to default state after patch is applied. So we need to disabled them manually

·         Disable Virtual Disk Service

o    After zoning is done and LUNs are visible in Disk management as Basic Online with Healthy status , Disable Virtual Disk Service from Services console. This would prevent this LUNs being visible to others accessing Backup host

Hope above clears your doubts

 Cheers !!!

 

 Cheers !!!

CJS

 

0
Login to vote
AbdulRasheed's picture

Thank you captain! 

Warm regards,

Abdul "Rasheed" Rasheed

Tweet me @AbdulRasheed127

-2
Login to vote
Rami Nasser's picture

Thanks AbdulRasheed for your efforts.

I have one question related to zoning between the VMware backup host and the data stores:

I'm using  the appliance 5220 as VMware backup host(SUSE Linux OS) ,when scaning the storage after zoning I can see the data stores WWN but I could not see the Paths to the LUN's.is this normal or I should see the path also? FYI the snapshot succed but the backup failed with status 23 then 6.I read many of comments regarding this but the SAN team said : as you can see the wwn of the storage this mean from there side no thing to do.kindly find attached snapshot .

Regards,

new added zone2.png new lscsii2.png
-4
Login to vote
AbdulRasheed's picture

Hi Rami, 

   You should see WWPN as well. It looks like the storage team had done LUN masking in such a way that the appliance is not an honored host to see the LUN. 

  I recommend getting Symantec's Technical Support team and your Storage team to be on a call/WebEx session to sort this out. 

PS: Sorry about the delay. This blog was written long time ago and I didn't realize that you had questions posted here. 

Warm regards,

Abdul "Rasheed" Rasheed

Tweet me @AbdulRasheed127

+2
Login to vote
Vikas D's picture

Hi Abdul

 

Thanks for such an informative post. We have a situtaion where we are using 5220 Appliance and need to backup Virtual Machines on VMware infrastrucutre. The SAN is running iSCSI protocol using Cisco 3750 switch. It seems that we cannot use SAN transport backup as we would not be able to connect the appliance to the SAN switch since iSCSI is not supported. 

The bandwidht on the box is also limited to 1G due to a single NIC that is default.

Can you please advise the best way to backup the VMs and use deduplication withouth affecting the ESX servers. We were thinking of the below few:

1. Increase bandwidth to 10G on the box using add on card and backup using 5220 as VM backup host.

2. Increase bandwidth to 10G on the box using add on card and backup by making vCenter server which is a physical server as backup host.

3. Do you know if there is any way to use iSCSI and connect directly to SAN from the appliance.

 

Thanks

Vikas

 

-2
Login to vote
captain jack sparrow's picture

Did you tried going with Client side de-dupe? VMware and hyper-v are best candidate for maximum de-dupe rates.

If you could wait for couple of weeks or say a month 7.6 is making its way as FA was recently announced. So it should be on the way to appliances. It has got accelerator too and there were alot improvements on vmware (saw on google hangouts).

If time is constraint for now,  Following should be best for you to try.

Measure current performance hit during backups on ESX. (single VM vs Concurrent)

With and Without client end Dedupe

1Gb should be enough as far as its dedicated for backups. and also depends on data size to be backed up vs backup window available and queued jobs.

Moving to 10GB is best , but you should also factor source disk read speed, CPU utilization and how much is Appliance is busy (after all it would be also busy writing data ;-).

Test, Tune, Result  and repeat .. This would be best  to achive optimum results for your domain.

 

 Cheers !!!

CJS

 

-6
Login to vote