Video Screencast Help
Search Video Help Close Back
to help
New in the Rewards Catalog: Vouchers for "Symantec Technical Specialist" and "Symantec Certified Specialist" exams.

VMWare backups are too slow

Updated: 27 Aug 2010 | 11 comments
Jay Son's picture
+1 1 Vote
Login to vote

NBU 6.5.3
Solaris (will soon be AIX)

Process = We take a snapshot of the VMFS LUN and schedule a backup (in NBU) to backup to tape. The Backup Selection we have is the path of the snapshot (on a Filer)

Problem = Our VMFS snapshot backups to tape are taking 25-30 hrs long This is all conected via fiber (disk (lun) to tape device). Our throughput is less than 10mb per second. Some of these snapshots are almost 800gb (w/ a throughput of 7-8mb per second). Unacceptable.

Qusetions = How can I speed things up (throughput) from disk to tape? What is the best way to backup VMware? Are there other technologies for backing up VMware in NetBackup? Can you tell me more about VCB?

Thanks
Jay Son

Comments

Stuart Green's picture
28
Jul
2009
1 Vote +1
Login to vote

6.5.4 fix available to address slowness

Worth a try....

Technote: 311193
GENERAL ERROR: VMWare Consolidated Backup(VCB) using NetBackup 6.5.x are slower than expected.

ENVIRONMENT/CONDITIONS:
Symantec uses a 32K Read Block size which can lower the performance of a VCB Backup. Because most disk are 64K or higher, a 32K Read Block can be insufficient.

SOLUTION/WORKAROUND:
ETA of Fix:
Symantec Corporation has acknowledged that the above-mentioned issue (Etrack 1402540 and 1401802 and 1424613) is present in the current version(s) of the product(s) mentioned at the end of this article. Symantec Corporation is committed to product quality and satisfied customers. This issue was scheduled to be addressed in the following release:
NetBackup 6.5 Release Update 4 (6.5.4)
When NetBackup 6.5.4 is released, please visit the following link for download and readme information:
http://www.symantec.com/enterprise/support/overvie...

Implementation information:
Buffer size tuning for this EEB is accomplished by creating the touch file 'SectorsPerBuffer' under <install_path>\Veritas\NetBackup\. Then add the desired buffer size in units of sectors. By default, most disks have a sector size of 512 bytes. To make a 2MB buffer, we add 4098 to the tuning file [512 b * 4098 = 2097152 b]

For Best Results:
1. Reformat VMware staging volume on the proxy server with an NTFS cluster size of 64KB.
2a. After applying the EEB into .\Veritas\NetBackup\bin
2b. Create the tuning file SectorsPerBuffer
3. Start with a tuning value of 4098 to set the internal buffer size of bpbkar32 to 2MB.
4. Set the disk storage unit fragment size to 2048 MB.

Tip: Get overview/document your NBU environment. Run 'nbsu' and review the output.

• If this provides help, please vote or mark appropriate solution.

Jay Son's picture
29
Jul
2009
1 Vote +1
Login to vote

Stuart - Do you know a George

Stuart - Do you know a George Winter? He will be presenting a "blueprint" on how to speed up you vmware backups. This will be held at San Francisco, CA. The session is "Creating the Fastest Possible Backups Using VMware Consolidated Backup - A Design Blueprint " I was hoping you can contact him and take some notes as to how this will help our issue. Unfortunately, I will be on vacation (finally). So, I will not be able to attend VMWorld 2009.

http://www.symantec.com/business/theme.jsp?themeid...

Thanks
Jay Son

Stuart Green's picture
29
Jul
2009
1 Vote +1
Login to vote

I wish I could be there. Pity

I wish I could be there. Pity the rewards points couldn't be exchanged for air miles.! Now there is an idea.!

This certainly looks an interesting session. George is the author of the white paper from Symantec on NetBackup for VMware.

I hope this presents new information and possible tweaks and tips to squeeze every drop of speed out of any VCB environment.

It might be made available online after VMWorld. Worth keeping an eye out for.

Tip: Get overview/document your NBU environment. Run 'nbsu' and review the output.

• If this provides help, please vote or mark appropriate solution.

Albert Widjaja's picture
28
Jul
2009
1 Vote +1
Login to vote

Backup Exec 12.5 + VCB v1.5

Hi All,

I'm also having problem with the Backup Exec 12.5 + VCB v1.5 to backup my VM LUN, the speed is so slow, to backup 800 GB it took 55 hours, I'm wondering if there is any solution for the Backup Exec 12.5 as well ?

Cheers,
Albert

/* Infrastructure Support Engineer */

Jay Son's picture
23
Sep
2009
0 Votes 0
Login to vote

Stuart - I just wanted to

Stuart - I just wanted to follow-up on our conversation about a month and a half ago. Do you know anymore information regarding the VMWorld 2009 Connference with George Winter? If there is, are their links to his presentation, white paper, Webcast, etc. that you can provideme, it would really help.

Thanks
Jay Son

wrobbins's picture
23
Sep
2009
1 Vote +1
Login to vote

Please sign-in to your VMworld account

Please sign-in to your VMworld account to view details for this session or lab.
If you do not have a VMworld account, please create a free VMworld account.

www.vmworld.com/docs/DOC-2965

~ Bill

Stuart Green's picture
29
Oct
2009
3 Votes +3
Login to vote

2 things since this post

1. Found a very good document on VMware's VIOP site called "Setting UP VCB".
In there the following statement

For FullVM type backups, copy speed to the Holding Tank is limited to about 1GB per minute. This is actually a limitation within “cmd.exe” and not VCB.

Roll on a Linux VCB proxy is what I say. But based on VMware's own product initiative this might not happen as vSphere has its own Data Recovery Appliance. There could be still hope for a Linux VCB proxy and file level backups, as mentioned as during session at VMworld 2008.
http://vmetc.com/2008/09/17/linux-strategy-and-roa...

Anyway heres the doc. Very well written. Helpful resource.
http://viops.vmware.com/home/docs/DOC-1392

2. I have since addressed the issue when performing a 3-FullVM and file for incremental by applying a 6.5.4 EEB that adds the monolithic export config to the snapshot options. Using this you are able to choose the 2GB chunk method.

netbackupforvmware_snapshotclient_options.png

Here is a recent published technote with the Etrack # for the EEB.
GENERAL ERROR: Large FullVM Snapshots using VMware Consolidated Backup(VCB) are slower than expected

Using the 2GB chunk I can confirm this speeds up the backup. From 36 hours to 3 hours for a 240GB LUN.

UPDATE: enhancements in NetBackup 7.0 if you are considering VMware and NetBackup going forward.

  • Fully integrated with VMware vStorage APIs (vSphere 4)
    VCB is no longer required. This feature eliminates the VCB copy of a VM snapshot to the VCB proxy staging area. NetBackup backs up the snapshot directly from the ESX server.
  • New VM and file recovery from incremental backups
    VMware vSphere now provides block-level incrementalVMbackups. Moreover, NetBackup now provides VMware and file-level recovery fromthe same set of full and incremental backups.
  • New Recovery Wizard
    This wizard simplifies the process of performing a direct or redirected VM recovery.
  • Tighter vCenter integration
    More VMware identifiers and attributes are available when configuring a NetBackup policy, including VMware hostname, display name, UUID, IP address, virtual center, datacenter, ESX platform, and others.

Tip: Get overview/document your NBU environment. Run 'nbsu' and review the output.

• If this provides help, please vote or mark appropriate solution.

Albert Widjaja's picture
29
Oct
2009
1 Vote +1
Login to vote

Slow NBD Transfer

Hi Stuart,

Regarding the Slow NBD transfer rate is, that because of the cmd.exe (Windows Server VCB Proxy) or NTFS file limitation or the shared link between ESXi console and the VMKernel (VMotion) ?

/* Infrastructure Support Engineer */

Karthikeyan Sundaram's picture
26
Feb
2010
0 Votes 0
Login to vote

Thanks Stuart

I just installed NBU 6.5.5 on my master server(acting as proxy server too), and disabled monolithic export option.

I have initiated 20 GB VM(Contains 10 GB of Data) Backup. I could see 11(s001,s002..) files in Mount Point. But i see the file size of those differs each.

How netbackup is sizing those files ?

Why did i get 11 Files ? ( is it based on Size of VM)

How this Option works ? 

       I am seeing, First s001 vmdk file copy started once that is completed, its starts s002 and so on.. but this is not initiating copies parallel.. Am i doing some wrong or is that how it works ?


Thanks, Karthikeyan Sundaram.

sdw303's picture
24
Sep
2009
2 Votes +2
Login to vote

Using v6.5.3.1 + EEBs for

Using v6.5.3.1 + EEBs for non-monolithic 2 GB chunks... NBU client buffer size of 64KB, and using the SectorsPerBuffer file for bpbkar32, and Windows formatting the staging area on the VCB server with NTFS cluster size of 64KB - and creating the staging area as RAID 0+1 with stripe set with 64KB stripe width across local 4 x 146 GB 15krpm UltraSCSI 320.

I've seen over 240 MB/s of VCB snapshot data coming up dual fabric SAN pipes (from NetApp 3070 FC-AL storage) onto the local fast disk - for three concurrent VCB backups.

Bpbkar32 tuning file on the server running VCB:
C:\Program Files\Veritas\NetBackup\SectorsPerBuffer
65536

...i.e. the quantitiy of 512 byte sectors to use.
           4096 =  2 MB
Therefore 65536 = 32 MB

...i.e. tells bpbkar32 to read the VMDK files in chunks of 32MB, before breaking the data into multiple client "BufferSize" fragments for transport to the media server's NET_BUFFER_SZ.

Zahid.Haseeb's picture
29
Oct
2009
1 Vote +1
Login to vote

 See

 See this
https://www-secure.symantec.com/connect/forums/vmware-backup-slow


Any comment will be appreciated. Mark as Solution if your query is resolved
__________________
Thanks in Advance
Zahid Haseeb

zahidhaseeb.wordpress.com