Backup of Hyper-V Guest OS fails intermittently with Status 156

Article:TECH129884  |  Created: 2010-01-11  |  Updated: 2010-01-11  |  Article URL http://www.symantec.com/docs/TECH129884
Article Type
Technical Solution


Environment

Issue



Backup of Hyper-V Guest OS fails intermittently with Status 156

Solution



Overview:
A Snapshot based Full backup of a virtual machine running on a Windows 2008 Hyper-V Server fails.  

Troubleshooting:
The Activity Monitor shows a series of these errors for the various streams involved:

2/16/2010 10:06:39 AM - Error bpbrm(pid=3808) from client MyGuest.bigdomain.com: ERR - Unable to create fvv_table for mapping: status 2059.
2/16/2010 10:06:39 AM - Error bptm(pid=6816) socket operation failed - 10054 (at child.c.1294)      
2/16/2010 10:06:39 AM - Error bptm(pid=6816) unable to perform read from client socket, connection may have been broken
2/16/2010 10:06:53 AM - Error bpbrm(pid=3808) could not send server status message      
2/16/2010 10:07:00 AM - end writing; write time: 00:00:30
the backup failed to back up the requested files(6)

===========

2/16/2010 10:07:40 AM - end Flash Backup Windows, Delete Snapshot On Exit; elapsed time: 00:00:40
2/16/2010 10:07:40 AM - begin Flash Backup Windows, End Notify Script
2/16/2010 10:07:40 AM - started process RUNCMD (4560)
2/16/2010 10:07:40 AM - ended process 0 (4560)
Status 0
2/16/2010 10:07:40 AM - end Flash Backup Windows, End Notify Script; elapsed time: 00:00:00
Status 6
2/16/2010 10:07:40 AM - end Flash Backup Windows, Delete Snapshot On Exit; elapsed time: 00:00:40
the backup failed to back up the requested files(6)

===========

2/16/2010 8:58:43 AM - Critical bpbrm(pid=5644) from client MyGuest.bigdomain.com FTL - snapshot creation failed, status 156  
2/16/2010 8:59:10 AM - end Flash Backup Windows, End Notify Script; elapsed time: 00:00:01
Status 156
2/16/2010 8:59:10 AM - end Flash Backup Windows, Delete Snapshot On Exit; elapsed time: 00:00:18
snapshot error encountered(156)


Details:

The above error is due to insufficient Shadow Storage space.  The Shadow Storage space exceeded the defined upper limit and Microsoft deleted the snapshot.

Full backups of virtual machines on a Hyper-V host require a large amount of free space on the Hyper-V host to store Shadow Storage snapshots of the virtual machines.

There are two possibilities.
1. There is not enough available disk space to house the snapshots in question
2. The configured Shadow Storage space is insufficient for the quantity of the virtual machines being backed up concurrently, and their accumulated Shadow Storage snapshots.

Although NetBackup uses Windows API's to tie into the Microsoft Volume Shadow Copy environment to create the Snapshot.  The Microsoft VSS environment will take the liberty of deleting the oldest snapshot if the total snapshot volume space exceeds the configured upper limit.

The errors above are directly related to Microsoft deleting the snapshot which NetBackup created due to exceeding the configured upper limit.  This caused subsequent references to the snapshot during the NetBackup job, to fail.  In the above example, NetBackup attempted to create a snapshot.  The act of creating the snapshot caused the upper size limit to be met or exceeded.  Microsoft then deleted the snapshot.  This caused the NetBackup job to fail.  Then, at the end of the job, the snapshot was not present and therefore NetBackup's attempt to delete it failed.

Solution / Workaround:
1. Examine the amount of free space on the Windows Volume which houses the virtual machines.

NOTE: A Virtual Machine snapshot will vary in size depending on the size and number of the vhd files per virtual machine. Multiply that with the number of concurrently run jobs. A general rule of thumb when determine the amount of Shadow Storage space is need is to have about 1 GB of free space for every 5 GB of .vhd file size.  So, if the .vhd file is 25 GB, you may want to have at least 5GB of free space in anticipation of the snapshot being around that size.

2. Examine the Storage Area settings on the Hyper-V host for the volume which houses the .vhd files. To do this, launch Windows Explorer > rt-click a volume > Configure Shadow Copies > select the proper volume > Settings.  Set the Storage Area size to "Use Limit" and specify an amount of space sufficient to house the combined Virtual Machine snapshots, but does not exceed the amount of available space on the volume.


Caution:  Setting the Storage Area size to No Limit can have unexpected consequences. See TechNotes 351298, 327791

3. Reduce the number of concurrent Virtual Machine backup jobs per Hyper-V host, so the combined Shadow Storage space does not exceed what is available and configured on the Hyper-V host.


Examination:
To get an idea of how large a snapshot actually is for a particular Hyper-V virtual machine, follow these steps:
1. On the Hyper-V host, open Windows Explorer and navigate into the \System Volume Information\ folder on whatever drive letter houses the Virtual Machines

NOTE: You may have to give yourself "Read" rights to view the contents of this folder
NOTE: The \System Volume Information\ folder is the location where VSS stores its snapshots

2. Launch a backup of a virtual machine housed on that particular Hyper-V host

3. The first thing the backup does, is take the VSS Snapshot.  Once the snapshot file appears in the \System Volume Information\ folder, you will be able to observe its size



Legacy ID



352206


Article URL http://www.symantec.com/docs/TECH129884


Terms of use for this information are found in Legal Notices