How to troubleshoot Exchange 2007 backups that is failing due to VSS (Volume Shadow Copy Service) Issues

Article:TECH58562

Created: 2010-01-05

Updated: 2013-05-08

Article URL http://www.symantec.com/docs/TECH58562

Article Type
Technical Solution

Product(s)

Show all

Languages

Show all

Problem

How to troubleshoot Exchange 2007 backups that is failing due to VSS (Volume Shadow Copy Service) Issues.

Solution

Below are the troubleshooting steps to resolve the VSS related issue with Exchange backups, following the steps in below given sequence should help resolve the issue:

Open the Windows Event Viewer and check system events for any VSS specific errors during the time of the backup.
Verify the VSS Writers are stable
Run the command right after the backup fails if at all possible.:
VSSADMIN LIST WRITERS
*If failure on any of the writers, step 3 should be considered. If Step 3 does not resolve or does not apply, Microsoft should be contacted to resolve VSS issues
Update the current VSS installation (IF POSSIBLE)
As of December 19, 2007 - A Microsoft VSS Hot fix (MKB 940349) is available.
Update VSS with the following patch or later if it is not already installed:

Availability of a Volume Shadow Copy Service (VSS) update rollup package for Windows Server 2003 to resolve some VSS snapshot issues
http://support.microsoft.com/kb/940349

Verify the latest version of the Microsoft Exchange Server MAPI Client and Collaboration Data Objects (MAPI/CDO) is installed on the Exchange Server. (Version 6.05.7888 or later) http://support.veritas.com/docs/287937

How to verify what the version of MAPI/CDO (6.05.7888) or greater, is installed on the Exchange Server.
Step 1. Search the Server for the CDOsys.dll file
Step 2. Check the version number of the file
If the file version is older than 6.05.7888, the latest version of the MAPI/CDO client as of 3/6/2008 can be downloaded here:

Microsoft Exchange Server MAPI Client and Collaboration Data Objects 1.2.1
http://www.microsoft.com/downloads/details.aspx?FamilyID=E17E7F31-079A-43A9-BFF2-0A110307611E&isplaylang=en&displaylang=en

(Note - The previous version of MAPI/CDO needs to be uninstalled before the new version is installed. A reboot of the Exchange Server is also required)

Verify the latest version of Management Tools is installed on the Backup Server.

(Note - If the Exchange Server is upgraded to Exchange SP1 and the Backup Server's Management Tools is not upgraded with it, this can cause issues on Exchange 2007 backups)
How to verify the what version of Management Tools is installed on the Backup Server:
Step 1. Click Start>Programs>Exchange>Exchange Management Shell
Step 2. Type the following command: Get -ExchangeServer
Step 3. Click Enter (This will show the number of Exchange Server's in the Environment)
Step 4. Under "AdminDisplayVersion" you will find the version for Management Tools.

If version 8.0 (SP1 for Management Tools is NOT installed)
If version 8.1 (SP1 for Management Tools is installed)

Below is an example:

In some extreme cases it may be necessary to debug Exchange VSS Writers at a higher level when VSS Writer failures cannot be determined or proven from the VSSADMIN LIST WRITERS command.

How to set the Exchange Server to 'Expert' level debugging to report in Windows Event Viewer.
Step 1. Before running the backup ensure the event logs for System and Application are saved and cleared.
Step 2. Click Start>Programs>Exchange>Exchange Management Shell
Step 3. Type the following command: cd \ and click Enter
Step 4. Type the following command:

Set-EventLogLevel "MSExchange Repl\Exchange VSS Writer" -level expert

Step 5. To verify run the command was successful type the following command:

Get-EventLogLevel "MSExchange Repl\Exchange VSS Writer"

(This will indicate the entry was successful if the 'expert' EventLevel is referenced)

Warning: Incorrect use of the Windows registry editor may prevent the operating system from functioning properly. Great care should be taken when making changes to a Windows registry. Registry modifications should only be carried-out by persons experienced in the use of the registry editor application. It is recommended that a complete backup of the registry and workstation be made prior to making any registry changes.

Step 6. Open the Registry on the Exchange Server and goto the following location:

HKLM>SYSTEM>CurrentControlSet>Services>MSExchangeIS>Diagnostics\9002 System

(Any time a registry change is expected, please backup the registry before moving forward)

Step 7. Set the following Registry Keys from default 0 to 7

- Exchange Writer

- Backup Restore

Step 8. Close the Regisrty and re-run the Exchange backup as before.

(Note - You can change the level of Event logging by setting the registry keys back to 0)

(Note - To change the Debugging level back to normal away from expert then run the command from the shell:)

Set-EventLogLevel "MSExchange Repl\Exchange VSS Writer" -level normal

*The above information should be able to determine or resolve VSS issues causing Exchange 2007 backups to fail. Should VSS errors report after performing the above mentioned items then Microsoft Support should be contacted.

Scenario where Exchange 2007 backups are failing with an "Unexpected Provider error" if the Exchange Database files are located on a NAS device. This will give following errors:

Error in the job log: AOFO: Initialization failure on: "\\\Microsoft Information Store\Advanced Open File Option used: Microsoft Volume Shadow Copy Service (VSS). VSS Snapshot error. The Microsoft Volume Shadow Copy Service (VSS) snapshot provider selected returned: Unexpected provider error".

Error from Application log on the Exchange server:

Event Type: Error
Event Source: VSS
Event Category: None
Event ID: 12293
Computer: EXCHANGE
Description:
Volume Shadow Copy Service error: Error calling a routine on the Shadow Copy
Provider {ddd3d232-a96f-4ac5-8f7b-250fd91fd102}. Routine details
CommitSnapshots({13849a84-74d3- 4c3e-8aff-09e9d7fc7786}) [hr = 0xc0040307].

Event Type: Error
Event Source: Navssprv
Event Category: None
Event ID: 4354
Computer: EXCHANGE
Description:
Data ONTAP VSS hardware provider failed in CommitSnapshots method
[SnapshotSetId={13849a84-74d3-4c3e-8aff-09e9d7fc7786}]. SnapDrive failed to create
snapshot.
Please check SnapDrive event logs.

Event Type: Warning
Event Source: SnapDrive
Event Category: Snapshot event
Event ID: 219

Computer: EXCHANGE
Description:
Failed to create a consistent snapshot({816328d6-f624-4f9a-8e1d-4341761d9b5f}) of
one or more virtual disk(s).
Please review event log for more information.

Event Id 4356:
Data ONTAP VSS hardware provider's AbortSnapshot method is called
[SnapshotSetId={787750f8-4a02-4755-ab6b-8c2932daa4e8}]. VSS has failed
the current shadow copy session because the previous errors are not.

This error is because Exchange Database files are on NAS device which has its own Snapshot Provider and the Hardware Snapshot Provider is picked for snapping the Exchange data in place of Microsoft Software Shadow Copy Provider.

For example a NETAPP device will have its own ONTAP VSS hardware provider.

To resolve such issues:

Run the VSSADMIN LIST PROVIDERS command on the Exchange server to get a list of installed snapshot provider. If the server has any other provider than Microsoft Software Shadow Copy Provider then the result will be like below (in e.g. below the server also shows NetApp ONTAP VSS hardware provider):

Provider name: 'Microsoft Software Shadow Copy provider 1.0'
Provider type: System
Provider Id: {b5946137-7b9f-4925-af80-51abd60b20d5}
Version: 1.0.0.7

Provider name: 'Data ONTAP VSS Hardware Provider'
Provider type: Hardware
Provider Id: {ddd3d232-a96f-4ac5-8f7b-250fd91fd102}
Version: 5. 0. 0. 1886

In Backup Exec, the Exchange 2007 Information Store backup are done using the Snapshot technology. By default the Microsoft Software Shadow Copy Provider is used to perform the snapshot backups. However if the Exchange server has more than one provider registered, the Volume Shadow Copy Service tries to start the providers in the following order:

Hardware provider
Software provider
System provider

If the Hardware provider has not been configured correctly or if the Hardware Provider is not supported by Backup Exec, the backup job may fail with the above mentioned errors.

To resolve such issues, edit the "Advanced Open File Option" section in the backup job properties for the Exchange Backup job to use the Microsoft Shadow Copy Provider explicitly. (Figure 1). This will force the job to use Microsoft Software Shadow Copy Provider.

Figure1.