How to troubleshoot Exchange VSS writer failures such as v-79-57344-65233 a failure occurred querying the Writer Status

Article:TECH209938  |  Created: 2013-08-26  |  Updated: 2014-05-29  |  Article URL http://www.symantec.com/docs/TECH209938
Article Type
Technical Solution

Product(s)

Subject

Issue



An Exchange VSS initialization failure causes an Exchange Agent backup or restore job to fail.

 


Error



v-79-57344-65233

0xe00fed1 - A failure occurred querying the Writer status.

 


Cause



As the error indicates, this is a potential problem with the writer at the time of the job running. Since the writer status could have changed since the job ran, it is important to note down the complete error from the Backup Exec Job log to get more information on the writer status at the time of failure.
 

Solution



To troubleshoot Exchange VSS writer errors, the following steps should be performed:
 
 
 
 
 

[ A ] Determine the current writer status:

 
To determine the writer status at the time of trouble-shooting, running the command VSSADMIN LIST WRITERS from a command prompt on the Exchange server. The status may show the Exchange Writer is in a "Retryable State" as shown:
  
 Writer name: 'System Writer'
 Writer Id: {e8132975-6f93-4464-a53e-1050253ae220}
 Writer Instance Id: {85fd6d21-d48e-4827-91df-d1bce6c15bed}
 State: [7] Failed
 Last error: No error
 Writer name: 'Microsoft Exchange Writer'
 Writer Id: {76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}
 Writer Instance Id: {11dcc128-2b8c-45ab-ae8b-fdc94fabc3ff}
 State: [1] Stable
 Last error: Retryable error
 
 The Exchange Writer could also show a "Failed State"
 
Writer name: 'Microsoft Exchange Writer'
Writer Id: {76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}
Writer Instance Id: {235de865-857d-4804-b87f-73f019f26d1d}
State: [7] Failed
Last error: No error
 
 
 

 

[ B ] Check Windows Event Logs on the failing Exchange server:

 
If the Writers have been found to be either of these states, review the Exchange Application (App) Event log and the Exchange System (Sys) Event log for errors from Event Source "MSExchangeIS", or "Volsnap" or "VSS" as listed with their resolutions below.
 
Click on one of the following to see the desired solution:
 
 
Event ID 2112, 2024, 8229 Event ID 9611, 459, 2007, 2002 Event ID 8230  Event ID 25
Event ID 12293 Event ID 9840, 9607, 9609, 9814, 2007 The Writer times out if there is another snapshot process in progress SBS 2011 Incremental backups
 
 
 
 
 

Event ID 2112, 2024, 8229

 

  • Source: MSExchangeRepl
  • Event id: 2112
  • Description: The Microsoft Exchange Replication service VSS Writer instance e62275a0-a66f-44af-ab12-474252089d29 failed with error code 80070020 when preparing for a backup of database 'XXX' 
  • Event id - 2024 
  • Source – MSExchangeRepl
  • Description: The Microsoft Exchange Replication service VSS Writer (Instance e62275a0-a66f-44af-ab12-474252089d29) failed with error 80070020 when preparing for a backup
  • Event id: 8229     
  • Source: VSS
  • Description: A VSS writer has rejected an event with error 0x800423f3, The writer experienced a transient error. If the backup process is retried, the error may not reoccur. Changes that the writer made to the writer components while handling the event will not be available to the requester. Check the event log for related events from the application hosting the VSS writer.
Cause
 
This error occurs during an Exchange 2010 DAG backup of a passive node.
 
Solution
 
Restart the Microsoft Exchange Replication service.
Re-try the backup job.
 
 
 
 

Event ID 9611, 459, 2007, 2002

Source: MSExchangeIS
Type: Error
Category: Exchange VSS Writer
Event ID: 9611
Description:
Error code 0x8004010f when freezing the Storage groups.

http://support.microsoft.com/kb/838864


Event Type: Error
Event Source: ESE
Event Category: Logging/Recovery
Event ID: 459
Computer: ComputerName
Description: Information Store (4920) storage group : The file drive:\logfilefolder\E0xxxxx.log is missing and could not be backed-up.

http://support.microsoft.com/kb/935634

Event Type: Error
Event Source: ESE
Event Category: ShadowCopy
Event ID: 2007
Computer: ComputerName
Description:
Information Store (244) Shadow copy instance 2 aborted.
 
Event Type: Error
Event Source: ESE
Event Category: ShadowCopy
Event ID: 2002
Computer: ComputerName
Description: Information Store (5992) Shadow copy instance number encountered error -2403 (in our case -569) on freeze.

http://support.microsoft.com/kb/933779

 
Cause
 
There are inconsistencies between the transaction log sequence for the Exchange database.

Solution
 
The transaction log sequence will need to be repaired, and potentially the Exchange database depending on the extent of the damage.  Below are several Microsoft articles with further recommendations.  For more information please contact Microsoft support.
 
 
Event ID 8230
Log Name:      Application
Source:        VSS
Event ID:      8230
Task Category: None
Level:         Warning
Description:
Volume Shadow Copy Service error: Failed resolving account spsearch with status 1376. Check connection to domain controller and VssAccessControl registry key.
Operation:
   Executing Asynchronous Operation
Context:
   Current State: DoSnapshotSet
Error-specific details:
   Error: NetLocalGroupGetMemebers(spsearch), 0x80070560, The specified local group does not exist.- From the bolg "http://social.msdn.microsoft.com/Forums/en-US/sharepoint2010general/thread/86a02a16-87a4-490d-bb3e-dc1491623a9f/"
Cause
 
In order for any writer to use the VSS infrastructure, the writer must run under an account that is a member of the local Administrators or Backup Operators group on the local computer. If this condition doesn't meet than the warnings would be displayed and the writers would fail.
 
Solution
 
Add the backup account to the following registry key:
 
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\VSS\VssAccessControl
 
Note that it is also possible that snapshots created outside of Backup Exec exist on the volume where the Exchange database or logs reside. It order to verify that no snapshots exist, follow these steps:
  1.  Open a command prompt on the Exchange server and type the command "Vssadmin list Shadows"
  2.  If a snapshot exists, it will be listed. Before attempting to delete the snapshot make a note of the time at which it was taken. This will help to determine if the orphaned snapshot was created during or outside a backup. If the snapshot was created outside a backup, check for scripts or application that could have created it (for example some applications take snapshots by creating Scheduled Windows tasks). 
 Make a note of the ShadowId and use the below command to delete it:
 Vssadmin delete shadows /Shadow=<ShadowId>
 
 
 
 
Event ID 25
Event ID: 25
Source: VolSnap
Description: The shadow copies of volume <drive lettter> were aborted because the diff area file could not grow in time.
Consider reducing the IO load on this system to avoid this problem in the future.
Cause
 
The errors may occur for one of the following reasons:
  • Volume Shadow Copy service writer time-out
  • Shadow copy deletion
  • Large audit log
  • Volsnap.sys driver has encountered excessive I/O activity and as a result the "diff" area cannot grow
  • Computer has run out of shadow copy storage space

 Solution

1.  Run a CHKDSK and defragment the volume named in Event ID 25
2.  Download and install the latest VSS patches from Microsoft
3. Contact Microsoft for further assistance with this error.
 
 
 
 
Event ID 12293
Log Name:   Application
Source:    VSS
Event ID:   12293
Description:
Volume Shadow Copy Service error: Error calling a routine on a Shadow Copy Provider b5946137-7b9f-4925-af80-51abd60b20d5. Routine details EndPrepareSnapshots(67378ac7-29ab-44d4-b577-0cc1907b0215) [hr = 0x8000ffff]. 
Cause
 
This can be caused by Microsoft VSS not being registered properly with the system, or the Shadow Copy associations for the volume are corrupted or misconfigured.
 
Solution
  1. Right-click any local disk.
  2. Select Configure Shadow Copies.
  3. Highlight Volume with Volume name reporting the Exception in the Backup Job Log
  4. Click on Settings.
  5. Set the Maximum Size to No Limit. Note: If No Limit is already selected click Use Limit and accept the default size. Click OK then OK. Then go back in and set the Maximum Size to No Limit and click OK.
  6. Attempt the Backup Job
 
 
 
Event ID 9840, 9607, 9609, 9814, 2007
Log Name:      Application
Source:        MSExchangeIS
Event ID:      9840
Task Category: General
Level:         Error
Description:
An attempt to prepare the storage group 'Support Storage Group' for backup failed because the storage group is already in the process of being backed up. The error code is -2403. (Note that if a backup was recently aborted, then it may take several minutes for the system to detect the aborted backup and initiate backup cleanup procedures, so this message may be generated if an attempt was made to backup a storage group before a previous backup attempt had fully terminated.)  
Event Type: Error
Event Source: MSExchangeIS
Event Category: Exchange VSS Writer
Event ID: 9607
Description: Error code 0x50d when preparing for backup. 
 
Log Name:      Application
Source:        MSExchangeIS
Event ID:      9814
Task Category: Exchange VSS Writer
Level:         Error
Description:
Exchange VSS Writer (instance 86453ccb-1181-45fe-9e7c-072b7494e440:8) failed with error code -2403 when preparing the database engine for backup of database 'Mailbox Database' (activation preference number).
 
Log Name:      Application
Source:        ESE
Event ID:      2007
Task Category: ShadowCopy
Level:         Error
Description: nformation Store (2064) Shadow copy instance 8 aborted.
 
Log Name:      Application
Source:        MSExchangeIS
Event ID:      9609
Task Category: Exchange VSS Writer
Level:         Error
Description:
Exchange VSS Writer (instance 2f5bbb68-ba32-42e1-8e03-e78ada2acbed:16) failed with error code -2403 when preparing for Snapshot
 
Method 1:
 
Dismount all databases in the storage group
Dismount all databases in the storage group
Use this method if the event ID 9840 that was logged on the Exchange 2007 server contained error code 2403.

To dismount all databases in the storage group, follow these steps:
  • Click Start, point to All Programs, point to Exchange Server 2007, and then click Exchange Management Console.
  • Expand Server Configuration, and then expand the storage group that is listed in the event ID 9840 message that was logged on the Exchange 2007 server.
  • Right-click each database in that storage group, and then click Dismount Database.
  • Click Yes to confirm the operation.
  • After the dismount the operation is complete. Right-click the database again, and then click Mount Database.
  • Try to perform the backup operation that resulted in event ID 9840.
  • If this method does not resolve the problem, try Method 2.
Method 2:
 
Restart the Microsoft Exchange Information Store service\ Use this method if the event ID 9840 that was logged on the Exchange 2007 server contained error code 1293. Use this method if Event ID 9607 was logged on Exchange 2003. Additionally, use this method if method 1 did not resolve the problem.

To restart the Microsoft Exchange Information Store service, follow these steps:
  • Click Start, point to All Programs, point to Administrative Tools, and then click Services.
  • Right-click Microsoft Exchange Information Store, and then click Restart.
  • Try to perform the backup operation that resulted in Event ID 9840.
    Reference : http://support.microsoft.com/?kbid=930800
     
Method 3:

If the dismount of the databases fails with error:
Couldn't dismount the database that you specified. Specified database: XXXXXXXXX; Error code: An Active Manager operation failed with a transient error. Please retry the operation. Error: Database action failed with transient error. Error: A transient error occurred during a database operation. Error: MapiExceptionBackupInProgress: Unable to unmount database. (hr=0x80004005, ec=1293)
 
Try to restart the Microsoft Exchange Information Store service, if it fails or stuck in stopping state/mode, reboot the Exchange server.
 
 
 
 
The writer times out if there is another snapshot process completing or in progress:
 
If running the command "VSSADMIN LIST WRITERS" from a command prompt, it might show Exchange writer status in Failed State 9.
 
Solution
 
Verify if a “third party” application is using VSS – this could be the Volume Shadow Copy running snapshots of the system drives. If so, this will need to be disabled.
Note: Disabling the shadow copies for the drive will delete all existing Shadow Copies.
 
 
 
 
SBS 2011 Incremental backups
 
Similar errors will occur in the job log:
 
Writer Name: Exchange Server, Writer ID: {76FE1AC4-15F7-4BCD-987E-8E1ACB462FB7}, Last error: The VSS Writer failed, the process can be repeated (0x800423f3), State: error during the backup preparation (7).

Writer Name: Exchange Server, Writer ID: {76FE1AC4-15F7-4BCD-987E-8E1ACB462FB7}, Last error: The VSS Writer failed, the process can be repeated (0x800423f3), State: error during the blocking process (9).
 
Cause:
This error is caused by Circular Logging being enabled on the Exchange server. A differential or Incremental backup cannot be done if circular logging is enabled.
 
Solution
To perform Incremental or differential type of backups, the logging method on Exchange needs to be changed by disabling the Circular Logging. Refer to www.symantec.com/docs/TECH11310
 
 
 
 

[ C ] Run the BETest Tool on the Exchange server to verify the functionality of the Exchange writer.

 
If Exchange writer issues are still present after having exhausted all of the suggested trouble-shooting, Microsoft's BETest Tool or Windows Server Backup (Windows 2008 only) may be utilized to verify the functionality of the Exchange Writers.
 
BETest Tool
  
If Service Pack 2 for Exchange 2007 is installed on a Windows 2008 Server, Windows Server Backup can backup Exchange 2007 using VSS writers.  Please review the following article from Microsoft on how to perform a backup of Exchange 2007 with Windows Server Backup.
 
 
Below is further instruction of how to operate the BETest Tool:
 

A. Download and install BETest Tool. This can be obtained by downloading and installing the Volume Shadow Copy Service SDK 7.2 from:
 

The default location for the i386 version of BETest is :
 
C:\Program Files (x86)\Microsoft\VSSSDK72\TestApps\BETest\obj\i386
 
Note: There are also AMD64 and IA64 compiled versions available.   Always change to the directory before running BETest.
 
 
B. Once the tool is installed, verify which writers are available, by running the following command:
 
Betest.exe > c:\writersAvailable.txt
 
 

C. Open the WritersAvailable file using Notepad.
 

Go to “Menu | Find” and type: Exchange Writer
 
Look for:
 
<writer-id>  
 
<component-logical-path>
 
Copy this information found under the Exchange Writer to create a Components.txt file, which is required by BETest Tool
 
<writer-id> is in standard GUID format
 
Ex: 76fe1ac4-15f7-4bcd-987e-8e1acb462fb7
 
<component-logical-path>   is either logical-path, logical-path\component-name or component-name-only (if there is no logical path)
 
Ex: Microsoft Exchange Server\Microsoft Information Store\SOBE2K3\7c82b8db-d03c-40ef-be0f-15cdc1f21e27
 
Example of how to add this information to the Components.txt file:
 
"{76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}":"Microsoft Exchange Server\Microsoft Information Store\SOBE2K3\7c82b8db-d03c-40ef-be0f-15cdc1f21e27";
 
This sample contains two GUIDS, the first one is the Exchange Writer, and must not be modified. The second GUID is for a specific storage group
 
Even if there is more than one storage group, select to run it against only one at a time, so the test will run faster. In this example, the tool will run the backup of only one Storage Group.
 
 

D. Is the server a cluster?
 

If this is a cluster server, the path will look similar to the following:
 

"Microsoft Exchange Server\Microsoft Information Store\Replica\EVS_Name\5df67a32-5f44-4585-ad0e-962b70f399d3","Microsoft Exchange Server\Microsoft Information Store\Replica\EVS_Name\35e64d4a-7c6b-41f8-a720-068d2798b908","Microsoft Exchange Server\Microsoft Information Store\Replica\EVS_Name\5afe57ab-c14d-4bf9-8a69-78691fad5a33";
 
The Component.txt will look like this for clusters:
 
"{76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}": "Microsoft Exchange Server\Microsoft Information Store\Replica\EVS_Name\5df67a32-5f44-4585-ad0e-962b70f399d3","Microsoft Exchange Server\Microsoft Information Store\Replica\EVS_Name\35e64d4a-7c6b-41f8-a720-068d2798b908","Microsoft Exchange Server\Microsoft Information Store\Replica\EVS_Name\5afe57ab-c14d-4bf9-8a69-78691fad5a33";
 

E. After the file Components.txt is created, run the backup using BETest by running the following command:
 

 
BETEST.exe /B /E /T 1 /S output.XML /C components.txt /D c:\betest > Output.txt
 
In this example, the backup will be created under the C:\Betest directory.
 
If this backup errors out, that may mean the Exchange writer is having a problem. To further trouble-shoot the issue, the Output.txt file will need to be sent to the Symantec Support for review.
 
After the above, run the following command without the /E switch.
 
BETEST.exe /B /T 1 /S output.XML /C components.txt /D c:\betest > Output2.txt
 
This file will need to be sent to Symantec Support as well.

F. More information about the switches:
 

BETest [/B] [/R] [/E] [/P] [/U/ [/Auth] [/T backup-type] [/S filename] [/C filename] [/D path]
 
/B   Performs backup only
 
/R   Performs restore only. Restore-only must be used with /S for a backup document file
 
/E   Excludes BETest test writer
 
/P   Create a persistent snapshot
 
/URemoteSnapshots  Adds to the set the share rather than the writer path
 
/Auth   perform an authorative restore for AD or ADAM
 
/T Backup type   Specify the backup type (FULL, LOG, COPY, INCREMENTAL, DIFFERENTIAL)
 
/S filename In case of backup, saves the backup document to file specified by Filename. In case of restore-only, loads the backup document from file
 
/C Filename   Selects which components to backup/restore based on the file
 
/D path   In case of backup it saves the files to be backed up to this location. In case of restore, restores the backed up files from this location.
 
/O   Specifies BootableSystemState backup
 

 
For further information on the BETest Tool please refer to the following knowledge base articles from Microsoft:
 
Exchange 2007 Data Backup and Volume Shadow Copy Services
 http://technet.microsoft.com/en-us/library/dd233256.aspx
 
 
If the Exchange server is running Windows 2003 – perform the following: 
 
BETest Tool for Windows 2003 only
 
BETest is a VSS requester that tests advanced backup and restore operations. This tool can be used to test an application's use of complex VSS features such as the following:
 
  • Incremental and differential backup
  • Complex restore options, such as authoritative restore
  • Rollforward options
 
NOTE: BETest is included in the Microsoft Windows Software Development Kit (SDK) for Windows Vista and later. The VSS 7.2 SDK includes a version of BETest that runs only on Windows Server 2003. For information about downloading the Windows SDK and the VSS 7.2 SDK, see Volume Shadow Copy Service  http://msdn.microsoft.com/en-us/library/bb968832(VS.85).aspx .
 
In the Windows SDK installation, the BETest Tool can be found in %ProgramFiles%\Microsoft SDKs\Windows\v6.0\bin\vsstools.
  
Running the BETest Tool
 
To run the BETest Tool from the command line, use the following syntax:
 
BETest command-line-options
 
The following usage example shows how to use the BETest tool together with the VSS Test Writer tool"
BETest Tool Usage Example
  1.  Create a test directory named C:\BETest. Copy the following files into this directory: 
  • Betest.exe
  • Vswriter.exe
  • BetestSample.xml
  • VswriterSample.xml
  1. Create a directory named C:\TestPath. Put some test data files in this directory.
  2. Create a directory named C:\BackupDestination. Leave this directory empty.
  3. Open two elevated command windows and set the working directory in each to C:\BETest.
  4. In the first command window, start the VSS Test Writer tool as follows:
     
vswriter.exe VswriterSample.xml
 
The vswriterSample.xml file configures the VSS Test Writer tool (vswriter) to report the contents of the c:\TestPath directory in preparation for a backup operation. Note that the VSS Test Writer tool will not produce output until it detects activity from a requester such as BETest Tool. To stop the VSS Test Writer tool, press CTRL+C.
  1. In the second command window, use the BETest Tool to perform a backup operation as follows In the first command window, start the VSS Test Writer tool as follows:
 
betest.exe /B /S backup.xml /D C:\BackupDestination /X BetestSample.xml
 
BETest Tool will back up the files from the C:\TestPath directory to the C:\BackupDestination directory. It will save the backup component document to C:\BETest\backup.xml.
 
  1. If the backup operation is successful, delete the contents of the C:\TestPath directory, and use the BETest Tool to perform a restore operation as follows:
     
betest.exe /R /S backup.xml /D C:\BackupDestination /X BetestSample.xml
 
For more information on the BETest Tool please refer to the following knowledge base article from Microsoft: 
 

 


Supplemental Materials

SourceUMI
Valuev-79-57344-65233
Description

A failure occurred querying the Writer status

 



Legacy ID



322617


Article URL http://www.symantec.com/docs/TECH209938


Terms of use for this information are found in Legal Notices