Video Screencast Help

Exchange DAG backup is getting failed with error 130

Created: 17 Jun 2012 | 19 comments
Saqib Alam's picture

Exchange DAG backup is getting failed with the following error. even i followed the Technote but still getting same.

http://www.symantec.com/business/support/index?pag...

6/17/2012 1:26:59 PM - Info nbjm(pid=7228) starting backup job (jobid=82) for client EX-ITU01-001, policy EXch_MB_TEST, schedule GRT_FULL  
6/17/2012 1:26:59 PM - Info nbjm(pid=7228) requesting STANDARD_RESOURCE resources from RB for backup job (jobid=82, request id:{06349E60-9B1A-4959-B5DD-46D7CE8F6BAF})  
6/17/2012 1:26:59 PM - requesting resource ExchangeGRT
6/17/2012 1:26:59 PM - requesting resource bs-itu01-001.kacst.ksa.NBU_CLIENT.MAXJOBS.EX-ITU01-001
6/17/2012 1:26:59 PM - requesting resource bs-itu01-001.kacst.ksa.NBU_POLICY.MAXJOBS.EXch_MB_TEST
6/17/2012 1:26:59 PM - granted resource bs-itu01-001.kacst.ksa.NBU_CLIENT.MAXJOBS.EX-ITU01-001
6/17/2012 1:26:59 PM - granted resource bs-itu01-001.kacst.ksa.NBU_POLICY.MAXJOBS.EXch_MB_TEST
6/17/2012 1:26:59 PM - granted resource MediaID=@aaaab;Path=F:\ExchangeGRT;MediaServer=bs-itu01-002.kacst.ksa
6/17/2012 1:26:59 PM - granted resource ExchangeGRT
6/17/2012 1:26:59 PM - estimated 0 Kbytes needed
6/17/2012 1:26:59 PM - begin Parent Job
6/17/2012 1:26:59 PM - begin Exchange 14 Snapshot, Step By Condition
Status 0
6/17/2012 1:26:59 PM - end Exchange 14 Snapshot, Step By Condition; elapsed time: 00:00:00
6/17/2012 1:26:59 PM - begin Exchange 14 Snapshot, Read File List
Status 0
6/17/2012 1:26:59 PM - end Exchange 14 Snapshot, Read File List; elapsed time: 00:00:00
6/17/2012 1:26:59 PM - begin Exchange 14 Snapshot, Create Snapshot
6/17/2012 1:26:59 PM - started
6/17/2012 1:27:00 PM - started process bpbrm (6952)
6/17/2012 1:27:07 PM - Info bpbrm(pid=6952) EX-ITU01-001 is the host to backup data from     
6/17/2012 1:27:07 PM - Info bpbrm(pid=6952) reading file list from client        
6/17/2012 1:27:07 PM - Info bpbrm(pid=6952) start bpfis on client         
6/17/2012 1:27:07 PM - begin Create Snapshot
6/17/2012 1:27:09 PM - Info bpfis(pid=36580) Backup started           
6/17/2012 1:28:20 PM - Critical bpbrm(pid=6952) from client EX-ITU01-001: FTL - snapshot creation failed - VSS volume snap failed., status 130
6/17/2012 1:28:20 PM - Info bpbrm(pid=6952) DB_BACKUP_STATUS is 156          
6/17/2012 1:28:20 PM - Warning bpbrm(pid=6952) from client EX-ITU01-001: WRN - NEW_STREAM0 is not frozen    
6/17/2012 1:28:20 PM - Warning bpbrm(pid=6952) from client EX-ITU01-001: WRN - Microsoft Information Store:\DB-MGR-01 is not frozen  
6/17/2012 1:28:20 PM - end writing
Status 130
6/17/2012 1:28:20 PM - end Create Snapshot; elapsed time: 00:01:13
6/17/2012 1:28:20 PM - begin Exchange 14 Snapshot, Stop On Error
Status 0
6/17/2012 1:28:20 PM - end Exchange 14 Snapshot, Stop On Error; elapsed time: 00:00:00
6/17/2012 1:28:20 PM - begin Exchange 14 Snapshot, Delete Snapshot
6/17/2012 1:28:21 PM - started process bpbrm (7036)
6/17/2012 1:28:27 PM - begin Delete Snapshot
6/17/2012 1:28:27 PM - Info bpfis(pid=0) Snapshot will not be deleted        
6/17/2012 1:28:30 PM - Info bpfis(pid=15344) Backup started           
6/17/2012 1:28:30 PM - Critical bpbrm(pid=7036) from client EX-ITU01-001: cannot open C:\Program Files\Veritas\NetBackup\online_util\fi_cntl\bpfis.fim.KACST-DAG001_1339928819.1.0      
6/17/2012 1:28:31 PM - end writing
Status 1542
6/17/2012 1:28:31 PM - end Delete Snapshot; elapsed time: 00:00:04
Status 130
6/17/2012 1:28:31 PM - end Exchange 14 Snapshot, Delete Snapshot; elapsed time: 00:00:11
system error occurred(130)

=======================================

VSSadmin Output from Mailbox.

vssadminlist providers

vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001-2005 Microsoft Corp.

Provider name: 'Microsoft Software Shadow Copy provider 1.0'
   Provider type: System
   Provider Id: {b5946137-7b9f-4925-af80-51abd60b20d5}
   Version: 1.0.0.7

Provider name: 'Data ONTAP VSS Hardware Provider'
   Provider type: Hardware
   Provider Id: {ddd3d232-a96f-4ac5-8f7b-250fd91fd102}
   Version: 6. 2. 1. 5029

 

========================================

vssadmis list writers

vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001-2005 Microsoft Corp.

Waiting for responses.
These may be delayed if a shadow copy is being prepared.

Writer name: 'Task Scheduler Writer'
   Writer Id: {d61d61c8-d73a-4eee-8cdd-f6f9786b7124}
   Writer Instance Id: {1bddd48e-5052-49db-9b07-b96f96727e6b}
   State: [1] Stable
   Last error: No error

Writer name: 'VSS Metadata Store Writer'
   Writer Id: {75dfb225-e2e4-4d39-9ac9-ffaff65ddf06}
   Writer Instance Id: {088e7a7d-09a8-4cc6-a609-ad90e75ddc93}
   State: [1] Stable
   Last error: No error

Writer name: 'Performance Counters Writer'
   Writer Id: {0bada1de-01a9-4625-8278-69e735f39dd2}
   Writer Instance Id: {f0086dda-9efc-47c5-8eb6-a944c3d09381}
   State: [1] Stable
   Last error: No error

Writer name: 'ASR Writer'
   Writer Id: {be000cbe-11fe-4426-9c58-531aa6355fc4}
   Writer Instance Id: {48980e05-ca1f-4836-a3f6-e136b76bd04b}
   State: [1] Stable
   Last error: No error

Writer name: 'Microsoft Exchange Replica Writer'
   Writer Id: {76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}
   Writer Instance Id: {0556c222-9ad1-4161-92f5-70c49844087b}
   State: [9] Failed
   Last error: Timed out

Writer name: 'System Writer'
   Writer Id: {e8132975-6f93-4464-a53e-1050253ae220}
   Writer Instance Id: {f3f3c65a-2be6-4197-aa44-c46cca6d6da5}
   State: [1] Stable
   Last error: No error

Writer name: 'IIS Metabase Writer'
   Writer Id: {59b1f0cf-90ef-465f-9609-6ca8b2938366}
   Writer Instance Id: {32ccda47-a147-4197-8019-c0e611ffc963}
   State: [1] Stable
   Last error: No error

Writer name: 'Microsoft Exchange Writer'
   Writer Id: {76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}
   Writer Instance Id: {1a020f73-5661-4a6e-b338-83f80ab8572a}
   State: [1] Stable
   Last error: No error

Writer name: 'Shadow Copy Optimization Writer'
   Writer Id: {4dc3bdd4-ab48-4d07-adb0-3bee2926fd7f}
   Writer Instance Id: {ce4f4439-7348-4d2d-8220-e15a0b2a36ce}
   State: [1] Stable
   Last error: No error

Writer name: 'Registry Writer'
   Writer Id: {afbab4a2-367d-4d15-a586-71dbb18f8485}
   Writer Instance Id: {dcaa6a35-bfcd-4ca2-9c7f-51eff7141f73}
   State: [1] Stable
   Last error: No error

Writer name: 'COM+ REGDB Writer'
   Writer Id: {542da469-d3e1-473c-9f4f-7847f01fc64f}
   Writer Instance Id: {ae618f76-cc8e-4d74-b66b-722b3a2aba15}
   State: [1] Stable
   Last error: No error

Writer name: 'IIS Config Writer'
   Writer Id: {2a40fd15-dfca-4aa8-a654-1f8c654603f6}
   Writer Instance Id: {55834471-acbd-4e40-bf43-8a38eeeda0f5}
   State: [1] Stable
   Last error: No error

Writer name: 'WMI Writer'
   Writer Id: {a6ad56c2-b509-4e6c-bb19-49d8f43532f0}
   Writer Instance Id: {1bbffbb1-6b60-43e0-9589-36ad4cdb557a}
   State: [1] Stable
   Last error: No error

Writer name: 'Cluster Database'
   Writer Id: {41e12264-35d8-479b-8e5c-9b23d1dad37e}
   Writer Instance Id: {c0151ad0-6dd1-4e85-9fb5-4604007e9f33}
   State: [1] Stable
   Last error: No error
 

Can any one provide any solution for this.

Enviornment
OS: Server 2008R2
Exchanage 2010
NBU Version: 7.5.0.1
 

 

 

Comments 19 CommentsJump to latest comment

captain jack sparrow's picture

is this full backup or diff/incr.. if diff/incr  check if circular logging is not enabled...

 

share more info about   exchange domain (how many mbx/cas/ht servers.. ) etc.

 Cheers !!!

CJS

 

Marianne's picture

I agree with speedfreak - we need more information.

Has this backup worked previously? Or new setup that has never worked before?

Are you seeing the same error in Event Viewer as per TECH181190?

How is policy configured? To backup Active or passive node? or Off-host backup?

If so, is Exchange management console (EMC) installed on all cluster members? Or off-host server/client?

 

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

Saqib Alam's picture

Same configuration which we have previously only change we made is from OLD Netbackup Master Server 7.0 on Server 2003 we migrate on 7.5.0.1 on Server 2008R2

Backup is full right now i am taking backup on one database not complete DAG with GRT and circular logging enabled.
No active and passive node is defined in policy no Off-host backup , snapshot is enabled.

 

captain jack sparrow's picture

Writer name: 'Microsoft Exchange Replica Writer'
   Writer Id: {76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}
   Writer Instance Id: {0556c222-9ad1-4161-92f5-70c49844087b}
   State: [9] Failed
   Last error: Timed out

 

Could you check why exchange replica writers are in failed state...

Check below links if it helps

http://blogs.technet.com/b/ehlro/archive/2010/02/1...

http://technet.microsoft.com/en-us/library/ee69229...

 

 Cheers !!!

CJS

 

Saqib Alam's picture

Now its in stable state as i reboot both my mailboxes servers.

captain jack sparrow's picture

Saqib

 

Just sharing my personal experience with various cases especially with Exchange.

Symantec recommends to have service account configured with necessary privileges for GRT based backups, however it is also required for non GRT backup.

Verify if below permissions are given to account u use

Privileged Account for EWS Access (Exchange 2010)

Account must be member of

  • Domain Admin
  • Domain Users
  • Organization Management

This account

  • Must be Unique in Forest (Initial 4 Chars) e.g. backupadmin etc.
  • Must have an Active Mailbox
  • Must not be Hidden from GAL

 

For Restoration, We need to Create Impersonation Rule on Exchange

Open EMS with Administrative Privileges

 

New-ManagementRole -Name EWSImpersonationRole -Parent ApplicationImpersonation

New-ManagementRoleAssignment -Role EWSImpersonationRole -User backupadmin

EWSImpersonationRoleAssignment

backupadmin refers to account you use

 

Replace Administrator with user created with above membership

Configure NetBackup Client and Legacy Services with Above Account

Specify this account under NetBackup Client host properties – Client Setting-Exchange

  • From Local Security Policy of server , In the User Rights Assignment, add the account to the following policies:
    • Allow log on locally
    • Log on as a service
    • Replace a process level token

Apply gpupdate /force after above policy is done

 

Also Within Policy attributes, Select Either Active or Passive copy to be backed up.

In Preferred servers  specify MBX servers

Note: Create a dummy policy (inactive) which includes your Hub , CAS and MBX Servers

Also make sure all your exchange server has same netbackup patch level (i.e. 7.5 or 7.1) same as Master/Media server

 

As you have took reboot of server, verify that writers are in stable state across Mailbox Servers. and Prior to running backup just validate above info. and Let us know backup output

 Cheers !!!

CJS

 

Nicolai's picture

Is the volumes where Exchnage located on Netapp Storeage ?

 

Provider name: 'Data ONTAP VSS Hardware Provider'

   Provider type: Hardware
   Provider Id: {ddd3d232-a96f-4ac5-8f7b-250fd91fd102}
   Version: 6. 2. 1. 5029

 

Assumption is the mother of all mess ups.

If this post answered your'e qustion -  Please mark as a soloution.

Saqib Alam's picture

Yes we have NetApp Storage we are using mount points not drive letters as we hve around 40 databases.

Saqib Alam's picture

It was working fine previously without having any issue the only change is hardware , OS and Netbackup Master Server Change.

Previous Envrioment :
OS: Server 2003x64 R2
NetBackup: 7.0.1

New
OS : Server 2008 x64 R2
NBU : 7.5.0.1

backupadmin have all the previllages.

 

captain jack sparrow's picture

hi Saqib

 

Is policy configured as stated in previous post.

Just to ensure, does backupadmin is able to browse Exchange Resources via BAR gui on MBX server.

this would confirm backupadmin has all the rights required.

Is Exchange virtualized?

 Cheers !!!

CJS

 

Saqib Alam's picture

 

Yes policy is configured as same as it was on my old master server. backup Admins have domain admin rights , member of exchange organization group. and is able to browse exhcange databases from Master & Media Server both.

Find the screen shot attached.

 

AttachmentSize
NBU_EXH_Policy.pdf 353.78 KB
captain jack sparrow's picture

Seems you did not went through Policy confirmation in my post. Seeing at snapshot you send, there is no Database Source defined,. (wonder if preferred servers are also updated)

Kindly enable either Active or Passive copy as Database Source, Under Enabled Granular Recovery Attribute

 

Also Browse Information Store via Mailbox Server (Login with backupadmin account) and Start Backup, Archive and Restore GUI (Run as Administrator)

Click on Start Backup  and Check if you are able to Expand the DAG till your Database

 

 

 Cheers !!!

CJS

 

Saqib Alam's picture

find the attached screen shot of BAR and bpfis logs from mailbox.

BAR.jpg
AttachmentSize
bpfis.zip 28.57 KB
ManikandanPC's picture

Check TCP chimney settings in master, media and exchange servers. Mostly this occurs due to network issues.

Marianne's picture

TCP Chimney disable applies to W2003.

Symantec recommends to leave it enabled for W2008.

See http://www.symantec.com/docs/TECH145791

Technotes, such as 294308  and 29008 it was recommend to disable TCP Chimney Offload. For Windows 2008 R2 platforms, performance gains have been realized when TCP Chimney Offload is enabled.

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

Will Restore's picture

 the only change is hardware , OS and Netbackup Master Server Change.

 

in other words almost everything changed

change is good when it's rattling around in your pocket, otherwise surprise

Will Restore -- where there is a Will there is a way

Saqib Alam's picture

Yes its resolved now it was VSS issue i change it System and now its working fine as by default Netbackup give precisendse to hardware VSS so hardware VSS was not working fine.

captain jack sparrow's picture

hi saqib mark the post as solved which helped you most

 Cheers !!!

CJS