Video Screencast Help
Symantec to Separate Into Two Focused, Industry-Leading Technology Companies. Learn more.

Exchange 2010 archive db backup fail

Created: 29 Mar 2013 | 33 comments

Hello, I'm trying to setup netback to backup exchange archive database. All other backup:sql, vmware and dag services are ok. But we have dedicated exchange server with 2 archive databses and backup of it failed with error: An existing snapshot is no longer valid and cannot be mounted for subsequent operations(1542). I already try bpfis query and it empty: 

INF - BACKUP START 6204

INF - FIS IDs:
INF - EXIT STATUS 0: the requested operation was successfully completed
 
vssadmin list writers before backup attempt is normal, after backup of corse it with waiting state.
Writer name: 'Microsoft Exchange Writer'
   Writer Id: {76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}
   Writer Instance Id: {d3f31f8b-893e-4fa2-9814-200192298952}
   State: [5] Waiting for completion
   Last error: No error
 
 
I already try to reregister vss and reinstall netbackup agent. many times reboot server, but no effect.
 
 
Bpfis log in attach. 
 
 
What should I do? Thanks!

 

Operating Systems:

Comments 33 CommentsJump to latest comment

inn_kam's picture

 

1.Which Netbackup version you are using?

2.Have You configured Netbackup Master and Client properties for exchange server correctly? Please follow step by step as described by Symantec on below videos

 

All versions videos for Exchange backups and restores with NetBackup

http://www.symantec.com/business/support/index?page=content&id=HOWTO41825

3.Is NetBackup Client Service running on the Exchange server?

 
Boolenat's picture

1. 7.5.0.5 

2. Yes, my backups of dag db are ok. 

3. Of course i check it:) 

 

I suppose that problem in vss but i have no idea what to do more. 

 

Thanks for reply!

Boolenat's picture

I have try this diskshadow script http://www.symantec.com/business/support/index?page=content&id=HOWTO60816 on my archive server and it's create shadow copy of my db ok. 

inn_kam's picture

again restart the exchange server and then check, if its Microsoft exchange writers problem

John Kozitzki's picture

I noticed you had a status code 130 in the log...whenever we had this error during snapshot with Exchange, we usually found a hung bpfis or tracker processes on some of the Exchange DAG servers.

Try stopping the Netbackup services on the Exchange DAG servers and, using bpps, see if there are any hung processes that did not stop. If there are, kill them and then restart the Netbackup services.

Boolenat's picture

Thaks for reccomendation, but with my dag servers backup all ok. my exchange archive server with information store have issue. and i already trying restart all netbackup services, also reinstall it. and many times reboot server at all. all procces works normaly. 

Mark_Solutions's picture

Just to clarify please - this is a 2010 DAG and yet your selection is Microsoft Information Store rather than being part of a DAG?

Have you browsed to the files selection to ensure it is selected correctly?

Also you have selected a snapshot backup type - so two questions here, have you tried it without it being a snapshot type (as your selection is an Information Store and not a DAG) and what is the storage used for that store? Some will call the storages own snapshot technology which won't always work - maybe try setting the snapshot type to 1 (system) to see if that helps.

Authorised Symantec Consultant

Don't forget to "Mark as Solution" if someones advice has solved your issue - and please bring back the Thumbs Up!!.

Boolenat's picture

in dag is users mailboxes and in information store is email archive mailboxes. i browsed for select for many times so it correct. i try both with and whiout snapshot. without snapshot message that it's unsupported type or something like that. also try different type of snapshots, and error the same.

storage is DD 860 and i have about 20 backup jobs successed every day on it (exchange dag2010, sql,vmware). only 1 problem with this exchange archive db server. i suppose it in connection information store + vss + netbackup problem. but don't know what to try more.

Mark_Solutions's picture

Have you tried it using the DAG name rather than the mailbox server name and browsing the DAG?

Do the mailbox servers have the NetBackup Legacy Network Service set to use the exchange admin account too?

Authorised Symantec Consultant

Don't forget to "Mark as Solution" if someones advice has solved your issue - and please bring back the Thumbs Up!!.

Boolenat's picture

but why should i choose dag if my db in information store on different server, which is not a part of dag?

service - yes, all started wich admin credential

Mark_Solutions's picture

Ok - so this is effectively a standalone exchange server not related to your DAG - that is fine then, just that usually Exchange 2010 tends to have a DAG of some sort rather than just a mailbox name, so uses bpresolver etc and if it cannot locate the database it wont work and leaves the VSS writer unstable

Can you run the following and show the outputs please:

Diskshadow

List Writers Status

List Shadows All

Thanks - just wondering if there is a corrupt shadow left behind somewhere that needs removing to solve this

Authorised Symantec Consultant

Don't forget to "Mark as Solution" if someones advice has solved your issue - and please bring back the Thumbs Up!!.

John Kozitzki's picture

I would try what Mark_Solutions suggests...

Somehow, you are getting mail from the DAG to the Archive Information Store...if it is tied to the DAG through a replication mechanism, the Archive Information Store would be referenced within the DAG.

If it is through some mail forwarding or smtp tap method, then the Archive Information Store is probably standalone and you would configure the policy as such.

Either method, there may be some residual reference to the DAG in the Archive Information Store that may be causing the issue...checking with the DAG name references may help to identify.

Boolenat's picture

Result in attach. Microsoft Exchange Writer in waiting mode, but if i restart information store service and try to backup the error will be the same.

AttachmentSize
cmd.txt 3.81 KB
Mark_Solutions's picture

OK - one last attempt for now before saying what Symantec will say (that it is a Microsoft VSS issue and you need a rollup package / hotfix - which I have yet to see anyone for whom that has worked! - I assume your systems are all up to date with WIndows updates and Exchange Service packs etc?) ...

Can you try

bpfis query

in case NetBackup has left something behind that needs clearing down

Authorised Symantec Consultant

Don't forget to "Mark as Solution" if someones advice has solved your issue - and please bring back the Thumbs Up!!.

Boolenat's picture

yes of course all systems up to date. i have an idea. i'll try now backupexec job on this server. before i setup netbackup, it was backupexec on it and jobs were done succsessfull. now i reinstall agents and try. let you know later.

>bpfis query

 

INF - BACKUP START 7688
INF - FIS IDs:
INF - EXIT STATUS 0: the requested operation was successfully completed
Boolenat's picture

As I supposed, backup exec 2010r3 job on this server begin normally, snapshot were successful created and backup started. So the problem is in netbackup. maybe it have problem in heterogeneous infrastructure? Because I have dag servers and dedicated standalone servers with mailbox database in my Exchange enviroment? 

Boolenat's picture

I think I should some thing to do with this:

16:34:01.966 [4520.5492] <4> bpfis: INF - Preparing freeze of Microsoft Information Store:\VGK_Archive using snapshot method VSS_Writer.
16:34:01.966 [4520.5492] <4> bpfis: INF - Created mount point C:\Program Files\Veritas\NetBackup\temp\_vrts_frzn_img_Microsoft Information Store_VGK_Archive_4520_1
16:34:42.982 [4520.5492] <32> bpfis: FTL - VfMS error 11; see following messages:
16:34:42.982 [4520.5492] <32> bpfis: FTL - Fatal method error was reported
16:34:42.982 [4520.5492] <32> bpfis: FTL - vfm_freeze_commit: method: VSS_Writer, type: FIM, function: VSS_Writer_freeze_commit
16:34:42.982 [4520.5492] <32> bpfis: FTL - VfMS method error 4; see following message:
16:34:42.982 [4520.5492] <32> bpfis: FTL - VSS_Writer_freeze_commit: snapshot create failed
16:34:42.982 [4520.5492] <32> bpfis: FTL - VfMS error 11; see following messages:
16:34:42.982 [4520.5492] <32> bpfis: FTL - Fatal method error was reported
16:34:42.982 [4520.5492] <32> bpfis: FTL - vfm_freeze_commit: method: VSS_Writer, type: FIM, function: VSS_Writer_freeze_commit
16:34:42.982 [4520.5492] <32> bpfis: FTL - VfMS method error 4; see following message:
16:34:42.982 [4520.5492] <32> bpfis: FTL - VSS_Writer_freeze_commit: snapshot create failed
16:34:42.982 [4520.5492] <4> bpfis: INF - Thawing NEW_STREAM0 using snapshot method VSS_Writer.
16:34:42.982 [4520.5492] <4> bpfis: INF - do_thaw return value: 0
16:34:42.982 [4520.5492] <4> bpfis: INF - Deleted mount point C:\Program Files\Veritas\NetBackup\temp\_vrts_frzn_img_NEW_STREAM0_4520_1
16:34:42.982 [4520.5492] <4> bpfis: INF - Thawing Microsoft Information Store:\VGK_Archive using snapshot method VSS_Writer.
16:34:42.982 [4520.5492] <4> bpfis: INF - do_thaw return value: 0
16:34:42.982 [4520.5492] <4> bpfis: INF - Deleted mount point C:\Program Files\Veritas\NetBackup\temp\_vrts_frzn_img_Microsoft Information Store_VGK_Archive_4520_1
16:34:42.982 [4520.5492] <16> bpfis: FTL - snapshot creation failed - Error attempting to gather metadata., status 130
16:34:42.982 [4520.5492] <4> bpfis: INF - Thawing NEW_STREAM0 using snapshot method VSS_Writer.
16:34:42.982 [4520.5492] <8> bpfis: WRN - NEW_STREAM0 is not frozen
16:34:42.982 [4520.5492] <4> bpfis: INF - Thawing Microsoft Information Store:\VGK_Archive using snapshot method VSS_Writer.
16:34:42.982 [4520.5492] <8> bpfis: WRN - Microsoft Information Store:\VGK_Archive is not frozen
16:34:42.982 [4520.5492] <8> bpfis: WRN - snapshot delete returned status 20
16:34:42.982 [4520.5492] <4> bpfis: INF - EXIT STATUS 130: system error occurred

any ideas?

Mark_Solutions's picture

Can you do a list providers so that we can see what it things is available for VSS - it seems to think that a backup is still in progress and faults

Also check this out just in case (though doesnt look like it from your logs)

http://www.symantec.com/docs/TECH154784

Finally - please work through this one to see if it helps:

http://www.symantec.com/docs/TECH191343

Authorised Symantec Consultant

Don't forget to "Mark as Solution" if someones advice has solved your issue - and please bring back the Thumbs Up!!.

Boolenat's picture

1) already done. i also create new db named 123. same error. and i try reinstall netbackup agent to folder c:\veritas.

2) checked. all ok. last part for dag, i'm not checking it. because its not dag db. 

Mark_Solutions's picture

At a loss now then unless the Backup Exec agent has left stuff behind that is conflicting with the NetBackup client causing the snapshot to fail

NetBackup tends to use Backup Exec binaries so unless the rights used by the NetBackup Client and Legacy Network Service are not the same as those used by Backup Exec I cannot see why one should work and one fail.

All I can say is to double check everything carefully as per the NetBackup Admin Guide etc. and make sure BUE was removed fully and the server rebooted before installing NetBackup

Also double check that the server does not appear in the Master Servers Host Properties - Client attributes section in case it has picked up a setting causing any issues

Authorised Symantec Consultant

Don't forget to "Mark as Solution" if someones advice has solved your issue - and please bring back the Thumbs Up!!.

Boolenat's picture

I install netbackup agent on all servers with previos installed backup exec. no problems. only here. also i many time recheck everything by admin guide. and every time same error:

An existing snapshot is no longer valid and cannot be mounted for subsequent operations(1542)

 

Mark_Solutions's picture

That message is usually one one of 2 things - VSS / Shadow Errors or a rights issue

 

Make sure everything is OK with Exchange itself (guessing it is as BUE works) and check for any application and system even log messages during the backup

It may be worth increase the client logging levels to see if bpfis will give a little more detail as to what is going wrong

Authorised Symantec Consultant

Don't forget to "Mark as Solution" if someones advice has solved your issue - and please bring back the Thumbs Up!!.

Boolenat's picture

Recheck all. Increase log level to 5. Same error. Log in attach

AttachmentSize
040513.txt 303.03 KB
Mark_Solutions's picture

A few things here but do please answer all of them...

 

Can you tell me about the server itself, what disk is being used here (local, SAN etc.), physical machine, virtual, and volume managers etc.

Did you clear out the C:\Veritas\NetBackup\temp\ and C:\Veritas\netbackup\online_util\fi_cntl directories  - just wondering why it says this in the log:

Have 10 snapshot list entries and 2 filesystem device list entries.

I assume "backup" is the user name for the services - is this account specifically set to be a local admin on the server?

Could i see a screen shot of your policy attributes tab as it also says this:

Database backup source not configured in the policy. Use backup passive database instance first.

Although it says it is not part of a DAG it does seem to think it has a CAS (cas.main.vgk) so not sure if this is confisuing things.

What I do see though is this line:

EseE14::GetTargetDatabaseCopyStatus(OUT) ... dag:No, status:"Mounted", active:"EXA", passive:"", healthy:No, mdb:Yes, cas:cas.main.vgk, clustered:No

Perhaps the database has issues and this is actually causing the failure.

Finally for now (a lot in the log file!) I see this line:

ERR - ubsStart_eseutil():CreateProcess() failed for "C:\Distr\exchange 2010 SP2\bin\eseutil.exe" /ml "\\?\GLOBALROOT.........

It indicates it cannot find eseutil.exe in the default path and looks like it is using one for the SP2 installation folder? This is well worth looking into to see why this should be happening as that version may not be suitable

Authorised Symantec Consultant

Don't forget to "Mark as Solution" if someones advice has solved your issue - and please bring back the Thumbs Up!!.

Boolenat's picture

1) this vmware 5.0 machine with mounted 3 datastore. max size of 1 datastore 2048 gb

2) yes, this folders empty

3) backup is admin + organization managment

4) in attach

5) cas - is client acces server, this information backup get from AD. EXA server have only mailbox role.

6) DB is active and working. with no errors.

7) i delete folder C:\Distr\exchange 2010 SP2. Eseutil.exe located in "c:\Program Files\Microsoft\Exchange Server\V14\Bin\eseutil.exe" and working (i can run it)

1.JPG 2.JPG 3.JPG 4.JPG
Mark_Solutions's picture

OK, now that you have cleared the eseutil could you try changing the snapshot type to 1 (system) - also make sure VMWare tools are up to date on the server as the snapshot and VSS will interface with VMWare

If both directories have been cleared down (C:\Veritas\NetBackup\temp\ and C:\Veritas\netbackup\online_util\fi_cntl), VMWare tools are up to date and policy set to VSS type 1 then re-try the backup

Rename the current bpfis log first and then if it fails post the new log file

Authorised Symantec Consultant

Don't forget to "Mark as Solution" if someones advice has solved your issue - and please bring back the Thumbs Up!!.

Boolenat's picture

but I espessially remove vmware vss addon on all servers. only ms vss left

Mark_Solutions's picture

OK - still worth trying it set to system - 1 just in case

Authorised Symantec Consultant

Don't forget to "Mark as Solution" if someones advice has solved your issue - and please bring back the Thumbs Up!!.

Boolenat's picture

well ok, i install vmware vss and chage snapshot type to 1. logs in attach

AttachmentSize
040513.txt 303.04 KB
Mark_Solutions's picture

Something still referencing it:

INF - ubsStart_eseutil():eseutil command issued - "C:\Distr\exchange 2010 SP2\bin\eseutil.exe" /ml "\\?\GLOBALROOT\De....
INF - ERR - ubsStart_eseutil():CreateProcess() failed for "C:\Distr\exchange 2010 SP2\bin\eseutil.exe" /ml "\\?\GLOBALROOT\.....
INF - ERR - ubsGet_lGenerationLog():ubsStart_eseutil() failed 0x4dd2c0.

INF - ERR - ubsGetExchangeLogsToBackup():ubsGet_lGenerationLog() failed - 0xc.
INF - ERR - GetExchangeLogsToBackup: ubsGetExchangeLogsToBackup() failed.
INF - ERR - EnumerateForLocalMetaData: GetExchangeLogsToBackup() failed.
INF - Informational: Closing Component 'Logs'
INF - Informational: All objects not processed. Forcing the status to 'false' for SetBackupSucceeded()
INF - EnumerateForLocalMetaData exiting.
INF - INF - EnumerateForLocalMetaData exiting.
INF - VSS_Writer_freeze_commit: snapshot create failed
INF - GetDiskFreeSpaceEx failed for NEW_STREAM0. Setting the snapshot size as default 2048 bytes

Something is just wrong here as it still wants to use that eseutil and also cannot get free disk space for the snapshot.

Needs to be looked at a lot deeper i think!

Authorised Symantec Consultant

Don't forget to "Mark as Solution" if someones advice has solved your issue - and please bring back the Thumbs Up!!.