Video Screencast Help
Symantec to Separate Into Two Focused, Industry-Leading Technology Companies. Learn more.

Issues with partitions not marking as backed up

Created: 10 Oct 2012 • Updated: 16 Oct 2012 | 14 comments
This issue has been solved. See solution.

I experiencing an issue with a recently created EV mailbox archiving partition.

I am using the "partitionsecurednotification.xml" trigger file as described in http://www.symantec.com/business/support/index?page=content&id=TECH35610 however despite the file being processed (marked as .old) and there being an event logged (ID 7130) stating successful validation of the trigger file the backup tab of the partition states the last item secured was only a couple of days after the creation of the partition (nearly 1 month ago).

I have 2 critical events logged in the status page the first states a partition has not been backed up for more than 1 day (ID 41259) and the other that there are more than 4.1 million savesets that have not been backed up (ID 41008)

I also see 3 events being logged in the EV log at the time the partitions are taken out of backup mode and the xml files are processed, I am unsure if these are related however. details below:

Event ID 13360

An error was detected while accessing the Vault Database 'SQL\Instance' (Internal reference: {CADODataAccess::ExecuteSQLCommand} [.\ADODataAccess.cpp, lines {1407,1409,1424,1442}, built Nov 11 21:02:49 2011]):

Description:  

Could not find stored procedure 'dbo.usp_ReRaiserror'.
 
 
SQL Command:
 uspd_Journals
 

Event ID 13360

An error was detected while accessing the Vault Database 'SQL\Instance' (Internal reference: {CADODataAccess::ExecuteSQLCommand} [.\ADODataAccess.cpp, lines {1407,1409,1424,1442}, built Nov 11 21:02:49 2011]):

Description:  

The 'uspd_Journals' procedure attempted to return a status of NULL, which is not allowed. A status of 0 will be returned instead.
 
 
SQL Command:
 uspd_Journals

Event ID 6796

A COM exception has been raised.
<0x80040e14>
Internal reference
{CVaultStoreDB::ClearExpiredJournalRecords} [.\VaultStoreDB.cpp, lines {12478,12487,12489,12490,12492}, built Nov 11 21:03:36 2011]
 
An exception is raised when a process encounters an unexpected fault.
 

 

thanks

Comments 14 CommentsJump to latest comment

Rich Jones's picture

The partition in question is on an NTFS drive but is backed up via Netapp snapshot technology hence use of xml trigger files.

ta

LCT's picture

What version of EV are you using and service pack? Has this configaration ever worked? What has changed recently to cause this issue?

 

Rich Jones's picture

oops forgot the vital info! sorry.

I am on V9.0.3 and the configuration works fine for the partition that was open prior to creating this new one (and indeed still does, just not on the new one.)

 

the only thing that has changed is the creation of the new partition (the one causing the issues) I cant think of any other changes that have been made.

Jeff Shotton's picture

The ReRaiseError 'error' is a bit of a red herring. In EV9 sp3 this stored proc was left out by accident. All it means is you get a different and less helpful error message if that one SP should throw an error (its use is just to report an error)

Have you tried using IgnoreArchiveBitTrigger.txt?

Do you use PartitionSecuredNotification.xml successfully on other partitions, or is this the first one? This file is created by (certain) backup software rather than the old ignorearchivebittrigger.txt which is created manually and is simply a blank file. Are you creating the file manually or allowing an integrated backup procedure to do it?

Regards,

Jeff

Jeff Shotton

Principal Consultant

Adept-tec Ltd

Website: here

Jeff Shotton's picture

And have you validated that the XML is a valid document?

Jeff Shotton

Principal Consultant

Adept-tec Ltd

Website: here

LCT's picture

I had a similar issue with EV9.0.3. have you seen the technote below? My issue affected newly opened partition and eventually existing recent closed partitions.

http://www.symantec.com/business/support/index?page=content&id=TECH180841

I am using IgnoreArchiveBitTrigger.txt. I guess you need to dtrace your issue and check the error in the dtrace and confirms it is the same issue as the technote.

Hope that helps.

SOLUTION
Rich Jones's picture

Jeff, cool. I will ignore the re-raise error bit for the moment then.

the xml file method is used successfully on a number of other partitions and is created via powershell script which copies it to all needed partitions and modifies the date. I am sure it is fine because it works on all the others and the event log throws up the event for the partition in question that advises it was validated.

 

LCT - I will have a read through that technote and run a dtrace to confirm and get back to you. thanks!

 

Rich Jones's picture

LCT - I get the following entry in my drace but not the corresponding exit exactly as per the tech article.

 

{CVSDBAccessor::GetUnsecuredSavesetsCount} (Entry) PartitionIdentity = [4]

 

do you know how I could confirm that partition identity = [4] corresponds to the partition I have the issue with?

 

below is what I think is the relevant part of the trace.

9921    11:14:21.445     [8584]    (StorageFileWatch)    <7944>    EV:L    {CVSDBAccessor::OpenConnection} (Entry)
9922    11:14:21.445     [8584]    (StorageFileWatch)    <7944>    EV:L    CADOContext::CreateConnection entry
9923    11:14:21.445     [8584]    (StorageFileWatch)    <7944>    EV:L    CADOContext::CreateConnection exit. source:Provider=SQLOLEDB;Server=SQL\Instance;Database=EVMailboxArchive;Integrated Security=SSPI hr=Success  (0)
9924    11:14:21.445     [8584]    (StorageFileWatch)    <7944>    EV:L    CVSDBAccessor::OpenConnection Information: Created a new ADOContext and DataAccess Object for Provider=SQLOLEDB;Server=SQL\Instance;Database=EVMailboxArchive;Integrated Security=SSPI connection string.
9925    11:14:21.445     [8584]    (StorageFileWatch)    <7944>    EV:L    {CVSDBAccessor::OpenConnection} (Exit) Status: [Success]
9926    11:14:21.445     [8584]    (StorageFileWatch)    <8012>    EV:L    CADODataAccess::GetParameterValue returned a VT_I4 value of 0
9927    11:14:21.445     [8584]    (StorageFileWatch)    <7944>    EV:L    {CWatchFileScanRequestBase::GetWatchFileEnumBatchSize} (Entry)
9928    11:14:21.445     [8584]    (StorageFileWatch)    <7944>    EV:L    CWatchFileScanRequestBase::GetWatchFileEnumBatchSize Warning: Could not read the chunk size for Watch file batch enumeration. VaultStoreIdentity = [2] and PartitionIdentity = [4].
9929    11:14:21.445     [8584]    (StorageFileWatch)    <7944>    EV:L    {CWatchFileScanRequestBase::GetWatchFileEnumBatchSize} (Exit)
9930    11:14:21.445     [8584]    (StorageFileWatch)    <7944>    EV:L    {CWatchFileScanRequestBase::GetMaxEntryInSISPartIdCache} (Entry)
9931    11:14:21.445     [8584]    (StorageFileWatch)    <7944>    EV:L    CWatchFileScanRequestBase::GetMaxEntryInSISPartIdCache Warning: Could not read the max entry count for sis part id cache. ValueUsed is [100000] VaultStoreIdentity = [2] and
.
9933    11:14:21.445     [8584]    (StorageFileWatch)    <7944>    EV:L    {CVSDBAccessor::GetUnsecuredSavesetsCount} (Entry) PartitionIdentity = [4]
9934    11:14:21.445     [8584]    (StorageFileWatch)    <7944>    EV:L    CVSDBAccessor::OpenConnection Information: An ADOContext Object already exists for Provider=SQLOLEDB;Server=SQL\Instance;Database=EVMailboxArchive;Integrated Security=SSPI connection string.
9935    11:14:21.445     [8584]    (StorageFileWatch)    <7944>    EV:L    CADODataAccess::CreateCommand entry
9936    11:14:21.445     [8584]    (StorageFileWatch)    <7944>    EV:L    CADODataAccess::CreateCommand exit. hr=Success  (0)
9937    11:14:21.445     [8584]    (StorageFileWatch)    <7844>    EV:L    {CWatchFileTimer::CheckTriggerFileExists}|Trigger file **removed**\PartitionSecuredNotification.xml not found so searching for .txt file
9938    11:14:21.445     [8584]    (StorageFileWatch)    <7844>    EV:L    {CWatchFileTimer::CheckTriggerFileExists}|Trigger file **removed**\IgnoreArchiveBitTrigger.txt also not found.
9939    11:14:21.445     [8584]    (StorageFileWatch)    <7844>    EV:L    {CWatchFileTimer::CheckTriggerFileExists} (Exit)
9940    11:14:21.445     [8584]    (StorageFileWatch)    <7844>    EV:L    {CWatchFileTimer::GetCurrentPartitionState} (Entry) VaultStoreIdentity = [6] PartitionIdentity = [0]
9941    11:14:21.445     [8584]    (StorageFileWatch)    <7844>    EV:L    {CWatchFileTimer::GetCurrentPartitionState} (Exit) Status: [Success]
9942    11:14:21.445     [8584]    (StorageFileWatch)    <7844>    EV:L    CWatchFileTimer::GetGreatestCommonDivisor Information: GCD of [60] and [60] is [60].
9943    11:14:21.445     [8584]    (StorageFileWatch)    <7844>    EV:L    {CWatchFileManager::PartitionScanRequired} (Entry) PartitionIdentity = [0] VaultStoreIdentity = [6] PartitionStatus = [2]
9944    11:14:21.445     [8584]    (StorageFileWatch)    <7844>    EV:L    {CWatchFileManager::PartitionScanRequired} (Exit)
9945    11:14:21.445     [8584]    (StorageFileWatch)    <7844>    EV:L    {CWatchFileTimer::IsPeriodicRequestQueued} (Entry)
9946    11:14:21.445     [8584]    (StorageFileWatch)    <7844>    EV:L    {CWatchFileTimer::IsPeriodicRequestQueued} (Exit) Status: [Success]
9947    11:14:21.445     [8584]    (StorageFileWatch)    <7844>    EV:L    {CWatchFileTimer::CheckTriggerFileExists} (Entry)
9948    11:14:21.445     [8584]    (StorageFileWatch)    <7844>    EV:L    {CWatchFileTimer::CheckTriggerFileExists}|Trigger file **removed**\PartitionSecuredNotification.xml not found so searching for .txt file
9949    11:14:21.445     [8584]    (StorageFileWatch)    <7844>    EV:L    {CWatchFileTimer::CheckTriggerFileExists}|Trigger file **removed**\IgnoreArchiveBitTrigger.txt also not found.
9950    11:14:21.445     [8584]    (StorageFileWatch)    <7844>    EV:L    {CWatchFileTimer::CheckTriggerFileExists} (Exit)
9951    11:14:21.445     [8584]    (StorageFileWatch)    <7844>    EV:L    CWatchFileTimer::StartRoutine Information: PSNScan going to sleep for [3600000] milliseconds for StorageService = [19F04B5681C84C14BB64060ECFF86388B1e10000EVSite]
9952    11:14:21.445     [8584]    (StorageFileWatch)    <4920>    EV:M    CCollector::CompactSparseCollections() - Sparse collection will require at least 2% of free disk space available
9953    11:14:21.460     [8584]    (StorageFileWatch)    <4920>    EV:M    EVCommon::IsRunningFromSISTools returned: false
9954    11:14:21.460     [8584]    (StorageFileWatch)    <4920>    EV:M    CCabFileMutex::wait[LOCK] :  Attempting to take a lock on file **removed**\2007\11-12\0\Collection1626.CAB.lock
9955    11:14:21.460     [8584]    (StorageFileWatch)    <4920>    EV:M    CCabFileMutex::wait :  handle: 95c
9956    11:14:21.460     [8584]    (StorageFileWatch)    <4920>    EV:M    CCabFileMutex::wait[LOCK] :  Lock taken on file **removed**\2007\11-12\0\Collection1626.CAB.lock, slept for 0 ms
9957    11:14:21.460     [8584]    (StorageFileWatch)    <5700>    EV:L    CADODataAccess::GetParameterValue returned a VT_I4 value of 0
9958    11:14:21.460     [8584]    (StorageFileWatch)    <4644>    EV:L    CADODataAccess::GetParameterValue returned a VT_I4 value of 0
9959    11:14:21.476     [8584]    (StorageFileWatch)    <4920>    EV:M    EVCabinet (ExtractFromCab) : Setting last-access time for Collection file **removed**\2007\11-12\0\Collection1626.CAB. HRESULT = 0
9960    11:14:21.476     [8584]    (StorageFileWatch)    <4056>    EV:L    CADODataAccess::GetParameterValue returned a VT_I4 value of 0
9967    11:14:21.491     [8584]    (StorageFileWatch)    <7228>    EV:L    CADODataAccess::GetParameterValue returned a VT_I4 value of 0
9968    11:14:21.538     [8584]    (StorageFileWatch)    <4644>    EV:L    CADODataAccess::GetParameterValue returned a VT_I4 value of 0
9969    11:14:21.538     [8584]    (StorageFileWatch)    <736>    EV:M    Collector is walking the directory tree for Partition 16715050F923DF742A936E1953A8745B61q10000EVSite
9970    11:14:21.554     [8584]    (StorageFileWatch)    <4056>    EV:L    CADODataAccess::GetParameterValue returned a VT_I4 value of 0
9971    11:14:21.554     [8584]    (StorageFileWatch)    <7228>    EV:L    CADODataAccess::GetParameterValue returned a VT_I4 value of 0
 

LCT's picture

Hi Rich,

What you are looking for in your dtrace is any sign of this line:

{CVSDBAccessor::GetUnsecuredSavesetsCount} (Exit) Status: [Success]

The line:

{CVSDBAccessor::GetUnsecuredSavesetsCount} (Entry) PartitionIdentity = [4]

this is your vault store partition identity which is number 4. the technote example is [0] which probably means it only has one vaultstore partition.

If you can't see the line below after your affected partition 

{CVSDBAccessor::GetUnsecuredSavesetsCount} (Exit) Status: [Success]

for the affected partition then I suspect you have the same problem as the technote described.

Backup your EV vaultstore DBs and then apply the workaround, reapply the xml file with backup and the restart EV services.

Jeff Shotton's picture

 

You will get the vault store identity from looking at the vaultstoreentry table:

use Enterprisevaultdirectory

select * from vaultstoreentry

Look through and find the vaultstoreentryid for the vault store in question.

 

The partitionentryid numbers are 0 based and in order added for a particular vault store, so you will be able to see which number partition this is in the VAC. (total -1 for the latest one)

The SQL patch is likely the solution.

 

Regards,

Jeff

 

 

Jeff Shotton

Principal Consultant

Adept-tec Ltd

Website: here

LCT's picture

oh, you need to go on the SQL server, in your EnterpriseVaultDirectory and then expand to Tables and then open PartitionEntry it will have all your vaultstore partitions details on their. You'll see partition id 0-4 on there (or more).

Rich Jones's picture

thats perfect guys. I have confirmed that partitionID 4 is the partition in question that does not seem to work and I have checked the trace and while the other partitions have:

{CVSDBAccessor::GetUnsecuredSavesetsCount} (Entry) PartitionIdentity = [X]

{CVSDBAccessor::GetUnsecuredSavesetsCount} (Exit) Status: [Success]

in fairly close order (sub 1 second) but after I see:

{CVSDBAccessor::GetUnsecuredSavesetsCount} (Entry) PartitionIdentity = [4]

I get no exit code for over 20 seconds (till the trace ends) most of the events are like the following

9997    11:14:21.554     [8584]    (StorageFileWatch)    <2516>    EV:L    The Collector for Partition 1530A0C3A0EC2C44DB677F57B32E0D9011q10000EVSite is parsing directory <Partition Location>\2010\07-19

I will perform a backup, run the script and let you know the results!
 

Rich Jones's picture

hi

 

it looks like the tech note at http://www.symantec.com/business/support/index?page=content&id=TECH180841 did indeed resolve the issue. I have applied it and re-ran the dtrace and I see the correct entries and the backlog is clearing.

 

thanks guys