Video Screencast Help

Unsecured SIS parts in other partitions.

Created: 12 Mar 2013 • Updated: 14 May 2013 | 8 comments
This issue has been solved. See solution.

Hi All,

The EV Operations Manager site shows me that “Number of Saveset files awaiting backup or replication” is 163. I know it’s not a very big number compared to some that I have seen described here but it has been static for quite a while which made me start looking into it.

All partitions are set to “Check for trigger file”, EV enters backup mode successfully, backups have completed successfully, trigger files have been created in all open and closed partitions and then EV is taken out of backup mode. After this, all the trigger files are automatically renamed to IgnoreArchiveBitTrigger.old by EV. The Status section on the Backup tab of each partition has a very recent date under “Last item secured”.

I have 3 partitions with "Unsecured SIS parts in other partitions". I could be wrong but logically I would expect there to be a similar number in "Unsecured SIS parts in local partition" on other partitions.

There are 163 items in the WatchFile table all with ItemSecured set to 1 and 163 items in the JournalArchive table with BackupComplete set to 0. There are 193 items in the WatchSISPartFile table. I’m guessing more than one SISPart for some of the items in the WatchFile as there are 163 distinct SavesetIdentity’s.

Can anyone offer any suggestions to take this further?

Thanks,

Mark.

Operating Systems:

Comments 8 CommentsJump to latest comment

JesusWept3's picture

DTrace StorageFileWatch and then go in and out of backup mode and look for the savesets that you saw in the watch file sis part table and see if there's an issue there, maybe the file no longer exists?

Advisor's picture

To add more, if files no longer exists, then run EVSVR verify operation to know exact count and more details about the error.

Mark_H's picture

Thanks for the suggestions.

JesusWept3, I was running DTrace yesterday but struggling a bit to actually follow the log. It seemed to bounce around looking at more than one saveset over various partitions simultaneously with collections trying to do things at the same time. I've narrowed the schedule on collections for now and will have another go with DTrace.

EV_Arjun, I’m not a frequent SQL user and the tables in the fingerprint database didn’t make much sense so I had a bit of a look at EVSVR to see if that might take this further. I ran it with the Verify operation and the Complete option over a small date range as a test run. It took 18 hours and with the date range I gave it, 2 partitions were covered (one closed and one open). The closed partition threw A LOT of the following error:

Fingerprint validation failed [0x80041b9f].

I found that error message here: http://www.symantec.com/business/support/index?page=content&id=TECH166430. If it's related, it's not helping much.

 The final entry for that partition was **** EVSVR ERROR: Failed to process Partition 'MailboxStore 001 Ptn8': Error: Unspecified error  [0x80004005]

The open partition didn't throw much information but also threw the error: **** EVSVR ERROR: Failed to process Partition 'MailboxStore 001 Ptn9': Error: Unspecified error  [0x80004005]

 

Thanks for your help,

Mark.
 

Mark_H's picture

Hi Rob,

Sorry, I forgot to come back and update this. I had a call open with Symantec and they used some internal SQL scripts to update the Secured flag of the SIS parts in question in the Fingerprint database. I was a bit uncomfortable with what felt like a hack but after providing DTrace's and deployment scanner logs a root cause wasn't found. I haven't had a chance (or a need) to revisit the issues I was having trying to use EVSVR though.

From memory the basic stages we went through are below in case anyone reads this thread in years to come.

If you have any thoughts on this process (positive or otherwise), I'd appreciate hearing them.

Thanks,
Mark.

--
We went through the following article (again):
Why items may get 'stuck' in a Pending State - Enterprise Vault (EV) Basic Archive Process: http://www.symantec.com/business/support/index?page=content&id=TECH66202

Once that was done, the engineer changed focus to a particular Saveset ... which explained the link between the vault store database and the fingerprint database member tables that previously stumped me. Here's the instructions from that conversation:

<<
1. Logon to SQL and change focus to the Vault Store database

2. Run the following SQL query to check that the DVS item is secured

SELECT SavesetIdentity,ItemSecured FROM WatchFile WHERE SavesetIdentity = 'SaveSetIdentity'

This will return the SaveSetIdentity and if it is secured (which it should be)

3. As the SIS parts are referenced in the fingerprint database, we need to locate which table in the database holds the SIS part reference

SELECT SavesetIdentity,FPDistinctionByte,ParentTransactionId FROM WatchSISPartFile WHERE SavesetIdentity = 'SaveSetIdentity'

This will return the SaveSetIdentity with the finger print MemberTable (FPDistinctionByte) and the item's ParentTransactionID

4. Knowing this information we can check that the SIS part has been secured.
Change focus to the fingerprint database

SELECT ParentTransactionId,Secured FROM MemberTable_FPDistinctionByte WHERE ParentTransactionId = 'ParentTransactionId'
>>

Following that he used evsvr with the dumpsaveset switch to confirm that the saveset and all its sisparts could be recombined successfully (I was concerned that if the Sisparts weren't "Secured" in the fingerprint database it may not even exist on disk).

Once my arm was twisted that the cause was probably due to a glitch during an upgrade at some point and no data was missing, we went ahead and updated the Secured flag on the SISPart.

The manual process was: UPDATE MemberTable_FPDistinctionByte SET Secured = 1 WHERE ParentTransactionId = 'ParentTransactionId'

The Engineer needed to get approval from backline support before he could use a bulk approach to perform this task and mentioned that this should NEVER be done without Symantec support supervision so I won't repeat all the details but the basics were:

Ran ‘CreateSPFindUnsecuredSISParts.sql’ script to create the stored procedure.
Ran ‘Select unsecured sis parts.sql’ script and took the count of unsecured SISparts.
Ran ‘Reset sis parts unsecured status.sql’ script and now all the SISparts are secured.
--

SOLUTION
DeadEyedJacks's picture

@MarkH

FYI, I've seen this issue with unsecured remote SIS parts a number of times now.

Although the remedial SQL scripts fix the flags, there's not been a root cause identified.

Also the issue does seem to reoccur, so keep the Select and Reset scripts handy.

Authorised Symantec Consultant on Archiving and eDiscovery ASC, STS, SCS, SSE+

Microsoft, NetApp and VMware certified professional MCTS, MCSE, MCSA, NCDA, NCIE-BR, VCP, VTSP

jgorman's picture

I might have found a possible root cause for this scenario(or at least our scenario). In our case we just finished a large migration of data from EV 7.5 to EV10 using TansVault. After the ingest of mail we had roughly 400-million savesets needing backup. We placed trigger files and flipped backup mode many times over the course of many weeks. The numbers dropped in batches and then would hang for a while and not change at all. Support got us a hotfix that broke the process down into smaller batches and that really cleared the savesets. We are now down to right above 2 million items needing backup. Our nightly archiving run pulls 200-300k new messages per day and after we flip backup mode with a trigger file in place we see those items cleared but we never get below 2 million. During the clearing of saveset we also removed a large batch of archives (a couple thousand former employees that are now gone). I suspect that these SISParts are missing because they were deleted prior to the SISPart being secured.

Am I in left field with this theory?