Video Screencast Help
Symantec to Separate Into Two Focused, Industry-Leading Technology Companies. Learn more.

Problems with error when moving archive.

Created: 11 Mar 2013 | 16 comments

Hi,

I´m trying to move an archive where I get a few errors:

Move Archive subtask 1DAE28830A14ED64E9F7E600D349E2E881013700evsite - 2013-03-06 16:05:49
Phase Item ID Source saveset ID Destination saveset ID Processing  error? Error code Error description Extended error code Extended error description Destination item deleted? Destination item deletion reason Source item deleted? Source item deletion reason Verification failure? Item subject Item folder path Item sent/delivery time Source item retrieval Destination item retrieval Source item URL Destination item URL 
Copying 361CE443AC564046A4101680C4271510 - - Yes 80040306 An internal failure occurred. Internal Error: 'Saveset decompression failed.   Error:%1   Context:%2      (0xc0041aa7)'. c0041aa7 Saveset decompression failed.   Error:%1   Context:%2      No - - - No - - - - - - - 
Copying 1881E5B8C6B5401EA5FA881E44B02C90 - - Yes 80040306 An internal failure occurred. Internal Error: 'Saveset decompression failed.   Error:%1   Context:%2      (0xc0041aa7)'. c0041aa7 Saveset decompression failed.   Error:%1   Context:%2      No - - - No - - - - - - - 
Copying 20BC3E57B72944489E75C52A4E6E9D50 - - Yes 80040306 An internal failure occurred. Internal Error: 'Saveset decompression failed.   Error:%1   Context:%2      (0xc0041aa7)'. c0041aa7 Saveset decompression failed.   Error:%1   Context:%2      No - - - No - - - - - - - 
Copying 858E611CC25144F89042CF7B62A65850 - - Yes 80040306 An internal failure occurred. Internal Error: 'Saveset decompression failed.   Error:%1   Context:%2      (0xc0041aa7)'. c0041aa7 Saveset decompression failed.   Error:%1   Context:%2      No - - - No - - - - - - - 

I have tried added <add key="NonFatalHexCodes" value="0x80040306,0xc0041aa7" /> in EVMoveArchiveTask.exe.config but I still get failures on the same files.

Does someone have an idea of how I can get EV to ignore the error?

/Henfo

Operating Systems:

Comments 16 CommentsJump to latest comment

Advisor's picture

EV version? Source and destination...

Henfo's picture

Hi,

Moving from EV 9.0.2 -> 10.0.2, I ran a Dtrace and got another errorcode that I put in EVMoveArchiveTask.exe.config so now the move is running :).

Advisor's picture

Glad to know... :)

Chris Warren's picture

Henfo,

The move archive 'NonFatalHexCodes' value should commonly only be used as a last resort as it will cause the Move process to ignore listed failures that would normally fail the move process.  9.0.2 had a couple of issues related to how storage handled damaged legacy items (pre-8.0 stored items) so I would recommend upgrading to 9.0.4 if you can and test without the NonFatalHexCodes set as this is a bandaid and not addressing the real issue.

Here's an example of one of the fixes:

http://www.symantec.com/docs/TECH169017
The Move Archive process fails during Copy phase with "Inner storage error code: 0x80004002"

When there is a Move Archive failure during the copy stage, you can test the individual items using EVSVR - Dumpsaveset to confirm the item is retrievable from storage.  See below for details:

http://www.symantec.com/docs/TECH75791
How to extract the Saveset and SIS parts of a specific archived item using EVSVR.exe

I hope this helps.

Henfo's picture

Ok,

The error I got in the log is:

"An internal failure occurred. Internal Error: '<0x80040e14>"

Dtrace of the move shows:

1124 15:05:40.660  [11888] (EVMoveArchiveTask) <8104> EV-H {MoveArchiveItem.HandleException} Exception thrown for item F1515667F34EB6DA6EDD7E18C9D1FA41. Exception: System.Runtime.InteropServices.COMException (0x80040306): An internal failure occurred. Internal Error: '<0x80040e14>'.|   at KVS.EnterpriseVault.Interop.IItem.Insert()|   at KVS.EnterpriseVault.MoveArchiveTask.ItemProcessor.CopyItem(QueuedSourceItem sourceitem, IContentManagementAPI4 cmAPImaThread, String& srcSSID, String& destSSID)

11125 15:05:40.660  [11888] (EVMoveArchiveTask) <8104> EV-H {MoveArchiveItem.HandleException} Item with transaction id F1515667F34EB6DA6EDD7E18C9D1FA41 could not be inserted. Error: . Error code: 0x80040e14. Non fatal error.

I can dump the saveset to disk and read message.

/Henfo

Chris Warren's picture

Henfo,

I've seen the "0x80040e14" error be the result of a SQL error on the destination.  Specifically, failure to insert a duplicate row in the destination vault store DB.  Check your event logs on the source for any 13360 errors with "Cannot insert duplicate key row in object 'dbo.Saveset' with unique index 'IX_IdTransaction'."

If this is what is happening, then check the destination Vault Store to confirm the transaction is already there and what archive it is in:

a. Take the transaction and convert it to IdTransaction format:
Example from the reference above - F1515667-F34E-B6DA-6EDD-7E18C9D1FA41

b. Perform the following SELECT query on the destination vault store DB:

SELECT IdTransaction, IdDateTime, ArchivePointIdentity
FROM Saveset
WHERE IdTransaction = 'F1515667-F34E-B6DA-6EDD-7E18C9D1FA41'

c. If this returns a result, record the ArchivePointIdentity

d. Run the following SQL Query against the Vault Store DB to get the ArchiveID of the user:

SELECT ArchivePointID, ArchivePointIdentity
FROM ArchivePoint
WHERE ArchivePointIdentity = 'VALUE FROM b'

e. Record the ArchivePointID.

f. In the Vault Admin Console (VAC), Hold Shift+Ctrl and Right-click Archives.
- This will give a new option to Find Archive or Folder.

g. Paste the ArchivePointID here to find what archive the item is associated with.

It is very rare, close to impossible, to have a duplicate transaction randomly generated. Most commonly, this can be due to there being 2 archives on the destination due to a reattempt to move the archive from scratch and selecting new archive, even though an archive already exists there.  If this item is located in one copy of the user's archive, yet you are now moving to a second instance of the users archive in the same site, the move will fail due to a duplicate record that does not exist in the archive you are moving to.

Is there more than one archive now in the destination site for this user?

I hope this helps.

Chris Warren's picture

Henfo - Are you still having the above error during the move?  Were you able to identify if there was 2 archives for the user on the destination?

Henfo's picture

Hi,

I checked but found no error 13360 in the log, and there is only one archive at destination site for the user.

/Henfo

Henfo's picture

Hi again,

Made a retry on the move and now I get alot of 13360 errors on the destination site, because of the items already moved before?.

No duplicate archive for user at destination.

/Henfo.

Chris Warren's picture

Henfo - The error you provided on the 12th, references that the transaction already exists on the destination vault store.  When you ran the following query, did it return any results on the destination?

SELECT IdTransaction, IdDateTime, ArchivePointIdentity
FROM Saveset
WHERE IdTransaction = 'F1515667-F34E-B6DA-6EDD-7E18C9D1FA41'

- If so, is the item referenced in the destination archive you are moving to, or a different archive?

When Move Archive moves items, it will use the same IdTransaction on the destination.  By default, when MA encounters a duplicate item on the destination archive, it will skip it and move on.  If it encounters a duplicate transaction on the destination Vault Store, yet NOT in the destination archive, it will cause the failure and error out. 

I have seen a situation where a 'legacy' method of move is performed via PST export and import will cause confusion with the MA when determining if the item already exists in the destination.  The reason I believe was because PST exports-imports will alter the archiveddate value while Move Archive will not.  So when Move Archive see's the destination item with an alternate archiveddate, it thinks it is different from the moved item and so will fail.

There is a config settting named 'SkipDuplicates'.  This setting alters how MA reacts when encountering a duplicate IdTransaction on the destination vault store that is not in the destination archive.  With this value at 0, this situation will fail the Move.  If this is set to 1, it will skip the item and move on.  Caution should be used on this option since it does skip moving items it perceives on the destination.

I hope this makes sense.

Henfo's picture

Hi, Chris

I ran the commands and got a result. Item points to the users archive at the new site.  So the item is there?

/Henfo

Chris Warren's picture

Henfo - Appears so. 

If the item exists in the destination archive, then normally that item will be skipped when you move the archive again to the same destination.  The alternative is that MA sees the destination Transaction ID and identifies this as being a different 'item' than the source copy (Some other property of the saveset differs from the source).  Perhaps you can run that query on both the source and destination, save off the results and post them here so we can see if they differ in the Saveset table.

Alternative:
Is the user still associated with the original source archive? 
If so, the alternative is that you can delete the destination archive, cancel the MA job and reattempt the move.

I hope this helps.

Henfo's picture

Chirs, what query should I run?

User is not associated with the original archive anymore.

Source connected to Hitachi hcp with Cifs and destination is connected with storage streamer api to the hcp could that have anything to do with this?.

Chris Warren's picture

Henfo - The Storage type is not related to the way the items are referenced within SQL.  As for the query I referenced, you could run the following on both Vault Store DBs:

SELECT * FROM Saveset WHERE IdTransaction = 'F1515667-F34E-B6DA-6EDD-7E18C9D1FA41'

From there we can look at the different values in the Savese table to determine if these are the same item.

Is this particular error affecting one specific user archive move, or a group of users?

Henfo's picture

Source:
F1515667-F34E-B6DA-6EDD-7E18C9D1FA41 70000000 2010060 37:47.0 -1 0 42065 1966 16 3767 54:00.0 2 24441087 NULL
Destination:
F1515667-F34E-B6DA-6EDD-7E18C9D1FA41 72589208 2010060 37:47.0 -1 0 41 11 49 3764 54:00.0 2 7072 NULL

This is the first user archive that we move to the new site in a new domain. (we have moved some test archives before, no error ).

Chris Warren's picture

Henfo,

Okay - so, even though the transaction is the same, the IdChecksumLow and the ItemSize are different.  Move Archive will believe this to be a duplicate transaction and different item from the source.