Video Screencast Help
Search Video Help Close Back
to help
New in the Rewards Catalog: Vouchers for "Symantec Technical Specialist" and "Symantec Certified Specialist" exams.

Failed to process all expired items in Vault

Updated: 21 May 2010 | 36 comments
AndrewB's picture
0 0 Votes
Login to vote
This issue has been solved. See solution.

I'm getting a recurring error on one specific EV server for one specific vault. It shows up once a day about 10 minutes before the storage expiry report (event 7085) is logged. I can't find anything in the KB regarding the error or the event id. How do I go about fixing this one?

Error, Event 6605, Storage Delete:

Failed to process all expired items in Vault.
Error reading Vault Entry from Vault Directory [0xc00418e7]
Vault Name: user, name
Vault Id: 178D330BDD62DAB4ABEBCA29C286C01271110000vault1

discussion Filed Under:

Comments

Scanner001's picture
09
Feb
2010
0 Votes 0
Login to vote

What version and Service Pack

What version and Service Pack of EV aer you using

Have you gone through any upgrades of earlier versions of EV

Have you run Storage Expiry before on your present version of EV

JesusWept2's picture
09
Feb
2010
0 Votes 0
Login to vote

yeah the version is

yeah the version is important, one thing i'd be interested in seeing, can you do a run now against the user?
do you get any 13360's at all?

There is an issue in EV8 where the VaultId doesn't exist in the vault table in the Storage database.

So what i would do is the following

1. Remote to the Enterprise Vault Server and open the Vault Admin Console
2. Hold CTRL-SHIFT and right click "Archives"
3. Select "Find Archives or Folders"
4. In the text box, enter "178D330BDD62DAB4ABEBCA29C286C01271110000vault1"
5. See if that comes back with any errors or whether it lists the user and folder correctly

6. Then in the VAC expand out Archives and then click Exchange Mailboxes
7. Find the problem user and then it will say what vault store the user exists in
8. you should see to the right of the user the Vault Store the user exists in

9. RDP to your SQL server or open a management studio
10. Run the following command and see if you get any hits

USE EVYourVaultStore
SELECT *
FROM Vault
WHERE VaultId = '178D330BDD62DAB4ABEBCA29C286C01271110000vault1'

If it doesn't return any results, then you will need to put on the following (if on EV8 SP3)
http://seer.entsupport.symantec.com/docs/341594.htm

AndrewB's picture
09
Feb
2010
0 Votes 0
Login to vote

JesusWept, thank you for the

JesusWept, thank you for the excellent instructions. I went step by step and the results are:
1. run-now on the user returned no errors or warnings.
2. find archives listed the user and folder correctly
3. the sql query returned the entry matching the vault id but i do see something suspicious:
the CreateDate and ModifiedDate are both '2007-03-02 07:52:05.930' and the DeletionRunDate is '1753-01-01 00:00:00.000'

Andy Becker | Authorized Symantec Consultant | Trace3 | Symantec Platinum Partner | www.trace3.com

AndrewB's picture
09
Feb
2010
0 Votes 0
Login to vote

EV8 SP1, upgraded pretty much

EV8 SP1, upgraded pretty much from every version since the KVS days. We've been doing storage expiry for several years now.

Andy Becker | Authorized Symantec Consultant | Trace3 | Symantec Platinum Partner | www.trace3.com

Scanner001's picture
09
Feb
2010
0 Votes 0
Login to vote

What kind of storage are you

What kind of storage are you using? Centera, NTFS, NAS?

AndrewB's picture
09
Feb
2010
0 Votes 0
Login to vote

CIFS served up from a NetApp

CIFS served up from a NetApp filer

Andy Becker | Authorized Symantec Consultant | Trace3 | Symantec Platinum Partner | www.trace3.com

JesusWept2's picture
09
Feb
2010
0 Votes 0
Login to vote

nt

nt

Scanner001's picture
09
Feb
2010
0 Votes 0
Login to vote

As JW can account for we have

As JW can account for we have been through a big expiry issue. Follow his instructions and lets see the outcome. Also get a DTrace when you can so we can see whats happening

JesusWept2's picture
09
Feb
2010
0 Votes 0
Login to vote

ok totally weird, sorry , get

ok totally weird, sorry , get a dtrace of StorageDelete like liam suggested.
The deletion date being 1753 just means that its never been run against the archive, but the column does need a date/time stamp so it creates the earliest one you can have

will look more in to this for you, just as a matter of interest, can you find that user and folder, give yourself access to them, open the mailbox in outlook or OWA and archive something in that folder and try again

AndrewB's picture
09
Feb
2010
0 Votes 0
Login to vote

our EV service account

our EV service account already had access so i used that to open the user's mailbox via outlook 2003 and manually archived an email from her sent items. not sure what it is that you wanted me to try again but nothing appears to have changed as far as the sql query results and our expiry runs at 6am so i'll keep an eye out tomorrow for that.

what's the best way to setup the dtrace to run at 6am tomorrow to catch the storage delete events? scheduled task?

Andy Becker | Authorized Symantec Consultant | Trace3 | Symantec Platinum Partner | www.trace3.com

Scanner001's picture
09
Feb
2010
0 Votes 0
Login to vote

You can put a trigger on the

You can put a trigger on the DTrace to look for the event ID that way it will only start logging when the event happens

Scanner001's picture
09
Feb
2010
0 Votes 0
Login to vote
JesusWept2's picture
09
Feb
2010
0 Votes 0
Login to vote

Sorry, the reason i wanted

Sorry, the reason i wanted you to manually archive an item is so that it updates the ArchiveFolder etc and update the modified dates because from what you said, that archive hasnt been touched in years, was just hoping it may kick start something, also i'm pretty sure wayne humphrey either in a blog or an article describes how to put evpm to run at a certain time

AndrewB's picture
09
Feb
2010
0 Votes 0
Login to vote

thanks for all the great

thanks for all the great advice. i'll get that dtrace setup and let you know how it goes.

Andy Becker | Authorized Symantec Consultant | Trace3 | Symantec Platinum Partner | www.trace3.com

AndrewB's picture
10
Feb
2010
0 Votes 0
Login to vote

scanner, JW, i've attached

scanner, JW,
i've attached the dtrace to my first post. thanks!

Andy Becker | Authorized Symantec Consultant | Trace3 | Symantec Platinum Partner | www.trace3.com

JesusWept2's picture
10
Feb
2010
0 Votes 0
Login to vote

Hey dude go back to my

Hey dude
go back to my original post and swap the 17D GUID for this one, 113C324F1457E384287BD251361DE86AD1110000vault1
Thats the folder thats failing

AndrewB's picture
10
Feb
2010
0 Votes 0
Login to vote

when i paste it in the VAC i

when i paste it in the VAC i get no results.

the sql query returns:
113C324F1457E384287BD251361DE86AD1110000vault1,99348,178,1,42,0,0,0,2008-03-21 05:50:01.707,NULL,1753-01-01 00:00:00.000

Andy Becker | Authorized Symantec Consultant | Trace3 | Symantec Platinum Partner | www.trace3.com

JesusWept2's picture
10
Feb
2010
0 Votes 0
Login to vote

whaaaaaaaaaat?! it can't be

whaaaaaaaaaat?!
it can't be found in the vac?
great, so it's an orphaned folder

dont suppose you ever restored the enterprisevaultdirectory but not the Vault stores at any point in time?

give me a few minutes to think what can be done

AndrewB's picture
10
Feb
2010
0 Votes 0
Login to vote

thankfully *knock on wood* we

thankfully *knock on wood* we have not had to do that.

Andy Becker | Authorized Symantec Consultant | Trace3 | Symantec Platinum Partner | www.trace3.com

AndrewB's picture
12
Feb
2010
0 Votes 0
Login to vote

do you think i should open a

do you think i should open a case for this issue?

Andy Becker | Authorized Symantec Consultant | Trace3 | Symantec Platinum Partner | www.trace3.com

TonySterling's picture
12
Feb
2010
0 Votes 0
Login to vote

Hi Andrew, I would recommend

Hi Andrew,

I would recommend getting a case going for this issue.  Sometimes a hands on look can find and resolve the issue faster than what can be done via the forum.

Do please report back your findings!  :)

cheers,

Tony Sterling

AndrewB's picture
12
Feb
2010
0 Votes 0
Login to vote

Ok Tony. Will do.

Ok Tony. Will do.

Andy Becker | Authorized Symantec Consultant | Trace3 | Symantec Platinum Partner | www.trace3.com

JesusWept2's picture
12
Feb
2010
0 Votes 0
Login to vote

they will tell you to put on

they will tell you to put on the hotfix that i mentioned above, then they will ask you to

1. Disable the user
2. Export the archive to a pst
3. Delete the archive
4. Re-enable the user
5. Reimport it and your problems will be resolved

AndrewB's picture
17
Feb
2010
0 Votes 0
Login to vote

just to update you on the

just to update you on the progress:

02/12/2010: opened case via MySupport portal with all pertinent details and a reference to this post.
02/15/2010: symantec engineer emailed that he reviewed all the details and asked if we use CA/DA. I replied no.
02/17/2010: symantec engineer emailed asking if EV backup mode overlaps storage expiry schedule. I replied no.

Andy Becker | Authorized Symantec Consultant | Trace3 | Symantec Platinum Partner | www.trace3.com

Scanner001's picture
17
Feb
2010
0 Votes 0
Login to vote

 How many items are you

 How many items are you expiring?

AndrewB's picture
17
Feb
2010
0 Votes 0
Login to vote

From the Storage Expiry

I'm not sure if this answers your question but from the Storage Expiry Report on the particular vault store where the problem archive sits:
Number of vaults processed: 2116
Total number of expired items deleted: 8577

Andy Becker | Authorized Symantec Consultant | Trace3 | Symantec Platinum Partner | www.trace3.com

Scanner001's picture
17
Feb
2010
0 Votes 0
Login to vote

 This is not many items. If

 This is not many items.

If there were 4 million then i would understand it giving this warning as we have seen such warnings recently when we deleted 18 million items so we needed up running expiry 4 times until it ran clean but don't think this is your issue 

AndrewB's picture
17
Feb
2010
0 Votes 0
Login to vote

It runs every day from

It runs every day from 6am-10am. Over the past month we had anywhere from 4,000 to 20,000 expired items deleted per run on that store.

Andy Becker | Authorized Symantec Consultant | Trace3 | Symantec Platinum Partner | www.trace3.com

Scanner001's picture
17
Feb
2010
0 Votes 0
Login to vote

 again small number of items

 again small number of items

JesusWept2's picture
17
Feb
2010
0 Votes 0
Login to vote

could you do a quick count on

could you do a quick count on the archivepoint table?
i'm wondering if when it hits this user it kills the entire Storage Expiry thread, because i seem to remember having that issue with liam some time ago

run the following query

USE EVyourvaultstore
SELECT COUNT(ArchivePointId) FROM ArchivePoint

Does it return 2116 results?

Scanner001's picture
17
Feb
2010
0 Votes 0
Login to vote

 Yeah our issue was missing

 Yeah our issue was missing folders info in the SQL database

AndrewB's picture
17
Feb
2010
0 Votes 0
Login to vote

JW: it returned 2116

JW: it returned 2116

Andy Becker | Authorized Symantec Consultant | Trace3 | Symantec Platinum Partner | www.trace3.com

AndrewB's picture
23
Feb
2010
0 Votes 0
Login to vote

I'm happy to report that the

I'm happy to report that the (escalated) Symantec Engineer was able to help resolve the issue. He pinpointed the error to one particular saveset in the archive that was missing it's vault entry. We found the cab file it was in, extracted the item for backup purposes and then deleted the corresponding entry from the saveset tables and the vault store. We manually ran the expiry and it completed successfully.

Andy Becker | Authorized Symantec Consultant | Trace3 | Symantec Platinum Partner | www.trace3.com

JesusWept2's picture
23
Feb
2010
0 Votes 0
Login to vote

he changed the refcount in

he changed the refcount in the collections table as well right?

AndrewB's picture
23
Feb
2010
0 Votes 0
Login to vote

he did not make any changes

he did not make any changes to the collections table. also, the dvs file itself is still inside the cab. he said it would be deleted automatically once an expiry threshold was met and the cab was rebuilt.

Andy Becker | Authorized Symantec Consultant | Trace3 | Symantec Platinum Partner | www.trace3.com

AndrewB's picture
25
Feb
2010
0 Votes 0
Login to vote

Per Symantec: "as far as

Per Symantec:
"as far as changing the reference count for the .cab file, as this is only one item it will not make much of a difference on when the cab is rebuilt. If you want to change this I would recommend waiting until the upgrade to SP4, and then run EVSVR."

Andy Becker | Authorized Symantec Consultant | Trace3 | Symantec Platinum Partner | www.trace3.com