Failed to process all expired items in Vault
Updated: 21 May 2010 | 36 comments
This issue has been solved. See solution.
I'm getting a recurring error on one specific EV server for one specific vault. It shows up once a day about 10 minutes before the storage expiry report (event 7085) is logged. I can't find anything in the KB regarding the error or the event id. How do I go about fixing this one?
Error, Event 6605, Storage Delete:
Failed to process all expired items in Vault.
Error reading Vault Entry from Vault Directory [0xc00418e7]
Vault Name: user, name
Vault Id: 178D330BDD62DAB4ABEBCA29C286C01271110000vault1
discussion Filed Under:
Comments
What version and Service Pack
What version and Service Pack of EV aer you using
Have you gone through any upgrades of earlier versions of EV
Have you run Storage Expiry before on your present version of EV
Liam Finn
http://www.plymouthrocklodge47.com
yeah the version is
yeah the version is important, one thing i'd be interested in seeing, can you do a run now against the user?
do you get any 13360's at all?
There is an issue in EV8 where the VaultId doesn't exist in the vault table in the Storage database.
So what i would do is the following
1. Remote to the Enterprise Vault Server and open the Vault Admin Console
2. Hold CTRL-SHIFT and right click "Archives"
3. Select "Find Archives or Folders"
4. In the text box, enter "178D330BDD62DAB4ABEBCA29C286C01271110000vault1"
5. See if that comes back with any errors or whether it lists the user and folder correctly
6. Then in the VAC expand out Archives and then click Exchange Mailboxes
7. Find the problem user and then it will say what vault store the user exists in
8. you should see to the right of the user the Vault Store the user exists in
9. RDP to your SQL server or open a management studio
10. Run the following command and see if you get any hits
USE EVYourVaultStore
SELECT *
FROM Vault
WHERE VaultId = '178D330BDD62DAB4ABEBCA29C286C01271110000vault1'
If it doesn't return any results, then you will need to put on the following (if on EV8 SP3)
http://seer.entsupport.symantec.com/docs/341594.htm
JesusWept, thank you for the
JesusWept, thank you for the excellent instructions. I went step by step and the results are:
1. run-now on the user returned no errors or warnings.
2. find archives listed the user and folder correctly
3. the sql query returned the entry matching the vault id but i do see something suspicious:
the CreateDate and ModifiedDate are both '2007-03-02 07:52:05.930' and the DeletionRunDate is '1753-01-01 00:00:00.000'
Andy Becker | Authorized Symantec Consultant | Trace3 | Symantec Platinum Partner | www.trace3.com
EV8 SP1, upgraded pretty much
EV8 SP1, upgraded pretty much from every version since the KVS days. We've been doing storage expiry for several years now.
Andy Becker | Authorized Symantec Consultant | Trace3 | Symantec Platinum Partner | www.trace3.com
What kind of storage are you
What kind of storage are you using? Centera, NTFS, NAS?
Liam Finn
http://www.plymouthrocklodge47.com
CIFS served up from a NetApp
CIFS served up from a NetApp filer
Andy Becker | Authorized Symantec Consultant | Trace3 | Symantec Platinum Partner | www.trace3.com
nt
nt
As JW can account for we have
As JW can account for we have been through a big expiry issue. Follow his instructions and lets see the outcome. Also get a DTrace when you can so we can see whats happening
Liam Finn
http://www.plymouthrocklodge47.com
ok totally weird, sorry , get
ok totally weird, sorry , get a dtrace of StorageDelete like liam suggested.
The deletion date being 1753 just means that its never been run against the archive, but the column does need a date/time stamp so it creates the earliest one you can have
will look more in to this for you, just as a matter of interest, can you find that user and folder, give yourself access to them, open the mailbox in outlook or OWA and archive something in that folder and try again
our EV service account
our EV service account already had access so i used that to open the user's mailbox via outlook 2003 and manually archived an email from her sent items. not sure what it is that you wanted me to try again but nothing appears to have changed as far as the sql query results and our expiry runs at 6am so i'll keep an eye out tomorrow for that.
what's the best way to setup the dtrace to run at 6am tomorrow to catch the storage delete events? scheduled task?
Andy Becker | Authorized Symantec Consultant | Trace3 | Symantec Platinum Partner | www.trace3.com
You can put a trigger on the
You can put a trigger on the DTrace to look for the event ID that way it will only start logging when the event happens
Liam Finn
http://www.plymouthrocklodge47.com
Here is the KB on how to
Here is the KB on how to setup a trigger in a DTrace
http://seer.entsupport.symantec.com/docs/284344.htm
Liam Finn
http://www.plymouthrocklodge47.com
Sorry, the reason i wanted
Sorry, the reason i wanted you to manually archive an item is so that it updates the ArchiveFolder etc and update the modified dates because from what you said, that archive hasnt been touched in years, was just hoping it may kick start something, also i'm pretty sure wayne humphrey either in a blog or an article describes how to put evpm to run at a certain time
thanks for all the great
thanks for all the great advice. i'll get that dtrace setup and let you know how it goes.
Andy Becker | Authorized Symantec Consultant | Trace3 | Symantec Platinum Partner | www.trace3.com
scanner, JW, i've attached
scanner, JW,
i've attached the dtrace to my first post. thanks!
Andy Becker | Authorized Symantec Consultant | Trace3 | Symantec Platinum Partner | www.trace3.com
Hey dude go back to my
Hey dude
go back to my original post and swap the 17D GUID for this one, 113C324F1457E384287BD251361DE86AD1110000vault1
Thats the folder thats failing
when i paste it in the VAC i
when i paste it in the VAC i get no results.
the sql query returns:
113C324F1457E384287BD251361DE86AD1110000vault1,99348,178,1,42,0,0,0,2008-03-21 05:50:01.707,NULL,1753-01-01 00:00:00.000
Andy Becker | Authorized Symantec Consultant | Trace3 | Symantec Platinum Partner | www.trace3.com
whaaaaaaaaaat?! it can't be
whaaaaaaaaaat?!
it can't be found in the vac?
great, so it's an orphaned folder
dont suppose you ever restored the enterprisevaultdirectory but not the Vault stores at any point in time?
give me a few minutes to think what can be done
thankfully *knock on wood* we
thankfully *knock on wood* we have not had to do that.
Andy Becker | Authorized Symantec Consultant | Trace3 | Symantec Platinum Partner | www.trace3.com
do you think i should open a
do you think i should open a case for this issue?
Andy Becker | Authorized Symantec Consultant | Trace3 | Symantec Platinum Partner | www.trace3.com
Hi Andrew, I would recommend
Hi Andrew,
I would recommend getting a case going for this issue. Sometimes a hands on look can find and resolve the issue faster than what can be done via the forum.
Do please report back your findings! :)
cheers,
Tony Sterling
Ok Tony. Will do.
Ok Tony. Will do.
Andy Becker | Authorized Symantec Consultant | Trace3 | Symantec Platinum Partner | www.trace3.com
they will tell you to put on
they will tell you to put on the hotfix that i mentioned above, then they will ask you to
1. Disable the user
2. Export the archive to a pst
3. Delete the archive
4. Re-enable the user
5. Reimport it and your problems will be resolved
just to update you on the
just to update you on the progress:
02/12/2010: opened case via MySupport portal with all pertinent details and a reference to this post.
02/15/2010: symantec engineer emailed that he reviewed all the details and asked if we use CA/DA. I replied no.
02/17/2010: symantec engineer emailed asking if EV backup mode overlaps storage expiry schedule. I replied no.
Andy Becker | Authorized Symantec Consultant | Trace3 | Symantec Platinum Partner | www.trace3.com
How many items are you
How many items are you expiring?
Liam Finn
http://www.plymouthrocklodge47.com
From the Storage Expiry
I'm not sure if this answers your question but from the Storage Expiry Report on the particular vault store where the problem archive sits:
Number of vaults processed: 2116
Total number of expired items deleted: 8577
Andy Becker | Authorized Symantec Consultant | Trace3 | Symantec Platinum Partner | www.trace3.com
This is not many items. If
This is not many items.
If there were 4 million then i would understand it giving this warning as we have seen such warnings recently when we deleted 18 million items so we needed up running expiry 4 times until it ran clean but don't think this is your issue
Liam Finn
http://www.plymouthrocklodge47.com
It runs every day from
It runs every day from 6am-10am. Over the past month we had anywhere from 4,000 to 20,000 expired items deleted per run on that store.
Andy Becker | Authorized Symantec Consultant | Trace3 | Symantec Platinum Partner | www.trace3.com
again small number of items
again small number of items
Liam Finn
http://www.plymouthrocklodge47.com
could you do a quick count on
could you do a quick count on the archivepoint table?
i'm wondering if when it hits this user it kills the entire Storage Expiry thread, because i seem to remember having that issue with liam some time ago
run the following query
USE EVyourvaultstore
SELECT COUNT(ArchivePointId) FROM ArchivePoint
Does it return 2116 results?
Yeah our issue was missing
Yeah our issue was missing folders info in the SQL database
Liam Finn
http://www.plymouthrocklodge47.com
JW: it returned 2116
JW: it returned 2116
Andy Becker | Authorized Symantec Consultant | Trace3 | Symantec Platinum Partner | www.trace3.com
I'm happy to report that the
I'm happy to report that the (escalated) Symantec Engineer was able to help resolve the issue. He pinpointed the error to one particular saveset in the archive that was missing it's vault entry. We found the cab file it was in, extracted the item for backup purposes and then deleted the corresponding entry from the saveset tables and the vault store. We manually ran the expiry and it completed successfully.
Andy Becker | Authorized Symantec Consultant | Trace3 | Symantec Platinum Partner | www.trace3.com
he changed the refcount in
he changed the refcount in the collections table as well right?
he did not make any changes
he did not make any changes to the collections table. also, the dvs file itself is still inside the cab. he said it would be deleted automatically once an expiry threshold was met and the cab was rebuilt.
Andy Becker | Authorized Symantec Consultant | Trace3 | Symantec Platinum Partner | www.trace3.com
Per Symantec: "as far as
Per Symantec:
"as far as changing the reference count for the .cab file, as this is only one item it will not make much of a difference on when the cab is rebuilt. If you want to change this I would recommend waiting until the upgrade to SP4, and then run EVSVR."
Andy Becker | Authorized Symantec Consultant | Trace3 | Symantec Platinum Partner | www.trace3.com
Would you like to reply?
Login or Register to post your comment.