SEPM v11.0.2000 MR2 broken links in 32GB database
I've just started a new job (2 months ago) and my employer uses Symantec Endpoint Protection 11.0 MR2. Its not a product I know a lot about, however I've been learning pretty fast. Fortunately this is a small organisation (about 100 clients). Unfortunately I have discovered a major issue with the installation of SEPM and have not been able to discover a satisfactory solution yet.
The management console and embedded database is installed on one of our 2 domain controllers (yeah, I know, bad idea, just remember I have inherited this!). The server is running Windows Server 2003 SP2 and all latest MS patches. The embedded database is 32GB in size and growing. I have only 4GB free disk space (so I doubt I will be able to 'dbunload' it to shrink it).
When I run 'dbvalidator.bat' it reports a total of four (4) errors comprising:
(a) Link is broken for [2] target ids:
---- 1 x LuPolicy object
---- 1 x IdsPolicy object
(b) Link is broken for [2] physical file ids:
---- 2 x ExPhysicalFileStream objects
I suspect that these errors are preventing the normal process of file deletion within the embedded database from working correctly, because when I finally managed to run the DBISQLC.EXE tool and login (that was hard, because nobody here remembered the original database install password, but luckily I guessed it!) ... when I ran DBISQLC.EXE and ran "SELECT count(*) AS mycount FROM binary_file WHERE deleted=1" and then repeated this for "deleted=0" and then again without the where clause, my results were:
Total records in BINARY_FILE: 1,659
Deleted records in BINARY_FILE: 1,608
Current records in BINARY_FILE: 51
In other words, 97% of that file is deleted junk. So that explains why the DB file is 32GB. But how to shrink it? If I understand correctly, 'dbunload' will only shrink the database when there a physically deleted records resulting in free space that can be recovered. But these 97% of records are only flagged as logically deleted, not physically deleted, so 'dbunload' will not touch them.
I tried another approach: I have installed SEPM also on my laptop and configured it as a new site being replicated from the main server. Replication took over 3 hours (!) but after it was done, the database on my laptop was just 1GB. Great! Not really. Now I have learned that if I delete SEPM off the main server, then try to reinstall it and replicate it back from the laptop, it won't work (something to do with "if you delete the 'main server' you can no longer replicate"). So I can't get any further with that idea.
Lastly, on my laptop with the nice 1GB database, I have run 'dbvalidator.bat' on it and, yes, the 4 errors that are in the original server are also in the replicated copy. So whilst it didn't bring across unnecessary "DELETED=1" records, it did copy over the corrupted records. Damn.
What I would really like to try is to run 'ludbfix' on my laptop embedded database, to see whether it can really fix the dud records, but after having talked this through with Symantec support here in Australia, they tell me that 'ludbfix' has been withdrawn and they won't help me further with that idea.
So what can I do now? I'd really like some advice. I don't feel capable of doing a complete reinstall, because I just don't know enough about the policies that have been configured on the existing server. I don't want to risk losing functionality, or worse, compromising the organisation's security through my ignorance.
Comments
It Is an issue of SEP 11.0.2
It is an issue of SEP 11.0.2, i.e, regarding the database file size, it keeps on growing, it is being solved in the later versions of the product, i would suggest you to take a backup of your database, and then go for the reinstallation and then restoring the database. Did you instal the latest version of SEP in your laptop?
Sapta wrote:> I would
Sapta wrote:
> I would suggest you to take a backup of your database,
> and then go for the reinstallation and then restoring the database.
> Did you install the latest version of SEP in your laptop?
Thanks for the suggestions. I don't have enough disk space on the server to take a backup. So from that point of view, I am at significant risk.
No, I didn't install the latest version of SEP on my laptop, I had to install MR2 because at the time I wanted to replicate the database over from the server to the laptop.
But that's okay, at least I have a copy of the database on the laptop that I can now experiment with. I was thinking of trying something like the following on the laptop:
-- break the replication link with the server
-- manually delete all the old files from the Binary File table
-- run 'dbvalidator.bat' to see whether manually deleting the files has introduced new errors into the database
-- if it does, I am no worse off, because the live server is still running (barely!)
-- if it does not, then I can consider:
---- manually deleting the old files from the Binary File table on the server
---- running 'dbunload' on the server to recover the lost disk space
If I can do that successfully, my immediate disk space problem is solved. I can then try the following on the server:
-- take a backup
-- export all existing policies
-- make careful notes of all settings on the server
-- upgrade from MR2 to MR4
-- if the upgrade works:
---- run 'dbvalidator.bat' to see if the upgrade has fixed the broken links problem
---- take a backup
---- upgrade from MR4 to MR4 MP2
---- if the upgrade works:
------ run 'dbvalidator.bat' to see if the upgrade has fixed the broken links problem
-- the the upgrade from MR2 to MR4 fails:
---- uninstall MR2
---- install MR4 MP2 as a clean install
---- set up groups as per previous install
---- import policies from previous install
---- configure all other settings as per previous install
---- distribute updated Sylink.xml file to clients
---- manually move clients to groups as they report in
Anyway, something like that ... I'm trying to be very careful so as not to risk losing settings/policies by being too hasty in trying to fix things.
The version which u r using
The version which u r using is pretty old and has some bugs in it u can go ahead with 11.0 MR4MP1/2.
U can try this as well
http://entsearch.symantec.com/search?p=R&srid=S10%2d1&lbc=symantec&w=drive%20space&url=http%3a%2f%2fservice1%2esymantec%2ecom%2fsupport%2fent%2dsecurity%2ensf%2f854fa02b4f5013678825731a007d06af%2feb82f51f1e184c7e88257394005fa76b%3fOpenDocument&rk=1&uid=311476324&sid=2&ts=c2&rsc=9AZnTaYiaV5KCPD-&ed=edn%5f54619&method=and&ed=edn%5f54619&af=dtype%3aarticle%20%20cat2%5fendpointsecurityincformersygateproducts%3a54619%20%20cat1%3aendpointsecurityincformersygateproducts&isort=score
Regards'
Ajit Jha
Technical Consultant
STS
Further information
I spent some considerable time yesterday experimenting with the database on the live server (risky, yes, but unfortunately necessary). I roughly followed this process:
-- having learned that if I replicated to my laptop but then deleted the server version I would lose the ability to have replication partners in the future, I abandoned that strategy, so I deleted the replication links from server and laptop and then uninstalled SEPM from my laptop
-- I was able to attach a USB drive to the server. I stopped both the Symantec Endpoint Protection Manager service and the Symantec Embedded Database service, then took a copy of the \db directory which has the 32GB sem5.db file and the log files in it and placed these on the temporary USB drive device (this took about 40 minutes to copy) ... so I now have some kind of limited backup capability in place
-- after taking the backup, I restarted the Symantec Embedded Database service and launched into a DBISQLC.EXE session
-- ran "SELECT count(*) AS MyCount FROM binary_file WHERE deleted=1" and confirmed the existence of roughly 1650 deleted records
-- ran 'dbvalidator.bat' and confirmed the existence of 4 link errors in the database
-- ran "SELECT id, time_stamp FROM binary_file WHERE deleted=1 ORDER BY time_stamp" to get a list of the IDs and the dates of the deleted records, sorted by date
-- ran "DELETE FROM binary_file WHERE id='<id>'" for the oldest time-stamped deleted record ... and confirmed that 1 record was physically deleted
-- ran 'dbvalidator.bat' again and found that there were no additional link errors evident
-- ran "DELETE FROM binary_file WHERE deleted=1 AND time_stamp < '<date>'" (where date is a number) ... I chose a date value that picked out about 50 of the oldest records in the database ... and confirmed that these 50 records were physically deleted
-- ran 'dbvalidator.bat' again and found that there were no additional link errors evident
-- restarted the Symantec Endpoint Protection Manager service
SO: it seems on the face of it that, in some limited sense, deleting these DELETED=1 records from the BINARY_FILE table in the embedded database doesn't do any visible, immediate or apparent harm. However this is a very tentative view and I am most reluctant to suggest that this really, genuinely is the case.
I decided to stop at that point and leave the database running overnight, so that I could see a Live Update (LUALL.EXE) session run, which happened this morning. It worked successfully.
My next plan is to take another backup of the raw database files (as I still do not have enough disk space to run a backup via the usual method) and then trash all the DELETED=1 files from the BINARY_FILE table in the database. After I have done that I will try a 'dbunload' process to see if I can shrink the database down to around 1GB or so. Of course, I will continue running 'dbvalidator' over the database to see if anything further gets broken as a result.
If I can get away with this and shrink the database, then even if it has errors in it, at least the database is small and I am no longer going to have to worry about the disk going full. I can then plan for a migration or full reinstallation at my leisure.
Not so fast
Well I ran: DELETE FROM binary_file WHERE deleted=1
Got as far as running 'dbunload' but it got part way through the process and hit a snag when it got part way through the BINARY_FILE table. Either those original broken links have tripped me up, or alternatively, there is some other kind of data corruption in the table.
I am now about to try to run the 'dbunload' on a fresh copy of SEPM installed on my laptop. I will stop the Symantec services, copy across the backup of my database, then restart the Embedded Database service.
If 'dbunload' fails to work then, its game over ... I will have to uninstall and reinstall SEPM on the server.
Double damn ...
I've got a copy of the
I've got a copy of the LUDBFIX tool. It's pretty old, from Feb. '09. I'll be happy to post it and give you the link for the same if you still need it.
Abhishek Pradhan, PMP, MCT
Consultant | Microsoft Corp.
Blog: http://blog.abhishekpradhan.net | SIG Lead - Pune IT Pro (Microsoft Pune User Group) | http://www.puneusergroup.org
Yes please!
Thanks very much for the offer. Yes, if you wouldn't mind me having a copy, I'd definitely love to give it a try. Given that I have now decided that I need to reinstall completely, I can't really make it any worse than it already is, even if it doesn't work! If you prefer to email it to me: stephen.frost@frostbyte.com.au.
Still searching on 5
Still searching on 5 terabytes of storage at home. Will PM you when I get the Zipped file, or better yet will mail it to you from Sydney next week if it's okay. I'll be in Australia for a week. :)
Bondi, here ah come babee..... ;)
Abhishek Pradhan, PMP, MCT
Consultant | Microsoft Corp.
Blog: http://blog.abhishekpradhan.net | SIG Lead - Pune IT Pro (Microsoft Pune User Group) | http://www.puneusergroup.org
Yes, that would be fine!
Thanks, yes, "next week" (now "this week") would be fine, no hurry at all. I spent a couple of hours trawling through the server last Friday and was able to free up about 10GB of extra free disk space, so my immediate problem of the disk possibly going full has been averted for a few months.
Have fun at Bondi! Although it is winter here (I'm down in Melbourne, which is quite a bit colder than Sydney).
Haha. Unfortunately my trip
Haha. Unfortunately my trip to Sydney (work) was cut short due to more work back home. Came back yesterday.
I've got the LUDBFIX zip file. I'll upload it on Skydrive and give you access in a days time.
Abhishek Pradhan, PMP, MCT
Consultant | Microsoft Corp.
Blog: http://blog.abhishekpradhan.net | SIG Lead - Pune IT Pro (Microsoft Pune User Group) | http://www.puneusergroup.org
Running out of time
Hi Abhishek,
I'm running short of time here. Any chance you could email it to me? (stephen.frost@frostbyte.com.au or sfrost@grv.org.au)
If I cannot get it by Thursday night, I think on Friday I will have to uninstall/reinstall SEPM, and I was hoping that 'ludbfix' would fix the data corruption and therefore help me compress the database instead of uninstalling the application.
Fingers crossed here!
Steve
Fingers still crossed ...
... and hoping against hope for a copy of 'ludbfix' ...
deleted
deleted
Hey Stephen,
Hey Stephen,
Sorry about the delay, but It's funny that you didn't get the Skydrive email.
I've sent you the link again in email from where you can download the LUDBFIX tools set.
Run it and let us know if it helps.
Abhishek Pradhan, PMP, MCT
Consultant | Microsoft Corp.
Blog: http://blog.abhishekpradhan.net | SIG Lead - Pune IT Pro (Microsoft Pune User Group) | http://www.puneusergroup.org
Got it this time
Thanks Abishek, got your email this time. Downloaded 'ludbfix' and ran it, however it seemed to have some kind of version mis-match with my edition of SEPM.
What I did discover was that the DBVALIDATOR tool essentially runs 'ludbfix' with a parameter of 'XMLValidator'. By comparing this with the files you sent me, I surmised that 'ludbfix' is still installed on my MR2 machine ... so I copied the DBVALIDATOR batch file and made a new LUDBFIX batch file ... and edited it to replace "XMLValidator" with "Main". When I ran this it did indeed run 'ludbfix' and it produced an LUDBFIX.LOG file in the \Log directory. Unfortunately for me, this also did not manage to fix my database corruption issues.
But who knows, this idea may help someone else some time (if they are desperate enough!).
FIXED!!!
I am pleased to report a solution has been found to the problem of the massive database in SEPM v11 MR2. You may recall that I had two fundamental issues I was trying to overcome:
1) massive 32GB embedded database file; and
2) database corruption which was preventing 'dbunload' from compressing the database
The approximate process I followed was:
1) used the dbiSQLC tool to delete all records from BINARY_FILE where the deleted flag was set to 1
2) upgraded from MR2 to MR4
3) upgraded from MR4 to MR4 MP2
4) ran a repair install of MR4 MP2 (just to make sure)
5) ran DBVALIDATOR (which confirmed that the upgrade had fixed the internal database errors -- so no more database corruption)
6) ran DBUNLOAD (which took more than 1 hour to complete)
7) distributed a new Sylink.XML file to all my clients (automatically to desktops via a script, manually to my servers)
After all this, the new database size is 3.6GB which is an almost 90% reduction. DBVALIDATOR comes up with a clean bill of health.
Good job!
Good job!
There is an online portal, save yourself the long hold times. Create ticket online, then call in with ticket # in hand :-) http://mysupport.symantec.com "We backup data to restore, we don't backup data just to back it up."
Further info
One more bit of additional information. After initially succeeding in packing the database down from >32GB to <4GB, I let things stabilise for a week or so. Yesterday I also deleted old copies of installation packages (v11.0.1000, v11.0.2000 and v11.0.3000) and then re-ran the DBUNLOAD process. After doing this, the database packed down to less than 1GB (720MB to be precise). I've been able to set up regular weekly database backups and they are running happily. So all is well in SEPM land at my workplace!
Would you like to reply?
Login or Register to post your comment.