SEPM Failing On A Daily Basis.
Hello All,
I need some advice on how to troubleshoot a problem that has started to occur recently with our SEPM installation. We are currently using SEPM 11.0.2000.1567, which I am sure many will note is an old old version. We did look at upgrading to the latest MR some time ago, but I believe it was to MR3 which from memory required more work then it was worth (recollection of having to upgrade to MR2 first the MR3 with some database related procedure needing following too), as we where having no problems (until the fabled New Year virus definitions bug that occurred) it did not seem that the effort required would return a tangible benefit.
Alas now we have a problem, which before I upgrade to MR5 (I believe it is the current version) I need to identify the cause of the problem and validate that MR5 will resolve the issue.
The issue is as follows, on a daily basis we open up the SEPM console to perform various management tasks, after entering the username and password to connect to the SEPM console, the progress bar is shown and it states it is loading, but it never proceeds further then a third of the way, no matter how long we leave it. To resolve this the Symantec Endpoint Protection Manager service needs restarted and also the Symantec Embedded Database service needs restarted. However when restarting the Symantec Embedded Database service it does not stop therefore the dbsrv9.exe process needs killed to force the restart of the embedded database service.
As a test I left the SEPM console for a week in the hung state to determine if it effects client computers, it does they fail to retrive the daily virus definition updates so it is not just the console application that is failing when this issue occurs, it appears to be the whole SEP environment.
My question is, how do I trace the cause of this, I know the SEP environment creates log files for various functions but reading them (as they appear to be in Java/Tomcat speak) is something that is above my level of knowledge? Also I dont know which log file of the many thier is I should be looking at (I looked at all of them and some do have errors reported) Can anybody help?
Also we do run the SEPM console on the same server we have WSUS installed on, however the WSUS management website is using port 8531, the only WSUS related system running on port 80 is the virtual SelfUpdate directory (which has to be running on port 80 for end clients to be able to update to a new version of the Windows Update Agent when WSUS has a new version to deploy). As this uses a specific URL path (http://windowsupdateserver.dns.name/SelfUpdate) I dont beleive running this virtual directory in the same website as the SEPM console has any effect?
Thanks.
Gary Hall.
Comments
Try by repairing the SEPM
Try by repairing the SEPM from add/remove programs..
The Symantec Endpoint Protection Manager hangs at the logon screen
Please don't forget to mark your thread solved with whatever answer helped you : ) Thanks & Regards Aravind
hi
this should be related to the version, its not that stable
if SEPM service or DB service fails your client wont get any update they all say sever offline
you can check this doc before we proceed further.
The Symantec Endpoint Protection Manager hangs at the logon screen
http://service1.symantec.com/support/ent-security.nsf/docid/2009082601032448?Open&seg=ent
Please don't forget to mark your thread solved with whatever answer helped you : ) Rafeeq
Reboot the machine and then
Reboot the machine and then repair the SEPM.
The scm.servero.log may be handy
It is under c:/Program files/symantec/SEPM/Tomcat/logs
Prachand Kumar MCSE-2003 Symantec Technical Specialist (SCTS)
Wow what a quick response,
Wow what a quick response, thanks for the information!
Going to proceed with a repair of the installation. Will let you know fo the outcome (and mark the thread as resolved if it does indeed resolve it).
For @symc_endpoint
Hello,
Was asked via Twitter (wow a first here to, a twitter communication to moi!) to upload the scm server logs as they could point to the cause of the problem that may not be repaired using the repair method (which I will still try). So here you go symc_endpoint@twitter.com!
Oh and thanks for Prachand who also pointed out these logs might help in determining the cause of the error.
Thanks.
Gary Hall.
hi
how did you install IIS?
The log says
IISCacheTask connect to secars failed: SERVICE NOT AVAILABLE
so basically IIS is not available.
do a iisrest should fix the issue.
http://service1.symantec.com/support/ent-security.nsf/854fa02b4f5013678825731a007d06af/4bd90f7f0f5b95c18825738c00660e10?OpenDocument
Please don't forget to mark your thread solved with whatever answer helped you : ) Rafeeq
Thanks for uploading the logs
Thanks for uploading the logs Gary.
Looks like your SEPM is getting connection refused when it tries to open a connection to the DB - typically that would indicate that the DB services arent running. They may be stopping due to corruption.
Can you take a look in the db folder of your SEPM install and post up your "out.log" file - that should tell us whats going on with the DB.
thanks!
Paul Murgatroyd
Principal Product Manager, Symantec Endpoint Protection
Endpoint twitter feed: http://twitter.com/symc_endpoint
Hello Rafeeq, Hmmm, but
Hello Rafeeq,
Hmmm, but before it hits the SERVICE NOT AVAILABLE error for IIS, I see a load of SQLException errors (Connection Refused) is that not related to the Adaptive Database Engine? Maybe the SERVICE NOT AVAILABLE error happens because the error occurring at the SQL stage is trying to read the meta data relating how to connect to IIS and seeing as it gets nothing it tries to connect to nothing resulting in the SERVICE NOT AVAILABLE error?
Just a thought.
Thanks.
Gary Hall.
Hello Paul, As requested
Hello Paul,
As requested attached is a zip file containing the out.logs.
Oh and I see you answered my query about the connection refused error before I event posted it! Thanks again.
Cheers.
Gary Hall.
hi
can you check the same in your IIS logs?
Please don't forget to mark your thread solved with whatever answer helped you : ) Rafeeq
@Rafeeq - IIS Traffic
Hi,
Checked the IIS logs and TBH there is nothing in the activity logs to even indicate any traffic being directed to SEPm, it's all SelfUpdate traffic (related to WSUS), so I checked the httperr log which is generated by IIS and took a look at the errors reported in the last 24 hours, they are as follows (can see some SEPm related activity errors in the log):
2010-03-09 01:01:17 10.191.10.16 3309 10.191.10.49 80 - - - - - Timer_ConnectionIdle -
2010-03-09 01:04:17 10.191.10.183 15230 10.191.10.49 80 - - - - - Timer_MinBytesPerSecond -
2010-03-09 02:51:48 10.191.10.101 3658 10.191.10.49 80 - - - - - Timer_ConnectionIdle -
2010-03-09 03:10:48 10.191.100.32 2086 10.191.10.49 8530 - - - - - Timer_ConnectionIdle -
2010-03-09 04:54:38 10.191.100.50 2577 10.191.10.49 8530 - - - - - Timer_ConnectionIdle -
2010-03-09 05:36:28 10.191.100.53 2209 10.191.10.49 8530 - - - - - Timer_ConnectionIdle -
2010-03-09 08:11:48 10.191.20.29 1118 10.191.10.49 8530 - - - - - Timer_ConnectionIdle -
2010-03-09 09:07:38 10.191.10.16 4026 10.191.10.49 80 - - - - - Timer_ConnectionIdle -
2010-03-09 09:47:42 10.191.20.90 1795 10.191.10.49 80 HTTP/1.0 GET /secars/secars.dll?h=2C8D125793ACE9EE9C7520B1AC033AFC0C989C9D360CF4A9CA23609AD8F356D16AB9D095294019FE9254A6F5156C95909B2FD0DA577583CCB19AF9A61AD70A4E8D5CC5BF4FC484847AEE55BC85D0B852AE009960A2120E7224E132829C4A597B23667B274BAA8541E66C118350B119D281220C94049EC21421402BB4853DD14F7228B6B5E8AB7D4769E2FBC9D555B01709F46B9EE52370926EB0C9AB31F1ED99313B6B1338EA677EDDDD7A6F51B180C6E49D51E41516EEF10720AAEA97112A99E93A153700A3DB4180B70F67FFB44FD00473BA1805905B4A0315E06F73568BDA85798F7B29AC0FA1CA309DAEA9B3AB68CB73B18D70FD232F9B91291C3F3039AA176DF643705BB46EAD6C81C25FDCA1FE - 1 Connection_Dropped DefaultAppPool
2010-03-09 14:20:34 10.191.20.39 1250 10.191.10.49 8530 - - - - - Timer_ConnectionIdle -
2010-03-09 15:40:19 10.191.51.10 4145 10.191.10.49 80 - - - - - Timer_MinBytesPerSecond -
2010-03-09 15:42:14 10.191.10.16 4795 10.191.10.49 80 - - - - - Timer_ConnectionIdle -
2010-03-09 17:03:12 10.191.20.80 4244 10.191.10.49 80 HTTP/1.0 GET /secars/secars.dll?h=2C8D125793ACE9EE9C7520B1AC033AFCE052DBEAA4244632D978B784D5ABF8E91578004FC8A126B2EC1328B55838AEDD3CEFD1300801E0D992BF9C04D36D863C0BE8FEFB632F1FBEB7D3B0FC585E050296691408F62DA7327F2C7940D18F594FD5963602EB3651EB393D91150CA8A7E53F8D236C5FF96DE606C312F490BFC57D0AC3214BA1CADECE0FDD9977A0D744CBAFE0608AC3878DB625973A248BEC49B221CD15CF2A927DD6510BC81F0275D7FD472B43E3953603C51B854E4E4F21981FE6D078DDA661884B0617A9240C46AC470473BA1805905B4A0315E06F73568BDA85798F7B29AC0FA1CA309DAEA9B3AB68CB73B18D70FD232F9B91291C3F3039AAF1669F0D464CF46B153CA0353AFBCDD6 - 1 Connection_Dropped DefaultAppPool
2010-03-09 17:09:45 10.191.100.1 2835 10.191.10.49 8530 - - - - - Timer_ConnectionIdle -
2010-03-09 17:25:25 10.191.10.16 4961 10.191.10.49 80 - - - - - Timer_ConnectionIdle -
2010-03-09 18:04:05 10.191.100.50 1569 10.191.10.49 8530 - - - - - Timer_ConnectionIdle -
2010-03-09 18:05:35 10.191.20.6 3897 10.191.10.49 8530 - - - - - Timer_ConnectionIdle -
2010-03-09 18:43:20 10.191.100.1 3699 10.191.10.49 8530 - - - - - Timer_MinBytesPerSecond -
2010-03-09 19:14:00 10.191.100.28 3476 10.191.10.49 8530 - - - - - Timer_ConnectionIdle -
2010-03-09 23:46:51 10.191.100.9 1208 10.191.10.49 8530 - - - - - Timer_ConnectionIdle -
2010-03-10 00:45:46 10.191.10.16 1652 10.191.10.49 80 - - - - - Timer_ConnectionIdle -
2010-03-10 00:50:21 10.191.10.180 4591 10.191.10.49 80 - - - - - Timer_MinBytesPerSecond -
2010-03-10 08:09:13 10.191.20.56 1166 10.191.10.49 80 HTTP/1.0 GET /secars/secars.dll?h=768F85B6F19810AC43793130BEB4AD5BEBA97B8282E47CC9258C1CAC6D40EFB85525B0FCA86B06674F81CF0ECB9577681CA3A1D202C2A192B3D4138919B218122E5F2A603363AEF60C33987910F02BFD2ACBE49717A9510E3C404DAECBFA6C8BEC4C29E9507B9619367A55BB94B76E7EC0EA9F429F5299592F64253D3B41D4D4BCAAEF005D30B795783A960007F20CFBED588C48C7FB1474B918D6868E23AABE437DC2A762725FCB93C68824CE31AA5ECC43445D9B89D4149B5CA1BD2219D420452F66A3BBC33934EAE06958AAB9C896F703F876A4E45AA652BCD4A648A089210B574AE1414BBD92FC0E97327C77E7F9E80AA86E599DB19776C54408D5FB02C77C8FB083812551515D635AC10FC474D3AAE2BA7EFF02B1D9A9AB356AE365B2828E8D7EB36339561F2FFC3B3A916BD24BF5ABA5144F93EE19E908AC1CD102322276409D5200AF4FA7B73AF2A9F7DFA1325D5BDCC52058C8D4ED0E0A62D557F8E1F566F7E5611F355CFEE9677EE47A466F - 1 Connection_Dropped DefaultAppPool
2010-03-10 08:38:42 10.191.100.1 2602 10.191.10.49 8530 - - - - - Timer_ConnectionIdle -
2010-03-10 08:55:43 10.191.20.35 1131 10.191.10.49 80 HTTP/1.0 GET /secars/secars.dll?h=FB14111941E6FB41A8F6475144D39F89EB44683BDC047E79664D0CF517B8976B100E7FD86B7D92CE923978928B3B1BCC9B573C26D4E38B2F2E4588F46A77FFC52E5F2A603363AEF60C33987910F02BFD2ACBE49717A9510E3C404DAECBFA6C8BEC4C29E9507B9619367A55BB94B76E7EC0EA9F429F5299592F64253D3B41D4D4BCAAEF005D30B795783A960007F20CFBED588C48C7FB1474B918D6868E23AABE437DC2A762725FCB93C68824CE31AA5ED31FDB4F98E950AA6274AED42B45E49261A7AEF38ECCE06071710DC1F12468D55105E4F4AAD71FF3F6F1E4C1036023DE21E952E08C69271B459CB808D590EA13C0AC93197B1ADD77073632AAF9FB639D57C0952977C32FE0B5E5BFF38E451916D0BA967E951332E71A64AC88943DE77EE90A88F61A4067A90CEA5E31D222C3DBBD1D715D948D43AC4B34EF077ACC03A51920E1B1F15E7D0047708FB8CBA61C18201E09A8284A2CB0C48CEF538645452BD11EFC9A43FFE2D9073784FCA02CA72D - 1 Connection_Dropped DefaultAppPool
2010-03-10 09:00:43 10.191.20.48 4517 10.191.10.49 80 HTTP/1.0 GET /secars/secars.dll?h=8523ED0E2A47534E7E6868AA100B10A039A9A24EFA02CB976DBDA32D8EA27E10C173407FA3249175BF97AF38A5E6A8603D9108E5687786B065F68ACDA14EBEC02E5F2A603363AEF60C33987910F02BFD2ACBE49717A9510E3C404DAECBFA6C8BEC4C29E9507B9619367A55BB94B76E7EC0EA9F429F5299592F64253D3B41D4D4BCAAEF005D30B795783A960007F20CFBED588C48C7FB1474B918D6868E23AABE437DC2A762725FCB93C68824CE31AA5ECC43445D9B89D4149B5CA1BD2219D420A0530744FA06A03CB93EF2BA138543C6789C773381829A72787E084C81ECFB02DB86AA7D0B866619BB677E5433981485E6867D926C900BF7900C983CB9C913537C1F88FC75FBD73B0E68C6B02AB90D45EBB2DB3481073BFBED179C3EA7AD74D5F42B817C6568170B95CF79CDC072AEF1CFA44E0D4C30846F1845CFAE952BA0C541DB8DEE5157C618BA8F76F58680058F61C0A8746F41305E36D42137CD99BF686192265855E3D1FE03869306539DE739 - 1 Connection_Dropped DefaultAppPool
2010-03-10 09:23:13 127.0.0.1 3676 127.0.0.1 80 HTTP/1.0 POST /Reporting/Monitors/ajax.php - 1 Connection_Dropped DefaultAppPool
2010-03-10 09:33:42 10.191.20.90 1358 10.191.10.49 80 HTTP/1.0 GET /secars/secars.dll?h=2C8D125793ACE9EE9C7520B1AC033AFC0C989C9D360CF4A9CA23609AD8F356D16AB9D095294019FE9254A6F5156C95909B2FD0DA577583CCB19AF9A61AD70A4E8D5CC5BF4FC484847AEE55BC85D0B852AE009960A2120E7224E132829C4A597B23667B274BAA8541E66C118350B119D281220C94049EC21421402BB4853DD14F7228B6B5E8AB7D4769E2FBC9D555B01709F46B9EE52370926EB0C9AB31F1ED99313B6B1338EA677EDDDD7A6F51B180C6E49D51E41516EEF10720AAEA97112A99E93A153700A3DB4180B70F67FFB44FD00473BA1805905B4A0315E06F73568BDA85798F7B29AC0FA1CA309DAEA9B3AB68CB73B18D70FD232F9B91291C3F3039AA95B7AD42CFA4DD6D33E594BDE7E3AC5B - 1 Connection_Dropped DefaultAppPool
2010-03-10 09:41:07 10.191.10.16 2442 10.191.10.49 80 - - - - - Timer_ConnectionIdle -
2010-03-10 10:40:52 10.191.20.80 1854 10.191.10.49 80 - - - - - Timer_MinBytesPerSecond -
2010-03-10 11:04:08 10.191.100.26 3197 10.191.10.49 8530 - - - - - Timer_ConnectionIdle -
2010-03-10 11:57:43 127.0.0.1 3721 127.0.0.1 80 HTTP/1.0 POST /Reporting/Dashboard/ajax.php - 1 Connection_Dropped DefaultAppPool
2010-03-10 11:57:43 127.0.0.1 3720 127.0.0.1 80 HTTP/1.0 POST /Reporting/Dashboard/ajax.php - 1 Connection_Dropped DefaultAppPool
Thanks.
Gary Hall.
hi
Your application pool dropped the connection, attaching a debugger will reveal the process thread.
I have been looking at him blogs from past two years:) he knows everything related to IIS
http://blogs.msdn.com/david.wang/archive/2005/08/29/HOWTO_Understand_and_Diagnose_an_AppPool_Crash.aspx#457928
Please don't forget to mark your thread solved with whatever answer helped you : ) Rafeeq
Thanks for the information
Thanks for the information Rafeeq, the David Wang blog makes for some interesting reading.
Ill wait to see if Paul Murgatroyd gets back within anything related to the database engine and then look at both sets of suggestions to make sure I am missing nothing out when resolving the problem.
Thanks.
Gary Hall.
hi
good to run a dbvalidator too,
http://service1.symantec.com/SUPPORT/ent-security.nsf/2326c6a13572aeb788257363002b62aa/a5f9153ff7a6add0882574430060dc04?OpenDocument
Your version had some issues with primary keys too; lets wait for Paul
http://service1.symantec.com/SUPPORT/ent-security.nsf/docid/2007121216360648
Please don't forget to mark your thread solved with whatever answer helped you : ) Rafeeq
The FInal Solution.
Firstly thanks to all for thier help, especially Rafeeq. I decided to go with an upgrade to MR5 as time was getting on and I started receiving virus definition out of date emails almost constantly for 20 computers (or more) when in actuality there was 3 computers showing in the SEPm console that had definitions older then one week. At this point I lost faith in our installation of SEPm.
Therefore I tried to perform an upgrade to MR5, first step backup the database. It got to 12GB in size and I became suspicious as how could we have a db that big for a total of around 300 computers and servers? I took a look at the actual db file (we used the built in db engine) and it was 128GB in size!!! Therefore I used Rafeeq's advice to perform a db validation in the hope that it would suggest we compact our 128GB sized database as it had a lot of wasted space. No joy, the validation tool stated we had validation errors and that we should restore from a backup.
At this point even though we have a backup I thought best to start a new, so removed SEP MR2 and installed MR5, before doing this I exported all out policies (AV, LiveUpdate, Centralised Exceptions etc) and then reimported them into MR5. Then using Symantec Altiris I deployed a Sylink Drop package to all our computers in order that they re-registered with the new SEP MR5 installation (as the domain id and all the group id's would have changed from the fresh install).
Therefore now have a MR5 installation with at least all our clients reporting back to a stable installation, next step is to use the upgrade wizard to start upgrading our clients to MR5.
Only time will tell if the SEPm console in MR5 will eventually start displaying the same fault as the previous MR version. I looked into the application pool dropping connections which appear to be mostly from clients rather then the SEPm server which can be expected I would say (computers being switched off incorrectly, unstable ISP links for when clients are connecting through VPN etc).
Thanks.
Gary.
Would you like to reply?
Login or Register to post your comment.