Endpoint Protection

Two SEPM servers running as peers on the same network. Configured mostly for load balancing I have 2 so that if one dies we still have at least 1 for managing things and for clients to recieve updates, policies, etc.
They were at version 12.1.4. I downloaded the latest SEPM and SEP upgrades available, 12.1.5.xxxx last week and proceeded to upgrade as always.
I went to SEPM2 and stopped the SEPM management and web services.
I went to SEPM1 and performed the SEPM upgrade. It went not only well, but quickly.
I then stopped the services on SEPM1 and went to SEPM2 and preformed the ugprade. SEPM2 failed due to not enough disk space. It had a lot free, but apparently not enough. I cleared up a ton of space and restarted the upgrade. I don't know why two identical servers one was fine and the other was filled with little space left but it appeared as if the content area was messed up and SEPM2 wasn't keeping things cleaned up. There was several gig difference in the free space but the servers were exactly the same.
This time instead of being so bloody simple (SEPM1 was the easiest and fasted upgrade I've ever done!) SEPM2 was crazy, it said it would use the recovery information, etc. etc. and looked like the old SEPM upgrades used to. With 1 exception, I knew it was the new system due to the additional service/port it was listing. SEPM1 never said a thing about any recovery info file, and only asked about the new port for process launcher, it never referred to anything else at all, it just ran.

SEPM2 said it was doing the database stuff, which of course it found as already done, and proceeded to move forward quickly - but at the apache service upgrade it choked. It kept failing stating it was unable to upgrade the web management part.
I checked the log files, hidden away - and the failure said nothing about looking at log files to see WHY it failed, just "we failed, OK?" message.

The apache log over and over each time I tried, showed this:
2014-10-13 08:08:18.680 THREAD 1 SEVERE: ================== Server Environment ===================
2014-10-13 08:08:18.695 THREAD 1 SEVERE: os.name = Windows Server 2008 R2
2014-10-13 08:08:18.695 THREAD 1 SEVERE: os.version = 6.1
2014-10-13 08:08:18.695 THREAD 1 SEVERE: os.arch = x64
2014-10-13 08:08:18.695 THREAD 1 SEVERE: java.version = 1.7.0_55
2014-10-13 08:08:18.695 THREAD 1 SEVERE: java.vendor = Oracle Corporation
2014-10-13 08:08:18.695 THREAD 1 SEVERE: java.vm.name = Java HotSpot(TM) Client VM
2014-10-13 08:08:18.695 THREAD 1 SEVERE: java.vm.version = 24.55-b03
2014-10-13 08:08:18.695 THREAD 1 SEVERE: java.home = C:\Program Files (x86)\Symantec\Symantec Endpoint Protection Manager\jre
2014-10-13 08:08:18.695 THREAD 1 SEVERE: catalina.home = C:\Program Files (x86)\Symantec\Symantec Endpoint Protection Manager\tomcat
2014-10-13 08:08:18.695 THREAD 1 SEVERE: java.user = null
2014-10-13 08:08:18.695 THREAD 1 SEVERE: user.language = en
2014-10-13 08:08:18.695 THREAD 1 SEVERE: user.country = US
2014-10-13 08:08:18.695 THREAD 1 SEVERE: scm.server.version = 12.1.5337.5000
2014-10-13 08:19:39.942 THREAD 33 WARNING: SemServiceManager> unInstallApacheService>> Uninstalling apache service...
2014-10-13 08:19:43.296 THREAD 33 WARNING: SemServiceManager> unInstallApacheService>> Uninstalling apache service done! Process return code = 0. Apache service account is deleted from 'log on as a service' option ? false
2014-10-13 08:19:43.296 THREAD 33 WARNING: SemServiceManager> installApacheService>> Installing Apache service, serverHome: C:\Program Files (x86)\Symantec\Symantec Endpoint Protection Manager\tomcat, serviceName: semwebsrv
2014-10-13 08:19:43.296 THREAD 33 WARNING: SemServiceManager> installApacheServiceOnce>> Installing Apache service... Current time = Mon Oct 13 08:19:43 CDT 2014
2014-10-13 08:19:45.512 THREAD 33 WARNING: SemServiceManager> installApacheServiceOnce>> Apache Service installation failed. Process return code ['0' means success] = 1
2014-10-13 08:19:45.512 THREAD 33 WARNING: SemServiceManager> installApacheServiceOnce>> Installing Apache service done! Process return code = 1, Current time = Mon Oct 13 08:19:45 CDT 2014
2014-10-13 08:19:46.526 THREAD 33 WARNING: SemServiceManager> installApacheService>> Installing Apache service failed. Installation process returned value ['0' means success]: 1 Retrying...
2014-10-13 08:19:46.526 THREAD 33 WARNING: SemServiceManager> unInstallApacheService>> Uninstalling apache service...
2014-10-13 08:19:50.020 THREAD 33 WARNING: SemServiceManager> unInstallApacheService>> Uninstalling apache service done! Process return code = 0. Apache service account is deleted from 'log on as a service' option ? false
2014-10-13 08:19:50.020 THREAD 33 WARNING: SemServiceManager> installApacheServiceOnce>> Installing Apache service... Current time = Mon Oct 13 08:19:50 CDT 2014
2014-10-13 08:19:51.767 THREAD 33 WARNING: SemServiceManager> installApacheServiceOnce>> Apache Service installation failed. Process return code ['0' means success] = 1
2014-10-13 08:19:51.767 THREAD 33 WARNING: SemServiceManager> installApacheServiceOnce>> Installing Apache service done! Process return code = 1, Current time = Mon Oct 13 08:19:51 CDT 2014
2014-10-13 08:19:52.782 THREAD 33 WARNING: SemServiceManager> installApacheService>> Installing Apache service failed. Installation process returned value ['0' means success]: 1 Retrying...
 

Can this be fixed or must I do a removal of the entire SEPM server and start all over again with all the security issues that go along with brand new SEPM servers? I do have SEPM1 working - I believe, need to check for sure, but saw no failure messages, but I'd rather not do a manual purge of SEPM2 and start from scratch if it can be avoided!

 

I don't know if this matters or not.......
2014-10-13 09:23:44.149 THREAD 13 SEVERE: ================== Server Environment ===================
2014-10-13 09:23:44.149 THREAD 13 SEVERE: os.name = Windows Server 2008 R2
2014-10-13 09:23:44.149 THREAD 13 SEVERE: os.version = 6.1
2014-10-13 09:23:44.149 THREAD 13 SEVERE: os.arch = x64
2014-10-13 09:23:44.149 THREAD 13 SEVERE: java.version = 1.7.0_55
2014-10-13 09:23:44.149 THREAD 13 SEVERE: java.vendor = Oracle Corporation
2014-10-13 09:23:44.149 THREAD 13 SEVERE: java.vm.name = Java HotSpot(TM) Client VM
2014-10-13 09:23:44.149 THREAD 13 SEVERE: java.vm.version = 24.55-b03
2014-10-13 09:23:44.149 THREAD 13 SEVERE: java.home = C:\Program Files (x86)\Symantec\Symantec Endpoint Protection Manager\jre
2014-10-13 09:23:44.149 THREAD 13 SEVERE: catalina.home = C:\Program Files (x86)\Symantec\Symantec Endpoint Protection Manager\tomcat
2014-10-13 09:23:44.149 THREAD 13 SEVERE: java.user = null
2014-10-13 09:23:44.149 THREAD 13 SEVERE: user.language = en
2014-10-13 09:23:44.149 THREAD 13 SEVERE: user.country = US
2014-10-13 09:23:44.149 THREAD 13 SEVERE: scm.server.version = 12.1.5337.5000

 

When doing the upgrade, the SEPM service on both needs to be stopped at the same time. Did you so this or only one at time? Not sure if this matters though...it's usually to avoid db schema corruption.

Can you review, and possibly post, the SEPM_INST.log as well?  This is saved to the %temp% folder of the account under which you attempted the upgrade.

I should re-word that.

I stop both services on both servers, I disable the two on the second server just in case for some reason it gets rebooted, they can't start by themselves.

I just stop them on the first as the first will simply start the services anyway and it has historically failed if I disable it as well on the first. Then I reverse the process, stop on the first so I can upgrade the second, but undisable the services on the second so the result is that they are stopped on both, not disabled on either when I do the upgrade on the second.

The upgrade process stops the services on the server being upgraded during all this anyway when you are performing the upgrade, it's just a paranoia thing.

The main important thing is to STOP and disable both on the second server, or the one I am NOT currently and actively upgrading. And again I disable the services on the second server, not just stop them, just in case someone else restarts a server or for some reason it happens to crash and reboot on its own, the services won't be left in automatic and restart in the middle of the upgrade to the SEPM1 server. I then reverse it all.

The process I use are the same I've used since I first started using SEP many years ago, the early 11 versions, and I've used every single upgrade since with very little trouble. They almost always simply work. This is the first failure I can recall in a very long time. The first server is verified working fine so far.

Yeah - I'll look at it and may need to redact some info if it gets too bloody specific with names and IP addresses and accounts and such.......

 

It's HUGE so it's attached, not pasted in here.

I changed LOG to TXT and added -copy to the name so I could leave the original in place with no changes. I did change IP addresses and a couple of SQL account related names.

 

Check my post here:

https://www-secure.symantec.com/connect/forums/multiple-issues-after-upadting-sep-121-ru5#comment-10771841

Some of these issues have been resolved thanks to support.

One, the av defs problem was an issue that they'd seen before and wrote a tool to fix it. One of the issues I found is actually a bug (introduced prior to 12.1.5)  And some of the issues were caused by SEPM upgrade to 12.1.5 making the SQL database grow SO FAST it filled the drive on the SQL server and caused some of the SQL files to hit their growth limits and beyond!

In any case, I now have 2 working SEPM servers again, the display in the firewall logs part of SEPM works on both servers, the SQL database explosive growth is being looked at, and the columns sorting issue was found to be a big I had uncovered and will be looked at as well.

 

Hello  ShadowsPapa can you please mention the tool you are talking about to clear the defs ? Regards

I'M not sure if you have to have an open ticket for the tool to be availble or not.

It was a tool that I was told to download from a link provided and then put into the "tools" folder on the server and run a batch file. It only took a minute or less but it sure fixed things as far as the virus defs problem!

 

What I am understanding from this please correct me if I am wrong . What this tool basically clears the corrupted defs automatically which we used to do by going into specific folders and then clearing some specific stuff within them , isn't it ?

Different animal. We'd tried all of the normal routes - more than once, on two servers. This was some weird problem that apparently takes place when a 12.1.5 upgrade goes south.

I had followed the first level tech all over the place, and I only got their support involved after following all possible leads on trying to manually clear out "bad defs" using the Symantec KB and tech docs and info from these forums. It wasn't even creating some piece that is normally supposed to be there. It downloaded the definitions fine in LU but wasn't able to make them available for the SEPM. The SEP defs were fine and on any given computer LU would run and update the AV defs. The AV defs for SEP running on the SEPM servers was fine - only the defs for the SEPM use - those given to the client computers from the SEPM servers were a problem. I bet we spent a couple of weeks trying to clean up the SEPM issue with the defs then the next level of support came in and we tried some things - and then he said "wait, the engineers say they've seen this before and are working on a tool to take are of it". He said they didn't know the cause, HOW it got that way (yet, at that time) but did know how to fix it if it happened. After a day or maybe only a few hours, I had the link, downloaded the tool, ran it on ONE SEPM (that's the thing, he said run on ONE SEPM server) and once I did that all was fine and still is.

Whatever it does, its' not for cleaning out defs of SEP, but the defs for SEPM.

Since you do it only on a single SEPM server I have to suspect it touches the SQL database where those things are stored for SEPM use.

The tool is a called ContentsCleanup... it has two files in it, a  *.jar file and a batch. It's used to clear stuck content from the database. I just had a similar issue with the Win32 defs not updating on the Managers and we fixed it with that cleanup tool. You need to get it from Symantec, as they want to ensure it addresses your issue and customize for that. My isssue was the Win32 virusdefs monikers.