Endpoint Protection

Odd and rare, but regular the last 2 days. These just started. Was it something I did perhaps? It first started at about  14:12:58 yesterday (2:12pm our time)

I get an email alert system event notification, unexpected server error ( do we ever expect them? ;-)   )

I went into monitors tab, logs, system, server activity and see what is possibly the problem - or at least *A* problem - and I see it for both SEPM servers, typically within seconds of each other, at times 3 or 4 minutes apart, but typically in pairs, one then the other.

The SEPM servers are at 12.1 RU2

 Note below the pairing - SEPM1 and SEPM2 in each I have bracketed with red. Never JUST one or the other, but always if it happens on one, it happens on the other within seconds or minutes.

Any changes made?

Are clients still communicating in to the SEPM? What the CPU under a heavy laod?

No changes to the HI part. Changes to the custom IPS as we keep trying to make SEP let the bluejeans plugin for IE work/install. HI, no. Nothing changed there.

Clients are communicating fine - HOWEVER, policies are really freaky lately, and one group that is sort of a test os SNAC alerted a fellow that he didn't comply with ANY of the HI checks, no SEP, no defs, group policy (AD) settings were incorrect, registry entries incorrect, it simply said every point failed. But it was lying - SEP was there and working, and the computer met all specs.

There are 2 SEPMs, both do this same thing, same errors, like shown above.

Heavy load? Nope - 2 dedficated SEP managers dealing with 350-400 client computers between them, so they never have any sort of a client load- not when SEPMs can deal with thousands, and we have a "few hundred".
 Spikes over 50% CPU are rare.  They are typically running 3-5% CPU, 1.55 gig memory used out of 6 gig "installed memory". They run 2 - 2.4 GHz AMD processors.  It's all they do - fully dedicated to SEPM and nothing else at all.
They are VM servers with Server 2008R2. Network utilization runs about .01% up to perhaps 2% or so of a 1Gbps connection. I've never seen over 5 or 10% network utilization.  My workstations work much harder.

 

Hi

Please repair and run management server wizard console

Regards

 sorry, ya lost me.

 "Please repair?"  Repair what?   Uh, that's what I'm asking about ;-)   WHAT needs repair, what triggered these errrors - why did they start? What step will solve them for sure.  Repair is what I'm asking about -  I don't know what needs repair.

 I also have to ask - What is a 'management server wizard console'?  I know of the configuration wizard - but honestly it's configured just fine as far as SQL and all the settings there are concerned.
All that configuration wizard changes is the connection, names, and credentials used. Sorry but I've never heard of a management server wizard console. Maybe that is something new with RU2?

Client computers communicate - that part isn't broken. The daily automatic backup of the SQL database works, the backup is then stored on one of the SEPM servers, that part works. The backup files are there and are about 17.5 GB (18,828,906,781 bytes) each. I keep two - the current and the one from the day before.
 I can get to and change policies, configuration info, I can see the groups and settings and configuration, I can move computers from group to group, I can push SEP installs to clients and it works, installs via a package assigned to a group work.

Those email alerts and the system logs like I showed. It's HI - *Host Integrity* related. It's something to do with JAVA and HI.

Anyway, please take another look - read the error log message lines and tailor a reply to a specific problem.   - JAVA is having trouble with the HOST INTEGRITY template and policy, or template OR policy, I can't tell which............. so there must be some issues there, but where is there?
 I don't know what it IS, but I know what it IS NOT. It IS host integrity template and/or policy related. It's not the setup of the SEPMs.  There is a problem that some JAVA process is having with the HI policy/template.

 I suppose it's entirely possible I'm among the few "power users" with SEP and everyone else takes it out of the box and uses all the defaults. Maybe that's why I see issues no one sees, and few even know about. I did a search and it seems that there's nothing related to this specific error out here. No one else uses SNAC or HI?

I'm not going to spend a lot of time "swapping engine parts until it runs". I can't stand "keep trying different stuff until you get lucky and it works". I prefer systematic repair once the problem is diagnosed and the cause found.

(As a side note - I hate JAVA, it's slow and VERY unsecure and it's a shame Symantec has chosen to use the security-hole laden Java for anything at all instead of writing clean code that can run with NO JAva installation.
There's a growing number of us fed up with weekly or monthly Java holes and patches and we're calling a halt to JAVA anything on any servers but for some really odd reason they've stuck with JAVA, so things run really slow and unreliably. We constantly must remove and reinstall the latest JAVA to patch things - and then stuff breaks.
Is it all Related? I don't know. I do find it ironic that the security company at the top of the charts - rated very high with Gartner, chooses to rely on, to base their security software on, the most unsecure piece of junk produced today besides Acrobat Reader - and that junk is JAVA.)

The HI policies push *.js files on the client,that runs locally for the checks.can you check if those JS files are getting blocked on the client.

If those files are blocked from executing then it might act differently. 

Sorry, but this is a pure server issue. I appreciate the try - and it does bring up something to look at - how to the policies get to clients? (I think it's XML, not JS, but it would be an interesting thing to look at)

 JS files are used widely in SEP (and other products) and are not blocked or restricted. If it was, I'd expect SEP to be acting totally bad. This is, or at least appears to be according to logs, where the server is compiling the JS policy files to push to the clients.
But by now I've pretty well figured out at least how to stop these errors - it was a shotgun approach, but after a week of no results or help, I had little choice......... We're too busy to mess around so I had to do something, even if it was wrong.

IMO,  this is 100% server. Nothing on the clients can cause an error on the management servers related to compiling policies, especially both servers within seconds of each other.

I temporarily resolved this, since it seems no one knows anything at all about this "problem" by simply deleting the HI policies completely. They are now totally gone, all HI policies, deleted from SEPM altogether. And guess what - the errors stopped.
I was right, it was a pure HI (Host Integrity), not related to AV or defs or anything on the client, 100% pure server problems.

We don't have this enabled on clients, and if it was a client issue, then I'd see pretty constant errors since the clients check in constantly through the day and night. At least that's my logic - clients with a heartbeat of 5 or 7 minutes checking in at random times to either server, yet the errors are logged on one server, then seconds later, the other, and not constantly through the day. Since a client is using one "parent" or the other, then it should hit just ONE of the servers if a client with a problem checks in with mother.
If a client checked in with mom, then 5 seconds later, checked in with dad, I could see both servers erring like that. But that's not how client check-in works, so that combined with the fact that the policy is not assigned and that this is a compile issue, and the policy assignment is one-way - it's pushed but the client doesn't respond instantly with a "try again, this one is bad" response, all of those pretty much rules out clients. Whatever it is, or now was, was pure server process.

This seems to be the way it's been going lately for SEP - I get a "please repair" with no explanation (seen all too often - and I know the reason as I predicted it a while back ), but otherwise nothing timely or of real consequence.  Do I not include enough information? Is the Symantec forum related to security lacking volunteers who really know SEP inside and out? Is it a bad time of year? Or am I just really super lucky and have these weird issues no one else ever sees? With that sort of luck, I guess I won't buy any lottery tickets, eh?  LOL!

Have you deleted the policy or took backup of policy and then deleted..may be you can try putting the same policy on a test SEPM or something to check if it has same behaviour..Possible policy corruption.