Endpoint Protection

We are having Windows servers spiking the CPU randomly; suddenly CPU spikes to 100% in all the servers at the same time, this is ~25 servers experience the issue simultaneously. All the servers have Symantec Endpoint Protection ver 12.1.6318.6100 installed. The issue is sporadic (happens once every 2 or 3 weeks) and unfortunately it cannot be reproduced at will.

The process spiking the CPU is the Goliath Performance Monitor agent from Breakout Technologies, the service is listed in Windows as "MonitoringIT Agent service") => either RpmAgent.exe or its parent process AgentService.exe are affected. When the issue occurs stopping or restarting the MonitoringIT Agent service resolves the issue.

The reason for this post is that recently we found that when the issue happens the affected process contains 2 instances of the SONAR engine (UMEngx86.dll), where servers that are not affected do not contain any (screenshot attached):

We also noticed shortly before the issue occurs (~15 mins) there's an isolated error on the Application log of every affected machine:
Event ID 74, Source: Symantec Antivirus, "SONAR has generated an error: code 0: description: Definition Failure"
We have memory dumps of the affected process in case they're necessary.

We don't see UMEngx86.dll hooked into any other processes, whether they are already running or when we start applications, so we wonder why is it for the Goliath agent.

We're still researching this and trying to repro in a test environment to find more patterns. For now we're posting this in hopes of getting some ideas/suggestions. Any assistance sincerely appreciated.

Thank you

Run the symdiag tool on the affected machines to do additional error checking:

Download SymDiag to detect Symantec product issues

I would also recommend getting a case open as this may require advance support.

you are best of creating a ticket with support and providing them with full memory dump as that's the first thing they are like to ask. and you many also want to work your monitoring tool vendor Breakout Technologies to get to the bottom of this.