Endpoint Protection

Good morning,

I have done a fair amount of searching about the problem we experianced this morning within one of our business units but I have come up empty handed. Because of the distributed nature of this business unit, bandwidth is always a concern. This morning, roughly 40 sepm clients (of roughly 200, so about 20% of them) decided to download between 94MB and 129MB from our SEPM server located in their domain.

After looking through these machines there seems to be no pattern, they are all windows xp or vista, they are all running various client versions (anwhere from 11.0.2 to 11.0.6) and some are updated to todays defs while others are still showing October 28th as their latest definition package.

The WAN team is inquiring with me why this happened, as they had to throttle SEP back on port 8014 manually as it was saturating many of the lines this morning. Since then the issue has resolved itself and the clients are no longer requesting all this data.

I am used to seeing 3-5MB going to machines on a monday morning but I have never seen 94-129MB of data being requested, for what seems to be no good reason.

What steps can I take to find out WHAT caused this, and how to ensure it doesn't happen in the future? Thanks for any help in advance.

I have seen this happen for at least two reasons, but it's possible there are more.

- 32-bit machines trying to download 64-bit definitions, or vice versa.

- Machines that request and download Intrusion Prevention signatures but can't apply them because NTP is broken, so on next heartbeat request them all over again.

A Sylink log would be able to tell you what's happening.  The output of the Support Tool would be able to let you know if NTP was broken.

How to enable Sylink Debugging for Symantec Endpoint Protection in the registry
http://www.symantec.com/docs/TECH104758

For those machines that are out there that are still at 11.0.2--I can't recommend strongly enough to get them updated.  Client communication problems galore have been resolved since then, plus there have been vulnerabilities identified with older builds (none have been exploited to my knowledge, but why leave that door open?).

Good luck,

sandra

One other thought to add to this...if you have SEPM set to hold 3 revisions (the default) and for whatever reason we have more than 3 revisions come out over the weekend and/or the machines are not turned on on Friday the clients will be older than the 3 revisions. This will cause the clients to request a full virus definition will which could cause the issue you are seeing.

Here's more info:

http://www.symantec.com/docs/TECH92225

http://www.symantec.com/docs/TECH92051

http://www.symantec.com/docs/TECH94916

Zone, I have come across the same issue as you, and was told by Symantec to throttle port 8014 via IIS. If I un-throttle the port the traffic increases but with the port throttled the traffic looks like there is no problem occurring.

You stated that your WAN team basically did the same thing and that the issue has since cleared up. Do you know if your WAN team still has port 8014 throttled and that is what is making look like it is cleared up? Or have they un-throttled port 8014 and they no longer see the large amount of traffic?

Thank you for the replies. Throttles are still in place until we can pinpoint what caused the issue.

John, I think you got it spot on. There was some space issues faced on the server in charge of the one affected domain so I turned it down temporarily to only house that exact number - 3. This was a few weeks ago, since then we have expanded the data drive on that unit. I am going to turn that number back up to 10 (our default) which I believe will fix this issue.

DCF, Sandra...thank you for your comments as well. I have been wanting to force machines to update to the latest SEP version via SCCM for literally years but it has never made its way to the top of the priority list. Maybe I should make it a priority!

Thanks again.