Endpoint Protection

And when I say "crippled," I mean employees were late clocking in because it took 5 minutes to load a web page with no pictures on it.

Here's what happened: I upgraded from 12.1 RU1 to 12.1.2 yesterday, and as soon as the install was finished everyone on the network started to notice that internet traffic came to just about a halt. I ran a speed test and found that we were getting somewhere around 0.2Mbps on a 3.0Mbps connection. That is bad, really really bad. Looking back at a bandwidth report from yesterday, there was over 2GB of traffic in under an hour all directed at my Symantec server, and from liveupdate.symantecliveupdate.com.

The reason this happened is that LiveUpdate was grabbing new virus defs because the ones on the server were about a month out of date. 12.1 RU1 stopped getting new definitions for some unknown reason (and this isn't the first time it's done this, either), so after trying various ways to get it to work I was ready to call Symantec tech support. I know that as soon as I called them they would want me to upgrade to the latest version of SEPM, so rather than argue about whether or not RU1 should be able to do something simple like get new virus definions, I figure'd I would go ahead and update. And fortunately, it did at least for now fix that problem. Last time it fixed the problem for about a week before I stopped getting definitions again.

Ok, lots of rambling, but here is what I need to know, if anyone can help: is SEPM/LiveUpdate *supposed* to start downloading as soon as it is installed, especially when it's out of date? And if that is normal, is there a way to control it? It is completely unacceptable for any computer to consume that much bandwidth on my network. This time it didn't kill our VOIP phones, but it did present a huge problem for the whole network.

If anyone knows a good/better way to control the LiveUpdate download, please let me know.

Hello,

Since Symantec Endpoint Protection Manager would always try to fetch the updates upon Migration for the Latest definitions as this is by design.

In this case, you could not perform any changes in Symantec Endpoint Protection Manager. However, you could try blocking the Liveupdate websites, which would give make sure that updates don't happen.

1. Liveupdate.symantecliveupdate.com
2. Liveupdate.symantec.com
3. Symantec.com

Secondly, Symantec makes LiveUpdate content available on the Internet through a partnership with the Akamai server network.  Akamai is a network of tens of thousands of servers scattered worldwide for more efficient distribution of content.  Symantec recommends specifying Symantec LiveUpdate servers via DNS name (fully qualified domain name), not via one or more static IP addresses. Use of static IP addresses to access Symantec LiveUpdate content is not supported or recommended.

For more information on the use of the Akamai network and means to create firewall rules for LiveUpdate in this manner, please consult Symantec knowledgebase article -

http://www.symantec.com/docs/TECH163079

Hope that helps!!

You have the ability to schedule updates during a certain window, When is it currently set to download updates?

Also, do you have multiple location or was this for only one? You can configure group update providers (GUPs) at remote locations to provide updates to clients so they don't come back over the WAN.

I believe the update from 12.1 to RU2 caused this traffic congestion - if the definitions were a month old - SEPM needed to update these right away after upgrade was finished. For future download you can set up SEPM to download only during specific timeframe from a specified Servers - either symantec Liveupdate or internal LUA.

Can you give us a bit more info surrounding the upgrade and when the network load hit happened please?  

Were clients being upgraded as well?  As part of the client install, they would automatically perform a Liveupdate unless the below settings changes were used:

http://www.symantec.com/docs/TECH184654
http://www.symantec.com/docs/TECH91474

Maybe can you explain abit about your network and SEPM architecture?

Like when did you schedule your LU running?

Did you tick the option to download from Symantec LU or only from management server?

etc etc

Thank you, that does help!

Temporarily blocking the Liveupdate domains during the update process could be a solution in my case, since this is not a frequent or recurring event (at least in normal circumstances). I will probably try that if I have this situation again.

Our updates are scheduled for evenings, i.e. non-production hours. Normally there isn't a problem because even if they all updated simultaneously, nobody is here to notice a bottleneck. Also, this was not a "regular" update. My SEPM had stopped updating virus definitions. Unfortunately, this has happened a few times and each time upgrading has seemed to fix the problem, at least temporarily.

In this case I had to do the update in the middle of the day (which again, isn't normal) and thus, the bottleneck. We have multiple locations, but run MPLS connections to our branch offices. They operate as if they were at the same physical location, just on slower connections to the network. In other words, it is one site/forest.

New virus definitions were not being downloaded by luall.exe. I could see this in the logs, so I knew where the problem was occuring, but couldn't identify the cause.

An hour or two after upgrading the server, employees started complaining of very slow web-related problems. I thought we were having connection issues because I didn't know of anything running that could be hogging that much bandwidth.

The day after the upgrade I checked a report that showed where all the traffic was coming from.

Clients were not supposed to be upgraded - prior to the server upgrade I moved all my clients into a control group on the server with no install packages. Since then I've been moving them back a few at a time.

Thanks for the articles, that will really help if I have to do this again!