Endpoint Protection

Coincidence? Well, check this out.
This is a chart of our bandwidth usage. Note when things started to go haywire - Friday afternoon, the 7th. Note that it's fine for a few hours each night starting at about midnight, then ending and going crazy again about 5 or 6 am.
Note then that the pattern changes Wednesday, and from then on, it's steady, no middle of the night drops in usage.
Guess what happened in that time........
On Friday the 7th, I built packages of RU6a, and assigned said packages to our main client group. I set the schedule for the packages but had my AM and PM wrong, so the package deployment was off from nidnight to 6am, when I intended it to not deploy from noon to 6. And note the dips in usage from midnight to about 6

Then on Wednesday, there were just a few not upgraded from RU5 to RU6a yet, so I set the deployment schedule for the assigned packages to be identical stop and start so it would just push out to all the few reaming workstations (And that's where the midnight dips in usage stop as well)
Now almost ALL clients are sitting at RU6a, but look at the bandwidth utilization for today, the 18th - spiked and stuck there.

My guess - the upgrade service has gone totally crazy and haywire in RU6a and it's totally KILLING our network.
We have a 5meg pipe. Utilization is SIX MEG!!!!!!!!!!!!!!!!
Take a REALLY good close look at this, and then note my package timing - package created and schedule set on the 7th, package schedule changed on the 12th.
Coincidence?
I doubt it!
RU6a, IMO, If I had it to do over, I'd not think about it yet.......... just too many questions, too many issues IMO.
I have no clue how this is going to get fixed now!
We've 40 offices around the entire state of Iowa. This is our pipe from those offices into this building with the servers, DCs, SEPM servers, etc.

Ouch! Making me think twice about upgrading to RU6a. I'm at RU6 in our pre deployment stage with about 100 clients. I will be pushing out to about 35K clients this year. Maybe I'll stay with Ru6 and wait for RU6 MP1

Hello ShadowsPapa,

The following document should at least ease the amount of strain the SEPM is putting on your network (if it is indeed the cause of your problems) until you are able to resolve the issue.

Title: 'How to throttle network bandwidth used by the Endpoint Protection Manager (SEPM) website in Microsoft's Internet Information Server (IIS)'
Document ID: 2008032806522248
> Web URL: http://service1.symantec.com/SUPPORT/ent-security.nsf/docid/2008032806522248?Open&seg=ent

Hope this helps.

Regards,
James

I would open a case with support

And this is our new server, a 2008R2 64bit dual processor 4gig RAM VMWare server.............
It's got ONLY SEPM and SEP client on it, otherwise, it's got nothing but OS on it.
It is one of two new servers, native RU6a install, not an upgrade.  It does this pretty much all day..............even after a reboot.
(RU5 NEVER acted like this)

I'm really curious to see what's causing this. Been on RU6a for a few weeks now and have yet to see issues like this. Your's and my server are nearly identical in terms of specs

I"m not certain yet, but the CPU spikes appear to be related to using the console! When I open the console on my workstation, and connect to a specific SEPM server with the JAVA console, the CPU spikes.
When I reboot the server and it kills the console connection and I DO NOT reconnect, but rather kill the JAVJAW or whatever processes on my client computer, the CPU use on the SEPM server it was connected to goes to normal. When I go to antoher computer, launch the console and connect to a different SEPM server, then THAT SEPM server spikes and stays there, until I log off and exit the console running on that workstation - and reboot the server.
Logging off the JAVA console isn't enough, you have to exit it and reboot the server.

So two major issues -
1. bandwidth use when using SEPM to assign client packages for the automated update,
2. and CPU spikes when using the JAVA console to connect to a SEPM server.
all appear to be introduced in RU6a.

DOES ANYONE here actually launch the JAVA console on a remote computer, say your DESKTOP computer, connect back to the SEPM server and leave the console running?
We use it so much we typically have the console running from 2 computers all day.

Nope, I must be wrong - one of the SEPM servers is starting to spike CPU again and the console isn't open anywhere, let alone connected back to it.............
It's setting off alarms on VMWare.

OK, this is so bad now that the boss insisted a case be opened.
CASE NUMBER = 412-326-460

FYI - I've done all the usual - I've throttled bandwidth in IIS to 1.5 meg! I've done what I thought I knew how to do.
The admin states it's saturating our pipe, 5 to 6 meg of a 5 meg pipe - mostly seen as chatter between client and server.
(but we have TWO identical SEPM servers! It appears it's only this one??? But then, VMWARE and Windows don't report any excess network interface use - only about .02% most of the time of a 1gig NIC)

ShadowsPapa,

You cannot open cases on the forums. Please call into support to receive assistance.

Regards,
James

Please let us know what tech support says. To stop the bleeding I would create a policy with traffic shaping enabled on your Internet facing firewall.

No GUP's in place?
Have you tweaked the Java heap sizes?
Change the clients from push to pull, and lengthen the interval as well?

See if you can get a bit more granular in where the traffic is coming from and going to, what ports, and anything else you can dig.  I'm more than inclined to lean towards a setting not set right...

Can you tell if packages are still being deployed?  It's a 90+MB package, sent to each and every machine in the group.  Thats a lot of bandwidth!!!  pre-staging it locally to a GUP helps a bit, but using a desktop software tool like Altiris, LANDesk, BigFix, etc is even better as they have some trick bandwidth tools for software deployment.  Think P2P software, with intelligence to only do P2P on local subnets, so the package is only sent once to each location, and shared amongst local hosts in that network.  

RU6 to RU6a is really not recommended unless you are experiencing one of the two issues it patches.

How many SEP clients point to the SEPM and what are the heartbeat settings?

Get wireshark running to really work out what the network traffic is.

Get Process Explorer from the sysinternals suite and work out exactly what processes are using the CPU time.

Then get back to us with more detail and I am sure we can get to the bottom of things.

Z

Ha-ha - guess you didn't read my message. I DID, and I even posted the case number   ;-)

No GUPs because, and even a SYMENTEC tech agrees, our setup isn't the best for GUPs - that would be more maintenance and overhead than it's worth. We've about 40 offices, with PCs ranging from 3 per office up to maybe 15 in an office. Hardly worth it, plus the fact we can't rely on computers being always on with so many notebooks, etc. The overhead and man-power to setup and maintain GUPS would be crazy.
PLUS - our servers keep SIXTEEN (16) defs versions, and our policies rarely change.
All but about 6 computers total are now on the latest, so update packages should not be rolling out. Those 6 are stubborn and won't update, so I'll have to push to them. Oddly enough, I can push to a whole list of computers and not choke bandwidth.......
Clients are on pull - and the heartbeat is 10 minutes. I set it to pull because with push, it keeps a constant "open connection" and I knew someone would have a cow, even though that worked for months here.
The issue JUST STARTED with RU6a. I went DIRECT to RU6a, skipping RU6, so these two SEPM servers are clean servers, never even had SEPM on them before, it's a clean UR6a install, NOT an upgrade from any other SEP.  One server, SEP1 was an upgrade, it's an older 2003 server but no one is complaining about it.
Here's the kicker - the admin states that SEPM2 is hogging bandwidth. He's using NTOP. He says it's making "too many contacts" yet Windows task manager and VMWare network utilization charts show this server only using .02% of the 1gig NIC connection.
i've got two apps, possibly a 3rd that say the server isn't doing it, but because he sees NTOP stating this server has more "contacts" than any other server, he's hit the roof. I say "utilization" from Windows task manager/network tab shows hardly any use, and it agrees 100% with VMWares monitor, so who is correct?
Well he is because he says so, and has convinced the boss.

So, I either need to prove myself correct, or him correct - and get to the bottom of it. Either way, the charts DO coincide with the SEPM installs, and the traffic pattern is VERY suspect. It's changed now from the original post now that all clients except that handful are upgraded, but it's still a nasty pattern in NTOP.

Yeah, I've tweaked the java heap sizes, but isn't that more for the console?????? The server doesn't use JAVA to deal with clientsor network traffic, correct?

>> I would create a policy with traffic shaping enabled on your Internet facing firewall.<<

Guess I don't know what that's referring to..................... esp since I don't deal with the Internet side equipment.

So far, I've only used the SEP_Support tool exe and sent back the package. I've also emailed them some CAP files from SEP's firewall (using the export function to export the SEP NTP packet log from the server)
I don't know how to read such files, but it just seems like although there's a ton of traffic, the log fills VERY fast, I don't see unusual things, unusual to me anyway. But I'm no pro at reading network sniffs either.

My mistake. I thought you were trying to reopen a case via the forums.

Regards,
James

This wasn't RU6 to RU6a, this was clean servers with no prior SEPM on them.
And the one that was upgraded, from RU5 to RU6a didn't have RU6 on it.
And yes, we're experiencing a lot of issues, including some close to what was described.

OK, all that being said, I think we've found another bug in SEP RU6a.
It appears to be coming from clients that are on PRE RU6a versions and that have not yet upgraded.
There are a stubborn few computers that won't get the RU6a upgrade like MOST of the computers did. Those few seem to be hammering away at the SEPM servers at an amazing and alarming rate! I mean they are really slamming hard - MEGABYTES of traffic!
When we turn off or reboot the computers that are not on RU6a yet, then the traffic to the SEPMs settles down quite a bit, but in a while, it picks back up again as those computers come back online.
They won't quit, they won't give up, and it WILL totally choke a 5meg pipe easily.
This is totally crazy, and has NEVER happened before. IT seems to be a bug introduced with RU6a or RU6 or both.
1. why won't these XP clients and W7 clients (mostly XP) upgrade from the group package? I have never seen such an issue - in the past, each other release has easily updated itself from the assigned package. When I PUSH a deployment, it upgrades!
2. why do they continiously hammer away at the SEPM servers with many megabytes of traffic, literally killing our network?  WOW, filling totally a 5 meg pipe!

Sir,

If you go on your server and look in C:\Program Files\Symantec\Symantec Endpoint Protection Manager\Inetpub\content\ (alpha numeric folder). And from there go into the newest content folder look at the files. Do you have any  0 byte delta definiton files?

Thank you,