Video Screencast Help

Runaway Upgrade Processes (obscene bandwidth consumption)

Created: 04 Jan 2012 • Updated: 04 Jan 2012 | 8 comments

So our network team recently noticed an interesting trend in our network throughput from our AV server.  They noticed that our servers where generating a combined, and sustained, throughput of about 500 Mbps between 1am and 7am.

Well that time frame matched an upgrade schedule that we performed quite a while back.  But that didn't seem right since almost all of the clients completed their upgrades a long time ago.  We have left the schedule in place to catch a very small percentage of straggling clients.  However when looking at this more deeply it appears that a small handful of machine (about 30-35) are in a very problematic state.  They can't seem to complete the upgrade and it is failing for some reason (we are still trying to determine why).  But instead of failing constructively and letting us know they have done so, they just seem to keep trying over, and over and over, etc.

By the end of each 6 hour window, each of these clients has downloaded between 35 and 40 GB of data (install package data).  Yes, that is not a typo, I did mean to say 35 to 40 Gigabytes.  Between these 30 some odd machines they are pulling a total of about 1.3 Terabytes of data from my servers.  And apparently they have been doing this nightly for quite a while.  Thank goodness this is all over a LAN.

Have anyone else seen this and if so, where you ever able to determine the cause and engineer a fix?

Thanks for your time.

Comments 8 CommentsJump to latest comment

pete_4u2002's picture

The figures seems to be on higher side.

What is the version of SEP ? Did you check if delta package being upgraded?

la_ripper's picture

First, you would need to identify what traffic is this....

 

Use a network monitor tool and check on what port is the traffic getting generated on .

Don't forget to mark your thread as 'solved'  or vote with the answer that best helped you!
 

Simpson Homer's picture

You could use Wireshark and check the traffic.

StiflingCobra's picture

I would try exporting the package you are deploying from the SEPM and try installing it directly on one of the problem clients and see if this generates any error's you can see during the install/upgrade.

Do you have a proxy server configured on the SYSTEM account on the machines that have the problems??  This can cause the traffic data to be malformed and the client machine to reject the data over and over again.

Check the following two registry keys to see if this is the case...

HKEY_USERS\.DEFAULT\Software\Microsoft\Windows\CurrentVersion\Internet Settings\ProxyEnable 

This should be set to 0 unless you have sofwtare that specifically requires the System account to have a Proxy Configured.

HKEY_USERS\.DEFAULT\Software\Microsoft\Windows\CurrentVersion\Internet Settings\Connections\DefaultConnectionSettings   

This should be deleted if you have set it to be 0 above.

Reboot the machine after changing the above then see if it updates ok.

DanielW 2's picture

I'll try the package idea and see if it gives us any clues.  It's not a proxy issue (none of the machines we have looked at so far have had a proxy configured).

Jason1222's picture

What version of SEP 11 are you running and what is the OS of the problematic machines?  32 or 64 bit?

Thanks.

mon_raralio's picture

You should also check the temp folders: user/temp and windows/temp for any installers.

They could probably be cleaned using Windows Disk Cleanup.

“Your most unhappy customers are your greatest source of learning.”

peter ashley's picture

I would appreciate your working with support on a case, so that if a product defect exists for this use case, it can either be identified as a duplicate or entered and fixed.