Runaway Upgrade Processes (obscene bandwidth consumption)
So our network team recently noticed an interesting trend in our network throughput from our AV server. They noticed that our servers where generating a combined, and sustained, throughput of about 500 Mbps between 1am and 7am.
Well that time frame matched an upgrade schedule that we performed quite a while back. But that didn't seem right since almost all of the clients completed their upgrades a long time ago. We have left the schedule in place to catch a very small percentage of straggling clients. However when looking at this more deeply it appears that a small handful of machine (about 30-35) are in a very problematic state. They can't seem to complete the upgrade and it is failing for some reason (we are still trying to determine why). But instead of failing constructively and letting us know they have done so, they just seem to keep trying over, and over and over, etc.
By the end of each 6 hour window, each of these clients has downloaded between 35 and 40 GB of data (install package data). Yes, that is not a typo, I did mean to say 35 to 40 Gigabytes. Between these 30 some odd machines they are pulling a total of about 1.3 Terabytes of data from my servers. And apparently they have been doing this nightly for quite a while. Thank goodness this is all over a LAN.
Have anyone else seen this and if so, where you ever able to determine the cause and engineer a fix?
Thanks for your time.