Endpoint Protection

Starting with the May 2nd, 2014 virus definitions, the environment I administer has had serious difficulties in updating virus defs in a normal manner. By this I mean, all of the endpoints are checking in within 24 hours but it is taking more than 24 hours for them to update. 

Example:

I have ~150,000 endpoints

Today my endpoints are updating to 6-1-2014 rev. 32 (we only update definitions once a day)

At the end of the day when we are going to update to 6-2 definitions, I'll still have 5000 or so endpoints on 6-1 definitions. They will stay on that date for 3 or 4 days slowly updating. On 6-2 I'll have 5000 on 6-1, on 6-3 I'll 4000, on 6-4 I'll have 3000 and so on. The endpoints are updating, they're just not doing it in a timely manner.

Where this gets really bad is the fact that its every definition. So I'll have 5000 on 6-1, 4000 on 5-31, 3000 on 5-30, 2000 on 5-29, 1000 on 5-28. In reality they don't update that quickly so what I end up with is something like 18,000 endpoints that are not within a 3 days of the current defnitions. This is inordanitely high as prior to May, we were looking at about 3000 endpoints out of date regularly.

Any ideas?

Thanks

They coming back to GUPs? How's the load on your SEPM?

A large portion are coming back to GUPs, I have about 2200 GUPs. And 14 SEPMs (in 2 sites). None of the SEPMs are overloaded. The GUPs are all updating but exhibit some of the same behaviour to a lesser amount. I'll have about 5 on N-3, 90 on N-2. So not bad, each GUP is responsible for about 20 endpoints so even with 100 GUPs not running today's or yesterday's defs, that is only going to account for 2000 endpoints.

Have you tried enabling sylink logging on an affected client to see what it reveals (if anything)?

No I have not. It is an idea to try. Just to cover the bases. I'll see if I can get that going. 

I don't have access to most endpoints. I do have access to the GUPs. They are all checking in during any given 24 hr period. And when I log into them they're all connected to a SEPM. 

Should give an idea, will just be more info in the GUP log than if it were a client.

Ok, I got a sylink debug from one of the 5 GUPs out of date today. This is exhibiting the same symptoms as the other endpoints. 

Attachment(s)

Sylink_3.7z   14 KB 1 version

I've been tracking my roughly 2200 GUPs for the past 2 weeks. I'm seeing the same thing with them as with the endpoints as a whole. 

I normally check them at 10am each day. Yesterday I did a second check at 4pm. This showed that most of the GUPs that were not up to date were now up to date as you would expect. There were about 10 that were not. 

This morning I have 10 that are not up to date but none of them match yesterday's report. As this has been happening for at least 2 weeks, I'm left to draw the conclusion that it is taking about 3 days for everything to update which is crazy.

I found the problem. One of the 14 SEPMs in the environment had a rogue BCP.exe process from 5/5 that was stopping it from handling incoming log files from endpoints. Would never have discovered it if the files hadn't finally filled up the drive. After deleting over 150,000 stale log and err files, restarting the services, traffic began flowing again. Now 2 days later instead of 18,000 out of date I have 3500 which is more in line with what they were before this happened.