I have a 2 SEPM load balanced environment. The 2 SEPMs share a DB and everything was humming along great using RU6MP3. About 2 weeks ago now I upgraded the SEPMs to RU7 and ever since then I have been experiencing some crazy issues.
Starting the Monday after the installation I had a DB lock and the SEPM service on my primary SEPM would not start. After working with support, we managed to correct this issue and the SEPM was back on-line. However, this was only the beginning of my fun as the environment has been running in a degraded state ever since.
Clients would fall out of date and update at an extremely slow pace for no reason, until I realized that that agentinfo folder on my secondary SEPM was filling up with hundreds of dat files. To combat this, we stopped the secondary SEPM and deleted the files and restarted the service. After this the dat files built right back up. Some more digging revealed that there were some *.err files in some of the log directories, so we deleted those as well, restarted the service and everything appeared to be working. The dat files were no longer building up in the agentinfo directory. As a result, we left things to bake over the weekend, checking in periodically and still the dat file numbers remained low, 1-5 files at a time.
But then Monday came and things went crazy again with the dat files building up and accumulating into the several hundreds. During this clinets would only slowly update and even clients talking to the primary SEPM would not update. As a test, I would stop the SEPM service on the secondary after which I would see a flood of clients flip to the latest definitions. We have roughly 17,000 endpoints and every few minutes I would see a few thousand show as updated until eventually all but a handful remained out of date.
Working with support, we have not been able to discover anything in the logs, the SEP support tool would generate corrupt data and so it was determined we should re-build the secondary SEPM as that appeared to be the culprit in this all. When the secondary was on, clients remained out of date, but then stop the services and clients update. Easy...
That is of course until this morning. Today when I stopped the SEPM services on the secondary, my clients have remained out of date. By out of date I am saying 2011-08-04 rev. 021, vs. 2011-08-04 rev. 048, but still it has been 3 hours and still only about 2000 endpoints have updated when normally by this time I would be mostly updated.
I am simply at my wits end and have no idea where else to go from here, so I am really hoping that someone else out there has experienced something similar to this and may have some suggestions as a possible fix.
Thanks