If you're in Push mode, that means the clients keep a constant connection to the SEPM so they can update policies as soon as a new one becomes available. This is the likely source of your problem.
With that many clients, this could cause some trouble. I would strongly suggest switching to Pull mode and setting a heartbeat of 15-30 minutes, especially if you don't update policies very often.
I would also suggest setting up GUP at the local site(s) to handle content distribution.
Here is a very similar thread here, please read thru it, especially the solution:
http://www.symantec.com/connect/forums/pull-mode-push-mode