How many content revisions are you keeping? See under Admin > Servers > Local Site > Edit Site Properties > LiveUpdate > Number of content revisions in your SEPM console.
If you have only 3 or 4 revisions, it's totally normal that some clients are pulling full content (as you say, about 160 MB). A client whose youngest content revision is older than the oldest content revision of the SEPM has to pull full content. That can happen after a long weekend or a short vacation. The reason is that SEPM can only produce tiny content packages (delta files) if the content revision times of SEPM and client overlap..
Symantec is delivering about 3 revisions per day. If your SEPM is downloading every 4 hours, you get 3 revisions per day. If you want to cover a week, you have to save 21 content revisions. Of course, if you are just updating once a day, 7 content revisions are enough.
The more content revisions you save, the rarer your clients will pull the full download.
But be aware that you need about 18GB for 21 content revisions (and the size grows slowly, but constantly). It's a trade-off.