Endpoint Protection

Hello gurus (I trust), I'm working in a corporate setting with around 13,000 XP clients and my job is to improve boot times. Sometimes we see boot times taking well over 10 minutes (even 20 in rare cases) particularly on a Monday morning.      Please bear with me. I'm new to this product.

By boot in this case we mean from pressing the power button to being able to read email with Lotus Notes.  We have analysed some of the components of the boot (multiple 3rd party processes run during start up) and find that removing Symantec improves boot by around 1 minute.

Why worse on a Monday morning ?   Well we discovered that users switch off on a Friday afternoon/evening. Symantec distributes definitions over the weekend. We deploy at 08:30.   User powers on after that time. Symantec client receives definitions, processes them, and performs a `quick scan`.  A quick scan is a full scan of all currently running/loaded processes and code.  This is  unfortunate timing because at that time there are many processes running to prepare the environment and they are all slowed down.

Anybody had the same experience ?  What action could you take ?




Secondly, I spoke with 2 supporters at Symantec technical support yesterday and learned something that surprised me.

Typically 3 but sometimes 4 times a day new definitions are pushed out by the vendor. They are received by our local management server. The local management server pushes them out 3 times a day by default. I understand this is configurable.

I've worked with different vendors in the past and today we would normally expect to see a tiny delta pushed out.

The supporter explained to me that the definitions arrive to
C:\Program Files\Common Files\Symantec Shared\VirusDefs\>DATE STAMP<

and are then processed by the client.  3 folders represent 3 days e.g.  right now we have

C:\Program Files\Common Files\Symantec Shared\VirusDefs\20100608.032
C:\Program Files\Common Files\Symantec Shared\VirusDefs\20100609.022
C:\Program Files\Common Files\Symantec Shared\VirusDefs\20100610.048

Each folder is 125 MB.   

Symantec support explained that in our default configuration the client receives 125MB 3 times per day to the date appropriate folder. The running client updates with whatever is new, presumably a tiny fraction of the 125MB.

Question: is this a correct of the process ?
Isn't there a delta method ?   125 MB * 3 * 13000 clients =  4875000 MB  =  4760 GB = 4.7 TB of bandwidth burned per day.   Please tell me this is wrong. I have it from 2 Symantec enterprise support people in India that this is the process but it just doesn't seem the right way to provide updates.


Thirdly ..... if we delete the folders listed above ... and restart the machine I trust the Symantec client will communicate with the Symantec local server and pull down and process what it is missing.


All comments very very warmly received.

THANKS

How much bandwidth is used by a SEP Client in One day ?
SEP will download only deltas.Refer above doc

Edit  AV/AS policy  and go to Administrator defined scans---->advanced Remove both "Run an active scan when new definition arrive" and " Run startup scan when user logged on ". And see whether it is making some difference...

Thanks for the quick reply Aravind. Much appreciated.

I read the linked document. Interesting that the delta is small (less than 10MB). This information differs from our Symantec enterprise support call yesterday.

Question 1)
So why does each folder

C:\Program Files\Common Files\Symantec Shared\VirusDefs\20100608.032
C:\Program Files\Common Files\Symantec Shared\VirusDefs\20100609.022
C:\Program Files\Common Files\Symantec Shared\VirusDefs\20100610.048

contain 125 MB of files ?     Each day ?   Where does 125 MB come from if the definitions/updates are much smaller ?

Changing the scan options is highly desirable. I have no access to the console to test this (different department) but we are in contact with the adminstrative group to test.this.

Question 2
We understand that when a PC is switched on SEP contacts the management server to check for new definitions. If there are new definitions then they are downloaded and installed and Active Scan commences.   I suspect this morning Active Scan (when people arrive for work) is causing a problem during the boot.
Is it configurable that the client will not accept new definitions or process them until say .... 11:00 am or 12:00  At this time the computer is fully initiated and people are starting to go for lunch. A great time for the update and scan.

I guess if we take your advice

Edit  AV/AS policy  and go to Administrator defined scans---->advanced Remove both "Run an active scan when new definition arrive" and " Run startup scan when user logged on "

then we can avoid the start up scanning and then specify definition deployment at 12:00 or "logon plus 2 hours"




regards

DK

Answer to Question 1: Each of those folders represents a full set of definitions, one revision from each day (6/8 rev. 032, 6/9 rev. 022, 6/10 rev 048).  (Does your SEPM update more than once a day?)  When the client checks in, the SEPM builds the delta package based on what the client reports that it has and what the newest available is.  If the SEPM doesn't have a revision old enough to create the delta, the client gets the full package.  You will probably want to increase your number of revisions to greater than 3.

Edit: To clarify, if you see these folder on the clients, it does not mean this amount was pulled down over the network.  As I understand it, the deltas are incorporated into what the client already has and copies it into another full revision.  That revision is what is presented to the SEPM the next time the client checks in for an update.  I hope that makes sense.

sandra

Sandra, thanks for your contribution. This casts further light on the issue.


For clarification: the three folders I listed above are on the XP clients, not server. I currently have nil access to the server.

> Does your SEPM update more than once a day?
I need to check with colleagues.

> You will probably want to increase your number of revisions to greater than 3.

You refer to the number of revisions on the SEPM server I presume ?    I don't know how many revisions are stored on the server. I can find out on Monday morning (I'm in central Europe).

I suspect the majority of clients are notebooks and are switched off over the weekend.


>As I understand it, the deltas are incorporated into what the client already has and copies it into another full revision. 

My expectation is that the client would only require one copy of definitions, not three. Why would the client need 3 folders ? No doubt there is a perfect reason but I don't see it right now.

>That revision is what is presented to the SEPM the next time the client checks in for an update. 

So what exactly is presented to the SEPM server ? A copy of all the folder is presented  or a refence number or a hash or ?

Finally ..... on the client folders they are named by date and revision number. What is the revision number? In the last 3 days we went from date plus rev 32 to rev 22 to rev 48 on a machine that has been switched on pretty much all the time (my own workstation). 
Do we need to be aware of the revision levels ?  I'm speculating that on a busy day the revision level will be higher than a 'quiet for viruses' day and the folder name simply reflects that. 


And finally again ................ 

we would dearly love to reproduce the Monday morning load by taking the definition level back a few days. Is there a way to regress the definition level on a client by some days ?  We would do that and then restart and with our tools measure how much time is added to the boot/logon processes when ACtive Scan is running. Is this doable?

Many thanks


Wishing you all a good weekend.

DK

You refer to the number of revisions on the SEPM server I presume ?

Yes, sorry.  Number of revisions stored on the SEPM is found under Admin > Servers > Local Site > Edit Site Properties > LiveUpdate tab > Disk Space Management for Downloads.  The greater the number of revisions, the more hard drive space is used on the SEPM, but the greater the chance the SEPM has what it needs to build a delta and NOT send a full definition package over the wire. 

I suspect the majority of clients are notebooks and are switched off over the weekend.

That does explain it.  If you have machines that shut down over the weekend and only store 3 revisions on the SEPM, those machines will likely need to pull down a full definition package when they come online on Monday.

The LiveUpdate schedule for the SEPM is on that same tab.  Default is every 4 hours.

Why would the client need 3 folders ?

I believe it is in the event you need to roll back, but I am not certain on that, and I can't find anything on a quick search.

So what exactly is presented to the SEPM server ? A copy of all the folder is presented  or a refence number or a hash or ?

This is highly simplified, but it's information exchanged during the heartbeat process via the information contained in the sylink profile.

Is there a way to regress the definition level on a client by some days ?

Title: 'How to Backdate Virus Definitions in Symantec Endpoint Protection Manager'
http://service1.symantec.com/SUPPORT/ent-security.nsf/docid/2007111515160948

From the look of it, you can only revert to what the SEPM is storing.

This may be useful to you:

Title: 'Disk Space Management procedures for the Symantec Endpoint Protection Manager'
http://service1.symantec.com/SUPPORT/ent-security.nsf/docid/2009091802033648

Thanks,
Sandra