Endpoint Protection

you turned off this already?

http://www.symantec.com/business/support/index?page=content&id=TECH106098

and users do not have any drives mappped? or does it only happen when drivers are mapped on to their profile?

Please also test disabling the rescanning of the file cache on arrival of new definitions:

http://www.symantec.com/docs/HOWTO27136

This is found in the Virus and protection policy, under Auto-Protect -> Advanced -> File Cache -> Untick the option to "rescan cache when new defintiions arrive".

Note: this rescanning of the cache is only a performance setting, and does not affect security.

Essentially, all the file cache does is keep track of files SEP believes to be clean.  The "rescan on new definitions" is there just to front-load the effort of making sure these files previously thought to be clean, are still clean.  Without the rescan, SEP just waits until the file is accessed again to scan it with Auto-Protect.

First off thanks for the very fast reply!

Unfortunately active scans on new definitions have been disabled from the get-go since implementing SEP several versions ago. Though users get their group- and personal shares mapped by our login script, network scanning on clients has been disabled; all the data being read is local.

The only noticeable change we've had on the software-side when this behaviour started was the upgrade to version 12. I'm currently waiting for another chance to tap into a system while the problem's happening to see which subprocess under 'System' is causing this, last time I checked it was ccSvchost.exe, but I want to double check to be sure.

"We've tried disabling file cache and rescanning the cache to no avail." 

I Should have been clearer pointing out those were SEP-client options we've disabled but we'v ebeen there, tried it, with unfortunately no change. Those were some of the first things we checked while trying to pinpoint the cause ourselves. Even tried things like disabling the Windows 7 SuperFetch-service, thinking SEP might've been confused with it's precaching algorithms. That appeared to be a dead end as well.

In that case, we'll have to go back to more general troubleshooting.

Can you provide more information about the affected endpoints, and how these appear to differ from those not experiencing the issue?  Also, have you tested going through the Auto-protect settings to amend it to "Scan on modify" to confirm if it is auto-protect that is causing the issue?

What is causing the actual disk reads?  Other than an actual scan, there's not really any other reason for SEP to read the disk.  Have you checked out the below article on unexpected scans?

http://www.symantec.com/docs/TECH171212

call support they have a tool which would remove all the previous scans. run it.

cleanwipe. install a fresh copy. 

"We've already tried using CleanWipe to remove all traces of the upgraded client, and completely reïnstalling the latest package from scratch."

I know the starting post is a long one, but that's only because we've been sinking our teeth into this for a while now, and have tried a lot of different things. Getting CleanWipe from support was one of the first.

Did you make sure that all the Symantec registry keys are removed after cleanwipe is run?

Have you had a chance to check out the link I posted earlier regarding unexpected scans?

http://www.symantec.com/docs/TECH171212

I'm working on it; interesting article by the way. Sadly only one GUID is listed under custom scans, with the 'Enabled' value under 'schedule' set to 0.

I'd like to add two exported registry-keys (HKLM\Symantec and HKLM\WoW6432Node\Symantec) from one of our troubled clients in case people are interested but they're pretty big and I don't want to make sensitive info public; is there anything I need to mask outside of company and user names, and IP-ranges?

As I said earlier there is a tool with support which will delete scans. I would like to run that first and install a fresh copy.

Oh I thought you meant CleanWipe itself, but it seems you're talking about a different tool.. do you happen to know it's name so I know what tool to ask support for? We've only gotten CleanWipe.

Yes, also a freshly deployed system exhibited the same 'phantom scanning' issue. 

I just accidentally ran into https://www-secure.symantec.com/connect/forums/disabling-triggered-scans-sep-121, where someone seems to have the exact vague problem, no other user profiles with scheduled scans here though :(  

I'm afraid without knowning what the keys were I can't really advise on what to blank out, as that Symantec key can potentially contain all Symantec Products.

As far as the sole identified GUID goes, if you highlight the HEX numbered key immediately below "Customer Tasks", you should be able to see a "StausDialogTitle" on the right.  What does it say?

I dont remember it either :) was something like SymDelscan or something. give them a call or create an online case, they should send that to you.

How to create a new case in MySupport

http://www.symantec.com/docs/TECH58873

How to Create and Validate a SymAccount for using Symantec's MySupport

http://www.symantec.com/docs/HOWTO31127

I just created a new case asking for the SymRmvScan-tool, referencing this topic as well. I fear this might not be the solution however, since the same problem exists after installing 12.1.3 on a freshly deployed, clean system, where no data of previous versions exists.

The custom task's StatusDialogTitle is 'Active Scan upon Startup', but in the Schedule subkey the Enabled-value equals 0. 

Could it be that the root of the problem lies in SEPM, since it's there that we generate the setup-packages installed on every managed client that exhibits the issue? Our SEPM-server has been upgraded a couple of times over the last few SEP-versions as well, to prepare for our clients.

As there do not appear to be any configured scans on the endpoint, then I would assume (at the moment) that something else is causing the disk I/O, but that the presence of Auto-Protect is exacerbating the matter.

Did you get a chance to test my earlier suggestion of switching Auto-protect to only scan on modify?

Have you ruled out Windows Indexing and the like?

There is only one time the client talk to SEPM. During the heartbeat. Have you set it in push mode or pull mode? Any list of exclusions. I head people saying if Exclusion are more the performance would take a leap. If you have scheduled  or assigned any package to the group(Autoupgrade) they might trying to download and run the install. if failed they would try installing again. but thats related to network usage.

Can you confirm if there are any packages assigned. Any machines in the group which do not face this issue? any diff in Service pack, OS, time zone ( which would create missed scan value and run scan over and over again)

Defs are distributed in Push-mode, packages are assigned, autoupgrade is applicable but also newly deployed systems which don't need an upgrade exhibit the symptoms.

I've just e-mailed the first SymRmvScan-logs to technical support; I already noticed a peculiar "Heuristic ReDetecion"-scan that was deleted. I'll be out of town until Monday, but will definetely check back as soon as I can. 

So that did delete few scans in the registry, lets see how it  progresses. Keep us posted.

We are experiencing the exact same issue with 12.1.3, I'll also be interested in your finding!

Just an update; unfortunately no efforts up to this point (running SymRmvScan-script, disabling client-side options, removing Network Threat Protection) have resolved the problem. Technical support is currently analyzing our forwarded logs, I have to say the servicedesk people from India are very polite and quick to respond, but sometimes the language barrier is an issue; our case had been closed and marked as 'resolved' while I asked for a few more days to test their advice. So on with a new case we now go, I'll keep updating in this thread as soon as I know more.

It's good to know we're not the only ones experiencing this, I hope you'll post here if and when you happen to come across a solution.

Hi there,

that's bad news. And still no solution...

We are making the same bad experience right now with all our 200 clients on which SEP 12.1.3 is installed.

Right after the new definitions are downloaded (in our case a mix between external LU Server and SEPM) the systems are completely unusable because of extreme I/O access (harddisk). It's not a cpu problem.

As you said, whatever settings you try to change, and there were a lot of ideas posted here - nothing stops SEP from doing what looks like a "scan" or "decompression" after the definitions are downloaded.

In our case it takes 10min of heavy load with unusable systems, driving the users completely mad.

Why isn't Symantec reacting on this huge issue?

Dear MOWCJ,

We're still testing and keeping our fingers crossed here, but it looks like recreating our client policies and explicitly disabling scanning on new definitions, scanning on user login, and disabling the file cache and shared insight cache completely seems to have finally done the trick for us. I hope the same will count for you. If not, we were able to submit our current policy-file to support for analysis on our request, maybe that's an option for you as well.

Please do let us know if the above helps you, so other admins can benefit as well. Thanks in advance.

Hi,

we are trying to rebuild a policy from scratch to see if it makes a difference.

I can't believe we are the only ones with these headaches. Let's hope they are working on a fix...

Hi,

I faced in the past this kind of issue and I fixed it by modifying the policy with these three steps:

1. "Run an Active Scan when new definitions arrive" > UNCHECKED

2. "Rescan cache when new definitions load" > UNCHECKED

3. Quarantine/When New Virus Definitions Arrive > Do nothing 

I'd also be interested in any new results.  I'm experiencing simlar issues.  I have 35 workstations that experience no additonal system memory increases, but the disk drive activity is out of control, and the pc's crawl to a halt while something in the background hoard the pc.  It occurs randomly but usually between 9-11am to all 35 pc's.

Still facing this annoying issue with 12.1.3001 clients...

We rebuilt the policies from scratch, and made sure the "rescan" options and so on were unchecked when new definitions arrive.

Did not work out. We are still facing that horrible I/O activity

(the sylink.xml was definitely updated)

I noticed when looking inside liveupdate of the client that there was only 1 set of definitions.
Can't change this because with most clients we do not want the updates to be distributed by the SEPM. So direct Liveupdate to Sym. updateserver is used - with no option to define how many definitions are to be kept on the client?

Maybe a delta problem in the definitions when directly downloading?

This is annoying...

It occurs at that moment the def.updates are being distributed (probably you set 10am +/- 1 hour)
When they land on the client the crazy I/O activity occurs (let's call it 'reorg' because we still don't know what is going on at that moment)

it looks like problem has been solved in SEP 12.1.4013. Any way now it works fine in our network.

Hi,

just note that the issue or residual scans was related to system migrated from SAV to SEP, I don't see such scenario and a request for that tool since 3-3.5 years already, i.e. it is so rare to see SAV installations around that I would not be so sure this is the final solution without questions and doubts.

it looks like problem has been solved...

Sorry. It was wrong information.

System-process are doing something after each Live Update, so my workstation works very slow for 30 minutes.

SEP v12.1.4013

Which system process?

Hello Yaroslav,

I agree with you - the problem remains the same. I expected it to be solved, but you guessed it...

First of all: I opened a case 1 week ago and now we are trying to found solution with Symantec support.

about your question

Which system process?

System process with name "System". Its PID=4. When I open Sysinternals Process Explorer - I see 2 thread "SRTSP.SYS+0x519d0" inside of it.

After each Live Update - it (the System process) begins to read a lot of data from my HDD. My computer works extremely slow at that time.

When I open SEP client and open Auto Protect Scan Statistic - I see that SEP are scanning my hard drive.

After about 30 minutes of that I see that it stopped scanning - at the same time the System process stop to read data from HDD.

That sounds like the real time storage protection service.  Can you verify you have the "rescan file cache" option disabled (as per my first post) and confirmed as applied to your clients (check policy serial number)?

Thank you, for your answer

1. Disable "rescan file cache" was first advice that I get from support. But it did not helped

2. Policy number are equal on SEPM and SEP Client: 78B3-01/24/2014 13:05:30 451

3. I did export policy... it is part of it:

<FileCacheOptions
ApplyModeFileCacheEntries="USER"
ApplyModeFileCacheSizeControl="USER"
ApplyModeRescanOnDefReload="USER" FileCacheEntries="30000"
FileCacheSizeControl="DEFAULT"
LockFileCacheEntries="0"
LockFileCacheSizeControl="0"
LockRescanOnDefReload="0"
RescanOnDefReload="0"/>

4. Yes, it looks like I have problem with "rescan file cache", because at problem time - SEP scans about 30'000 files after each Live Update.

Can't understand how to fix it.

I have an idea. I will try to "close the lock", update policy on a client and wait for new Live Update.

Hello Yaroslav,

I am sure that doesn't change anything. The lock is just for user-control yes/no? - and you saw the policy had the setting inside, so...
What's interesting is that we noticed the I/O activity only with XP systems- for now
Did you notice that behaviour also in Win7 systems?

OT: did you notice this thread is not being updated without users being logged in in symantec connect? I wonder why? (without being logged in it shows me 34 comments, logged in I see 41 comments?  )

What's interesting is that we noticed the I/O activity only with XP systems- for now
Did you notice that behaviour also in Win7 systems?

some Win7 users also complains but not much so I did not investigate their workstations, it looks like only WinXP users have really big problems with SEP client

I am sure that doesn't change anything. The lock is just for user-control yes/no? - and you saw the policy had the setting inside, so...

Any way I will wait for new virus definition update. Who knows...

Also I sent log files to support and I am waiting for answer soon.

OT: did you notice thread is not being updated without users being logged in in symantec connect?

Yes. It looks like caching or (more probably) waiting for moderator check

As advised - I tried to disable "Rescan cache when new definitions load" and "Run an Active Scan when new definitions arrive" - but it did not help.

It looks like I found solution yesterday.
Yes, it was "Rescan cache when new definitions load". I disabled it in SEP Manager - Clients got new policy, but (I don't know why) used it's own (SEP client) settings and ignore policy settings.

I don't know is it a bug or feature. Only after I "closed the lock" - clients used this settings.

After that - 2 most problem workstation works very well after Live Update. Before that I had problem after each bases update.

I think it was not a bug

I foundout that I have "Client control" on my SEP policy.

http://www.symantec.com/business/support/index?page=content&id=TECH185903
(see 2. Locking the client user interface)

Probably it is not default settings.

Yeah, if it persisted, I was going to ask you to check out the RescanOnDefReload regkey to see if it was correctly applying your policy.

As you say, the reason it hadn't applied your policy is because Client Control had been given and the recscan setting had been applied on the client side at some point.

Glad you found the issue in your environment, hopefully this helps the OP too.

SEP 12.1.4 Small Business Edition - approximately 50 managed clients

We have been experiencing the same issue on most if not all of our systems; heavy system lag mainly due to HD I/O after a definition update. This lasts for an average of half an hour, significantly longer when users attempt to operate their systems normally during this time (such as opening Outlook, databases, or anything that also has moderate to heavy HD usage).

This is a very annoying issue, especially for laptop users that bring their machine's home for the night. When they plug their machines into the network each morning, the computers download the updates from the previous night (for numerous reasons, the server has to do the definition updates each night) and do this unstoppable rescan each and every day before becoming usable.

Turning off Tamper Protection and manually changing the RescanOnDefReload registry key to 0 does seem to eliminate this problem on initial testing using manual definition updates. 

However, this setting resets each time the computer is rebooted, even when not in contact with the manager. Additionally, SBE does not have this rescan setting in the manager nor in the client UI.

Is there anyway to manually adjust the SEP client group profiles on the server to push and keep this change to the clients? At the very least, is there a way on each machine that this registry setting can be kept from resetting on each reboot?

Hi ERCA,

How many content revisons do you save in SEPM. If the count is more then it will delta and the file to download will be of a lesser size.

As of freezing the registry, you could block the access to 'RescanOnDefReload registry key' through ADC and do not allow any program to access that including SEP process.

Hello ABN,

The delta versus full definition download is not our current issue. Our revisions are currently set to 5 with a single update each day. We have tried 10 as well which just bloated the server system drive.

The long boots occur during both delta and full updates (if a computer is out of the office for a week it will do a full update which does take a little longer) and the heavy disk IO happens again if additional (definitely delta) updates are run later in the day.

As I touched on in my previous post, I have tested by running manual definition updates with the registry setting left at 1 and updates with the setting changed to 0, and this does appear to drastically reduce the amount of time that IO is high.

Regarding the registry key, I did not word that correctly. Symantec has a sizable number of registry values within the RealTimeScan registry key (where the RescanOnDefReload value resides) and a number of subkeys, so I did not want to lock the entire key. I do not believe you can lock a single registry value.

Rather than fighting my antivirus with Windows locks, I wanted to know if there was a way to edit a Symantec config file to stop it from overwriting that one specfic value or redefine the default for that value to 0 so that it would rewrite the value the less system destroying way.

As I have read in previous posts in this thread, this option is available in the non-SBE SEPM, so I was hoping there would be a way to force this change even though it isn't available in the UI. SBE users should not be saddled with this just because they are smaller customers. I would like to stay with Symantec, but if this cannot be fixed we will have to change to a different solution.

Thank you ERCA for the detail. At this point a Future Modification Request is the best way.

You may post the setting as an IDEA and we can vote it to get more attention. May be they woudl add that in the next release. Fingers crossed

Thank you ABN for your assistance and for the previous users whose tips led me to the correct registry setting.

I have managed to locate the group config files on the SEPM server and have begun testing on manually adjusting this RescanOnDefReload value within these config files. So far this seems to be a functional workaround for this issue. One thing I have yet to test is whether I (likely) will have to manually re-adjust these config files if any settings are changed in the UI to the SEPM computer groups.

I would think that the FileCacheEntries setting could be dangerous if the RescanOnDefReload is disabled. How long will those cached files go untested against newer definitions? Do I understand correctly that the cached files get skipped during AutoProtect scans? Would those cached files be rescanned during each full scan?

Hello emmanuel944,

In our case, the files to adjust are located in the following folder on our server that runs the SEP manager software:

'C:\Program Files\Symantec\Symantec Protection Center\data\outbox\agent'

Within the 'agent' folder there are folders with apparently random hexadecimal names such as 'E6B27A700C00000402BF6482CA2B67C0'. We have one of these folders per computer group we have within the SEPM plus one extra for the encompassing "My Company" container.

Within each of these folders, there is a file called 'Profile.xml'. We opened each of these Profile.xml files in WordPad. On the 4th line, it contains 'Name=' and then the name of the computer group for which this file appears to contain the settings.

Doing a search within the file for 'rescan' provides 3 results, all on the same line of the file. The last entry on the line is 'RescanOnDefReload="1"/>'. We changed this '1' to a '0'. As we are unsure how drastic a security risk this might pose for the Cached Files, we also changed value for the 'FileCacheEntries=' to '0' (this is a few items earlier on the same line).

After making these changes, we saved the Profile.xml files. Eventually the server pushed these out to the clients and everything seems to run much smoother ever since.

Keep in mind that making changes in the SEPM program overwrites the Profile.xml files, so we have to double check them after making adjustments in the manager software.

This is how it worked for us, unfortunately I cannot say whether this will work for others. Good luck!