Endpoint Protection

Dear friends, we are running Symantec Endpoint Protection 12.1 on Windows Server 2008 R2 File Server but, randomly, the network shares hangs and the network applications stop responding.

The current configuration is:

• Symantec Endpoint Protection
  • ​Enabled Features: File System Antivirus only
  • Version: 12.1.4100.4126
  • Exceptions: Correctly defined to exclude important folders, files and processes from any kind of scan
  • Scan: Only modified files are being scanned
• Windows Server 2008 R2 x64 with Service Pack 1
  • Hyper-V Enabled
  • File Server Resource Management is Enabled
  • Symantec Endpoint Protection Manager is Installed (ASA database is defragmented)
• Server: IBM System x3550 M4
  • 32GB RAM
  • 2 Processors
  • RAID Volumes: RAID 1 - Operating System, RAID 5 - User Files, RAID 1 - Virtual Machines (all disks are 6Gbps SAS)

Beside the slow access to the network shares, when the server is restarted the server stucks during shut down. A hard shutdown is needed.

When Symantec Endpoint Protection is removed from server, the problem does not happen.

Right now, after a reboot on last saturday - 14th june, 2014 -, the server is fully functional and I'm waiting for the problem to happen again to try to use Process Monitor capture server activity and confirm that the problem is being caused by SEP (http://blogs.technet.com/b/markrussinovich/archive/2010/12/07/3373406.aspx). I also enabled VPDebug Logging (http://www.symantec.com/business/support/index?page=content&id=TECH102939) to check if SEP shows some sort of error message on it's working log.

Any help will be appreciated.

Best regards to everyone! 

Disable only the SEP firewall and try again

other thing you'll need to do is call support, they will have you run the symhelp tool and re-produce the issue. It will collect the info and send to support. You will also need to run a packet trace while re-producing the issue.

We are having this EXACT same issue right now and have been fighting this.  We have gone so far as to create a VM dump for Microsoft and have a case open with Symantec as well.  We are not running the SEP firewall, just default windows firewall.

Microsoft suggested a hotfix specifically for the srv2.sys driver, but we tried that on of our failing servers late last week and it just crashed tonight.

One thing you will find on these servers I bet as we did, if you revert to SMB1 shares (connect to shares from XP or Win2K3) you'll find that all the shares are there an still work.  This seems to just be limited to SMB2 shares.

If we find anything out I will post here a well.

This has been an issue in the past with SEP. I can remember once related to the firewall for sure and auto-protect beig another.

Good route on calling Symantec, they should be able to get the root cause, however, it would be a code fix most likely to truely correct the issue.

We have had this EXACT same problem, we have logged support tickets with Symantec to no avail. They want to reproduce the problem, but we cannot it is random.

We have had the problem occur on 2 different Windows 2008 R2 clusters running the file services role. And a standalone 2008 R2 server as well.

We have a near identical setup to Roger above in terms of Symantec version and Windows OS. We are also ONLY running the AV scanning features. The only difference is that we are running VMWare instead of Hyper-V.

Due to the critical nature of our file services clusters and the fact that it happened 3 times on us on one of the clusters, we temporarily uninstalled Symantec and the problem has not occured since.

We noticed the problem right after installing the latest version, which for us was a small version upgrade from what we were running before.

Mark, yeah we went from 12.1.4013.4013 to 12.1.4100.4126 last Sunday during our production server windows patching (staged/pushed install from the SEPM for upgrade - restart needed) and started having the issue w/ SMB2 shares on most Windows 2008 R2 file servers.

Can I ask what changes you guys have tried in your troubleshooting?

As it's been narrowed down to the AV only component, have you tested:

• enabling/disabling the network scan option within Auto-Protect, and the further options it contains (i.e. trust remote auto-protect and the network caching)
• disabling all unscheduled scans (active scan on def updates, file cache scans, etc)

What I find interesting is the different environments.  The OP's setup suggests this is happening on a physical Hyper-V VM Host (please correct me if I'm wrong), while markcloud's description suggests his file server is a VMWare virtual guest.

I'll be keen to see how this progresses.

We've sent a full memory dump to Microsoft when the issue occurred. From that memory dump on an affected Windows 2008 R2 server, they were able to determine that "...Most of the srv2 worker threads are waiting for an oplock. Hence new SMB2 request from the clients are not being served...It looks like a deadlock is in the symantec driver SRTSP64.sys..".

Again, we only went from v12.1.4013.4013 to v12.1.4100.4126 and started having these issues. There was no changes made in SEPM policies, settings, etc. Just a slight version upgrade in the client.

In the interm, we've done one of the following:
1. Stop the SMC/service on the server.
2. Remove the SEP client entirely.
3. Run CleanWipe, restart, install pervious version of 12.1.4013.4013.

We are only running the 'Basic Install Feature Set' (Virus, Spyware, and Basic Download Protection) features on our Windows servers.

As a note about Hyper-V vs VMWare:

Our Windows Clusters are on VMWare and we have the issue arise there.

Our standalone Windows 2008 R2 machine was a physical server and we saw the issue there as well.

We tried disabling auto-protect, def update scans, cache scans, etc. We thought that was the problem early on due to high I/O of definition files during the outages, but after disabling all of the above, we conitnued to see the problem, therefore we did a full uninstall of the SEP client on our critical file clusters (not happy about doing that, but we saw no other option).

Also we've noticed the problem as soon as 8 hours after a reboot on a very lightly used server (maybe like 5 SMB sessions opened) as well as on a heavily used server thousands of SMB connections. So it doesn't appear to be load/uptimed related as was another one of our theories.

Dear friends, before saying anything, I'd like to thank you for your opinions.

In a quite similar situation as Jeff, we upgraded from version 12.1.4013.4013 to 12.1.4100.4126 and started having these issues.

The server spent almost a month without Symantec Endpoint Protection installed and nothing happened. The server worked very well during this period.

I define every single best practice documented by Symantec and other software developers (Microsoft and Computer Associates) to avoid performance problems. For example:

1. Important folders and files from operating system and applications are removed from all types of scan.
2. Network scan is disabled on file servers and workstations.
3. Antivirus file cache is disabled on file servers.
4. Only file system antivirus is enabled on servers.
5. Only modified files are scanned.

As I told you, since last saturday it didn't happen again.

We are now waiting for the problem to happen to use the procedures I told you on the beginning of this thread. With the Procmon Analysis in my hands proving that the problem is being caused by some sort of Symantec file system driver - which due to the tests made I have no doubt, I'll be able to call Symantec and open a case with information in hands for their analysis.

I'll keep you informed about any updates.

Best regards for everyone!

Does it happen when definitions are updated?

We didn't observe if it happens during definition updates.

Just seeing this post and checking to see if anyone has an update on this.  We are in the EXACT same scenario with multiple file and RDS servers and I have gone through 3 months of this and have suffered DFS corruption from dirty shutdowns and shares going offline (millions of files).  We just opened a call with Microsoft, and they're suggesting memory dumps etc - but it looks like someone else has gone through this rodeo.  We're going to open up a case with Symantec and ask to immediately escalate.  Thanks in advance...

We are also having the same issues, I have removed 12.1.4100.4126 from one server to test and we are not having the issue.

What I am looking at doing is, push the old version of the client back out (12.1.4013.4013). The only concern I have is going backwards, I have not done this before. Anyone have input on this?

Symantec – do you have a fix for this?

As long as the newer version is alrady removed, it should be as easy as installing the "old" version.

Gotcha - not looking forward to this....  we have about 800 nodes with SEP, including about 150 servers.  While the file share issue is only affecting our core file servers, the 'shutting down' hang-up is spread througout our servers and affects things randomly.  Servers not coming up from updates, etc....  I'm going to have to script out clean-wipe then push the old version.  Just a lot of work for something that is a clear bug and we'd expect it to be fixed....

See GadJeff notes above, he is a co-worker of mine and our primary SEP administrator (and when I use 'We' for the rest of this it was all him doing set-up and testing).

We automated the process with SCCM, used the cleanwipe utility to remove SEP and do initial reboot.  Then let it clean-up left over files and install the older version and a final reboot.  I think we rolled back about 700 servers in total over 3 weekends.

We had about a 5% fail rate on the automated process.  Most of those were due to SCCM not knowing the removal steps finished out, and maybe 10 servers where the host would not reboot due to the SEP sevices not stopping properly and had to hard reset the machine.

Glenn - regarding the 'downgrade' - we have been working with a Symantec engineer, and they are suggesting that maybe this was a result of upgrading from a previous version.  For this reason, they're suggesting we cleanwipe, then install the SAME version (4126).  Although I hate being the guinnea pig.....  I don't know that I've had any success with doing an inline downgrade....

Hi,

We also have the SAME ISSUE with Windows Server 2008 R2. Share is intermittent and we cannot replicate the issue since it is "intermittent". Shares, printers become inaccessible, you can RDP, ping, and telnet 445 service but you cannot browse the shares. we've been sending logs to the support but it seems like they havent figured out anything yet for 2 months now. We are running version 12.1.4100.4126 without the Firewall component and created basic package as the AV component only but still occuring randomly.

We have removed SEP client temporarily from the file servers that are having issues.

Has symantec support found anything yet?

No, Symantec has yet to recreate the issue in their 'own' lab (to my knowledge). I keep insisting that they do so. We (as well as yourself and other customers...) have removed that version and reverted to back to v12.1.4013.4013.

Thanks for the response GadJeff. I will try the lower version (12.1.4013.4013) since we have totally removed sep client (cleanwipe) to the server that is having issues.

I have upgrade to 12.1.4112.4156.105 and my file servers have stop having this issue.

but,

on one of my networks the new version was pushed to the clients, but it did not uninstall the older version (12.1.3001.165.105) - Oh the joy...

How do you mean it did not uninstall it?

By default SEP upgrade put all the files for each version in its own folder (titled 12.1.3001.165.105 and 12.1.4112.4156.105 in your case) and swaps round the active files (i.e. the ones that SEP is running on) during a reboot.

Only after this mandatory reboot and swap over is the new version up and running.  With the older versions files now free, SEP will evetually delete the old files when the machine is free (this can take a few hours, but the 12.1.3001.165.105 folder will eventually be deleted).

Assuming the upgrade and swap over didn't bomb out, you just need to wait...

on some servers I see both version folders, and after mulit-reboots, the new version is still not installing.

If the swap over itself is not happening, then it won't free up the older versions' files for deletion.

I'd recommend checking the article below on the logs for troubleshooting the client install:

http://www.symantec.com/docs/TECH164067

thanks for the info, but it did not help. when i look at the properties of the client with SEPM, i see that 'Deployment targer version' is 12.1.4112.4156 and the 'Deployment running version' is 12.1.3001.165.  the only install packages that are on the SEMP server is 12.1.4112.4156 - i have removed 12.1.3001.165.

Hello, All - So we've been going back and forth, trying to get a response from Symantec.  I saw the notes about the 'b' version, 12.1.4112.4156 - we've downloaded it and have begun testing.  We called into Engineering in Symantec and asked for release notes for this version to see if the share/shutdown hangup was identified in this version - they told us that there were NO release notes and pointed us to a page that referenced this upgrade as a vulnerability fix.  All that said, I have the smoking gun: specifically for our hung up on shuttind down issue, Microsoft has analyzed a memory dump and has clrealy identified Symantec to be the issue:

Debug summary:

SRTSP64.sys is preventing the computer from shutting down 

To say I'm frustrated would be an understatement.  We have now traced this issue back to failings in our file servers, DFS (because of dirty shutdown), fax server (zetafax), document management server (Worldox), site file servers, and more.  This has literally caused us to spend hundreds upon hundreds of hours troubleshooting with vendors only to find it was Symantec.  I'm passing this on to our case manager asking for a supervisor to provide a response if we cannot get an engineer.

We're ready to move forward with the new version, but would like some kind of acknowledgement that this has been resolved.

Good luck with that, i have been banging my head against the wall of "Symantec" about this issue.

IT's BACK -

I have installed 12.1.4112.4156 (2 weeks ago) and along with other ongoing issues, we are back at the same issue of Network Shares Stop Responding Randomly on Windows Server 2008, I have tried to removed 12.1.4112.4156 from one server to test but having issues removing SEP, so i am getting HBSS to push McAfee A/V to the server to see if this will uninstall SEP. then to watch if the Network Shares Stop Responding.

Symantec – do you have a fix for this? or will my fix be HBSS???????

Dear friends, I found a Microsoft Article (KB2582112 - http://support.microsoft.com/kb/2582112), which reports that Windows may stop responding on High Network I/O operations through SMB shares.

I believe that if SEP has some sort of issue on its file system driver, maybe this Windows problem can be turning things worse, and probably this hotfix must be needed to be applied on the server and workstations to solve part of the problem.

This hotfix does not come on any Service Pack, update or update rollup from Microsoft.

We removed Symantec Antivirus from the file server last week. Today it stopped responding, but was a very fast outage (5 seconds - not as long as with Symantec installed).

I applied this hotfix and we will be monitoring server to check if it is going to happen again.

Best regards to everyone!

Dear friends, I have found the following article at Microsoft Support Site: http://support.microsoft.com/kb/2582112/en-us .This article says that Windows Server 2008 R2 and Windows Seven operating systems stop responding during high network I/O SMB operations.

Details about it: this hotfix is from 2011(!!!).

I believe that if Symantec is having some sort of issue on its file system driver, together with this Windows Platform issue it can increase the consequences of high network I/O at the operating system side and, so, making things worse.

Last week we uninstalled SEP 12.1.4a from file server and installed Microsoft Security Essentials. On last saturday, 08/23/2014, we experienced a very quick outage (5 or 10 seconds) on file server.

I applied this hotfix, as it does not come with any Service Pack or Update or Update Rollup (it's not completely tested yet).

I'll keep everyone informed if I have any update about this issue.

Best regards!

We have been having this same problem with version 12.1.4100.4126. I happened to notice by pure coencidence one day when this was happening on a particular server that a virus definition update was being sent out by our management server. I've limited the live updates of the management server to a period at night now and it seems to have improved things. I'm still waiting on it to happen again.

Dear All,

i have tried to install this KB2582112 to one of our FS, however the issue re-occured after 2 weeks. does any one had a fix already?

I think my open case at symantec support will just age..(nothings happening, they want me to replicate the issue. <crap>)

We are having the same issue. On two heavily used file server the shares starting to slow down, and they eventually stop working. the server not showing anything in the resource explorer, no IO/memory or CPU usage, the shares just stopped working.

KB2582112 did not solve this, neither a restart, because after about an hour the shares slow down and stops again.

Another experience is the server stucks in the "Shutting down..." phase forever, until manual reset.

Rolled back to 4013.4013, all of the problems went away.

Update - I got an Symantec engineer on the phone and provided them with a copy of the dump and the diagnosis from Microsoft.  He mentioned that this week is jammed because of staff cuts and some training, and that we should expect to hear back next week.  At this point, we've basically identified the issue - they just need to fix it.

Thank you for keeping us posted!

Just wondering if you have heard from Symantec? I dont want to install the newer client in case it doenst fix it!

Possibly, from the RU5 fix notes:

File system performance degradation when Symantec Endpoint Protection client access a Distributed File System

Fix ID: 3291764

Symptom: File system performance to a Distributed File System (DFS) drive or share may be degraded when Symantec Endpoint Protection is installed on the client.

Solution: A registry value has been added to allow customers affected by this issue to disable filename normalization across the network.

Nope, that is nothing like this error. We don't even have DFS.

Anyway, next week I'll test the new client on our servers.

I asked the engineer working our case "...I don't see in the release notes that our issue was 'fixed' with this revision though. Is that correct?"

His email reply =  "...I did not see the issue reported in the release notes.  I have not yet seen the dump analysis.  I have again pinged the development team asking for status." = this was as of 9/23 @11:06am.

We are also experiencing issues on our File Server clusters since upgrading to 12.1.4100.4126. We are experiencing the following issues.

1. On a file server cluster when trying to move a file service to another node the move fails and the following event is created "Cluster resource 'Cluster Disk 3' (resource type '', DLL 'clusres.dll') either crashed or deadlocked. The Resource Hosting Subsystem (RHS) process will now attempt to terminate, and the resource will be marked to run in a separate monitor."

2. When shutting down a file server in the cluster the shutdown hangs. Usually we have to just hard reboot the server. However, if you just let the server hang at the shutdown eventually the server will blue screen.

3. On two of our file servers in the cluster the server has blue screened out of nowhere. When analyzing the dump file it states that the the file SRTSP64.sys caused the crash.

These issues didin't occur until we upgraded to 12.1.4100.4126

We are having the same issue, windows 2008 R2 file and printer sharing gets hung, server requires a reset.

Same problem on Win2003 file servers. Any success story with RU5?

Just installed on a minor fileserver, if the stress-tests are successfull, I will continue with the bigger ones, which has bigger network I/O in work hours.

UPDATE: The fileserver did not show any kind of SMB errors under load today, altough it is a minor fileserver, it is a good sign. Will come back with updates in the next few days.

UPDATE2: Bad news. 12.1.5 is still producing errors. Only with XP machines, but the symptoms are the same.

All - good news (well, kind of) - for all those suffering from this issue, I've made my way up the chain at Symantec and now have the attention of technical principals, development, and account management.  Hoping for some traction sooner than later.  Will keep everyone posted.

Dear all,

thanks for shedding some light on this issue. I believe we are suffering from the same problem.

For us, this also happens on two of our Server 2008 R2 machines. Connection to the network shares drop / seemingly take forever and the Server hangs on shutdown afterwards. The issue has not yet occured on any OS other than 2008 R2 for us (our 2003 R2 Fileserver, which is heavily frequented, seems to be unaffected). The machines are also hosted on VMware.

SEP Version is 12.1.4100.4126, Symantec Firewall is disabled on one of the two affected servers.

I have noticed that once the issue occurs, even locally on the server itself I cannot open or copy specific files inside the shared folder (it takes "forever" to calculate before copying). Only some files in the shared folder are affected and explorer crashes every time I try to access these file. From a remote computer, the whole share is unavailible. As for you, after a reboot, everything is working as expected.

Are you experiencing the same behaviour (copying the shared folder that became unavailible locally on the server crashes explorer or takes forever to calculate)? Can somebody please try that, once the issue occurs for you again? I just want to make sure our problem has the same cause...

Hi Everyone,

There are many different reasons that network shares could stop responding or hang.  Some of them may be related to our software, others may not.  Symantec is aware of and investigating an issue that happens after installing/upgrading to Symantec Endpoint Protection 12.1.4.1 (RU4 MP1) or 12.1.5 (RU5).  If you believe that you are experiencing this please open a Support ticket so that we can assist you in troubleshooting and determining root cause.  If you've opened a ticket that you feel isn't getting appropriate attention, please send me a message with your case number, and I'll look into it personally.

Depending on your symptoms there may be one or more viable workarounds for your circumstances.  Your support technician can help you determine what is best in your environment.

Faraz Zobairi
Manager, Symantec Corporation
Enterprise Security, Mobility & Management

Has anyone been able to duplicate this with stress testing?  We are having the issue, but when I tried to open an issue with Symantec they just wanted to do some basic things "and see if it happens again".  We can't really take more downtime since we have already narrowed it down to symantec endpoint.

Also apparently network threat protection isn't recommended to be installed on the servers, only workstations, seems crazy to me, because the most important data is on servers, but support said it would cause these issues and they don't recommend it, lol.

Hi!

No, we've rolled back almost instantly. With our primary fileserver, it happens 2-3 hours after work start time.

We don't have network protection on, and I don't have time nor resources to test it. It's symantec endpoint 100%, but I can't reproduce it, because It will bring dopwn the whole fileserver again. You can't test it on a testserver because it needs hell of a lot SMB connections.

Hi Joe,

The issue in this conversation is not related to NTP.  There are instances where Network Threat Protection (NTP) may cause performance issues on server operating systems.  This is not a blanket rule.  Many of our customers run NTP on servers without issues. .  The following document has some information about installing SEP on servers:

Best Practices for Installing Symantec Endpoint Protection (SEP) on Windows Servers

Pleae let me know if you have a case number you'd like me to take a look at.

Faraz Zobairi
Manager, Symantec Corporation
Enterprise Security, Mobility & Management

Has anyone disabled SYMefa.sys yet?  This sounds like is is version specific.  And if it *IS* version specific it will be a static component so we can elminate :

Autoprotect, SRTSP, SRTSPX, NAVENG NAVEX15.

that leaves on a AV only install :
SERVICE_NAME: symds
SERVICE_NAME: symefa
SERVICE_NAME: symevent
SERVICE_NAME: symiron
SERVICE_NAME: srtspx                not static
SERVICE_NAME: srtsp                  not static
SERVICE_NAME: navex15             not static
SERVICE_NAME: naveng               not static
SERVICE_NAME: bhdrvx86
SERVICE_NAME: symnets
SERVICE_NAME: idsvix86 

of those elements, ones we can preclude are sysplant, and IDS,  they should *not* be loaded on file servers.  we also remove the non-static elements marked non-static.

this leaves us SymDS, Symwefa, Symevent, Symiron, BHDrv***, symnets and idsvix86 on a AV only install of SEP 12.x.x.x

Of those the common culprits are : SYMefa,  BHdrvx86, BHdrvx64, SymDS

I would test as follows ::
uninstall sep 12
reboot
cleanwipe
reboot
install AV only package, disable EVERY option but : Core files + virus, spyware, and basic download protection only, all other msi check boxes should be 'x' red out of the package.
 

then run the command 

Another thought that occurs is *if* at some point Email tools were installed (just installed, not even enabled) or the firewall was installed you may have a instance of symtdi, symtdiv loaded.  I would ensure these components are removed via uninstall, and re-install with the aforementioned AV only package for file servers. 

I have only the AV package, never installed any other module. Unfortunately I don't have the luxury to test with the only problematic fileserver, what is under heavy load (and that's the only thing that triggers the error, lot of SMB connections wich is impossible to recreate in test environments)

12.1.5 solves the SMB2 connections freeze (at least we did not detect it) so win7 computers won't be affected, but XP computers with SMB connections will trigger the error, freeze the network shares on XP machines, and still can't shut down the server without a hard reset.

hello to all,

we are encounterig the same problems on file servers with win 2008 r2 and 2012 r2.

same network performance issues/freezes using smb shares.

same problem when shutting down the file servers (freeze on "windows is shutting down" and of course we need to power off).

the problem is surely present with both 12.1.4 and 12.1.5 and persist also disabling ntp by policy.

we also encoutered problems stopping symantec services with smc -stop (the command sometimes hangs, other time is apparentely completed but looking into windows services we found in both case the services still running). this problems with smc -stop  is present only in file servers or other windows server machines, never encountered on clients.

At the moment we have solved all the problems removing completely sep from the windows file server machines.. we are gonna to open a ticket because the situation is not acceptable.

We also have encoutered some performance issues using smb shares that can be solved only disabling/removing sep from clients in particulary for those situation where users need to copy gigabytes over network performances are not accettable and we need to disable the sep also on the client computers.

PS: more in general we are encoutering problems/hangs in particulary with firewall module since we upgraded to 12.x from 11.06 about two years ago, we opened lot of tickets where the most frequent answer was to wait the next upgrade, but things are going even worst!!

We too are experiencing the same freezing of SMB on Windows 2008 R2 file servers and new SEP clients 12.1.4112.  We upgraded from 11.x versions and never saw the issue with the old client.

In addition, we are seeing failures doing SQL restores from a remote SMB share using a multi-threaded SQL backup tool.  Single threaded SQL restores using Management Studio are not a problem.  We will be validating that this condition is the same as the file server issue with high SMB and IO later this afternoon.

Our Symantec Support technician has directed us to switch our Auto-Protect Advanced Scanning and Monitoring policy from "Scan when a file is accessed or modified" to "Scan when a file is modified". 

We've change that policy of of this morning but I don't have high hopes that this will correct the issue.   I was hoping some of you experiencing the same condition could comment back as to what your comparable policy was set to when you experienced the issue.

A third possibly related issue we seeing since upgrading SEP clients to 12.1.4112 is on our Exchange 2010 mailbox servers.  CPU on the mailbox servers was stable for years before upgrading.  Seemingly at random times, CPU kicks up and pegs CPU at 100% (60% SEP and %40 Exchange).  The technician is suggestion we disable "Rescan cache when new definitions load" within Auto-Protects File Caching policy.

Any thoughts on that suggestion?  I again am interested in what you all have set for your policy.

These are detailes on where that policy is located....

Symantec Endpoint Protection Manager's - Virus and Spyware Policy, Advanced tab, File cache and uncheck the option
"Rescan cache when new definitions load"

We are still experiencing the same problem with our 2008r2 servers, SEP RU5, and Win 7 and 8 (Surface 2Pro) clients. Redirected profiles/folders on DFS shares.

I just upgraded a client's Windows 2008 R2 server to SEPM 12.1.5 last weekend.  I updated the SEP client on the server to 12.1.5 this past weekend.

My client now complains:

Our network has been extremely slow for the past few days.  Is there something wrong?  It takes like 10 minutes for print jobs to print.. to access a file on the network it takes forever. So long that the computer list is not responding.

At this point, I am not inclined to release the 12.1.5 client to the Win 7 desktops at this site.

Aside from opening up a case and running SymHelp - and then not getting any sort of definitive answer - what sort of actions should I take?

Thanks!

Per some replies above, rolling back to a working version, 12.1.4

Do I simply install the previous installation package?

Or is there actually documentation for this type of task?

Thanks!

There is no downgrade or rollback

Needs to be an uninstall/reinstall

I have been doing the following:-

1. Uninstall problem version and reboot.

2. Run cleanwipe and reboot

3. Install version 12.1.4013.4013

The reason I have run cleanwipe after the uninstall is because I have had issues with the previous install still hanging around and causing the same issues eventhough 12.1.4013.4013 was installed after. I have followed this process on numerous servers and the problem hasnt re-occurred.

Hello,

If you're experiencing this issue, please do open a case so we can help.  If you case isn't getting traction, send me a message, and I can help.

Faraz Zobairi
Manager, Symantec Corporation
Enterprise Security, Mobility & Management

Hi

We are also having the same problem

2 file server Windows Server 2008 R2 on wmvare, sep 12.1.5xxx "Basic Protection for servers" installation

firewall, "intrusion prevention", "application and device control" disabled policy

2500 clients with windows 7 pro & premium sep 12.1.5xxx "Full Protection for clients"

firewall disabled policy

1 sepm 12.1.5xxx

when sep is uninstalled on the file servers is there no problem

When sep is installed on the server

after some hours is the server connection not responding to the shares and when the servers is restarted it hung up so we need to use power down.

I followed suggestion from Tech Support re file share: http://www.symantec.com/docs/TECH225417  At first it seemed to work, but another call told me it didn't.

In this particular case, a look at the Event Viewer showed SEPM taking - on average - 45 minutes to process the live updates.  In a few cases, this ran as long as 112 minutes (max 178).  The longest time was processing the 12.1.5 defs, which are not even rolled out, but have been defined as being available.  This being the SBE version, I can't delete these packages from the database!

The longer the processing takes, the longer the network is unavailable to my client.  I'm going to recommend that for such a small site, he switch to the cloud version.  That would eliminate the SEPM altogether.

Thank you for posting a link to the article.  I meant to do it earlier, but got busy and it slipped my mind.  The Article was published externally earlier this week, and has a workaround that we've had quite a bit of success with for customers experiencing unresponsive network shares since upgrading to RU4 MP1 or later.

Network shares stop responding with Symantec Endpoint Protection Auto-protect installed

In order to know with 100% certainty whether or not the issue you're experiencing is the same as the one mentioned in this article, we'd still need a memory dump to confirm what happening.  That said, if the workaround resolves the issue, there's a high likelihood your are experiencing the issue being tracked in this document.  A code fix is planned for a later build.

I'd still encourage anyone with this problem to contact support.  Once we've verified the issue, we can add you to the list of customers to notify once an updated build is available.

Regards,

Faraz Zobairi
Manager, Symantec Corporation
Enterprise Security, Mobility & Management

for 7 days ago I uninstall 12.1.5337.5000 on our 2 file servers and installed 12.1.2100.2093

after that there was no problem with lost network share respons.

I have tried the workaround suggested by TECH225417, didn't work for us. 

It did appear a bit better, server will only lose shares once a week.  Previously without the workaround the shares would stop responding 1-2 times a day under heavy load.

Will revert back to a previous version, can't recommend anyone use this version in production even with the workaround.

Neither for us. Client updating is stopped at us until I see the changelog mentioning this exact same issue resolved. (nice long and old thread btw)

We've seen 4 different types of symptoms with the 12.1.4112 client and we agree, the symptoms are not resolved by disabling deferred scanning as described in TECH225417.  We've provided multiple crash dumps to Microsoft and they point to a bad Symantec driver with push locks.  Symantec is still reviewing our complete crash dumps.

We are having to cleanwipe the system, then reinstall the 12.1.4013 client as that seems much more stable, I just hope its not going to kick us in some other way.

We opened a case a couple months back, and long story short another defect along these lines was introduced in RU4 MP1, not resolved in RU5, and so is expected to be resolved in RU6 (circa Q1 sometime in 2015), no known ETA.

I'd suggest your best bet is to keep pursuing via your support case, in case this issue is yet another defect not like ours.

After several months with problems in our file server, we find this post, we are having the same problem here in the company, look for an urgent solution.

Good luck!

only 2 solutions that I know of:

Revert to version 12.1.4013, be aware that you will be vulnerable to CVE-2014-3434, you should disable application and device control to mitigate this issue, not so much of an issue on normal servers, might be an issue on terminal servers though.

Upgrade windows 2008 servers to windows 2012, windows 2012 does not appear to be affected by the issue.

revert, and be patient a little. Beta is out, wait for the release. (and hope they fixed his)

FYI I have had the problem with Symantec on Windows 2012 R2. I had to uninstall Symantec using cleanwipe and install 12.1.4013.4013. I havent had a problem since.

has anyone heard anything from symantec? having the same problem...

Good morning, all.  We have been working extensively with a top-level Symantec engineer on this issue for some time.  The two issues, for us, have been resolved.  To recap:

Shutdown Issue

This was the issue where the SEP driver would hang up on shutdown and the machine would freeze.  This was resolved sometime in November (I beleive) in a Bash update that was rolled out to both 12.1.4x and 12.1.5x.

Network Share not Responding Issue

Having gone through mutliple iterations of 12.1.4 and 12.1.5, the ultimate resolution was not specific to a version of software or a patch, but a registry key.  We tried this on multiple machines, physical and virtual, 2008 R2 and 2012 R2 - all of them have been healthy for weeks if not months since applying this.  Prior to, we were having lock-ups weekly or better.  Here is the key:

By default, Deferred Scan feature is Enabled and it will delay to scan. To disable Deferred Scan feature, 1. Disable Tamper Protection.

2. Create registry key to disable this option.

3. Click Start > Run

4. Type in: regedit and click OK

5. Navigate to:

"HKEY_LOCAL_MACHINE\SOFTWARE\Symantec\Symantec Endpoint Protection\AV\Storages\Filesystem\RealTimeScan" (32 bit Operating System) or “HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Symantec\Symantec Endpoint Protection\AV\Storages\Filesystem\RealTimeScan” (64 bit Operating System)

6. Click Edit > New > DWORD Value

7. Name the new value: DeferredScanning

8. Open DeferredScanning

9. In the Value data field add “0” (1 = on, 0 = off)

10. Click OK

11. Close the Registry Editor window.

12. Enable Tamper Protection again.

Hope this helps everyone!  Symantec Engineers - chime in if I got something wrong here.  This was the result of my open case with one of the principal tech support engineers.

Found this artcle:

How to disable deferred scanning in Auto-Protect for Symantec Endpoint Protection

For the issue we experienced on this area, we were told a code fix was needed, and that it would come in RU6. RU6 is now in beta, but we're still waiting on confirmation the fix was included before we spend the time on trying to verify the fix.

This issue looks being addressed through the article TECH228086 - Windows computers sometimes hang on startup after installing Symantec Endpoint Protection 12.1 RU4 MP1 or newer

This issue occurs when Symantec Endpoint Protection's Auto-Protect component does not properly release its opportunistic lock (oplock).

Disabling Deferred Scanning for Symantec Endpoint Protection's Auto-Protect component on affected machines
Follow the instructions in knowledgebase document TECH224108 - https://www.symantec.com/docs/TECH224108

Please, read the whole thread before posting the same thing the hundredth time. Thank You.

Does anyone know when RU6 is going to be released?

Currently in beta, still a couple months out most likely

You can expect in the month of April'15.

We tried the DeferredScanning registry change and had no luck. We have experienced this issue on both 2008R2 and 2012R2 on a VM platform using 12.1.5 Basic Server.

Judging from the difficulty I had unistalling the client I suspect that when it is either performing a scan or updating defs it has no ability to throttle down to accomodate the resources available. I suspect its CPU usage that is the culprit.

It does require "workday" level usage to occur.

Noticed that 12.1 RU6 SEP content was added to our LiveUpdate (LUA) server's product catalog about a week ago. Any planned release date this month?

has anyone seen this RU6 yet with the supposed fix?

I'm having the exact same issues for the past 4 months on a set of newly created Win2012r2 file servers running 12.1.4112.4156

RU6 is still in development, some are predicting its release to come in May, but nothing has been confirmed.

On April 1st RU6 Beta 2 was released. I would suggest reaching out to your Symantec representatives to request access to the beta program so that you can try verify if the defect is fixed.

We are also experiencing the same issue with network shares dropping, using Version: 12.1.4100.4126 on a Windows 2008 R2 file server.  This has happened to us twice now, about three months apart and each time we have notice the virus definitions fail to load on the file server. 

We called tech support and referenced this thread but the tech suggested we update to 12.1.5.  I am hesitant to do so because of others that have already stated 12.1.5 produces the same issues.

Has anyone else had any luck with the deferred scanning registry edit over a period of a few months?

I would like to hear if anyone else out there was able to solve this problem.

Thanks.

Hy Joe,

I have a 2012 Server which has had same problems.

So upgrading from 2008 to 2012, might solve the issue, but is not 100%, i had problems even on 2012.

After i completely removed RU5 and downgraded to RU4, the server is ok, no problems after the downgrade.

Hi Brian,

DFS Server on our Network is Freezing,  Due to Which Users are loosing their Home Drives mapped for them.

SEP 12.1.6 is been disabled on these Servers. No Scans being Run

Servers are part of an Exception Group and had only AV & AS Feature Set Enabled.

Still this morning there is a similar issue, Does it have anything to do with SYMANTEC AV?

How did you disable SEP? Did you turn off all components? Did you also do 'smc -stop' from a cmdline?