Endpoint Protection

With SEP 12.1.4013.4013, I am encountering extremely slow file copy between servers caused by the NTP Firewall. When copying a 2 GB file, with the SEP FW enabled, it is taking over 3 hours to copy. Once I disable the SEP FW, the copy happens fast (2 minutes). When I re-enable the SEP FW, it slows to a grinding halt. I've relaxed the SEP FW rules in every way I can imagine (1 rule...allow ANY..ANY), and I still get the same slow behavior. I don't see any traffic blocks. Any ideas on what could be happening? Thanks!

You're best option here is to check with support since you're on the latest version. This was a known issue a few versions ago but has since been fixed. Was this an upgraded client? Creating an any <> any rule won't help. I've been down this road before. Call support, they will have you turn on advanced logging and do packet traces when the problem is reproduced which they can then look at.

Also, do you have the option to "scan files on remote computers" configured? This is in the AV policy but it sounds like you've narrowed it down to the fw exclusively?

Hi

What are the components installed ?

Regards

 

Yes, this is definitely narrowed down to a FW issue. I spent quite a bit of time before checking auto-protect options, as well as other SEP components, before finally narrowing it down to a FW issue. All SEP components are installed. The file copy is from a Win 2008 server to a 2003 server. I'll check into the SMB tip. Any other ideas before contacting Tech .Support ? :( 

Thanks! 

Check the smb issue but it's not really a fix but a workaround. Suport is the best way to go especially if it's a bug

Hi,

Thank you for posting in Symantec community.

We have heard about this issue and might have fixed in SEP 12.1 RU4 MP1 release.

Time being downgrade to SEP 12.1 RU1 & check the performance.

Best Regards,

Chetan

Will do. Do you know when RU4 MP1 is expected to be released? Thanks!

Hi,

It's a difficult to question to answer. Because we are not allowed to disclose release dates in advance.

You can expect it very soon.

Check this article: How do I sign-up to recieve an email bulletin when a new product is released?

http://www.symantec.com/business/support/index?page=content&id=TECH105781&actp=search&viewlocale=en_US&searchid=1296752026497

Chetan, I am experiencing this same problem with SEP 12.1 RU4 MP1a and it is extremely frustrating. Support has been of no help to us and just run us around in circles with changes to anti-virus policies that are completely unrelated to the actual issue.

I have narrowed down the problem to a specific scenario: A Server 2012 Standard on an HP DL385p Gen8 server with broadcom NICs and any Hyper-V VM guests on this server. I have Dell poweredge servers and all of them are fine. The problem is with the firewall on the server initiating the file copy to another server (disabling the firewall on the destination server has no effect)

Both the Host and the Guests were slow. I switched the host to Basic protection which removed the firewall and improved performance for the host, but the guests are still slow. Changing Chimney Offloading and related settings did not appear to make a difference. 

 

Copying with Firewall enabled is in the 300Kbps range and is 10Mbps with it disabled - even when ALL rules for the firewall are set to allow the problem occurs. The issue began sometime after upgrading from 12.1.2 (not sure in which release of 12.1.3 or 12.1.4, but probably the latter)

This appears related to the SNAC network driver as best as I can tell.

Other possibly relevant information: Backup Exec agent for backup exec 2014 is also installed on this server and VM Guests including the related pure disk drivers. Backup Exec 2013 experienced the same problems after the SEP upgrade, so the issue is definitely with  with SEP.

 

PLEASE help us resolve this problem!!! I have logs and can perform any testing you might need to get a proper cause for this - just let me know what you need. Case #06959284

Hi, we had this problem and were promised that it would be fixed in 12.1.RU4 MP1. It turned out it WAS but ONLY if you also disable Denial of Service detection in your policies for clients and servers. Just upgrading to that version wasn't enough to fix it, but once we disabled DoS too we haven't seen the issue since.

Hope that helps

Hi,

Sad to hear that! We are aware of this issue, it's been reported with SEP 12.1 RU4 MP1 as well but workaround have been provided.

I hope support engineer would have applied the steps given in this article:

SMB transfer speeds decrease after installing Symantec Endpoint Protection 12.1 RU2 MP1 or higher

http://www.symantec.com/docs/TECH201555 

If you followed all the steps given in this article correctly SMB speed should go up. Make sure SEPM and SEP clients both are on same version.

OR else you need to downgrade to SEP 12.1 RU2.

let me know if you want to test with SEP 12.1 RU2 version.

 

You are correct. In order for the fix to work, Denial of Service Protection feature of SEP must be disabled:

More info you will find in this article.

SMB transfer speeds decrease after installing Symantec Endpoint Protection 12.1 RU2 MP1 or higher

http://www.symantec.com/docs/TECH201555 

Hi ukDavidC,

 

Thanks for the tip.  I actually checked this last night when reviewing all our settings and the info here: http://www.symantec.com/business/support/index?page=content&id=TECH201555

 

DoS protection was on, but turning it off didn't seem to fix the problem for the client version 12.1.4100.4126. However, I checked a server that was still running 12.1.4013.4013 and I it seems to be working as expected!

 

Can you confirm exactly which version number you are running where this works for you? I'm wondering if maybe there was a regression when or around when they patched the heartbleed problem.

Hi abcr,

We're on 12.1.4100.4126 (which is RU4 MP1 but not 1a).  From memory they fixed the heartbleed issue in 1a...

Upgrade SEPM from RU4 (12.1.4013.4013 ) to RU4 MP1 (12.1.4100.4126.)

Upgrade all the clients to RU4 MP1. I believe provide workaround should work after having both SEPM and SEP clients on the 12.1 RU4 MP1 version.

Yeh, like I said.

Thanks for the quick update Chetan, Our SEPM is already at MP1a. I think perhaps the driver was still doing something problematic in memory even after the policy was disabled and the client policy updated. I rebooted the problem server and thus far the file copy speeds are much more normal. So for anyone else having this problem: give the server a reboot after disabling the DoS protection before determining if the problem is fixed or not.

I will post back if I have more problems, but I think disabling DoS Protection + reboot was the key.

This is also one of the important note: The fix will not work on any firewall rule which has packet logging enabled. If required just cross verify.

Hi Chetan,

I can confirm this plague is still a problem for us in client 12.1.4.4100.4126 with the latest SEPM installed. File copy speeds start out fine, but within a few hours of running the server the copy speeds go to a few hundred KBps. If we try to run backup exec on one of the affected servers in starts off at over 1200MB per minute but drops down to 25-40MB per minute part way through the backup (See attached example of file copy). Disabling the firewall checkbox instantly returns the copy perofrmance to normal.

 

We've had a case open for weeks with support. Backup exec support pointed us to Endpoint support who have given us the runaround, and we can't even get a status update about the case. 

 

My higher-ups are asking what the next steps are going to be for solving this problem as it is preventing us from completing backups, and unfortunately the only thing I have to tell them at this point is that we will need to start looking at other possible vendors after over a decade of using Symantec products. 

 

 

I think you should test with SEP 12.1 RU2 (12.1.2015.2015) client version. Downgrade few clients to RU2 & test the performance.

I have a question when I completely check your discussion as found you have latest version . yes Symantec had an reported problem with older version. Issue is when copy data file from one system to another or from file server or try to access excel file through network it works slow and it got fixed in sep 12.1 RU4a and. But as I see that you are referring BE backup and it is slow. Do you have issue only BE or can you give examples with screenshot. It helps If that is the case yes Symantec firewall and BE program have conflict. And the primary issue which I was facing was, if I schedule backup to local drive it is fast but if we schedule to take backup system status file it works slow and every day I used to have such problem. Every day if I reboot my server the BE works fine. Can you check is such similar problem you have?

Yes, this sounds like the problem we are having. What is interesting is before backup exec runs file copy is fast. After backup exec starts to slow down smb file copy is also very slow. We also had problems with the backup causing the windows internald database to hang - you were no longer able to connect with any of the dependent services (WSUS or ADFS 2). The only solution was to restart the service.

Thank you for the recommendation. Where can we obtain this version? On flexnet only the latest version is available.

Just want to add my 2 cents here.  We are also experiencing this issue, although it seems to be limited to a single server for us at this point; (Just finished updating all SEP clents, so still monitoring performance).

We experience 2 scenarios that I can attribute to the SEP (and maybe BE) software - our dispatching/ERP software (which launches from within a network share) sometimes becomes painfully slow requiring not just a reboot of the server (VM) but a reboot of the entire physical host before performance returns to normal.
The other scenario that occurs (typically in conjunction with the painfully slow application launch) is the backups (using Backup Exec 2014) run at about 16MB/s as opposed to the typical speed which is roughly 1,000MB/s+ when backing up the same server.

I've created exceptions for the primary applications running on that server (SQL) as well as the entire network share from which the ERP/Dispatching application is executed.  I've also followed a Symantec KB article about creating exception rules in the SEP firewall for the BE applications, but since we don't run the Sym firewall on servers, that is even kind-of moot.  I'll be looking into the suggestions about disabling DDoS features sometime this week as I have time to test recreating the issue.  So far that seems to be the only suggested workaround that anyone has had luck with, although I plan on also attempting some in-depth troubleshooting.

Are there any documented SEP debug logging commands that we can use to capture performance information without having to open yet another case with support?  I ask because it seems that enough customers have opened support cases regarding this or similar anomolies and the responsivness seems to have been less than desirable either in a lack of knowledge about the issue or time wasted performing alot of basic and often un-related/un-necessary troubleshooting steps.

               Thanks,

Ok, I'm still waiting on a resolution from support while we evaluate options from other vendors. This is clearly a widespread issue as it appears in numerous threads

For example:

http://forum.support.veritas.com/connect/fr/forums/slow-network-after-upgrading-endpoint-protection-1212-mp1from-previous-version?page=1

https://www-secure.symantec.com/connect/forums/symantec-endpoint-protection-slows-file-transfer-and-network-speed-vms-vmware-environment-fac

 

Others have stated that this is related to the teefer driver for network threat protection and I can confirm this is the problem as smc -disable -ntp resolves the issue (but then there is no network threat protection so what are we paying for exactly?)

 

Symantec why the heck can't you roll back the teefer driver to a working version or run a diff between when it worked and now? What the heck!

Looks the same with SEP 12.1.5...

Sorry to hear that Hans. We had this problem and 12.1.5 actually did fix it for us, but make sure you follow these steps in addtion to upgrading:

http://www.symantec.com/docs/TECH201555

 

If that doesn't work I highly suggest opening a ticket with symantec. Stay with it, the first level or 2 of support is painful but you will eventually get to someone that can help. In our case it took about 2 months, but we eventually got it resolved.

Thanks for the udpate.

Thanks abcr :)

Disabling "Denial of Service Protection" solved it :)

Maybe that function should have a "WARNING THIS FUNCTION REDUCES NETWORK BANDWITH TO 40%"

I agree, if this is so fundamentally broken please either put a warning on the option or (preferably) get it fixed!!

 

That said it seems like they have some folks working on it, so hopefully we will see some permanent fixes soon. Symantec has had some serious quality problems over the last couple years, but it seems like they are at least trying to improve lately. Hopefully the trend will continue for their own sake!