Endpoint Protection

Summary of our issue:  We have a new 10 Gb virtual Infrastructure running ESXi 5.5 U3.  We have installed fresh copies of Windows Server 2012 R2 as 2 Virtual Machines running on the same host.  Both are connected to the same VSwitch.  Using JPerf to test the network throughput we are getting a solid 8Gbps throughput between these clients.  The clients only had Java, JPerf, and VMware Tools for the VMxnet3 adapter drivers.  (I believe 8Gbps is the maximum Windows will do by default with a single thread due to a QOS GPO setting)  After installing Symantec Endpoint Protection Client 12.1.6 a (or 12.1.4 or seems any version of 12.1) our bandwidth is severly imparied.  With 12.1.6a installed on both VM's our throughput is kicked down to 1.8 Gbps maximum throughput using the same tests.  The only fix for this that I have found is to uninstall SEP completely although this is not a preferred fix. 

 

Today I worked with Symantec (Still have an open case) and they claim the SEPM (Management Console) is the culprit and we upgraded it to the latest version.  I inquired as to why it would have anything to do with client to client traffic,  her response was this link. 

http://www.symantec.com/connect/articles/tips-installing-sep-low-bandwidth-environment

I cannot see the link between this article and our issue but we updated to the latest version as we needed to anyway.

I figured if its SEPM is the culprit then I will install 2 unmanaged clients with their latest version of the client.  SEP 12.1.6 a

After I did this my servers that were getting 8 Gbps continuous throughput were bottlenecked down to 1.8 Gbps.  It is clearly an issue with their client.  Uninstalling the client completely restores full network bandwidth.  (I also uninstalled each piece of SEP one by one to see if it was the IPS or Firewall but nothing fully restored the full bandwidth except completely uninstalling.)

I am still reaching out to Symantec on this but thought I would share and start a discussion. 

If I had to take a guess you too will experience this slowness on a 10Gb network using Symantec.  Curious if anyone else has experienced this as well.

I understand there will be some impact on system performance with a tool like AV running but this seems excessive and needs addressed.

There have been issues like this in the past with the SEP client firewall.

Is the SEP firewall enabled? If so, try disabling and see what the result is.

In any event, support should've had you enable advanced debugging and do packet traces so they can review.

Brian,  Thanks for your reply,  I have read many articles that you have been part of and appreciate all those. (specifically this article https://support.symantec.com/en_US/article.TECH201555.html) and I have disabled the fierwalls on both clients and the bottlenecks are still present.  I actually uninstalled each piece of the SEP client one by one all the way back to the core files.  The results were interesting but not ideal,  It seemed that after each uninstall of the NTP, PTP, Virus Spyware and Basic Download functions the throughput would increase about 1 Gbps.  When the Core Files were the only thing installed my thoughput would reach just under 4 Gbps.  It was only after completely removing SEP that I would reach 8Gbps again.

Again Thanks for your reply.  If we can get to the bottom of this I imagine it will be very beneficial for a lot of people.  I am betting anyone wiht a 10Gb network would experience the same issues.

I will continue to work with support.  They did not pull any logs or debugging information.  I think they saw "poor performace" in the ticket and went to their goto solution which was to upgrade SEPM.

Still an open Case.

Darren

I would ask to have your ticket escalated to backline support.

I dealt with this a couple years back with a version of 12.1.2 and advanced debugging and packet traces were key in identifying root cause. With that said, it looks to be a design/code issue.

I agree it appears to be a design/code issue.  Something that may work well enough in a 1Gb network but has adverse affects on high bandwidth networks. I will work on requesting an escallation this afternoon.  Thanks again for your input.

Darren

Hi Darren,

 

you might want to have a look at the below article. in any case I (as an Ex symantec Technician) can tell you that this case is not headed in the right direction. I second brain here, have the case handled by a senior technican or get it escalated to advanced engineer as we cannot directly work with backline engineer.

 Best Practices for the Intrusion Prevention System component of Symantec Endpoint Protection (SEP) on high-availability/high bandwidth servers.

 

Thank you for your suggestions. I have escalated this on the Symantec side and we are going to try and escalate this through our vendors inside contacts. At this point I don't see how we can correct this unless there is a registry setting or config file that can change this behavior.

Try unchecking the Download Insight feature in the Antivirus policy.

Even if you don't install the "Download Insight" feature component on the Agent,  SEP will do some kind of checks if that policy is enabled. I saw that it improved the download speeds on 10Gbps interfaces.

It is also important that you do a Security vs Availibility assesment. Do you really need 10Gbps traffic on all your servers?

Unless you do media streaming, high intense backup or other types of heavy traffic. 4Gbps is still a very decent speed :)

 

 

 

 

You need to make sure that the Symantec IPS driver is not present on the servers - it is called: IDSvia64.sys

The driver is running even if the feature is disabled in SEPM, so make sure that your installation package and client only contains "Virus, Spyware and Basic Download Protection".

Also see this article from Microsoft: http://blogs.technet.com/b/craigf/archive/2014/02/03/a-backup-server-flooded-by-dpcs.aspx

I appreciate your input however I described that the only way to achieve 4Gbps was to remove all functions of the AV client except the core files.  The Core SEP installation doesnt protect you and is only to be used for troubleshooting.  Regardless of the use of our servers it is unacceptable that our 10Gb network is throttled down to 1Gb or less when SEP is installed.  We are in the process is escalating this issue on our side as a client and from our vendors side to see if we can get the backline engineers to take a look. 

Again I understand there is a hit in performace when using security software but this is excessive in terms of machine resources.

Curious what others performance has been using SEP. 

Darren

Thanks for your suggestions.  I will give this a try and post my results.

Darren

Any idea where this IDSvia64.sys file is located?  I cannot find this on my system.  I did however uninstall all symantec features except the basic virus, spyware protection.  I am getting an average of 4Gbps throughput but that is still only half what the system can do without syamntec installed.  I am thinking this file doesnt exist in Symantec Endpoint 12.1.6a

If anyone knows if this file should exist I would love to know.

Darren

IDSVia64.sys is stored in the IPS definition cache, similar to:
C:\ProgramData\Symantec\Symantec Endpoint Protection\12.1.6318.6100.105\Data\Definitions\IPSDefs\20150924.011

You can check if it is loading as a system driver, via Device Manager.
Make sure to enable View - Show Hidden Devices
Look in Non-Plug and Play Drivers
If you see IDSVia64 entry, it is loading.

Keen to know your outcome as we are looking at moving multiple servers to 10Gb links, from 2 x 1Gb.

This article seems to show the IPS is know to be an issue:
https://support.symantec.com/en_US/article.TECH162135.html

Following this post with interest - please keep us updated.

Thanks.

Thanks for following.  I will post Symantec's official response after I get it. Currently It is escalated and handed off to the Backline Engineers.  I had another support vendor of ours who I was working with before we realized it was Symantec causing our slowness.  They have recreated this in their labs.  I have asked Symantec to recreate this in their labs.  At this point that is what they are doing.  I have requested daily updates but was told due to the nature of this issue it could take a week or two to get more information.  Very frustrating problem.  I have been playing with different components being installed and here are some interesting results.  The best I can do is to install the client with basic protection only and no download insight (Which I am ok with on a server) but I am not ok with losing 3 Gbps of my total possible bandwidth.

Symantec Testing	Average Throughput in Gbps
Without SEP installed	7.75
With Core Files only	4.8
With Basic Virus Protection ONLY and No Download Insight	4.8
Embedded with Basic Virus Protection ONLY and No Download Insight	4.8
Basic with Download Insight (no PTP or NTP)	2
Basic with Download Insight and PTP (No NTP)	2
Basic with Download Insight and PTP and NTP (Default)	1.75
With Default SEP Client	1.75
All Features NO Download Insight or SONAR (Canot have SONAR without DI)	1.8
	1.8

 

We have been exploring the possibility of using VShield and Symantec Virtual Appliance.  We got everything running and found that we are still required to use the SEP Client on our virtual machines.  I can see how this may help in large Virtual footprints (VDI) but this wont help in our situation.  The issue is that the client still needs installed on each VM.

Interesting we are actually seeing the worst network performance yet using this setup.  Getting 1.3 Gbps throughput. 

More Info from others thinking this was an Agentless setup.  It is NOT:

https://www-secure.symantec.com/connect/forums/vshield-agent-less-sep

Darren

Thank you for your reply.  I do not have this driver loaded on my system and it is not listed in the Device Manager.  I have removed the IPS function and posted results below but it is not the only culprit for these performance issues.  I will continue to post any new information I am getting.

Darrem

I installed a competitors endpoint agent and performed the same tests.  The results were much better,  I was able to get full or nearly full bandwidth using a non symantec product.  I also tested its functionality by downloading the eicar.org test virus and it caught it as we would expect. 

I will give Symantec a week or so but if I keep getting the run-around I will be looking for a different security solution.

Darren. 

If you want to test completly agentless I would recommend looking into Symantec DataCenter Security: This product provides agentless Antivirus with Insight support and Agentless IPS. Note that it requires Vmware NSX.

http://www.symantec.com/data-center-security/

 

I appreciate the suggestion and would love to go agentless however for this specific scenario we are running high performance physical hosts with only 2 Virtual Servers on them.  This solution would be overkill in out situation.  I can see the value in this for large dense virtual environments.  Possibly some day we will consider this as an option for I dont think this is the soltuion for us at this time.

Perhaps we will give it a shot at least to test and see how we can benefit from it.

Darren

For those following:

Still working with Backline Support at Symantec.  Performing many tests wtih the Symantec WPP Debugging tool.

Have found that the SepMasterService could be the culprit...disabling that service in the registry seems to restore network performance.  Working with them at this point to narrow down the specific cause of the slowness so they can look trough their code.

I plan on performing similar tests on our 1Gb network to see if the same performance issues occur using SEP.  It is possible anyone using their client is suffering from performance issues but arent aware of it.  I will post those results once I have them.

I have found that for my tests using Jperf it doest matter if the jperf client machine has SEP installed at all,  The issues appear to be from the jperf server side. 

Darren

Hi Darren,

Just to verify, in the great Gbps-throughput table you made and testing you've done, are you doing this testing by installing SEP on a server or a client workstation?

Thanks,

Tom.

Tom, thanks for your reply.  All tests were done between 2 Windows Server 2012 R2 VM's.  I have been using Jperf to test my throughput.  One VM is acting as a Jperf client and the other is acting as a Jperf Server.  There is no impact on performance whether SEP is installed on the VM acting as the Jperf client or not.  The issues are when SEP is installed on the VM acting as my Jperf Server.  All scenarios in my table above reflect the changes made to the VM acting as the Jperf Server. 

None of my testing has involved workstation operating systems at this time.  (plan on doing that to see if I am suffering performance issues on my production 1 Gb network, Kind of afraid of what I may find)

Darren

 

 

Thank you for the update and for working with them to narrow down the issue - hopefully Symantec will find and resolve the issue.

I appreciate that.  I have already spent more time on this than I should have but I would imagine if this is a core product design/code issue it could help...well everyone using this product.

Update:  It was suggested to try multiple CPUs on my VM's (I was not hopeful that I would see improvement as we first noticed this issue in a VM with 8 VCPUs) but for the sake of working through this I gave it a try.  My results were interesting but not incredible.  Without SEP on a VM with 2 VCpus I get over 10Gbps throughput.  With Core Files only (no protection at all) on a VM with 2 VCpus I was getting 5.8 Gbps average throughput which is an improvement from 4.8 Gbps but again where is my missing 4.2 Gbps.

I performed the same tests on my VM's using a competitor product and I get over 10Gbps with their agent installed and with protection enabled.  I am starting to get the feeling that the official answer from Symantec will be "We will look into improving performance in future revisions but for now you are out of luck"  I am still actively working with SEP and they have not gave me an official response but I am losing hope and getting frustrated due to the amount of time spent on this issue.  I could have switched to the competitor product and fully implemented it by now.

Darren

Test Results Below

SEP:

Competitor:

For anyone following,  I am still working wtih Symantec on this issue.  Haven't made much progress in the last 3 weeks.  The issue is now with the development team.  They had me try a few things today but the results were the same.  Installing their product signifigantly slows down network throughput.

Update:

Yes I am still here and still working with Symantec on this issue.

Today I decided to see what impact this is having on our 1 Gbps network.  Although there is a performance hit it isnt nearly as signifigant.  Lost about 15 Mbps of throughput so a drop between 10 - 20% is what I experienced on my 1 Gbps network.  for comparison I am losing nearly 80% of my throughput on my 10Gbps network.

Test details below:

See a drop on a 1 Gbps network in performance when SEP is installed.  It is not nearly as bad but still noticeable.  Lost about 15 mbps.

 

Tests were done between my laptop (Win7) and a desktop PC Dell Optiplex 7010 (win7) both connected to the same 1 Gbps switch.

 

I only tested with the Typical Unmanaged SEP installation taking all the defaults.

 

These tests make me feel a bit better about having SEP on all our servers and workstations on the 1 Gbps network.

 

On 1 Gbps network:

Get about 95Mbps without SEP

Get about 80Mbps with SEP installed.

 

On 10Gbps network: Much more significant.

Get about 10.5 GBps without SEP installed

Get about 2 GBps with SEP installed. 

 

Without SEP or security software on a 1 Gbps network.

 

 

With SEP installed typical install unmanaged:

 

 

 

 

Has Symantec determined a root cause? I guess I'm surprised this has dragged on this long for you...

Nope,  It is with Development.  I have even supplied them with my Virtual Servers to test with.  Last thing they had me try was to stop the SEP Framework in task manager.  That killed the System Tray icon...which resulted in no change.  SEP being on the machine is still affecting network performance.

Somehow they must have a driver installed that wraps itself around the NIC Driver.  I really dont know what they are looking at.  I have my technical contact who calls me about twice a week to give me updates and try the occasional troubleshooting task. 

I too am surprised this is taking so long when working with a Global Company like symantec.  I am relatively small potatoes compared to them and figured they would have a top notch team of experts working on this.

I am determined to work with them at this point.  Kind of a personal challenge.  This issue affects ANYONE using this product in high bandwidth environments.

Darren

Thanks for your work on this Darren.

Tom.

I am surprised as well. The good news is there seems to be an opportunity to make improvements to the product which will benefit customers going forward. The bad news is you're the guinea pig :(

Yeah,  I have moved onto a competetitor product for these 6 Servers I have in production in this new environment.  Minimul impact on network throughput.  I do have capacity for test machines in this environment to continue work on this.  I dont mind at this point to continue to work with them.  I didnt want to take on a new Endpoint security project at this time.  If we can get a fix for this we can continue using SEP and everyone will benefit.  Hopefully they will get somewhere soon.  They seem to be having trouble creating a 10Gbps test environment.  I am thinking this could be one factor holding them up.  I am still baffled that this is not easily achievable for the technical experts at a company the size of Symantec.

Darren.

Update:  Still working through the issue but progress has slowed way down.  Last thing we tried was disabling the Base Filtering Engine in Windows.  After doing this and still having SEP installed I was able to get my full network throughput.  At least we know now that the issue is somehow related to the BFE and how SEP client interacts with this Windows Service.

I will share more information as I get it.

More infor about BFE:

https://msdn.microsoft.com/en-us/library/windows/desktop/aa363967(v=vs.85).aspx

Base Filtering Engine (BFE)

A service that controls the operation of the Windows Filtering Platform. It performs the following tasks. Accepts filters and other configuration settings for the platform. Reports the current state of the system, including statistics. Enforces the security model for accepting configuration in the platform. For example, a local administrator can add filters but other users can only view them. Plumbs configuration settings to other modules in the system. For example, IPsec negotiation polices go to IKE/AuthIP keying modules, filters go to the filter engine.

Adding this thread from reddit for reference:

https://www.reddit.com/r/sysadmin/comments/3vz48t/symantec_endpoint_filters_all_traffic_to_1_gbs/

Brian,  Thanks for posting the reddit thread. 

Update on my issue for those still following,  Still an issue but still working with Backline Engineers on the issue.

We have found there is an issue during the uninstallation of various components of SEP that leaves registry components behind. 

I have successfully installed AV Basic virus and malware protection only on my servers and achieved 10 Gbps throughput. 

Process:  Install all components of SEP, Reboot, Uninstall SEP, REboot,  Install only AV Basic protection and reboot.  You will have bare minimum protection but get your bandwidth.

Issue:  This does not resolve the problem at hand.  If I repeat the process but include AV Basic protection and IPS I lose 80% of my throughput.  So we cannot utilize any features other than basic virus protection without suffering from tremendous bandwidth loss on a 10Gb network.

Continuing to work with SEP.

Darren

Hi,

Just a short message to inform you that we have the same issue. Your 2 workarounds are working for us.

Cazza

 

 

Glad to hear.  This is an acceptable workaround for High Performance Server solutions but not a fix for their overall product.  I am still pushing them to resolve their horrendous issues with IPS and its affect on High Bandwidth networks.

Darren

 

 

 

 

Update: I knew Symantec would either acknowledge the issue and correct it or acknowledge the issue and say we will fix it later.  The latter has occurred.  Symantec is aware that the IPS feature severly degrades high bandwidth networks however since the IPS feature,(to paraphrase SEP Support), "Has not been QA tested in 10 Gbps networks it is not supported in those environments".  I am waiting on the official response and technical article from them however there you have it folks.

10Gbps networks are not supported with the SEP IPS feature.

All I can say is I tried to get this resolved.  I am a bit frustrated as I spent an incredible amount of my time troubleshooting this along with the SEP Engineers to help them come to a resolution and they decided to take this route.  What can you do when you are a smaller company.  Try alternate products thats what.

I will post another final update with their technical reasoning once I have it.

Darren

 

 

 

...and considering "other" products don't seem to have this issue and suport 10Gbps, you would think it would be possible to correct. I don't know the code specifics of SEP and the man hours it would take to fix but I'm guessing it's not feasible. Still odd considering it's Symantec but I'm not going to speculate.

It sucks you spent all the time you did, but it is appreciated by us customers because at least we have an answer, even though it's not what we want to hear.

OK For anyone still following I wanted to post my final comment.  Officially Symantec has not QA tested IPS in high bandwidth networks.  As such they will not supoort it in high bandwidth networks.

I believe they will address this in future major revisions of SEP but for anyone running 12.x and earlier (current versions) dont expect to use anything but basic virus protection if you are running a 10Gb network.

This is the official article they provided me explaining this.

https://support.symantec.com/en_US/article.TECH92440.html

Thanks for following and I hope this information above proves useful to others.

Darren