Video Screencast Help

Symantec Endpoint Protection slows file transfer and network speed on VMs in VMware environment by a factor of four

Created: 25 Jul 2013 | 12 comments

We are experiencing significant reduction in file transfer rate and network speed with in our LAN between Virtual Machines running Windows 7 and Windows Server 2008 R2 guest OSs, Symantec Endpoint Protection (SEP) installed is 12.1.2015.2015.

VMs with all features of SEP installed have file transfer speed of about 30 MB/sec vs 120 MB/sec with no SEP installed.

Network speeds measured using the iperf utility shows a similar speed degradation of 4 times, 350 Mb/sec vs 1400 Mb/sec.

To simplify and exclude all extraneous factors we performed file transfer and network speed test where all VMs are hosted on the same VMware ESXi virtualization hosts (Version ESXi 5.1.0 Build 1117900). All VMs are x64 and the ethernet adapters are VMXNET 3, VMWare tools are installed and updated to the latest versions. Virtualization Host CPU usage is 20% and Memory Usage is 40% during the test.

The only article I found on the subject was http://www.symantec.com/connect/forums/sep-121-ru2.... We already had the power setting to high performance so the solution did not help our case.

We tried enabling only the relevant features of SEP, it did not result in any significant improvement. Only installing SEP Core or unistalling SEP completely seem to be the only solution.

This seems to be a much bigger trade off between Security and Network Speed than anticipated. Any suggestions and comments are welcome.

 

Operating Systems:

Comments 12 CommentsJump to latest comment

.Brian's picture

What happens with the firewall disabled?

Have you tried 12.1 RU3?

 

Please click the "Mark as solution" link at bottom left on the post that best answers your question. This will benefit admins looking for a solution to the same problem.

sumit7890's picture

Brian, Thanks for your recommendation, I did try the test earlier with Firewall feature uninstalled.

Virus, Spyware and Basic Download Protection: All features Installed and On

Proactive Threat Protection: All features Installed and On

Network Threat Protection: Installed and On
Sub-feature - Intrusion Prevention On
Sub-feature - Firewall uninstalled.

Results:

File transfer rate at 36 MB/sec vs 130 MB/sec
Iperf network speed 913 Mb/sec vs 2385 Mb/sec

As expected Updating to 12.1.3 might be an initial suggestion and perhaps a viable Hit and Trial. But SEP 12.1.2015 being a stable release has anyone else faced similar issues? Does SEP 12.1.3 addresses these issues?

 

.Brian's picture

Support will be able to determine this.

You will need to enable WPP logging via the SymHelp tool and provide Wireshark traces. They can than make the determination as to what's going on.

I only suggest Ru3 because it's the latest (and this may have been addressed) but if it still occurs than there may be a previously undiscovered bug.

Please click the "Mark as solution" link at bottom left on the post that best answers your question. This will benefit admins looking for a solution to the same problem.

Mithun Sanghavi's picture

Hello,

Please check these 2 recent Articles:

SEP 12.1.2 Best Practices on Citrix Virtual Desktops ( Provisioning Services) -Part 1-

SEP 12.1.2 Best Practices on Citrix Virtual Desktops ( Provisioning Services) -Part 2-

I would suggest you to check the Virtual Image Exception (VIE) tool - 

The Symantec Endpoint Protection (SEP) 12.1 client checks for this attribute before scanning files and skips scanning any files that are marked as "known good" by the VIE tool. Scans on VDI clients created with images processed by the VIE tool will experience lower I/O load, CPU usage, and network bandwidth usage during scheduled and manual scans.

The Virtual Image Exception (VIE) tool was created specifically for VDI environments deployed using shared base images. The VIE tool provides the ability to exempt the files in a base image from SEP client scans once the image is deployed. If the files are updated or changed in any way, the updated/changed files will be scanned as usual.

It is suggested that VM admins either record their VIE exceptions list prior to their VM template machine being added to the domain, or place the computer account for the VM template machine into an OU with no GPOs applied.  Once the VIEtool's exceptions list has been created, GPOs can then be applied to the system as normal. 

Please see the following article for more information on use of the VIE tool:

http://www.symantec.com/business/support/resources/sites/BUSINESS/content/staging/DOCUMENTATION/4000/DOC4335/en_US/2.0/sep_virtual_image_exception.pdf

Here are the Steps and Action:

Step 1: On the base image, perform a full scan all of the files to ensure that the files are clean. If the Symantec Endpoint Protection client quarantines infected files, you must repair or delete the quarantined files to remove them from quarantine.

Step 2: Ensure that the client's quarantine is empty. 

Step 3: Run the Virtual Image Exception tool from the command line to mark the base image files. Check the Article:

Step 4: Enable the feature in Symantec Endpoint Protection Manager so that your clients know to look for and bypass the marked files when a scan runs.

Step 5: Remove the Virtual Image Exception tool from the base image.

The Virtual Image Exception tool supports fixed, local drives. It works with the files that conform to the New Technology File System (NTFS) standard.

Reference: 

Symantec Endpoint Protection Virtual Image Exception User Guide 12.1

http://www.symantec.com/docs/DOC4335

About the Symantec Virtual Image Exception tool

http://www.symantec.com/docs/TECH172218

Symantec Endpoint Protection 12.1 - Virtualization Best Practices

http://www.symantec.com/docs/TECH173650

SEP 12.1 & Virtualization

https://www-secure.symantec.com/connect/articles/sep-121-virtualization

Hope that helps!!

Mithun Sanghavi
Senior Consultant
MIM | MCSA | MCTS | STS | SSE | SSE+ | ITIL v3

Don't forget to mark your thread as 'SOLVED' with the answer that best helped you.

hforman's picture

Actually, that version is NOT a stable release.  It causes problems with the teefer driver and slows down your network.  I have turned on all the firewalls and have verified that 12.1.3 fixes the known issue found in 12.1. RU2.  Your choices, just after RU2 came out were documented as either turn off the firewall, wait for RU3 or revert to the previous version.  Reverting was awful.  Everything OK with RU3.

sumit7890's picture

Brian, thanks for your suggestion. I have support to assist me with SymHelp and Wireshark.

Mithun, I looked up the Best Practices documents and articles you provided. Though valuable,I am not sure if they will apply to this case because I am not in Citrix environment and I am not trying to optimize scans. I can create exceptions for certain files but I could not find the files for VMware environment.

The slow network and file transfer speed that we are experiencing are during normal operation of machine and not during the scans.

.Brian's picture

Please update this thread with your progress if you can.

Please click the "Mark as solution" link at bottom left on the post that best answers your question. This will benefit admins looking for a solution to the same problem.

Mithun Sanghavi's picture

Hello,

Could you please let us know if this file transfer issue occurying from Server to client machine or vice versa or both ways?

Secondly, could you try installing the AV/AS component only and disable the symtdi.sys driver from the machines and check if that helps.

The SEP firewall components will not protect a VMware guest operating system. 

If the VMware guest operating system requires SEP protection, it must be installed directly to the VMware guest Operating System.

For Vmware Environment, check these Articles:

Guidelines for installing and running the Symantec Endpoint Protection Manager (SEPM) in a VMware image.

http://www.symantec.com/docs/TECH132456

Best Practices for Symantec Endpoint Protection in Virtual Environments

http://www.symantec.com/docs/TECH95300

Using Symantec Endpoint Protection in virtual infrastructures

http://www.symantec.com/docs/HOWTO81060

Best Practice for Symantec Endpoint Protection Scheduled Scans in VMWare

http://www.symantec.com/docs/TECH95928

SEPM: poor database performance

http://www.symantec.com/docs/TECH155046

Hope that helps!!

Mithun Sanghavi
Senior Consultant
MIM | MCSA | MCTS | STS | SSE | SSE+ | ITIL v3

Don't forget to mark your thread as 'SOLVED' with the answer that best helped you.

smakovits's picture

cor089f702

N iperf -s cor089f701 N iperf -c 172.29.67.55 -P 10 -t 240 -w 130000 [SUM]  0.0-240.0 sec   149 GBytes  5.32 Gbits/sec Same host, same vLAN, Same vSwitch
cor089f702 N iperf.exe -s -P 0 -i 1 -p 5001 -f k cor089f701 N iperf.exe -c 172.29.67.55 -P 1 -i 1 -p 5001 -f k -t 10 -T 1 [164]  0.0-10.0 sec  0.96 GBytes  0.83 Gbits/sec Same host, same vLAN, Same vSwitch
cor089f702 N iperf.exe -s -P 0 -i 1 -p 5001 -f k cor089f701 N iperf.exe -c 172.29.67.55 -P 10 -i 1 -p 5001 -f k -t 10 -T 1 [SUM]  0.0-10.3 sec  3.72 GBytes  3.10 Gbits/sec Same host, same vLAN, Same vSwitch
cor089f702 N file copy cor089f701 N file copy 75MB/s (both directions) Same host, same vLAN, Same vSwitch
cor089f702 N iperf -s cor089f701 Y iperf -c 172.29.67.55 -P 10 -t 240 -w 130000 [SUM]  0.0-274.2 sec  35.7 GBytes  1.12 Gbits/sec Same host, same vLAN, Same vSwitch
cor089f702 N file copy cor089f701 Y file copy 28MB/s (F702-F701) 21MB/s (F701-F702) Same host, same vLAN, Same vSwitch
cor089f702 N iperf -s cor089f701 Y iperf -c 172.29.67.55 -P 10 -t 240 -w 130000 [SUM]  0.0-240.5 sec  5.08 GBytes   181 Mbits/sec Same host, same vLAN, Same vSwitch
cor089f702 N file copy cor089f701 Y file copy 16MB/s (both directions) Same host, same vLAN, Same vSwitch
cor089f702 NTP removed iperf -s cor089f701 NTP removed iperf -c 172.29.67.55 -P 10 -t 240 -w 130000 [SUM]  0.0-240.3 sec  27.4 GBytes   978 Mbits/sec Same host, same vLAN, Same vSwitch
cor089f702 NTP removed file copy cor089f701 NTP removed file copy 30-60MB/s (F702-F701) Same host, same vLAN, Same vSwitch
cor089f702 AV only iperf -s cor089f701 AV only iperf -c 172.29.67.55 -P 10 -t 240 -w 130000 [SUM]  0.0-240.0 sec  33.8 GBytes  1.21 Gbits/sec Same host, same vLAN, Same vSwitch
cor089f702 AV only file copy cor089f701 AV only file copy 50-60MB/s (F702-F701) Same host, same vLAN, Same vSwitch
cor089f702 N iperf -s cor089f701 N iperf -c 172.29.67.55 -P 10 -t 240 -w 130000 [SUM]  0.0-241.3 sec   128 GBytes  4.56 Gbits/sec Same host, same vLAN, Same vSwitch

 

I am most curious about your results as well, because after some exhaustive testing, I can say SEP definitely has a major impact on my systems.

Mithun Sanghavi's picture

Hello,

In your case, I would request you to please open a Case with Symantec Technical Support Team and PM me the Case #.

How to create a new case in MySupport

http://www.symantec.com/docs/TECH58873

Phone numbers to contact Tech Support:-

Regional Support Telephone Numbers:

United States: 800-342-0652 (407-357-7600 from outside the United States)
Australia: 1300 365510 (+61 2 8220 7111 from outside Australia)
United Kingdom: +44 (0) 870 606 6000

Additional contact numbers: http://www.symantec.com/business/support/contact_techsupp_static.jsp

Hope that helps!!

Mithun Sanghavi
Senior Consultant
MIM | MCSA | MCTS | STS | SSE | SSE+ | ITIL v3

Don't forget to mark your thread as 'SOLVED' with the answer that best helped you.

sumit7890's picture

Mithun, I did create a case with Symantec Tech Support back in June but don't have a resolution yet. I have sent you the case # in PM.

Posting my issue in this community forum is an alternate attempt to find a solution. Or perhaps understand that file transfer and network speed being four times slower is normal and expected behavior on Virtual Machines with Symantec Endpoint Protection installed in VMware environment.