SEP Slows Down Visual Studio Build
Greetings,
Our corporate IT organization has decided to rollout SEP to our R&D organization. A small minority of the developers that are part of the early beta test are seeing their Visual Studio build times seriously degrade after installed SEP. With SEP installed they are seeing build times on the order of 60 minutes. Once they uninstall SEP build times decrease to 12-15 minutes. I'm concerned that more developers are going to see this issue when IT rolls this out company wide next week.
Some more details, IT is requiring us to use SEP version 11.0.4014.25. We really don't have any control over the version since it's being pushed out by a central IT group. Does anyone have any ideas for improving performance on the impacted clients? I don't see any obvious solution in the following KB artical, which deals with PC performance issues.
http://service1.symantec.com/SUPPORT/ent-security....
Regards,
Jason
Comments
SEP Slows Down Visual Studio Build
Hi,
If you could let us know the features of endpoint that are installed on the client machines in question. Is it only AV/AS or is it with NTP & PTP.
Thanks & Regards Sandip C Sali
Please upgarde to 11.0.4202
Please upgarde to 11.0.4202 i.e Mr4 MP2 & see it should help.:)
In my opnion an upgrade to
In my opnion an upgrade to SEP MR4MP2 will help as I was working on a similar case and the issue got resolved with an upgrade.
Prachand Kumar MCSE-2003 Symantec Technical Specialist (SCTS)
Hi Sandip, AV/AS, NTP, and
Hi Sandip,
AV/AS, NTP, and NTP are all turned on.
Kavin,
At this point IT is rolling out 1.0.4014.25. I don't have much control over the version.
Regards,
Jason
It is the issue with File
It is the issue with File System Auto-Protect ..the real time scanning scan the files of BS project build and whenever you try to rebuild or change all the files are monitored and when they are changed they are again scanned..
Since the project files keep on changing the projetc files continuos scanning process slows it down..
best option would be to make necessary Centralized exceptions for Visual Studio..
VMWARE-- SEP 12.1 vs McAfee vs Trend Micro
Hi Vikram, That seems very
Hi Vikram,
That seems very plausible. Is there a way I can disable File System Auto-Protect on the client as a test case?
Cheers,
Jason
Disabling the file system
Disabling the file system auto protect means disabling the Antivirus hence I wont recommend that.
Try to upgarde to MR4 MP2 that should fix your issue.
Since the client will be a
Since the client will be a managed client I don't think you will have an option to disable the Auto Protect
You can find Auto Protect by opening the SEP UI. Click on Options AV/AVS ---> Change setting---> File system Auto Protect
Above all it is not reccomended to disable AP.
Prachand Kumar MCSE-2003 Symantec Technical Specialist (SCTS)
Thanks folks. I've requested
Normal
0
false
false
false
EN-US
X-NONE
X-NONE
/* Style Definitions */
table.MsoNormalTable
{mso-style-name:"Table Normal";
mso-tstyle-rowband-size:0;
mso-tstyle-colband-size:0;
mso-style-noshow:yes;
mso-style-priority:99;
mso-style-qformat:yes;
mso-style-parent:"";
mso-padding-alt:0in 5.4pt 0in 5.4pt;
mso-para-margin:0in;
mso-para-margin-bottom:.0001pt;
mso-pagination:widow-orphan;
font-size:11.0pt;
font-family:"Calibri","sans-serif";
mso-ascii-font-family:Calibri;
mso-ascii-theme-font:minor-latin;
mso-fareast-font-family:"Times New Roman";
mso-fareast-theme-font:minor-fareast;
mso-hansi-font-family:Calibri;
mso-hansi-theme-font:minor-latin;}
MSBuild.exe - The Microsoft build tool (probably the most important)
Csc.exe - Visual Studio C# compiler
Cl.exe - Visual Studio C++ compiler
Thanks for the help,
Jason
Hi Jason, I think you are
Hi Jason,
I think you are following the correct path with creating the exceptions for Visual Studio, but I am confused a little bit at why you are adding exceptions for the Style Definitions?? Or are you? I guess the way you posted it confused me, because those are not executibles are they? Anyway I would think adding the exception for the three compilers would be enough. What you should do it test it with just the exception for the three compilers and see how much your build times decrease. If you are happy with the results then that is it. If you don't see the reduced build time you should try using a tool such as the task manager or a better free one called process explorer to see what other .exe are running during the builds. This might give you a better idea of what you need to exlude. The link to the process explorer download is here: http://download.cnet.com/Process-Explorer/3000-209... . Don't forget to mark someones answer as the "solution" if it ends up working. Probably whoever was the one that suggested creating the exceptions above, I think that was the correct answer.
Grant-
Please don't forget to mark your thread solved with whatever answer helped you : )
I think creating this much
I think creating this much exceptions should help you.
But Again try to install MR4 MP2 on one of the client & see if that helps?
Upgrading is not going to
Upgrading is not going to help.
We're also seeing the same issue crop up with a lot of people using VSTS, VSTFS, Visual Studio 2005 and Visual Studio 2008.
I've worked around the issue by disabling the PTP component, and excluding the VS folders from being scanned. This helped a lot.
You may actually want to change the feature to "Scan files when accessed / modified" to "Scan Files when they are backed up". This prevents SEP from injecting it's own code into the core runtime files reqd. for VS and helpe to alleviate the problem.
HTH
Abhishek Pradhan, PMP, MCT
Consultant | Microsoft Corp.
Blog: http://blog.abhishekpradhan.net | SIG Lead - Pune IT Pro (Microsoft Pune User Group) | http://www.puneusergroup.org
The NAC add on was a solution
The NAC add on was a solution for one customer that was using IBM clearcase.
When Clearcase was running the following happened.
1. A firewall rule came up to block all traffic except to the AD server, email, and the Clearcase build servers.
2. Auto-protect was turned off.
When Clearcase was exited..
1. Everything reverted back.
An alternative, would be the NAC add-on, but just create a firewall rule that blocks all HTTP traffic, when developing, and revert back when done. Most viruses come from web surfing anyways...
There is an online portal, save yourself the long hold times. Create ticket online, then call in with ticket # in hand :-) http://mysupport.symantec.com "We backup data to restore, we don't backup data just to back it up."
Adding MSBuild and the other
Adding MSBuild and the other build components to the exception list didn't help.
Abhishek, the development environment you described sounds exactly like what we have in our lab. We use just about every aspect of Team Foundation including SCM, Team Build, and WIT. I'm going to forward on your email to our IT group, but I have a feeling those changes will prove to be too drastic for them to consider. I just hope this is a fairly isolated problem since SEP is being rolled out to a few hundred developers this week.
Regards,
Jason
Hi Jason, I really think it
Hi Jason,
I really think it would be helpful if you could run some sort of process checker like the windows task manager (there are better ones out their google will help you find tons) to see exactly what process is eating up the CPU during your builds. As you stated that the build time go down again after you uninstall SEP then it is most likely a SEP process, but figuring out which one could help narrow down exactly what is going on. With a jump in build times from 10 min or so to 30 min+ you should definitely see an increase in CPU usage of one or more processes. Again try a better process checker than the task manager if you can. Sometimes it is like trying to find a needle in a haystack if you use the standard windows task manager.
Cheers
Grant
Please don't forget to mark your thread solved with whatever answer helped you : )
what exact did you add exception for VS?
Some folks are talks to excude the scan on Visual Studio, has anyone done yet? what exact did you add exception? how confident you feel safe about the setting?
Thanks
Hi Grant, We ran Process
Hi Grant,
We ran Process Explorer when we were first debugging the problem (I like to see the process tree view). That's how we came up with the list of tasks to exclude. MsBuild.exe always runs, with csc.exe being the primary subprocess. Based on Abhishek's email, IT just sent me an email saying they:
1) Removed the lock settings for PTP
2) Changed the scan settings to "Scan Files when modified" and also removed the settings lock.
We'll see if any of this helps...
Chenh,
The processes IT added to the exception list are:
MSBuild.exe - The Microsoft build tool (probably the most important)
Csc.exe - Visual Studio C# compiler
Cl.exe - Visual Studio C++ compiler
However, it didn't appear to help. Sorry I don't know how they added them so I have only minimal confidence they were added correctly. I guess if you want to get technical the Visual Studio process is actually devenv, but it has little to do with the actual build.
Regards,
Jason
Making the changes Abhishek
Making the changes Abhishek recommended helped, but didn't fix the issue. Here is where we're at:
Default IT configuration build time: ~60 minutes
Applying Abhishek's changes build time: ~44 minutes
Uninstalling SEP build time: ~15 minutes
At this point I'm going to recommend any developer experiencing slow builds after installing SEP just uninstall.
Seriously, the NAC add-on
Seriously, the NAC add-on was the fix for one customer. A Symantec SE can show your IT group how simple an install it is along with config.
You may be surprised to find that the add-on is actually included in some of the Protection Suite bundles... Your company may already own it...
AV poses a problem period, and no matter what you do, build times will be affected. But you can get really creative on the solution, short of brute force (exclusions) that actually compromise security. SNAC will give the flexibility you need, additional security, and better options down the road for offering compliance checking to machines.
There is an online portal, save yourself the long hold times. Create ticket online, then call in with ticket # in hand :-) http://mysupport.symantec.com "We backup data to restore, we don't backup data just to back it up."
If you are using source safe,
If you are using source safe, or if your project files are located on the network, you can try turning off network scanning to see if it helps.
Hi Jason
Just wondering how SEP held up once you deployed to all your devs? Are you still seeing the same slowdown?
Grant-
Please don't forget to mark your thread solved with whatever answer helped you : )
Hi Grant, 8 of the 10
Hi Grant,
8 of the 10 developers on the project are still seeing the issue. Most of them have SEP, but one or two even see the problem with NAV 10. I'm going to work with our IT group again tomorrow in creating a new exclusion list, but I'm not optimistic it will fix the issue. I did find that disabling some of the hidden devices (symevent, symredrv, NAVENG, NAVEX15) in Device Manager will restore performance, but of course this also disables the realtime protection. Even in NAV we can't just disable virus scanning we have to disable the devices. Are there any tools/utilities we might have installed that interfer or slow down the hidden filter devices? Do you have any other advice for what I should try tomorrow with the IT group?
Regards,
Jason
I can't think of anything
I can't think of anything that you could do to SEP itself to increase your build times, but I think that if you build your exceptions list correctly it would have to increase the build times back to the ~15 min mark. Really if SEP is excluding every bit of your compiler then there should be no reason to see the increased build times UNLESS SEP is eating up such a huge part of your CPU or Memory that the computer just can't handle it. But I am assuming that since you are writing programs that take that long to compile you are using fairly new computers with good CPU's and lots of memory. Is SEP (or some other process) eating up one of your cores on the CPU or lots of memory during the compile? What is the CPU and memory usage of the cores during the compile and what process is using them? I am not trying to be skeptical of what you are doing, but again if SEP is really excluding the processes then there should be no reason to see the slowdown. Also maybe you should try to exclude the workspace (temporarily) of the project to see if SEP is trying to on access scan the files as you are trying to compile/read from them.
Also as a side note you said that once you uninstalled SEP you saw the half hour decrease in build times, but does this mean there was no AV installed at all when you were doing this? Have you tried other AV's? I guess I am really asking what were you guys using before the switch to SEP?
Grant-
Please don't forget to mark your thread solved with whatever answer helped you : )
It appears that disabling the
It appears that disabling the "scan for security risks" under the File System Auto-Protect options fixes the build performance issue. According to the SEP help documentation this option, "Scans for security risks such as adware and spyware". Also, according to the documentation it seems reasonable to make this a user controlled setting so that is what I'm pursueing with our IT group right now. Currently the setting is locked to 'on' in our configuration.
Does this new development bring any other issues to mind that might be the utlimate root cause? The documentation is fairly thin on what the option actually does, how does it differ from the real-time virus scanning?
Regards,
Jason
We have been experiencing the
We have been experiencing the same problems with Visual Studio on Windows XP 32-bit. In our case the slow down in compilation performance increased about 10-fold, from 45mins to several hours!!! This behavior renders SEP useless in our environment.
I have seen someone asking to check for CPU/RAM usage etc. The issue actually lies with SEP blocking the harddisk access (while monitoring?) to other processes, so the CPU usage goes down as "everyone" waits for a piece of hard-disk I/O bandwidth.
Initially, we thought that something else blocks the harddisk access, since for the test case we have disabled SEP, using its menus. Apparently, this doesn't stop the monitoring!!! Only recently we figured out that SEP is at fault, once we started to reinstall systems to find out the problematic application.
Also, during the tests we have found out that disabling SEP's items one by one including Network Monitoring (has to be done from its client GUI window) "recovers" about 10%-20% of the build performance. Still this is a far cry from the original benchmark.
We haven't found any way of fixing the issue, beside uninstalling SEP.
With ANY development
With ANY development platform, look at their support site and search for AV... You'll find countless articles about settings to use or performance issues found.
Frankly, the way a complier works, it's understandable why there is a slowdown the way AV engines scan files.
That said, either you turn off auto-protect, but you compromise security or you use something like the SNAC plug-in that I've successfully deployed to solve one customer issue using IBM's ClearCase.
There is an online portal, save yourself the long hold times. Create ticket online, then call in with ticket # in hand :-) http://mysupport.symantec.com "We backup data to restore, we don't backup data just to back it up."
Would you like to reply?
Login or Register to post your comment.