Endpoint Protection

Our client version is 11.0.6300.803 and since a week or two, out of the blue, I have multiple help calls of workstations running low of memory errors.

After some research msiexec.exe is trying to re-install the SEP client and it does this non-stop until you reboot. Then after a while it starts in all over again. No errors, it just reinstalls and then starts over again over and over and over.

Has there been a windows update or some definition file that could have started this problem? This is happening on client that I have all over the world. The only way I've found to fix it is to uninstall/reboot/remove all folder related to symantec/reinstall. So far that is working but if I have to do this to 800 PCs...I don't want to think about it.  Help!

Forgot to add that this is, as far as I can see, only happening on XP SP3.

See if this helps

http://www.symantec.com/business/support/index?page=content&id=TECH104575

OR try this

try this

go to start ->run

type msiexec /unregserver

and then

type msiexec /regserver

try running the installation..hope this helps

if not try this

Restart in safe mode your PC go to c:\windows\system32 and rename msi.dll,msiexec.exe and msihnd.dll you can chance the ext .dll to .old
Then restart windows to normal mode and reinstal windows installer and try the installation
( you can find the windows installer in CD1 of symante endpoint protection )

Hello,

On how many machines this is happening?

How did you push the package through GPO??..Do you have any rule to delete sep shortcut or something after installation is done?

Check this Article:

http://www.symantec.com/docs/TECH96345 

Were these client machines upgrade / migrated from the Previous version of SEP v.11??

If yes, try the steps below:

Hello,

I too had similar issue and was solved with the same article which Mithun has suggested above.

As far as I can tell so far all the PCs that have this problem have been upgraded from an earlier version of 11.x and the upgrade was done through SEPM. I have not seen any Windows 7 or Windows servers having this problem (so far), like I said in my first post this just started happening and I've been on this version since July.

Also, as I mentioned above, I know how to fix it on a one PC at a time scenario. How do I fix this on the dozens (close to 100 so far) this is happening to without touching each PC! That's just ridiculous! You should not have to do this with a corporate product!

If others are having this issue I want to know why this is doing this! And I would like a patch or something I can push out.

When I upgrade to 12.1 and push out the new client via SEPM, will this happen again?

I am experiencing this same issue and it's becoming a real annoyance. Many more machines in my office are starting to get affected by this issue. The computer would slow down to a crawl while MSI processes attempt to install/reinstall/reconfigure the client and it would eventually lead to virtual memory being used, and crashing the computer.

Beginning a Windows Installer transaction: {5E2E4797-502A-4FFD-81EC-F9BA8BF0C581}. Client Process Id: 8780.

Product: Symantec Endpoint Protection -- Configuration completed successfully.

Windows Installer reconfigured the product. Product Name: Symantec Endpoint Protection. Product Version: 11.0.7000.975. Product Language: 1033. Reconfiguration success or error status: 0.

Ending a Windows Installer transaction: {5E2E4797-502A-4FFD-81EC-F9BA8BF0C581}. Client Process Id: 8780.

Detection of product '{5E2E4797-502A-4FFD-81EC-F9BA8BF0C581}', feature 'Core' failed during request for component '{30466A58-8174-4ED4-9171-A4D739E84E3A}'

I'll attempt the msiexec /unregserver and reregister it and let you know how it goes....

You got to be kidding me...once again Symantecrap demonstrates their complete incompentence!

We are having the same problem here, with no fix in sight.

I should note that this appears to be happening on computers that were previously infected/cleaned.

Have you checked the machine if there was any type of malware infection on them. If found ou can submit them or temporarily block using app control.

If you are having issue on such a large number of machines Open a technical support case 

If the machine were to have malware infection on it, it's sad that the SEP client breaks instead of finding said infection.

If the several machines that are affected by this issue have had malware infections on it in the past I wouldn't have a job. lol.

unregistering/reregistering the windows installer failed to resolve the issue on my own machine. I cleanwiped it and pushed 11.0.7 and it seems to have done the trick.

I've begun to roll out 11.0.7 over installations which are affected to see if it resolves the issue. I will hestitate greatly to go desk to desk and run a cleanwipe - it's just not feasible until I can find out for sure if doing an installation over a previous one won't fix the issue permanently.

When you say "clean wiped" do you mean a new OS image or a uninstall/reinstall of the SEP client?

fyi 11.0.7 hasn't made a difference. On one machine, it would go into this loop, then be ok for a day, then back to this problem.

This issue is currently being investigated.

Hello all.  Our engineering team are actively looking into this and trying to understand the issue.  What would help immensely is more log files and more information.

Have any of you logged support cases for this issue yet?  If so, can we the case ID's so we can review internally and work towards the solution.

thanks!

CleanWipe is an application to remove Symantec if the normal uninstallation procedure fails. Usually done when the installation data on the PC gets corrupted.

If Symantec is stuck in a reinstall loop, this usually means that there is an error in writing the new files. This usually happens during remote installations. We had the same problem when deploying over the LAN using a different method. Although regardless of the methods, I think the procedure remains the same: Copy the installation files, run setup.exe. And sometimes, the files does not reach the host in one piece.

You could try using the deployment wizard and add the SEPPrep in the package. Found in \Tools\NoSupport\CompetitiveUninstall of the SEP installer. It would give you additional log information and get the SEP_INST.log together with the log generated by the SEPPrep.

But if that fails, you need to remove the processes on the affected PCs. Which I hope you can still reach remotely. The admins blocked specific ports on our LAN so I can't do a taskkill and have to fix them manually.

yep, its an app which completely uninstalls all traces of symantec endpoint protection, liveupdate, etc.

Mine are clients that were updated to a newer Sep client (usually from 11.0.5) up to 11.0.6 and newer with AV/PTP/NTP. They ran fine for a while till about a week ago.

PTP gets disabled and NTP just shows a couple Exclamation Points in the client. Then the MSIexec.exe process is continually running.

A reboot will sometimes fix the issue until it just starts again. I havent had a chance to get my hands on a client until today and had to get it working so the employee could get back to work. I uninstalled the client and reinstalled with the files on the local machine, as was done previously.

I suspect the issue will reappear soon enough.

We are experincing the same issue on 300+ machines.  

I am curious to find out how many companies are affected.. 

Did everyone's problems start around the 20th?

Btw, we fixed this by reinstallation. Trying to stop the msiexec from looping is your first task. Then it's either cleanwipe, or manual removal followed by registry cleaning. Then install from a package. I usually make new packages quarterly. Just because. :D

I believe it started around the 20th for us as well.

@datacenterdan & Chsu: Did your installation roll back on the 20th? Not sure if definition updates does that. But highly unlikely. Maybe a new package would do the trick. Not to be confused by definition updates. If that's the case, force update the clients with newer definitions.

Hi,

It seems to be a known issue, if the issue exists, open a support case and provide the logs.

Hello all.  Our engineering

Hi,

I came across this KB

http://www.symantec.com/business/support/index?page=content&id=TECH170667&locale=en_US

I have heard that they are working with Microsoft now to get a Hot Patch that will fix the issue.

Hi,

Workaround suggested ( from the KB), keep checking

http://www.symantec.com/business/support/index?page=content&id=TECH170667&locale=en_US

Solution

Hi all,

I wanted to update you on the status of this issue:

The memory leak was identified to be tied to the trust verification process.  Further analysis indicated that the leak itself is actually in Microsoft’s crypt32.  There was an update for crypt32.dll that went out on the 19^th, which coincides with the beginning of the reports from our customers (http://support.microsoft.com/kb/2616676).   This update has both embedded signature changes, as well as code changes for the DigiNotar signature types, because these were previously not handled (http://technet.microsoft.com/en-us/security/advisory/2607712).

We submitted our UMDH dumps to our Microsoft TAM, and he was able to tie the leak to a known bug that Microsoft is already working on.  The Windows Sustained Engineering team are currently working on a hotfix for this. It is set to Priority 0 which is the highest level.  We are awaiting an ETA for delivery.

Our current suggestion is to do one of the following:

1. Disable running verification on every process launch to slow the leak. This is a configuration in PTP that can be done from the management console.
2. Revert the DigiNotar patch (http://support.microsoft.com/kb/2616676  )

More details in this KB: http://www.symantec.com/docs/TECH170667

thanks!

See, this is concrete proof that the war on malware if as effective and successful as the war on drugs. Symantec & MS, 2 huge entities with vast amounts of engineering & tech know how...basically broke their own stuff.

And even better...we pay for it!

And in the meantime, XP Antivirus 20...15 or whatever it is up to now...starts without issue whatsoever on any machine it's installed on I'm sure!

Sad, sad sad.

But keep trying guys...and we'll keep buying I'm sure.

Time to Upgrade to Linux

Haha. If more companies move to Linux, that might make the malware authors target linux. And being a robust OS to begin with, we'll probably be dealing with expert coders here and not simple script kiddies. - From a Linux home user. :D

I have this exact issue on about 30 of 400 Win XP Pcs. I thought I could fix the issue by intalling 11.7000 but still am getting errors like below

Detection of product '{5E2E4797-502A-4FFD-81EC-F9BA8BF0C581}', feature 'Core' failed during request for component '{30466A58-8174-4ED4-9171-A4D739E84E3A}'

There is a hotfix available from Microsoft that addresses this memory leak in crypt32.dll. See the following Microsoft technical article:

http://support.microsoft.com/kb/959658 ("A memory leak problem occurs when you run an application that uses the HttpSendRequest function of the WinHTTP API or of the WinINet API to send Secure Sockets Layer requests in Windows XP Service Pack 3").

This hotfix is only for customers experiencing this issue and is available only by request to Microsoft.

Applying hotfix-968730 updates the version of Crypt32.dll to 5.131.2600.5779 19-Mar-2009 http://support.microsoft.com/kb/968730

Where as hotfix959658 updates the version of Crypt32.dll to 5.131.2600.5707 07-Nov-2008 http://support.microsoft.com/kb/959658 

Note: Please check the version of crypt32.dll on system it will be higher than the versions mentioned in the above hotfix.

As an alternative to applying the Microsoft hotfix in the Solution section above, you may do one of the following:

• In the Symantec Endpoint Protection TruScan settings, disable the "Scan new processes immediately" option to slow the leak. To further slow the leak, increase the TruScan interval (default is 1 hour).
  
  Monitor rtvscan.exe memory usage on the Endpoint Protection client. If it goes above 300MB then restart the rtvscan.exe service (the "Symantec Endpoint Protection" service) . No reboot should be necessary. After monitoring memory usage you may identify an hourly interval at which you can schedule a service restart.
  
  You may instead remove the Proactive Threat Protection component of the Endpoint Protection Client.
   
• Or remove the DigiNotar patch: http://support.microsoft.com/kb/2616676

more info : http://www.symantec.com/business/support/index?page=content&id=TECH170667&locale=en_US 
 

Solution

There is a hotfix available from Microsoft that addresses this memory leak in crypt32.dll. See the following Microsoft technical article:

http://support.microsoft.com/kb/959658 ("A memory leak problem occurs when you run an application that uses the HttpSendRequest function of the WinHTTP API or of the WinINet API to send Secure Sockets Layer requests in Windows XP Service Pack 3").

This hotfix is only for customers experiencing this issue and is available only by request to Microsoft. 800-936-4900

United States/Canada:+1-425-704-3638 | UK:0808 2349667 | Ireland:1-800-760-647 | Australia:1-800-151-075 |India:1800 266 3050 |New Zealand:0800-451917 |Singapore:800-1205517 |Malaysia:1-800-815217 |South Africa:080-09-83720 

Applying hotfix KB959658 downgrades the file version of Crypt32.dll to 5.131.2600.5707 07-Nov-2008

Applying hotfix KB968730 downgrades the file version of Crypt32.dll to 5.131.2600.5779 19-Mar-2009

Note: File version of Crypt32.dll on effected system will be higher than mentioned above something similar to “5.131.2600.6149"

As an alternative to applying the Microsoft hotfix in the Solution section above, you may do one of the following:

• In the Symantec Endpoint Protection TruScan settings, disable the "Scan new processes immediately" option to slow the leak. To further slow the leak, increase the TruScan interval (default is 1 hour).
  
  Monitor rtvscan.exe memory usage on the Endpoint Protection client. If it goes above 300MB then restart the rtvscan.exe service (the "Symantec Endpoint Protection" service) . No reboot should be necessary. After monitoring memory usage you may identify an hourly interval at which you can schedule a service restart.
  
  You may instead remove the Proactive Threat Protection component of the Endpoint Protection Client.
   
• Or remove the DigiNotar patch: http://support.microsoft.com/kb/2616676

There is a hotfix available from Microsoft that addresses this memory leak in crypt32.dll. See the following Microsoft technical article:

http://support.microsoft.com/kb/959658 ("A memory leak problem occurs when you run an application that uses the HttpSendRequest function of the WinHTTP API or of the WinINet API to send Secure Sockets Layer requests in Windows XP Service Pack 3").

http://support.microsoft.com/kb/968730 ("Windows Server 2003 and Windows XP clients cannot obtain certificates from a Windows Server 2008-based certification authority (CA) if the CA is configured to use SHA2 256 or higher encryption
")Applying hotfix KB959658 downgrades the file version of Crypt32.dll to 5.131.2600.5707 07-Nov-2008

Note: File version of Crypt32.dll on effected system will be higher than mentioned above something similar to “5.131.2600.6149"

Applying hotfix KB959658 downgrades the file version of Crypt32.dll to 5.131.2600.5707 07-Nov-2008

Applying hotfix KB968730 downgrades the file version of Crypt32.dll to 5.131.2600.5779 19-Mar-2009

Anyone get this hotfix to install?

I apply the patch, reboot and the file is back to 5.131.2600.6149

That hot fix is not the answer. It does not install most of the time. It needs too much hand holding.

And crippling SEP is not the answer either.

Until I see a patch from Symatnec or MS that permanently fixes the problem, I'm not marking anything in this tread as a "Solution".

I'm fast forwarding my SEP 12.1 rollout because of this debacle.

Ok, I must be missing something here.

Symantec says the problem is due to a memory leak in a file released as part of a Sept. 2011 Microsoft Security update, and posts this paragraph:

"We submitted our UMDH dumps to our Microsoft TAM, and he was able to tie the leak to a known bug that Microsoft is already working on.  The Windows Sustained Engineering team are currently working on a hotfix for this. It is set to Priority 0 which is the highest level.  We are awaiting an ETA for delivery."

So I backoff the PTP protection from my workstations and await the fix.

Then we are told that the fix is to install a hotfix from 2008, and, oh by the way, if a new fix comes out we may have to uninstall this fix.

If Microsoft is working on it, as stated above, then is the suggested fix from 2008 just a temporay fix?  Does installing it effect the security update that caused all this in the first place?

I need to have a solid direction on what the true solution is to this problem.  I can only calm upper managment for a limited amount of time....

There are a number of fixes posted in the Symantec article, although yes most of them do seem temporary. The "true" fix that you are looking for comes in two flavors.

1) Apply the hotfix, which rolls back Crypt32.dll. Since this is considered a hotfix, if Microsoft releases a fix for this to General Availability (AFAIK, Windows Update) then you will need to remove this hotfix in order to apply the GA patch.

2) Upgrade to 12.1

IMHO, Any of the other options listed reduce protection and/or increase vulernabilities.

Solid direction? You are asking a lot RC.

It's obvious they just want you to upgrade to 12.1

I'm looking at different products. Upgrading my entire inferstructure everytime there is a problem is not an option.

You are suggesting that my motive telling you to upgrade to 12.1 is to try and get your money, but from my understanding of it (I'm in support, not sales nor licensing) if you have a valid support contact you should already have access to 12.1. Even if you don't have access to 12.1 or don't want to upgrade to 12.1 you have another option, apply the Microsoft hotfix (SEP does require some properly working operating system components). We cannot fix a defect that is outside of our product (as we've stated before this is a Microsoft defect), nor do we control whether they release a patch as a hotfix or make it General Availability.

No I'm not suggestion that at all. I already have a 12.1 license.

What I'm saying is there really is no real fix except to upgrade.

The roll back and the unsupported hot fix are not the answer. Neither is crippling the client.

You guys should just bite the bullet and stop beating around the bush and tell people to upgrade, since that is the only real solution as far as I can see.

You know it's not our fuault that your product ties into a MS dll (not smart IMHO) that corrupts your product.

Yes, I will be upgrading my server and 600 clients within the next week.

We have thousands of XP SP3 machines just upgraded to SEP 11 RU7, and this issue is getting pretty serious on our end. Another thread about this, which describes the logs we're seeing, but is 'solved' is here (pertains specifically to RU7):

http://www.symantec.com/connect/forums/sep-client-1107000-reconfiguring-every-2-minutes

We're in the same boat RE: the hotfix. If it's going to need uninstallation later, just to upgrade the original wonky patch, AND it's not recommended unless you actually have the issue, it's not something our security team likes. Rolling back security, and increasing complexity, for something that shouldn't happen anyway is hard to get approved.

We've done some testing and found a few interesting tidbits. Here's some info we have around the crypt32.dll file this involves.

Clean XP SP3 VM -- crypt32.dll version .5512

After patch 2607712 -- crypt32.dll version .6147

After patch 2616676v2 -- crypt32.dll version .6149 dated 9/9/2011 2:12 am, 585k

After HotFix 959658 -- crypt32.dll version .6149 dated 9/9/2011 2:11 am, 586k

crypt32.dll file INSIDE the HotFix 959658 package -- version .5707 dated 11/07/08

We're wondering why the hotifx CHANGED the date modified, and size (and hash) of the crypt32.dll, but not the version number. And why the version number remained the NEWEST one when the crypt32.dll file inside the HotFix package is clearly a rollback to .5707 from three years ago.

Long story short - we are still seeing this issue in our environment, and see no advantage to deploying the HotFix at this moment. And we will not disable the security features this affects, either. We have reached out to Microsoft as well and hope for a much better answer than we've gotten so far.

kirk...

Firt of all, if you are having issues with the installation of symantec, due to the CRYPT32.DLL provided by Microsoft, it is not a Symantec issue and should be treated as an OS issue.

To workaround this, you can disable the requirement for the sysstem to check for Root Certificates, get the installation done and stop complaining about a 2 year old hot fix.  There are many ways to remove the Root Certificate checks, by GPo for everyone or by local machine - more work.  Of course, seeing as how this is controlled by GPO, there is also a registry hack that can be performed.  Doing this will indeed indicate 100% if the problem you are experiencing is indeed as most believe it to be.

However, should you encounter the SAME PROBLEM after removing the verification for Root Certificates (crypt32.dll) than likely there is another underlyinh problem as to why the system is looping.

* * * *

When this has happened to me in the past, it was because of a rights issue in the %userprofile% of the user logged into the machine.  (%userprofile%\Local Settings\Application Data\Symantec).

Either Uninstalling Symantec and wiping the folder completely; or fixing the permissions on the            %userprofile% usually corrected this problem.

* * * * * * *

In my environment, my user profiles have been moved to the D: drive. 

By running from a command prompt: "Chkdsk /X d:" which unmounts the D: drive (user profile) the system will always loop into a continuous reinstallation of Symantec, until the scan is done and the drive remounted.  Incorrect permissions to write to this folder can also be linked to the same behavior.

So, can someone please try removing the check for "root certificates" and confirm, as Paul M. has stated that this is the case.

Thank you.

I decided to install the hotfix on the impacted clients. The good news here is that we plan on phasing out our Windows XP boxes soon and upgrading to 12.1 when 12.1 MP1 comes out in the next few days. I didn't see this thread until I posted something similar and found out it was a known issue.