Endpoint Protection

Hi All,

it took us only two weeks to find out the root cause of failure for one of our standard 32-bit application running in our servers (having 64-bit Windows Server 2008 R2 SP1 in them).
All the windows installations are licensed (no hacked software and threats), we have an "enterprise" anti-virus system installed, which is Symantec Endpoint Protection (build 12.1.2100.2093).

After Windows updates which came in September we got one of our common-used application failed to start with error code 0xc0000005 (the application was unable to start correctly). Maybe some internal updates came also for Symantec Endpoint Protection itself (our system administrators looks like too busy to provide all the information, so we had to investigate the issue by ourselves).

Anyhow after restarting the servers on 1st/2nd of October we got this error started and we spend much time on investigations.

Appsight showed us that the appication is failing when accessing C:\Windows\Microsoft.NET\Framework\v2.0.50727\dw20.exe

First we were trying to remove updates, then we started to check all the support-level applications, including Anti-virus and the only thing which helped is Uninstalling the Symantec Endpoint Protection (just switching it off was not enough).

 

We will try to install new version and so on, but I just decided to share this information for those who maybe searching for an answer for similar problem.
I will try to update this topic if we have more information.


BR,

Vlad

Did you have Proactive Threat Protection component installed? Was there anything in the log for it? Anything in the AV risk log?

Did you try creating an exclusion?

Hello,

Let us know if the issue is resolved after installing the Latest version of SEP 12.1.RU3?

Did you check the Symantec Endpoint Protection Risk Logs?

Awaiting your reply.

As I understood from another topics Risk Logs you mentioned are in C:\Documents And Settings\All Users\... I have no permissions to this folder in our application servers. I will ask our infrastructure to get the logs, but I am not sure if they can find time for it.
In Event Logs there is no warnings or errors related to Symantec - only info about updates coming and installing.

And, as I said before, even when AV was switched off the issue was still there untill we uninstalled it, so maybe this is some kind of this particular application issue caused by the way it is loading shared libraries (so maybe it is not common issue).

I will let you know after installing new version.

I'm just more curious to see if there was a conflict as opposed to the program actually being flagged as "malware"

Was wondering what adding and exception would do.

Exception will not do anything (and it is not doing) because even when AV is off (so no checking is done at all) the issue is still there.

I can see now that Risk log could be also reviewed from GUI, so there's nothing there.

I would suggest opening a support case than. Keep the thread updated as time permits.

And I have found the root cause.
After profilling the application with Dependency Walker when it works normally and when it fails I found that it takes one of symantec libraries instead of standard windows dll when fails:

Loaded "c:\programdata\symantec\symantec endpoint protection\12.1.2100.2093.105\data\definitions\bashdefs\20130924.011\UMENGX86.DLL" at address 0x71610000.  Successfully hooked module.

When it works normally it is not taking this library at all (so that's why it works after uninstalling Symantec Enpoint Protection completelly).

then diff gives me several instances of symantec library instead of calling internal system functions:

DllMain(0x71610000, DLL_PROCESS_ATTACH, 0x00000000) in "c:\programdata\symantec\symantec endpoint protection\12.1.2100.2093.105\data\definitions\bashdefs\20130924.011\UMENGX86.DLL" called.
GetProcAddress(0x768C0000 [c:\windows\syswow64\KERNEL32.DLL], "FlsAlloc") called from "c:\programdata\symantec\symantec endpoint protection\12.1.2100.2093.105\data\definitions\bashdefs\20130924.011\UMENGX86.DLL" at address 0x716194D6 and returned 0x768D4F13.
GetProcAddress(0x768C0000 [c:\windows\syswow64\KERNEL32.DLL], "FlsGetValue") called from "c:\programdata\symantec\symantec endpoint protection\12.1.2100.2093.105\data\definitions\bashdefs\20130924.011\UMENGX86.DLL" at address 0x716194E3 and returned 0x768D1252.
GetProcAddress(0x768C0000 [c:\windows\syswow64\KERNEL32.DLL], "FlsSetValue") called from "c:\programdata\symantec\symantec endpoint protection\12.1.2100.2093.105\data\definitions\bashdefs\20130924.011\UMENGX86.DLL" at address 0x716194F0 and returned 0x768D41F0.
GetProcAddress(0x768C0000 [c:\windows\syswow64\KERNEL32.DLL], "FlsFree") called from "c:\programdata\symantec\symantec endpoint protection\12.1.2100.2093.105\data\definitions\bashdefs\20130924.011\UMENGX86.DLL" at address 0x716194FD and returned 0x768D357F.
DllMain(0x71610000, DLL_PROCESS_ATTACH, 0x00000000) in "c:\programdata\symantec\symantec endpoint protection\12.1.2100.2093.105\data\definitions\bashdefs\20130924.011\UMENGX86.DLL" returned 1 (0x1).


for "normal process":

DllMain(0x71D50000, DLL_PROCESS_ATTACH, 0x00000000) in "c:\windows\syswow64\VERSION.DLL" called.
DllMain(0x71D50000, DLL_PROCESS_ATTACH, 0x00000000) in "c:\windows\syswow64\VERSION.DLL" returned 1 (0x1).
DllMain(0x747B0000, DLL_PROCESS_ATTACH, 0x00000000) in "c:\windows\winsxs\x86_microsoft.vc80.crt_1fc8b3b9a1e18e3b_8.0.50727.6195_none_d09154e044272b9a\MSVCR80.DLL" called.
GetProcAddress(0x76AC0000 [c:\windows\syswow64\KERNEL32.DLL], "FlsAlloc") called from "c:\windows\winsxs\x86_microsoft.vc80.crt_1fc8b3b9a1e18e3b_8.0.50727.6195_none_d09154e044272b9a\MSVCR80.DLL" at address 0x747B3001 and returned 0x76AD4F13.
GetProcAddress(0x76AC0000 [c:\windows\syswow64\KERNEL32.DLL], "FlsGetValue") called from "c:\windows\winsxs\x86_microsoft.vc80.crt_1fc8b3b9a1e18e3b_8.0.50727.6195_none_d09154e044272b9a\MSVCR80.DLL" at address 0x747B300E and returned 0x76AD1252.
GetProcAddress(0x76AC0000 [c:\windows\syswow64\KERNEL32.DLL], "FlsSetValue") called from "c:\windows\winsxs\x86_microsoft.vc80.crt_1fc8b3b9a1e18e3b_8.0.50727.6195_none_d09154e044272b9a\MSVCR80.DLL" at address 0x747B301B and returned 0x76AD41F0.


I want to highlight that on one of our application servers is still working fine, so the logs are from there (not all the windows updates weer applied there). So looks like windows updates together with latest symantec updates, which came on 24-Sep-2013 (library is in that folder) are making som strange things.
 

Normally when I see a application fail with the 0xc0000005 it is because of application and device control (ADC). Have you tried adding a ADC exception within the SEPM for this program? You could also isolate to verify it is this portion of the product by uninstalling just ADC.

Mick2009 did a great article on this, check it and see if you see a pattern

https://www-secure.symantec.com/connect/articles/crreating-application-control-exclusions-symantec-endpoint-protection-121

Perhaps an ADC exception is needed (assuming you have the ADC component installed)

As I understand our infrastructure guys are contacting you directly, but just to add some info in this thread - even reinstalling Symantec Endpoint Protection without any additional components (removing SONAR and ADC) we still have the same issue.

In dependency walker I can still see that
Loaded "c:\programdata\symantec\symantec endpoint protection\12.1.2100.2093.105\data\definitions\bashdefs\20130924.011\UMENGX86.DLL" at address 0x71BC0000.  Successfully hooked module.

after SYSFER.
And looks like this library makes the issue happen as the only differences to successfull running are calls for functions from this library.

And by the way - this library was installed in Symantec update on 01-Oct-2013.
Next day (after the servers were restarted) the issue has appeared for the first time.