Endpoint Protection

This morning i've got an overwhelming count of messages from our employees. Many of them were receiving messages that Virus definitions are missing and then some of them have carshed (BSOD) repeatedly 2-3 times. Crash dumps point to MS drivers (mup.sys, fltmgr.sys, rdbss.sys). Makes no sense. Maybe some older fault SEP definition can be the cause and as most has updated already they are not experiencing this issue?

What Exact SEP version are you using ?

You will collect the Full Memory Dump files and create a Case symantec Support Team.

QuickStart Guide - Create and Manage Support Cases in SymWISE

http://www.symantec.com/docs/HOWTO31132

How to update a support case and upload diagnostic files with MySupport

http://www.symantec.com/docs/TECH71023

OR

Regional Support Telephone Numbers:

United States: https://support.broadcom.com (407-357-7600 from outside the United States)

Australia: 1300 365510 (+61 2 8220 7111 from outside Australia)

United Kingdom: +44 (0) 870 606 6000

Additional contact numbers: http://www.symantec.com/business/support/contact_techsupp_static.jsp

See below thread

https://www-secure.symantec.com/connect/forums/bsod-due-upgrade-sep-12153375000

 

SEP 12.1.5337.5000

I'm so far not interested in wasting my time with filing a case, if it is a temporary issue. Just want a confirmation from users, if this happened not just to us.

See one of thread raised that issue are resolved next sep release 12.1.6

https://www-secure.symantec.com/connect/forums/ru6-sep-121

We've been using RU5 for quite some time without crashes.

I suggest you can contact symantec support and submit Full Memory Dump for permanent issue.

Haven't seen or heard anything yet. Usually something will show up in the public knowledge base though or even a blog.

I've not heard from users either but running 12.1.5 and with the past few issues this version has not been affected.

Yes, we have had at least 185 machines experience the BSOD since Friday.  We opened tickets with both Symantec and Microsoft and sent memory dumps to both.  We've been running SEP 12.1.5 since November or earlier, and this just started Friday.

We are testing various approaches.  Running CleanWipe and reinstall seems to fix it.  We don't know if rolling back AV or IPS defs can fix it.  Symantec hasn't yet accepted responsibility, although it seems pretty clear from the memory dumps it is SEP.

What type of hardware is everyone running who experiences this?

You say it is clearly SEP, what driver exactly? In my case my debugger points to various MS drivers. Though i kind of blaming SEP also as there were no Windows/Office updates recently. I think it could be some faulty update in the morning, so most users got the latest update during the day. But the ones who left earlier on Friday didn't yet have this newer version, so after booting their PCs something went wrong, until they finally managed to auto-update definitions. Some have crashed and rebooted 2-3 times in 30 minutes.

I think all machines affected by this were Dell E5520 laptops with Windows 7.

Lenovo with windows 7.  We have both laptops and desktops impacted. 

we see different drivers with different memory dumps.  sometimes Mup.sys, which is a windows driver, and sometimes srtsp.sys, which is a Symantec driver.

The thing that pointed us to symantec is that uninstalling symantec appears to resolve it. 

Today we only had one case of a crash (yesterday it was 3 PCs) and one PC still showed that there are no virus definitions (it was low on space in C disk, so maybe that was the cause, that it couldn't auto-update). I have reinstalled SEP on both PCs and waiting for issues. If it doesn't reappear i have nothing to file with support so far.

we have had it come back even after reinstalling SEP on systems.

Has Symantec publicly ack'd this?

Does it appear to be related to defs?

Rolling back IPS defs to 2/25 consistently resolves the problem for us.

We are continuing to work with Symantec and Microsoft to run memory dumps and try to isolate the component.

The number of systems continue to grow.

 

 

Thanks for the tip Mister Paul...any hope for relief is welcomed.

On my SEPM I do not have IPS defs dated 02/25/2015, so I selected 02/24/2015 r12. Hopefully they will be more well behaved (assuming the IPS defs are the culprit).

-Mike

Hello,

we have the same problem with our XenApp 6.5 Farm. Since Friday any servers crash randomly with BSOD (Bugcheck 3B, 50 or 27 - some refer to rdbss.sys). "BlueScreenView" points to ntoskrnl.exe.

The servers has only AV installed. We rolled back AV definitions to 24.02.2014. I hope that helps.

Have anybody new informations about that problem?

 

Regards

Thomas

 

Dunno if it's related, but I'm facing some BSOD issues as reported below.

 

We also have random BSOD since friday, occured on 25% of our workstations so far. Crash dumps point to MS drivers (mup.sys & rdbss.sys) as well. 

I hope people at symantec are taking this very seriously and are working on a fix.. some feedback please ?

For me yesterday was the last crash (only 1 occurance). And this machine didn't had definitions for Proactive Threat Protection and Network Threat Protection. Those were in "Waiting for updates" state. So i have reinstalled SEP on this machine and everything is fine since. Are you using the latest definitions for AV, NTP? Today we have 2015.03.03 R20/R11 and everything seems ok.

Today 7 of 38 Servers are crashed with AV signature 2015.03.03 r20. After Rollback to 2015.02.24 r2 I have had so, only 2 crashes.

Your servers have a minimal or a full installation ? Because none of our servers crashed so far, but they have minimal installation (no SONAR & IPS, no antimalware, no anti intrusion)

Bad pool caller is frequently displayed on the blue screen we see.

We also have a minimal installation on servers and have not had problems there.

We also have a minimal installation on servers and have not had problems there.

Anyone else who has this problem - please notify Symantec support. As of Tuesday they were telling us we were the only ones who had called in with it. Thanks! Mr Paul

I opened, simultaneously with my question at the forum, also a ticket with Symantec.

By the way, we have a minimal installation - only AV.

Hello! I confirm frequent crashes since March 1. All our minidumps contains reference to symefasi.sys (Symantec Extended File Attributes) driver. We have tried to uninstall this driver ("sc delete symefasi") and reboot comuter. All crashes are gone. SEP client does not report any problem after driver deletion. This temporary solution helps us.

I can confirm that we didn't had issues with the servers so far (which have only AV module installed). But our servers are running all the time. I suspect the issue is appearing only after a fresh boot (at least it looks like with workstations).

While we're waiting some info from Symantec, what is the side effect of the deletion of that driver?

Side effects are unknown. Visually SEP client shows green "checkcircle". No error messages. Virus detection, firewall and device management does not suffer.

So, i've just had one crash today. IT was a Lenovo laptop (Win7), which was off since yesterday. I have booted it and updated Office program and then it crashed with symefasi.sys as a cause. I think it already had yesteday's definitions (03.03). So it looks like Symantec is trying to fix 02.27 problem and this is something new (haven't seen symefasi.sys in minidumps so far). Will file a case and hope tomorrow i won't have 200+ crashes..

Definitions udpate causing systems to crash, this is new..

I analyse a crashdump with the following results.

These output is from bugcheck 3B and 7E:

SYSTEM_SERVICE_EXCEPTION (3b)
An exception happened while executing a system service routine.
Arguments:
Arg1: 00000000c0000005, Exception code that caused the bugcheck
Arg2: fffff800018bc445, Address of the instruction which caused the bugcheck
Arg3: fffff88005707a30, Address of the context record for the exception that caused the bugcheck
Arg4: 0000000000000000, zero.

Debugging Details:
...

ADDITIONAL_DEBUG_TEXT:  
You can run '.symfix; .reload' to try to fix the symbol path and load symbols.

FAULTING_MODULE: fffff80001850000 nt

DEBUG_FLR_IMAGE_TIMESTAMP:  537fbee6

EXCEPTION_CODE: (NTSTATUS) 0xc0000005 - Die Anweisung in 0x%08lx verweist auf Speicher 0x%08lx. Der Vorgang %s konnte nicht im Speicher durchgef hrt werden.

FAULTING_IP:
nt!ExpInterlockedPopEntrySList+25
fffff800`018bc445 498b08          mov     rcx,qword ptr [r8]

CONTEXT:  fffff88005707a30 -- (.cxr 0xfffff88005707a30;r)
rax=0000000012820001 rbx=0000000000000000 rcx=fffff8800157e500
rdx=d5d8d8ffcfdedf01 rsi=000000000010d800 rdi=0000000000000000
rip=fffff800018bc445 rsp=fffff88005708410 rbp=fffff8a00aecf2f0
 r8=d5d8d8ffcfdedf00  r9=fffff880009c0180 r10=fffff8800157e500
r11=fffffa800d9aabb0 r12=fffff88001554928 r13=fffff8a00896ee50
r14=0000000000000007 r15=fffff88005708590
iopl=0         nv up ei ng nz na po nc
cs=0010  ss=0018  ds=002b  es=002b  fs=0053  gs=002b             efl=00010286
nt!ExpInterlockedPopEntrySList+0x25:
fffff800`018bc445 498b08          mov     rcx,qword ptr [r8] ds:002b:d5d8d8ff`cfdedf00=????????????????
Last set context:
rax=0000000012820001 rbx=0000000000000000 rcx=fffff8800157e500
rdx=d5d8d8ffcfdedf01 rsi=000000000010d800 rdi=0000000000000000
rip=fffff800018bc445 rsp=fffff88005708410 rbp=fffff8a00aecf2f0
 r8=d5d8d8ffcfdedf00  r9=fffff880009c0180 r10=fffff8800157e500
r11=fffffa800d9aabb0 r12=fffff88001554928 r13=fffff8a00896ee50
r14=0000000000000007 r15=fffff88005708590
iopl=0         nv up ei ng nz na po nc
cs=0010  ss=0018  ds=002b  es=002b  fs=0053  gs=002b             efl=00010286
nt!ExpInterlockedPopEntrySList+0x25:
fffff800`018bc445 498b08          mov     rcx,qword ptr [r8] ds:002b:d5d8d8ff`cfdedf00=????????????????
Resetting default scope

DEFAULT_BUCKET_ID:  WIN7_DRIVER_FAULT

BUGCHECK_STR:  0x3B

CURRENT_IRQL:  0

ANALYSIS_VERSION: 6.3.9600.17298 (debuggers(dbg).141024-1500) amd64fre

LAST_CONTROL_TRANSFER:  from fffff880015b6fc8 to fffff800018bc445

STACK_TEXT:  
fffff880`05708410 fffff880`015b6fc8 : fffffa80`0d9aabb0 fffff880`05708480 fffffa80`00000005 00000000`00000000 : nt!ExpInterlockedPopEntrySList+0x25
fffff880`05708420 fffff880`015b6f82 : 00000000`00000000 fffff880`01578798 00000000`00000000 00000000`00000000 : symefasi+0x16efc8
fffff880`05708470 fffff880`015b5ab9 : 00000000`00000000 fffff8a0`0896eea0 fffffa80`0cf4a1b0 fffff800`018d53cc : symefasi+0x16ef82
fffff880`057084c0 fffff880`015b5fd6 : 00000000`00000106 fffff880`012c750c 00000000`00000106 fffff8a0`0896eec8 : symefasi+0x16dab9
fffff880`05708560 fffff880`015a8123 : fffff8a0`0896ed40 fffff8a0`0896ed40 fffffa80`0eeb5d90 fffffa80`0a309070 : symefasi+0x16dfd6
fffff880`05708610 fffff880`015adb1b : fffff8a0`0896ed40 00000000`00000000 00000000`00000000 fffff8a0`0896ee48 : symefasi+0x160123
fffff880`05708650 fffff880`015ad39e : 00000000`00000000 fffff880`01578798 00000000`00000000 00000000`00000001 : symefasi+0x165b1b
fffff880`057086d0 fffff880`015aa095 : fffff8a0`0000004b 00000000`00000000 fffff8a0`15c409f0 00000000`000007ff : symefasi+0x16539e
fffff880`05708770 fffff880`0158cc9f : fffffa80`0acdb040 fffffa80`0d917988 00000000`00000000 fffff880`05708b60 : symefasi+0x162095
fffff880`05708890 fffff800`01be2fe7 : fffffa80`0b953b00 fffff880`05708b60 fffffa80`0b953b00 fffffa80`0d917870 : symefasi+0x144c9f
fffff880`057088d0 fffff800`01be3846 : 00000000`054be788 00000000`00001c04 00000000`00000001 00000000`05b9ed1c : nt!NtMapViewOfSection+0x15f7
fffff880`05708a00 fffff800`018c37d3 : 00000000`00000000 fffff880`05708b60 00000000`00000000 fffff800`01bb122b : nt!NtDeviceIoControlFile+0x56
fffff880`05708a70 00000000`74ac2e09 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!KeSynchronizeExecution+0x3a23
00000000`054bf038 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : 0x74ac2e09

FOLLOWUP_IP:
symefasi+16efc8
fffff880`015b6fc8 4885c0          test    rax,rax

SYMBOL_STACK_INDEX:  1

SYMBOL_NAME:  symefasi+16efc8

FOLLOWUP_NAME:  MachineOwner

MODULE_NAME: symefasi

IMAGE_NAME:  symefasi.sys

IMAGE_VERSION:  5.0.1.29

STACK_COMMAND:  .cxr 0xfffff88005707a30 ; kb

BUCKET_ID:  WRONG_SYMBOLS

FAILURE_BUCKET_ID:  WRONG_SYMBOLS

ANALYSIS_SOURCE:  KM

FAILURE_ID_HASH_STRING:  km:wrong_symbols

FAILURE_ID_HASH:  {70b057e8-2462-896f-28e7-ac72d4d365f8}

Followup: MachineOwner
---------

 

 

We tried 12.1.5337.5000 on Windows 7 and had issues.  R5 seems to work fine on  Windows 8/8.1 and 2012/R2.  Our Windows 7 clients still seem to be stable with R3.  Same goes for our Windows 2008/R2 servers, R3 is stable so we are sticking with it till the next release. 

We started with experimenting with R5 on Windows 7 clients in January and Febuary, but were unable to resolve the BSOD issue, even after opening a Symantec support case.  Also, our laptop and desktops are all Lenovo Thinkpad/ThinkCentre.  Based on our testing, I can only summarize:

R3 good for Win7/2008/R2

R5 good for Win8/8.1/2012/R2

Well, unfortunately my company has signed up a 3 year contract with Symantec for AV protection... I hate Symantec, nothing they do works fine the first time... exclusions still get scanned, server reboots, server crashes... but this is not relevant atm... our Windows 7 computers are also crashing, randomly BSOD multiple times a day. The only solution we have found was to uninstall Symantec 12.1.5 with CleanWype. All issues we are experiencing are the same ones mister paul and wroot have described above. Thanks for creating the post guys. 

I have tried multiple times to communicate with Symantec to open a ticket online and this is what i get:

https://my.symantec.com/static/sitemaintenance.html

"MySymantec Maintenance

We apologize for the inconvenience, but the site is currently unavailable while we are making updates. While the system is down, you can view articles and forum posts in our user community, Symantec Connect. You do not need to log in to browse the forums and read the posts and articles."

Then i try the phone to give them a call, get on hold for a long time and i get disconnected... what a joke this company is. 

All computers that are crashing are showing this error:

IMAGE_NAME:  symefasi.sys

ERROR_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%08lx referenced memory at 0x%08lx. The memory could not be %s...

Well, i will keep trying to contact them... and making sure i don't renew the AV when the time comes. 

http://www.customerservicescoreboard.com/Symantec

 

we tested removing that driver and it broke symantec for us.  So probably a feature we're using that you are not? Perhaps IPS?  - Paul

Symantec tells us that IPS defs released today, 04-March-2015 rev. 12 (20150301.12), should resolve our issue. Hopefully it will resolve yours too. Please test and report back.  We won't know our results 'till tomorrow.

I am going to post this in all the BSOD threads, so apologies for duplication...

Paul

Does anyone know whether the IPS signatures are required if only AV is installed? The IPS signature 04032015.12 I have installed . Today (sincs 3 Hours), I have not had any crashes.

 

Thomas

Since my last post I had no more crashes. From Symantec I got a confirmation that the error has been corrected with the new IPS signatures. Even if only AV is installed, the IPS signatures are used .

Unfortunately, my BSOD started on Friday, Feb. 27,2015. I work for the largest school system in NJ and is already affecting well over a hundred workstations. Unfortunately, Symantec is refusing to accept blame, too much liabilty.

The problem sometimes gets temporarly resolved by rebooting multiple times but always reappears. I upgraded tp SEP 12.1.5 about 2 weeks ago and then the nightmare started on this past Friday. You can open the SEP GUI and watch the virus definitions update and then on the next reboot it Blue Screens.

I have been administering Symantec AV for 12 years and this always happens once in a while. To resolve my issue I have users or tech support over the phone, have them boot into Safe Mode with Networking using their domain accounts. Connecting to a mapped drive through a logon script I have them run Clean Wipe which solves the problem on the next reboot. Then they log in normally and either tech support or the user now reinstalls SEP 12.1.4 until Symantec fixes the problem. At this point BSOD is gone and you can remote access the PC.

Symantec, for security reasons, I haven't tried recently, but does not have a silent function for CleanWipe.

I would do this via a GPO if it would work in Safe Mode with Networking but I will try uninstalling SEP 12.1.5 during the logon script if I can get the PC to run the MSI. I will get back with other ideas when I find something better to use.

And Yes, the Symantec tech wants a memory dump file but can't get the FTP site to except my 4GB file.

I have not been in work yet since yesterday due to the snow storm (NJ). Yesterday PCs were still Blue Screening after updating.

Nick

 

Yes.  The IPS signature is still required even with just AV.  You should be fine with the 04032015.012.

It might be too soon to tell but ever since we updated SEPM we have had no more BSOD. 

Hi all,

This issue looks like resolved through a CIDS-SONAR definitions rollback. This rollback was triggered on March 4th. If your Live Update is running properly you certainly will have this issue addressed. Also there is a workarounf for the problem:

Rollback the defs to a date before Feb, 26th.

How to Backdate Virus Definitions in Symantec Endpoint Protection Manager
Article URL http://www.symantec.com/docs/TECH102935 

Be sure that the cause of the BSOD is the SymEFA component. A Full Dump is capable of collecting this data.

Fell free to open a ticket within our support team for further assistance.

Thanks for the update Lucas...what would be curious to know is what the CIDS-SONAR defs were conflicting with?

Out of my 20K+ machines, less than 50 (thank goodness) were *affected.

Affected = BSOD and/or Slow Boot and/or Black Screen of Death (freeze on a black screen while booting)

Rolling back to RU4 MP1 resolved the issue in most cases...

For us, this all appeared to start after we pushed the last MS "patch Tuesday" updates...so we had a lot of finger pointing between our team and the patch pushers.

Comments??

Thanks again for keeping us informed,

-Mike

 

In work yesterday, I went back to trying to resolve a Blue Screen on a computer I left that way on purpose. I eventually, removed the SEP12.1.5 client with Cleanwipe in "Safe Mode with Networking" and reinstalled the client. And yes the only difference I see now is after the "Waiting for updates.." on the Proactive Threat Protection and Network Threat Protection complete their updating. I now see that instead of late February date on Network Threat Protection, it now displays March 4, 2015 and no longer Blue Screens.Since none of my servers have that component installed "Basic Threat Protection for servers", I never had a Blue Screen on servers. At some point, I will reonstall 12.1.5. Thanks, By the way Symantec tech support still couldn"t resolve my saving the 4GB Memory dump file on their FTP server because of a permission issue.

Long story short. I went to work yesterday and looked again at one of the Blue Screen computers. Uninstalled 12.1.5 via "Safe Mode with networking" and reinstalled 12.1.5. This time after the "Waiting for updates.." on Proactive Threat Protection and Network Threat Protection, the date for Network Threat Protection was now march 4, 2015 and no longer the end of February. This appears to be what stopped the Blue Screen for me.

Nick

I have closed the case. There were no crashes since 03.04. Looks like CIDS defs rollback did the thing.

That issue resolved on sepm 12.1.6

Blue screen error after upgrading SEP

Fix ID: 3649959

Symptom: A blue screen occurs with BugCheck 3b on Symantec Endpoint Protection client, which points to a SymEFA component.

Solution: Fixed a performance issue which caused the blue screen.

https://support.symantec.com/en_US/article.TECH230558.html