Endpoint Protection

See http://www.symantec.com/business/support/index?page=content&id=TECH192811 for further details on this issue.

After a full evaluation and root cause analysis of the issue, we have determined that the issue was limited to machines running a combination of Windows XP, the latest version of the SONAR technology, the July 11^th rev11 SONAR signature set, and certain third party software.  Only customers running this combination of technologies and who downloaded the July 11^th rev11 SONAR signature set via LiveUpdate between 6:25PM PT and 2:51AM PT on July 12^th were affected.

The root cause of the issue was an incompatibility due to a three way interaction between some third party software that implements a file system driver using kernel stack based file objects – typical of encryption drivers, the SONAR signature and the Windows XP Cache manager.  The SONAR signature update caused new file operations that create the conflict and led to the system crash.

Symantec understands the consequences of this type of issue to our customers and goes to great length to prevent them.  The quality assurance process for SONAR signatures is extensive.  The process includes:

• Peer review and vetting of all signatures
• True positive testing
• False positive testing
• Functional testing of all signature content
• Compatibility testing

The compatibility testing part of the quality assurance process for SONAR signatures missed catching this compatibility issue.  It is this part of our process that we will be improving to avoid future issues.  We are currently restructuring our testing process to improve compatibility testing and will not be releasing new SONAR signatures until this new process is in place. 

Regards 

Orla Cox
Symantec Security Response

On July 11th, 2012 Symantec Security Response started receiving reports of customers experiencing blue screens after applying the July 11th revision 18 definitions. Machines may continue to blue screen after they reboot.This problem only appears to occur on Windows XP machines. The root cause of the problem is unknown at this time.

Hi

Do you run SEP 12.1 and do you have a solution to this.

We are experiencing htis same issue on some XP computers, same symptoms as above.

Thanks

We are also experiencing this issue but after many investigations it doesn't seems to be related to virus definition but to SONAR definitions. Disabling Sonar the issue disappears

This morning we got problem with many XP computers.

They came in to a loop

I did this with some computer and after that they was ok

Booted up in fail safe mode, chose yesterdays date 

unplug network and rebooted

Uninstalled NETWORK THREAT PROTECTION

Plug in to network and restarted

OK

Maybe a drastic work around

12.1.1101.401

Hi,

Current workarounds are:

-  Roll back to definitions released before the 11^th 

Follow the steps below to roll back virus definitions in Symantec Endpoint Protection Manager:

1. Click Policies
2. Select View Policies
3. Click LiveUpdate.
4. Double-click your current LiveUpdate Content Policy Under the "LiveUpdate Content" tab. The LiveUpdate Content Policy Overview dialog box appears.
5. From the "LiveUpdate Content" section, click Security Definitions.
6. Enable the Select a revision option located in the "AntiVirus and AntiSpyware definitions" section,
7. Click the Edit button. The Select Revision - Antivirus and AntiSpyware definitions dialog box appears.
8. Expand the drop-down list and browse to the appropriate (32-bit or 64-bit) definition set.
9. Click the desired rollback definition date.
10. Click OK.
11. Click OK to close the "Security Definitions" dialog box and return to the "Policies" tab.

or

-  Temporarily disable the BASH driver on affected machine (BHDrvx86)

sc config bhdrvx86 start= disabled (This command disables the BASH driver.)

sc config bhdrvx86 start= system (This command enables the BASH driver.)

We too had the same issue.

We have roll backed the defintion to 11th July 2012 R003 for Antivirus and Antispyware from SEPM console as per Symantec suggested workaround. For systems getting blue memory dump, we restarted with last good known configuration and logged in with local admin account and made the registry entry changes as mentioned below..

Navigate to HKLM/System/CurrentControlSet/Services

Select the driver BHDrvx86 and change the start value to 4

Reboot the system the system .

Pls check whether the problematic systems are Fat 32 or NTFS. We have addressed the same for FAT32 systems till now.

Regards,

SAM

Hello,

In this case, as suggested above, you could also try Rollback of Virus definitions -

How to Backdate Virus Definitions in Symantec Endpoint Protection Manager

http://www.symantec.com/docs/TECH102935

Symantec Endpoint Protection 12.1: How to roll back the BASH definitions to a known good version

http://www.symantec.com/docs/HOWTO53366

To Disable Bash Drivers via a command prompt (with local administrator privileges):

• 32-bit Windows: "sc config bhdrvx86 start= disabled" (without the quotes)
• 64-bit Windows: "sc config bhdrvx64 start= disabled" (without the quotes)

Restart the system once you have successfully completed the command.

To re-enable BASH, please repeat the same command used to disable it, but replace "disabled" with "system". The machine will need to be restarted for the change to take effect.

To confirm if bash is running, please use either "sc query bhdrvx86" or "sc query bhdrvx64".

NOTE: Incase of these steps would does not assist, please create a case with Symantec Technical Support.

We have had this issue from about 11:20am WST (+8)
It seems to have only appeared on computers that are a Dell Optiplex 780.

Have applied the fix as mentioned above and it works.

Will this registry setting revert to default when new definitions are installed?

I just had the issue on a Dell Latitude D630 laptop w/ XP SP3.

Our company has more than 40 Win XP,Restart the computer after the bluescreen....

MGD...

In safemode, I uninstall SEP Client with CleanWipe-v12.1.1000.157 , reboot the system,it`s OK....

We had 2 XP machines and now a Windows 7 machine get the blue screen. 

I hope and really really wanted to believe that Windows 7 BSOD is not related about this issue.

We have Tech Support on the phone. They are saying the problem is with SONAR.

The blue screens stopped once we removed Symantec from the pc

Mark,

Please inform us with the news.

I was able to get my clients reverted back to Rev 3 this morning.  I've only had one computer have the BSOD issue.

Hello,

Symantec Security Response has published BASH signatures (PTP Definition should now read as: 11^th July, 2012 Rev: 012) so you should see PTP definitions date as 11th July, 2012 Rev: 012 (Earlier it was 11^th July, 2012 Rev: 011). 

Check this: http://www.symantec.com/security_response/definitions.jsp

Hope that helps!!

Do your clients update from the manager or from GUPs?

Does anyone know if GUPs are capable of rolling a client back?

I'm testing SONAR definitions 07/11/2012 r12 and are working fine. The issue seems fixed.

...is that I wrote at the beginning of the thread.

Both.  The GUPS reverted back to R3 and eventually the clients the GUPS services did too.  It looks like Symantec resolved this with a new definition update, but I'm waiting till the weekend before updating everything.

So is PTP the problem? The lack of information on this issue is amazing, although not surprising.

The above info is the correct fix and we have pushed out R12 to as many PCs as we can but over 30% of our PC environment still will not boot. We got several up using Last Known Good... but we're still working on the rest.

Great news Fabio and Mark,

I hope you'll fix them less effort.

Hello,

SONAR is part of Proactive Threat Protection on your SEP 12.1 client computers. 

SONAR is part of Proactive Threat Protection on your client computers. You manage SONAR settings as part of a Virus and Spyware Protection policy.

Hope that helps!!

I understand what SONAR and PTP are. We are trying to get confirmation that the definition files for those components are what is causing this.

We believe the problem may be related to some Proactive Threat Protection (PTP) updates which were released yesterday but haven't fully confirmed root cause. As a workaround we have removed those updates and posted new PTP definitions which no longer include those updates. These are dated July 11 2012, revision 12. Our recommendation is that affected users run LiveUpdate to get these PTP definitions.

We're continuing to treat this as a priority and confirm root cause.

Orla

Symantec Security Response

Hello Orla,

From your last post, as i understand you're talking about just PTP definitions. So, reverting back from 07/11/2012 R18 to R3(for example) will not solve issue ? 

And also you're not talking about version of SEP and you just mentioned as "PTP", you mean also SONAR includded or just SONAR. Both SEP 11 and 12.1 are affected ?

32-Bit or 64-Bit? We've only had issues with 32-bit machines so far, that I'm aware of.

Why does it affect WinXP only ?

Why it doesn't affect SEP11 ?

What is in the PTP updates released yesterday ?

I'm having the same issues as described.  For me only fixable by booting all affected WXP PC's (150+ on several locations) into safe mode and doing sc  config  bhdrvx86  start= disabled

But now I've got two Windows 2003 servers (32b)  with the same issue. Disabled bhdrvx86 /PTP as well, reboot, servers are working again.

So as far as i can see, Windows XP and Windows server 2003 are affected. I haven't got any problems with W7 or W2K8

Though I do appreciate the complexity of SEP, crippling half of my PC's is really not acceptable. This has cost us (and others as well obviously) massive amounts of time and money on IT support and employees not being able to work.

Symantec, will you compensate us (being your clients) in any way?

If it is an issue with PTP Why didn't it affect SEP11 ?

 

SONAR is used in SEP 12.1, SEP 11.x uses Truscan.

So which one is the bad revision?  In my Content Revison's Area I can select 7/11/2012 r18... is this the good one or the bad one?

We were hit with this today to, to be honest its a bit of a farce really...

To fix it, we opened device manager and disabled "bhdrvx86" and rebooted. (You will need to turn on "Show hidden devices to see it). It was conflicting with this driver and causing the crash.

Once this was done they all worked again.

This whole episode is a joke, had the issue been a conflict with a random device driver then I could maybe slightly more sympathetic, BUT for it to conflict with its OWN symantec related drivers and cause this issue is a total farce. Who tested it before release? Was it even tested? I hope the person responsible is in the whole world of crap that we were in this morning, maybe that person/s should be invoiced for all the lost time/work that we had today because quite frankly its unacceptable.

Even phoning Symantec support this morning was the start of the hell we went through, first line support was more interested in asking ridiculous questions instead of attempting to do something about it (Where they even aware of it?). She then said that someone will contact us within 24 hours as its not classed as a major incident? Really? So your software cripples our network and its not a major incident? After arguing the toss we got it down as a critical incident and where passed through to a technical support representative who said "Oh yes we are aware of an issue" The support is a joke, the quality control is a joke and the software is not much better. Yes I know these things happen, but any half decent quality control/testing process would surely of highlighted the issue?

For anyone still having the issue, the above solution will work, it just took us a couple of hours woth of lost time to resolve it...Lets hope tomorrow goes slightly easier!

Hi;

Are there any exact solution regarding this issue.

While a root cause analysis is still underway, the following article has been published to provide customers with a workaround to recover from the issue.

SEP 12.1 Win XP Users Experiencing Blue Screen when running Proactive Threat Protection Definitions July 11th 2012 rev 11.

http://www.symantec.com/docs/TECH192811

That is the one causing issues according to the original post but the link below says r.11 is the problem child.

If you update the PTP definitions to July 11th 2012 rev 12 it seems to have resolved the issue

Good morning.

We resolved the problem disabling de SONAR in the registry, but, how is the following steps?

Did Symantec comunicate the solution formally?

Thanks a lot

Knowledge base article here:

http://www.symantec.com/business/support/index?page=content&id=TECH192811

I also had many computers down yesterday because of Symantec.

I replied to my support ticket with... " How is Symantec going to compensate customers for the hours of lost worker production and the time and effort taken by IT staffs to rectify this huge error by Symantec?  Does Symantec even test updates before they push them out to customers?  I need a detailed explanation on why this happened and how Symantec will keep it from happening again.  I want someone to call me with the explanation.  I expect a call within an hour."

I got a call about 20 minutes later from a Support Mgr they are working on a compensation package for me.

I encourage everyone to ask to be compensated for the time and effort it took all of us the fix Symantec's software.

Symantec Technical Support is aware of the July 11 PTP definition issue related to SEP 12.1 on Windows XP machines. 

As noted earlier on this forum, further technical information as well as remediation steps can be found in the Knowledge Base article at:

http://www.symantec.com/business/support/index?page=content&id=TECH192811.

As more information becomes available, we will be adding it to this existing KB document. 

If you need further assistance with this issue or have additional technical quesitons, please contact Symantec Technical Support.  Contact information can be found at: http://www.symantec.com/support/contact_techsupp_static.jsp

Hello megamanVI,

No. This Registry key will need to be manually changed to its original value of "1" (no quotes).

Regards,

James

I would like to express my concerns regarding the Norton update. This is because the update crashed my sister's Windows 7 PC as well. Hence, not only Windows XP PCs are affected but also Windows 7 too. Booting the PC to safe mode did not do the trick.

Oh really? My sister's PC restarted on its own to install Norton updates and it was working fine prior to updating. Afterwards her Windows 7 PC crashed. Hence the problem is affecting other operating systems too. How do I know? Her PC was working perfectly up into around now where her computer crashed not once but twice following the Norton update. A hard reboot on the system only resolved the issue for 30 minutes then it crashed 31 minutes later while she was reading the news. 

This problem is affecting my sister's Windows 7 laptop very badly. After this particular update, her laptop griefed twice. I hope that Symantec will investigate problems with Windows 7 PCs because considering Windows 7 is a new OS problems there are being overlooked. This is because employess would be carried away trying the resolve problems in the XP OS that they are likely to overlook flaws with other operating systerms such as Windows Vista and Windows 7. As a result Norton must anaylze if this update is affecting other operating systems. 

I feel sorry for you, my sister's Windows 7 PC crashed twice on her and I am not willing to follow techincal support advice. 

Get right on the root case as soon as feasible. This problem is affecting other operating systems besides Windows XP. 

I would like to also to mention that this issue is affecting Windows 7 PCs. My sister's Windows 7 PC worked perfectly fine until her computer restarted by itself and the update was installed on her system. As soon the update was finished the blue screen showed up. My laptop with the same operating system, however was left unscathed because I have not been updating my laptop for the past five weeks A quick resolve shall be well appreciated.

I have published a new post and blog with more details on our research and root cause analysis of the incident. https://www-secure.symantec.com/connect/forums/summary-july-11-2012-symantec-endpoint-protection-blue-screen-incident

Hello Himiyuki,

This issue only affects Windows XP systems.  What you are seeing on the Windows 7 system is likely related to something else.  Booting to safe mode will disable the SEP drivers so if it is still blue screening when booting to safe mode it is not related to this issue.  I recommend calling support to get help with the Windows 7 system.

Thanks,

Hello Himiyuki,

As Elisha noted, this issue is specific to the Windows XP environment.  If your sister's machine crashed, unfortunately, it was caused by something else.

If you are still experiencing problems, you can contact Norton technical support at http://us.norton.com/support/.  Additionally, there are forums for Norton issues at: http://community.norton.com/.

Please feel free to reach out to either the community or support for assistnace.

Hi Himiyuki,

I work for Symantec in Norton Support and wanted to make sure that you received help with your sister's computer problem. Blue screens are a severe issue and we want to make sure you get the help you need.

There are many causes of a blue screen and since our engineering team has determined the root cause of this particular blue screen happens only on Windows XP, I am pretty sure that it is not the cause of the problem on your sister's computer. However, one way to definately rule this out is to follow the steps in our remediation document www.norton.com/vd0711 . If Windows will not start after followig the steps, then this SONAR definition issue is not what caused the problem.

Having said that, we also know how frustrating a blue screen can be and would like to help you resolve the problem if it is caused by our Norton product or point you to the right support option if it is not.

Have you already contacted Norton Support about this?

Let me know and also indicate which country you are in and I can provide you with the different contact options.

You can also get quick help for any Norton product issue on our Norton product forums

I look forward to your reply and the chance to help.

RCave

Richard Cave

Supervisor, Norton Product Support Management

Hello Everyone,

Following article is updated on 2012-07-14 with detailed information about the blue screen issue.

SEP 12.1 Win XP Users Experiencing Blue Screen when running Proactive Threat Protection Definitions July 11th 2012 rev 11.

http://www.symantec.com/docs/TECH192811

Please check the same to learn more about it.

Both of those link refer to "some third-party software".  Is there any location that states which third party software was the contributing factor in the ticket(s) they investigated?

The root cause of the issue was an incompatibility due to a three-way interaction between some third-party software that implements a file system driver using kernel stack based file objects – typical of encryption drivers, the SONAR signature and the Windows XP Cache manager.  The SONAR signature update caused new file operations that create the conflict and led to the system crash. 

On the blog post regarding this issue, you've mentioned a third-party software which implements a system driver using encryption that in combination with SEP12.1 and WinXP generates the BSOD.

I want to know what are these software's ?

Any examples ?

My company does not use any disk encryption software. Is Microsoft BitLocker installed by default with Windows XP? And if it is installed with XP how could that be considered "third party"?   The only software installed on our computers is; Microsoft Office, IBM AS400 client software, VNC remote control and SEP.

The blog post mentioned "typical of encryption drivers, the SONAR signature and the Windows XP Cache manager."   The encryption drivers described are for products implementing "whole disk encryption" which manage encryption of all the data on the hard disk.  Examples of products which implement this type of functionality include Novell ZenWorks, PGP WDE and Sophos LanCrypt.  Several different WDE product drivers are correlated to the incident, and investigation is continuing on the root cause to understand the specific interaction.

Peter, I noticed your post about this issue was changed.  Microsoft BitLocker was removed from your post as a possible third party program that causes this issue. My company does not use any whole disk encryption programs. So again why is Symantec saying this issues root cause was an incompatibility due to a three way interaction between some third party software?  Its time to stop the blame game. Symantec needs to take ownership of this issue. 

It took us most of the morning but we were able to get all our PCs to boot using Last Known Good and apply r12 successfully. No issues since.

Hi Scott,

BitLocker is simply an example of the type of product that is showing up as interacting and was provided for illustrative purposes only, so customers could understand this class of software.   Currently, we have not received customer reports that indicate BitLocker as one of the interacting products for this incident (since BitLocker is not supported on Windows XP).   I removed this product because listing a non-interactive product is admittedly confusing (… my apologies).

You can log a support case, if you have not already done so.  Support can collect information on your specific environment and then Security Response will check out the specific software you are running to better understand your incompatibility mechanism.

In regards to “third party,” this was simply a phrase to indicate an additional software product.  My apologies if it was confusing.  We have clarified this language in an update to that post.

PS: Michael Marfise just updated his overview post to include a list of known interactions and clarify “third party” language.

https://www-secure.symantec.com/connect/forums/summary-july-11-2012-symantec-endpoint-protection-blue-screen-incident

We are having Windows XP machines without PTP we just used only Antivirus for XP machines..whether this SONAR definition is for Only Antivirus?

Please tell me..

Hello Srikanth,

No.  This issue will not happen if you do not have PTP installed or if you are using only the antivirus component.  Also it happened with the content released between 6:25PM PT on July 11th and 2:51AM PT on July 12th.  The latest signatures on LiveUpdate do not have this issue.

Thanks,

Ok Then Thanks I can check and let back

Hello Srikanth,

This issue will not happen if PTP is not installed & if using only AV components.

As you can see SONAR is a sub component of PTP, screenshot is attached for reference.

The problem has been identified as a compatibility issue in SONAR definitions released July 11th at 6:25PM PT. Once the cause of the issue was discovered, the signature was removed from the definition set and an updated definition set was published.  This “rollback” of signatures was done on July 12th at 2:51AM PT. Once the signature was rolled back, no new issues were reported from the field.

The KB document on this issue has been updated to include information on what happened, why it happened and what we are doing to avoid this issue in the future. For more information, review the following document:

SEP 12.1 Win XP Users Experiencing Blue Screen when running Proactive Threat Protection Definitions July 11th 2012 rev 11.

http://www.symantec.com/docs/TECH192811

Check following blog as well.

https://www-secure.symantec.com/connect/forums/summary-july-11-2012-symantec-endpoint-protection-blue-screen-incident

Thats why only i asked..if it happen SONAR is not installed..Thanks for the update

Its a bit presumptious for Symantec to be blaming the issue on disk encryption software aint it?

Our desktops are not encrypted, laptops are but desktops are not, so it cant be an issue with any enryption driver. Maybe they should stop looking for excuses, find out what actually caused it and let us know. Then improve their testing and quality control to ensure it doesnt happen again, because from what I can see, the issue was their software, conflicting with their drivers which caused the issue in the first place.

Takes a big person to admit their mistakes and a very small person to point blame

Hi Andrew,

As you may have seen, our initial investigations pointed us to believe this issue was related to the way in which a new SONAR signature interacted with drivers from some encryption software and thats what we let customers know.  However, further investigation has meant we have now changed our wording appropriately as you can see in the blog posting here: https://www-secure.symantec.com/connect/forums/summary-july-11-2012-symantec-endpoint-protection-blue-screen-incident to quote:

"The root cause of the issue was an incompatibility due to a three-way interaction between software that implements a file system driver using kernel stack-based file objects.  The three-way interaction is between the software that implements a file system driver (using kernel stack-based file objects), the SONAR signature and the Windows XP Cache manager.  The SONAR signature update caused new file operations that create the conflict and led to the system crash. 

We have confirmed examples of this interaction with the following products:

• Novell ZenWorks
• PGP Whole Disk Encryption
• Sophos LanCrypt
• SlySoft Virtual Clone Drive"

So we absolutely know what happened, and are taking the appropriate steps to make sure it doesn't happen again.  There was no blame pointed, we were simply letting our customers know what we believed was causing the problem and giving an example based on the cases we already had.  Now we know more, we have updated our information.

Hi,

Just out of curiosity then, can you tell me how this issue managed to get hrough testing, get through quality control (assuming there is some) and get released to end users to cause the issues it did?

Also, can you tell me why your support seemed to have no idea what was going on, until i went through to the "technical Support" who seemed to know all about it and rather annoyingly said "Oh so your aware of the issue then?" and then went on to explain it.

Surely it would have made sense to have the first line aware of the issue, because the first person i spoke to didnt have a clue, yet it seemed more important for her to take details that were not relevant to the issue at all, thus wasting more time and then saying the job is a certain priority and someone will contact me in 24 hours...It was actually a major issue with most of the site down...so I had to argue with for a higher priority and got put through to someone.

We do understand the consequences of this type of issue to our customers and go to great lengths to prevent them.  The quality assurance process for SONAR signatures is extensive.  The process includes:

• Peer review and vetting of all signatures
• True positive testing
• False positive testing
• Functional testing of all signature content
• Compatibility testing

The compatibility testing part of the quality assurance process for SONAR signatures missed catching this compatibility issue.  It is this part of our process that we will be improving to avoid future issues.  We are currently restructuring our testing process to improve compatibility testing and will not be releasing new SONAR signatures until this new process is in place.

On the subject of support, I'm sorry you had problems with our support lines, but as I am sure you can imagine, when a quick spreading issue like this hits, there is always potential for people to miss communications or notifications (e.g. first line just came in for the day and started answering calls before checking their email, etc.).  As soon as we knew what was happening we took steps to make sure everyone was aware and cases were dealt with as quickly as possible (looking at some cases, I see customers being responded to within 30 minutes of logging their call regardless of what level of severity it was logged at).

Hi Chetan

We are facing this problem.

I culdn't find registry setting mentioned in other comments may be ther are for spem 12 version.

But we are using sepm 11 and sep 11.

Could you tell us how to solve this issue.

Hi Pavan, can I suggest to quickly contact our tech support to check on the issue you are seeing. A couple of thing you could help to speed things up.

1. Memory dump, if it's generated

2. If you have manage to recover the computer, download http://www.symantec.com/business/support/index?page=content&id=TECH105414 to collect the logs, and file information.

The tech support member, would be able to walk you thru, if you are unable to peform this collection.

Thanks.

Hi Pavan,

Have you tried given steps in the following article?

http://www.symantec.com/business/support/index?page=content&id=TECH192811

We have following information as of now.

Which Enterprise Products are Impacted?

Based on our root cause analysis, we determined the problem is isolated to some Windows XP machines with file system drivers (usually encryption) running:

• Symantec Endpoint Protection Small Business Edition (SEP SBE) 12.1
• Symantec Endpoint Protection (SEP) 12.1
• Symantec Endpoint Protection.cloud (SEP.cloud)

This issue has not been reported for SEP 11 on any operating system, it might have different root cause.

However as AL76 suggested pls quickly contact support to check on the issue you are seeing.

Regional Support Telephone Numbers:
United States: https://support.broadcom.com (407-357-7600 from outside the United States)
Australia: 1300 365510 (+61 2 8220 7111 from outside Australia)
United Kingdom: +44 (0) 870 606 6000
India: Toll-Free 000 800 4401 456                                                                                                

Contact Symantec Customer Care on: http://www.symantec.com/support/assistance_care.jsp

Additional contact numbers: http://www.symantec.com/business/support/contact_techsupp_static.jsp

Meanwhile you can provide following information

1) Since you are facing this issue?

1) How many computers are affected?

2) SEPM downloaded definitions date and revision number ?

3) Have you seen this issue with specific operating system ?

If you are a experiencing a seemingly similar issue, may be worth contacting Symantec Technical Support to get assistance with troubleshooting it.

I awoke on Thursday morning (7/12/2012) with all but three of our restaurants with blue screens.  That is 48 restaurants in 5 states.  It took two days and thousands of dollars to get our computers back up and running.  This fiasco cost us in the neighborhood of $100,000. 

We spend all of this money on an antivirus solution to protect ourselves from just this sort of threat, than to turn around and have that software create the biggest threat I've seen.  This is just completely unacceptable.

What steps need to be taken so that we can get properly compensated for Symantecs tremendous FAIL.

Hi Carl,

Are you still having any technical issues?  If so, I can have someone from technical support contact you. 

Additionally, I'll have someone contact you to discuss your compensation concern. 

Regards,

Jon

I am working with technical support to bring the SEPM back online.  I have already heard from the Director, Product Management, to discuss other issues.  Thanks.