Updated July 16th, 2012 10:30 AM PST: Additional details provided on confirmed examples where we have seen the blue screen issue.
Updated July 31st, 2012 1:36 PM PST: Issue was also experienced on Windows Server 2003 and has been confirmed by Symantec.
What Happened?
On July 11, 2012 at approximately 10:30 PM PT, Security Response started receiving reports of customers experiencing blue screens on Windows XP and Windows Server 2003 machines after applying definitions July 11th revision 18 and SONAR definitions July 11th rev11. This update only contained signature updates and no change to the SONAR driver.
The problem has been identified as a compatibility issue in SONAR definitions released July 11th at 6:25PM PT. Once the cause of the issue was discovered, the signature was removed from the definition set and an updated definition set was published. This “rollback” of signatures was done on July 12th at 2:51AM PT. Once the signature was rolled back, no new issues were reported from the field.
Why Did the Blue Screen Issue Happen? (Updated July 16th, 2012 10:30 AM PST)
After a full evaluation and root cause analysis of the issue, we have determined that the issue was limited to machines running a combination of Windows XP or Windows Server 2003, the latest version of the SONAR technology, the July 11th rev11 SONAR signature set, and certain software. Only customers running this combination of technologies and who downloaded the July 11th rev11 SONAR signature set via LiveUpdate between 6:25PM PT and 2:51AM PT on July 12th were affected.
The root cause of the issue was an incompatibility due to a three-way interaction between software that implements a file system driver using kernel stack-based file objects. The three-way interaction is between the software that implements a file system driver (using kernel stack-based file objects), the SONAR signature and the Windows XP Cache manager. The SONAR signature update caused new file operations that create the conflict and led to the system crash.
We have confirmed examples of this interaction with the following products:
- Novell ZenWorks
- PGP Whole Disk Encryption
- Sophos LanCrypt
- SlySoft Virtual Clone Drive
How Will Symantec Prevent this from Happening Again?
Symantec understands the consequences of this type of issue to our customers and goes to great lengths to prevent them. The quality assurance process for SONAR signatures is extensive. The process includes:
- Peer review and vetting of all signatures
- True positive testing
- False positive testing
- Functional testing of all signature content
- Compatibility testing
The compatibility testing part of the quality assurance process for SONAR signatures missed catching this compatibility issue. It is this part of our process that we will be improving to avoid future issues. We are currently restructuring our testing process to improve compatibility testing and will not be releasing new SONAR signatures until this new process is in place.
Which Enterprise Products are Impacted?
Based on our root cause analysis, we determined the problem is isolated to some Windows XP machines with file system drivers (usually encryption) running:
- Symantec Endpoint Protection Small Business Edition (SEP SBE) 12.1
- Symantec Endpoint Protection (SEP) 12.1
- Symantec Endpoint Protection.cloud (SEP.cloud)
Mac customers are not impacted. Additionally, Symantec Endpoint Protection 11 is not impacted.
This issue has not been reported on any other operating system or other version of Symantec’s Enterprise security products.
How do Customers Know if they are at Risk?
If customers have not experienced a blue screen it is highly unlikely they are at risk. If customers have any concerns, they should make sure they are running the latest definitions and, if they are, they should have no risk of experiencing blue screens.
How do Customers Remediate this Situation?
Customers running Symantec Endpoint Protection (SEP) 12.1 or Symantec Endpoint Protection Small Business Edition 12.1, should refer to the knowledge based (KB) article for specific details on how to resolve this issue. If customers continue to experience issues or have further questions, they should contact technical support via their regular support channels.
http://www.symantec.com/business/support/index?page=content&id=TECH192811
How can Symantec Endpoint Protection.cloud Customers Resolve this Issue?
If a Symantec Endpoint Protection.cloud customer is experiencing this issue they should contact Symantec.cloud technical support.
http://www.symanteccloud.com/supportcentre/information/contact_global_support
Symantec Endpoint Security Team