Endpoint Protection

 View Only
  • 1.  12.1.4100.4126 Active Scan Causing Server Instability

    Posted Apr 21, 2014 11:18 AM

    We recently started upgrading our servers from Symantec Endpoint Protection 12.1 RU4 (12.1.4013.4013) to version 12.1 RU4 MP1 (12.1.4100.4126).

    The day after being updated, one of the servers started generating scary events in the System event log:

    Many instances of...
    An I/O operation initiated by the Registry failed unrecoverably. The Registry could not read in, or write out, or flush, one of the files that contain the system's image of the Registry.

    ...mixed in with many instances of...
    The file system structure on the disk is corrupt and unusable. Please run the chkdsk utility on the volume C:.

    The System event log also became corrupt.

    The first time this happened, I scheduled a chkdsk /f and restarted the computer.  Then the next morning, the same thing was found to have occurred over night.  

    I tracked this down to the scheduled "Active Scan" that runs at 12:30 AM.  Knowing this, I tried manually running an active scan and was able to induce the problem at will.

    The next thing I tried was to uninstall the client, run CleanWipe, then reinstall it.  This did not fix the problem.  Running an active scan still caused the errors to be generated and the event log to become corrupted.

    So, finally, I uninstalled the client, ran CleanWipe and reinstalled 12.1 RU4 (12.1.4013.4013).  After this, I lost the ability to replicate the problem.

    Our other servers aren't experiencing this problem.  The details of this particular server are:
    Type: Virtual (Hyper-V)
    OS: Windows Server 2003 (fully patched)
    Role: Terminal Server

    Our other servers aren't terminal servers.  But, we took this server out of load balancing when the problem developed.  So, this isn't a multiple user issue.  And I was able to replicate the problem right after a boot.  So, it wasn't an issue of something slowly happening.

    I had installed the client as "Basic Protection for Servers" and only "Virus and Spyware Protection" was showing in the client Status.  So, there was nothing exotic installed.

    It's my opinion that something got introduced in the 12.1.4100.4126 build that is not playing nice.



  • 2.  RE: 12.1.4100.4126 Active Scan Causing Server Instability

    Posted Apr 21, 2014 11:22 AM

    You may want to just go ahead and repot this to support so they can investigate,



  • 3.  RE: 12.1.4100.4126 Active Scan Causing Server Instability

    Posted Apr 29, 2014 02:11 PM

    So, I did open a support case. While I rolled back the production system to 12.1 RU4, I had an identically configured test system that I allocated for allowing Symantec to troubleshoot this case.

    Here's my experience so far.  I hope this will help you understand why I didn't want to bother.


    04/22 9:33 AM ET: Opened support case on the web portal.  I pretty much copied and pasted this post's text into the case description.  We have "basic" support.  So, support is supposed to be available during business hours.

    04/22 6:04 PM ET: 8 hours later, I get the initial response asking me to run SymHelp.  

    04/23 9:49 AM ET: I run SymHelp and upload the file.

    04/23 10:05 AM ET: Even though I included the case number in the SymHelp submission, I receive an email that support can't find files and they ask me to provide the upload path.  I don't have this, so I rerun SymHelp at 10:39 AM ET and email the FTP path.

    04/24 5:05 PM ET: 31 hours after submitting the second SymHelp file.  I receive an email asking when I can be reached by phone.

    04/24 5:06 PM ET: I reply that I'm available right now.  If not now then M-F any day at 9:00 AM ET.

    04/24 7:34 PM ET: A Symantec rep tells me she will be monitoring the case over the weekend.

    04/28 4:12 PM ET: A Symantec rep tries to call me.  I'm not at the office and this is not the time I listed as being available.

    04/29 1:22 PM ET: A Symantec rep calls me.  This is still not close to the time I stated I was available, but I am able to stop what I'm doing and take the call.  The first thing she asks is: "So, you updated to 12.1 RU4 MP1 and it won't accept your old policies?"  Now mind you, the text of this first post is what I put in the case description.  Yet, the rep clearly doesn't understand the problem.

    We conduct a WebEx session and I show the event logs.  She then agrees to escalate to the next level.  The rep has me run SymHelp again.  Then, I'm asked to redo it, this time with scanning for all data.  Then I'm asked to redo it again a third time, this time with WPP logging enabled.  We only save and she only downloads the third version.


    It's been almost a week since the initial support request.  These error messages are serious.  I would be stressed out waiting like this if we had a production system with this problem.  Even now, I'm not sure if I will get out of this without having to rebuild the test system. 



  • 4.  RE: 12.1.4100.4126 Active Scan Causing Server Instability

    Posted May 29, 2014 08:34 AM

    05/02 10:13 AM ET: I was asked to run the SymHelp utility again and this time also upload .etl files from " C:\Documents and Settings\All Users\Application Data\Symantec\Symantec Endpoint Protection\<version>\Data\Logs\"

    05/07 9:40 AM ET: A Symantec rep calls while I am in a meeting.

    05/07 10:09 AM ET: I email that I am out of the meeting and available.

    05/07 1:47 PM ET: The Symantec rep emails me that he is busy with another customer and will try to call either after the current call or in the morning.

    05/08 11:37 AM ET: The Symantec rep calls me.  This rep appears to have better technical skills than the past reps and I have an overall feeling of confidence in this rep.  We collect more logs through a Webex session.

    There has been no progress since 05/08.  The case appears to have stalled.



  • 5.  RE: 12.1.4100.4126 Active Scan Causing Server Instability

    Posted May 29, 2014 08:48 AM

    If you ever find yourself in a postition where you think no progress is being made, I'd recommend calling into Support again and asking for the Duty Manager.

    They're usually quite helpful in getting a case the attention it deserves.



  • 6.  RE: 12.1.4100.4126 Active Scan Causing Server Instability

    Posted May 30, 2014 08:16 AM

    Thanks for the tip.



  • 7.  RE: 12.1.4100.4126 Active Scan Causing Server Instability

    Posted Jul 16, 2014 10:12 AM

    I'm writing what may be the final update on this case.

    06/10 3:20 PM: I hear back from the support rep.  He wishes to discuss the case with me.  Note that he just wants to talk.  He does not want to conduct a remote session.

    06/11 12:53 PM: I receive a voice mail from the support rep.  The logs have shown there is an issue occurring when C:\pagefile.sys is scanned.  Since that is an OS file, he suggests this is not a Symantec issue and advises I open a support case with Microsoft.  

    06/11/ 1:47 PM: My response to the support rep's attempt to dispose of the case:

    "If you would like to get Microsoft involved, I would be fine with that.  I'm not going to open up a case with them myself, though.  It would be much better for you to use the connections Symantec has with Microsoft than for me to try to open a case as a customer.  The first step Microsoft has a customer do when troubleshooting system stability issues is to uninstall third party AV.  I've already demonstrated that the problem goes away when we uninstall SEP 12.1.4100.4126 and reinstall an older version.  Microsoft would just point the finger back at you guys.

    Let's not overlook this point.  Older versions of SEP work fine."

    06/19: The support rep contacts me wanting to collect more logs.  He wants to get logs using Process Monitor on the server experiencing the problem and then logs from a server that is functioning correctly.  My availability is limited at this time.  So, further delays are my fault.  

    07/09: I connect with the support rep.  We attempt to collect the logs, but find that the symptoms are no longer occurring.  Now this server has been powered off since the last time I worked with the support rep.  I did notice that it downloaded new SEP defs when it booted up.  Other than that, no changes were made.

    The server has been running ever since and the problem has not returned.  So...it appears that Symantec resolved this problem with a def update.  But, I think this experience does show why I didn't want to open a support case.  A lot of my time was spent on this and I had to fend off one attempt to deflect the case to Microsoft.

    One thing that was particularly irritating is Symantec's insistence on randomly calling.  Other vendors I deal with send me meeting invitations.  This allows me to plan my schedule and is respectful of my time.  Symantec support wants me to drop whatever I'm doing at a moment's notice.