Endpoint Protection

I have done some searching in Google, here, and Symantec Tech articles and cannot locate any information on this....

Windows XP Pro SP3 (Attached to a domain)
Install of SEP Client goes flawlessly and even pulls in the policy updates and does the FIRST reboot after the initial install does not cause any issues.

The system can run ALL day and longer with no issues, BUT the moment the system reboots it presents with a BSOD during load up...  as such:
STOP: c0000218 {REGISTRY FILE FAILURE}
Cannot load hive (file): \SystemRoot\System32\Configure\Software

Begin mem dump blah blah blah
---------------------------------------------------

Before you say you say it, yes the registry is corrupted and requires a system restore point (b4 SEP 11 installed) to recover the system to a useable state (Safe mode still works to run system restore)

At 1st I though bad HDD is corrupting data, nope
CHKDSK finds nothing wrong (except the reg file)

As long as SEP 11 is NOT on this system it is fine. So, anyone come across this before? Why is it corrupting the registry on reboot and how can i resolve this?

I have installed SEP 11 MR4 on 70+ workstations and only this ONE is causing me grief....

EDIT: Have tried with all options all the way down to just Anti-Virus/Anti-Spyware options

EDIT: Attached Dump File

We have similar issue, where we have installed SEP MR4 first and then installed Win XP SP3. System become unstable after SP# installation

Hi Tejas:
Can you capture the memory dump contents. I'll do my best to track the BSOD.
-Suren 

I have a client that is experiencing the same issue on 80% of their machines.


Have had a ticket with Symantec for 3 months now (including full memory dump) and they still have not resolved the issue. They just keep saying they arte still analyzing the dump.

Has anyone found a fix for this? It's embarrasing selling a solution to a client and touting it as one of the best, just to have it crash all their computers after installing it...

Three months for a ticket and still no solution? do u have SLA with them? three months seems is way too long.

@ StephenW, can you give me your case number? I can then check the status of your dump analysis.



Thomas

@Wirebug - If you haven't done so already, I would recommend opening a case with support and providing them with a full memory dump. With a full memory dump, we are usually able to find root cause. We have a team in place that does nothing but read memory dumps and they are usually fairly quick at coming back with an answer.

@StephenW - As Cycletech has said, if you provide a case number, we can check the status of the case for you and find out what the current status of your case is at.

I don't know if we have an SLA, but we do have a "Essential Support" Support contract that IS VALID.

The case number is 312-156-857. The individual who I have been working with at Symantec is great, it's just that they have been analyzing a memory dump for 3+ months and my client is starting to question if it was a good idea to purchase this software.

Please keep in mind we are running the Symantec Multi-Tier Protection Small Business Edition. And this is occuring on 80% of the workstations at my client's office.

Used the cleansweep app to remove any old reminents, I've personally checked the registry for anyhting odd (manually, not with software), but still no luck. After installing, on the 2nd or 3rd reboot, the system BSODs on every boot stating that the registry is corrupt when it is indeed not corrupt.

Typically to get the workstation back I either have to restart it 10+ times to get it to boot (which I am then in a hurry to uninstall the software because eventually it BSODs), or I have to enter safemode, disable the symantec product manually in the software and also in the registry, then restart. After doing this sometimes it allows me to boot properly and uninstall it, or I have to go back to safe mode and issue a quick chkdsk for it to flag the registry as clean.

Please note that it is NOT the registry. Even after backing up a good copy, and restoring it, it still says it's corrupt. And also, I don't think that the registry is corrupt on multiple machines (80% of the office). They are in decent working order, and have no software out of the ordinary.

Any updates would be great.

Thanks

Stephen

I looked at the your case,and the developer is working on analyzing the dump. The last case comments were entered on Friday. I have pinged development for an update. Stay tuned for more.

Thomas

Did the other guy ever get his issue resolved? Or has he just disapeered?

Steven, 

WireBug never responded back on this thread. I just sent him a PM asking for an update on his issue.

Thomas

Have not found a resolution for this issue yet. Sorry I was not monitoring this thread as I assumed I would be emailed. I see I am not the only one suffering this issue. I will see about getting a memory dump if I can. For the time being I been running an older product on the system in question but would really like to get the SEP going. 1 PC out 70 and all the rest are running great.

I am going to explore a possible security issue in the registry that may have been caused by our previous AV client, but I have my doubts it is the cause since 69 other PC's took SEP just fine.

How would I supply the Memory dump? I am pretty sure I have support coverage for the year but seeing as how the previous person to open a case has not got a resolution I am hesitant to start a ticket myself since the issue is already being looked into.

There could be so many different reasons for your BSOD issue. It would be best if you gave us your dump for analysis.
I will send you a PM with upload instructions.

Thomas

\SystemRoot\System32\Configure\Software
Corruption causing at kernel level.There is a chance that this can be a Rootkit.
Use IceSword to find out what are sys files loading in kernel level and if you find anything suspicious either submit it or try to remove it.

According to the dump the crash happens while the "system" is trying to load the software registry hive.
Have you tried on another computer as it seems this particular machine got a corrupted hive.

From the dump:
ERROR_CODE: (NTSTATUS) 0xc0000218 - {Registry File Failure} The registry cannot load the hive (file)
ADDITIONAL_DEBUG_TEXT: \SystemRoot\System32\Config\SOFTWARE - hive could not be loaded.


f64e4c34 8060c09c c0000218 00000001 00000001 nt!ExRaiseHardError+0x13e
f64e4dac 8057aeff 00000002 00000000 00000000 nt!CmpLoadHiveThread+0x1e8
f64e4ddc 804f88ea 805bab07 00000002 00000000 nt!PspSystemThreadStartup+0x34

Vikram,
Sounds more like corruption rather then "evil" code.

I've scanned the system with every tool I can find.... it's clean

Please understand this BSOD happens after a reboot, the reboot is not caused by BSOD. As long as SEP 11 MR4 is NOT installed the system behaves just fine.

After install 1st reboot goes just fine but when user powers down for the day and returns in the AM they are greeted with the BSOD and corrupt registry.

Same behaviour here too.

Only with me, it's not one machine, its 10+ machines.

Each of the computers this happens with, they ALL have the exact same BSOD. And all the behaviour is the same!

I have this issue on 6 machines as well. For us, this issue began on June 9. All are Dell T3400's with LNE100tx network card.

For now I have uninstalled SEP 11 and the computers work great!

cono_sur,

Any chance you can call Symantec, submit a ticket, and do a memory dump. I'm sure we would all like to get this taken care of soon as possible!

Also, for the Symantec pplz, Any word on the memory dump analyzation?

Unfortunatly I am currently unable to provide a full memory dump. This would require me to break a system that is used all day everyday. I am on VAC next week so when I return I will look into "re-breaking" the system to get a full dump but it may be 2 weeks from now before I can provide it.

If there is anyone that currently has a system experiencing this I am sure Symantec ppl would love to get a full mem dump file from you :-)

I have the full memory dump as well as the mini dump. How can I send these to you for investigation?

Has anyone tried using the MDOP toolkit to analyze / check what could be the issue here?

I also tried with XP SP3, but did not face any issue with the client installation and reboot. Could you let me know the patch level of the affected systems?

I'd recommend that you open a case with Microsoft Support's Perf or Setup team and have someone from MS's end also look into this issue.

I ran CleanWipe, then re-installed the latest version and all seems to be fine now.

Just for the record to help resolve this issue,

I have ran CleanWipe on multiple machines, multiple times to make sure that the old software may have been removed.

I've even manually checked the registry for entries that could have anything to do with Symantec and/or virus definitions with no luck....

However what is odd, is that one machine in the office that it installed on to with no problems, one day there was a virus definition update and BAM, the issue occured with that machine.

I'm wondering if this will get fixed. It's making me look like a fool infront of my client as I'm the Symantec Partner who sold them their Symantec software....

Hi is NTP installed?

After a couple of days, the problem cropped up again and I've had to uninstall SEP. Have not had any problems since uninstalling.

NTP = Network Time Protocol?

NTP also stands for "Network Threat Protection"

Did you open a case with Symantec? If so, can you give us your case number?

Oh yeah. Duh.

Yes, Network Threat Protection is installed. I have not opened a case with Symantec yet, but will do so if I do not hear back from our corporate IT group.

@ Cono_sur, I will create a fileshare for you to upload the dump. Look for a PM in the next 30 minutes.

If anyone else can provide a dump, please let me know.

Regards,
Thomas

Ok. Thanks. I go on vacation in less than 2 hours. =)

Thank you!

Thanks, Hopefully we can get to the root of all this. I will update here when we have something to report. Have a great vacation. : )

Thomas

@  cono_sur

The dump that you provided was a mini dump. I realize you are leaving, but we really need a Full dump to find root cause. There is a reference to LNE100V5.sys a a possible cause of the crash. Looks like an old linksys Ethernet driver.

Can someone else in your organization provide the full dump?

Folks, We here at Symantec are committed to helping our customers.

After review of cono_sur's mini dump it has been been found to be a different issue from StevenW' s BSOD.

@ conosur, please open a case with Symantec support when you return from vacation. We will need to collect a full dump from you to find root cause of your issue.

@ StevenW, Please upload your latest full dump to the FTP site. Support will continue to troubleshoot your BSOD issue. We will get to the bottom of this.

If anyone else is experiencing this type of BSOD, it is recommended that you open a case with support. Please be prepared to provide the full dump for analysis.

Thanks,
Thomas

I had similar issue with other folks posted on this Forum machine experienced BSOD issue on some of my machines with SEP 11.0 installed. My issues started to happen when I started to deploy device control to manage the removable storage.

I have one policy that withdraw Application and Device Control, which means there is no device control to cleints, i call it device enable group. I also multiple groups, either disable certain or disable all removable storage devices. 

when I moved the client to the device disable group to disable removalbe storage devices, my client contacted helpdesk reporting machine blue screen and reboot looping at the next reboot time.

In my "blocked devices", I blocked all the removable devices. a long list including USB, CD/DVD. Floppy, etc and in my "devices excluded from blocking" I allow different type of "Human  Interface Devices", and IDE Disk,  Device name: IDE DISK, Identification: IDE*

My understanding is SEP 11.0 code to be able recognize which drive is the OS boot drive, with my devices exclusion configuration, I found this might not be true, by checking the logs on the blue screen machine, I found machine hard disk was blocked for OS access, I updated my devices exlusion policy to exclude "IDE\DISK*", on the blue screen machine F8 to boot into option and choose safe mode with network, once logon, update the SEP policy, then reboot the machine back into normally mode and machine never experienced blue screeen since then. The blue screen does not happen to all the machines, it could be SEP 11.0 does not work as designe on certain hard disks, I have seen the issue on some WD and Sumsung drives. This fix might not fix your issue, but you at least to have some clue where to start to troubleshoot this type of issues.

Also I saw couple of posting that people stated it took more than 3 months for Symantec to perform dump analysis and still no result, I do not think this is acceptable SLA, I hope Symantec really can improve their post sell services.  

We have similar issue, which has no answer yet. We have installed SEP MR4 first and then installed Win XP SP3. System become unstable after SP3 installation

@ Tejas Shah,

Do you have a case open with Symantec? If so, can you give us your case number?

What did you install? MR4 MP1a? we had no issues with it on XP SP3

Please open a case with Symantec support when you return from vacation. We will need to collect a full dump from you to find root cause of the BSOD issue.

Thanks,
Thomas

But we were getting a different kind of BSOD (Stop 50 error) on every reboot after the initial one and ended up disabling Tamper Protect to resolve it. The only simularities to your issue are the BSOD and rebooting, but I thought I'd mention our "fix" just in case it helped.

-Mike

Greetings

I have been testing SEP 11.0.4014.26 which I believe is SEP 11 MR4 MP1a and the installs went beautifuly on all the machines save one.  This machine BSOD on a user login (user wiht admin rights) but not on the administrator account.  The BSOD complains about the ipsecw2k.sys driver.  After some google'ing I found some relationship to the NORTEL VPN client.  After removing the VPN client the machine boots without incident.

Any ideas why the VPN client would cause and incompatability with SEP 11?

Many Thanks for your feedback Paul

Greetings Again

Re-installing the Nortel VPN clinet made it crash and un-installing it fixed the problem again so it looks like this is the root cause of the problem.  More work to do I appreciate your comments and suggestions.

Paul

How about updating/upgrading to MR4 MP2? Btw what version of Nortel VPN client are you using? (You should get the updated drivers)

Please use XP SP3

Check the SEP release notes, I believe this issue has been resolved wth MR3.

http://service1.symantec.com/SUPPORT/ent-security.nsf/docid/2007121216360648

I provided the full dump in a ZIP file (MEMORY.zip, 68 MB) . Did you not receive that?

cono_sur

Sorry, I was unaware that you uploaded the file. I am in the process of downloading now.
Stay tuned for updates.

Thanks,
Thomas

@ cono_sur,

You dump has been submitted to our engineers for analysis. I will update here when i have more info.

Regards,
Thomas

@ cono_sur,

The BSOD was a stack overflow crash caused by a very old linksys driver (2001) using way too much kernel stack. The kernel stack is 12k only and does not grow.
Once it is exhausted, the system bugchecks. The linksys driver, LNE100V5.sys, is using over 4k of the 12k stack.

Updating your Linksys driver will resolve the issue.

Regards,
Thomas

Thanks for looking into it, Thomas!

I'll look into updating the driver. However, if I recall correctly, there were no driver updates for this network card (which we purchased brand new only last November!). I'll check again.

cono_sur

A couple things to note, when i searched the Linksys site, the driver version you are using is dated from 2005. The data sheet does not list Vista as a supported OS.

http://www.linksysbycisco.com/US/en/support/LNE100TX/download

Thomas

I am seeing this on multiple HP desktops and notebooks Windows XP SP2 w/ SEP 11.0.4000.2295.  BSOD stating REGISTRY ERROR.  Workstations seem to be fine but if you reboot them, they do not come back up.  I have seen workstations BSOD after manually running Live Update, and I have seen them BSOD when trying to uninstall via Add/Remove Programs.  All stop errors state the same thing. REGISTRY ERROR.  I should also mention that it this is totally random, and seems to be only on new builds with a good image.  I do not have a fix, but I have created a workaround that gets my machines back up and running in less than 10 minutes with no system restore or reinstallation of SEP11.  I would think our first step would be up upgrade to RU5 but I am doing the below in the mean time.

1.  Boot to SystemRescueCD or any other live distro.
2.  type ntfs-3g /dev/sda1 /mnt/windows to mount the Windows partition.  Then type cd /mnt/windows/WINDOWS/system32/config to change to the registry directory.
3.  type ren software software.old to rename the software registry hive to software.old.
4.  Reboot the workstation into Windows.
5.  Let the Windows loading screen disappear and wait for the hdd activity light to stop.  The time this takes will depend on the speed of your workstation.
6.  Reboot to SystemRescueCD and type ntfs-3g /dev/sda1 /mnt/windows.
7.  type cd /mnt/windows/WINDOWS/system32/config
8.  type ls and make sure the SOFTWARE (case sensitive) file exists.  If you do not see SOFTWARE then repeat steps 4-8 again.
9.  type rm SOFTWARE to delete the SOFTWARE file.
10.  type ren software.old software to replace the original software registry hive.

Windows should now boot successfully with a message stating the registry was restored from backup or log.  Works for me about 10 times a week, 3 times just yesterday.  I have actually scripted the sysresccd to make it easier,  try it out.  Cant say it will work for everyone, but in my environment with 600 or so devices, this is much easier than imaging and hoping it doesnt repeat.

Hi Br_ndon,

You should make a new thread for your issue. The thread that you tagged onto is 30+ weeks old and will most likely be ignored by the majority of users here in the forums. So please make a new thread, and cut and paste what you wrote above. Feel free to link back to this thread if you feel that it is relavent. I think you will get faster responses this way.

Thanks
Grant