Endpoint Protection

I have a very strange situation that is occuring on a handful of systems and it is a little worrysome.  I have a mixture of Windows 2003 and 2008 servers that have been affected.  Overall I have several thousand endpoints that have deployed OK, but for some reason there are a few that dont seem to like the upgrade.

Some were running a form of v11 and others 12.1.1101.401, the target for all is 12.1.2015.2015.  Essentially what has happened is the NIC is essentially offline and you cannot access it via RDP and you need to console into the machine.  Once on the sytem, there is nothing obvious to point to SEP, but long story short, each time I have needed to run cleanwipe to remove SEP.  After removing and rebooting the system is OK.  On 2 of the systems I have been able to go back in, clean out a few left over Symantec Reg keys, reboot and install 12.1.2 without issue.

I need to probably do the same to the others to get them deployed, but the concern is that the install is randomly taking down production systems even after what appears to be a successfull installation.  The fact that I can run cleanwipe to restore function, delete registry keys, reboot and install tells me that SEP is OK in the end.  Or so it appears, but what is happening to cause the systems to go completely offline after the initial deployment,

Any thoughts or ideas would be great.  If there is something in the log I can look for, I can, but the simple fact that it completes successfully makes me think i will be searching for the unknown without a symhelp tool at the time a system is in the failed state.

If someone else has seen this and has ideas, I am all ears.  But like I said, it has been completely random, newer 2008 servers, older 2003 and a mix of 12.1.1 and v11.x that all yielded the same failed state.  In total that number is probably fewer than 10, but I still must seek some answers if they exist.  Thanks.

I've seen this on some of our machines that run an older version of 11.x (11.0.4 and lower) and were upgraded to 12.1.

Now it has only happened on workstations, because we only run AV on servers. I assume you have either NTP or PTP or both on these machines?

In our case we just ran cleanwipe and was done with it. We didn't have time to troubleshoot it. Out of curiosity, have you tried upgrading to a newer version of 11.x than going to 12.1? I know there were soe issues back in the older version of 11.x that have since been fixed.

I believe the oldest version is probably 11.0.6300 with others at 11.0.7000 and then 12.1 as noted.

11.x for the most part did not have PTP and NTP installed.  12.1.1 had everything installed, AV, PTP, NTP.

So again, nothing that jumps out as a potential cause.  My current theory is that the failure is with the installation of the teefer driver for NTP.  Because some systems aren't running NTP at install teefer should not be there requiring upgrade, while others have it and need removed and installed.

Either way, this is the only thing that seems to make sense.  Something during the installation is failing and breaking the network.  The fact that the GUI shows the FW is malfunctioning makes me believe that too.

One thing to note while writing this I was actually dealing with a failed system.  I was actually successful running the uninstall, without requiring cleanwipe.  The client showed as 12.1.1101.401, it uninstalled, I rebooted, installed 12.1.2015.2015, rebooted, system came up with the network functional, the GUI said it needed another reboot so I rebooted again and now everythign seems happy.  At this time, it would appear the issue might most like be a failed install, which then requires an uninstall and re-install to correct the issue.  Now if only I can determine what caused the failure in the first place.

I did run a symhelp on it and will submit it to support and if it contains anything useful I will report it.

OK, so it has been concluded that this is a known issue.  There is a brand new Technote that was just published but it is not searchable yet (03/11/2013), TECH202029.

The issue is to be resolved in 12.1RU2MP1.  Essentially, 12.1RU2 is not gracefully deleting Teefer, causing the network stack to become corrupted.  MP1 is due at month end, give or take.

Current work-around include cleanwipe or deleting the registry via the process listed below:
 
Once networking has been lost:
1.        Add the firewall component, if it is not present, and reboot.
2.        Remove the firewall component, but do not reboot.
3.        Back up the registry and delete the following registry keys if present:
HKLM\SYSTEM\CurrentControlSet\Control\Network\{4d36e974-e325-11ce-bfc1-08002be10318}\{72891E7B-0A3D-4541-BDCB-3DA62E25B6A8}
HKEY_LOCAL_MACHINE\SYSTEM\ControlSet002\services\eventlog\System\Teefer3
HKEY_LOCAL_MACHINE\SYSTEM\ControlSet002\services\Teefer3
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\eventlog\System\Teefer3
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Teefer3
4.        Reboot the system and networking should be restored.

Looking in the Symhelp in the sis-inst log, it shows the presence of of the above registry keys.

It’s hard to predict if it will happen or not, in developments testing notes they could only get it to reproduce 1 out of 3 times, so it sounds like it’s a timing issue.

Good to know. Thanks a lot for sharing this.