Endpoint Protection

Dear Symantec Community,

I am new to this forum and I make some mistakes as a newbie, please do advise me on best practices when using this forum. Please bear with me.

I was handed a relatively old workstation in my office to perform several upgrades. One of them involved upgrading the computer's Symantec Endpoint Protection to the latest version in the 11.x line.

The computer possesses an unmanaged client of Symantec Endpoint Protection 11.0.5002.333 (RU5) and is running Windows 7 Professional 32 bit. During the upgrade process, I stopped the Symantec Management Client,  before first upgrading the platform to SEP RU6 11.0.6000.550, and then to SEP RU6a 11.0.6003.833. At this point, I was told to restart the computer, and this was where the problems started to surface; each time the computer boots into normal mode OR Last Known Good Configuration, the Blue Screen of Death DRIVER_IRQL_NOT_LESS_OR_EQUAL error pops out, effectively rendering it unbootable. The affected file appears to be WpsHelper.sys, the IPS component of SEP.

I tried to uninstall SEP in safe mode (the only bootable mode left) which obviously does not work because the Windows Installer Service doesn't ever start in safe mode. Applying patches at this point is futile even though I am aware that there is supposedly a fix for this file that prevents the error, but I believed that the computer has never connected to LiveUpdate for at least 4 years because this is a spare unit and is on temporary loan to a client.

What steps should I take to rescue this system, without having to reinstall the entire operating system? System restore was turned off at the time of the installation so going back to an earlier image of Windows is not possible.

Any advice is kindly appreciated. Thank you!

-it_geek

Can you get SEP off the machine with cleanwipe?

This a very old, end of life version which was known to be buggy.

Try manually removing:

Manual uninstall documents for Symantec Endpoint Protection

You can run uninstall in safe mode.

Add these keys and unistall SEP first

https://www-secure.symantec.com/connect/blogs/windows-installer-safe-mode

once thats done.

Run the setup.exe

select custom

install only Antivirus / antispyware components, check if issue persists.

That is precisely what I am trying to do.

As I have mentioned earlier:

Normal/Last Known Good Config: System blue screens before I get a chance to access appwiz.cpl

Safe Mode: Windows Installer Service is unavailable.

EDIT: Yes, I am aware of the bugginess of this version of SEP. Why my organisation has refused to upgrade is a good question, though. I can't connect to the internet with that computer in that state anyway.

And I can't seem to find the page for cleanwipe, could you please provide a link here? Thanks.

Try the manual removal:

Manual uninstall documents for Symantec Endpoint Protection

Ok, this sounds like a promising solution. I will execute the commands and update you where possible.

Also, please advise on best practices on upgrade paths pertaining to SEP 11.0 RU5 to the last available release 11.0.7 (RU7), to avoid this error in the future.

You can upgrade right from 11.05 to 11.07

Just keep in mind 11.x is end of life starting in January. You're best off moving to 12.1.5 as soon as possible

That should be possible once I get rid of this issue, and restore access to the internet. I would be better off installing a fallback version of Symantec Endpoint Protection, get basic protection for the workstation before I connect this to the SEP Manager later and upgrade it accordingly. As I have mentioned, this workstation is old and unused for 4 years, so things can happen.

I say this because my organization has seen an increase in the number of IPS attacks, notably one of the Bash CVE attacks (don't remember the code), so it is probably it wiser decision for now.

EDIT: I can't seem to find a page which shows 11.0.5 to 11.0.7, or is it this one?:

http://www.symantec.com/business/support/index?page=content&id=TECH165494&key=54619&basecat=DOWNLOADS&actp=LIST

And then I upgrade to 11.0.7.4 from 11.0.7?

http://www.symantec.com/business/support/index?page=content&id=TECH213507&key=54619&basecat=DOWNLOADS&actp=LIST

Please clarify, thank you.

See the info here:

https://www-secure.symantec.com/connect/articles/latest-symantec-endpoint-protection-releasesd-sep-ru7-mp4-11074001398

Yes, you need to get to 11.0.7 first then move to the latest and final release 11.0.7 RU7 MP4

Download using your serial # from here:

https://symantec.flexnetoperations.com

https://fileconnect.symantec.com

download the latest version of 11.7 you could find

run the setup.exe , install only AV

replace sylink make it connected to 11.6 SEPM. there is no schema change so SEPM will be able to manage this verson

Follow Solution 3:

http://www.symantec.com/business/support/index?page=content&id=TECH92556

That it for now,when you are ready for the upgrade , upgrade everything.

This method is same for all the version

http://www.symantec.com/business/support/index?page=content&id=TECH92556

The computer is taking unusually longer to start up than usual, but thanks for the assistance. Will update where possible.

On a side note, is there any known malware that makes use of the Interactive Services Detection to exploit session 0, especially by making use of comctl32.dll to display a window to run its payload?

Have you tried running a malware scan with a LiveCD such as SERT or Norton power eraser?

https://security.symantec.com/nbrt/npe.aspx

How to make the Symantec Endpoint Recovery Tool boot from a USB memory stick

@Rafeeq I am only upgrading an unmanaged client, NOT SEPM. SEPM has already long been migrated to 12.1.5 (I think). My colleague is in charge of that, so I have rather restricted access to the license keys.

To avoid repeating this same error, should I install back the base client (11.0 RU5), perform a liveupdate, before upgrading to 11.0 RU7, etc, etc... When I did that with another PC it got a different error message, something like "SMC Service could not start as the file version is not the same as the one installed", or some error like that. Does this have to do with the fact that I did not turn off the Symantec Management Client Service before running the patch installer?

For this PC I didn't even connect it to liveupdate, I just installed the patches first and I got into this almost unrescuable error.

Please advise on the procedure.

Because it's unmanaged, just do a complete removal. Get cleanwipe if you need it. Then install RU7 MP4. It will just make thing much easier and efficient. No reason to do multiple upgrades.

Uninstall the installed version first

directly install 11.0.7( better not to have 11.5) it had lot of issues., run LU

I belive last time you would have installed client only patches.that might cause some issues.

I currently only have the 11.0.5 installer with me (client only), looks like I don't have a choice until I get back to office tomorrow. Also, the client needs this tomorrow so I am trying to make do with whatever I have, and the client upgrade patches seem to be the only downloads available to me as an unmanaged client. I would like to stress that my position in my organisation doesn't grant me access to the license key needed to access fileconnect, because that is under my colleague.

Also, regarding what Brian said, I found some rather unusual activity on this computer...

On the 13th of November there was a pop-up by Windows 7's interactive services detection, prompting that comctl32.dll, one of the Windows 7 components, wanted to display a message. When I clicked show message I was taken to session 0, but I saw nothing... I call this unusual because it has never happened before, and it is pretty weird.

Two days later, Proactive Threat Protection suddenly started flooding me with a horde of 168 notifications. Apparently, it started with it detecting a trojan horse A*******.TMP found in C:\ProgramData\Symantec\Symantec Endpoint Protection\xfer, but what's most interesting about this trojan horse is that it aggressively recreates itself even after getting quarantined by Symantec Endpoint Protection, and the notifications only stopped after I unplugged from the internet. I have submitted a sample of the original trojan horse to symantec's malware response center under the quarantine window, submitted under the email address "solar*******(at)gmail(dot)com" <email redacted to protect identity> but have yet to receive a reply. Maybe you could update me on this?

I am not sure if these two incidents are actually related, but if they actually are, which I do suspect, we could possibly looking at a rather sophisticated and even more so dangerous 0-day exploit here. I have no doubt that you, always being updated with the security landscape, know about the Darkhotel APT report by Kaspersky which involved very sophisticated malicious injections designed to steal data. I can confirm that one of my organisation's high level employee was indeed hit by this attack at some hotel in Shanghai, and traces of the malware were only detected upon reconnecting to our organisation's coroporate internet. Obviously, the dormant malware turned active knows its targets well and was intent on stealing information from possibly our data servers, so our organisation has been quite on high alert since. However, it may have actually spread quite a bit through our network before it was recognized and deleted by SEP. That's why I got really worried when these two incidents started to pop up.

I have run NPE, SERT, Rootkit Sysinternals, and SEP's standard malware scan but no threats were revealed.

I am wondering if there's any buzz on the Symantec Community regarding the Darkhotel APT, and if there is, I would be quite happy to hop on board the conversation over the weekend!

Regarding the ability to scan keyloggers, is it true that SEP 11.0 RU5 does not support keylogger scanning on Windows 7 x86? I have another corporate PC running this platform and the option to scan for keyloggers is grayed out. Is this the limitation of SEP 11.0 RU5, a malware attack, or just the need to upgrade to RU7?

Let me know what you think of this, thank you. Also, regarding the file submitted to the Symantec Malware Response Center, if there's any information such as analysis on that malware please notify me!

-it_geek.

The location of those TMP files indicate this to a known issue / false positive. This was a problem in 11.x and still continues in 12.1.

In regards to the submission, Symantec should've sent you an email with a tracking number

So the original TMP file itself is a false positive too...? If that is the case, why did the notifications only stop once the workstation was disconnected from the internet?

UPDATE: Uninstalling SEP 11.0 RU6a now, reboot pending.

Judging by the info you provided that looks to be the case but would need to see some logs.

Notifications for what and to where? The SEPM?

Notifications generated by Auto-Protect. Regarding logs, those were wiped out when I had to reinstall SEP following a serious conflict with Lenovo's Power Management Driver, but I might have a csv file I exported of the threat log... if I can find it I will upload here.

EDIT: Damn, looks like I can't find it for now. Too many files to sort. All I do remember, however, is that SEP also detected an attack matching the IPS Signature "GNU Bash Attack CVE 2014....", and then just 2 minutes later the notification mayhem started.

That's the shellshock vulnerability.

Is there any reason then, to believe that IT DID manage to punch through IPS even after being detected? I think my IPS definition files were on the latest release just 2 hours before the incident.

IPS was likely alerting it blocked the attempt

Unless I can find the csv file I can't be for sure. But this is not the only problem causing software, Lenovo's built in software is giving me even bigger problems (but this is out of your scope so don't bother)

was the uninstall /reinstall success?

UPDATE: Uninstall is indeed a success, but that is no cause for any joy. Now when I reboot the system, it seems to boot normally, until it begins to load the desktop and then it goes to a black screen. No errors this time whatsoever. And then it freezes. Seems that there is more than one issue afflicting this workstation, and is beyond your control. I am probably looking at a hardware issue (possibly caused by Lenovo's Power Manager Driver) Thank you for your assistance thus far, and I will post any further updates (if it is required).

The correct solution has already been marked by me.

My previous comment somehow disappeared.... (or is it still being moderated?)

Yes, it's in the queue to be released. Should happen some time today.

Anyways, I think this topic should be put to rest, seems indeed the problem is now outside your scope. Thanks for the assistance. Just don't be too eager to lock the topic, I might post further updates or questions pertaining to this topic later. Thanks!

Topic will lock/close automatically after 6 months of inactivity.

Ok, I am actually really at crossroads with the issue... it could easily be a malware issue or a software bug with one of Lenovo's software. Currently, I have opened a case on Lenovo (see: http://forums.lenovo.com/t5/W-Series-ThinkPad-Laptops/Out-of-the-Frying-Pan-Into-the-Fire-Lenovo-W510-encounters/m-p/1801327#M49325) which pertains to diagnosing possible bugs with their software.

I am now struggling to figure out whether or not it is a software malfunction or malware attack causing the issue, but here are the current symptoms:

Now, after removing Symantec Endpoint Protection, when I login to the desktop in Normal Windows Mode or Last Known Good Configuration, the computer will load up to a certain point and then freeze up (i.e. he HID devices including the internal keyboard, internal and external pointing devices (both touchpad and mouse) will completely refuse to respond. And most of the time the screen will also black out. Only once or twice have I seen the screen return back to normal (but still hung), only to be replaced by the following events followed by an absolute freeze:

1) The computer attempts to readjust its resolution, and then it hangs before showing a BSoD displaying:

"Attempt to reset display driver and recover from time out failed."

There is no error code available for this BSoD, strangely, BUT the link to the system event log in this post may be able to provide some idea. I suspect it is either error 116 or 117, though.

2) In another case, the display comes back from the black screen, but immediately after that, a window pops up saying "Hidden Window: SmartAudio Controller Driver" and then an error message comes up saying that "The application has encountered a fatal error." And then it hangs.

3) In the early stages of the system failure, I would also see the brightness display increasing, decreasing and that after that freezing. At first, I thought that this would have something to do with ThinkVantage Power Manager (since it is mostly responsible for managing automatic brightness during logon/logoff and shutdown/startup.) Even after uninstalling ThinkVantage Power Manager and its related Power Management Driver, it still fails to solve the problem.

Apart from that, the system has always gone into a state of blackout and devices fail to respond. The computer only functions in safe mode... I have tried to disable all startup items and services in msconfig to attempt to diagnose, but even that does not help to restore normal mode to its normality.

As I have mentioned earlier, there are also two possible signs that could indicate malware:

On the 13th of November there was a pop-up by Windows 7's interactive services detection, prompting that comctl32.dll, one of the Windows 7 components, wanted to display a message. When I clicked show message I was taken to session 0, but I saw nothing... I call this unusual because it has never happened before, and it is pretty weird.

Two days later, Proactive Threat Protection suddenly started flooding me with a horde of 168 notifications. Apparently, it started with it detecting a trojan horse A*******.TMP found in C:\ProgramData\Symantec\Symantec Endpoint Protection\xfer, but what's most interesting about this trojan horse is that it aggressively recreates itself even after getting quarantined by Symantec Endpoint Protection, and the notifications only stopped after I unplugged from the internet. It sounds like a false positive judging by what the .Brian mentioned, but why did it only stop when the computer was unplugged from the internet?

So, does this actually seem like a malware issue, or just a case of poorly coded software/hardware drivers? Please do refer to the Lenovo Forum link enclosed in this post to stay updated on the issue, and I am hoping to find closure to this problem.

Enclosed are the links to the Event Viewer logs. These will only be accessible for the next 48 hours, following these will be removed to restrict circulation:

https://drive.google.com/open?id=0B3tjsi7FePplUi14Mkc3b2MxdE0&authuser=0

I am sorry to be quite annoying, but the post I have made almost 10 hours earlier has not shown up...? Did it get lost in cyberspace, or was the post seen as spam?

Showing as spam, needs to be released by an admin

Or repost but take any links out, that may help

Erm, but if I take the links out... how are you guys to see the event logs for Windows (they are really huge, and I can't upload via the inline attachment system here, so I had to upload it via Google Drive). Won't you be able to make an exception for this?

Then wait until an admin releases your post.

Ok then... I will just have to wait....

Yes, it's been marked as a SPAM. I will try to check with an admin to make it publish. Trojen detection under temp folder was a known issue and has been fixed in the later releases. 

If possible upgrade to the latest version of SEP. Can you repair the Operating system?.

Hello,

Sorry for the extremely late reply, what's interesting here is that after 1 entire week of starting the services and its dependencies one by one I finally isolated the services that's causing the computer to display the abovementioned symptoms:

Wired Autoconfig

WLAN Autoconfig

WWAN Autoconfig

This is probably the funniest joke for the cause of a computer crash, but this joke is quite crippling too, because I can't use my LAN or WLAN without these services!!!

Any idea what's actually causing the problems now? I suspect SEP might be causing issues, especially with the Network Threat Protection component. @Chetan I appreciated your input in the Lenovo Forum, however, I also do want to listen to a second perspective from other Symantec Employees regarding a compatibility issue with the network drivers and SEP.