Hi all.

After upgrading over 1600 machines to SEP 12.1 RU1 MP 1 we are now getting reports from users saying their machines are unresponsive to anything. This is since upgrading SEP to SEP 12.1.

This happens to a lot of machines directly after upgrading but is also happening to machines that have been rebuilt with a fresh install of SEP.

One thing to note - running SMC -stop instantly fixes the issue - the hard disk stops been solid and machne becomes responsive. Also I've since noticed that if we remove the network cable from the machine the machine instantly becomes responsive and the hard disk light settles down.

It's clear that this is a result of upgrading to SEP 12.1 RU1 MP1 - but why?

If I completely remove SEP and reinstall problem still exists. I've checked scan logs and no scans have run since upgrade on most machines so that's not the cause either

downgrading to SEP 11 fixes the problem

any ideas?

Thanks, Mark.

update on this

As a test - I've removed a component at a time to see which part could be causing the issuse - I've now removed every component, each time restarting and seeing if the problem occurs - the only component left is the base antivirus and antispyware - and the probkem is still happening.

Also....before I get asked about machine specs, free disc space etc.... these are all well spec machines, plenty of free space, plenty of RAM.... etc

please do check is there any policy is enabled under "Application and Device Control" for blocking any devices.. disable all policies under "Application and Device Control" and check.

Hello,

You said, "if we remove the network cable from the machine the machine instantly becomes responsive and the hard disk light settles down." and this is seen when only AV/AS is installed as well.

So it is clear that there is some network activity is being performed.

Could you please try Running the Process Explorer from Microsoft ( http://technet.microsoft.com/en-us/sysinternals/bb896653.aspx ) and check which process is taking the highest CPU and Memory usage.

Also check if there are any change in the AV/AS policies are applied which may be causing this issue. Is the AV/AS policy applied default ?

What happens when we disable the Network Settings Option in the AV/AS policy?

Awaiting reply.

strange thing is - there is no CPU activity - during this time it's only at 1%

we don't use the applicaion and device policy - set to disable

yes, this only hppens with AV running...

ok - some more news on this - I think it's resolved now

after slowly removing components and changing policies I've seen that disabling the Download Insight and SONAR fix this problem

as soon as I changed that and checked with machines, when they got the latest policy the problem was gone.

Wonder if it's SONAR or Download protection

either way, it's a poor show

cheers

Hello,

Did you check the CPU usage from Task Manager? If yes, I would then recommend you to check this via Process Explorer.

Process Explorer http://technet.microsoft.com/en-us/sysinternals/bb896653.aspx

also check what processes are running on this machine?

Did you check if there are any change in the AV/AS policies are applied which may be causing this issue. Is the AV/AS policy applied default ?

What happens when we disable the Network Settings Option in the AV/AS policy?

Could you run the Symantec Support Tool (SST) and check if it finds anything suspicious?

Using Symantec Support Tool, how do we Collect the Suspicious Files and Submit the same to Symantec Security Response Team.

Awaiting reply.

Even though you don't use application and device control, was it still installed? Should be installed under Proactive Threat Protection under Add/Remove Programs.

If so, try just removing it and see what the result is.

Hello,

SONAR is part of Proactive Threat Protection on your client computers. You manage SONAR settings as part of a Virus and Spyware Protection policy.

Whereas; Advanced Download Protection (Download Insight) is a new advanced protection feature included with the SEP 12.1 client. This feature allows the SEP client to leverage Symantec's Cloud-based reputation database when files are downloaded or executed directly from popular Web browsers. 

Check these Articles: 

Managing SONAR http://www.symantec.com/docs/HOWTO55215

really odd - disabling this seems to have fixed some machines - but not others....

and I'd removed the other fetures before and it did nothing - just bare bones antivirus and anti spyware were left on

we seem to be seeing the same situation.  Reports of lost network, trouble with the machines firewall.  Hard freezing upon boot with a black screen.   So far uninstalling AV is the only consistent solution.

You could use Process Monitor to see what's being scanned to hopefully narrow it down a bit:

How to Configure Sysinternals' Process Monitor to Record Symantec's Auto-Protect Events

http://www.symantec.com/business/support/index?page=content&id=TECH98079

Ok - well this is still continuing

some went away when I changed the policies - some did not....so I exported new packages which contained the new policies and used these....this works for some machines not all of them

I believe this is more to do with connection to the SEPM servers....

As I mentioned before - if you remove the network cable the hard disk LED stops being solid and the machine springs to life..... I tested the hunch that it was the connection to the management servers by taking them offline - this has the same result as unplugging the network cable.

this is now getting to be a big enough problem - it was a hanful of machines before, but  it's growing

Just an update on this......

I have further proven that it is something to do with the communication with the managment servers.

If I change the communication settings policy and remove the tick from the "download policies and content from the management server" then everything is fine....

as soon as I put that tick back in - when the client refreshes its policy it goes back to solid HDD LED and being unresponsive...

I've got a call open with support but I thought I'd update on here.... maybe someone has found a fix for this?

Thanks,

ok - we have no proven that this is definitely to do with communication with the SEPM servers....

basically - the SEP clients are trying to download updates from the SEPM server over and over and over and over - but each time they fail....so it tries again...and again...

this is causing the heavy writes to the disk....

my workaround so far is to either put them in a group that excludes communication with the SEPM server - not a perm fix obviously.....or download and run the latest intelligent updater form Symantec which forces the updates on.... Symantec thought that would be the fix.....but.............

if the laptop is off for a few days and therefore out of date - it tries again to download from SEPM and fails and locks the machine again

So far - this is only noticeable with Lenovo T-410 models.... of which we have plenty.... all other models so far seem to be uneffected.

I think I saw this when we first started seeing the new 12.1.1 clients get installed.  There was no HDD light issue but we don't comnnect the clients to SEPM for updates; we go straight to Symantec.  We have Bluecoat controlling access to the Internet so we saw repeated Internet signons popping up without the user having a browser open.

We had to make sure we had exclusions (both whitelisting in Blue Coat and in exceptions).  The two URLs were:

ent-shasta-mr-clean.symantec.com

ent-shasta-rrs.symantec.com

(I thought they would have *.symantec.com whitelisted but no, just specific ones)

Also check the reboot status (needs reboot) of some of the machines.  We kept getting "needs reboot to install features of Network Threat Protection".  Note that Sonar and Insight seem to be related to the Anti-virus product more than NTP.  Finally, I also checked the policy boxes to always disable Windows Firewall when running Symantec (re-enables upon uninstall).  Unfortunately, could tell what fixed most of the issue but our systems settled down after all that.  Hope that any of this will help.

Howie

Same problem here with some new Lenovo W520 laptops.

It sometimes takes laptops over 40 minutes to stop this activity after being turned on, even if only off for a day.

Symantec, where are you?  It's time for a fix.

I'm glad there's someone out there sharing my pain.

Chrish1 - if you check event viewer logs, under symantec endpoint protection - are you seeing content download fail from management server over and over and over

I will check that.  These laptops are used for training, and there is a training session happening now.  I will check later.

Thanks.

Hmmm...

Seems the clients are lopping when trying to update.  As you said, removing the management server tick fixes the problem.

Have you tried removing the definitions from the SEPM server and re-downloading them?

https://www-secure.symantec.com/connect/articles/how-clear-corrupt-virus-definitions-sepm

There might be an issue where a definition is corrupted and causing the machines to freak out.

Hi.

Yes - I've tried clearing the little blighters out.... there was no joy with that.

We also have more than 1 management server and it's happening no matter which one they connect to

I've got a beta version of the latest client - I'm tempted to install that on a couple of the machines and see if it's resolved

Well boys, I'm testing out Kaspersky AV as nothing I've tried with Symantec lately has helped.  We have been experiencing lots of program hangs and Not Responding messages with all (150) our Revit desktops since updating Symantec to the latest version.  These are Core i7 16gb RAM mac daddy workstations.

We no longer have time to be hunting down the various Symantec-related problems.

Our renewal for Symantec happens in December, and by then we will be using a different product.  SEP/SAV has just gone downhill in the last 5 years and there's no end to this kind of crap in sight. 

Good luck.

Hello,

I would request you to please PM me the Symantec Support Case #, if any.

Incase, if you haven't created any Symantec Support Case, I would recommend you to Create a Case with Symantec Technical Support online.

How to create a new case in MySupport

http://www.symantec.com/business/support/index?page=content&id=TECH58873

OR

Call Symantec Technical Support 

Phone numbers to contact Tech Support:-

Regional Support Telephone Numbers:

United States: https://support.broadcom.com (407-357-7600 from outside the United States)

Australia: 1300 365510 (+61 2 8220 7111 from outside Australia)

United Kingdom: +44 (0) 870 606 6000

BREAKING NEWS

or at least I think - I've just made a discovery. The effected machines all have something in common that all our other machines don't  - they all have solid state hard disks.

Which isn't the norm for us. This is news to me I've only just accidentally found out

So - all Lenovo T-410's. All running Win XP..... all have solid state disks.

Just going to feed this back to Symantec now.