Endpoint Protection

Many - not all. but it's a fair share - 1/3 at least. Forever may be described as 15 to 45 minutes.
Currently at SEP 12.1 RU1 - version 12.1.1000.157
Moving to SEP RU1 MP1 version 12.1.1101.401

This is how I almost ALWAYS do this - this is not new, been doing this since we started with SEP 11.xx

I created a group with relaxed policies - SEP security is turned off, tamper protection is disabled, passwords removed, etc. etc. etc.
I created install packages for SEP 32 and 64 bit as well as SNAC.
I made the installs silent, refresh logs and policies - full protection for clients.
I assigned the install packages to my special SEP upgrade/install group.
I set it "Download the client package from the management server"
I configured using "Upgrade schedule" - 17:00 to 7:00 over 3 days.
They get a notification it's happening.

The install settings for the package are:
silent, install to default folder, enable logging, submit reputation information, add to start menu, remove all previous logs and policies and reset communications settings. (did that last one as a couple of clients this summer had certificate mis-matches and I had to manually import communications settings to get them fixed)

The Schedule Reboot tab is:
Custom restart, at a scheduled time/day, hard restart, and restart immediately if user not logged in.

So far, this part seems to be ok..... The PROBLEM is that when this happens, and I was able to witness this finally on a computer here today, the user says, yeah, sure, go ahead, then a couple minutes later is prompted to reboot, or snooze the reboot, she chose to reboot - and it took 20 minutes before she had her desktop back! It sat at the starting Windows screen for a good 20 minutes, NO KIDDING.
At 6:06am, auto-upgrade agent is installed.
At 6:06 am event ID 1040 MSIINSTALLER kicked off SEP MSI.
At 6:07 migration service started,
At 6:09 the auto-upgrade agent stopped.
At 6:14 ccSvcHost.exe initiated a restart of the computer - Legacy API shutdown.
6:10-6:15 I can see where services are stopping, and then later kernel messages and services are starting -
At 6:16 there's warning message about hpdskflt
Teefer2 loads, then the network drivers load at 6:17.
Then notihng until 6:25 when the Filter Manager says SymEFA has loaded then eeCtrl loads, then nothing again until 6:34
At 6:34, among other things, SRTSP fails to load.
At these times - about 6:15 to 6:35, the user can do nothing but watch the starting Windows screen.
At 6:37:52, SRTSP finally loads.

 there's a huge gap in the logs - all Windows logs - between about 6:17 and 6:34 save for the two lines at about 6:25 when the SymEFA and other thing loads.
The application log is empty between 6:09 and 6:15 and then again until 6:34 -
For all of us here in IT, and for the user, it appears as if SEP upgrades are taking 15 to 45 minutes to install, however, when looking at the logs, it appears more to be some bad interaction going on where things simply are sitting and not loading.

Is ANYONE ELSE seeing this same bad behavour??
Thoughts? Ideas? Fixes? Please.............. it's leaving a terrible sour taste in the mouths of management and users here.
This started with 12.1.
This did not happen with any 11 updates - they went fast and smooth. This happened last spring when I put 12.1 out to begin with.

 

Not sure what is going on with RU1 MP1 but it seems to be causing some issues. I've stayed away due to PCs not shutting down properly due to ccSvcHost.exe hanging. Had no issues with RU1.

What happens if you export a package and install locally (no push from SEPM)?

This is not specific to RU1 MP1.

This is just happening with it exactly as it happened when I rolled out RU1 last winter. I started that process in December 2011, wrapped it up in February - and yes, it took that long on 350 computers because it was a pure nightmare - disaster. Computers got stuck in between, or rolled back, we often had to visit the site, manually uninstall and clean it up, and reinstall the 12.1 RU1, others went "ok". But this is where I FIRST found this same issue - computers hanging at the "starting Windows" screen for 15 to 45 minutes (one user reported a FULL HOUR).
So I first saw it with 12.1 when we FIRST rolled it out.
I'm seeing it again - but this time, I'm armed and ready - and asking users to report directly to me the instant it happens so I can track it minute-by-minute and gather all the logs - SEP and Windows logs, for that time period.
This AM I was lucky - it was my own wife who called me and said "my computer is.........." and I knew the computer, she works just a couple offices away, and I started gathering logs as fast as I could get to the computer.

Interesting, too - during this time, the computer can't be pinged or remotely accessed. It seems only after the SRTSP loads does it then finally seem to ping and allow remote management access (remote registry, remove computer management, etc.)
 

After roughly 6:10am and before 6:37am today, I could not access it or ping it, and she was just able to get to her desktop at 6:36/6:37.
Otherwise, for all intents and purposes - the computer was "hung" for 20 minutes during this upgrade. Again, not new with RU1 MP1, it happened to us last winter with our very first 12.1 rollout. It's a 12.1 thing for us.

And same result if you run an upgrade package locally?

Have not tried that as most of our computers are a few miles away - from 30 minutes to 4 hours.
Hope to resolve as we REALLY REALLY need the next upgrade soon to come - RU2.

It will fix some important issues for us in other areas - hoping to resolve this now so that I can smoothly upgrade to that level later -  like I said - I had a great reputation with SEP 11 as the upgrades went so smooth, most never realized it was even happening, and there were NO complaints, and believe it or not - you'd heard that "10% rule" where if you push or upgrade or whatever, 10% will fail? My failures with SEP 11.xx in upgrades, etc. was less than 1%. It made me look like Scotty on Star Trek. This is making me look more like Stan Laurel........... 

I has issues in 11.x with the upgrade schedule, would generally bury our WAN so I quit using. For no more than 40 clients, it would take days on end to deploy. But everything was silent so I can't say whether or not the install took that long as our users would've never known.

Best course of action may be a support call...

Hi,

SEP 12.1 RU2 is released in the month of Nov'12.

New fixes and enhancements in Symantec Endpoint Protection 12.1 Release Update 2

http://www.symantec.com/docs/TECH199676

Hi Bill,

Can you please share the Install log of any one client..and are these machines good on RAM/CPU part ?

Wow, months later............ Too little, way too late.
The problem started December 2011, and went through first part of November 2012, nearly a full year of SEP upgrade #@$%. It's a bit late, now.

I've known about and had SEP RU2 in my hands for a couple of months, so waiting and replying with "RU2 is out and solves a lot of problems" months later isn't a good feeling.
RU2 isn't the point, and it came way late - well after we had all the problems for over a year. The problem should have been solved months ago. And not by RU2. It should have come way before that.
I hope most problems aren't answered this late in the game. It was so bad we came close to "no more SEP upgrades from now on".
The logs will be long since gone. There really isn't any use trying to troubleshoot it a year later.
The issue wasn't taking forever for the files to get TO the computers,
it was AFTER the files were copied to the computer that the problems came into play.
When the computer rebooted is when the nightmare on 12th Street started. Whatever SEP installs did after the files were copied into a parallel position to the prior version, then the computer rebooted it took in some cases 30 minutes to get control of the computer back. One poor person waited FORTY minutes. The only thing going on was nothing, literally - Windows was not writing to ANY logs during that time. SEP was done writing to logs. But whatever happens immediately after the computer reboots was nothing but trouble.  Further, many installs failed. Some "install failed, rolled back", some simply failed to update then lost all connection with the SEPMs. The only cure in each and every case was a full uninstall and reinstall. Basically, it started when we moved from 11. The very first SEP 12.xxx we installed as upgrades over 11 were nothing but trouble. It was a disaster.  It even caused some people to report computer screens going black at random times, then coming back. This would happen the time of the install after the reboot, and now and then during the day, then by the 2nd or 3rd day after the SEP upgrade, the computer would behave again. SEP even seemed to be killing video processes.
I still to this day find a computer now and then that for some reason appears, but SEP is broken. The management piece seems to ping the server so I get a current last check-in time, but that's it.

The nighmare is over thanfully - at least we THINK it is - I'm now in testing mode of RU2, not at all willing to run head-first into yet another SEP total nightmare, I'm taking it VERY slow, a computer here and there just to see if we get complaints again. Imagine rebooting your computer and having to wait 30 minutes, or the upgrade going out over night and you come in to log in and your computer takes a half hour to respond and allow you to work. That was the norm for us - with EVERY upgrade we made.  Basically SEP and I were put on a software upgrade diet - very limited upgrades or updates, only if absolutely needed, and only with full controls and tons of testing. Even then there were problems.

I'm shocked that for 3 months there wsn't a single reply  (no, not really, because frankly, no one even in Symantec knew what was going on and in the online community, when there's no answer, you get no reply), suddenly the answer is SEP RU2? No, SEP RU1 was broken, MP1 was broken, and everything in between was b roken as far as we're concerned - at least as far as the install process. So having to wait a full year until RU2 wasn't the answer. It is now as we've got no choice, but when we were in the middle of the problems, there was no answer at all.
........... SEP RU2 is old news. Logs will be long since gone, and I scoured them myself, inclding every single log Windows makes and found nothing other than SEP upgrade process was doing "something" mysterious after the reboot - not before, not during file copy. And during that time, no logs were written to. On ones that "install failed, rolling back" happened, that's pretty much all the install logs indicated. Failure, rolling back. No other clues.  For the ones that sat for 30 minutes, most of those were successful, thus no errors were ever entered in any logs. It's sort of like a trip to the store - and the store is only 1 mile away, but it took you 2 hours to get there. Did you have any breakdowns or problems? No, you simply crawled there and took your time, sat down now and then for a rest, then finally decided to sprint the last 10 feet.

Can't blame our computers - we install more heavy stuff than SEP - MS office, Office and Windows service packs, 300 meg of patches, etc. - those all go very well every single time. The one and only application we've ever hated the upgrade process on was SEP 12. When heavy Microsoft service packs and patches install more reliably, there's a problem.  But it seems over a year later they might finally have it right - after SEP earned a very bad reputation with us for their horrible upgrade processes.