Endpoint Protection

Bad upgrade issues moving from 11.0.7000.975 to SEM 12.1 RU1 12.1.1000.157

Issues number one is that it won't upgrade.
I've assigned packages until exhausted, for over a week now, I've removed, added, deleted, added, reconfigured, changed, edited, and to no avail.
Before we get to assuming it's basics - I've dealt with the product for several years and knew 11.xx almost inside and out and pull some pretty neat tricks with it. I figured that if I did all the prerequesits, like disabling tamper protection, disabling registry protection, removing passwords, etc. etc. and gave it a few DAYS to take effect, even then manually pusing new settings, it should work, right? I created a sub-group under the main group, made sure the settings were safe for upgrade, assigned a package to that sub-group, then moved clients into that group. I set the package deployment so the times were the same, meaing instant push, and a week later, nothing. Not a single client is getting upgraded rto 12.1 from 11.07

I did this for several groups, several computer types, even a couple of servers in thier own group. They simply will not upgade to 12.1 These are all Windows 7 32bit. The auto-deploy using packages assigned to groups has always worked great in the past. At most I might have a 5% or less fail rate. Out of 350, I might have to do a dozen manually. In this case, it's a 100% fail rate.

Issue number 2
I thought, ok, I can cry UNCLE and do it the old-fashioned way - PUSH using the drop down from the home page of the console.... so I did that. OUCH, it broke each computer I did that to. Not only did they not properly upgrade, not SEP 11.07 isn't working either. In fact, here's a good one-  now ALL Symantec services on each of those computers is set to MANUAL. The endpoint protection service I was able to remotely set it to automatic and launch it, but can't do that with anything else. So now we have machines that won't communicate with the servers and the protection is broken, and it won't auto-upgrade and even a push really messes things up.

I've never ever had such a bad time upgrading any Symantec product. Moving from 11.07 to SEP 12.1 RU1 has been nothing but disaster and headache. I'm so surprised, I'm almost in shock. I've used SEP for years, never any real problems with upgrades, updates, or rollouts, even auto-upgrade has always worked very well.

Any clues? I've done the due-dilligence, meaning searched the forums here for over an hour, and nothing even remotely related!

The SEPM servers are NOT over loaded with only about 170 clients each and 5 meg pipes. Besides, I'm only taking 4 or 5 clients at a time in the test groups that have the packages applied.

Recall I did have a disasterous install on one of the SEPM servers, but I blew that one away, built a new VM, and installed SEPM12.1 from scratch. It now works well.   However, I'm now stuck, totally unable to use any method to upgrade from 11.0.7000.975 to 12.1

Hi,

look in the Smcinst.log.  Smcinst.log contains information related to the “AutoUpgrade” feature in SEP. look for any errors that show up when Autoupgrade is failing.

Is that log to look at on the management servers or on the clients?

What is the path - all  of these clients are in far away places so I have to remote into the C$ to get to any logs so it would be great to know the exact path to navigate to ;-)

Siunce I can't push this out since it trashes the services in each and every case I've tried so far, I need to rely on auto-upgrade.

Yes, you've been a good contributor and resource here; so I would trust that you've done the basics.    Sorry I don't have specific solutions for you; but I can identify with your problems.  When dealing with SAV and SEP products I have always done an uninstall/reboot/install/reboot.  With 12.1 I tried letting SEPM do the install on a computer as a test; and it broke the computer -- services got set to manual and I could not start them or repair them.  And I could not get SEP 11x or 12.1 to work on it after that.  I finally reimaged the computer.

So far the uninstall/reboot/install/reboot has been working.  Eventually I will use SCCM to mass-deploy to the rest of the enterprise.

Ah, so they do have an "undocumented bug". Interesting. I wondered as I keep up with things so well on our systems, and they've been so trouble-free for so long, I knew there was nothing at all wrong with the workstations or the SEP 11 installs - they are perfect, and the management and consoles are fresh installs on one server, a successful upgrade on the other, and the management piece is working "as designed" ;-)

Unfortunately, we don't have the ability to do the uninstall, reboot, install, reboot sequence. The reasons are that our 383 (at last count in the console) computers are scattered all over the state in nearly 40 offices. We do not have staff in those offices with any rights. They can't even do an IP address refresh or install a new screensaver let alone uninstall or install SEP. We also don't have much in the line of IT staff here being government on a budget where things are so tight, we've even stopped ordering pencils.

That leaves us up a really big creek. We have SCCM, but the person who handles that isn't an expert and can't do any installs unless it's a package supplied ready to go as an MSI. So even using SCCM is out. Even Microsoft app and OS updates are a fun event.

Worse, with us having deployed BIT-LOCKER, this means if SCCM DID uninstall and reboot - the computer would be left hanging at the bitlocker login screen - and when the user returned to unlock the computer, it would be totally 150% unprotected. That means it's literally impossible for us to push out a fix for this via SCCM or any other remote facility as the computers after the first reboot would simply sit at the bitlocker screen until or unless someone went to that office.

WOW, this is a disaster I'm NOT looking forward to informing our boss about. Once I do, maintenance will need to come do some ceiling repair when he's done going through the roof.

Unless I get some great tips and advice and a solution via this post today, looks like we'll be opening a case and running this one up the pole as there's absolutely no excuse for sending out an app upgrade that breaks every computer it would touch and force such a sequence of events as a hands-on uninstall and install on over 300 computers.

How about anyone else with this same issue letting me know of their excitement as well?

I'm almost in shock as I've never had such an experience with Symantec software. It just has never happened to me in 20 years - for the first time, I may be almost at a loss for words! ;-)

On a client that won't "autoupgrade", check the following registry key:

HKEY_LOCAL_MACHINE\SOFTWARE\Symantec\Symantec Endpoint Protection\SMC\Language

Is it blank?

See the following KB doc:

http://www.symantec.com/docs/TECH153398

MJD

Thanks for the tip - will check......... however, the issue is "apparently" with every computer we have.... none will auto-upgrade. I say apparently as every computer I've tried, and it's been a few now, none will auto-upgrade.

I'm concerned, however, that if they DID - would they also break because of what the push upgrade does to them all?

As far as this earlier tip:
>>

Hi,

look in the Smcinst.log.  Smcinst.log contains information related to the “AutoUpgrade” feature in SEP. look for any errors that show up when Autoupgrade is failing.<<

There is no such log on any computers I've looked at - servers or desktop PCs. I even did a search of the C: drives of these computers for *.log and find nothing other than the daily events of SEP - defs updates, and so on. There is no smcinst.log file existing on the C: drive of our computers. Why, I do not know.

I checked - so far the ones I see it's "English"

What's REALLY REALLY weird is that today now one of the machines that totally broke when I tried to do the push after the auto-upgrade failed - it's up and running and reporting in, and all the services are set to automatic, and started! What the heck? Auto-heal?? Whew! 

Before you read this registry key export, please keep in mind that these computers have been rebooted multiple times. So this message is in error............

This is a registry key export from one of the machines that will not auto-upgrade, below that is an export from another machine - same thing apparently.

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SOFTWARE\Symantec\Symantec Endpoint Protection\SMC\SYLINK\SyLink\OpState]
"DeployStatus"=dword:12075004
"DeployMessage"="Symantec Endpoint Protection has detected that there are pending system changes that require a reboot.  Please reboot the system and rerun the installation.
"
"DeployPreviousVersion"="11.0.7000.975"
"DeployTargetVersion"="12.1.1000.157"
"DeployRunningVersion"="11.0.7000.975"
"RebootReason"=dword:00000000
"RebootRequestor"=""

-----------------------------------------------------------------------------------

Key Name:          HKEY_LOCAL_MACHINE\SOFTWARE\Symantec\Symantec Endpoint Protection\SMC\SYLINK\SyLink\OpState
Class Name:        <NO CLASS>
Last Write Time:   1/12/2012 - 9:12 AM
Value 0
  Name:            DeployStatus
  Type:            REG_DWORD
  Data:            0x12075004

Value 1
  Name:            DeployMessage
  Type:            REG_SZ
  Data:            Symantec Endpoint Protection has detected that there are pending system changes that require a reboot.  Please reboot the system and rerun the installation.

Value 2
  Name:            DeployPreviousVersion
  Type:            REG_SZ
  Data:            11.0.7000.975

Value 3
  Name:            DeployTargetVersion
  Type:            REG_SZ
  Data:            12.1.1000.157

Value 4
  Name:            DeployRunningVersion
  Type:            REG_SZ
  Data:            11.0.7000.975

Value 5
  Name:            RebootReason
  Type:            REG_DWORD
  Data:            0

Value 6
  Name:            RebootRequestor
  Type:            REG_SZ
  Data:           

Value 7
  Name:            OpStateData
  Type:            REG_SZ
  Data:            <?xml version="1.0" encoding="UTF-8" ?>
 

I feel your pain. I've been fighting with multiple issues I need to get resolved before I even push the clients.

Just curious, as I didn't see it mentioned; are the computers being fully rebooted after the upgrade?  I didn't catch on initially that a reboot had to happen for the new client to even *start* installing.  I was used to the previous upgrades where all the pieces installed prior to rebooting.

In the past the auto-upgrade had simply worked, no reboot needed really........ the install started cleanly, and finished. If a reboot was needed, it would STILL show the new version but would state "reboot needed".

In this case, the machines have all been rebooted, some many times since I started this "test". You asked if they were rebooted after the upgrade? They never upgraded.

The ones I pushed SEP 12 to WERE rebooted, again, sometimes more than one time, but they never showed they had the new version, and further, the services were all set to manual, SEP was dead, and I could not do anything about it.

You may want to consider this a blessing in disguise. We have had more problems since upgrading clients to 12RU1 than we can count. I'm in the same boat, we've used this product on 10,000+ clients for years and have a good handle on it. Going from v11 to v12 has been a nightmare in every way imaginable. I have no solutions for you but I do feel your pain.

I think the log file kimberlyw was referring to is the SEPINST.LOG file. You should find it in C:\Windows\Temp, unless you installed manually when logged on as someone. Then it will be located in their temp directory.

Went to a few Security user group meetings where they said that the upgrade to v12 would be a "parallel" upgrade. IOW, v12.1 would install and then a migration would take place by uninstalling v11. My migrations never completed. Still trying to track down why. There don't seem to be any logs for the migration.

In my case, I thought it was related to VMs since none of my physical PCs are having the issue. So far, I've been successful in running an older CleanWipe and then manually removing the rest. I then install SEP 11. I've tried SEP 12.1 RU1 again, but it just fails again.

I'm thinking I'll be staying at v11 for some time yet.