Video Screencast Help

Rolling Upgrade

Created: 11 Aug 2009 • Updated: 08 Sep 2010 | 5 comments
WizzardS10's picture
4 Agree
1 Disagree
+3 5 Votes
Login to vote
Status: On Roadmap

It is often difficult to schedule outages on clusters with more than 1 active nodes.  This is a High availability environment after all.

Please provide an upgrade path to allow upgrading a single node at a time.  Granted all of the new features may not be available until after all of the nodes have been upgraded.

Comments 5 CommentsJump to latest comment

Tomer Gurantz's picture

VCS has been built with the ability to stop the cluser without shutting down applications (via "hastop -force" type commands), which effectively means you can upgrade any VCS component without any application downtime whatsoever. VCS itself cannot be upgraded via a 'rolling upgrade', but any application that supports such ability is handled by VCS.

I cannot see any fault with the way in which VCS is currently upgraded, as a rolling upgrade for VCS components themselves (like LLT, GAB, Fencing, and HAD, etc.) would cause additional complexity of inter-version supportability, and: 1. isn't worth the effot, 2. would complicate VCS code and introduce additional bugs (which is the one thing you never want to have happen in your clustering product).

For this reaon I "disagree" with this enhancement. 

Senior Principal Technical Education Consultant with Symantec Corporation

Login to vote
Jay Kim's picture

Well, technically speaking, rolling upgrade is possible but more steps are needed.
Obviously, a reboot is required after the upgrade of SFW so there will be a downtime.
But at least it is possible to reduce the amount of downtime by the method below. 
P.S. I'm only talking about SFW HA / VCS for Windows here as that's what I know.

eg. 2 node cluster = 1 Active node. 1 Passive node.

On passive node:
1. Stop HAD / LLT service. ("hastop -local -force" then "net stop llt")
2. Upgrade SFWHA / VCS but don't reboot after.
3. Set HAD service to manual.
4. Reboot.
5. Stop LLT (net stop llt)

On active node:
1. Offline the APP service groups.
2. Stop HAD / LLT service.

On Passive node
1. Start LLT and HAD (need to manually seed gab via "gabconfig -cx")
2. Online the APP service groups.
3. Set HAD service back to Automatic.

On active node:
1. Upgrade SFW HA / VCS but don't reboot after.
2. Set HAD service to automatic.
3. Reboot.


As you can see, my method above is basically isolating each node of any VCS communication during the upgrade. I've excluded setting LLT/GAB services to manual but it can also be done for being even more sure of reducing any possibility of conflicts during version mismatch between nodes. (This requires using the "sc" command)  I didn't go too much into detail of each command but hope it's easy enough to see what is possible.

Personally, I upgrade one by one just incase something does indeed go wrong during or after the install / upgrade. (rather have 1 node down after a bad upgrade instead of having all nodes down) But it is always best to schedule a full downtime whenever possible as you never know what may happen in any upgrade scenario. So it's best to "know" when it'll be down instead of causing any unexpected downtime.

At the end, rollup upgrade is possible and isn't really limited. It's just that we don't generally recommend it due to possible conflicts and/or issues that may arise between upgraded and non-upgraded machines if they do try and communicate.

Please mark my post as solution to help others also if it resolved your issue.

Login to vote
Darren James's picture

Be careful upgrading one VCS node at a time. This is not a supported method of doing upgrade officially.

The possible issue is different versions of gab communicating over LLT between upgraded/non upgraded nodes, which could potentially lead to cluster interconnect/heartbeat issues.

Login to vote
Kimberley's picture

From the product team: This is coming in the next release for UNIX. We'll update status for Windows, as the platforms are on different development schedules.

Thanks for participating in the community!

Login to vote
Lisa Erickson's picture

Considering for next major release

Login to vote