procedure and aprox estimated down time regarding single node cluster/two node cluster
Expected environment
OS = Solaris 10
HA = 6.0
Cluster Node = SIngle node cluster
Queries
1.) We have a plan to go with single node cluster and after few days we will add one node with single node cluster.
Any step which we can follow before going with single node cluster which can keep my service group up/No downtime when I add the new node with single node cluster after few days. As per the below TN we must need downtime when adding a node with single node cluster.
https://sort.symantec.com/public/documents/vcs/5.1...
2.) Any suggested TN instead of above ? I feel confuse with the step no#7 under below document. (I cant find the files in single node cluster then how can I move it)
https://sort.symantec.com/public/documents/vcs/5.1...
3.) Suppose on a single node cluster we run the cluster configure wizard and create a new cluster and add both nodes. This will make a new two node cluster. Just we will lose our service group(which only have four resouces). But this will make a two node cluster quick if we compare this point/way with the first link. See also below the link for reference:
https://sort.symantec.com/public/documents/vcs/5.1...
4.) If we have two node cluster running fine. We want to remove one node(which has old hardware) from it and want to add another node(with advance hardware) in cluster. Do we need any down time ? (If any TN please share)
Comments 27 Comments • Jump to latest comment
Did you look at following link for adding a node to VCS cluster?
https://sort.symantec.com/public/documents/vcs/6.0/solaris/productguides/html/vcs_install/ch33.htm
To use the above link, You should have install node-A using following information.
https://sort.symantec.com/public/documents/vcs/6.0/solaris/productguides/html/vcs_install/apes02s01.htm
https://sort.symantec.com/public/documents/vcs/6.0/solaris/productguides/html/vcs_install/apes02s02.htm
The node-B is also set-up using the same instructions, except that you may not start VCS on node-B.
Beyond that you will use the information given in first link above.
and following for removing a node:
https://sort.symantec.com/public/documents/vcs/6.0/solaris/productguides/html/vcs_install/ch33s05.htm
Thanks and Warm Regards,
Amit Rangari
If this post helped you resolving the issue, please mark it as solution. _____________________________________________________________________________
https://sort.symantec.com/public/documents/vcs/6.0...
As per the above steps , can we run the below command on a single node cluster ?
Any comment will be appreciated. Mark as Solution if your query is resolved
__________________
Thanks in Advance
Zahid Haseeb
zahidhaseeb.wordpress.com
the above assumes that you have a single-node cluster running configured as "single node cluster to join a larger cluster".
if you have configured single-node cluster "to be a stand-alone single node cluster", then you will have to follow the instructions given at following link.
(pasting 6.0 solaris link instead of 5.1sp1 link provided in first post)
https://sort.symantec.com/public/documents/vcs/6.0/solaris/productguides/html/vcs_install/ch32.htm
this may have the downtime.
The suggestion i gave are assuming that you going to install fresh VCS 6.0 node and hence you have option to install single-node cluster to join a larger cluster.
i hope the difference between the following two is understood -
https://sort.symantec.com/public/documents/vcs/6.0/solaris/productguides/html/vcs_install/apes02s01.htm
[snip]
You can use the installer program to install a cluster on a single system for either of the two following purposes:
To prepare the single node cluster to join a larger cluster
To prepare the single node cluster to be a stand-alone single node cluster
[/snip]
I think customer is in first situation.
Thanks and Warm Regards,
Amit Rangari
If this post helped you resolving the issue, please mark it as solution. _____________________________________________________________________________
First thanks arangari for your kind replies.
Got it. What I asume is below:
To prepare the single node cluster to join a larger cluster
The above term means suppose there is a two/three or more node cluster configured and service group are online on it .. and we install vcs on a new machine and add this new machine to a two/three or more node cluster.
To prepare the single node cluster to be a stand-alone single node cluster
The above term means we installed vcs on a single machine and configured this machine as a single node cluster and service group is online on it... (In this phase if I want to make this single node cluster to a two nodes cluster then we must need down time as per the below document)
https://sort.symantec.com/public/documents/vcs/6.0...
=====================================
Any comment will be appreciated. Mark as Solution if your query is resolved
__________________
Thanks in Advance
Zahid Haseeb
zahidhaseeb.wordpress.com
Zahid,
your understanding is correct. however the larger cluster could also have just a single node to start with.
also the groups should be present on original cluster and the new node should get added into the SystemList of the SGs after new node joins.
Thanks and Warm Regards,
Amit Rangari
If this post helped you resolving the issue, please mark it as solution. _____________________________________________________________________________
You made my both point same amit :( .. but still I am very thankful of yours to repling on my queries.
Let me make very simple and to the point:
( Right now there is no cluster implemented just planning on the behalf of SYMANTEC CONNECT comments which I always feel very informative )
a,.) In a single node cluster can we join/add a new node to make this single node cluster in to two node cluster without any down time ?
b.) In a two node cluster (whose HB are connected back to back and no switch is installed in between two node cluster so we cant add a third node).... can we remove passive node(who hardware is actually old) and add a new node( whose hardware is as same as active node which we get after few weeks ) in place of passive node without any down time ?
Any comment will be appreciated. Mark as Solution if your query is resolved
__________________
Thanks in Advance
Zahid Haseeb
zahidhaseeb.wordpress.com
Zahid:
a: Yes. There is no downtime needed for node to be joined. There may be a down-time for joining node. If the new node is fresh, then i do not see any issue there either. This is all assuming that you have made sure that node to be joined was created for joining into larger cluster. Essentially, it is already using LLT and GAB, and hence extending should be no issue.
b. lets call HB as LLT links. If the LLT links are connected without any switch, back-to-back between two nodes, then one can shut-down the node to be replaced. This will make sure that (based on configuration), the service groups running will be evacuated to the other node. once the LLT links are connected, one can run 'installvcs -addnode'. Please note that the original node name may still be present in /etc/llthosts and in VCS configuration, which need to be removed by editing the file, and deleteing system (hasys -delete).
Let me know if you get into any issues here.
Thanks and Warm Regards,
Amit Rangari
If this post helped you resolving the issue, please mark it as solution. _____________________________________________________________________________
point a
- There may be a down-time for joining node. If the new node is fresh,
Could nt understand what are you saying
- I means that single node cluster means there is a cluster which has only one node. Then how you assume that LLT and GAB is running in a cluster which has only one node. ( Can I install a single node cluster with LLT and GAB ?) Essentially, it is already using LLT and GAB, and hence extending should be no issue.
Any comment will be appreciated. Mark as Solution if your query is resolved
__________________
Thanks in Advance
Zahid Haseeb
zahidhaseeb.wordpress.com
Yes- you could start cluster with one node with GAB and LLT.
Just try following:
Set llttab normally
Set llthosts with only one node
Gabtab will have
'gabconfig -c -n1'
Start VCS without -onenode
Thanks and Warm Regards,
Amit Rangari
If this post helped you resolving the issue, please mark it as solution. _____________________________________________________________________________
Zahid, i will attempt responding to you by monday - with proper illustrations. :)
Thanks and Warm Regards,
Amit Rangari
If this post helped you resolving the issue, please mark it as solution. _____________________________________________________________________________
Creating a two node cluster in two steps:
Configure node-A:
Follow instructions in https://sort.symantec.com/public/documents/vcs/6.0/solaris/productguides/html/vcs_install/apes02s02.htm
When you select "Answer y if you plan to incorporate the single node cluster into a multi-node cluster in the future", the installer would have installed LLT/GAB and also configured the same, to be used as one node.
When the configuration is complete and installer starts the services, LLT, GAB and VCS is started.
Confirm that 'port-a' and 'port-h' is showing up in output of the command '/sbin/gabconfig -a' .
You can configure service groups under this node-A and bring them online.
Add node-B to form two node cluster:
When ready to add node-B into the cluster with node-A, install the same version of VCS as present on node-A, on node-B. Do not configure node-B.
Please confirm the following:
Follow instruction at
https://sort.symantec.com/public/documents/vcs/6.0/solaris/productguides/html/vcs_install/ch33s02.htm
to add node-B to create 2-node cluster with node-A.
Replacing node with old hardware:
There is downtime for service groups online on node-A. There is no downtime for service groups online on node-B.
Remove cross-cables from node-A and put them into new node.
Install VCS on new node and follow the instructions from section "Add node-B to form two node cluster" for new node.
Modify the service groups to include the new node as possible failover system if required.
Thanks and Warm Regards,
Amit Rangari
If this post helped you resolving the issue, please mark it as solution. _____________________________________________________________________________
Thanks arangari for your kind words. Let me do this at my side and will share the result. Again bundle of thanks for your kind interest on my queries.
Any comment will be appreciated. Mark as Solution if your query is resolved
__________________
Thanks in Advance
Zahid Haseeb
zahidhaseeb.wordpress.com
Zahid - let me know if you tried the procedure I suggested? if so, does it resolve the issue you were facing?
Thanks and Warm Regards,
Amit Rangari
If this post helped you resolving the issue, please mark it as solution. _____________________________________________________________________________
Yes sure. I will definately share the result and thanks :)
Any comment will be appreciated. Mark as Solution if your query is resolved
__________________
Thanks in Advance
Zahid Haseeb
zahidhaseeb.wordpress.com
One more thing I would like to ask. When I install HA on single system with GAB and LLT. What should I need to verify (to cross check this single node) that in future I will be able to add another node without any down time/problem.
Any comment will be appreciated. Mark as Solution if your query is resolved
__________________
Thanks in Advance
Zahid Haseeb
zahidhaseeb.wordpress.com
Just check llttab and gabtab files exist and LLT is running (lltstat -nvv). If llttab does not exist and llt is not running, then this means VCS is started using "hastart -onenode" and you need to configure LLT and GAB before adding nodes. Having to use "hastart -onenode" is another indicator - i.e if you try to start VCS on a one-node cluster where LLT and GAB are not configured using "hastart" then it won't work as you have to use "hastart -onenode".
Mike
UK Symantec Consultant in VCS, GCO, SF, VVR, VxAT on Solaris, AIX, HP-ux, Linux & Windows
If this post has helped you, please vote or mark as solution
Thanks Mike. Here's the output I can see
bash-3.00# cat gabtab
/sbin/gabconfig -c -n1
bash-3.00# cat llttab
set-node Solaris2
set-cluster 60964
link qfe0 /dev/qfe:0 - ether - -
link qfe1 /dev/qfe:1 - ether - -
link-lowpri aggr1 /dev/aggr:1 - ether - -
bash-3.00# lltstat -nvv
LLT node information:
Node State Link Status Address
* 0 Solaris2 OPEN
qfe0 UP 08:00:20:C7:6D:18
qfe1 UP 08:00:20:C7:6D:19
aggr1 UP 00:00:00:00:00:00
1 CONNWAIT
qfe0 DOWN
qfe1 DOWN
aggr1 DOWN
2 CONNWAIT
qfe0 DOWN
qfe1 DOWN
aggr1 DOWN
3 CONNWAIT
qfe0 DOWN
qfe1 DOWN
aggr1 DOWN
4 CONNWAIT
qfe0 DOWN
qfe1 DOWN
aggr1 DOWN
5 CONNWAIT
qfe0 DOWN
qfe1 DOWN
aggr1 DOWN
6 CONNWAIT
qfe0 DOWN
qfe1 DOWN
aggr1 DOWN
7 CONNWAIT
qfe0 DOWN
qfe1 DOWN
aggr1 DOWN
8 CONNWAIT
qfe0 DOWN
qfe1 DOWN
aggr1 DOWN
9 CONNWAIT
qfe0 DOWN
qfe1 DOWN
aggr1 DOWN
10 CONNWAIT
qfe0 DOWN
qfe1 DOWN
aggr1 DOWN
11 CONNWAIT
qfe0 DOWN
qfe1 DOWN
aggr1 DOWN
12 CONNWAIT
qfe0 DOWN
qfe1 DOWN
aggr1 DOWN
13 CONNWAIT
qfe0 DOWN
qfe1 DOWN
aggr1 DOWN
14 CONNWAIT
qfe0 DOWN
qfe1 DOWN
aggr1 DOWN
15 CONNWAIT
qfe0 DOWN
qfe1 DOWN
aggr1 DOWN
16 CONNWAIT
qfe0 DOWN
qfe1 DOWN
aggr1 DOWN
17 CONNWAIT
qfe0 DOWN
qfe1 DOWN
aggr1 DOWN
18 CONNWAIT
qfe0 DOWN
qfe1 DOWN
aggr1 DOWN
19 CONNWAIT
qfe0 DOWN
qfe1 DOWN
aggr1 DOWN
20 CONNWAIT
qfe0 DOWN
qfe1 DOWN
aggr1 DOWN
21 CONNWAIT
qfe0 DOWN
qfe1 DOWN
aggr1 DOWN
22 CONNWAIT
qfe0 DOWN
qfe1 DOWN
aggr1 DOWN
23 CONNWAIT
qfe0 DOWN
qfe1 DOWN
aggr1 DOWN
24 CONNWAIT
qfe0 DOWN
qfe1 DOWN
aggr1 DOWN
25 CONNWAIT
qfe0 DOWN
qfe1 DOWN
aggr1 DOWN
26 CONNWAIT
qfe0 DOWN
qfe1 DOWN
aggr1 DOWN
27 CONNWAIT
qfe0 DOWN
qfe1 DOWN
aggr1 DOWN
28 CONNWAIT
qfe0 DOWN
qfe1 DOWN
aggr1 DOWN
29 CONNWAIT
qfe0 DOWN
qfe1 DOWN
aggr1 DOWN
30 CONNWAIT
qfe0 DOWN
qfe1 DOWN
aggr1 DOWN
31 CONNWAIT
qfe0 DOWN
qfe1 DOWN
aggr1 DOWN
32 CONNWAIT
qfe0 DOWN
qfe1 DOWN
aggr1 DOWN
33 CONNWAIT
qfe0 DOWN
qfe1 DOWN
aggr1 DOWN
34 CONNWAIT
qfe0 DOWN
qfe1 DOWN
aggr1 DOWN
35 CONNWAIT
qfe0 DOWN
qfe1 DOWN
aggr1 DOWN
36 CONNWAIT
qfe0 DOWN
qfe1 DOWN
aggr1 DOWN
37 CONNWAIT
qfe0 DOWN
qfe1 DOWN
aggr1 DOWN
38 CONNWAIT
qfe0 DOWN
qfe1 DOWN
aggr1 DOWN
39 CONNWAIT
qfe0 DOWN
qfe1 DOWN
aggr1 DOWN
40 CONNWAIT
qfe0 DOWN
qfe1 DOWN
aggr1 DOWN
41 CONNWAIT
qfe0 DOWN
qfe1 DOWN
aggr1 DOWN
42 CONNWAIT
qfe0 DOWN
qfe1 DOWN
aggr1 DOWN
43 CONNWAIT
qfe0 DOWN
qfe1 DOWN
aggr1 DOWN
44 CONNWAIT
qfe0 DOWN
qfe1 DOWN
aggr1 DOWN
45 CONNWAIT
qfe0 DOWN
qfe1 DOWN
aggr1 DOWN
46 CONNWAIT
qfe0 DOWN
qfe1 DOWN
aggr1 DOWN
47 CONNWAIT
qfe0 DOWN
qfe1 DOWN
aggr1 DOWN
48 CONNWAIT
qfe0 DOWN
qfe1 DOWN
aggr1 DOWN
49 CONNWAIT
qfe0 DOWN
qfe1 DOWN
aggr1 DOWN
50 CONNWAIT
qfe0 DOWN
qfe1 DOWN
aggr1 DOWN
51 CONNWAIT
qfe0 DOWN
qfe1 DOWN
aggr1 DOWN
52 CONNWAIT
qfe0 DOWN
qfe1 DOWN
aggr1 DOWN
53 CONNWAIT
qfe0 DOWN
qfe1 DOWN
aggr1 DOWN
54 CONNWAIT
qfe0 DOWN
qfe1 DOWN
aggr1 DOWN
55 CONNWAIT
qfe0 DOWN
qfe1 DOWN
aggr1 DOWN
56 CONNWAIT
qfe0 DOWN
qfe1 DOWN
aggr1 DOWN
57 CONNWAIT
qfe0 DOWN
qfe1 DOWN
aggr1 DOWN
58 CONNWAIT
qfe0 DOWN
qfe1 DOWN
aggr1 DOWN
59 CONNWAIT
qfe0 DOWN
qfe1 DOWN
aggr1 DOWN
60 CONNWAIT
qfe0 DOWN
qfe1 DOWN
aggr1 DOWN
61 CONNWAIT
qfe0 DOWN
qfe1 DOWN
aggr1 DOWN
62 CONNWAIT
qfe0 DOWN
qfe1 DOWN
aggr1 DOWN
63 CONNWAIT
qfe0 DOWN
qfe1 DOWN
aggr1 DOWN
bash-3.00#
Any comment will be appreciated. Mark as Solution if your query is resolved
__________________
Thanks in Advance
Zahid Haseeb
zahidhaseeb.wordpress.com
So this meets all criteria - llttab and gabtab exist and llt is running, so you can add node without having to stop llt and gab. Note there is another criteria that you don't have an "exclude" line in your llttab file - myself (as a Symantec consultant) and several colleages of mine use to add a line "exclude 2-31" to llttab for 2-node clusters which means the lltstat -nvv output is much shorter as just shows 2 nodes (0 and 1) and llt is slightly more efficient as it is not always checking for other nodes being added. I subsequently changed this "best practice" to use exclude with 2 spare nodes so for a 2-node cluster I would use "exclude 3-63" in llttab so that 2 nodes could be added in the future without stopping gab and llt.
But even if you have to stop llt and gab, this does not mean an outage unless you are using CVM/CFS as you can force stop VCS and leave applications up, while you restart LLT and GAB with a new config.
Mike
UK Symantec Consultant in VCS, GCO, SF, VVR, VxAT on Solaris, AIX, HP-ux, Linux & Windows
If this post has helped you, please vote or mark as solution
As per the below suggested link by arangari
https://sort.symantec.com/public/documents/vcs/6.0...
Would you please highlight the below:-
Updates the following configuration files and copies them on the new node:
/etc/llthosts
/etc/gabtab
/etc/VRTSvcs/conf/config/main.cf
Any comment will be appreciated. Mark as Solution if your query is resolved
__________________
Thanks in Advance
Zahid Haseeb
zahidhaseeb.wordpress.com
As per the below TN following three things are required:
=======================================================
Updates the following configuration files and copies them on the new node:
/etc/llthosts
/etc/gabtab
/etc/VRTSvcs/conf/config/main.cf
https://sort.symantec.com/public/documents/vcs/6.0/solaris/productguides/html/vcs_install/ch33s02.htm
=======================================================
/etc/llthosts
existing cluster node output:
# cat /etc/llthosts
0 Solaris2
step #1
Add another node on the existing file on existing cluster node
# vi /etc/llthosts
0 Solaris2
1 Solaris1
step #2
copy the /etc/llthosts file from existing cluster node to new node
===========
/etc/gabtab
existing cluster node output:
# cat /etc/gabtab
/sbin/gabconfig -c -n1
step #1
Replace the entry n1 with n2 as the cluster node will be two
vi /etc/gabtab
/sbin/gabconfig -c -n2
step #2
copy the /etc/gabtab file from existing cluster node to new node
==========
/etc/VRTSvcs/conf/config/main.cf
What should I update on the existing cluster node ?
Any comment will be appreciated. Mark as Solution if your query is resolved
__________________
Thanks in Advance
Zahid Haseeb
zahidhaseeb.wordpress.com
Zahid:
please note that the TN indicated is stating what installer is doing:
Adding nodes using the VCS installer
The VCS installer performs the following tasks: <<----
Verifies that the node and the existing cluster meet communication requirements.
Verifies the products and packages installed on the new node.
Discovers the network interfaces on the new node and checks the interface settings.
Creates the following files on the new node:
please follow the below instructions as suggested in the technote
To add the node to an existing VCS cluster using the VCS installer
Thanks and Warm Regards,
Amit Rangari
If this post helped you resolving the issue, please mark it as solution. _____________________________________________________________________________
Ahh. Its nothing to do just run the addnode command. Very simple
Thanks arangari for your kind support :)
Any comment will be appreciated. Mark as Solution if your query is resolved
__________________
Thanks in Advance
Zahid Haseeb
zahidhaseeb.wordpress.com
While setting up first node in cluster(cluster with single node). I am seeing something which I would like to ask. See the below snapshot for reference first:
==============================================================
1) Configure heartbeat links using LLT over Ethernet
2) Configure heartbeat links using LLT over UDP
b) Back to previous menu
How would you like to configure heartbeat links? [1-2,b,q,?] (1)
Discovering NICs on Solaris ................................................................................................ Discovered bge0 qfe0 qfe1 qfe2 qfe3
Enter the NIC for the first private heartbeat link on Solaris: [b,q,?] (bge0) qfe0
Would you like to configure a second private heartbeat link? [y,n,q,b,?] (y) qfe1
Invalid selection. Please re-enter
Would you like to configure a second private heartbeat link? [y,n,q,b,?] (y)
Enter the NIC for the second private heartbeat link on Solaris: [b,q,?] (bge0) qfe1
Would you like to configure a third private heartbeat link? [y,n,q,b,?] (n)
Do you want to configure an additional low-priority heartbeat link? [y,n,q,b,?] (n) y
Enter the NIC for the low-priority heartbeat link on Solaris: [b,q,?] bge0
Checking media speed for qfe0 on Solaris
.................................................................................................................. Down
Checking media speed for qfe1 on Solaris .................................................................................................................. Down
CPI WARNING V-9-40-2959 installer can not detect media speed for the selected private NICs properly. Consult your Operating System manual for information on how to set
the Media Speed.
Do you want to continue with current heartbeat configuration? [y,n,q] (y)
==============================================================
Is this any problem if I continue with
Checking media speed for qfe1 on Solaris Down
Checking media speed for qfeo on Solaris Down
Any comment will be appreciated. Mark as Solution if your query is resolved
__________________
Thanks in Advance
Zahid Haseeb
zahidhaseeb.wordpress.com
one can go ahead without the above check passing. There are normally no issues with this and LLT works without any trouble.
Thanks and Warm Regards,
Amit Rangari
If this post helped you resolving the issue, please mark it as solution. _____________________________________________________________________________
Successfully Done. But hagui did not show me the second node.
What I did:
From the System Manager I did add the Node which was showing me the Available Systems in System Manager, Then I am able to see it.
Thanks again for your kind input arangari.
Any comment will be appreciated. Mark as Solution if your query is resolved
__________________
Thanks in Advance
Zahid Haseeb
zahidhaseeb.wordpress.com
"installvcs -addnode" adds a node to the cluster - i.e does neccessary LLT and GAB stuff and runs "hasys -add". It is not meant to update service groups as service groups do not need to contain all the systems in the cluster so for example in a 3-node cluster you might have one service group containing sys1 & sys3 and another containing sys2 & sys3.
Using system manager is basically running "hagrp -modify SystemList (and hagrp -modify "AutoStartList") to allow the service group to fail to the new system.
Mike
UK Symantec Consultant in VCS, GCO, SF, VVR, VxAT on Solaris, AIX, HP-ux, Linux & Windows
If this post has helped you, please vote or mark as solution
Thanks Mike for kind input. I appriciate :)
Any comment will be appreciated. Mark as Solution if your query is resolved
__________________
Thanks in Advance
Zahid Haseeb
zahidhaseeb.wordpress.com
Would you like to reply?
Login or Register to post your comment.