Video Screencast Help
Symantec to Separate Into Two Focused, Industry-Leading Technology Companies. Learn more.

VCS Cluster not starting.

Created: 13 Jan 2014 • Updated: 16 Jan 2014 | 10 comments
This issue has been solved. See solution.

Hi

I am facing problem while trying to start VCS .

From LOG :

==============================================================

tail /var/VRTSvcs/log/engine_A.log

    2014/01/13 21:39:14 VCS NOTICE V-16-1-11050 VCS engine version=5.1
    2014/01/13 21:39:14 VCS NOTICE V-16-1-11051 VCS engine join version=5.1.00.0
    2014/01/13 21:39:14 VCS NOTICE V-16-1-11052 VCS engine pstamp=Veritas-5.1-10/06/09-14:37:00
    2014/01/13 21:39:14 VCS INFO V-16-1-10196 Cluster logger started
    2014/01/13 21:39:14 VCS NOTICE V-16-1-10114 Opening GAB library
    2014/01/13 21:39:14 VCS NOTICE V-16-1-10619 ‘HAD’ starting on: nsscls01
    2014/01/13 21:39:16 VCS INFO V-16-1-10125 GAB timeout set to 30000 ms
    2014/01/13 21:39:16 VCS NOTICE V-16-1-11057 GAB registration monitoring timeout set to 200000 ms
    2014/01/13 21:39:16 VCS NOTICE V-16-1-11059 GAB registration monitoring action set to log system message
    2014/01/13 21:39:31 VCS CRITICAL V-16-1-11306 Did not receive cluster membership, manual intervention may be needed for seeding

=============================================================================================

root@nsscls01# hastatus -sum
VCS ERROR V-16-1-10600 Cannot connect to VCS engine
VCS WARNING V-16-1-11046 Local system not available

 

Please advice how can I start the VCS.

Operating Systems:

Comments 10 CommentsJump to latest comment

Gaurav Sangamnerkar's picture

Hello,

can you post more details, how many node this cluster has ? is VCS already running on other nodes or this is the first node you are trying to start VCS ? what is operating system version ?

also paste below from all the nodes

# gabconfig -a

# cat /etc/gabtab

 

from the initial looks, it appears that GAB on this node is not able to communicate with GAB on other nodes & untill your GAB is successfully started you can't start HAD (VCS)

Below GAB sits LLT (heartbeat), you might want to look at LLT as well if all the heartbeats are working & connected, below output can give view on LLT status

# lltstat -vvn

 

G

PS: If you are happy with the answer provided, please mark the post as solution. You can do so by clicking link "Mark as Solution" below the answer provided.
 

Marianne's picture

 2014/01/13 21:39:31 VCS CRITICAL V-16-1-11306  Did not receive cluster membership, manual intervention may be needed for seeding

As per Gaurav's excellent post - you firstly need to check if LLT and GAB are running.

Cluster comms must first of all be established before VCS (had) will start.

Is this a new cluster? 
Or a cluster that has worked fine previously?

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

mikebounds's picture

As Gaurav and Marianne have said, there is an issue with LLT or GAB.

By default VCS will not start until all nodes are up and can communicate with each other via the heartbeats, so most likely is that another node is not up or there is problem with LLT communication so run "lltstat -nvv" to check that you have no nodes which have links in the DOWN state. It could also be that LLT is ok and GAB is not running on one of the nodes, but this is less likely as GAB is started by the O/S running "/etc/gabtab" file on boot up, (unless you have manaully disabled GAB startup)

Mike 

UK Symantec Consultant in VCS, GCO, SF, VVR, VxAT on Solaris, AIX, HP-ux, Linux & Windows

If this post has answered your question then please click on "Mark as solution" link below

Sun-Ray's picture

root@nsscls02#  hastatus -sum

– SYSTEM STATE
– System               State                Frozen

A  nsscls01              UNKNOWN              0
A  nsscls02              RUNNING              0

==============================================================
root@nsscls01# lltstat -nvv
LLT node information:
Node                 State    Link  Status  Address
* 0 nsscls01           OPEN
nxge1   UP      08:00:28:16:39:46
nxge2   UP      08:00:28:23:AA:8A
1 nsscls02           CONNWAIT
nxge1   DOWN
nxge2   DOWN

root@nsscls02# lltstat -nvv
LLT node information:
Node                 State    Link  Status  Address
0 nsscls01           CONNWAIT
nxge1   DOWN
nxge2   DOWN
* 1 nsscls02           OPEN
nxge1   UP      08:00:23:0E:29:74
nxge2   UP      08:00:23:15:CD:85

===============================================================
root@nsscls01# gabconfig -a
GAB Port Memberships
===============================================================

root@nsscls02# gabconfig -a

GAB Port Memberships
================================================================
Port a gen  1c77f01 membership ;1
Port b gen  1c77f04 membership ;1
Port h gen  1c77f03 membership ;1

In Gabtab :
/sbin/gabconfig -c -n2

 

Please advice.

Sun-Ray's picture

root@nsscls02#  hastatus -sum

– SYSTEM STATE
– System               State                Frozen

A  nsscls01              UNKNOWN              0
A  nsscls02              RUNNING              0

==============================================================
root@nsscls01# lltstat -nvv
LLT node information:
Node                 State    Link  Status  Address
* 0 nsscls01           OPEN
nxge1   UP      08:00:28:16:39:46
nxge2   UP      08:00:28:23:AA:8A
1 nsscls02           CONNWAIT
nxge1   DOWN
nxge2   DOWN

root@nsscls02# lltstat -nvv
LLT node information:
Node                 State    Link  Status  Address
0 nsscls01           CONNWAIT
nxge1   DOWN
nxge2   DOWN
* 1 nsscls02           OPEN
nxge1   UP      08:00:23:0E:29:74
nxge2   UP      08:00:23:15:CD:85

===============================================================
root@nsscls01# gabconfig -a
GAB Port Memberships
===============================================================

root@nsscls02# gabconfig -a

GAB Port Memberships
================================================================
Port a gen  1c77f01 membership ;1
Port b gen  1c77f04 membership ;1
Port h gen  1c77f03 membership ;1

In Gabtab :
/sbin/gabconfig -c -n2

Gaurav Sangamnerkar's picture

Hello,

Clearly the issue is at LLT layer ... as you can see from llstat output you have posted, node 1 says that node 2 LLT is down, & node 2 says that node 1 LLT is down .. couple of possibilities here

1. Either the physical connection itself has issues, you can use tools like dlpiping or lltping to determine the status of LLT links. These tools are helpful because LLT works at mac layer. Alternatively to test, you can plumb some IPs on both the sides & try test ping. for e.g plumb 1.1.1.1 on nxge1 on node 1 & 1.1.1.2 on nxge1 on node 2 & you can ping to confirm connectivity.

Link for dlpiping

http://sfdoccentral.symantec.com/sf/5.0MP3/aix/man...

 

2. If connectivity is found right, just try to start all the components manually on node 1

# /etc/init.d/llt start

# /etc/init.d/gab start

I have observed that sometimes LLT status is now shown correctly unless GAB is started correct. once these are started, check the "gabconfig -a" output again. If GAB starts & shows membership with other node, you will need to start IOFencing

# /etc/init.d/vxfen start

post this you would be able to execute "hastart" in order for VCS to start

 

G

PS: If you are happy with the answer provided, please mark the post as solution. You can do so by clicking link "Mark as Solution" below the answer provided.
 

SOLUTION
Sun-Ray's picture

Hi

I am not sure if anybody is able to see my posting. I have posted twice with the output but its not visible for me. So posting once more. If you are able to see please respond with resolution.

 

root@nsscls02#  hastatus -sum

– SYSTEM STATE
– System               State                Frozen

A  nsscls01              UNKNOWN              0
A  nsscls02              RUNNING              0

==============================================================
root@nsscls01# lltstat -nvv
LLT node information:
Node                 State    Link  Status  Address
* 0 nsscls01           OPEN
nxge1   UP      08:00:28:16:39:46
nxge2   UP      08:00:28:23:AA:8A
1 nsscls02           CONNWAIT
nxge1   DOWN
nxge2   DOWN

root@nsscls02# lltstat -nvv
LLT node information:
Node                 State    Link  Status  Address
0 nsscls01           CONNWAIT
nxge1   DOWN
nxge2   DOWN
* 1 nsscls02           OPEN
nxge1   UP      08:00:23:0E:29:74
nxge2   UP      08:00:23:15:CD:85

===============================================================
root@nsscls01# gabconfig -a
GAB Port Memberships
===============================================================

root@nsscls02# gabconfig -a

GAB Port Memberships
================================================================
Port a gen  1c77f01 membership ;1
Port b gen  1c77f04 membership ;1
Port h gen  1c77f03 membership ;1

In Gabtab :
/sbin/gabconfig -c -n2

Gaurav Sangamnerkar's picture

I am able to see your posts in a "red dotted line sqaure" ... not sure why this is appearing like this, I will check with admins on this..

G

PS: If you are happy with the answer provided, please mark the post as solution. You can do so by clicking link "Mark as Solution" below the answer provided.
 

mikebounds's picture

To test heartbeat network you can temporarily plumb some IPs on the interfaces:

Choose an interface, say nxge1 and plumb 1.1.1.1 netmask 255.255.255.0 on one node and 1.1.1.2 netmask 255.255.255.0 on the other node and then test you can ping 1.1.1.1/2 between the two nodes.

Mike

UK Symantec Consultant in VCS, GCO, SF, VVR, VxAT on Solaris, AIX, HP-ux, Linux & Windows

If this post has answered your question then please click on "Mark as solution" link below

Sun-Ray's picture

Hi Gaurav,

 

I have checked the llt connectivity okay. Stop and start the llt in the problem node. Then manually start remaining process manually and start the cluster. Now everythig looks green. 

 

root@nsscls02#  hastatus -sum

– SYSTEM STATE
– System               State                Frozen

A  nsscls01              RUNNING              0
A  nsscls02              RUNNING              0

==============================================================

Thanks to all for your support. :)

 

Cheers ....