VCS CRITICAL V-16-1-10037 VxFEN driver not configured. VCS Stopping. Manually restart VCS after configuring fencing

Article:TECH187188  |  Created: 2012-04-24  |  Updated: 2012-07-28  |  Article URL http://www.symantec.com/docs/TECH187188
Article Type
Technical Solution



Issue



After a node reboot, throwing following error, unable to start the VCS further.

2012/03/03 20:51:47 VCS CRITICAL V-16-1-10037 VxFEN driver not configured. Retrying...
2012/03/03 20:52:02 VCS CRITICAL V-16-1-10037 VxFEN driver not configured. Retrying...
2012/03/03 20:52:17 VCS CRITICAL V-16-1-10037 VxFEN driver not configured. Retrying...
2012/03/03 20:52:32 VCS CRITICAL V-16-1-10037 VxFEN driver not configured. Retrying...
2012/03/03 20:52:47 VCS CRITICAL V-16-1-10037 VxFEN driver not configured. Retrying...
2012/03/03 20:53:02 VCS CRITICAL V-16-1-10037 VxFEN driver not configured. Retrying...
2012/03/03 20:53:17 VCS CRITICAL V-16-1-10031 VxFEN driver not configured. VCS Stopping. Manually restart VCS after configuring fencing <<<< Problem, showing related to fencing.


Error



2012/03/03 20:51:47 VCS CRITICAL V-16-1-10037 VxFEN driver not configured. Retrying...
2012/03/03 20:52:02 VCS CRITICAL V-16-1-10037 VxFEN driver not configured. Retrying...
2012/03/03 20:52:17 VCS CRITICAL V-16-1-10037 VxFEN driver not configured. Retrying...
2012/03/03 20:52:32 VCS CRITICAL V-16-1-10037 VxFEN driver not configured. Retrying...
2012/03/03 20:52:47 VCS CRITICAL V-16-1-10037 VxFEN driver not configured. Retrying...
2012/03/03 20:53:02 VCS CRITICAL V-16-1-10037 VxFEN driver not configured. Retrying...
2012/03/03 20:53:17 VCS CRITICAL V-16-1-10031 VxFEN driver not configured. VCS Stopping. Manually restart VCS after configuring fencing <<<< Problem, showing related to fencing.


Environment



SFHA 5.0MP3, Linux RHEL 5.


Cause



########## Node 1 /var/adm/messages file ###########

2012/03/03 20:46:24 VCS INFO V-16-1-10196 Cluster logger started
2012/03/03 20:46:24 VCS NOTICE V-16-1-11022 VCS engine (had) started
2012/03/03 20:46:24 VCS NOTICE V-16-1-11050 VCS engine version=5.0
2012/03/03 20:46:24 VCS NOTICE V-16-1-11051 VCS engine join version=5.0.30.2
2012/03/03 20:46:24 VCS NOTICE V-16-1-11052 VCS engine pstamp=Veritas-5.0MP3RP2-01/22/09-15:17:00
2012/03/03 20:46:24 VCS NOTICE V-16-1-10114 Opening GAB library
2012/03/03 20:46:24 VCS NOTICE V-16-1-10619 'HAD' starting on: Node0
2012/03/03 20:46:24 VCS INFO V-16-1-10125 GAB timeout set to 15000 ms
2012/03/03 20:46:38 VCS CRITICAL V-16-1-11306 Did not receive cluster membership, manual intervention may be needed for seeding
2012/03/03 20:51:47 VCS INFO V-16-1-10077 Received new cluster membership
2012/03/03 20:51:47 VCS NOTICE V-16-1-10112 System (Node0) - Membership: 0x3, DDNA: 0x0
2012/03/03 20:51:47 VCS NOTICE V-16-1-10322 System  (Node '1') changed state from UNKNOWN to INITING
2012/03/03 20:51:47 VCS NOTICE V-16-1-10086 System Node0 (Node '0') is in Regular Membership - Membership: 0x3
2012/03/03 20:51:47 VCS NOTICE V-16-1-10086 System  (Node '1') is in Regular Membership - Membership: 0x3
2012/03/03 20:51:47 VCS NOTICE V-16-1-10453 Node: 1 changed name from: '' to: 'Node1'
2012/03/03 20:51:47 VCS NOTICE V-16-1-10322 System Node1 (Node '1') changed state from INITING to CURRENT_DISCOVER_WAIT
2012/03/03 20:51:47 VCS NOTICE V-16-1-10322 System Node0 (Node '0') changed state from CURRENT_DISCOVER_WAIT to LOCAL_BUILD
2012/03/03 20:51:47 VCS NOTICE V-16-1-10322 System Node1 (Node '1') changed state from CURRENT_DISCOVER_WAIT to CURRENT_PEER_WAIT  <<<< Node 1 waiting to join Node 0
2012/03/03 20:51:47 VCS CRITICAL V-16-1-10037 VxFEN driver not configured. Retrying...
2012/03/03 20:52:02 VCS CRITICAL V-16-1-10037 VxFEN driver not configured. Retrying...
2012/03/03 20:52:17 VCS CRITICAL V-16-1-10037 VxFEN driver not configured. Retrying...
2012/03/03 20:52:32 VCS CRITICAL V-16-1-10037 VxFEN driver not configured. Retrying...
2012/03/03 20:52:47 VCS CRITICAL V-16-1-10037 VxFEN driver not configured. Retrying...
2012/03/03 20:53:02 VCS CRITICAL V-16-1-10037 VxFEN driver not configured. Retrying...
2012/03/03 20:53:17 VCS CRITICAL V-16-1-10031 VxFEN driver not configured. VCS Stopping. Manually restart VCS after configuring fencing <<<< Problem, showing related to fencing.
2012/03/04 15:40:32 VCS NOTICE V-16-1-11022 VCS engine (had) started   <<<< Here the VCS was started again.
2012/03/04 15:40:32 VCS INFO V-16-1-10196 Cluster logger started
2012/03/04 15:40:32 VCS NOTICE V-16-1-11050 VCS engine version=5.0


The above log's show problem related to Fencing. Checked for the fencing specific log's,


Sat Mar  3 20:29:13 HKT 2012 vxfenconfig -U returned 1
Sat Mar  3 20:29:13 HKT 2012 vxfenconfig -U output is VXFEN vxfenconfig ERROR V-11-2-1071 Ioctl VXFEN_IOC_CLUSTSTAT failed!
Sat Mar  3 20:29:13 HKT 2012 output of vxfenadm is
Device Name: /dev/sdq
Total Number Of Keys: 4
key[0]:
        Key Value [Numeric Format]:  66,45,45,45,45,45,45,45
        Key Value [Character Format]: B-------
key[1]:
        Key Value [Numeric Format]:  66,45,45,45,45,45,45,45
        Key Value [Character Format]: B-------
key[2]:
        Key Value [Numeric Format]:  65,45,45,45,45,45,45,45
        Key Value [Character Format]: A-------
key[3]:
        Key Value [Numeric Format]:  65,45,45,45,45,45,45,45
        Key Value [Character Format]: A-------
Device Name: /dev/sds
Total Number Of Keys: 4
key[0]:
        Key Value [Numeric Format]:  66,45,45,45,45,45,45,45
        Key Value [Character Format]: B-------
key[1]:
        Key Value [Numeric Format]:  66,45,45,45,45,45,45,45
        Key Value [Character Format]: B-------
key[2]:
        Key Value [Numeric Format]:  65,45,45,45,45,45,45,45
        Key Value [Character Format]: A-------
key[3]:
        Key Value [Numeric Format]:  65,45,45,45,45,45,45,45
        Key Value [Character Format]: A-------
Device Name: /dev/sdr
Total Number Of Keys: 4
key[0]:
        Key Value [Numeric Format]:  66,45,45,45,45,45,45,45
        Key Value [Character Format]: B-------
key[1]:
        Key Value [Numeric Format]:  66,45,45,45,45,45,45,45
        Key Value [Character Format]: B-------
key[2]:
        Key Value [Numeric Format]:  65,45,45,45,45,45,45,45
        Key Value [Character Format]: A-------
key[3]:
        Key Value [Numeric Format]:  65,45,45,45,45,45,45,45
        Key Value [Character Format]: A-------
Device Name: /dev/sdg
Total Number Of Keys: 4
key[0]:
        Key Value [Numeric Format]:  66,45,45,45,45,45,45,45
        Key Value [Character Format]: B-------
key[1]:
        Key Value [Numeric Format]:  66,45,45,45,45,45,45,45
        Key Value [Character Format]: B-------
key[2]:
        Key Value [Numeric Format]:  65,45,45,45,45,45,45,45
        Key Value [Character Format]: A-------
key[3]:
        Key Value [Numeric Format]:  65,45,45,45,45,45,45,45
        Key Value [Character Format]: A-------
Device Name: /dev/sdi
Total Number Of Keys: 4
key[0]:
        Key Value [Numeric Format]:  66,45,45,45,45,45,45,45
        Key Value [Character Format]: B-------
key[1]:
        Key Value [Numeric Format]:  66,45,45,45,45,45,45,45
        Key Value [Character Format]: B-------
key[2]:
        Key Value [Numeric Format]:  65,45,45,45,45,45,45,45
        Key Value [Character Format]: A-------
key[3]:
        Key Value [Numeric Format]:  65,45,45,45,45,45,45,45
        Key Value [Character Format]: A-------
Device Name: /dev/sdh
Total Number Of Keys: 4
key[0]:
        Key Value [Numeric Format]:  66,45,45,45,45,45,45,45
        Key Value [Character Format]: B-------
key[1]:
        Key Value [Numeric Format]:  66,45,45,45,45,45,45,45
        Key Value [Character Format]: B-------
key[2]:
        Key Value [Numeric Format]:  65,45,45,45,45,45,45,45
        Key Value [Character Format]: A-------
key[3]:
        Key Value [Numeric Format]:  65,45,45,45,45,45,45,45
        Key Value [Character Format]: A-------
Sat Mar  3 20:29:13 HKT 2012 vxfenconfig -U returned with 1. exiting
Sat Mar  3 20:41:19 HKT 2012 \nstarting in vxfen-startup
Sat Mar  3 20:41:19 HKT 2012 calling regular vxfenconfig
Sun Mar  4 16:22:45 HKT 2012 \nstarting in vxfen-startup
Sun Mar  4 16:22:45 HKT 2012 calling regular vxfenconfig

After going through the log's, there was a reservation key's present when the server was booting up, and hence Fencing was not coming up.


Solution



Clear these keys to start I/O fencing, using these commands:

# /opt/VRTSvcs/vxfen/bin/vxfenclearpre

# reboot

Note: Reboot all nodes in the cluster.




Article URL http://www.symantec.com/docs/TECH187188


Terms of use for this information are found in Legal Notices