Video Screencast Help
Symantec to Separate Into Two Focused, Industry-Leading Technology Companies. Learn more.

solution needed for vxfen issue

Created: 04 Feb 2013 • Updated: 06 Feb 2013 | 16 comments
This issue has been solved. See solution.

 

there is a two node cluster and we split two node cluster for upgrade. The isolated node is not coming up as vxfen is not starting

/02/01 11:52:05 VCS CRITICAL V-16-1-10037 VxFEN driver not configured. Retrying...
2013/02/01 11:52:20 VCS CRITICAL V-16-1-10037 VxFEN driver not configured. Retrying...
2013/02/01 11:52:35 VCS CRITICAL V-16-1-10037 VxFEN driver not configured. Retrying...
2013/02/01 11:52:50 VCS CRITICAL V-16-1-10037 VxFEN driver not configured. Retrying...
2013/02/01 11:53:05 VCS CRITICAL V-16-1-10037 VxFEN driver not configured. Retrying...
2013/02/01 11:53:20 VCS CRITICAL V-16-1-10031 VxFEN driver not configured. VCS Stopping. Manually restart VCS after configuring fencing
^C

IOFENCING configuration seems okay on node 2 as /etc/vxfentab has the entry for  co-ordinator disks and /etc/vxfendg has diskgroup entry vxfendg2 and these disks and diskgroup are visible too.

DEVICE       TYPE            DISK         GROUP        STATUS
c0t5006016047201339d0s2 auto:sliced     -            (ossdg)      online
c0t5006016047201339d1s2 auto:sliced     -            (sybasedg)   online
c0t5006016047201339d2s2 auto:sliced     -            (vxfendg1)   online
c0t5006016047201339d3s2 auto:sliced     -            (vxfendg1)   online
c0t5006016047201339d4s2 auto:sliced     -            (vxfendg1)   online
c0t5006016047201339d5s2 auto:sliced     -            (vxfendg2)   online
c0t5006016047201339d6s2 auto:sliced     -            (vxfendg2)   online
c0t5006016047201339d7s2 auto:sliced     -            -            online
c2t0d0s2     auto:SVM        -            -            SVM
c2t1d0s2     auto:SVM        -            -            SVM

On checking vxfen.log

 

nvoked S97vxfen. Starting
Fri Feb  1 11:50:37 CET 2013 starting vxfen.. 
Fri Feb  1 11:50:37 CET 2013 calling start_fun
Fri Feb  1 11:50:38 CET 2013 found vxfenmode file
Fri Feb  1 11:50:38 CET 2013 calling generate_vxfentab
Fri Feb  1 11:50:38 CET 2013 checking for /etc/vxfendg
Fri Feb  1 11:50:38 CET 2013 found  /etc/vxfendg.
Fri Feb  1 11:50:38 CET 2013 calling generate_disklist
Fri Feb  1 11:50:38 CET 2013 Starting vxfen.. Done. 
Fri Feb  1 11:50:38 CET 2013 starting in vxfen-startup
Fri Feb  1 11:50:38 CET 2013 calling regular vxfenconfig
Fri Feb  1 11:50:38 CET 2013 return value from above operation is 1
Fri Feb  1 11:50:38 CET 2013 output was VXFEN vxfenconfig ERROR V-11-2-1003 At least three coordinator disks must be defined
Log Buffer: 0xfffffffff4041090

refadm2-oss1{root} # cat /etc/vxfendg
vxfendg2

and there are below mentioned two disks in vxfendg2

c0t5006016047201339d5s2 auto:sliced     -            (vxfendg2)   online
c0t5006016047201339d6s2 auto:sliced     -            (vxfendg2)   online

 

is it due to two  disks in coordinator diskgroup? Is it a known issue ?

 

 

Comments 16 CommentsJump to latest comment

arangari's picture

could you please provide output of 'gabconfig -a'  from both the nodes?

 

 

 

Thanks and Warm Regards,

Amit Rangari

If this post helped you resolving the issue, please mark it as solution. _____________________________________________________________________________

symsonu's picture

refadm2-oss1{root} # gabconfig -a
GAB Port Memberships
===============================================================
Port a gen   d22e01 membership 0

refadm2-oss1{root} # gabconfig -a
GAB Port Memberships
===============================================================
Port a gen   d22e01 membership 0

Both the nodes are split and cannot see eachother.

mikebounds's picture

The coordinator diskgroup must have at least 3 disks.  It is ok if a node can only see 2 of the 3 disks, must the diskgroup must contain 3 disks, and it looks as though vxfendg only contains 2 disks.

 

What version of VCS is this as with upgrades FROM 5.1 or higher you can do rolling upgrades, which means you do not need to split cluster - you can upgrade one node and VCS on the other node can still communicate with the other node even though it is on a different version.

Mike

UK Symantec Consultant in VCS, GCO, SF, VVR, VxAT on Solaris, AIX, HP-ux, Linux & Windows

If this post has answered your question then please click on "Mark as solution" link below

symsonu's picture

version is 5.1SP1RP2P2.

 

how can i add the third disk in the coordinator diskgroup?

symsonu's picture

I am trying to add   c0t5006016047201339d7s2 in vxfendg2 by importing vxfendg2.but it failed.

vxdisk -o alldgs list
DEVICE       TYPE            DISK         GROUP        STATUS
c0t5006016047201339d0s2 auto:sliced     -            (ossdg)      online
c0t5006016047201339d1s2 auto:sliced     -            (sybasedg)   online
c0t5006016047201339d2s2 auto:sliced     -            (vxfendg1)   online
c0t5006016047201339d3s2 auto:sliced     -            (vxfendg1)   online
c0t5006016047201339d4s2 auto:sliced     -            (vxfendg1)   online
c0t5006016047201339d5s2 auto:sliced     -            (vxfendg2)   online
c0t5006016047201339d6s2 auto:sliced     -            (vxfendg2)   online
c0t5006016047201339d7s2 auto:sliced     -            -            online
c2t0d0s2     auto:SVM        -            -            SVM
c2t1d0s2     auto:SVM        -            -            SVM
refadm2-oss1{root} # vxdg -t import vxfendg2
VxVM vxdg ERROR V-5-1-10978 Disk group vxfendg2: import failed:  
Disk for disk group not found
refadm2-oss1{root} #  
 

mikebounds's picture

You could try to import with "-f" to see if there is just one disk that can't be seen or if it is all the disks.

Mike

UK Symantec Consultant in VCS, GCO, SF, VVR, VxAT on Solaris, AIX, HP-ux, Linux & Windows

If this post has answered your question then please click on "Mark as solution" link below

symsonu's picture

 

 

refadm2-oss1{root} # vxdg -t import vxfendg2
VxVM vxdg ERROR V-5-1-10978 Disk group vxfendg2: import failed:  
Disk for disk group not found
refadm2-oss1{root} # vxdg -tfC import vxfendg2
VxVM vxdg WARNING V-5-1-560 Disk c0t5006016C47201339d7: Not found, last known location: c0t5006016047201339d7s2
refadm2-oss1{root} # vxdg list
NAME         STATE           ID
vxfendg2     enabled              1340186515.29.refadm1-oss1
refadm2-oss1{root} # vxdisk -o alldgs list
DEVICE       TYPE            DISK         GROUP        STATUS
c0t5006016047201339d0s2 auto:sliced     -            (ossdg)      online
c0t5006016047201339d1s2 auto:sliced     -            (sybasedg)   online
c0t5006016047201339d2s2 auto:sliced     -            (vxfendg1)   online
c0t5006016047201339d3s2 auto:sliced     -            (vxfendg1)   online
c0t5006016047201339d4s2 auto:sliced     -            (vxfendg1)   online
c0t5006016047201339d5s2 auto:sliced     c0t5006016C47201339d5  vxfendg2     online
c0t5006016047201339d6s2 auto:sliced     c0t5006016C47201339d6  vxfendg2     online
c0t5006016047201339d7s2 auto:sliced     -            -            online
c2t0d0s2     auto:SVM        -            -            SVM
c2t1d0s2     auto:SVM        -            -            SVM
-            -         c0t5006016C47201339d7 vxfendg2     failed was:c0t5006016047201339d7s2
refadm2-oss1{root} #  
 

symsonu's picture

Can you pls suggest me the action plan ?

 

 

Marianne's picture

Try 'vxreattach' command.

See http://www.symantec.com/docs/TECH175491

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

symsonu's picture

refadm2-oss1{root} # /etc/vx/bin/vxreattach -c c0t5006016C47201339d7
VxVM vxreattach ERROR V-5-2-238 No matching Volume Manager disk and device IDs found for c0t5006016047201339d7s2
refadm2-oss1{root} # /usr/lib/vxvm/bin/vxreattach  
refadm2-oss1{root} # vxdisk -o alldgs list
DEVICE       TYPE            DISK         GROUP        STATUS
c0t5006016047201339d0s2 auto:sliced     -            (ossdg)      online
c0t5006016047201339d1s2 auto:sliced     -            (sybasedg)   online
c0t5006016047201339d2s2 auto:sliced     -            (vxfendg1)   online
c0t5006016047201339d3s2 auto:sliced     -            (vxfendg1)   online
c0t5006016047201339d4s2 auto:sliced     -            (vxfendg1)   online
c0t5006016047201339d5s2 auto:sliced     c0t5006016C47201339d5  vxfendg2     online
c0t5006016047201339d6s2 auto:sliced     c0t5006016C47201339d6  vxfendg2     online
c0t5006016047201339d7s2 auto:sliced     -            -            online
c2t0d0s2     auto:SVM        -            -            SVM
c2t1d0s2     auto:SVM        -            -            SVM
-            -         c0t5006016C47201339d7 vxfendg2     failed was:c0t5006016047201339d7s2
refadm2-oss1{root} #  
 

Marianne's picture

Try: 

vxdg -k -g <group> adddisk <disk>=<device> 

vxdg -k -g vxfendg2 adddisk c0t5006016047201339d7s2=c0t5006016047201339d7s2

(We don't normally use Device name for Disk name, but it seems this is what you've done for other disks in vxfendg2).

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

SOLUTION
AHerr's picture

You could:
1) disable fencing (/opt/VRTS/install/installvcs -fencing) and then enable fencing.
2) run vxfenswap to change fencing devices (https://sort.symantec.com/public/documents/sfha/6....)

symsonu's picture

Hello,

 

what does -k  option specifies in  below coomand

 

vxdg -k -g vxfendg2 adddisk c0t5006016047201339d7s2=c0t5006016047201339d7s2

 

I mean the use of -k option please.

 

 

g_lee's picture

re: vxdg -k option - from manpage:

https://sort.symantec.com/public/documents/sfha/5....

----------
KEYWORDS

adddisk

[...]

If the -k option is specified, the disk media name must represent a disk media record that was previously dissociated from its disk access record with -k rmdisk; otherwise, a new disk media record is created to represent the disk. With the -k option, plexes requiring recovery are flagged as stale.
----------

minor amendment to Marianne's command as the other dmnames don't use s2:

# vxdg -g vxfendg2 -k adddisk c0t5006016047201339d7=c0t5006016047201339d7s2

If the adddisk doesn't work, please provide the output from:

# vxdisk list c0t5006016047201339d7s2

If this post has helped you, please vote or mark as solution

symsonu's picture

I have added the disk with below command

 

vxdg -g vxfendg2 -o coordinator -kf adddisk c0t5006016C47201339d7=c0t5006016047201339d7s2.

 

a slight modification needed to add -o coordinator as its a coordinator diskgroup

Thanks all for the excellent support

 

 

g_lee's picture

Please mark Marianne's post as solution since she pointed out the command.

If this post has helped you, please vote or mark as solution