Video Screencast Help

VCS 5.1 - DG import fails to the new node

Created: 15 Jan 2014 • Updated: 22 Jan 2014 | 9 comments
This issue has been solved. See solution.

Hi,

 

I have recently added a new node to a cluster and it joined the cluster with no problem... but whe I tried to import a DG to the node it fails...

Note that this DG is available and imported in a different node.

Here si the log when it WORKED in a existing node:

2014/01/16 00:49:42 VCS NOTICE V-16-1-10301 Initiating Online of Resource md_DG (Owner: unknown, Group: mdm) on System D-node6
2014/01/16 00:49:44 VCS WARNING V-16-10001-1014 (D-node6) DiskGroup:md_DG:online:Diskgroups will be imported without reservations
2014/01/16 00:49:46 VCS NOTICE V-16-10001-1009 (D-node6) DiskGroup:md_DG:online:vxdg import succeeded on Disk Group md_DG
2014/01/16 00:49:47 VCS NOTICE V-16-10001-1010 (D-node6) DiskGroup:md_DG:online:Volumes in Disk Group md_DG are started. Any mirrors are updated in background
 

And here is when it FAILED in the new node:

2014/01/16 00:45:11 VCS NOTICE V-16-1-10301 Initiating Online of Resource md_DG (Owner: unknown, Group: mdm) on System d-node9
2014/01/16 00:45:12 VCS WARNING V-16-10001-1014 (d-node9) DiskGroup:md_DG:online:Diskgroups will be imported without reservations
2014/01/16 00:45:12 VCS WARNING V-16-10001-1016 (d-node9) DiskGroup:md_DG:online:vxdg import (clear flag) failed.
2014/01/16 00:45:12 VCS WARNING V-16-10001-1017 (d-node9) DiskGroup:md_DG:online:Trying force import for the diskgroup.
2014/01/16 00:45:12 VCS ERROR V-16-10001-1003 (d-node9) DiskGroup:md_DG:online:** ERROR: vxdg import (force) failed on Disk Group md_DG
2014/01/16 00:45:12 VCS ERROR V-16-10001-1004 (d-node9) DiskGroup:md_DG:online:** ERROR: vxdg import failed on Disk Group md_DG after vxdctl enable
2014/01/16 00:45:13 VCS INFO V-16-2-13716 (d-node9) Resource(mdm_DG): Output of the completed operation (online)
==============================================
VxVM vxdg ERROR V-5-1-10978 Disk group md_DG: import failed:
No valid disk found containing disk group
VxVM vxdg ERROR V-5-1-10978 Disk group md_DG: import failed:
No valid disk found containing disk group
VxVM vxdg ERROR V-5-1-10978 Disk group md_DG: import failed:
No valid disk found containing disk group
==============================================
 

Looking into the old dicussions I have tried this but still didnt work...: changed before importing

# vxdmpadm settune dmp_cache_open=off
Tunable value will be changed immediately
# vxdmpadm gettune all | grep cache
dmp_cache_open                          off               on
 

But I got the same error.

Same thing o var/adm/mesages:

Jan 16 00:45:12 d-node9 Had[17566]: [ID 702911 daemon.notice] VCS ERROR V-16-1-1003 (dp-node9) DiskGroup:mdm_DG:online:** ERROR: vxdg import (force) failed on Disk Group mdm_DG
Jan 16 00:45:12 d-node9 vxdmp: [ID 631182 kern.notice] NOTICE: VxVM vxdmp V-5-0-0 removed disk array DISKS, datype = Disk
Jan 16 00:45:12 d-node9 vxdmp: [ID 803759 kern.notice] NOTICE: VxVM vxdmp V-5-0-34 added disk array DISKS, datype = Disk
Jan 16 00:45:12 d-node9 Had[17566]: [ID 702911 daemon.notice] VCS ERROR V-16-1-1004 (dp-node9) DiskGroup:mdm_DG:online:** ERROR: vxdg import failed on Disk Group mdm_DG after vxdctl enable
 

Any suggestion?

 

Tks,

Joao

 

 

 

Operating Systems:

Comments 9 CommentsJump to latest comment

Gaurav Sangamnerkar's picture

Hi Joao,

Can yo give more details , is this a CFS/CVM setup or its a failover setup ? Please give OS & SF versions as well, also to start, paste following outputs

1. attach main.cf

2. # gabconfig -a

3. # cat /etc/VRTSvcs/conf/config/main.cf | grep -i usefence

4. # hastatus -sum

 

G

PS: If you are happy with the answer provided, please mark the post as solution. You can do so by clicking link "Mark as Solution" below the answer provided.
 

mikebounds's picture

Can you provide output of "vxdisk -o alldgs list" on problem node - does it show the md_DG diskgroup?

Mike

UK Symantec Consultant in VCS, GCO, SF, VVR, VxAT on Solaris, AIX, HP-ux, Linux & Windows

If this post has answered your question then please click on "Mark as solution" link below

joaotelles's picture

From the faulty node:

# vxdisk -o alldgs list
DEVICE       TYPE            DISK         GROUP        STATUS
disk_0       auto            -            -            error
disk_1       auto            -            -            error
st2540-0_0   auto:cdsdisk    -            (oradg01)    online
st2540-0_1   auto:cdsdisk    -            (oradg01)    online
st2540-0_2   auto:cdsdisk    -            (oradg01)    online
st2540-0_3   auto:cdsdisk    -            (oradg01)    online
st2540-0_4   auto:cdsdisk    -            (oradg01)    online
st2540-0_5   auto:cdsdisk    -            (oradg01)    online
st2540-0_6   auto:cdsdisk    -            (oradg01)    online
 

From the old/working node:

vxdisk -o alldgs list
DEVICE       TYPE            DISK         GROUP        STATUS
disk_0       auto:SVM        -            -            SVM
disk_1       auto:SVM        -            -            SVM
st2540-0_0   auto:cdsdisk    -            (mastera_DG) online
st2540-0_1   auto:cdsdisk    bomserver_DG01  bomserver_DG online
st2540-0_2   auto:cdsdisk    wsm_DG01     wsm_DG       online
st2540-0_3   auto:cdsdisk    -            (hds_DG)     online
st2540-0_4   auto:cdsdisk    -            -            online
st2540-0_5   auto:cdsdisk    -            -            online
st2540-0_6   auto:cdsdisk    -            -            online
st2540-0_7   auto:cdsdisk    -            -            online
st2540-0_8   auto:cdsdisk    -            -            online
st2540-0_9   auto:cdsdisk    -            -            online
st2540-0_10  auto:cdsdisk    -            -            online
st2540-0_11  auto:cdsdisk    -            (sfm1a_DG)   online
st2540-0_12  auto:cdsdisk    -            (wig2a_DG)   online
st2540-0_13  auto:cdsdisk    wig3a_DG01   wig3a_DG     online
st2540-0_14  auto:cdsdisk    wig5a_DG01   wig5a_DG     online
st2540-0_15  auto:cdsdisk    md_DG01     md_DG       online
st2540-0_16  auto:cdsdisk    md_DG02     md_DG       online
 

=====

 

From the faulty node:

# gabconfig -a
GAB Port Memberships
===============================================================
Port a gen   40691c membership 012 4
Port h gen   406934 membership 012 4
 

I dont use fence.

# cat /etc/VRTSvcs/conf/config/main.cf | grep -i usefence
#

hastatus -sum

-- SYSTEM STATE
-- System               State                Frozen

A  D-node4             RUNNING              0
A  D-node5             RUNNING              0
A  D-node6             RUNNING              0
A  d-node9             RUNNING              0
 

B  md             D-node5             Y          N               OFFLINE
B  md             D-node6             Y          N               ONLINE
B  md             d-node9             Y          N               OFFLINE

 

AttachmentSize
main.zip 1.63 KB
joaotelles's picture

On the new/faulty node:

# vxdisk -o alldgs list
DEVICE       TYPE            DISK         GROUP        STATUS
disk_0       auto            -            -            error
disk_1       auto            -            -            error
st2540-0_0   auto:cdsdisk    -            (oradg01)    online
st2540-0_1   auto:cdsdisk    -            (oradg01)    online
st2540-0_2   auto:cdsdisk    -            (oradg01)    online
st2540-0_3   auto:cdsdisk    -            (oradg01)    online
st2540-0_4   auto:cdsdisk    -            (oradg01)    online
st2540-0_5   auto:cdsdisk    -            (oradg01)    online
st2540-0_6   auto:cdsdisk    -            (oradg01)    online
 

On the old/working node:

#  vxdisk -o alldgs list
DEVICE       TYPE            DISK         GROUP        STATUS
disk_0       auto:SVM        -            -            SVM
disk_1       auto:SVM        -            -            SVM
st2540-0_0   auto:cdsdisk    -            (mastera_DG) online
st2540-0_1   auto:cdsdisk    bomserver_DG01  bomserver_DG online
st2540-0_2   auto:cdsdisk    wsm_DG01     wsm_DG       online
st2540-0_3   auto:cdsdisk    -            (hds_DG)     online
st2540-0_4   auto:cdsdisk    -            -            online
st2540-0_5   auto:cdsdisk    -            -            online
st2540-0_6   auto:cdsdisk    -            -            online
st2540-0_7   auto:cdsdisk    -            -            online
st2540-0_8   auto:cdsdisk    -            -            online
st2540-0_9   auto:cdsdisk    -            -            online
st2540-0_10  auto:cdsdisk    -            -            online
st2540-0_11  auto:cdsdisk    -            (sfm1a_DG)   online
st2540-0_12  auto:cdsdisk    -            (wig2a_DG)   online
st2540-0_13  auto:cdsdisk    wig3a_DG01   wig3a_DG     online
st2540-0_14  auto:cdsdisk    wig5a_DG01   wig5a_DG     online
st2540-0_15  auto:cdsdisk    md_DG01     mdm_DG       online
st2540-0_16  auto:cdsdisk    md_DG02     mdm_DG       online

=====

On the new node:

# gabconfig -a
GAB Port Memberships
===============================================================
Port a gen   40691c membership 012 4
Port h gen   406934 membership 012 4
 

# cat /etc/VRTSvcs/conf/config/main.cf | grep -i usefence
#
 

hastatus -sum

-- SYSTEM STATE
-- System               State                Frozen

A  D-node4             RUNNING              0
A  D-node5             RUNNING              0
A  D-node6             RUNNING              0
A  d-node9             RUNNING              0

-- GROUP STATE
-- Group           System               Probed     AutoDisabled    State

B  ClusterService  D-node4             Y          N               OFFLINE
B  ClusterService  D-node5             Y          N               ONLINE
B  ClusterService  D-node6             Y          N               OFFLINE
B  ClusterService  d-node9             Y          N               OFFLINE
B  bomserver       D-node5             Y          N               OFFLINE
B  bomserver       D-node6             Y          N               ONLINE
B  bomserver       d-node9             Y          N               OFFLINE
B  hds             D-node5             Y          N               ONLINE
B  hds             D-node6             Y          N               OFFLINE
B  mastera         D-node5             Y          N               ONLINE
B  mastera         D-node6             Y          N               OFFLINE
B  md             D-node5             Y          N               OFFLINE
B  md             D-node6             Y          N               ONLINE
B  md             d-node9             Y          N               OFFLINE
B  nic             D-node4             Y          N               ONLINE
B  nic             D-node5             Y          N               ONLINE
B  nic             D-node6             Y          N               ONLINE
B  nic             d-node9             Y          N               ONLINE
B  oracle          D-node4             Y          N               ONLINE
B  sfm1a           D-node5             Y          N               ONLINE
B  sfm1a           D-node6             Y          N               OFFLINE
B  sysmona         D-node4             Y          N               ONLINE
B  sysmona         D-node5             Y          N               ONLINE
B  sysmona         D-node6             Y          N               ONLINE
B  wig2a           D-node5             Y          N               ONLINE
B  wig2a           D-node6             Y          N               OFFLINE
B  wig3a           D-node5             Y          N               OFFLINE
B  wig3a           D-node6             Y          N               ONLINE|STOPPING
B  wig3a           d-node9             Y          N               OFFLINE
B  wig5a           D-node5             Y          N               OFFLINE
B  wig5a           D-node6             Y          N               ONLINE
B  wig5a           d-node9             Y          N               OFFLINE
B  wsm             D-node5             Y          N               OFFLINE
B  wsm             D-node6             Y          N               ONLINE
B  wsm             d-node9             Y          N               OFFLINE
 

 

Gaurav Sangamnerkar's picture

Faulty node is not having disks or md_dg visible, it can only see oradg01 in deported state, you need to make sure that below two disks are visible to faulty node as well ...

st2540-0_15  auto:cdsdisk    md_DG01     md_DG       online
st2540-0_16  auto:cdsdisk    md_DG02     md_DG       online

don't rely on the names st2540xxxxx, this name may be different on different nodes, you can run a /etc/vx/diag.d/vxscsiinq on good node to find serial number of below luns & make sure that same luns are visible to faulty node as well

 

G

PS: If you are happy with the answer provided, please mark the post as solution. You can do so by clicking link "Mark as Solution" below the answer provided.
 

joaotelles's picture

Just did for the good node on the md_dg devices but I got no serial number...

 

# /etc/vx/diag.d/vxscsiinq st2540-0_15

Inquiry for /dev/vx/rdmp/st2540-0_15, evpd 0x0, page code 0x0
        Vendor id                        : SUN
        Product id                       : LCSM100_F
        Revision                         : 0670
        Serial Number                    :

# /etc/vx/diag.d/vxscsiinq st2540-0_16

Inquiry for /dev/vx/rdmp/st2540-0_16, evpd 0x0, page code 0x0
        Vendor id                        : SUN
        Product id                       : LCSM100_F
        Revision                         : 0670
        Serial Number                    :
 

Is this expected...? this is the good node... I will try to get the list of LUNs visible on the faulty one now.

Gaurav Sangamnerkar's picture

I would expect to get a serial number ... can you try the same with raw devices once

Find raw device with "vxdisk -e list" & you would get the native name in right column ... you can then run a vxscsiinq on raw device with /dev/rdsk/cxtxdx number & see the serial number

 

G

PS: If you are happy with the answer provided, please mark the post as solution. You can do so by clicking link "Mark as Solution" below the answer provided.
 

joaotelles's picture

Did as you told but no luck. Only for the disk at c0. Not for the c6 disk.

I have tried to access the CAM to see these disks but the diskarray is out of commuication to the CAM.. maybe this is the problem?

# vxdisk -e list
DEVICE       TYPE           DISK        GROUP        STATUS               OS_NATIVE_NAME   ATTR
disk_0       auto:SVM       -            -           SVM                  c0t5000CCA0153684F4d0s2 -
disk_1       auto:SVM       -            -           SVM                  c0t5000CCA015393DACd0s2 -
st2540-0_0   auto:cdsdisk   -            -           online               c6t202400A0B84930EDd4s2 -
st2540-0_1   auto:cdsdisk   bomserver_DG01  bomserver_DG online               c6t202400A0B84930EDd5s2 -
st2540-0_2   auto:cdsdisk   wsm_DG01     wsm_DG      online               c6t202400A0B84930EDd0s2 -
st2540-0_3   auto:cdsdisk   -            -           online               c6t202400A0B84930EDd1s2 -
st2540-0_4   auto:cdsdisk   -            -           online               c6t202400A0B84930EDd2s2 -
st2540-0_5   auto:cdsdisk   -            -           online               c6t202400A0B84930EDd3s2 -
st2540-0_6   auto:cdsdisk   -            -           online               c6t202400A0B84930EDd6s2 -
st2540-0_7   auto:cdsdisk   -            -           online               c6t202400A0B84930EDd7s2 -
st2540-0_8   auto:cdsdisk   -            -           online               c6t202400A0B84930EDd8s2 -
st2540-0_9   auto:cdsdisk   -            -           online               c6t202400A0B84930EDd9s2 -
st2540-0_10  auto:cdsdisk   -            -           online               c6t202400A0B84930EDd10s2 -
st2540-0_11  auto:cdsdisk   -            -           online               c6t202400A0B84930EDd11s2 -
st2540-0_12  auto:cdsdisk   -            -           online               c6t202400A0B84930EDd12s2 -
st2540-0_13  auto:cdsdisk   wig3a_DG01   wig3a_DG    online               c6t202400A0B84930EDd13s2 -
st2540-0_14  auto:cdsdisk   wig5a_DG01   wig5a_DG    online               c6t202400A0B84930EDd14s2 -
st2540-0_15  auto:cdsdisk   mdm_DG01     mdm_DG      online               c6t202400A0B84930EDd15s2 -
st2540-0_16  auto:cdsdisk   mdm_DG02     mdm_DG      online               c6t202400A0B84930EDd16s2 -
root@DP-node6 # etc/vx/diag.d/vxscsiinq c6t202400A0B84930EDd15s2

Inquiry for /dev/rdsk/c6t202400A0B84930EDd15s2, evpd 0x0, page code 0x0
        Vendor id                        : SUN
        Product id                       : LCSM100_F
        Revision                         : 0670
        Serial Number                    :
root@DP-node6 # etc/vx/diag.d/vxscsiinq c6t202400A0B84930EDd16s2

Inquiry for /dev/rdsk/c6t202400A0B84930EDd16s2, evpd 0x0, page code 0x0
        Vendor id                        : SUN
        Product id                       : LCSM100_F
        Revision                         : 0670
        Serial Number                    :
root@DP-node6 # etc/vx/diag.d/vxscsiinq c0t5000CCA0153684F4d0s2

Inquiry for /dev/rdsk/c0t5000CCA0153684F4d0s2, evpd 0x0, page code 0x0
        Vendor id                        : HITACHI
        Product id                       : H103030SCSUN300G
        Revision                         : A2A8
        Serial Number                    : 1105GYZHBE
 

mikebounds's picture

Veritas Volume Manager only sees the disks the O/S sees, so if Solaris "format" command only see the 7 disks from oradg01 and the 2 internal disks and "format" cannot see the 2 disks for the md_DG diskgroup, then you need to fix issue with your O/S not seeing the disks, which could be an issue with the O/S, the FC switches or the disk array.

Mike

UK Symantec Consultant in VCS, GCO, SF, VVR, VxAT on Solaris, AIX, HP-ux, Linux & Windows

If this post has answered your question then please click on "Mark as solution" link below

SOLUTION