Video Screencast Help
Symantec to Separate Into Two Focused, Industry-Leading Technology Companies. Learn more.

need help

Created: 14 Oct 2012 • Updated: 16 Oct 2012 | 5 comments

while booting a sfrac 4.1 cluster node, getting below error and it is stuck

 

 

ct 14 15:30:00 rmgdbp09 vcsmm: VCS RAC ERROR V-10-1-15013 vcsmm_ioctl: driver not configured
Oct 14 15:32:17 rmgdbp09 scsi: WARNING: /pci@8,600000/SUNW,qlc@1/fp@0,0/ssd@w50060e8006cfa802,0 (ssd226):
Oct 14 15:32:17 rmgdbp09 SCSI transport failed: reason 'timeout': giving up
Oct 14 15:38:17 rmgdbp09 scsi: WARNING: /pci@8,600000/SUNW,qlc@1/fp@0,0/ssd@w50060e8006cfa802,0 (ssd226):
Oct 14 15:38:17 rmgdbp09 SCSI transport failed: reason 'timeout': giving up
Oct 14 15:45:00 rmgdbp09 vcsmm: VCS RAC ERROR V-10-1-15013 vcsmm_ioctl: driver not configured
Oct 14 16:00:00 rmgdbp09 vcsmm: VCS RAC ERROR V-10-1-15013 vcsmm_ioctl: driver not configured
 

 

Discussion Filed Under:

Comments 5 CommentsJump to latest comment

symsonu's picture

Oct 14 16:25:29 rmgdbp09 login: ROOT LOGIN /dev/console
Last login: Sun Oct 14 13:09:15 on console
Oct 14 16:25:30 rmgdbp09 arp: ar_query_xmit: Could not find the ace
Oct 14 16:25:30 rmgdbp09 arp: [ID 994748 kern.notice] ar_query_xmit: Could not find the ace
Oct 14 16:25:30 rmgdbp09 arp: [ID 994748 kern.notice] ar_query_xmit: Could not find the ace
Sun Microsystems Inc.   SunOS 5.9       Generic May 2002
You have new mail.
root@rmgdbp09 #
root@rmgdbp09 #
root@rmgdbp09 #
root@rmgdbp09 #
root@rmgdbp09 # dmesg

Sun Oct 14 16:25:41 UTC 2012
Oct 14 14:22:18 rmgdbp09 vxdmp: [ID 917986 kern.notice] NOTICE: VxVM vxdmp V-5-0-112 disabled path 118/0x710 belonging to the dmpnode 286/0xc8
Oct 14 14:23:52 rmgdbp09 vxdmp: [ID 736771 kern.notice] NOTICE: VxVM vxdmp V-5-0-148 enabled path 118/0x710 belonging to the dmpnode 286/0xc8
Oct 14 14:29:16 rmgdbp09 vxdmp: [ID 917986 kern.notice] NOTICE: VxVM vxdmp V-5-0-112 disabled path 118/0x700 belonging to the dmpnode 286/0x50
Oct 14 14:33:18 rmgdbp09 scsi: [ID 107833 kern.warning] WARNING: /pci@8,600000/SUNW,qlc@1/fp@0,0/ssd@w50060e8006cfa802,32 (ssd223):
Oct 14 14:33:18 rmgdbp09        offline
Oct 14 14:33:52 rmgdbp09 vxdmp: [ID 736771 kern.notice] NOTICE: VxVM vxdmp V-5-0-148 enabled path 118/0x700 belonging to the dmpnode 286/0x50
Oct 14 14:38:18 rmgdbp09 vxdmp: [ID 917986 kern.notice] NOTICE: VxVM vxdmp V-5-0-112 disabled path 118/0x6f8 belonging to the dmpnode 286/0x80
Oct 14 14:38:52 rmgdbp09 vxdmp: [ID 736771 kern.notice] NOTICE: VxVM vxdmp V-5-0-148 enabled path 118/0x6f8 belonging to the dmpnode 286/0x80
Oct 14 14:39:45 rmgdbp09 vxesd[141]: [ID 360244 daemon.notice] Event Source daemon started
Oct 14 14:39:45 rmgdbp09 syseventd[75]: [ID 617319 daemon.error] SIGHUP caught - reloading modules
Oct 14 14:39:46 rmgdbp09 syseventd[75]: [ID 661968 daemon.error] Daemon restarted
Oct 14 14:41:14 rmgdbp09 pseudo: [ID 129642 kern.info] pseudo-device: laner0
Oct 14 14:41:14 rmgdbp09 genunix: [ID 936769 kern.info] laner0 is /pseudo/laner@0
Oct 14 14:46:17 rmgdbp09 scsi: [ID 107833 kern.warning] WARNING: /pci@8,600000/SUNW,qlc@1/fp@0,0/ssd@w50060e8006cfa802,0 (ssd226):
Oct 14 14:46:17 rmgdbp09        SCSI transport failed: reason 'timeout': giving up
Oct 14 14:49:17 rmgdbp09 scsi: [ID 107833 kern.warning] WARNING: /pci@8,600000/SUNW,qlc@1/fp@0,0/ssd@w50060e8006cfa802,0 (ssd226):
Oct 14 14:49:17 rmgdbp09        SCSI transport failed: reason 'timeout': giving up
Oct 14 14:51:17 rmgdbp09 scsi: [ID 107833 kern.warning] WARNING: /pci@8,600000/SUNW,qlc@1/fp@0,0/ssd@w50060e8006cfa802,0 (ssd226):
Oct 14 14:51:17 rmgdbp09        SCSI transport failed: reason 'timeout': giving up
Oct 14 14:53:17 rmgdbp09 scsi: [ID 107833 kern.warning] WARNING: /pci@8,600000/SUNW,qlc@1/fp@0,0/ssd@w50060e8006cfa802,0 (ssd226):
Oct 14 14:53:17 rmgdbp09        SCSI transport failed: reason 'timeout': giving up
Oct 14 14:56:18 rmgdbp09 scsi: [ID 107833 kern.warning] WARNING: /pci@8,600000/SUNW,qlc@1/fp@0,0/ssd@w50060e8006cfa802,0 (ssd226):
Oct 14 14:56:18 rmgdbp09        SCSI transport failed: reason 'timeout': giving up
Oct 14 14:58:21 rmgdbp09 llt: [ID 144861 kern.notice] LLT INFO V-14-1-10009 LLT Protocol available
Oct 14 14:58:32 rmgdbp09 lmx: [ID 504452 kern.notice] LMX Multiplexor available
Oct 14 14:58:34 rmgdbp09 root: [ID 702911 daemon.error] CAPTURE_UPTIME ERROR: /var/opt/SUNWsrsrp missing
Oct 14 14:58:34 rmgdbp09 savecore: [ID 686316 auth.error] no dump device configured
Oct 14 14:58:34 rmgdbp09 last message repeated 1 time
Oct 14 14:58:34 rmgdbp09 genunix: [ID 454863 kern.info] dump on /dev/dsk/c1t0d0s1 size 32793 MB
Oct 14 14:58:34 rmgdbp09 ntpdate[1207]: [ID 558275 daemon.notice] adjust time server 193.88.157.124 offset -0.367765 sec
Oct 14 14:58:35 rmgdbp09 pseudo: [ID 129642 kern.info] pseudo-device: tod0
Oct 14 14:58:35 rmgdbp09 genunix: [ID 936769 kern.info] tod0 is /pseudo/tod@0
Oct 14 14:58:35 rmgdbp09 pseudo: [ID 129642 kern.info] pseudo-device: pm0
Oct 14 14:58:35 rmgdbp09 genunix: [ID 936769 kern.info] pm0 is /pseudo/pm@0
Oct 14 14:58:35 rmgdbp09 gab: [ID 872190 kern.notice] GAB INFO V-15-1-20021 GAB available
Oct 14 14:58:35 rmgdbp09 gab: [ID 222459 kern.notice] GAB INFO V-15-1-20026 Port a registration waiting for seed port membership
Oct 14 14:58:36 rmgdbp09 gab: [ID 222459 kern.notice] GAB INFO V-15-1-20026 Port d registration waiting for seed port membership
Oct 14 14:58:36 rmgdbp09 llt: [ID 860062 kern.notice] LLT INFO V-14-1-10024 link 1 (ce5) node 0 active
Oct 14 14:58:36 rmgdbp09 llt: [ID 860062 kern.notice] LLT INFO V-14-1-10024 link 0 (ce1) node 0 active
Oct 14 14:58:36 rmgdbp09 xntpd[1314]: [ID 702911 daemon.notice] xntpd 3-5.93e Mon Sep 20 15:47:11 PDT 1999 (1)
Oct 14 14:58:36 rmgdbp09 xntpd[1314]: [ID 301315 daemon.notice] tickadj = 5, tick = 10000, tvu_maxslew = 495, est. hz = 100
Oct 14 14:58:37 rmgdbp09 xntpd[1314]: [ID 798731 daemon.notice] using kernel phase-lock loop 0041
Oct 14 14:58:37 rmgdbp09 last message repeated 1 time
Oct 14 14:58:40 rmgdbp09 gab: [ID 316943 kern.notice] GAB INFO V-15-1-20036 Port d gen   2d7c05 membership 01
Oct 14 14:58:40 rmgdbp09 gab: [ID 316943 kern.notice] GAB INFO V-15-1-20036 Port a gen   2d7c06 membership 01
Oct 14 15:00:01 rmgdbp09 vcsmm: [ID 743917 kern.notice] VCS RAC ERROR V-10-1-15013 vcsmm_ioctl: driver not configured
Oct 14 15:07:12 rmgdbp09 scsi: [ID 107833 kern.warning] WARNING: /pci@8,600000/SUNW,qlc@1/fp@0,0/ssd@w50060e8006cfa802,0 (ssd226):
Oct 14 15:07:12 rmgdbp09        SCSI transport failed: reason 'timeout': giving up
Oct 14 15:15:00 rmgdbp09 vcsmm: [ID 743917 kern.notice] VCS RAC ERROR V-10-1-15013 vcsmm_ioctl: driver not configured
Oct 14 15:22:17 rmgdbp09 scsi: [ID 107833 kern.warning] WARNING: /pci@8,600000/SUNW,qlc@1/fp@0,0/ssd@w50060e8006cfa802,0 (ssd226):
Oct 14 15:22:17 rmgdbp09        SCSI transport failed: reason 'timeout': giving up
Oct 14 15:30:00 rmgdbp09 vcsmm: [ID 743917 kern.notice] VCS RAC ERROR V-10-1-15013 vcsmm_ioctl: driver not configured
Oct 14 15:32:17 rmgdbp09 scsi: [ID 107833 kern.warning] WARNING: /pci@8,600000/SUNW,qlc@1/fp@0,0/ssd@w50060e8006cfa802,0 (ssd226):
Oct 14 15:32:17 rmgdbp09        SCSI transport failed: reason 'timeout': giving up
Oct 14 15:38:17 rmgdbp09 scsi: [ID 107833 kern.warning] WARNING: /pci@8,600000/SUNW,qlc@1/fp@0,0/ssd@w50060e8006cfa802,0 (ssd226):
Oct 14 15:38:17 rmgdbp09        SCSI transport failed: reason 'timeout': giving up
Oct 14 15:45:00 rmgdbp09 vcsmm: [ID 743917 kern.notice] VCS RAC ERROR V-10-1-15013 vcsmm_ioctl: driver not configured
Oct 14 16:00:00 rmgdbp09 vcsmm: [ID 743917 kern.notice] VCS RAC ERROR V-10-1-15013 vcsmm_ioctl: driver not configured
Oct 14 16:08:22 rmgdbp09 gab: [ID 316943 kern.notice] GAB INFO V-15-1-20036 Port b gen   2d7c0a membership 01
Oct 14 16:08:22 rmgdbp09 gab: [ID 316943 kern.notice] GAB INFO V-15-1-20036 Port o gen   2d7c09 membership 01
Oct 14 16:08:22 rmgdbp09 vcsmm: [ID 357760 kern.notice] VCS RAC INFO V-10-1-15047 mmpl_reconfig_ioctl: dev_ioctl failed, vxfen may not be configured
Oct 14 16:08:22 rmgdbp09 vxfen: [ID 725375 kern.notice] NOTICE: VCS FEN INFO V-11-1-35 Fencing driver going into RUNNING state
Oct 14 16:08:42 rmgdbp09 snmpXdmid: [ID 723131 daemon.error] Error in Adding Row for Subscription Table Entry
Oct 14 16:08:42 rmgdbp09 snmpXdmid: [ID 132663 daemon.error] Failed to add filter to SP for Event delivery
Oct 14 16:09:49 rmgdbp09 syslog[4934]: [ID 702911 daemon.notice] VCS INFO V-16-1-11240 Command Server: running with security OFF
Oct 14 16:09:49 rmgdbp09 Had[4933]: [ID 702911 daemon.notice] VCS NOTICE V-16-1-10619 'HAD' starting on: rmgdbp09
Oct 14 16:09:50 rmgdbp09 Had[4933]: [ID 702911 daemon.notice] VCS NOTICE V-16-1-10620 Waiting for local cluster configuration status
Oct 14 16:09:51 rmgdbp09 Had[4933]: [ID 702911 daemon.notice] VCS ERROR V-16-1-10624 Local cluster configuration stale
Oct 14 16:09:51 rmgdbp09 last message repeated 1 time
Oct 14 16:09:51 rmgdbp09 Had[4933]: [ID 702911 daemon.notice] VCS NOTICE V-16-1-11034 Registering for cluster membership
Oct 14 16:09:51 rmgdbp09 Had[4933]: [ID 702911 daemon.notice] VCS NOTICE V-16-1-11035 Waiting for cluster membership
Oct 14 16:09:56 rmgdbp09 gab: [ID 316943 kern.notice] GAB INFO V-15-1-20036 Port h gen   2d7c19 membership 01
Oct 14 16:09:56 rmgdbp09 Had[4933]: [ID 702911 daemon.notice] VCS INFO V-16-1-10077 Received new cluster membership
Oct 14 16:09:56 rmgdbp09 Had[4933]: [ID 702911 daemon.notice] VCS NOTICE V-16-1-10086 System  (Node '0') is in Regular Membership - Membership: 0x3
Oct 14 16:09:56 rmgdbp09 Had[4933]: [ID 702911 daemon.notice] VCS NOTICE V-16-1-10086 System rmgdbp09 (Node '1') is in Regular Membership - Membership: 0x3
Oct 14 16:09:56 rmgdbp09 Had[4933]: [ID 702911 daemon.notice] VCS NOTICE V-16-1-10075 Building from remote system
Oct 14 16:09:58 rmgdbp09 vxfen: [ID 214757 kern.notice] NOTICE: VCS FEN INFO V-11-1-34 The ioctl VXFEN_IOC_CLUSTSTAT returned 0
Oct 14 16:09:58 rmgdbp09 Had[4933]: [ID 702911 daemon.notice] VCS NOTICE V-16-1-10066 Entering RUNNING state
Oct 14 16:09:58 rmgdbp09 Had[4933]: [ID 702911 daemon.notice] VCS NOTICE V-16-1-50311 VCS Engine: running with security OFF
Oct 14 16:10:00 rmgdbp09 Had[4933]: [ID 702911 daemon.notice] VCS ERROR V-16-1-1005 (rmgdbp09) CVMCluster:???:monitor:node - state: out of cluster
Oct 14 16:10:04 rmgdbp09 vxfen: [ID 214757 kern.notice] NOTICE: VCS FEN INFO V-11-1-34 The ioctl VXFEN_IOC_CLUSTSTAT returned 0
Oct 14 16:10:09 rmgdbp09 gab: [ID 316943 kern.notice] GAB INFO V-15-1-20036 Port u gen   2d7c1b membership ;1
Oct 14 16:10:09 rmgdbp09 gab: [ID 674723 kern.notice] GAB INFO V-15-1-20038 Port u gen   2d7c1b k_jeopardy 0
Oct 14 16:10:09 rmgdbp09 gab: [ID 513393 kern.notice] GAB INFO V-15-1-20040 Port u gen   2d7c1b    visible 0
Oct 14 16:10:14 rmgdbp09 gab: [ID 316943 kern.notice] GAB INFO V-15-1-20036 Port v gen   2d7c1a membership 01
Oct 14 16:10:15 rmgdbp09 pseudo: [ID 129642 kern.info] pseudo-device: devinfo0
Oct 14 16:10:15 rmgdbp09 genunix: [ID 936769 kern.info] devinfo0 is /pseudo/devinfo@0
Oct 14 16:10:20 rmgdbp09 vxvm:vxconfigd: [ID 456610 daemon.notice] V-5-1-7900 CVM_VOLD_CONFIG command received
Oct 14 16:10:20 rmgdbp09 vxvm:vxconfigd: [ID 699813 daemon.notice] V-5-1-7899 CVM_VOLD_CHANGE command received
Oct 14 16:10:20 rmgdbp09 vxvm:vxconfigd: [ID 322665 daemon.notice] V-5-1-7961 establishing cluster
Oct 14 16:10:25 rmgdbp09 gab: [ID 316943 kern.notice] GAB INFO V-15-1-20036 Port w gen   2d7c20 membership 01
Oct 14 16:10:31 rmgdbp09 vxfen: [ID 214757 kern.notice] NOTICE: VCS FEN INFO V-11-1-34 The ioctl VXFEN_IOC_CLUSTSTAT returned 0
Oct 14 16:11:16 rmgdbp09 last message repeated 40 times
Oct 14 16:11:25 rmgdbp09 Had[4933]: [ID 702911 daemon.notice] VCS ERROR V-16-1-13027 (rmgdbp08) Resource(srvm_vol_dg) - monitor procedure did not complete within the expected time.
Oct 14 16:11:45 rmgdbp09 Had[4933]: [ID 702911 daemon.notice] VCS ERROR V-16-1-13027 (rmgdbp08) Resource(RMAN_VOL01) - monitor procedure did not complete within the expected time.
Oct 14 16:11:45 rmgdbp09 Had[4933]: [ID 702911 daemon.notice] VCS ERROR V-16-1-13027 (rmgdbp08) Resource(PPUB_VOL_arch) - monitor procedure did not complete within the expected time.
Oct 14 16:11:45 rmgdbp09 Had[4933]: [ID 702911 daemon.notice] VCS ERROR V-16-1-13027 (rmgdbp08) Resource(PPTL_VOL_oracle) - monitor procedure did not complete within the expected time.
Oct 14 16:11:45 rmgdbp09 Had[4933]: [ID 702911 daemon.notice] VCS ERROR V-16-1-13027 (rmgdbp08) Resource(PPTL_VOL_arch) - monitor procedure did not complete within the expected time.
Oct 14 16:11:45 rmgdbp09 Had[4933]: [ID 702911 daemon.notice] VCS ERROR V-16-1-13027 (rmgdbp08) Resource(PAVS_VOL_arch) - monitor procedure did not complete within the expected time.
Oct 14 16:11:45 rmgdbp09 Had[4933]: [ID 702911 daemon.notice] VCS ERROR V-16-1-13027 (rmgdbp08) Resource(PPUB_VOL_oracle) - monitor procedure did not complete within the expected time.
Oct 14 16:11:45 rmgdbp09 Had[4933]: [ID 702911 daemon.notice] VCS ERROR V-16-1-13027 (rmgdbp08) Resource(PAVS_VOL_oracle) - monitor procedure did not complete within the expected time.
Oct 14 16:13:35 rmgdbp09 vxio: [ID 535583 kern.warning] WARNING: VxVM vxio V-5-0-164 Failed to join cluster rmgprod, aborting
Oct 14 16:13:35 rmgdbp09 gab: [ID 397130 kern.notice] GAB INFO V-15-1-20032 Port u closed
Oct 14 16:13:35 rmgdbp09 gab: [ID 397130 kern.notice] GAB INFO V-15-1-20032 Port v closed
Oct 14 16:13:35 rmgdbp09 vxvm:vxconfigd: [ID 955251 daemon.error] V-5-1-8765 cluster join for node 1 failed: Connection timed out
Oct 14 16:13:39 rmgdbp09 gab: [ID 397130 kern.notice] GAB INFO V-15-1-20032 Port w closed
Oct 14 16:13:39 rmgdbp09 vxvm:vxconfigd: [ID 741513 daemon.notice] V-5-1-7901 CVM_VOLD_STOP command received
Oct 14 16:13:39 rmgdbp09 vxvm:vxconfigd: [ID 778436 daemon.error] V-5-1-4109 -1 returned from volcvm_establish
Oct 14 16:13:39 rmgdbp09 vxvm:vxconfigd: [ID 886039 daemon.warning] V-5-1-4852 cluster_establish: timed out
Oct 14 16:13:39 rmgdbp09 vxvm:vxconfigd: [ID 453237 daemon.error] V-5-1-11178 kernel_fail_join() : join timed out during reconfiguration (12, -1)
Oct 14 16:13:39 rmgdbp09 vxvm:vxconfigd: [ID 565473 daemon.notice] V-5-1-9543 Timeout is not reset: another reconfig in progress
Oct 14 16:14:10 rmgdbp09 Had[4933]: [ID 702911 daemon.notice] VCS ERROR V-16-1-1005 (rmgdbp09) CVMCluster:???:monitor:node - state: out of cluster
Oct 14 16:14:10 rmgdbp09 reason: request to join failed
Oct 14 16:15:00 rmgdbp09 vcsmm: [ID 832650 kern.notice] VCSMM: mm_deblog_sz = 1048576
Oct 14 16:15:00 rmgdbp09 vcsmm: [ID 706457 kern.notice] VCSMM: mm_msglog_sz = 128
Oct 14 16:15:00 rmgdbp09 vcsmm: [ID 838746 kern.notice] VCSMM: mm_slave_max = 1024
Oct 14 16:15:10 rmgdbp09 Had[4933]: [ID 702911 daemon.notice] VCS ERROR V-16-1-1005 (rmgdbp09) CVMCluster:???:monitor:node - state: out of cluster
Oct 14 16:15:10 rmgdbp09 reason: request to join failed
Oct 14 16:15:12 rmgdbp09 Had[4933]: [ID 702911 daemon.notice] VCS ERROR V-16-1-13066 (rmgdbp09) Agent is calling clean for resource(cvm_clus) because the resource is not up even after online completed.
Oct 14 16:15:13 rmgdbp09 vxfen: [ID 214757 kern.notice] NOTICE: VCS FEN INFO V-11-1-34 The ioctl VXFEN_IOC_CLUSTSTAT returned 0
Oct 14 16:15:17 rmgdbp09 gab: [ID 316943 kern.notice] GAB INFO V-15-1-20036 Port u gen   2d7c22 membership ;1
Oct 14 16:15:17 rmgdbp09 gab: [ID 674723 kern.notice] GAB INFO V-15-1-20038 Port u gen   2d7c22 k_jeopardy 0
Oct 14 16:15:17 rmgdbp09 gab: [ID 513393 kern.notice] GAB INFO V-15-1-20040 Port u gen   2d7c22    visible 0
Oct 14 16:15:22 rmgdbp09 gab: [ID 316943 kern.notice] GAB INFO V-15-1-20036 Port v gen   2d7c1c membership 01
Oct 14 16:15:28 rmgdbp09 vxvm:vxconfigd: [ID 456610 daemon.notice] V-5-1-7900 CVM_VOLD_CONFIG command received
Oct 14 16:15:28 rmgdbp09 vxvm:vxconfigd: [ID 699813 daemon.notice] V-5-1-7899 CVM_VOLD_CHANGE command received
Oct 14 16:15:28 rmgdbp09 vxvm:vxconfigd: [ID 322665 daemon.notice] V-5-1-7961 establishing cluster
Oct 14 16:15:33 rmgdbp09 gab: [ID 316943 kern.notice] GAB INFO V-15-1-20036 Port w gen   2d7c22 membership 01
Oct 14 16:15:39 rmgdbp09 vxfen: [ID 214757 kern.notice] NOTICE: VCS FEN INFO V-11-1-34 The ioctl VXFEN_IOC_CLUSTSTAT returned 0
Oct 14 16:16:44 rmgdbp09 last message repeated 23 times
Oct 14 16:16:45 rmgdbp09 Had[4933]: [ID 702911 daemon.notice] VCS ERROR V-16-1-13027 (rmgdbp08) Resource(PAVS_VOL_arch) - monitor procedure did not complete within the expected time.
Oct 14 16:16:45 rmgdbp09 Had[4933]: [ID 702911 daemon.notice] VCS ERROR V-16-1-13027 (rmgdbp08) Resource(PPTL_VOL_arch) - monitor procedure did not complete within the expected time.
Oct 14 16:16:45 rmgdbp09 Had[4933]: [ID 702911 daemon.notice] VCS ERROR V-16-1-13027 (rmgdbp08) Resource(PPUB_VOL_arch) - monitor procedure did not complete within the expected time.
Oct 14 16:16:45 rmgdbp09 Had[4933]: [ID 702911 daemon.notice] VCS ERROR V-16-1-13027 (rmgdbp08) Resource(PPTL_VOL_oracle) - monitor procedure did not complete within the expected time.
Oct 14 16:16:45 rmgdbp09 Had[4933]: [ID 702911 daemon.notice] VCS ERROR V-16-1-13027 (rmgdbp08) Resource(RMAN_VOL01) - monitor procedure did not complete within the expected time.
Oct 14 16:16:45 rmgdbp09 Had[4933]: [ID 702911 daemon.notice] VCS ERROR V-16-1-13027 (rmgdbp08) Resource(PPUB_VOL_oracle) - monitor procedure did not complete within the expected time.
Oct 14 16:16:45 rmgdbp09 Had[4933]: [ID 702911 daemon.notice] VCS ERROR V-16-1-13027 (rmgdbp08) Resource(PAVS_VOL_oracle) - monitor procedure did not complete within the expected time.
Oct 14 16:16:52 rmgdbp09 vxfen: [ID 214757 kern.notice] NOTICE: VCS FEN INFO V-11-1-34 The ioctl VXFEN_IOC_CLUSTSTAT returned 0
Oct 14 16:17:02 rmgdbp09 last message repeated 16 times
Oct 14 16:17:25 rmgdbp09 Had[4933]: [ID 702911 daemon.notice] VCS ERROR V-16-1-13027 (rmgdbp08) Resource(srvm_vol_dg) - monitor procedure did not complete within the expected time.
Oct 14 16:18:43 rmgdbp09 vxio: [ID 535583 kern.warning] WARNING: VxVM vxio V-5-0-164 Failed to join cluster rmgprod, aborting
Oct 14 16:18:43 rmgdbp09 gab: [ID 397130 kern.notice] GAB INFO V-15-1-20032 Port u closed
Oct 14 16:18:43 rmgdbp09 gab: [ID 397130 kern.notice] GAB INFO V-15-1-20032 Port v closed
Oct 14 16:18:43 rmgdbp09 vxvm:vxconfigd: [ID 955251 daemon.error] V-5-1-8765 cluster join for node 1 failed: Connection timed out
Oct 14 16:18:47 rmgdbp09 gab: [ID 397130 kern.notice] GAB INFO V-15-1-20032 Port w closed
Oct 14 16:18:47 rmgdbp09 vxvm:vxconfigd: [ID 741513 daemon.notice] V-5-1-7901 CVM_VOLD_STOP command received
Oct 14 16:18:47 rmgdbp09 vxvm:vxconfigd: [ID 778436 daemon.error] V-5-1-4109 -1 returned from volcvm_establish
Oct 14 16:18:47 rmgdbp09 vxvm:vxconfigd: [ID 886039 daemon.warning] V-5-1-4852 cluster_establish: timed out
Oct 14 16:18:47 rmgdbp09 vxvm:vxconfigd: [ID 453237 daemon.error] V-5-1-11178 kernel_fail_join() : join timed out during reconfiguration (12, -1)
Oct 14 16:18:47 rmgdbp09 vxvm:vxconfigd: [ID 565473 daemon.notice] V-5-1-9543 Timeout is not reset: another reconfig in progress
Oct 14 16:19:18 rmgdbp09 Had[4933]: [ID 702911 daemon.notice] VCS ERROR V-16-1-1005 (rmgdbp09) CVMCluster:???:monitor:node - state: out of cluster
Oct 14 16:19:18 rmgdbp09 reason: request to join failed
Oct 14 16:20:18 rmgdbp09 Had[4933]: [ID 702911 daemon.notice] VCS ERROR V-16-1-1005 (rmgdbp09) CVMCluster:???:monitor:node - state: out of cluster
Oct 14 16:20:18 rmgdbp09 reason: request to join failed
Oct 14 16:20:19 rmgdbp09 Had[4933]: [ID 702911 daemon.notice] VCS ERROR V-16-1-13066 (rmgdbp09) Agent is calling clean for resource(cvm_clus) because the resource is not up even after online completed.
Oct 14 16:20:20 rmgdbp09 vxfen: [ID 214757 kern.notice] NOTICE: VCS FEN INFO V-11-1-34 The ioctl VXFEN_IOC_CLUSTSTAT returned 0
Oct 14 16:20:25 rmgdbp09 gab: [ID 316943 kern.notice] GAB INFO V-15-1-20036 Port u gen   2d7c24 membership ;1
Oct 14 16:20:25 rmgdbp09 gab: [ID 674723 kern.notice] GAB INFO V-15-1-20038 Port u gen   2d7c24 k_jeopardy 0
Oct 14 16:20:25 rmgdbp09 gab: [ID 513393 kern.notice] GAB INFO V-15-1-20040 Port u gen   2d7c24    visible 0
Oct 14 16:20:30 rmgdbp09 gab: [ID 316943 kern.notice] GAB INFO V-15-1-20036 Port v gen   2d7c1e membership 01
Oct 14 16:20:36 rmgdbp09 vxvm:vxconfigd: [ID 456610 daemon.notice] V-5-1-7900 CVM_VOLD_CONFIG command received
Oct 14 16:20:36 rmgdbp09 vxvm:vxconfigd: [ID 699813 daemon.notice] V-5-1-7899 CVM_VOLD_CHANGE command received
Oct 14 16:20:36 rmgdbp09 vxvm:vxconfigd: [ID 322665 daemon.notice] V-5-1-7961 establishing cluster
Oct 14 16:20:41 rmgdbp09 gab: [ID 316943 kern.notice] GAB INFO V-15-1-20036 Port w gen   2d7c24 membership 01
Oct 14 16:20:47 rmgdbp09 vxfen: [ID 214757 kern.notice] NOTICE: VCS FEN INFO V-11-1-34 The ioctl VXFEN_IOC_CLUSTSTAT returned 0
Oct 14 16:21:44 rmgdbp09 last message repeated 20 times
Oct 14 16:21:46 rmgdbp09 Had[4933]: [ID 702911 daemon.notice] VCS ERROR V-16-1-13027 (rmgdbp08) Resource(PPTL_VOL_oracle) - monitor procedure did not complete within the expected time.
Oct 14 16:21:46 rmgdbp09 Had[4933]: [ID 702911 daemon.notice] VCS ERROR V-16-1-13027 (rmgdbp08) Resource(PPUB_VOL_arch) - monitor procedure did not complete within the expected time.
Oct 14 16:21:46 rmgdbp09 Had[4933]: [ID 702911 daemon.notice] VCS ERROR V-16-1-13027 (rmgdbp08) Resource(PPUB_VOL_oracle) - monitor procedure did not complete within the expected time.
Oct 14 16:21:46 rmgdbp09 Had[4933]: [ID 702911 daemon.notice] VCS ERROR V-16-1-13027 (rmgdbp08) Resource(PAVS_VOL_oracle) - monitor procedure did not complete within the expected time.
Oct 14 16:21:46 rmgdbp09 Had[4933]: [ID 702911 daemon.notice] VCS ERROR V-16-1-13027 (rmgdbp08) Resource(PPTL_VOL_arch) - monitor procedure did not complete within the expected time.
Oct 14 16:21:46 rmgdbp09 Had[4933]: [ID 702911 daemon.notice] VCS ERROR V-16-1-13027 (rmgdbp08) Resource(PAVS_VOL_arch) - monitor procedure did not complete within the expected time.
Oct 14 16:21:46 rmgdbp09 Had[4933]: [ID 702911 daemon.notice] VCS ERROR V-16-1-13027 (rmgdbp08) Resource(RMAN_VOL01) - monitor procedure did not complete within the expected time.
Oct 14 16:21:50 rmgdbp09 vxfen: [ID 214757 kern.notice] NOTICE: VCS FEN INFO V-11-1-34 The ioctl VXFEN_IOC_CLUSTSTAT returned 0
Oct 14 16:22:10 rmgdbp09 last message repeated 19 times
Oct 14 16:22:24 rmgdbp09 Had[4933]: [ID 702911 daemon.notice] VCS ERROR V-16-1-13027 (rmgdbp08) Resource(srvm_vol_dg) - monitor procedure did not complete within the expected time.
Oct 14 16:23:20 rmgdbp09 sshd[8133]: [ID 800047 auth.error] error: PAM: System error for illegal user sumitsg from 193.88.157.75
Oct 14 16:23:25 rmgdbp09 last message repeated 1 time
Oct 14 16:23:29 rmgdbp09 sshd[8180]: [ID 800047 auth.error] error: ssh_msg_send: write
Oct 14 16:23:51 rmgdbp09 vxio: [ID 535583 kern.warning] WARNING: VxVM vxio V-5-0-164 Failed to join cluster rmgprod, aborting
Oct 14 16:23:51 rmgdbp09 gab: [ID 397130 kern.notice] GAB INFO V-15-1-20032 Port u closed
Oct 14 16:23:51 rmgdbp09 gab: [ID 397130 kern.notice] GAB INFO V-15-1-20032 Port v closed
Oct 14 16:23:51 rmgdbp09 vxvm:vxconfigd: [ID 955251 daemon.error] V-5-1-8765 cluster join for node 1 failed: Connection timed out
Oct 14 16:23:55 rmgdbp09 gab: [ID 397130 kern.notice] GAB INFO V-15-1-20032 Port w closed
Oct 14 16:23:55 rmgdbp09 vxvm:vxconfigd: [ID 741513 daemon.notice] V-5-1-7901 CVM_VOLD_STOP command received
Oct 14 16:23:55 rmgdbp09 vxvm:vxconfigd: [ID 778436 daemon.error] V-5-1-4109 -1 returned from volcvm_establish
Oct 14 16:23:55 rmgdbp09 vxvm:vxconfigd: [ID 886039 daemon.warning] V-5-1-4852 cluster_establish: timed out
Oct 14 16:23:55 rmgdbp09 vxvm:vxconfigd: [ID 453237 daemon.error] V-5-1-11178 kernel_fail_join() : join timed out during reconfiguration (12, -1)
Oct 14 16:23:55 rmgdbp09 vxvm:vxconfigd: [ID 565473 daemon.notice] V-5-1-9543 Timeout is not reset: another reconfig in progress
Oct 14 16:24:26 rmgdbp09 Had[4933]: [ID 702911 daemon.notice] VCS ERROR V-16-1-1005 (rmgdbp09) CVMCluster:???:monitor:node - state: out of cluster
Oct 14 16:24:26 rmgdbp09 reason: request to join failed
Oct 14 16:25:26 rmgdbp09 Had[4933]: [ID 702911 daemon.notice] VCS ERROR V-16-1-1005 (rmgdbp09) CVMCluster:???:monitor:node - state: out of cluster
Oct 14 16:25:26 rmgdbp09 reason: request to join failed
Oct 14 16:25:27 rmgdbp09 Had[4933]: [ID 702911 daemon.notice] VCS ERROR V-16-1-13066 (rmgdbp09) Agent is calling clean for resource(cvm_clus) because the resource is not up even after online completed.
Oct 14 16:25:28 rmgdbp09 Had[4933]: [ID 702911 daemon.notice] VCS ERROR V-16-1-1005 (rmgdbp09) CVMCluster:???:monitor:node - state: out of cluster
Oct 14 16:25:28 rmgdbp09 reason: request to join failed
Oct 14 16:25:29 rmgdbp09 Had[4933]: [ID 702911 daemon.notice] VCS ERROR V-16-1-10303 Resource cvm_clus (Owner: unknown, Group: cvm) is FAULTED (timed out) on sys rmgdbp09
Oct 14 16:25:30 rmgdbp09 arp: [ID 994748 kern.notice] ar_query_xmit: Could not find the ace
Oct 14 16:25:30 rmgdbp09 last message repeated 1 time
Oct 14 16:25:31 rmgdbp09 Had[4933]: [ID 702911 daemon.notice] VCS ERROR V-16-1-10205 Group cvm is faulted on system rmgdbp09
root@rmgdbp09 #

Marianne's picture

Really don't know what to say to you...

Version 4.1 reached EOSL more than a year ago - so I doubt if Support will be able to assist...

You seem to have a hardware error on one of your disks:

WARNING:/pci@8,600000/SUNW,qlc@1/fp@0,0/ssd@w50060e8006cfa802,0 (ssd226):
Oct 14 14:46:17 rmgdbp09        SCSI transport failed: reason 'timeout': giving up

Have you asked you hardware support team to investigate?

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

g_lee's picture

see also https://www-secure.symantec.com/connect/forums/nee...

(duplicate post by OP)

also moved this post from Storage and Clustering Documentation forum to Cluster Server forum

If this post has helped you, please vote or mark as solution

symsonu's picture

Hello Marianne

 

This error disk is not in use....

Yasuhisa Ishikawa's picture

 

Check this tech note and try it.
 
http://www.symantec.com/docs/TECH67749

Authorized Symantec Consultant(ASC) Data Protection in Tokyo, Japan