Video Screencast Help
Scheduled Maintenance: Symantec Connect is scheduled to be down Saturday, April 19 from 10am to 2pm Pacific Standard Time (GMT: 5pm to 9pm) for server migration and upgrade.
Please accept our apologies in advance for any inconvenience this might cause.

Need to know the root cause for filesystem failure in SFS NAS

Created: 30 Jul 2013 • Updated: 31 Jul 2013 | 1 comment

detected an issue with SFS NAS cluster and found that filesystems a cannot be confirmed on df, and general users cannot login to the node.

 

 

SFS event log:->

 

84441) 2013 Jun 19 17:42:39 kyornas051_01 sfsfs_event.network.alert: Node kyornas051_02 went offline.
84442) 2013 Jun 19 17:45:01 kyornas051_01 sfsfs_event.storage.alert.master: Slave node(s) successfully fenced out from cluster.
84443) 2013 Jun 19 18:19:49 kyornas051_01 sfsfs_event.network.info: Node kyornas051_01 joined the cluster.
84444) 2013 Jun 19 18:19:49 192.168.13.159 sfsfs_event.network.info: Node kyornas051_01 joined the cluster.
84445) 2013 Jun 19 18:21:49 kyornas051_01 sfsfs_event.network.info.master: cluster show currentload successful
84446) 2013 Jun 19 18:22:36 kyornas051_01 sfsfs_event.network.info.master: ip addr show successful
84447) 2013 Jun 19 18:23:40 kyornas051_01 sfsfs_event.network.info.master: services show successful
84448) 2013 Jun 19 18:24:20 kyornas051_01 sfsfs_event.network.alert: Interface bond0 is down on system kyornas051_02.
84449) 2013 Jun 19 18:25:09 kyornas051_01 sfsfs_event.network.info.master: IO Fencing Status: Enabled with SCSI3 Persistent Reservations
84450) 2013 Jun 19 18:25:15 kyornas051_01 sfsfs_event.network.info.master: Checked status of IO Fencing on the coordinator disks
84451) 2013 Jun 19 18:25:22 kyornas051_01 sfsfs_event.network.alert: Interface bond0 is up on system kyornas051_02.
84452) 2013 Jun 19 18:25:26 kyornas051_02 sfsfs_event.network.info: Node kyornas051_02 joined the cluster.
84453) 2013 Jun 19 18:27:01 kyornas051_01 sfsfs_event.network.info.master: cluster show currentload successful
84454) 2013 Jun 19 18:27:21 kyornas051_01 sfsfs_event.network.info.master: cluster show currentload successful
84455) 2013 Jun 19 18:27:56 kyornas051_01 sfsfs_event.network.info.master: cluster show currentload successful
84456) 2013 Jun 19 18:35:05 kyornas051_01 sfsfs_event.network.info.master: cluster show currentload successful

==================================================================

 

message file

 

 

message 51_01

2013 Jun 19 17:42:20 kyornas051_01 kernel: LLT INFO V-14-1-10205 link 1 (priveth1) node 1 in trouble
2013 Jun 19 17:42:20 kyornas051_01 kernel: LLT INFO V-14-1-10205 link 0 (priveth0) node 1 in trouble
2013 Jun 19 17:42:26 kyornas051_01 kernel: LLT INFO V-14-1-10032 link 1 (priveth1) node 1 inactive 8 sec (1698350566)
2013 Jun 19 17:42:26 kyornas051_01 kernel: LLT INFO V-14-1-10032 link 0 (priveth0) node 1 inactive 8 sec (1698356198)
2013 Jun 19 17:42:27 kyornas051_01 kernel: LLT INFO V-14-1-10032 link 1 (priveth1) node 1 inactive 9 sec (1698350566)

then it took sometime for llt to expire

013 Jun 19 17:42:33 kyornas051_01 kernel: LLT INFO V-14-1-10509 link 1 (priveth1) node 1 expired
2013 Jun 19 17:42:33 kyornas051_01 kernel: LLT INFO V-14-1-10510 sent hbreq (NULL) on link 0 (priveth0) node 1. 0 more to go.
2013 Jun 19 17:42:33 kyornas051_01 kernel: LLT INFO V-14-1-10032 link 0 (priveth0) node 1 inactive 15 sec (1698356198)
2013 Jun 19 17:42:34 kyornas051_01 kernel: LLT INFO V-14-1-10509 link 0 (priveth0) node 1 expired
2013 Jun 19 17:42:38 kyornas051_01 kernel: GAB INFO V-15-1-20036 Port h gen  1132317 membership 0
2013 Jun 19 17:42:38 kyornas051_01 kernel: GAB INFO V-15-1-20036 Port v gen  113231a membership 0
2013 Jun 19 17:42:38 kyornas051_01 kernel: GAB INFO V-15-1-20036 Port w gen  113231c membership 0
2013 Jun 19 17:42:38 kyornas051_01 kernel: GAB INFO V-15-1-20036 Port a gen  1132305 membership 0
2013 Jun 19 17:42:38 kyornas051_01 kernel: GAB INFO V-15-1-20036 Port b gen  1132314 membership 0
2013 Jun 19 17:42:38 kyornas051_01 kernel: GAB INFO V-15-1-20036 Port f gen  113231e membership 0
2013 Jun 19 17:42:38 kyornas051_01 Had[30829]: VCS INFO V-16-1-10077 Received new cluster membership
2013 Jun 19 17:42:38 kyornas051_01 kernel: VXFEN INFO V-11-1-68 Completed ejection of leaving node(s) from data disks.
2013 Jun 19 17:42:38 kyornas051_01 vxvm:vxconfigd: V-5-1-7899 CVM_VOLD_CHANGE command received
2013 Jun 19 17:42:38 kyornas051_01 vxvm:vxconfigd: V-5-1-13170 Preempting CM NID 1
2013 Jun 19 17:42:38 kyornas051_01 vxvm:vxconfigd: V-5-1-0 Calling join complete
2013 Jun 19 17:42:38 kyornas051_01 vxvm:vxconfigd: V-5-1-8062 master: not a cluster startup
2013 Jun 19 17:42:38 kyornas051_01 vxvm:vxconfigd: V-5-1-10994 join completed for node 0
2013 Jun 19 17:42:38 kyornas051_01 vxvm:vxconfigd: V-5-1-4123 cluster established successfully
2013 Jun 19 17:42:39 kyornas051_01 Had[30829]: VCS ERROR V-16-1-10079 System kyornas051_02 (Node \'1\') is in Down State - Membership: 0x1
2013 Jun 19 17:42:39 kyornas051_01 Had[30829]: VCS ERROR V-16-1-10322 System kyornas051_02 (Node \'1\') changed state from RUNNING to FAULTED
2013 Jun 19 17:42:39 kyornas051_01 sfsfs_event.network.alert: Node kyornas051_02 went offline.

2013 Jun 19 17:42:41 kyornas051_01 kernel: vxfs: msgcnt 617 Phase 0 - /dev/vx/dsk/sfsdg/_nlm_ - Blocking buffer reads for recovery. gencnt 1 primary 0 leavers: 0x2 0x0 0x0 0x0
2013 Jun 19 17:42:41 kyornas051_01 kernel:
2013 Jun 19 17:42:41 kyornas051_01 kernel: vxfs: msgcnt 618 Phase 0 - /dev/vx/dsk/sfsdg/oss2a-segment1 - Blocking buffer reads for recovery. gencnt 1 primary 0 leavers: 0x2 0x0 0x0 0x0
2013 Jun 19 17:42:41 kyornas051_01 kernel:
2013 Jun 19 17:42:41 kyornas051_01 kernel: vxfs: msgcnt 619 Phase 0 - /dev/vx/dsk/sfsdg/oss2a-sgwcg - Blocking buffer reads for recovery. gencnt 1 primary 0 leavers: 0x2 0x0 0x0 0x0

2013 Jun 19 17:42:43 kyornas051_01 kernel: vxfs: msgcnt 695 - /dev/vx/dsk/sfsdg/_nlm_ - PNOLT bitmap in vrt: 0x3 0x0 0x0 0x0, Log replay for per node logs: 0x1 0x0 0x0 0x0
013 Jun 19 17:42:43 kyornas051_01 kernel: vxfs: msgcnt 724 Phase 2 - /dev/vx/dsk/sfsdg/_nlm_ - Buffer reads allowed
2013 Jun 19 17:42:43 kyornas051_01 kernel: vxfs: msgcnt 726 Phase 9 - /dev/vx/dsk/sfsdg/_nlm_ - Set Primary nodeid to 0
2013 Jun 19 17:42:43 kyornas051_01 kernel:
2013 Jun 19 17:42:43 kyornas051_01 kernel: vxfs: msgcnt 727 Phase 10 -/dev/vx/dsk/sfsdg/_nlm_ - Processing extended operations.
2013 Jun 19 17:42:43 kyornas051_01 kernel:

2013 Jun 19 17:43:06 kyornas051_01 kernel: nfsd: last server has exited
2013 Jun 19 17:43:06 kyornas051_01 kernel: nfsd: unexporting all filesystems
2013 Jun 19 17:43:06 kyornas051_01 rpc.mountd: Caught signal 15, un-registering and exiting.
2013 Jun 19 17:43:07 kyornas051_01 AgentFramework[30925]: VCS ERROR V-16-1-13067 Thread(4146072480) Agent is calling clean for resource(nasgw_nfs) because the resource became OFFLINE unexpectedly, on its own.
2013 Jun 19 17:43:07 kyornas051_01 Had[30829]: VCS ERROR V-16-1-13067 (kyornas051_01) Agent is calling clean for resource(nasgw_nfs) because the resource became OFFLINE unexpectedly, on its own.
2013 Jun 19 17:43:08 kyornas051_01 kernel: nfsd: failed to unregister export cache
2013 Jun 19 17:43:08 kyornas051_01 kernel: Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
2013 Jun 19 17:43:08 kyornas051_01 kernel: NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory
2013 Jun 19 17:43:08 kyornas051_01 kernel: NFSD: starting 90-second grace period
2013 Jun 19 17:43:08 kyornas051_01 logger: event_notify.sh will insert a syslog marker now!
2013 Jun 19 17:43:08 kyornas051_01 sfsfs_event.marker.info: =============== Event Log Marker ===============
2013 Jun 19 17:43:14 kyornas051_01 kernel: svc: unknown version (4)
2013 Jun 19 17:43:15 kyornas051_01 logger: 2013 Jun 19 17:42:39 [kyornas051_01,alert,sfs] Node kyornas051_02 went offline.
2013 Jun 19 17:43:37 kyornas051_01 kernel: svc: unknown version (4)
2013 Jun 19 17:43:59 kyornas051_01 kernel: svc: unknown version (4)
2013 Jun 19 17:44:08 kyornas051_01 AgentFramework[30925]: VCS ERROR V-16-1-13006 Thread(4143971232) Resource(nasgw_nfs): clean procedure did not complete within the expected time.
2013 Jun 19 17:44:08 kyornas051_01 Had[30829]: VCS ERROR V-16-1-13006 (kyornas051_01) Resource(nasgw_nfs): clean procedure did not complete within the expected time.
2013 Jun 19 17:44:22 kyornas051_01 kernel: svc: unknown version (4)
2013 Jun 19 17:45:01 kyornas051_01 /usr/sbin/cron[25755]: (root) CMD (LOCKDISABLED=LOCKDISABLED /opt/VRTSnasgw/scripts/rrdtool.pl update >/dev/null 2>&1)
2013 Jun 19 17:45:01 kyornas051_01 /usr/sbin/cron[25773]: (root) CMD (/opt/VRTSnasgw/scripts/sav_event_notify.sh addevents >/dev/null 2>/dev/null)
2013 Jun 19 17:45:01 kyornas051_01 sfsfs_event.storage.alert.master: Slave node(s) successfully fenced out from cluster.
 

=============================================================================================

 

engine log

 

2013/06/19 17:42:33 VCS WARNING V-16-1-11155 LLT heartbeat link status changed. Previous status = 0x3; Current status = 0x1.
2013/06/19 17:42:38 VCS INFO V-16-1-10077 Received new cluster membership
2013/06/19 17:42:38 VCS NOTICE V-16-1-10112 System (kyornas051_01) - Membership: 0x1, DDNA: 0x0
2013/06/19 17:42:38 VCS NOTICE V-16-1-10034 RECONFIG received. VCS waiting for I/O fencing to be completed
2013/06/19 17:42:39 VCS NOTICE V-16-1-10036 I/O fencing completed
2013/06/19 17:42:39 VCS ERROR V-16-1-10079 System kyornas051_02 (Node '1') is in Down State - Membership: 0x1
2013/06/19 17:42:39 VCS ERROR V-16-1-10322 System kyornas051_02 (Node '1') changed state from RUNNING to FAULTED
2013/06/19 17:42:39 VCS NOTICE V-16-1-10446 Group NFS is offline on system kyornas051_02
2013/06/19 17:42:39 VCS NOTICE V-16-1-10446 Group Phantomgroup_priveth0 is offline on system kyornas051_02
2013/06/19 17:42:39 VCS NOTICE V-16-1-10446 Group VIPgroup1 is offline on system kyornas051_02
2013/06/19 17:42:39 VCS NOTICE V-16-1-10446 Group VIPgroup10 is offline on system kyornas051_02
2013/06/19 17:42:39 VCS NOTICE V-16-1-10446 Group VIPgroup11 is offline on system kyornas051_02
2013/06/19 17:42:39 VCS NOTICE V-16-1-10446 Group VIPgroup12 is offline on system kyornas051_02
2013/06/19 17:42:39 VCS NOTICE V-16-1-10446 Group VIPgroup2 is offline on system kyornas051_02
2013/06/19 17:42:39 VCS NOTICE V-16-1-10446 Group VIPgroup3 is offline on system kyornas051_02
2013/06/19 17:42:39 VCS NOTICE V-16-1-10446 Group vrts_vea_cfs_int_cfsmount1 is offline on system kyornas051_02
2013/06/19 17:42:39 VCS NOTICE V-16-1-10446 Group vrts_vea_cfs_int_cfsmount2 is offline on system kyornas051_02
2013/06/19 17:42:39 VCS NOTICE V-16-1-10446 Group vrts_vea_cfs_int_cfsmount3 is offline on system kyornas051_02
2013/06/19 17:42:39 VCS NOTICE V-16-1-10446 Group vrts_vea_cfs_int_cfsmount4 is offline on system kyornas051_02
2013/06/19 17:42:39 VCS NOTICE V-16-1-10446 Group vrts_vea_cfs_int_cfsmount5 is offline on system kyornas051_02
2013/06/19 17:42:39 VCS NOTICE V-16-1-10446 Group vrts_vea_cfs_int_cfsmount6 is offline on system kyornas051_02
2013/06/19 17:42:39 VCS NOTICE V-16-1-10446 Group vrts_vea_cfs_int_cfsmount7 is offline on system kyornas051_02
2013/06/19 17:42:39 VCS NOTICE V-16-1-10446 Group vrts_vea_cfs_int_cfsmount8 is offline on system kyornas051_02
2013/06/19 17:42:39 VCS NOTICE V-16-1-10446 Group vrts_vea_cfs_int_cfsmount9 is offline on system kyornas051_02
2013/06/19 17:42:39 VCS NOTICE V-16-1-10446 Group vrts_vea_cfs_int_cfsmount10 is offline on system kyornas051_02
2013/06/19 17:42:39 VCS NOTICE V-16-1-10446 Group vrts_vea_cfs_int_cfsmount11 is offline on system kyornas051_02
2013/06/19 17:42:39 VCS NOTICE V-16-1-10446 Group vrts_vea_cfs_int_cfsmount12 is offline on system kyornas051_02
2013/06/19 17:42:39 VCS NOTICE V-16-1-10446 Group vrts_vea_cfs_int_cfsmount13 is offline on system kyornas051_02
2013/06/19 17:42:39 VCS NOTICE V-16-1-10446 Group vrts_vea_cfs_int_cfsmount14 is offline on system kyornas051_02
2013/06/19 17:42:39 VCS NOTICE V-16-1-10446 Group vrts_vea_cfs_int_cfsmount15 is offline on system kyornas051_02
2013/06/19 17:42:39 VCS NOTICE V-16-1-10446 Group vrts_vea_cfs_int_cfsmount16 is offline on system kyornas051_02
2013/06/19 17:42:39 VCS NOTICE V-16-1-10446 Group vrts_vea_cfs_int_cfsmount17 is offline on system kyornas051_02
2013/06/19 17:42:39 VCS NOTICE V-16-1-10446 Group vrts_vea_cfs_int_cfsmount18 is offline on system kyornas051_02
2013/06/19 17:42:39 VCS NOTICE V-16-1-10446 Group vrts_vea_cfs_int_cfsmount19 is offline on system kyornas051_02
2013/06/19 17:42:39 VCS NOTICE V-16-1-10446 Group vrts_vea_cfs_int_cfsmount20 is offline on system kyornas051_02
2013/06/19 17:42:39 VCS NOTICE V-16-1-10446 Group vrts_vea_cfs_int_cfsmount21 is offline on system kyornas051_02
2013/06/19 17:42:39 VCS NOTICE V-16-1-10446 Group vrts_vea_cfs_int_cfsmount22 is offline on system kyornas051_02
2013/06/19 17:42:39 VCS NOTICE V-16-1-10446 Group vrts_vea_cfs_int_cfsmount23 is offline on system kyornas051_02
2013/06/19 17:42:39 VCS NOTICE V-16-1-10446 Group vrts_vea_cfs_int_cfsmount24 is offline on system kyornas051_02
2013/06/19 17:42:39 VCS NOTICE V-16-1-10446 Group vrts_vea_cfs_int_cfsmount25 is offline on system kyornas051_02
2013/06/19 17:42:39 VCS NOTICE V-16-1-10446 Group vrts_vea_cfs_int_cfsmount26 is offline on system kyornas051_02
2013/06/19 17:42:39 VCS NOTICE V-16-1-10446 Group vrts_vea_cfs_int_cfsmount27 is offline on system kyornas051_02
2013/06/19 17:42:39 VCS NOTICE V-16-1-10446 Group vrts_vea_cfs_int_cfsmount28 is offline on system kyornas051_02
2013/06/19 17:42:39 VCS NOTICE V-16-1-10446 Group vrts_vea_cfs_int_cfsmount29 is offline on system kyornas051_02
2013/06/19 17:42:39 VCS NOTICE V-16-1-10446 Group CanHostConsole is offline on system kyornas051_02
2013/06/19 17:42:39 VCS NOTICE V-16-1-10446 Group CanHostNLM is offline on system kyornas051_02
2013/06/19 17:42:39 VCS NOTICE V-16-1-10446 Group Phantomgroup_bond0 is offline on system kyornas051_02
2013/06/19 17:42:39 VCS NOTICE V-16-1-10446 Group cvm is offline on system kyornas051_02
2013/06/19 17:42:39 VCS NOTICE V-16-1-10446 Group ReconfigGroup is offline on system kyornas051_02
2013/06/19 17:42:39 VCS INFO V-16-1-10493 Evaluating kyornas051_01 as potential target node for group NFS
2013/06/19 17:42:39 VCS INFO V-16-1-50010 Group NFS is online or faulted on system kyornas051_01
2013/06/19 17:42:39 VCS INFO V-16-1-10493 Evaluating kyornas051_02 as potential target node for group NFS
2013/06/19 17:42:39 VCS INFO V-16-1-10494 System kyornas051_02 not in RUNNING state
2013/06/19 17:42:39 VCS INFO V-16-1-10493 Evaluating kyornas051_01 as potential target node for group Phantomgroup_priveth0
2013/06/19 17:42:39 VCS INFO V-16-1-50010 Group Phantomgroup_priveth0 is online or faulted on system kyornas051_01
2013/06/19 17:42:39 VCS INFO V-16-1-10493 Evaluating kyornas051_02 as potential target node for group Phantomgroup_priveth0
2013/06/19 17:42:39 VCS INFO V-16-1-10494 System kyornas051_02 not in RUNNING state
2013/06/19 17:42:39 VCS INFO V-16-1-10493 Evaluating kyornas051_01 as potential target node for group VIPgroup1
2013/06/19 17:42:39 VCS INFO V-16-1-10493 Evaluating kyornas051_02 as potential target node for group VIPgroup1
2013/06/19 17:42:39 VCS INFO V-16-1-10494 System kyornas051_02 not in RUNNING state
2013/06/19 17:42:39 VCS INFO V-16-1-10493 Evaluating kyornas051_01 as potential target node for group VIPgroup10
2013/06/19 17:42:39 VCS INFO V-16-1-10493 Evaluating kyornas051_02 as potential target node for group VIPgroup10
2013/06/19 17:42:39 VCS INFO V-16-1-10494 System kyornas051_02 not in RUNNING state
2013/06/19 17:42:39 VCS INFO V-16-1-10493 Evaluating kyornas051_01 as potential target node for group VIPgroup11
2013/06/19 17:42:39 VCS INFO V-16-1-10493 Evaluating kyornas051_02 as potential target node for group VIPgroup11
2013/06/19 17:42:39 VCS INFO V-16-1-10494 System kyornas051_02 not in RUNNING state
2013/06/19 17:42:39 VCS INFO V-16-1-10493 Evaluating kyornas051_01 as potential target node for group VIPgroup12
2013/06/19 17:42:39 VCS INFO V-16-1-10493 Evaluating kyornas051_02 as potential target node for group VIPgroup12
2013/06/19 17:42:39 VCS INFO V-16-1-10494 System kyornas051_02 not in RUNNING state
2013/06/19 17:42:39 VCS INFO V-16-1-10493 Evaluating kyornas051_01 as potential target node for group VIPgroup2
2013/06/19 17:42:39 VCS INFO V-16-1-10493 Evaluating kyornas051_02 as potential target node for group VIPgroup2
2013/06/19 17:42:39 VCS INFO V-16-1-10494 System kyornas051_02 not in RUNNING state
2013/06/19 17:42:39 VCS INFO V-16-1-10493 Evaluating kyornas051_01 as potential target node for group VIPgroup3
2013/06/19 17:42:39 VCS INFO V-16-1-10493 Evaluating kyornas051_02 as potential target node for group VIPgroup3
2013/06/19 17:42:39 VCS INFO V-16-1-10494 System kyornas051_02 not in RUNNING state
2013/06/19 17:42:39 VCS INFO V-16-1-10493 Evaluating kyornas051_01 as potential target node for group CanHostConsole
2013/06/19 17:42:39 VCS INFO V-16-1-50010 Group CanHostConsole is online or faulted on system kyornas051_01
2013/06/19 17:42:39 VCS INFO V-16-1-10493 Evaluating kyornas051_02 as potential target node for group CanHostConsole
2013/06/19 17:42:39 VCS INFO V-16-1-10494 System kyornas051_02 not in RUNNING state
2013/06/19 17:42:39 VCS INFO V-16-1-10493 Evaluating kyornas051_01 as potential target node for group CanHostNLM
2013/06/19 17:42:39 VCS INFO V-16-1-50010 Group CanHostNLM is online or faulted on system kyornas051_01
2013/06/19 17:42:39 VCS INFO V-16-1-10493 Evaluating kyornas051_02 as potential target node for group CanHostNLM
2013/06/19 17:42:39 VCS INFO V-16-1-10494 System kyornas051_02 not in RUNNING state
2013/06/19 17:42:39 VCS INFO V-16-1-10493 Evaluating kyornas051_01 as potential target node for group Phantomgroup_bond0
2013/06/19 17:42:39 VCS INFO V-16-1-50010 Group Phantomgroup_bond0 is online or faulted on system kyornas051_01
2013/06/19 17:42:39 VCS INFO V-16-1-10493 Evaluating kyornas051_02 as potential target node for group Phantomgroup_bond0
2013/06/19 17:42:39 VCS INFO V-16-1-10494 System kyornas051_02 not in RUNNING state
2013/06/19 17:42:39 VCS INFO V-16-1-10493 Evaluating kyornas051_01 as potential target node for group ReconfigGroup
2013/06/19 17:42:39 VCS INFO V-16-1-50010 Group ReconfigGroup is online or faulted on system kyornas051_01
2013/06/19 17:42:39 VCS INFO V-16-1-10493 Evaluating kyornas051_02 as potential target node for group ReconfigGroup
2013/06/19 17:42:39 VCS INFO V-16-1-10494 System kyornas051_02 not in RUNNING state
2013/06/19 17:42:39 VCS INFO V-16-6-15025 (kyornas051_01) hatrigger:invoking nfs_preonline
2013/06/19 17:42:39 VCS INFO V-16-6-15076 (kyornas051_01) hatrigger:invoking regular preonline trigger if it exists
2013/06/19 17:42:39 VCS INFO V-16-6-15025 (kyornas051_01) hatrigger:invoking nfs_preonline
2013/06/19 17:42:39 VCS INFO V-16-6-15025 (kyornas051_01) hatrigger:invoking nfs_preonline
2013/06/19 17:42:39 VCS INFO V-16-6-15076 (kyornas051_01) hatrigger:invoking regular preonline trigger if it exists
2013/06/19 17:42:39 VCS INFO V-16-6-15025 (kyornas051_01) hatrigger:invoking nfs_preonline
2013/06/19 17:42:39 VCS INFO V-16-6-15025 (kyornas051_01) hatrigger:invoking nfs_preonline
2013/06/19 17:42:39 VCS INFO V-16-6-15076 (kyornas051_01) hatrigger:invoking regular preonline trigger if it exists
2013/06/19 17:42:39 VCS INFO V-16-6-15076 (kyornas051_01) hatrigger:invoking regular preonline trigger if it exists
2013/06/19 17:42:39 VCS INFO V-16-6-15076 (kyornas051_01) hatrigger:invoking regular preonline trigger if it exists
2013/06/19 17:42:39 VCS INFO V-16-6-15025 (kyornas051_01) hatrigger:invoking nfs_preonline
2013/06/19 17:42:39 VCS INFO V-16-6-15076 (kyornas051_01) hatrigger:invoking regular preonline trigger if it exists
2013/06/19 17:42:40 VCS INFO V-16-1-50135 User root fired command: hagrp -clear CIFSgroup10  from localhost
2013/06/19 17:42:40 VCS INFO V-16-1-50135 User root fired command: hagrp -clear CIFSgroup11  from localhost
2013/06/19 17:42:41 VCS INFO V-16-1-50135 User root fired command: hagrp -clear CIFSgroup3  from localhost
2013/06/19 17:42:42 VCS INFO V-16-1-50135 User root fired command: hagrp -clear CIFSgroup1  from localhost
2013/06/19 17:42:42 VCS INFO V-16-1-50135 User root fired command: hagrp -clear CIFSgroup12  from localhost
2013/06/19 17:42:42 VCS INFO V-16-1-50135 User root fired command: hagrp -clear CIFSgroup2  from localhost
2013/06/19 17:42:44 VCS WARNING V-16-1-11155 LLT heartbeat link status changed. Previous status = 0x1; Current status = 0x0.
2013/06/19 17:42:44 VCS INFO V-16-1-50135 User root fired command: hagrp -online CanHostConsole  kyornas051_01  from localhost
2013/06/19 17:42:44 VCS INFO V-16-1-50135 User root fired command: hagrp -online CanHostNLM  kyornas051_01  from localhost
2013/06/19 17:42:44 VCS INFO V-16-6-15002 (kyornas051_01) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/sysoffline kyornas051_02 FAULTED   successfully
2013/06/19 17:42:53 VCS INFO V-16-1-50135 User root fired command: hagrp -online VIPgroup11  kyornas051_01  from localhost
2013/06/19 17:42:53 VCS NOTICE V-16-1-10166 Initiating manual online of group VIPgroup11 on system kyornas051_01
2013/06/19 17:42:53 VCS NOTICE V-16-1-10460 Clearing start attribute for resource VIP11 of group VIPgroup11 on node kyornas051_02
2013/06/19 17:42:53 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group VIPgroup11 on all nodes
2013/06/19 17:42:53 VCS NOTICE V-16-1-10187 Received -nopre online command for group VIPgroup11 on system kyornas051_01
2013/06/19 17:42:53 VCS NOTICE V-16-1-10301 Initiating Online of Resource VIP11 (Owner: unknown, Group: VIPgroup11) on System kyornas051_01
2013/06/19 17:42:53 VCS INFO V-16-1-50135 User root fired command: hagrp -online VIPgroup10  kyornas051_01  from localhost
2013/06/19 17:42:53 VCS NOTICE V-16-1-10166 Initiating manual online of group VIPgroup10 on system kyornas051_01
2013/06/19 17:42:53 VCS NOTICE V-16-1-10460 Clearing start attribute for resource VIP10 of group VIPgroup10 on node kyornas051_02
2013/06/19 17:42:53 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group VIPgroup10 on all nodes
2013/06/19 17:42:53 VCS NOTICE V-16-1-10187 Received -nopre online command for group VIPgroup10 on system kyornas051_01
2013/06/19 17:42:53 VCS NOTICE V-16-1-10301 Initiating Online of Resource VIP10 (Owner: unknown, Group: VIPgroup10) on System kyornas051_01
2013/06/19 17:42:53 VCS INFO V-16-1-50135 User root fired command: hagrp -online VIPgroup12  kyornas051_01  from localhost
2013/06/19 17:42:53 VCS NOTICE V-16-1-10166 Initiating manual online of group VIPgroup12 on system kyornas051_01
2013/06/19 17:42:53 VCS NOTICE V-16-1-10460 Clearing start attribute for resource VIP12 of group VIPgroup12 on node kyornas051_02
2013/06/19 17:42:53 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group VIPgroup12 on all nodes
2013/06/19 17:42:53 VCS NOTICE V-16-1-10187 Received -nopre online command for group VIPgroup12 on system kyornas051_01
2013/06/19 17:42:53 VCS NOTICE V-16-1-10301 Initiating Online of Resource VIP12 (Owner: unknown, Group: VIPgroup12) on System kyornas051_01
2013/06/19 17:42:53 VCS INFO V-16-6-15002 (kyornas051_01) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/preonline kyornas051_01 VIPgroup11 MANUAL  successfully
2013/06/19 17:42:53 VCS INFO V-16-1-50135 User root fired command: hagrp -online VIPgroup1  kyornas051_01  from localhost
2013/06/19 17:42:53 VCS NOTICE V-16-1-10166 Initiating manual online of group VIPgroup1 on system kyornas051_01
2013/06/19 17:42:53 VCS NOTICE V-16-1-10460 Clearing start attribute for resource VIP1 of group VIPgroup1 on node kyornas051_02
2013/06/19 17:42:53 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group VIPgroup1 on all nodes
2013/06/19 17:42:53 VCS NOTICE V-16-1-10187 Received -nopre online command for group VIPgroup1 on system kyornas051_01
2013/06/19 17:42:53 VCS NOTICE V-16-1-10301 Initiating Online of Resource VIP1 (Owner: unknown, Group: VIPgroup1) on System kyornas051_01
2013/06/19 17:42:53 VCS INFO V-16-1-50135 User root fired command: hagrp -online VIPgroup2  kyornas051_01  from localhost
2013/06/19 17:42:53 VCS NOTICE V-16-1-10166 Initiating manual online of group VIPgroup2 on system kyornas051_01
2013/06/19 17:42:53 VCS NOTICE V-16-1-10460 Clearing start attribute for resource VIP2 of group VIPgroup2 on node kyornas051_02
2013/06/19 17:42:53 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group VIPgroup2 on all nodes
2013/06/19 17:42:53 VCS NOTICE V-16-1-10187 Received -nopre online command for group VIPgroup2 on system kyornas051_01
2013/06/19 17:42:53 VCS NOTICE V-16-1-10301 Initiating Online of Resource VIP2 (Owner: unknown, Group: VIPgroup2) on System kyornas051_01
2013/06/19 17:42:53 VCS INFO V-16-1-50135 User root fired command: hagrp -online VIPgroup3  kyornas051_01  from localhost
2013/06/19 17:42:53 VCS NOTICE V-16-1-10166 Initiating manual online of group VIPgroup3 on system kyornas051_01
2013/06/19 17:42:53 VCS NOTICE V-16-1-10460 Clearing start attribute for resource VIP3 of group VIPgroup3 on node kyornas051_02
2013/06/19 17:42:53 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group VIPgroup3 on all nodes
2013/06/19 17:42:53 VCS NOTICE V-16-1-10187 Received -nopre online command for group VIPgroup3 on system kyornas051_01
2013/06/19 17:42:53 VCS NOTICE V-16-1-10301 Initiating Online of Resource VIP3 (Owner: unknown, Group: VIPgroup3) on System kyornas051_01
2013/06/19 17:42:53 VCS INFO V-16-6-15002 (kyornas051_01) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/preonline kyornas051_01 VIPgroup10 MANUAL  successfully
2013/06/19 17:42:53 VCS INFO V-16-6-15002 (kyornas051_01) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/preonline kyornas051_01 VIPgroup12 MANUAL  successfully
2013/06/19 17:42:53 VCS INFO V-16-6-15002 (kyornas051_01) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/preonline kyornas051_01 VIPgroup1 MANUAL  successfully
2013/06/19 17:42:53 VCS INFO V-16-6-15002 (kyornas051_01) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/preonline kyornas051_01 VIPgroup2 MANUAL  successfully
2013/06/19 17:42:53 VCS INFO V-16-6-15002 (kyornas051_01) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/preonline kyornas051_01 VIPgroup3 MANUAL  successfully
2013/06/19 17:42:59 VCS INFO V-16-1-50135 User root fired command: hares -modify cfsmount28  Primary  kyornas051_01  from localhost
2013/06/19 17:43:01 VCS INFO V-16-1-10298 Resource VIP1 (Owner: unknown, Group: VIPgroup1) is online on kyornas051_01 (VCS initiated)
2013/06/19 17:43:01 VCS NOTICE V-16-1-10447 Group VIPgroup1 is online on system kyornas051_01
2013/06/19 17:43:05 VCS INFO V-16-1-10298 Resource VIP12 (Owner: unknown, Group: VIPgroup12) is online on kyornas051_01 (VCS initiated)
2013/06/19 17:43:05 VCS NOTICE V-16-1-10447 Group VIPgroup12 is online on system kyornas051_01
2013/06/19 17:43:06 VCS INFO V-16-1-50135 User root fired command: hares -clear VIP1  from localhost
2013/06/19 17:43:06 VCS INFO V-16-1-50135 User root fired command: MSG_RES_PROBE nasgw_nfs  kyornas051_01  from localhost
2013/06/19 17:43:06 VCS WARNING V-16-10031-7017 (kyornas051_01) NFS:nasgw_nfs:monitor:nfsd filesystem not mounted, returning offline
2013/06/19 17:43:07 VCS ERROR V-16-2-13067 (kyornas051_01) Agent is calling clean for resource(nasgw_nfs) because the resource became OFFLINE unexpectedly, on its own.
2013/06/19 17:43:10 VCS INFO V-16-1-50135 User root fired command: hares -clear VIP12  from localhost
2013/06/19 17:43:10 VCS INFO V-16-6-15002 (kyornas051_01) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/postonline kyornas051_01 VIPgroup12   successfully
2013/06/19 17:43:12 VCS INFO V-16-1-50135 User root fired command: hares -modify cfsmount22  Primary  kyornas051_01  from localhost
2013/06/19 17:43:18 VCS INFO V-16-1-50135 User root fired command: hares -modify cfsmount29  Primary  kyornas051_01  from localhost
2013/06/19 17:43:22 VCS INFO V-16-1-50135 User root fired command: hares -modify cfsmount23  Primary  kyornas051_01  from localhost
2013/06/19 17:43:37 VCS INFO V-16-1-50135 User root fired command: hares -modify cfsmount24  Primary  kyornas051_01  from localhost
2013/06/19 17:43:50 VCS INFO V-16-1-50135 User root fired command: hares -modify cfsmount25  Primary  kyornas051_01  from localhost
2013/06/19 17:44:00 VCS INFO V-16-2-13001 (kyornas051_01) Resource(NicMonitor_bond0): Output of the completed operation (monitor)
WARNING: pinging broadcast address
2013/06/19 17:44:08 VCS ERROR V-16-2-13006 (kyornas051_01) Resource(nasgw_nfs): clean procedure did not complete within the expected time.
2013/06/19 17:44:53 VCS INFO V-16-6-15002 (kyornas051_01) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/postonline kyornas051_01 VIPgroup1   successfully
2013/06/19 17:45:00 VCS INFO V-16-2-13001 (kyornas051_01) Resource(NicMonitor_bond0): Output of the completed operation (monitor)
WARNING: pinging broadcast address
2013/06/19 17:47:00 VCS INFO V-16-2-13001 (kyornas051_01) Resource(canHostConsole_consoleNIC): Output of the completed operation (monitor)
WARNING: pinging broadcast address
2013/06/19 17:47:54 VCS WARNING V-16-2-13012 (kyornas051_01) Resource(VIP10): online procedure did not complete within the expected time.
2013/06/19 17:47:54 VCS ERROR V-16-2-13065 (kyornas051_01) Agent is calling clean for resource(VIP10) because online did not complete within the expected time.
2013/06/19 17:47:54 VCS WARNING V-16-2-13012 (kyornas051_01) Resource(VIP11): online procedure did not complete within the expected time.
2013/06/19 17:47:54 VCS WARNING V-16-2-13012 (kyornas051_01) Resource(VIP2): online procedure did not complete within the expected time.
2013/06/19 17:47:54 VCS ERROR V-16-2-13065 (kyornas051_01) Agent is calling clean for resource(VIP2) because online did not complete within the expected time.
2013/06/19 17:47:54 VCS WARNING V-16-2-13012 (kyornas051_01) Resource(VIP3): online procedure did not complete within the expected time.
2013/06/19 17:47:54 VCS ERROR V-16-2-13065 (kyornas051_01) Agent is calling clean for resource(VIP11) because online did not complete within the expected time.
2013/06/19 17:47:54 VCS ERROR V-16-2-13065 (kyornas051_01) Agent is calling clean for resource(VIP3) because online did not complete within the expected time.
2013/06/19 17:47:55 VCS INFO V-16-2-13068 (kyornas051_01) Resource(VIP10) - clean completed successfully.
2013/06/19 17:47:55 VCS INFO V-16-2-13068 (kyornas051_01) Resource(VIP11) - clean completed successfully.
2013/06/19 17:47:55 VCS INFO V-16-2-13068 (kyornas051_01) Resource(VIP3) - clean completed successfully.
2013/06/19 17:47:55 VCS INFO V-16-2-13068 (kyornas051_01) Resource(VIP2) - clean completed successfully.
2013/06/19 17:47:55 VCS INFO V-16-2-13072 (kyornas051_01) Resource(VIP10): Agent is retrying online (attempt number 1 of 2).
2013/06/19 17:47:55 VCS INFO V-16-2-13072 (kyornas051_01) Resource(VIP11): Agent is retrying online (attempt number 1 of 2).
2013/06/19 17:47:55 VCS INFO V-16-2-13072 (kyornas051_01) Resource(VIP2): Agent is retrying online (attempt number 1 of 2).
2013/06/19 17:47:55 VCS INFO V-16-2-13072 (kyornas051_01) Resource(VIP3): Agent is retrying online (attempt number 1 of 2).
2013/06/19 17:48:00 VCS INFO V-16-2-13001 (kyornas051_01) Resource(canHostConsole_consoleNIC): Output of the completed operation (monitor)
WARNING: pinging broadcast address
2013/06/19 17:49:00 VCS INFO V-16-2-13001 (kyornas051_01) Resource(canHostConsole_consoleNIC): Output of the completed operation (monitor)
WARNING: pinging broadcast address
2013/06/19 17:50:59 VCS ERROR V-16-2-13027 (kyornas051_01) Resource(NicMonitor_priveth1) - monitor procedure did not complete within the expected time.
2013/06/19 17:50:59 VCS ERROR V-16-2-13027 (kyornas051_01) Resource(nlmmasterNIC) - monitor procedure did not complete within the expected time.
2013/06/19 17:50:59 VCS ERROR V-16-2-13027 (kyornas051_01) Resource(canHostConsole_consoleNIC) - monitor procedure did not complete within the expected time.
2013/06/19 17:50:59 VCS ERROR V-16-2-13027 (kyornas051_01) Resource(NicMonitor_bond0) - monitor procedure did not complete within the expected time.
2013/06/19 17:50:59 VCS ERROR V-16-2-13027 (kyornas051_01) Resource(NicMonitor_pubeth1) - monitor procedure did not complete within the expected time.
2013/06/19 17:50:59 VCS ERROR V-16-2-13027 (kyornas051_01) Resource(NicMonitor_pubeth4) - monitor procedure did not complete within the expected time.
2013/06/19 17:50:59 VCS ERROR V-16-2-13027 (kyornas051_01) Resource(canHostNLM_nlmmaster_device) - monitor procedure did not complete within the expected time.
2013/06/19 17:50:59 VCS ERROR V-16-2-13027 (kyornas051_01) Resource(NicMonitor_pubeth3) - monitor procedure did not complete within the expected time.
2013/06/19 17:50:59 VCS ERROR V-16-2-13027 (kyornas051_01) Resource(NicMonitor_pubeth2) - monitor procedure did not complete within the expected time.
2013/06/19 17:51:01 VCS ERROR V-16-2-13027 (kyornas051_01) Resource(NicMonitor_priveth0) - monitor procedure did not complete within the expected time.
2013/06/19 17:51:08 VCS ERROR V-16-2-13027 (kyornas051_01) Resource(nasgw_smweb) - monitor procedure did not complete within the expected time.
2013/06/19 17:52:01 VCS ERROR V-16-2-13027 (kyornas051_01) Resource(consoleNIC) - monitor procedure did not complete within the expected time.
2013/06/19 17:52:01 VCS ERROR V-16-2-13027 (kyornas051_01) Resource(NicMonitor_pubeth0) - monitor procedure did not complete within the expected time.
2013/06/19 17:52:01 VCS ERROR V-16-2-13027 (kyornas051_01) Resource(phantomNIC_bond0) - monitor procedure did not complete within the expected time.
2013/06/19 17:52:01 VCS ERROR V-16-2-13027 (kyornas051_01) Resource(NicMonitor_pubeth5) - monitor procedure did not complete within the expected time.
2013/06/19 17:52:56 VCS WARNING V-16-2-13012 (kyornas051_01) Resource(VIP2): online procedure did not complete within the expected time.
2013/06/19 17:52:56 VCS WARNING V-16-2-13012 (kyornas051_01) Resource(VIP3): online procedure did not complete within the expected time.
2013/06/19 17:52:56 VCS WARNING V-16-2-13012 (kyornas051_01) Resource(VIP11): online procedure did not complete within the expected time.
2013/06/19 17:52:56 VCS WARNING V-16-2-13012 (kyornas051_01) Resource(VIP10): online procedure did not complete within the expected time.
2013/06/19 17:52:56 VCS ERROR V-16-2-13065 (kyornas051_01) Agent is calling clean for resource(VIP11) because online did not complete within the expected time.
2013/06/19 17:52:56 VCS ERROR V-16-2-13065 (kyornas051_01) Agent is calling clean for resource(VIP10) because online did not complete within the expected time.
2013/06/19 17:52:56 VCS ERROR V-16-2-13065 (kyornas051_01) Agent is calling clean for resource(VIP2) because online did not complete within the expected time.
2013/06/19 17:52:56 VCS ERROR V-16-2-13065 (kyornas051_01) Agent is calling clean for resource(VIP3) because online did not complete within the expected time.
2013/06/19 17:53:57 VCS ERROR V-16-2-13006 (kyornas051_01) Resource(VIP10): clean procedure did not complete within the expected time.
2013/06/19 17:53:57 VCS ERROR V-16-2-13006 (kyornas051_01) Resource(VIP3): clean procedure did not complete within the expected time.
2013/06/19 17:53:57 VCS ERROR V-16-2-13006 (kyornas051_01) Resource(VIP11): clean procedure did not complete within the expected time.
2013/06/19 17:53:57 VCS ERROR V-16-2-13006 (kyornas051_01) Resource(VIP2): clean procedure did not complete within the expected time.
2013/06/19 17:57:08 VCS ERROR V-16-2-13210 (kyornas051_01) Agent is calling clean for resource(nasgw_smweb) because 4 successive invocations of the monitor procedure did not complete within the expected time.
2013/06/19 17:57:09 VCS INFO V-16-2-13068 (kyornas051_01) Resource(nasgw_smweb) - clean completed successfully.
2013/06/19 17:58:10 VCS ERROR V-16-2-13077 (kyornas051_01) Agent is unable to offline resource(nasgw_smweb). Administrative intervention may be required.
2013/06/19 18:18:42 VCS NOTICE V-16-1-11022 VCS engine (had) started
2013/06/19 18:18:42 VCS INFO V-16-1-10196 Cluster logger started
2013/06/19 18:18:42 VCS NOTICE V-16-1-11050 VCS engine version=5.0
2013/06/19 18:18:42 VCS NOTICE V-16-1-11051 VCS engine join version=5.0.40.0
2013/06/19 18:18:42 VCS NOTICE V-16-1-11052 VCS engine pstamp=Veritas-5.0MP4HF2-08/02/10-23:10:00
2013/06/19 18:18:42 VCS NOTICE V-16-1-10114 Opening GAB library
2013/06/19 18:18:42 VCS NOTICE V-16-1-10619 'HAD' starting on: kyornas051_01
2013/06/19 18:18:42 VCS INFO V-16-1-10125 GAB timeout set to 15000 ms
2013/06/19 18:18:46 VCS INFO V-16-1-10077 Received new cluster membership
2013/06/19 18:18:46 VCS NOTICE V-16-1-10112 System (kyornas051_01) - Membership: 0x1, DDNA: 0x0
2013/06/19 18:18:46 VCS NOTICE V-16-1-10086 System kyornas051_01 (Node '0') is in Regular Membership - Membership: 0x1
2013/06/19 18:18:46 VCS NOTICE V-16-1-10322 System kyornas051_01 (Node '0') changed state from CURRENT_DISCOVER_WAIT to LOCAL_BUILD
2013/06/19 18:18:48 VCS NOTICE V-16-1-10032 VxFEN driver available. Local node id=0
2013/06/19 18:18:48 VCS NOTICE V-16-1-10322 System kyornas051_01 (Node '0') changed state from LOCAL_BUILD to RUNNING
2013/06/19 18:18:48 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/Application/ApplicationAgent for resource type Application successfully started at Wed Jun 19 18:18:48 2013
2013/06/19 18:18:48 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/CFSMount/CFSMountAgent for resource type CFSMount successfully started at Wed Jun 19 18:18:48 2013
2013/06/19 18:18:48 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/CFSfsckd/CFSfsckdAgent for resource type CFSfsckd successfully started at Wed Jun 19 18:18:48 2013
2013/06/19 18:18:48 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/CVMCluster/CVMClusterAgent for resource type CVMCluster successfully started at Wed Jun 19 18:18:48 2013
2013/06/19 18:18:48 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/CVMVolDg/CVMVolDgAgent for resource type CVMVolDg successfully started at Wed Jun 19 18:18:48 2013
2013/06/19 18:18:48 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/CVMVxconfigd/CVMVxconfigdAgent for resource type CVMVxconfigd successfully started at Wed Jun 19 18:18:48 2013
2013/06/19 18:18:48 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/IP/IPAgent for resource type IP successfully started at Wed Jun 19 18:18:48 2013
2013/06/19 18:18:48 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/NFS/NFSAgent for resource type NFS successfully started at Wed Jun 19 18:18:48 2013
2013/06/19 18:18:48 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/NFSRestart/NFSRestartAgent for resource type NFSRestart successfully started at Wed Jun 19 18:18:48 2013
2013/06/19 18:18:48 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/NIC/NICAgent for resource type NIC successfully started at Wed Jun 19 18:18:48 2013
2013/06/19 18:18:48 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/NetBios/NetBiosAgent for resource type NetBios successfully started at Wed Jun 19 18:18:48 2013
2013/06/19 18:18:48 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/Phantom/PhantomAgent for resource type Phantom successfully started at Wed Jun 19 18:18:48 2013
2013/06/19 18:18:48 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/Proxy/ProxyAgent for resource type Proxy successfully started at Wed Jun 19 18:18:48 2013
2013/06/19 18:18:48 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/SambaServer/SambaServerAgent for resource type SambaServer successfully started at Wed Jun 19 18:18:48 2013
2013/06/19 18:18:48 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/Share/ShareAgent for resource type Share successfully started at Wed Jun 19 18:18:48 2013
2013/06/19 18:18:48 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/HostMonitor for resource type HostMonitor successfully started at Wed Jun 19 18:18:48 2013

 

==================================================

 

I can see that one of the node 51_02 got fenced out due to issue with hearbeat.

But I could not understand , why there was an issue with filesystems. why they were affected by this as this is a parallel cluster

 

 

 

 

Operating Systems:

Comments 1 CommentJump to latest comment

Kimberley's picture

Moved to VCS forum.

Thanks for participating in the community!