Video Screencast Help
Symantec to Separate Into Two Focused, Industry-Leading Technology Companies. Learn more.

Cluster shutdown SFS01 impacts to NFS services

Created: 21 Sep 2012 • Updated: 02 Oct 2012 | 3 comments
This issue has been solved. See solution.

Hi Experts,

Kindly need your help. We're using Symantec FileStore (SFS) 5.6P3 for NFS services for serving our 3 NFS clients (using Solaris 10). Our SFS servers are running Suse Linux 10 with Cluster (VCS).

SFS01> upgrade show
5.6P3 ENTERPRISE EDITION  (Thu Apr 21 04:09:48 2011),      Installed on Fri Jun 29 17:20:07 JST 2012
SFS01>

We experienced NFS service interuption for about 4 minutes when we did "#cluster shutdown <SFS02>" in oder to switchover all services to other SFS server (SFS01). Usually NFS service interuption would take only 4-6 seconds and no impacts to our application. We need to run "cluster shudown <SFS02>" since we want to replace faulty memory at one of our SFS server.

Here is the VCS (engine_A.log) log during failover services group from SFS02 to SFS01 server. As we can see, there are many messages "CFSMount:cfsmountxxx:offline:Umount Failed". It seems the CFSMount was busy. Then SFS tried to run "FUSER kill" command to release these CFSMount.

- Is anyone know what is the root cause this issue? why SFS took very long to failover the services group to other SFS server?

- Is there any way to avoid this kind of problem instead of using "cluster shutdown <SFS02>" command, especially if we want to reduce the NFS service interuption?

- How to find out that CFSMount is busy because is being used by processes/users? so we could find the best time to run "cluster shutdown <SFS02>" command.

Please find attached the main.cf for your reference.

Thank you

 VCS Log:

2012/09/20 15:13:01 VCS INFO V-16-1-10307 Resource share_180 (Owner: unknown, Group: NFS) is offline on SFS02 (Not initiated by VCS)
2012/09/20 15:13:01 VCS INFO V-16-6-15002 (SFS02) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/resfault SFS02 share_178 ONLINE  successfully
2012/09/20 15:13:04 VCS INFO V-16-6-15002 (SFS02) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/resfault SFS02 share_180 ONLINE  successfully
2012/09/20 15:13:04 VCS ERROR V-16-20011-5503 (SFS02) CFSMount:cfsmount168:offline:Umount Failed : Mount Point : /vx/nmsrc3a-eba_rtt
2012/09/20 15:13:04 VCS ERROR V-16-20011-5503 (SFS02) CFSMount:cfsmount158:offline:Umount Failed : Mount Point : /vx/orasts1-sql_anywhere
2012/09/20 15:13:04 VCS ERROR V-16-20011-5503 (SFS02) CFSMount:cfsmount159:offline:Umount Failed : Mount Point : /vx/orasts1-sybase_iq
2012/09/20 15:13:04 VCS ERROR V-16-20011-5503 (SFS02) CFSMount:cfsmount170:offline:Umount Failed : Mount Point : /vx/nmsrc3a-nms_cosm
2012/09/20 15:13:04 VCS ERROR V-16-20011-5503 (SFS02) CFSMount:cfsmount160:offline:Umount Failed : Mount Point : /vx/orasts1-sw
2012/09/20 15:13:04 VCS ERROR V-16-20011-5503 (SFS02) CFSMount:cfsmount162:offline:Umount Failed : Mount Point : /vx/nmsrc3a-segment1
2012/09/20 15:13:04 VCS ERROR V-16-20011-5503 (SFS02) CFSMount:cfsmount167:offline:Umount Failed : Mount Point : /vx/nmsrc3a-eba_ebss
2012/09/20 15:13:04 VCS NOTICE V-16-20011-5510 (SFS02) CFSMount:cfsmount168:offline:Attempting fuser TERM : Mount Point : /vx/nmsrc3a-eba_rtt
2012/09/20 15:13:04 VCS NOTICE V-16-20011-5510 (SFS02) CFSMount:cfsmount170:offline:Attempting fuser TERM : Mount Point : /vx/nmsrc3a-nms_cosm
2012/09/20 15:13:04 VCS NOTICE V-16-20011-5510 (SFS02) CFSMount:cfsmount159:offline:Attempting fuser TERM : Mount Point : /vx/orasts1-sybase_iq
2012/09/20 15:13:04 VCS NOTICE V-16-20011-5510 (SFS02) CFSMount:cfsmount160:offline:Attempting fuser TERM : Mount Point : /vx/orasts1-sw
2012/09/20 15:13:04 VCS NOTICE V-16-20011-5510 (SFS02) CFSMount:cfsmount158:offline:Attempting fuser TERM : Mount Point : /vx/orasts1-sql_anywhere
2012/09/20 15:13:04 VCS NOTICE V-16-20011-5510 (SFS02) CFSMount:cfsmount167:offline:Attempting fuser TERM : Mount Point : /vx/nmsrc3a-eba_ebss
2012/09/20 15:13:04 VCS NOTICE V-16-20011-5510 (SFS02) CFSMount:cfsmount162:offline:Attempting fuser TERM : Mount Point : /vx/nmsrc3a-segment1
2012/09/20 15:13:04 VCS ERROR V-16-20011-5503 (SFS02) CFSMount:cfsmount172:offline:Umount Failed : Mount Point : /vx/nmsrc3a-ddc_data
2012/09/20 15:13:04 VCS NOTICE V-16-20011-5510 (SFS02) CFSMount:cfsmount172:offline:Attempting fuser TERM : Mount Point : /vx/nmsrc3a-ddc_data
2012/09/20 15:13:04 VCS ERROR V-16-20011-5503 (SFS02) CFSMount:cfsmount173:offline:Umount Failed : Mount Point : /vx/nmsrc3a-home
2012/09/20 15:13:04 VCS NOTICE V-16-20011-5510 (SFS02) CFSMount:cfsmount173:offline:Attempting fuser TERM : Mount Point : /vx/nmsrc3a-home
2012/09/20 15:13:08 VCS WARNING V-16-10031-11502 (SFS02) Share:share_184:monitor:Could not access path:/vx/nmsrc2a-eba_ebsw.
2012/09/20 15:13:08 VCS ERROR V-16-2-13067 (SFS02) Agent is calling clean for resource(share_184) because the resource became OFFLINE unexpectedly, on its own.
2012/09/20 15:13:09 VCS INFO V-16-2-13068 (SFS02) Resource(share_184) - clean completed successfully.
2012/09/20 15:13:09 VCS INFO V-16-1-10307 Resource share_184 (Owner: unknown, Group: NFS) is offline on SFS02 (Not initiated by VCS)
2012/09/20 15:13:10 VCS WARNING V-16-10031-11502 (SFS02) Share:share_185:monitor:Could not access path:/vx/nmsrc2a-eba_ebsg.
2012/09/20 15:13:10 VCS WARNING V-16-10031-11502 (SFS02) Share:share_186:monitor:Could not access path:/vx/nmsrc2a-eba_rede.
2012/09/20 15:13:10 VCS ERROR V-16-2-13067 (SFS02) Agent is calling clean for resource(share_185) because the resource became OFFLINE unexpectedly, on its own.
2012/09/20 15:13:10 VCS ERROR V-16-2-13067 (SFS02) Agent is calling clean for resource(share_186) because the resource became OFFLINE unexpectedly, on its own.
2012/09/20 15:13:11 VCS INFO V-16-2-13068 (SFS02) Resource(share_185) - clean completed successfully.
2012/09/20 15:13:11 VCS INFO V-16-2-13068 (SFS02) Resource(share_186) - clean completed successfully.
2012/09/20 15:13:11 VCS INFO V-16-1-10307 Resource share_186 (Owner: unknown, Group: NFS) is offline on SFS02 (Not initiated by VCS)
2012/09/20 15:13:11 VCS INFO V-16-1-10307 Resource share_185 (Owner: unknown, Group: NFS) is offline on SFS02 (Not initiated by VCS)
2012/09/20 15:13:12 VCS WARNING V-16-10031-11502 (SFS02) Share:share_187:monitor:Could not access path:/vx/nmsrc2a-eba_rtt.
2012/09/20 15:13:12 VCS ERROR V-16-2-13067 (SFS02) Agent is calling clean for resource(share_187) because the resource became OFFLINE unexpectedly, on its own.
2012/09/20 15:13:12 VCS INFO V-16-6-15002 (SFS02) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/resfault SFS02 share_184 ONLINE  successfully
2012/09/20 15:13:13 VCS INFO V-16-2-13068 (SFS02) Resource(share_187) - clean completed successfully.
2012/09/20 15:13:13 VCS INFO V-16-1-10307 Resource share_187 (Owner: unknown, Group: NFS) is offline on SFS02 (Not initiated by VCS)
2012/09/20 15:13:14 VCS INFO V-16-6-15002 (SFS02) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/resfault SFS02 share_185 ONLINE  successfully
2012/09/20 15:13:14 VCS INFO V-16-6-15002 (SFS02) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/resfault SFS02 share_186 ONLINE  successfully
2012/09/20 15:13:16 VCS INFO V-16-6-15002 (SFS02) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/resfault SFS02 share_187 ONLINE  successfully
2012/09/20 15:13:22 VCS INFO V-16-1-50135 User root fired command: hares -modify cfsmount166  Primary  SFS01  from localhost
2012/09/20 15:13:23 VCS INFO V-16-1-50135 User root fired command: hares -modify cfsmount125  Primary  SFS01  from localhost
2012/09/20 15:13:23 VCS NOTICE V-16-20011-5510 (SFS02) CFSMount:cfsmount168:offline:Attempting fuser TERM : Mount Point : /vx/nmsrc3a-eba_rtt
2012/09/20 15:13:23 VCS NOTICE V-16-20011-5510 (SFS02) CFSMount:cfsmount167:offline:Attempting fuser TERM : Mount Point : /vx/nmsrc3a-eba_ebss
2012/09/20 15:13:23 VCS NOTICE V-16-20011-5510 (SFS02) CFSMount:cfsmount158:offline:Attempting fuser TERM : Mount Point : /vx/orasts1-sql_anywhere
2012/09/20 15:13:23 VCS NOTICE V-16-20011-5510 (SFS02) CFSMount:cfsmount159:offline:Attempting fuser TERM : Mount Point : /vx/orasts1-sybase_iq
2012/09/20 15:13:23 VCS NOTICE V-16-20011-5510 (SFS02) CFSMount:cfsmount162:offline:Attempting fuser TERM : Mount Point : /vx/nmsrc3a-segment1
2012/09/20 15:13:23 VCS NOTICE V-16-20011-5510 (SFS02) CFSMount:cfsmount170:offline:Attempting fuser TERM : Mount Point : /vx/nmsrc3a-nms_cosm
2012/09/20 15:13:23 VCS NOTICE V-16-20011-5510 (SFS02) CFSMount:cfsmount160:offline:Attempting fuser TERM : Mount Point : /vx/orasts1-sw
2012/09/20 15:13:23 VCS NOTICE V-16-20011-5510 (SFS02) CFSMount:cfsmount172:offline:Attempting fuser TERM : Mount Point : /vx/nmsrc3a-ddc_data
2012/09/20 15:13:23 VCS NOTICE V-16-20011-5510 (SFS02) CFSMount:cfsmount173:offline:Attempting fuser TERM : Mount Point : /vx/nmsrc3a-home
2012/09/20 15:13:29 VCS INFO V-16-1-50135 User root fired command: hares -modify cfsmount1  Primary  SFS01  from localhost
2012/09/20 15:13:29 VCS INFO V-16-1-50135 User root fired command: hares -modify cfsmount126  Primary  SFS01  from localhost
2012/09/20 15:13:34 VCS ERROR V-16-10031-7027 (SFS01) NFS:nasgw_nfs:monitor:Daemon [mountd] not running version [1] on TCP port
2012/09/20 15:13:38 VCS WARNING V-16-10031-11502 (SFS02) Share:ishare_126:monitor:Could not access path:/vx/nmsrc2a-segment1.
2012/09/20 15:13:38 VCS WARNING V-16-10031-11502 (SFS02) Share:ishare_129:monitor:Could not access path:/vx/nmsrc2a-eba_ebsg.
2012/09/20 15:13:38 VCS WARNING V-16-10031-11502 (SFS02) Share:ishare_130:monitor:Could not access path:/vx/nmsrc2a-eba_ebsw.
2012/09/20 15:13:38 VCS WARNING V-16-10031-11502 (SFS02) Share:ishare_128:monitor:Could not access path:/vx/nmsrc2a-eba_rede.
2012/09/20 15:13:38 VCS ERROR V-16-2-13067 (SFS02) Agent is calling clean for resource(ishare_129) because the resource became OFFLINE unexpectedly, on its own.
2012/09/20 15:13:38 VCS ERROR V-16-2-13067 (SFS02) Agent is calling clean for resource(ishare_126) because the resource became OFFLINE unexpectedly, on its own.
2012/09/20 15:13:38 VCS ERROR V-16-2-13067 (SFS02) Agent is calling clean for resource(ishare_128) because the resource became OFFLINE unexpectedly, on its own.
2012/09/20 15:13:38 VCS ERROR V-16-2-13067 (SFS02) Agent is calling clean for resource(ishare_130) because the resource became OFFLINE unexpectedly, on its own.
2012/09/20 15:13:39 VCS INFO V-16-2-13068 (SFS02) Resource(ishare_129) - clean completed successfully.
2012/09/20 15:13:39 VCS INFO V-16-2-13068 (SFS02) Resource(ishare_126) - clean completed successfully.
2012/09/20 15:13:39 VCS INFO V-16-2-13068 (SFS02) Resource(ishare_128) - clean completed successfully.
2012/09/20 15:13:39 VCS INFO V-16-2-13068 (SFS02) Resource(ishare_130) - clean completed successfully.
2012/09/20 15:13:39 VCS WARNING V-16-10031-11502 (SFS02) Share:ishare_132:monitor:Could not access path:/vx/nmsrc2a-eba_rtt.
2012/09/20 15:13:39 VCS ERROR V-16-2-13067 (SFS02) Agent is calling clean for resource(ishare_132) because the resource became OFFLINE unexpectedly, on its own.
2012/09/20 15:13:39 VCS INFO V-16-1-10307 Resource ishare_130 (Owner: unknown, Group: NFS) is offline on SFS02 (Not initiated by VCS)
2012/09/20 15:13:39 VCS INFO V-16-1-10307 Resource ishare_128 (Owner: unknown, Group: NFS) is offline on SFS02 (Not initiated by VCS)
2012/09/20 15:13:39 VCS INFO V-16-1-10307 Resource ishare_126 (Owner: unknown, Group: NFS) is offline on SFS02 (Not initiated by VCS)
2012/09/20 15:13:39 VCS INFO V-16-1-10307 Resource ishare_129 (Owner: unknown, Group: NFS) is offline on SFS02 (Not initiated by VCS)
2012/09/20 15:13:40 VCS INFO V-16-2-13068 (SFS02) Resource(ishare_132) - clean completed successfully.
2012/09/20 15:13:40 VCS INFO V-16-1-10307 Resource ishare_132 (Owner: unknown, Group: NFS) is offline on SFS02 (Not initiated by VCS)
2012/09/20 15:13:40 VCS NOTICE V-16-20011-5510 (SFS02) CFSMount:cfsmount158:offline:Attempting fuser KILL : Mount Point : /vx/orasts1-sql_anywhere
2012/09/20 15:13:40 VCS NOTICE V-16-20011-5510 (SFS02) CFSMount:cfsmount160:offline:Attempting fuser KILL : Mount Point : /vx/orasts1-sw
2012/09/20 15:13:40 VCS NOTICE V-16-20011-5510 (SFS02) CFSMount:cfsmount162:offline:Attempting fuser KILL : Mount Point : /vx/nmsrc3a-segment1
2012/09/20 15:13:40 VCS NOTICE V-16-20011-5510 (SFS02) CFSMount:cfsmount168:offline:Attempting fuser KILL : Mount Point : /vx/nmsrc3a-eba_rtt
2012/09/20 15:13:40 VCS NOTICE V-16-20011-5510 (SFS02) CFSMount:cfsmount167:offline:Attempting fuser KILL : Mount Point : /vx/nmsrc3a-eba_ebss
2012/09/20 15:13:40 VCS NOTICE V-16-20011-5510 (SFS02) CFSMount:cfsmount170:offline:Attempting fuser KILL : Mount Point : /vx/nmsrc3a-nms_cosm
2012/09/20 15:13:40 VCS NOTICE V-16-20011-5510 (SFS02) CFSMount:cfsmount159:offline:Attempting fuser KILL : Mount Point : /vx/orasts1-sybase_iq
2012/09/20 15:13:40 VCS NOTICE V-16-20011-5510 (SFS02) CFSMount:cfsmount172:offline:Attempting fuser KILL : Mount Point : /vx/nmsrc3a-ddc_data
2012/09/20 15:13:40 VCS NOTICE V-16-20011-5510 (SFS02) CFSMount:cfsmount173:offline:Attempting fuser KILL : Mount Point : /vx/nmsrc3a-home
2012/09/20 15:13:41 VCS INFO V-16-1-50135 User root fired command: hares -modify cfsmount133  Primary  SFS01  from localhost
2012/09/20 15:13:42 VCS INFO V-16-6-15002 (SFS02) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/resfault SFS02 ishare_126 ONLINE  successfully
2012/09/20 15:13:42 VCS INFO V-16-6-15002 (SFS02) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/resfault SFS02 ishare_128 ONLINE  successfully
2012/09/20 15:13:42 VCS INFO V-16-6-15002 (SFS02) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/resfault SFS02 ishare_129 ONLINE  successfully
2012/09/20 15:13:42 VCS INFO V-16-6-15002 (SFS02) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/resfault SFS02 ishare_130 ONLINE  successfully
2012/09/20 15:13:43 VCS INFO V-16-6-15002 (SFS02) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/resfault SFS02 ishare_132 ONLINE  successfully
2012/09/20 15:13:44 VCS WARNING V-16-10031-11502 (SFS02) Share:share_207:monitor:Could not access path:/vx/nmsrc2a-eba_rede.
2012/09/20 15:13:44 VCS WARNING V-16-10031-11502 (SFS02) Share:share_208:monitor:Could not access path:/vx/nmsrc2a-eba_rtt.
2012/09/20 15:13:44 VCS ERROR V-16-2-13067 (SFS02) Agent is calling clean for resource(share_208) because the resource became OFFLINE unexpectedly, on its own.
2012/09/20 15:13:44 VCS ERROR V-16-2-13067 (SFS02) Agent is calling clean for resource(share_207) because the resource became OFFLINE unexpectedly, on its own.
2012/09/20 15:13:45 VCS INFO V-16-2-13068 (SFS02) Resource(share_207) - clean completed successfully.
2012/09/20 15:13:45 VCS INFO V-16-2-13068 (SFS02) Resource(share_208) - clean completed successfully.
2012/09/20 15:13:45 VCS INFO V-16-1-10307 Resource share_208 (Owner: unknown, Group: NFS) is offline on SFS02 (Not initiated by VCS)
2012/09/20 15:13:45 VCS INFO V-16-1-10307 Resource share_207 (Owner: unknown, Group: NFS) is offline on SFS02 (Not initiated by VCS)
2012/09/20 15:13:47 VCS INFO V-16-1-50135 User root fired command: hares -modify cfsmount163  Primary  SFS01  from localhost
2012/09/20 15:13:47 VCS INFO V-16-1-50135 User root fired command: hares -modify cfsmount165  Primary  SFS01  from localhost
2012/09/20 15:13:47 VCS INFO V-16-1-50135 User root fired command: hares -modify cfsmount164  Primary  SFS01  from localhost
2012/09/20 15:13:47 VCS INFO V-16-1-50135 User root fired command: hares -modify cfsmount161  Primary  SFS01  from localhost
2012/09/20 15:13:48 VCS INFO V-16-6-15002 (SFS02) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/resfault SFS02 share_207 ONLINE  successfully
2012/09/20 15:13:48 VCS INFO V-16-6-15002 (SFS02) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/resfault SFS02 share_208 ONLINE  successfully
2012/09/20 15:13:49 VCS WARNING V-16-10031-11502 (SFS02) Share:share_143:monitor:Could not access path:/vx/nmsrc2a-eba_rtt.
2012/09/20 15:13:49 VCS WARNING V-16-10031-11502 (SFS02) Share:share_140:monitor:Could not access path:/vx/nmsrc2a-eba_ebsg.
2012/09/20 15:13:49 VCS WARNING V-16-10031-11502 (SFS02) Share:share_139:monitor:Could not access path:/vx/nmsrc2a-eba_rede.
2012/09/20 15:13:49 VCS WARNING V-16-10031-11502 (SFS02) Share:share_141:monitor:Could not access path:/vx/nmsrc2a-eba_ebsw.
2012/09/20 15:13:49 VCS ERROR V-16-2-13067 (SFS02) Agent is calling clean for resource(share_139) because the resource became OFFLINE unexpectedly, on its own.
2012/09/20 15:13:49 VCS ERROR V-16-2-13067 (SFS02) Agent is calling clean for resource(share_140) because the resource became OFFLINE unexpectedly, on its own.
2012/09/20 15:13:49 VCS ERROR V-16-2-13067 (SFS02) Agent is calling clean for resource(share_143) because the resource became OFFLINE unexpectedly, on its own.
2012/09/20 15:13:49 VCS ERROR V-16-2-13067 (SFS02) Agent is calling clean for resource(share_141) because the resource became OFFLINE unexpectedly, on its own.
2012/09/20 15:13:50 VCS INFO V-16-2-13068 (SFS02) Resource(share_139) - clean completed successfully.
2012/09/20 15:13:50 VCS INFO V-16-2-13068 (SFS02) Resource(share_140) - clean completed successfully.
2012/09/20 15:13:50 VCS INFO V-16-2-13068 (SFS02) Resource(share_143) - clean completed successfully.
2012/09/20 15:13:50 VCS INFO V-16-2-13068 (SFS02) Resource(share_141) - clean completed successfully.
2012/09/20 15:13:50 VCS INFO V-16-1-10307 Resource share_141 (Owner: unknown, Group: NFS) is offline on SFS02 (Not initiated by VCS)
2012/09/20 15:13:50 VCS INFO V-16-1-10307 Resource share_139 (Owner: unknown, Group: NFS) is offline on SFS02 (Not initiated by VCS)
2012/09/20 15:13:50 VCS INFO V-16-1-10307 Resource share_140 (Owner: unknown, Group: NFS) is offline on SFS02 (Not initiated by VCS)
2012/09/20 15:13:50 VCS INFO V-16-1-10307 Resource share_143 (Owner: unknown, Group: NFS) is offline on SFS02 (Not initiated by VCS)
2012/09/20 15:13:53 VCS INFO V-16-6-15002 (SFS02) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/resfault SFS02 share_143 ONLINE  successfully
2012/09/20 15:13:53 VCS INFO V-16-6-15002 (SFS02) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/resfault SFS02 share_140 ONLINE  successfully
2012/09/20 15:13:53 VCS INFO V-16-6-15002 (SFS02) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/resfault SFS02 share_139 ONLINE  successfully
2012/09/20 15:13:53 VCS INFO V-16-6-15002 (SFS02) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/resfault SFS02 share_141 ONLINE  successfully
2012/09/20 15:13:55 VCS NOTICE V-16-20011-5510 (SFS02) CFSMount:cfsmount158:offline:Attempting fuser KILL : Mount Point : /vx/orasts1-sql_anywhere
2012/09/20 15:13:55 VCS NOTICE V-16-20011-5510 (SFS02) CFSMount:cfsmount162:offline:Attempting fuser KILL : Mount Point : /vx/nmsrc3a-segment1
2012/09/20 15:13:55 VCS NOTICE V-16-20011-5510 (SFS02) CFSMount:cfsmount170:offline:Attempting fuser KILL : Mount Point : /vx/nmsrc3a-nms_cosm
2012/09/20 15:13:55 VCS NOTICE V-16-20011-5510 (SFS02) CFSMount:cfsmount168:offline:Attempting fuser KILL : Mount Point : /vx/nmsrc3a-eba_rtt
2012/09/20 15:13:55 VCS NOTICE V-16-20011-5510 (SFS02) CFSMount:cfsmount159:offline:Attempting fuser KILL : Mount Point : /vx/orasts1-sybase_iq
2012/09/20 15:13:55 VCS NOTICE V-16-20011-5510 (SFS02) CFSMount:cfsmount160:offline:Attempting fuser KILL : Mount Point : /vx/orasts1-sw
2012/09/20 15:13:55 VCS NOTICE V-16-20011-5510 (SFS02) CFSMount:cfsmount167:offline:Attempting fuser KILL : Mount Point : /vx/nmsrc3a-eba_ebss
2012/09/20 15:13:55 VCS NOTICE V-16-20011-5510 (SFS02) CFSMount:cfsmount172:offline:Attempting fuser KILL : Mount Point : /vx/nmsrc3a-ddc_data
2012/09/20 15:13:55 VCS NOTICE V-16-20011-5510 (SFS02) CFSMount:cfsmount173:offline:Attempting fuser KILL : Mount Point : /vx/nmsrc3a-home
2012/09/20 15:14:08 VCS NOTICE V-16-20011-5504 (SFS02) CFSMount:cfsmount158:offline:Force Umount : Mount Point : /vx/orasts1-sql_anywhere
2012/09/20 15:14:08 VCS NOTICE V-16-20011-5504 (SFS02) CFSMount:cfsmount162:offline:Force Umount : Mount Point : /vx/nmsrc3a-segment1
2012/09/20 15:14:08 VCS NOTICE V-16-20011-5504 (SFS02) CFSMount:cfsmount168:offline:Force Umount : Mount Point : /vx/nmsrc3a-eba_rtt
2012/09/20 15:14:08 VCS NOTICE V-16-20011-5504 (SFS02) CFSMount:cfsmount167:offline:Force Umount : Mount Point : /vx/nmsrc3a-eba_ebss
2012/09/20 15:14:08 VCS NOTICE V-16-20011-5504 (SFS02) CFSMount:cfsmount159:offline:Force Umount : Mount Point : /vx/orasts1-sybase_iq
2012/09/20 15:14:08 VCS NOTICE V-16-20011-5504 (SFS02) CFSMount:cfsmount160:offline:Force Umount : Mount Point : /vx/orasts1-sw
2012/09/20 15:14:08 VCS NOTICE V-16-20011-5504 (SFS02) CFSMount:cfsmount170:offline:Force Umount : Mount Point : /vx/nmsrc3a-nms_cosm
2012/09/20 15:14:08 VCS NOTICE V-16-20011-5504 (SFS02) CFSMount:cfsmount172:offline:Force Umount : Mount Point : /vx/nmsrc3a-ddc_data
2012/09/20 15:14:08 VCS NOTICE V-16-20011-5504 (SFS02) CFSMount:cfsmount173:offline:Force Umount : Mount Point : /vx/nmsrc3a-home
2012/09/20 15:14:09 VCS INFO V-16-6-15002 (SFS01) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/postonline SFS01 NLMGroup   successfully
2012/09/20 15:14:19 VCS ERROR V-16-20011-5503 (SFS02) CFSMount:cfsmount174:offline:Umount Failed : Mount Point : /vx/nmsrc3a-ora_pm_vol
2012/09/20 15:14:19 VCS NOTICE V-16-20011-5510 (SFS02) CFSMount:cfsmount174:offline:Attempting fuser TERM : Mount Point : /vx/nmsrc3a-ora_pm_vol
2012/09/20 15:14:19 VCS ERROR V-16-20011-5503 (SFS02) CFSMount:cfsmount38:offline:Umount Failed : Mount Point : /vx/nms1a-segment1
2012/09/20 15:14:19 VCS ERROR V-16-20011-5503 (SFS02) CFSMount:cfsmount39:offline:Umount Failed : Mount Point : /vx/nms1a-sgwcg
2012/09/20 15:14:19 VCS ERROR V-16-20011-5503 (SFS02) CFSMount:cfsmount40:offline:Umount Failed : Mount Point : /vx/nms1a-eba_rede
2012/09/20 15:14:19 VCS NOTICE V-16-20011-5510 (SFS02) CFSMount:cfsmount39:offline:Attempting fuser TERM : Mount Point : /vx/nms1a-sgwcg
2012/09/20 15:14:19 VCS NOTICE V-16-20011-5510 (SFS02) CFSMount:cfsmount38:offline:Attempting fuser TERM : Mount Point : /vx/nms1a-segment1
2012/09/20 15:14:19 VCS NOTICE V-16-20011-5510 (SFS02) CFSMount:cfsmount40:offline:Attempting fuser TERM : Mount Point : /vx/nms1a-eba_rede
2012/09/20 15:14:19 VCS ERROR V-16-20011-5503 (SFS02) CFSMount:cfsmount41:offline:Umount Failed : Mount Point : /vx/nms1a-eba_ebsg
2012/09/20 15:14:19 VCS ERROR V-16-20011-5503 (SFS02) CFSMount:cfsmount42:offline:Umount Failed : Mount Point : /vx/nms1a-eba_ebsw
2012/09/20 15:14:19 VCS NOTICE V-16-20011-5510 (SFS02) CFSMount:cfsmount41:offline:Attempting fuser TERM : Mount Point : /vx/nms1a-eba_ebsg
2012/09/20 15:14:19 VCS NOTICE V-16-20011-5510 (SFS02) CFSMount:cfsmount42:offline:Attempting fuser TERM : Mount Point : /vx/nms1a-eba_ebsw
2012/09/20 15:14:19 VCS ERROR V-16-20011-5503 (SFS02) CFSMount:cfsmount43:offline:Umount Failed : Mount Point : /vx/nms1a-eba_ebss
2012/09/20 15:14:19 VCS NOTICE V-16-20011-5510 (SFS02) CFSMount:cfsmount43:offline:Attempting fuser TERM : Mount Point : /vx/nms1a-eba_ebss
2012/09/20 15:14:19 VCS ERROR V-16-20011-5503 (SFS02) CFSMount:cfsmount44:offline:Umount Failed : Mount Point : /vx/nms1a-eba_rtt
2012/09/20 15:14:19 VCS NOTICE V-16-20011-5510 (SFS02) CFSMount:cfsmount44:offline:Attempting fuser TERM : Mount Point : /vx/nms1a-eba_rtt
2012/09/20 15:14:20 VCS ERROR V-16-20011-5503 (SFS02) CFSMount:cfsmount45:offline:Umount Failed : Mount Point : /vx/nms1a-pm_storage
2012/09/20 15:14:20 VCS NOTICE V-16-20011-5510 (SFS02) CFSMount:cfsmount45:offline:Attempting fuser TERM : Mount Point : /vx/nms1a-pm_storage
2012/09/20 15:14:34 VCS ERROR V-16-10031-7027 (SFS01) NFS:nasgw_nfs:monitor:Daemon [mountd] not running version [1] on TCP port
2012/09/20 15:14:35 VCS ERROR V-16-2-13066 (SFS01) Agent is calling clean for resource(nasgw_nfs) because the resource is not up even after online completed.
2012/09/20 15:14:36 VCS WARNING V-16-10031-7002 (SFS01) NFS:nasgw_nfs:clean:Sending SIGKILL signal to nfsd (26022 26023 26024 26025 26026 26028 26029 26030 26031 26032 26033 26034 26035 26036 26037 26038 26039 26040 26041 26042 26043 26044 26045 26046 26047 26048 26049 26050 26051 26052 26053 26054 26055 26056 26057 26058 26059 26060 26061 26062 26063 26064 26065 26066 26067 26068 26069 26070 26071 26072 26073 26074 26075 26076 26077 26078 26079 26080 26081 26082 26083 26084 26085 26086 26087 26088 26089 26090 26091 26092 26093 26094 26095 26096 26097 26098 26099 26100 26101 26102 26103 26104 26105 26106 26107 26108 26109 26110 26111 26112 26113 26114 26115 26116 26117 26118).
2012/09/20 15:14:38 VCS NOTICE V-16-20011-5510 (SFS02) CFSMount:cfsmount174:offline:Attempting fuser TERM : Mount Point : /vx/nmsrc3a-ora_pm_vol
2012/09/20 15:14:38 VCS NOTICE V-16-20011-5510 (SFS02) CFSMount:cfsmount39:offline:Attempting fuser TERM : Mount Point : /vx/nms1a-sgwcg
2012/09/20 15:14:38 VCS NOTICE V-16-20011-5510 (SFS02) CFSMount:cfsmount40:offline:Attempting fuser TERM : Mount Point : /vx/nms1a-eba_rede
2012/09/20 15:14:38 VCS NOTICE V-16-20011-5510 (SFS02) CFSMount:cfsmount38:offline:Attempting fuser TERM : Mount Point : /vx/nms1a-segment1
2012/09/20 15:14:38 VCS NOTICE V-16-20011-5510 (SFS02) CFSMount:cfsmount41:offline:Attempting fuser TERM : Mount Point : /vx/nms1a-eba_ebsg
2012/09/20 15:14:38 VCS NOTICE V-16-20011-5510 (SFS02) CFSMount:cfsmount42:offline:Attempting fuser TERM : Mount Point : /vx/nms1a-eba_ebsw
2012/09/20 15:14:38 VCS NOTICE V-16-20011-5510 (SFS02) CFSMount:cfsmount43:offline:Attempting fuser TERM : Mount Point : /vx/nms1a-eba_ebss
2012/09/20 15:14:39 VCS NOTICE V-16-20011-5510 (SFS02) CFSMount:cfsmount44:offline:Attempting fuser TERM : Mount Point : /vx/nms1a-eba_rtt
2012/09/20 15:14:39 VCS NOTICE V-16-20011-5510 (SFS02) CFSMount:cfsmount45:offline:Attempting fuser TERM : Mount Point : /vx/nms1a-pm_storage
2012/09/20 15:14:55 VCS NOTICE V-16-20011-5510 (SFS02) CFSMount:cfsmount174:offline:Attempting fuser KILL : Mount Point : /vx/nmsrc3a-ora_pm_vol
2012/09/20 15:14:55 VCS NOTICE V-16-20011-5510 (SFS02) CFSMount:cfsmount39:offline:Attempting fuser KILL : Mount Point : /vx/nms1a-sgwcg
2012/09/20 15:14:55 VCS NOTICE V-16-20011-5510 (SFS02) CFSMount:cfsmount40:offline:Attempting fuser KILL : Mount Point : /vx/nms1a-eba_rede
2012/09/20 15:14:55 VCS NOTICE V-16-20011-5510 (SFS02) CFSMount:cfsmount38:offline:Attempting fuser KILL : Mount Point : /vx/nms1a-segment1
2012/09/20 15:14:55 VCS NOTICE V-16-20011-5510 (SFS02) CFSMount:cfsmount41:offline:Attempting fuser KILL : Mount Point : /vx/nms1a-eba_ebsg
2012/09/20 15:14:55 VCS NOTICE V-16-20011-5510 (SFS02) CFSMount:cfsmount42:offline:Attempting fuser KILL : Mount Point : /vx/nms1a-eba_ebsw
2012/09/20 15:14:56 VCS NOTICE V-16-20011-5510 (SFS02) CFSMount:cfsmount43:offline:Attempting fuser KILL : Mount Point : /vx/nms1a-eba_ebss
2012/09/20 15:14:56 VCS NOTICE V-16-20011-5510 (SFS02) CFSMount:cfsmount44:offline:Attempting fuser KILL : Mount Point : /vx/nms1a-eba_rtt
2012/09/20 15:14:56 VCS NOTICE V-16-20011-5510 (SFS02) CFSMount:cfsmount45:offline:Attempting fuser KILL : Mount Point : /vx/nms1a-pm_storage
2012/09/20 15:15:10 VCS NOTICE V-16-20011-5510 (SFS02) CFSMount:cfsmount174:offline:Attempting fuser KILL : Mount Point : /vx/nmsrc3a-ora_pm_vol
2012/09/20 15:15:10 VCS NOTICE V-16-20011-5510 (SFS02) CFSMount:cfsmount39:offline:Attempting fuser KILL : Mount Point : /vx/nms1a-sgwcg
2012/09/20 15:15:10 VCS NOTICE V-16-20011-5510 (SFS02) CFSMount:cfsmount38:offline:Attempting fuser KILL : Mount Point : /vx/nms1a-segment1
2012/09/20 15:15:10 VCS NOTICE V-16-20011-5510 (SFS02) CFSMount:cfsmount40:offline:Attempting fuser KILL : Mount Point : /vx/nms1a-eba_rede
2012/09/20 15:15:11 VCS NOTICE V-16-20011-5510 (SFS02) CFSMount:cfsmount41:offline:Attempting fuser KILL : Mount Point : /vx/nms1a-eba_ebsg
2012/09/20 15:15:11 VCS NOTICE V-16-20011-5510 (SFS02) CFSMount:cfsmount42:offline:Attempting fuser KILL : Mount Point : /vx/nms1a-eba_ebsw
2012/09/20 15:15:11 VCS NOTICE V-16-20011-5510 (SFS02) CFSMount:cfsmount43:offline:Attempting fuser KILL : Mount Point : /vx/nms1a-eba_ebss
2012/09/20 15:15:11 VCS NOTICE V-16-20011-5510 (SFS02) CFSMount:cfsmount44:offline:Attempting fuser KILL : Mount Point : /vx/nms1a-eba_rtt
2012/09/20 15:15:11 VCS NOTICE V-16-20011-5510 (SFS02) CFSMount:cfsmount45:offline:Attempting fuser KILL : Mount Point : /vx/nms1a-pm_storage
2012/09/20 15:15:23 VCS NOTICE V-16-20011-5504 (SFS02) CFSMount:cfsmount174:offline:Force Umount : Mount Point : /vx/nmsrc3a-ora_pm_vol
2012/09/20 15:15:24 VCS NOTICE V-16-20011-5504 (SFS02) CFSMount:cfsmount40:offline:Force Umount : Mount Point : /vx/nms1a-eba_rede
2012/09/20 15:15:24 VCS NOTICE V-16-20011-5504 (SFS02) CFSMount:cfsmount39:offline:Force Umount : Mount Point : /vx/nms1a-sgwcg
2012/09/20 15:15:24 VCS NOTICE V-16-20011-5504 (SFS02) CFSMount:cfsmount38:offline:Force Umount : Mount Point : /vx/nms1a-segment1
2012/09/20 15:15:24 VCS NOTICE V-16-20011-5504 (SFS02) CFSMount:cfsmount41:offline:Force Umount : Mount Point : /vx/nms1a-eba_ebsg
2012/09/20 15:15:24 VCS NOTICE V-16-20011-5504 (SFS02) CFSMount:cfsmount42:offline:Force Umount : Mount Point : /vx/nms1a-eba_ebsw
2012/09/20 15:15:24 VCS NOTICE V-16-20011-5504 (SFS02) CFSMount:cfsmount43:offline:Force Umount : Mount Point : /vx/nms1a-eba_ebss
2012/09/20 15:15:24 VCS NOTICE V-16-20011-5504 (SFS02) CFSMount:cfsmount44:offline:Force Umount : Mount Point : /vx/nms1a-eba_rtt
2012/09/20 15:15:24 VCS NOTICE V-16-20011-5504 (SFS02) CFSMount:cfsmount45:offline:Force Umount : Mount Point : /vx/nms1a-pm_storage
2012/09/20 15:15:26 VCS INFO V-16-1-10305 Resource cfsmount128 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount128) is offline on SFS02 (VCS initiated)
2012/09/20 15:15:26 VCS NOTICE V-16-1-10300 Initiating Offline of Resource cvmvoldg128 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount128) on System SFS02
2012/09/20 15:15:26 VCS INFO V-16-1-10305 Resource cfsmount131 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount131) is offline on SFS02 (VCS initiated)
2012/09/20 15:15:26 VCS NOTICE V-16-1-10300 Initiating Offline of Resource cvmvoldg131 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount131) on System SFS02
2012/09/20 15:15:26 VCS INFO V-16-1-10305 Resource cfsmount130 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount130) is offline on SFS02 (VCS initiated)
2012/09/20 15:15:26 VCS NOTICE V-16-1-10300 Initiating Offline of Resource cvmvoldg130 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount130) on System SFS02
2012/09/20 15:15:26 VCS INFO V-16-1-10305 Resource cfsmount135 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount135) is offline on SFS02 (VCS initiated)
2012/09/20 15:15:26 VCS NOTICE V-16-1-10300 Initiating Offline of Resource cvmvoldg135 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount135) on System SFS02
2012/09/20 15:15:26 VCS INFO V-16-1-10305 Resource cfsmount134 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount134) is offline on SFS02 (VCS initiated)
2012/09/20 15:15:26 VCS NOTICE V-16-1-10300 Initiating Offline of Resource cvmvoldg134 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount134) on System SFS02
2012/09/20 15:15:26 VCS INFO V-16-1-10305 Resource cfsmount141 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount141) is offline on SFS02 (VCS initiated)
2012/09/20 15:15:26 VCS NOTICE V-16-1-10300 Initiating Offline of Resource cvmvoldg141 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount141) on System SFS02
2012/09/20 15:15:26 VCS INFO V-16-1-10305 Resource cfsmount137 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount137) is offline on SFS02 (VCS initiated)
2012/09/20 15:15:26 VCS NOTICE V-16-1-10300 Initiating Offline of Resource cvmvoldg137 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount137) on System SFS02
2012/09/20 15:15:26 VCS INFO V-16-1-10305 Resource cfsmount124 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount124) is offline on SFS02 (VCS initiated)
2012/09/20 15:15:26 VCS NOTICE V-16-1-10300 Initiating Offline of Resource cvmvoldg124 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount124) on System SFS02
2012/09/20 15:15:26 VCS INFO V-16-1-10305 Resource cfsmount140 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount140) is offline on SFS02 (VCS initiated)
2012/09/20 15:15:26 VCS NOTICE V-16-1-10300 Initiating Offline of Resource cvmvoldg140 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount140) on System SFS02
2012/09/20 15:15:26 VCS INFO V-16-1-10305 Resource cfsmount144 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount144) is offline on SFS02 (VCS initiated)
2012/09/20 15:15:26 VCS NOTICE V-16-1-10300 Initiating Offline of Resource cvmvoldg144 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount144) on System SFS02
2012/09/20 15:15:26 VCS INFO V-16-1-10305 Resource cfsmount139 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount139) is offline on SFS02 (VCS initiated)
2012/09/20 15:15:26 VCS NOTICE V-16-1-10300 Initiating Offline of Resource cvmvoldg139 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount139) on System SFS02
2012/09/20 15:15:26 VCS INFO V-16-1-10305 Resource cfsmount136 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount136) is offline on SFS02 (VCS initiated)
2012/09/20 15:15:26 VCS NOTICE V-16-1-10300 Initiating Offline of Resource cvmvoldg136 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount136) on System SFS02
2012/09/20 15:15:26 VCS INFO V-16-1-10305 Resource cfsmount147 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount147) is offline on SFS02 (VCS initiated)
2012/09/20 15:15:26 VCS NOTICE V-16-1-10300 Initiating Offline of Resource cvmvoldg147 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount147) on System SFS02
2012/09/20 15:15:26 VCS INFO V-16-1-10305 Resource cfsmount146 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount146) is offline on SFS02 (VCS initiated)
2012/09/20 15:15:26 VCS NOTICE V-16-1-10300 Initiating Offline of Resource cvmvoldg146 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount146) on System SFS02
2012/09/20 15:15:26 VCS INFO V-16-1-10305 Resource cfsmount138 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount138) is offline on SFS02 (VCS initiated)
2012/09/20 15:15:26 VCS NOTICE V-16-1-10300 Initiating Offline of Resource cvmvoldg138 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount138) on System SFS02
2012/09/20 15:15:26 VCS INFO V-16-1-10305 Resource cfsmount151 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount151) is offline on SFS02 (VCS initiated)
2012/09/20 15:15:26 VCS NOTICE V-16-1-10300 Initiating Offline of Resource cvmvoldg151 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount151) on System SFS02
2012/09/20 15:15:26 VCS INFO V-16-1-10305 Resource cfsmount142 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount142) is offline on SFS02 (VCS initiated)
2012/09/20 15:15:26 VCS NOTICE V-16-1-10300 Initiating Offline of Resource cvmvoldg142 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount142) on System SFS02
2012/09/20 15:15:26 VCS INFO V-16-1-10305 Resource cfsmount150 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount150) is offline on SFS02 (VCS initiated)
2012/09/20 15:15:26 VCS NOTICE V-16-1-10300 Initiating Offline of Resource cvmvoldg150 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount150) on System SFS02
2012/09/20 15:15:26 VCS INFO V-16-1-10305 Resource cfsmount149 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount149) is offline on SFS02 (VCS initiated)
2012/09/20 15:15:26 VCS NOTICE V-16-1-10300 Initiating Offline of Resource cvmvoldg149 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount149) on System SFS02
2012/09/20 15:15:26 VCS INFO V-16-1-10305 Resource cfsmount153 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount153) is offline on SFS02 (VCS initiated)
2012/09/20 15:15:26 VCS NOTICE V-16-1-10300 Initiating Offline of Resource cvmvoldg153 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount153) on System SFS02
2012/09/20 15:15:26 VCS INFO V-16-1-10305 Resource cfsmount145 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount145) is offline on SFS02 (VCS initiated)
2012/09/20 15:15:26 VCS NOTICE V-16-1-10300 Initiating Offline of Resource cvmvoldg145 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount145) on System SFS02
2012/09/20 15:15:26 VCS INFO V-16-1-10305 Resource cfsmount148 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount148) is offline on SFS02 (VCS initiated)
2012/09/20 15:15:26 VCS NOTICE V-16-1-10300 Initiating Offline of Resource cvmvoldg148 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount148) on System SFS02
2012/09/20 15:15:26 VCS INFO V-16-1-10305 Resource cfsmount152 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount152) is offline on SFS02 (VCS initiated)
2012/09/20 15:15:26 VCS NOTICE V-16-1-10300 Initiating Offline of Resource cvmvoldg152 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount152) on System SFS02
2012/09/20 15:15:26 VCS INFO V-16-1-10305 Resource cfsmount143 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount143) is offline on SFS02 (VCS initiated)
2012/09/20 15:15:26 VCS NOTICE V-16-1-10300 Initiating Offline of Resource cvmvoldg143 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount143) on System SFS02
2012/09/20 15:15:26 VCS INFO V-16-1-10305 Resource cfsmount133 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount133) is offline on SFS02 (VCS initiated)
2012/09/20 15:15:26 VCS NOTICE V-16-1-10300 Initiating Offline of Resource cvmvoldg133 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount133) on System SFS02
2012/09/20 15:15:26 VCS INFO V-16-1-10305 Resource cfsmount156 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount156) is offline on SFS02 (VCS initiated)
2012/09/20 15:15:26 VCS NOTICE V-16-1-10300 Initiating Offline of Resource cvmvoldg156 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount156) on System SFS02
2012/09/20 15:15:26 VCS INFO V-16-1-10305 Resource cfsmount157 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount157) is offline on SFS02 (VCS initiated)
2012/09/20 15:15:26 VCS NOTICE V-16-1-10300 Initiating Offline of Resource cvmvoldg157 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount157) on System SFS02
2012/09/20 15:15:26 VCS INFO V-16-1-10305 Resource cfsmount163 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount163) is offline on SFS02 (VCS initiated)
2012/09/20 15:15:26 VCS NOTICE V-16-1-10300 Initiating Offline of Resource cvmvoldg163 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount163) on System SFS02
2012/09/20 15:15:26 VCS INFO V-16-1-10305 Resource cfsmount154 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount154) is offline on SFS02 (VCS initiated)
2012/09/20 15:15:26 VCS NOTICE V-16-1-10300 Initiating Offline of Resource cvmvoldg154 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount154) on System SFS02
2012/09/20 15:15:26 VCS INFO V-16-1-10305 Resource cfsmount165 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount165) is offline on SFS02 (VCS initiated)
2012/09/20 15:15:26 VCS NOTICE V-16-1-10300 Initiating Offline of Resource cvmvoldg165 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount165) on System SFS02
2012/09/20 15:15:26 VCS INFO V-16-1-10305 Resource cfsmount155 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount155) is offline on SFS02 (VCS initiated)
2012/09/20 15:15:26 VCS NOTICE V-16-1-10300 Initiating Offline of Resource cvmvoldg155 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount155) on System SFS02
2012/09/20 15:15:26 VCS INFO V-16-1-10305 Resource cfsmount164 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount164) is offline on SFS02 (VCS initiated)
2012/09/20 15:15:26 VCS NOTICE V-16-1-10300 Initiating Offline of Resource cvmvoldg164 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount164) on System SFS02
2012/09/20 15:15:26 VCS INFO V-16-1-10305 Resource cfsmount161 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount161) is offline on SFS02 (VCS initiated)
2012/09/20 15:15:26 VCS NOTICE V-16-1-10300 Initiating Offline of Resource cvmvoldg161 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount161) on System SFS02
2012/09/20 15:15:26 VCS INFO V-16-1-10305 Resource cfsmount166 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount166) is offline on SFS02 (VCS initiated)
2012/09/20 15:15:26 VCS NOTICE V-16-1-10300 Initiating Offline of Resource cvmvoldg166 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount166) on System SFS02
2012/09/20 15:15:26 VCS ERROR V-16-2-13064 (SFS02) Agent is calling clean for resource(cfsmount167) because the resource is up even after offline completed.
2012/09/20 15:15:26 VCS ERROR V-16-2-13064 (SFS02) Agent is calling clean for resource(cfsmount168) because the resource is up even after offline completed.
2012/09/20 15:15:26 VCS ERROR V-16-2-13064 (SFS02) Agent is calling clean for resource(cfsmount159) because the resource is up even after offline completed.
2012/09/20 15:15:31 VCS INFO V-16-1-10305 Resource cvmvoldg135 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount135) is offline on SFS02 (VCS initiated)
2012/09/20 15:15:31 VCS NOTICE V-16-1-10446 Group vrts_vea_cfs_int_cfsmount135 is offline on system SFS02
2012/09/20 15:15:31 VCS INFO V-16-1-10305 Resource cvmvoldg137 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount137) is offline on SFS02 (VCS initiated)
2012/09/20 15:15:31 VCS NOTICE V-16-1-10446 Group vrts_vea_cfs_int_cfsmount137 is offline on system SFS02
2012/09/20 15:15:31 VCS INFO V-16-1-10305 Resource cvmvoldg144 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount144) is offline on SFS02 (VCS initiated)
2012/09/20 15:15:31 VCS NOTICE V-16-1-10446 Group vrts_vea_cfs_int_cfsmount144 is offline on system SFS02
2012/09/20 15:15:31 VCS INFO V-16-1-10305 Resource cvmvoldg140 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount140) is offline on SFS02 (VCS initiated)
2012/09/20 15:15:31 VCS NOTICE V-16-1-10446 Group vrts_vea_cfs_int_cfsmount140 is offline on system SFS02
2012/09/20 15:15:32 VCS INFO V-16-1-10305 Resource cvmvoldg131 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount131) is offline on SFS02 (VCS initiated)
2012/09/20 15:15:32 VCS NOTICE V-16-1-10446 Group vrts_vea_cfs_int_cfsmount131 is offline on system SFS02
2012/09/20 15:15:32 VCS INFO V-16-1-10305 Resource cvmvoldg141 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount141) is offline on SFS02 (VCS initiated)
2012/09/20 15:15:32 VCS NOTICE V-16-1-10446 Group vrts_vea_cfs_int_cfsmount141 is offline on system SFS02
2012/09/20 15:15:32 VCS INFO V-16-1-10305 Resource cvmvoldg134 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount134) is offline on SFS02 (VCS initiated)
2012/09/20 15:15:32 VCS NOTICE V-16-1-10446 Group vrts_vea_cfs_int_cfsmount134 is offline on system SFS02
2012/09/20 15:15:32 VCS INFO V-16-1-10305 Resource cvmvoldg130 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount130) is offline on SFS02 (VCS initiated)
2012/09/20 15:15:32 VCS NOTICE V-16-1-10446 Group vrts_vea_cfs_int_cfsmount130 is offline on system SFS02
2012/09/20 15:15:32 VCS INFO V-16-1-10305 Resource cvmvoldg128 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount128) is offline on SFS02 (VCS initiated)
2012/09/20 15:15:32 VCS NOTICE V-16-1-10446 Group vrts_vea_cfs_int_cfsmount128 is offline on system SFS02
2012/09/20 15:15:32 VCS INFO V-16-1-10305 Resource cvmvoldg124 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount124) is offline on SFS02 (VCS initiated)
2012/09/20 15:15:32 VCS NOTICE V-16-1-10446 Group vrts_vea_cfs_int_cfsmount124 is offline on system SFS02
2012/09/20 15:15:33 VCS INFO V-16-1-10305 Resource cvmvoldg151 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount151) is offline on SFS02 (VCS initiated)
2012/09/20 15:15:33 VCS NOTICE V-16-1-10446 Group vrts_vea_cfs_int_cfsmount151 is offline on system SFS02
2012/09/20 15:15:33 VCS INFO V-16-1-10305 Resource cvmvoldg136 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount136) is offline on SFS02 (VCS initiated)
2012/09/20 15:15:33 VCS NOTICE V-16-1-10446 Group vrts_vea_cfs_int_cfsmount136 is offline on system SFS02
2012/09/20 15:15:33 VCS INFO V-16-1-10305 Resource cvmvoldg138 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount138) is offline on SFS02 (VCS initiated)
2012/09/20 15:15:33 VCS NOTICE V-16-1-10446 Group vrts_vea_cfs_int_cfsmount138 is offline on system SFS02
2012/09/20 15:15:33 VCS INFO V-16-1-10305 Resource cvmvoldg147 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount147) is offline on SFS02 (VCS initiated)
2012/09/20 15:15:33 VCS NOTICE V-16-1-10446 Group vrts_vea_cfs_int_cfsmount147 is offline on system SFS02
2012/09/20 15:15:33 VCS INFO V-16-1-10305 Resource cvmvoldg139 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount139) is offline on SFS02 (VCS initiated)
2012/09/20 15:15:33 VCS NOTICE V-16-1-10446 Group vrts_vea_cfs_int_cfsmount139 is offline on system SFS02
2012/09/20 15:15:33 VCS INFO V-16-1-10305 Resource cvmvoldg150 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount150) is offline on SFS02 (VCS initiated)
2012/09/20 15:15:33 VCS NOTICE V-16-1-10446 Group vrts_vea_cfs_int_cfsmount150 is offline on system SFS02
2012/09/20 15:15:33 VCS INFO V-16-1-10305 Resource cvmvoldg146 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount146) is offline on SFS02 (VCS initiated)
2012/09/20 15:15:33 VCS NOTICE V-16-1-10446 Group vrts_vea_cfs_int_cfsmount146 is offline on system SFS02
2012/09/20 15:15:33 VCS INFO V-16-1-10305 Resource cvmvoldg153 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount153) is offline on SFS02 (VCS initiated)
2012/09/20 15:15:33 VCS NOTICE V-16-1-10446 Group vrts_vea_cfs_int_cfsmount153 is offline on system SFS02
2012/09/20 15:15:34 VCS INFO V-16-1-10305 Resource cvmvoldg145 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount145) is offline on SFS02 (VCS initiated)
2012/09/20 15:15:34 VCS NOTICE V-16-1-10446 Group vrts_vea_cfs_int_cfsmount145 is offline on system SFS02
2012/09/20 15:15:34 VCS INFO V-16-1-10305 Resource cvmvoldg148 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount148) is offline on SFS02 (VCS initiated)
2012/09/20 15:15:34 VCS NOTICE V-16-1-10446 Group vrts_vea_cfs_int_cfsmount148 is offline on system SFS02
2012/09/20 15:15:34 VCS INFO V-16-1-10305 Resource cvmvoldg156 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount156) is offline on SFS02 (VCS initiated)
2012/09/20 15:15:34 VCS NOTICE V-16-1-10446 Group vrts_vea_cfs_int_cfsmount156 is offline on system SFS02
2012/09/20 15:15:34 VCS INFO V-16-1-10305 Resource cvmvoldg143 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount143) is offline on SFS02 (VCS initiated)
2012/09/20 15:15:34 VCS NOTICE V-16-1-10446 Group vrts_vea_cfs_int_cfsmount143 is offline on system SFS02
2012/09/20 15:15:34 VCS INFO V-16-1-10305 Resource cvmvoldg152 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount152) is offline on SFS02 (VCS initiated)
2012/09/20 15:15:34 VCS NOTICE V-16-1-10446 Group vrts_vea_cfs_int_cfsmount152 is offline on system SFS02
2012/09/20 15:15:34 VCS INFO V-16-1-10305 Resource cvmvoldg142 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount142) is offline on SFS02 (VCS initiated)
2012/09/20 15:15:34 VCS NOTICE V-16-1-10446 Group vrts_vea_cfs_int_cfsmount142 is offline on system SFS02
2012/09/20 15:15:34 VCS INFO V-16-1-10305 Resource cvmvoldg133 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount133) is offline on SFS02 (VCS initiated)
2012/09/20 15:15:34 VCS NOTICE V-16-1-10446 Group vrts_vea_cfs_int_cfsmount133 is offline on system SFS02
2012/09/20 15:15:34 VCS INFO V-16-1-10305 Resource cvmvoldg149 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount149) is offline on SFS02 (VCS initiated)
2012/09/20 15:15:34 VCS NOTICE V-16-1-10446 Group vrts_vea_cfs_int_cfsmount149 is offline on system SFS02
2012/09/20 15:15:34 VCS INFO V-16-1-10305 Resource cvmvoldg163 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount163) is offline on SFS02 (VCS initiated)
2012/09/20 15:15:34 VCS NOTICE V-16-1-10446 Group vrts_vea_cfs_int_cfsmount163 is offline on system SFS02
2012/09/20 15:15:34 VCS INFO V-16-1-10305 Resource cvmvoldg157 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount157) is offline on SFS02 (VCS initiated)
2012/09/20 15:15:34 VCS NOTICE V-16-1-10446 Group vrts_vea_cfs_int_cfsmount157 is offline on system SFS02
2012/09/20 15:15:36 VCS INFO V-16-1-10305 Resource cvmvoldg166 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount166) is offline on SFS02 (VCS initiated)
2012/09/20 15:15:36 VCS NOTICE V-16-1-10446 Group vrts_vea_cfs_int_cfsmount166 is offline on system SFS02
2012/09/20 15:15:36 VCS INFO V-16-2-13003 (SFS01) Resource(nasgw_nfs): Output of the timed out operation (clean)
mount: nfsd already mounted or /proc/fs/nfsd busy
mount: according to mtab, nfsd is already mounted on /proc/fs/nfsd

2012/09/20 15:15:36 VCS ERROR V-16-2-13006 (SFS01) Resource(nasgw_nfs): clean procedure did not complete within the expected time.
2012/09/20 15:15:38 VCS ERROR V-16-20011-5503 (SFS02) CFSMount:cfsmount46:offline:Umount Failed : Mount Point : /vx/nms1a-home
2012/09/20 15:15:38 VCS INFO V-16-1-10305 Resource cvmvoldg161 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount161) is offline on SFS02 (VCS initiated)
2012/09/20 15:15:38 VCS NOTICE V-16-1-10446 Group vrts_vea_cfs_int_cfsmount161 is offline on system SFS02
2012/09/20 15:15:38 VCS INFO V-16-1-10305 Resource cvmvoldg155 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount155) is offline on SFS02 (VCS initiated)
2012/09/20 15:15:38 VCS NOTICE V-16-1-10446 Group vrts_vea_cfs_int_cfsmount155 is offline on system SFS02
2012/09/20 15:15:38 VCS INFO V-16-1-10305 Resource cvmvoldg154 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount154) is offline on SFS02 (VCS initiated)
2012/09/20 15:15:38 VCS NOTICE V-16-1-10446 Group vrts_vea_cfs_int_cfsmount154 is offline on system SFS02
2012/09/20 15:15:38 VCS INFO V-16-1-10305 Resource cvmvoldg164 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount164) is offline on SFS02 (VCS initiated)
2012/09/20 15:15:38 VCS NOTICE V-16-1-10446 Group vrts_vea_cfs_int_cfsmount164 is offline on system SFS02
2012/09/20 15:15:38 VCS INFO V-16-1-10305 Resource cvmvoldg165 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount165) is offline on SFS02 (VCS initiated)
2012/09/20 15:15:38 VCS NOTICE V-16-1-10446 Group vrts_vea_cfs_int_cfsmount165 is offline on system SFS02
2012/09/20 15:15:38 VCS ERROR V-16-20011-5503 (SFS02) CFSMount:cfsmount54:offline:Umount Failed : Mount Point : /vx/orasts1-log
2012/09/20 15:15:38 VCS ERROR V-16-20011-5503 (SFS02) CFSMount:cfsmount65:offline:Umount Failed : Mount Point : /vx/orasts1-archive
2012/09/20 15:15:38 VCS ERROR V-16-20011-5503 (SFS02) CFSMount:cfsmount47:offline:Umount Failed : Mount Point : /vx/nms1a-ora_pm_vol
2012/09/20 15:15:38 VCS ERROR V-16-20011-5503 (SFS02) CFSMount:cfsmount72:offline:Umount Failed : Mount Point : /vx/orasts1-rejected
2012/09/20 15:15:40 VCS ERROR V-16-20011-5503 (SFS02) CFSMount:cfsmount74:offline:Umount Failed : Mount Point : /vx/orasts1-snapshot
2012/09/20 15:15:50 VCS ERROR V-16-2-13027 (SFS01) Resource(share_167) - monitor procedure did not complete within the expected time.
2012/09/20 15:15:52 VCS INFO V-16-1-50135 User root fired command: hares -modify cfsmount169  Primary  SFS01  from localhost
2012/09/20 15:15:54 VCS INFO V-16-1-50135 User root fired command: hares -modify cfsmount168  Primary  SFS01  from localhost
2012/09/20 15:16:05 VCS ERROR V-16-2-13027 (SFS01) Resource(ishare_145) - monitor procedure did not complete within the expected time.
2012/09/20 15:16:05 VCS ERROR V-16-2-13027 (SFS01) Resource(ishare_151) - monitor procedure did not complete within the expected time.
2012/09/20 15:16:05 VCS ERROR V-16-2-13027 (SFS01) Resource(ishare_108) - monitor procedure did not complete within the expected time.
2012/09/20 15:16:05 VCS ERROR V-16-2-13027 (SFS01) Resource(share_173) - monitor procedure did not complete within the expected time.
2012/09/20 15:16:05 VCS ERROR V-16-2-13027 (SFS01) Resource(ishare_107) - monitor procedure did not complete within the expected time.
2012/09/20 15:16:05 VCS ERROR V-16-2-13027 (SFS01) Resource(share_216) - monitor procedure did not complete within the expected time.
2012/09/20 15:16:05 VCS ERROR V-16-2-13027 (SFS01) Resource(share_246) - monitor procedure did not complete within the expected time.
2012/09/20 15:16:05 VCS ERROR V-16-2-13027 (SFS01) Resource(ishare_163) - monitor procedure did not complete within the expected time.
2012/09/20 15:16:05 VCS ERROR V-16-2-13027 (SFS01) Resource(cfsmount169) - monitor procedure did not complete within the expected time.
2012/09/20 15:16:05 VCS ERROR V-16-2-13210 (SFS01) Agent is calling clean for resource(cfsmount169) because 1 successive invocations of the monitor procedure did not complete within the expected time.
2012/09/20 15:16:05 VCS ERROR V-16-2-13027 (SFS01) Resource(share_227) - monitor procedure did not complete within the expected time.
2012/09/20 15:16:05 VCS INFO V-16-2-13068 (SFS01) Resource(cfsmount169) - clean completed successfully.
2012/09/20 15:16:05 VCS ERROR V-16-2-13064 (SFS02) Agent is calling clean for resource(cfsmount173) because the resource is up even after offline completed.
2012/09/20 15:16:06 VCS INFO V-16-2-13082 (SFS01) Resource(cfsmount169) recovered from fault, on its own.
2012/09/20 15:16:06 VCS INFO V-16-2-13068 (SFS02) Resource(cfsmount173) - clean completed successfully.
2012/09/20 15:16:06 VCS ERROR V-16-2-13064 (SFS02) Agent is calling clean for resource(cfsmount158) because the resource is up even after offline completed.
2012/09/20 15:16:06 VCS INFO V-16-2-13068 (SFS02) Resource(cfsmount158) - clean completed successfully.
2012/09/20 15:16:06 VCS ERROR V-16-2-13064 (SFS02) Agent is calling clean for resource(cfsmount172) because the resource is up even after offline completed.
2012/09/20 15:16:06 VCS INFO V-16-2-13068 (SFS02) Resource(cfsmount172) - clean completed successfully.
2012/09/20 15:16:06 VCS ERROR V-16-2-13064 (SFS02) Agent is calling clean for resource(cfsmount160) because the resource is up even after offline completed.
2012/09/20 15:16:06 VCS NOTICE V-16-20011-5510 (SFS02) CFSMount:cfsmount46:offline:Attempting fuser TERM : Mount Point : /vx/nms1a-home
2012/09/20 15:16:06 VCS INFO V-16-2-13068 (SFS02) Resource(cfsmount160) - clean completed successfully.
2012/09/20 15:16:06 VCS ERROR V-16-2-13064 (SFS02) Agent is calling clean for resource(cfsmount162) because the resource is up even after offline completed.
2012/09/20 15:16:06 VCS INFO V-16-2-13068 (SFS02) Resource(cfsmount162) - clean completed successfully.
2012/09/20 15:16:06 VCS ERROR V-16-2-13064 (SFS02) Agent is calling clean for resource(cfsmount170) because the resource is up even after offline completed.
2012/09/20 15:16:06 VCS INFO V-16-2-13068 (SFS02) Resource(cfsmount170) - clean completed successfully.
2012/09/20 15:16:06 VCS ERROR V-16-2-13064 (SFS02) Agent is calling clean for resource(cfsmount39) because the resource is up even after offline completed.
2012/09/20 15:16:06 VCS INFO V-16-2-13068 (SFS02) Resource(cfsmount39) - clean completed successfully.
2012/09/20 15:16:06 VCS ERROR V-16-2-13064 (SFS02) Agent is calling clean for resource(cfsmount174) because the resource is up even after offline completed.
2012/09/20 15:16:06 VCS INFO V-16-2-13068 (SFS02) Resource(cfsmount174) - clean completed successfully.
2012/09/20 15:16:06 VCS ERROR V-16-2-13064 (SFS02) Agent is calling clean for resource(cfsmount38) because the resource is up even after offline completed.
2012/09/20 15:16:06 VCS INFO V-16-2-13068 (SFS02) Resource(cfsmount38) - clean completed successfully.
2012/09/20 15:16:06 VCS ERROR V-16-2-13064 (SFS02) Agent is calling clean for resource(cfsmount42) because the resource is up even after offline completed.
2012/09/20 15:16:06 VCS INFO V-16-2-13068 (SFS02) Resource(cfsmount42) - clean completed successfully.
2012/09/20 15:16:06 VCS ERROR V-16-2-13064 (SFS02) Agent is calling clean for resource(cfsmount41) because the resource is up even after offline completed.
2012/09/20 15:16:06 VCS INFO V-16-2-13068 (SFS02) Resource(cfsmount41) - clean completed successfully.
2012/09/20 15:16:06 VCS ERROR V-16-2-13064 (SFS02) Agent is calling clean for resource(cfsmount40) because the resource is up even after offline completed.
2012/09/20 15:16:06 VCS INFO V-16-2-13068 (SFS02) Resource(cfsmount40) - clean completed successfully.
2012/09/20 15:16:06 VCS ERROR V-16-2-13064 (SFS02) Agent is calling clean for resource(cfsmount43) because the resource is up even after offline completed.
2012/09/20 15:16:06 VCS INFO V-16-2-13068 (SFS02) Resource(cfsmount43) - clean completed successfully.
2012/09/20 15:16:06 VCS ERROR V-16-2-13064 (SFS02) Agent is calling clean for resource(cfsmount44) because the resource is up even after offline completed.
2012/09/20 15:16:09 VCS INFO V-16-2-13068 (SFS02) Resource(cfsmount44) - clean completed successfully.
2012/09/20 15:16:13 VCS NOTICE V-16-20011-5510 (SFS02) CFSMount:cfsmount74:offline:Attempting fuser TERM : Mount Point : /vx/orasts1-snapshot
2012/09/20 15:16:13 VCS ERROR V-16-2-13064 (SFS02) Agent is calling clean for resource(cfsmount45) because the resource is up even after offline completed.
2012/09/20 15:16:13 VCS NOTICE V-16-20011-5504 (SFS02) CFSMount:cfsmount168:clean:Force Umount : Mount Point : /vx/nmsrc3a-eba_rtt
2012/09/20 15:16:13 VCS NOTICE V-16-20011-5510 (SFS02) CFSMount:cfsmount72:offline:Attempting fuser TERM : Mount Point : /vx/orasts1-rejected
2012/09/20 15:16:13 VCS NOTICE V-16-20011-5504 (SFS02) CFSMount:cfsmount159:clean:Force Umount : Mount Point : /vx/orasts1-sybase_iq
2012/09/20 15:16:13 VCS NOTICE V-16-20011-5510 (SFS02) CFSMount:cfsmount54:offline:Attempting fuser TERM : Mount Point : /vx/orasts1-log
2012/09/20 15:16:13 VCS NOTICE V-16-20011-5510 (SFS02) CFSMount:cfsmount47:offline:Attempting fuser TERM : Mount Point : /vx/nms1a-ora_pm_vol
2012/09/20 15:16:13 VCS NOTICE V-16-20011-5510 (SFS02) CFSMount:cfsmount65:offline:Attempting fuser TERM : Mount Point : /vx/orasts1-archive
2012/09/20 15:16:13 VCS NOTICE V-16-20011-5504 (SFS02) CFSMount:cfsmount167:clean:Force Umount : Mount Point : /vx/nmsrc3a-eba_ebss
2012/09/20 15:16:14 VCS INFO V-16-2-13068 (SFS02) Resource(cfsmount45) - clean completed successfully.
2012/09/20 15:16:15 VCS INFO V-16-2-13068 (SFS02) Resource(cfsmount167) - clean completed successfully.
2012/09/20 15:16:15 VCS INFO V-16-2-13068 (SFS02) Resource(cfsmount159) - clean completed successfully.
2012/09/20 15:16:15 VCS INFO V-16-2-13068 (SFS02) Resource(cfsmount168) - clean completed successfully.
2012/09/20 15:16:15 VCS INFO V-16-1-10305 Resource cfsmount158 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount158) is offline on SFS02 (VCS initiated)
2012/09/20 15:16:15 VCS NOTICE V-16-1-10300 Initiating Offline of Resource cvmvoldg158 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount158) on System SFS02
2012/09/20 15:16:15 VCS INFO V-16-1-10305 Resource cfsmount173 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount173) is offline on SFS02 (VCS initiated)
2012/09/20 15:16:15 VCS NOTICE V-16-1-10300 Initiating Offline of Resource cvmvoldg173 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount173) on System SFS02
2012/09/20 15:16:15 VCS INFO V-16-1-10305 Resource cfsmount172 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount172) is offline on SFS02 (VCS initiated)
2012/09/20 15:16:15 VCS NOTICE V-16-1-10300 Initiating Offline of Resource cvmvoldg172 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount172) on System SFS02
2012/09/20 15:16:15 VCS INFO V-16-1-10305 Resource cfsmount162 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount162) is offline on SFS02 (VCS initiated)
2012/09/20 15:16:15 VCS NOTICE V-16-1-10300 Initiating Offline of Resource cvmvoldg162 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount162) on System SFS02
2012/09/20 15:16:15 VCS INFO V-16-1-10305 Resource cfsmount160 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount160) is offline on SFS02 (VCS initiated)
2012/09/20 15:16:15 VCS NOTICE V-16-1-10300 Initiating Offline of Resource cvmvoldg160 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount160) on System SFS02
2012/09/20 15:16:15 VCS INFO V-16-1-10305 Resource cfsmount170 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount170) is offline on SFS02 (VCS initiated)
2012/09/20 15:16:15 VCS NOTICE V-16-1-10300 Initiating Offline of Resource cvmvoldg170 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount170) on System SFS02
2012/09/20 15:16:15 VCS INFO V-16-1-10305 Resource cfsmount39 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount39) is offline on SFS02 (VCS initiated)
2012/09/20 15:16:15 VCS NOTICE V-16-1-10300 Initiating Offline of Resource cvmvoldg39 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount39) on System SFS02
2012/09/20 15:16:15 VCS INFO V-16-1-10305 Resource cfsmount174 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount174) is offline on SFS02 (VCS initiated)
2012/09/20 15:16:15 VCS NOTICE V-16-1-10300 Initiating Offline of Resource cvmvoldg174 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount174) on System SFS02
2012/09/20 15:16:15 VCS INFO V-16-1-10305 Resource cfsmount38 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount38) is offline on SFS02 (VCS initiated)
2012/09/20 15:16:15 VCS NOTICE V-16-1-10300 Initiating Offline of Resource cvmvoldg38 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount38) on System SFS02
2012/09/20 15:16:15 VCS INFO V-16-1-10305 Resource cfsmount42 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount42) is offline on SFS02 (VCS initiated)
2012/09/20 15:16:15 VCS NOTICE V-16-1-10300 Initiating Offline of Resource cvmvoldg42 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount42) on System SFS02
2012/09/20 15:16:15 VCS INFO V-16-1-10305 Resource cfsmount41 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount41) is offline on SFS02 (VCS initiated)
2012/09/20 15:16:15 VCS NOTICE V-16-1-10300 Initiating Offline of Resource cvmvoldg41 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount41) on System SFS02
2012/09/20 15:16:15 VCS INFO V-16-1-10305 Resource cfsmount43 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount43) is offline on SFS02 (VCS initiated)
2012/09/20 15:16:15 VCS NOTICE V-16-1-10300 Initiating Offline of Resource cvmvoldg43 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount43) on System SFS02
2012/09/20 15:16:15 VCS INFO V-16-1-10305 Resource cfsmount40 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount40) is offline on SFS02 (VCS initiated)
2012/09/20 15:16:15 VCS NOTICE V-16-1-10300 Initiating Offline of Resource cvmvoldg40 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount40) on System SFS02
2012/09/20 15:16:15 VCS INFO V-16-1-10305 Resource cfsmount44 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount44) is offline on SFS02 (VCS initiated)
2012/09/20 15:16:15 VCS NOTICE V-16-1-10300 Initiating Offline of Resource cvmvoldg44 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount44) on System SFS02
2012/09/20 15:16:16 VCS INFO V-16-1-10305 Resource cfsmount167 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount167) is offline on SFS02 (VCS initiated)
2012/09/20 15:16:16 VCS NOTICE V-16-1-10300 Initiating Offline of Resource cvmvoldg167 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount167) on System SFS02
2012/09/20 15:16:16 VCS INFO V-16-1-10305 Resource cfsmount45 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount45) is offline on SFS02 (VCS initiated)
2012/09/20 15:16:16 VCS NOTICE V-16-1-10300 Initiating Offline of Resource cvmvoldg45 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount45) on System SFS02
2012/09/20 15:16:16 VCS INFO V-16-1-10305 Resource cfsmount168 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount168) is offline on SFS02 (VCS initiated)
2012/09/20 15:16:16 VCS NOTICE V-16-1-10300 Initiating Offline of Resource cvmvoldg168 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount168) on System SFS02
2012/09/20 15:16:16 VCS INFO V-16-1-10305 Resource cfsmount159 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount159) is offline on SFS02 (VCS initiated)
2012/09/20 15:16:16 VCS NOTICE V-16-1-10300 Initiating Offline of Resource cvmvoldg159 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount159) on System SFS02
2012/09/20 15:16:16 VCS INFO V-16-1-10305 Resource cfsmount169 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount169) is offline on SFS02 (VCS initiated)
2012/09/20 15:16:16 VCS NOTICE V-16-1-10300 Initiating Offline of Resource cvmvoldg169 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount169) on System SFS02
2012/09/20 15:16:19 VCS INFO V-16-1-50135 User root fired command: hares -modify cfsmount42  Primary  SFS01  from localhost
2012/09/20 15:16:19 VCS INFO V-16-1-50135 User root fired command: hares -modify cfsmount46  Primary  SFS01  from localhost
2012/09/20 15:16:19 VCS INFO V-16-1-50135 User root fired command: hares -modify cfsmount45  Primary  SFS01  from localhost
2012/09/20 15:16:19 VCS INFO V-16-1-50135 User root fired command: hares -modify cfsmount43  Primary  SFS01  from localhost
2012/09/20 15:16:19 VCS INFO V-16-1-50135 User root fired command: hares -modify cfsmount44  Primary  SFS01  from localhost
2012/09/20 15:16:22 VCS INFO V-16-1-50135 User root fired command: hares -modify cfsmount41  Primary  SFS01  from localhost
2012/09/20 15:16:22 VCS INFO V-16-1-50135 User root fired command: hares -modify cfsmount173  Primary  SFS01  from localhost
2012/09/20 15:16:22 VCS INFO V-16-1-50135 User root fired command: hares -modify cfsmount38  Primary  SFS01  from localhost
2012/09/20 15:16:22 VCS INFO V-16-1-50135 User root fired command: hares -modify cfsmount40  Primary  SFS01  from localhost
2012/09/20 15:16:22 VCS INFO V-16-1-50135 User root fired command: hares -modify cfsmount172  Primary  SFS01  from localhost
2012/09/20 15:16:22 VCS INFO V-16-1-50135 User root fired command: hares -modify cfsmount39  Primary  SFS01  from localhost
2012/09/20 15:16:22 VCS INFO V-16-1-50135 User root fired command: hares -modify cfsmount170  Primary  SFS01  from localhost
2012/09/20 15:16:22 VCS INFO V-16-1-50135 User root fired command: hares -modify cfsmount167  Primary  SFS01  from localhost
2012/09/20 15:16:27 VCS INFO V-16-1-10305 Resource cvmvoldg162 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount162) is offline on SFS02 (VCS initiated)
2012/09/20 15:16:27 VCS NOTICE V-16-1-10446 Group vrts_vea_cfs_int_cfsmount162 is offline on system SFS02
2012/09/20 15:16:28 VCS INFO V-16-1-10305 Resource cvmvoldg174 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount174) is offline on SFS02 (VCS initiated)
2012/09/20 15:16:28 VCS NOTICE V-16-1-10446 Group vrts_vea_cfs_int_cfsmount174 is offline on system SFS02
2012/09/20 15:16:28 VCS INFO V-16-1-10305 Resource cvmvoldg160 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount160) is offline on SFS02 (VCS initiated)
2012/09/20 15:16:28 VCS NOTICE V-16-1-10446 Group vrts_vea_cfs_int_cfsmount160 is offline on system SFS02
2012/09/20 15:16:28 VCS INFO V-16-1-10305 Resource cvmvoldg39 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount39) is offline on SFS02 (VCS initiated)
2012/09/20 15:16:28 VCS NOTICE V-16-1-10446 Group vrts_vea_cfs_int_cfsmount39 is offline on system SFS02
2012/09/20 15:16:28 VCS INFO V-16-1-10305 Resource cvmvoldg172 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount172) is offline on SFS02 (VCS initiated)
2012/09/20 15:16:28 VCS NOTICE V-16-1-10446 Group vrts_vea_cfs_int_cfsmount172 is offline on system SFS02
2012/09/20 15:16:28 VCS INFO V-16-1-10305 Resource cvmvoldg173 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount173) is offline on SFS02 (VCS initiated)
2012/09/20 15:16:28 VCS NOTICE V-16-1-10446 Group vrts_vea_cfs_int_cfsmount173 is offline on system SFS02
2012/09/20 15:16:28 VCS INFO V-16-1-10305 Resource cvmvoldg158 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount158) is offline on SFS02 (VCS initiated)
2012/09/20 15:16:28 VCS NOTICE V-16-1-10446 Group vrts_vea_cfs_int_cfsmount158 is offline on system SFS02
2012/09/20 15:16:28 VCS INFO V-16-1-10305 Resource cvmvoldg170 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount170) is offline on SFS02 (VCS initiated)
2012/09/20 15:16:28 VCS NOTICE V-16-1-10446 Group vrts_vea_cfs_int_cfsmount170 is offline on system SFS02
2012/09/20 15:16:28 VCS INFO V-16-1-10305 Resource cvmvoldg42 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount42) is offline on SFS02 (VCS initiated)
2012/09/20 15:16:28 VCS NOTICE V-16-1-10446 Group vrts_vea_cfs_int_cfsmount42 is offline on system SFS02
2012/09/20 15:16:28 VCS INFO V-16-1-10305 Resource cvmvoldg38 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount38) is offline on SFS02 (VCS initiated)
2012/09/20 15:16:28 VCS NOTICE V-16-1-10446 Group vrts_vea_cfs_int_cfsmount38 is offline on system SFS02
2012/09/20 15:16:28 VCS INFO V-16-1-10305 Resource cvmvoldg44 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount44) is offline on SFS02 (VCS initiated)
2012/09/20 15:16:28 VCS NOTICE V-16-1-10446 Group vrts_vea_cfs_int_cfsmount44 is offline on system SFS02
2012/09/20 15:16:28 VCS INFO V-16-1-10305 Resource cvmvoldg41 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount41) is offline on SFS02 (VCS initiated)
2012/09/20 15:16:28 VCS NOTICE V-16-1-10446 Group vrts_vea_cfs_int_cfsmount41 is offline on system SFS02
2012/09/20 15:16:30 VCS INFO V-16-1-10305 Resource cvmvoldg43 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount43) is offline on SFS02 (VCS initiated)
2012/09/20 15:16:30 VCS NOTICE V-16-1-10446 Group vrts_vea_cfs_int_cfsmount43 is offline on system SFS02
2012/09/20 15:16:30 VCS INFO V-16-1-10305 Resource cvmvoldg40 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount40) is offline on SFS02 (VCS initiated)
2012/09/20 15:16:30 VCS NOTICE V-16-1-10446 Group vrts_vea_cfs_int_cfsmount40 is offline on system SFS02
2012/09/20 15:16:30 VCS INFO V-16-1-10305 Resource cvmvoldg168 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount168) is offline on SFS02 (VCS initiated)
2012/09/20 15:16:30 VCS NOTICE V-16-1-10446 Group vrts_vea_cfs_int_cfsmount168 is offline on system SFS02
2012/09/20 15:16:30 VCS INFO V-16-1-10305 Resource cvmvoldg167 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount167) is offline on SFS02 (VCS initiated)
2012/09/20 15:16:30 VCS NOTICE V-16-1-10446 Group vrts_vea_cfs_int_cfsmount167 is offline on system SFS02
2012/09/20 15:16:30 VCS INFO V-16-1-10305 Resource cvmvoldg159 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount159) is offline on SFS02 (VCS initiated)
2012/09/20 15:16:30 VCS NOTICE V-16-1-10446 Group vrts_vea_cfs_int_cfsmount159 is offline on system SFS02
2012/09/20 15:16:30 VCS INFO V-16-1-10305 Resource cvmvoldg45 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount45) is offline on SFS02 (VCS initiated)
2012/09/20 15:16:30 VCS NOTICE V-16-1-10446 Group vrts_vea_cfs_int_cfsmount45 is offline on system SFS02
2012/09/20 15:16:30 VCS INFO V-16-1-10305 Resource cvmvoldg169 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount169) is offline on SFS02 (VCS initiated)
2012/09/20 15:16:30 VCS NOTICE V-16-1-10446 Group vrts_vea_cfs_int_cfsmount169 is offline on system SFS02
2012/09/20 15:16:32 VCS NOTICE V-16-20011-5504 (SFS02) CFSMount:cfsmount46:offline:Force Umount : Mount Point : /vx/nms1a-home
2012/09/20 15:16:39 VCS NOTICE V-16-20011-5504 (SFS02) CFSMount:cfsmount74:offline:Force Umount : Mount Point : /vx/orasts1-snapshot
2012/09/20 15:16:39 VCS NOTICE V-16-20011-5504 (SFS02) CFSMount:cfsmount65:offline:Force Umount : Mount Point : /vx/orasts1-archive
2012/09/20 15:16:39 VCS NOTICE V-16-20011-5504 (SFS02) CFSMount:cfsmount47:offline:Force Umount : Mount Point : /vx/nms1a-ora_pm_vol
2012/09/20 15:16:39 VCS NOTICE V-16-20011-5504 (SFS02) CFSMount:cfsmount54:offline:Force Umount : Mount Point : /vx/orasts1-log
2012/09/20 15:16:39 VCS ERROR V-16-10031-7029 (SFS01) NFS:nasgw_nfs:monitor:Daemon [rpc.mountd] is not running.
2012/09/20 15:16:39 VCS NOTICE V-16-20011-5504 (SFS02) CFSMount:cfsmount72:offline:Force Umount : Mount Point : /vx/orasts1-rejected
2012/09/20 15:16:40 VCS WARNING V-16-10031-7002 (SFS01) NFS:nasgw_nfs:clean:Sending SIGKILL signal to nfsd (5804 5805 5806 5807 5808 5809 5810 5811 5812 5813 5814 5815 5816 5817 5818 5819 5820 5821 5822 5823 5824 5825 5826 5830 5831 5832 5833 5834 5835 5836 5837 5838 5839 5840 5841 5842 5843 5844 5845 5846 5847 5848 5850 5851 5852 5853 5854 5855 5856 5857 5858 5859 5860 5861 5862 5863 5864 5865 5866 5867 5868 5869 5870 5871 5872 5873 5874 5875 5876 5877 5878 5879 5880 5881 5882 5883 5884 5885 5886 5887 5888 5889 5890 5891 5892 5893 5894 5895 5896 5897 5898 5899 5900 5901 5902 5903).
2012/09/20 15:16:40 VCS INFO V-16-2-13001 (SFS01) Resource(nasgw_nfs): Output of the completed operation (clean)

Comments 3 CommentsJump to latest comment

Loic Bouquet's picture

Hi AntonA,

Unfortunatly, this Filestore version is an old version, the current is 5.6RP1P4HF1.

- How many disk are mapped on the system ? I have 80 disks and when i restart a node, sometime the other nodes loose CVM... not good.

- How many File System are designed ? Symantec reccommend to have a maximum of 10 FS on an cluster. If you have more than 10, maybe the CVM can crash.

Maybe the best way is to upgrade your cluster in a newer 5.6 version : https://sort.symantec.com/patch/detail/6512

Also, i didin't found the 5.6P3 version in Sort... maybe it's unsupported now ?

AntonA's picture

Hi Loic,

Thank you for your reply. Actually we're planning to upgrade to 5.7, but maybe will be done next year.

- How many disk are mapped on the system ?

We have 12 disks mapped on our system

Disk             sfs01_01 sfs01_02
====             ========  ======== 
emc_clariion0_100 OK        OK       
emc_clariion0_101 OK        OK       
emc_clariion0_102 OK        OK       
emc_clariion0_2  OK        OK       
emc_clariion0_22 OK        OK       
emc_clariion0_23 OK        OK       
emc_clariion0_28 OK        OK       
emc_clariion0_29 OK        OK       
emc_clariion0_3  OK        OK       
emc_clariion0_4  OK        OK       
emc_clariion0_43 OK        OK       
emc_clariion0_44 OK        OK  

- How many File System are designed ?

We have 68 Filesystem on our system, which being mounted and shared (NFS) for about 10 NFS clients (running in solaris 10). I'm worry about this, since you told me that Symantec recommendation is to have max 10 Filesystem. How I could get this kind dimensioning/limitation information ?

Yes, I can not find this 5.6P3 patch as well at symantec patch list. But, I'm not sure whether 5.6 P3 still supported or not. But from your given patch link, this 5.6P3 seems still mentioned in some 5.6 patch release note.

"After you apply this patch, you cannot uninstall it. The 5.6RP1P4HF1 patch can only be installed on 5.6 or 5.6 P1 or 5.6 P2 or 5.6 P3 or 5.6 RP1 or 5.6 RP1 P1 or 5.6 RP1 P2 or 5.6 RP1 P3 or 5.6 RP1 P4."
 

- Based on your experience, do you have any idea, how long NFS service interuption would take during service failover from one node to another which ? Especially for Software level 5.6.

Actually this issue occur randomly, and I'm still not sure what causing this. And sometime, when we run "cluster> shutdown nodename1", the service failover would be hanging, and not able to failover completely to other servers. Then we need to manually restart (poweroff/poweron) the both servers.

- What is the proper actions if service failover hangs and services could not be online at other server ?

Thanks again.

Loic Bouquet's picture

Hi,

The best way for your issue is certainly to ask Symantec support.

I can't find the limitation of FS on a SFS in documentations, but i have one customer with this issue (and my cluster have this issue in 5.6RP1P4HF1).

Normaly, the failover occur in 4-6 seconds, like you, if all is good. Sometimes, we need to reboot all the cluster (like you).

When the ressources are failed, you can try to online them in support, with traditionnals HA commands, like hares or hagrp. do a hastatus -sum to have the list of failed ressources.

I don't know if the CVM version is better in 5.7 distributions, but maybe you can test it on virtual machine to have an idea.

Best regards,

Loïc

SOLUTION