VCS 4 node Cluster experiencing chronic scsi "reservation conflict" warnings; vxdisk-list indicates VxVM vxdisk ERROR V-5-1-539 Device tagmastore-usp0_2113: get_contents failed

Article:TECH185490  |  Created: 2012-03-30  |  Updated: 2012-07-28  |  Article URL http://www.symantec.com/docs/TECH185490
Article Type
Technical Solution


Environment

Issue



vxdisk list command line shows the error. VxVM vxdisk ERROR V-5-1-539 Device tagmastore-usp0_2113: get_contents failed:;  pegsdb25 scsi: [ID 107833 kern.warning.... (ssd205):     reservation conflict    i/o to invalid geometry


Error



Multiple nodes experiencing similar scsi errors:

Feb 29 15:13:10 nodedb25 scsi: [ID 107833 kern.warning] WARNING: /pci@2,600000/SUNW,emlxs@0,1/fp@0,0/ssd@w50060e80047e0432,59 (ssd205): reservation conflict

Feb 29 15:13:10 nodedb25 scsi: [ID 107833 kern.warning] WARNING: /pci@2,600000/SUNW,emlxs@0,1/fp@0,0/ssd@w50060e80047e0432,59 (ssd205): i/o to invalid geometry

Feb 29 15:13:10 nodedb25 scsi: [ID 107833 kern.warning] WARNING: /pci@2,600000/SUNW,emlxs@0,1/fp@0,0/ssd@w50060e80047e0432,59 (ssd205): reservation conflict

Feb 29 15:13:10 nodedb25 scsi: [ID 107833 kern.warning] WARNING: /pci@2,600000/SUNW,emlxs@0,1/fp@0,0/ssd@w50060e80047e0432,59 (ssd205): reservation conflict

Feb 29 15:13:10 nodedb25 scsi: [ID 107833 kern.warning] WARNING: /pci@2,600000/SUNW,emlxs@0,1/fp@0,0/ssd@w50060e80047e0432,59 (ssd205): i/o to invalid geometry

 

Mar 15 05:45:53 nodedb01 scsi: [ID 107833 kern.warning] WARNING: /pci@0,600000/pci@0/pci@9/SUNW,emlxs@0/fp@0,0/ssd@w50060e80047e0401,24 (ssd211): reservation conflict

Mar 15 05:45:53 nodedb01 scsi: [ID 107833 kern.warning] WARNING: /pci@0,600000/pci@0/pci@9/SUNW,emlxs@0/fp@0,0/ssd@w50060e80047e0401,24 (ssd211): reservation conflict

Mar 15 05:45:53 nodedb01 scsi: [ID 107833 kern.warning] WARNING: /pci@0,600000/pci@0/pci@9/SUNW,emlxs@0/fp@0,0/ssd@w50060e80047e0401,24 (ssd211):

Mar 15 05:45:53 nodedb01 scsi: [ID 107833 kern.warning] WARNING: /pci@0,600000/pci@0/pci@9/SUNW,emlxs@0/fp@0,0/ssd@w50060e80047e0401,24 (ssd211):

Mar 15 05:45:53 nodedb01 scsi: [ID 107833 kern.warning] WARNING: /pci@0,600000/pci@0/pci@9/SUNW,emlxs@0/fp@0,0/ssd@w50060e80047e0401,24 (ssd211): reservation conflict


Environment



----Cluster node SFHA  is in upgrade transition from 5.0MP3 to 5.1SP1RP1P1

----SF is being upgraded on nodes in phases while production service group migrated between upgraded nodes

----VCS will remain 5.0MP3 until SF is upgraded

----Solaris MPxIO is also being replaced by VxDMP

----also under VxVM 5.1SP1RP1P1 "dmp_native_support" is enabled to support ZFS LUNs

$ egrep 'SunOS' */uname_a ; cat */etc/release
SunOS nodedb01 5.10 Generic_144488-12 sun4u sparc SUNW,SPARC-Enterprise
                   Oracle Solaris 10 9/10 s10s_u9wos_14a SPARC
     Copyright (c) 2010, Oracle and/or its affiliates. All rights reserved.
                            Assembled 11 August 2010
$ egrep 'vx|llt|gab|emcp' */modinfo
 29  1380800  50c10 317   1  vxdmp (VxVM 5.1SP1RP1P1 DMP Driver)
 31 7be00000 21a918 318   1  vxio (VxVM 5.1SP1RP1P1 I/O driver)
 33  13caa80   1190 319   1  vxspec (VxVM 5.1SP1RP1P1 control/status)
226 7bbf7308    d40 320   1  vxportal (VxFS 5.1_SP1RP1 portal driver)
227 7a600000 1fe098  21   1  vxfs (VxFS 5.1_SP1RP1 SunOS 5.10)
228 7ab18000  26470 323   1  llt (LLT 5.0MP3)
229 7ab38000  4a028 324   1  gab (GAB device 5.0MP3)
230 7ab6a000  51c10 325   1  vxfen (VRTS Fence 5.0MP3)
$

 

bash-3.00# uname -a
SunOS nodedb03 5.10 Generic_138888-06 sun4u sparc SUNW,SPARC-Enterprise

bash-3.00# modinfo | egrep 'vx|llt|gab|emcp'
 36  13428a8  3e4e0 317   1  vxdmp (VxVM 5.0MP3: DMP Driver)
 38 7be00000 209248 318   1  vxio (VxVM 5.0MP3 I/O driver)
 40 7bfe90f0    c78 319   1  vxspec (VxVM 5.0MP3 control/status driv)
220 7af953f0    cb0 320   1  vxportal (VxFS 5.0_REV-5.0MP3A25_sol port)
221 7aa00000 1d89e0  21   1  vxfs (VxFS 5.0_REV-5.0MP3A25_sol SunO)
222 7abc8000  26470 322   1  llt (LLT 5.0MP3)
223 7a600000  4a028 323   1  gab (GAB device 5.0MP3)
224 7a632000  51c10 324   1  vxfen (VRTS Fence 5.0MP3)


# uname -a
SunOS nodedb25 5.10 Generic_144488-12 sun4u sparc SUNW,SPARC-Enterprise

# modinfo | egrep 'vx|llt|gab|emcp'
 29  1380800  50c10 324   1  vxdmp (VxVM 5.1SP1RP1P1 DMP Driver)
 31 7be00000 21a918 325   1  vxio (VxVM 5.1SP1RP1P1 I/O driver)
 33  13caa80   1190 326   1  vxspec (VxVM 5.1SP1RP1P1 control/status)
227 7bbf5308    d40 327   1  vxportal (VxFS 5.1_SP1RP1 portal driver)
228 7a600000 1fe098  21   1  vxfs (VxFS 5.1_SP1RP1 SunOS 5.10)
229 7ab1e000  26470 329   1  llt (LLT 5.0MP3)
230 7ab3e000  4a028 330   1  gab (GAB device 5.0MP3)
231 7ab70000  51c10 331   1  vxfen (VRTS Fence 5.0MP3)
232 7aba2000  21ec0 332   1  vxglm (VxGLM 5.0MP3 (SunOS 5.10))
233 7ae1c000   5418 333   1  vxgms (VxGMS 5.0MP3 (SunOS))

bash-3.00# uname -a
SunOS nodedb24 5.10 Generic_138888-06 sun4u sparc SUNW,SPARC-Enterprise

bash-3.00# modinfo | egrep 'vx|llt|gab|emcp'
 36  13428a8  3fdc0 322   1  vxdmp (VxVM 5.0MP3RP4: DMP Driver)
 38 7be00000 211590 323   1  vxio (VxVM 5.0MP3RP4 I/O driver)
 40 7bff0428    c78 324   1  vxspec (VxVM 5.0MP3RP4 control/status d)
222 7ae9d180    cb0 325   1  vxportal (VxFS 5.0_REV-5.0MP3RP4g_sol por)
223 7aa00000 1dd7d0  21   1  vxfs (VxFS 5.0_REV-5.0MP3RP4g_sol Sun)
224 7abcc000  26470 327   1  llt (LLT 5.0MP3)
225 7a600000  4a028 328   1  gab (GAB device 5.0MP3)
226 7a632000  51c10 329   1  vxfen (VRTS Fence 5.0MP3)
 


Cause



---When VCS managed VxVM disk group is vxdg-imported on one node,   VCS i/o fencing applies SCSI-3 PGR registration and write-exclusive reservation keys on the disk group data disks;  for example:

# vxfenadm -g /dev/rdsk/c1t50060E80047E0427d71s2

Reading SCSI Registration Keys...

Device Name: /dev/rdsk/c1t50060E80047E0427d71s2

Total Number Of Keys: 2

key[0]:

Key Value [Numeric Format]: 67,86,67,83,0,0,0,0

Key Value [Character Format]: CVCS

key[1]:

Key Value [Numeric Format]: 67,86,67,83,0,0,0,0

Key Value [Character Format]: CVCS

 

vxfenadm -r /dev/rdsk/c1t50060E80047E0427d71s2

Reading SCSI Reservation Information...

Device Name: /dev/rdsk/c1t50060E80047E0427d71s2

Total Number Of Keys: 1

Key[0]:

Reservation Type: SCSI3_RESV_WRITEEXCLUSIVEREGISTRANTSONLY

Key Value [Numeric Format]: 67,86,67,83,0,0,0,0

Key Value [Character Format]: CVCS

---- but other nodes on the cluster experience "reservation conflict" warnings such as................

Feb 29 15:13:10 nodedb25 scsi: [ID 107833 kern.warning] WARNING: /pci@2,600000/SUNW,emlxs@0,1/fp@0,0/ssd@w50060e80047e0432,59 (ssd205): reservation conflict

Feb 29 15:13:10 nodedb25 scsi: [ID 107833 kern.warning] WARNING: /pci@2,600000/SUNW,emlxs@0,1/fp@0,0/ssd@w50060e80047e0432,59 (ssd205): i/o to invalid geometry

Mar 15 05:45:53 nodedb01 scsi: [ID 107833 kern.warning] WARNING: /pci@0,600000/pci@0/pci@9/SUNW,emlxs@0/fp@0,0/ssd@w50060e80047e0401,24 (ssd211): reservation conflict

Mar 15 05:45:53 nodedb01 scsi: [ID 107833 kern.warning] WARNING: /pci@0,600000/pci@0/pci@9/SUNW,emlxs@0/fp@0,0/ssd@w50060e80047e0401,24 (ssd211): reservation conflict

Mar 15 05:45:53 nodedb01 scsi: [ID 107833 kern.w


Solution



---- While array was placing reservation key SCSI3_RESV_WRITEEXCLUSIVEREGISTRANTSONLY the reservation key was behaving READEXCLUSIVEREGISTRANTSONLY instead. This prevented other hosts from even reading the LUNs imported to the other host.  Commands like prtvtoc and dd read would fail. 

----Discovered the Hitachi Tagmastore UPS600 host mode option settings were not complete.  HMO2 was not set.  Required settings to support SFHA for both DMP and VCS I/O Fencing are:

Host mode 09 =  Solaris hosts

Host mode option 02  =  VCS, CVM-CFS, SFRAC I/O fencing configured

Host mode option 07  =  Auto-recognize array LUN resize;

                                      *  requires host mode option 09  and  latest Solaris 10  HBA drivers

Host mode option 22   =  VCS cluster with common disk groups for failover or parallel LUN I/O access; (if AIX requires host mode 0F)




Article URL http://www.symantec.com/docs/TECH185490


Terms of use for this information are found in Legal Notices