Solaris 10 kernel update 8 patch and VxVM upgrade to 5.0MP3RP1HF6; vxdisk-list EFI LUNs becomes status error

Article:TECH143696  |  Created: 2010-11-08  |  Updated: 2012-07-28  |  Article URL http://www.symantec.com/docs/TECH143696
Article Type
Technical Solution

Product(s)

Environment

Issue



 

Unable to make VxVM claim new provisioned LUNs that are over one terabyte.
Problems with Solaris 10 EFI labeled LUNs with paths being declared incorrectly with active-passive controller arrays.

Error



 

-----VxVM vxdisk-list indicates “error” status for recently provisioned LUNs greater than one terabyte containing EFI labels:
 
# vxdisk list
 

DEVICE       TYPE            DISK         GROUP        STATUS

 
c1t5006016039A01736d0s2 auto:cdsdisk    -            -            online
c1t5006016039A01736d1s2 auto:cdsdisk    -            -            online
c1t5006016039A01736d2s2 auto            -            -            error
c1t5006016039A01736d3s2 auto            -            -            error
c1t5006016039A01736d4 auto:sliced     -            -            online
c1t5006016039A01736d5 auto:sliced     -            -            online
c1t5006016039A01736d6s2 auto            -            -            error
c1t5006016039A01736d7s2 auto            -            -            error

. . .

 
. . .
 
----The error status presumably means the LUN has either not been labeled via format, or the LUN is SAN configured improperly, or this is an indication of an HBA hardware or driver problem.
 
---- Our first hint from format is that the errored LUN(s) appear to have device paths describing the size of a terabyte or more.
# echo |format

AVAILABLE DISK SELECTIONS:

 
       0. c1t5006016A39A01736d0 <DGC-RAID5-0326 cyl 13052 alt 2 hd 255 sec 63>
          /pci@8,600000/fibre-channel@1/fp@0,0/ssd@w5006016a39a01736,0
       1. c1t5006016A39A01736d1 <DGC-RAID5-0322 cyl 51198 alt 2 hd 256 sec 16>
          /pci@8,600000/fibre-channel@1/fp@0,0/ssd@w5006016a39a01736,1
       2. c1t5006016A39A01736d2 <DGC-RAID 3-0326-1.79TB>
          /pci@8,600000/fibre-channel@1/fp@0,0/ssd@w5006016a39a01736,2
       3. c1t5006016A39A01736d3 <DGC-RAID 3-0326-1.79TB>
          /pci@8,600000/fibre-channel@1/fp@0,0/ssd@w5006016a39a01736,3
       4. c1t5006016A39A01736d4 <DGC-RAID3-0326 cyl 59851 alt 2 hd 255 sec 252>
          /pci@8,600000/fibre-channel@1/fp@0,0/ssd@w5006016a39a01736,4
       5. c1t5006016A39A01736d5 <DGC-RAID3-0326 cyl 59851 alt 2 hd 255 sec 252>
          /pci@8,600000/fibre-channel@1/fp@0,0/ssd@w5006016a39a01736,5
       6. c1t5006016A39A01736d6 <DGC-RAID 3-0326-1.79TB>
          /pci@8,600000/fibre-channel@1/fp@0,0/ssd@w5006016a39a01736,6
       7. c1t5006016A39A01736d7 <DGC-RAID 3-0326-1.79TB>
          /pci@8,600000/fibre-channel@1/fp@0,0/ssd@w5006016a39a01736,7

. . .

 
. . .
 
----On closer inspection of all the format device paths for a single EFI 1.79TB LUN ---say for example LUN “d20” --- we discover format declares “1.79TB” properly for some of the d20 device paths but not for others of this twelve device path LUN named “d20”.
 
 
$ cat format |egrep "d20" |more
      20. c1t5006016A39A01736d20 <DGC-RAID3-0326 cyl 59851 alt 2 hd 255 sec 252>
     103. c1t5006016239A01736d20 <DGC-RAID 3-0326-1.79TB>
     104. c1t5006016839A01736d20 <DGC-RAID3-0326 cyl 59851 alt 2 hd 255 sec 252>
     105. c1t5006016039A01736d20 <DGC-RAID 3-0326-1.79TB>
     192. c2t5006016A39A01736d20 <DGC-RAID3-0326 cyl 59851 alt 2 hd 255 sec 252>
     275. c2t5006016839A01736d20 <DGC-RAID3-0326 cyl 59851 alt 2 hd 255 sec 252>
     276. c2t5006016039A01736d20 <DGC-RAID 3-0326-1.79TB>
     277. c2t5006016239A01736d20 <DGC-RAID 3-0326-1.79TB>
     364. c4t5006016B39A01736d20 <DGC-RAID3-0326 cyl 59851 alt 2 hd 255 sec 252>
     447. c4t5006016339A01736d20 <DGC-RAID 3-0326-1.79TB>
     448. c4t5006016939A01736d20 <DGC-RAID3-0326 cyl 59851 alt 2 hd 255 sec 252>
     449. c4t5006016139A01736d20 <DGC-RAID 3-0326-1.79TB>
 
 
----Correspondingly, the device driver-link build for the /dev/dsk device tree is inconsistent. We see the distinguishing “wd” (whole disk) EFI ioctl paths for some entries but also SMI partition “h” defined for other ioctl paths. 
 
$ ls –al /dev/dsk  |egrep "1736d20" |egrep ":wd|:h"
lrwxrwxrwx   1 root     root          77 Dec 23 17:51 c1t5006016039A01736d20 -> ../../devices/pci@8,600000/fibre-channel@1/fp@0,0/ssd@w5006016039a01736,14:wd
lrwxrwxrwx   1 root     root          77 Dec 23 17:51 c1t5006016239A01736d20 -> ../../devices/pci@8,600000/fibre-channel@1/fp@0,0/ssd@w5006016239a01736,14:wd
lrwxrwxrwx   1 root     root          76 Dec 23 17:51 c1t5006016839A01736d20s7 -> ../../devices/pci@8,600000/fibre-channel@1/fp@0,0/ssd@w5006016839a01736,14:h
lrwxrwxrwx   1 root     root          76 Dec 23 17:51 c1t5006016A39A01736d20s7 -> ../../devices/pci@8,600000/fibre-channel@1/fp@0,0/ssd@w5006016a39a01736,14:h
lrwxrwxrwx   1 root     root          79 Dec 23 17:52 c2t5006016039A01736d20 -> ../../devices/pci@8,600000/fibre-channel@1,1/fp@0,0/ssd@w5006016039a01736,14:wd
lrwxrwxrwx   1 root     root          79 Dec 23 17:52 c2t5006016239A01736d20 -> ../../devices/pci@8,600000/fibre-channel@1,1/fp@0,0/ssd@w5006016239a01736,14:wd
lrwxrwxrwx   1 root     root          78 Dec 23 17:52 c2t5006016839A01736d20s7 -> ../../devices/pci@8,600000/fibre-channel@1,1/fp@0,0/ssd@w5006016839a01736,14:h
lrwxrwxrwx   1 root     root          78 Dec 23 17:52 c2t5006016A39A01736d20s7 -> ../../devices/pci@8,600000/fibre-channel@1,1/fp@0,0/ssd@w5006016a39a01736,14:h
lrwxrwxrwx   1 root     root          79 Dec 23 17:53 c4t5006016139A01736d20 -> ../../devices/pci@8,600000/fibre-channel@2,1/fp@0,0/ssd@w5006016139a01736,14:wd
lrwxrwxrwx   1 root     root          79 Dec 23 17:52 c4t5006016339A01736d20 -> ../../devices/pci@8,600000/fibre-channel@2,1/fp@0,0/ssd@w5006016339a01736,14:wd
lrwxrwxrwx   1 root     root          78 Dec 23 17:53 c4t5006016939A01736d20s7 -> ../../devices/pci@8,600000/fibre-channel@2,1/fp@0,0/ssd@w5006016939a01736,14:h
lrwxrwxrwx   1 root     root          78 Dec 23 17:53 c4t5006016B39A01736d20s7 -> ../../devices/pci@8,600000/fibre-channel@2,1/fp@0,0/ssd@w5006016b39a01736,14:h
 
These EFI verses SMI device path inconsistencies causes VxVM ‘vxdisk –list’ to declare status “error” for the dmpnodes we see with vxdisk-list output.

Environment



 

---- With the release of Solaris 10 3/05 LUNS greater than 1 terabyte became supported with the Extensible Firmware Interface (EFI) label surpassing the 1 terabyte limit of the Sun Microsystems, Inc (SMI) label default. 
 
---- With the release of Solaris 10 kernel update 8 10/09 (patch 141444-09) new problems with EFI LUNs surfaced, particularly with storage arrays with “active-passive” controller environments.  (for kernel updates please see  http://blogs.sun.com/patch/entry/solaris_10_kernel_patchid_progression)
 
---- Solaris 10 update 8 also included extending the SMI VTOC (EXTVTOC) 1 terabyte restriction to 2 terabytes allowing the support for Solaris boot drives up to 2TB. This introduced new problems when the OS tries to identify and size SMI verses EFI disk labels particularly when there are multiple active-passive device paths to the LUN.
 
---- Oracle-Sun subsequently confirmed the problem as the bug CR 6912703:  ssd lun with EFI label presents SMI minor device :h rather than EFI device :wd;  involves active-passive arrays. 
 
---- Oracle-Sun support recommended the temporary work-around: return to a pre-update 8 version such as Solaris 10 kernel update 7 (141414-10).
 
---- Effort to resolve EFI bugs for various customers resulted in Oracle-Sun IDRs (Interim Diagnostic/Relief (IDR) i.e. hotfix for OS kernel) that evolved to today’s latest (11/1/2010) IDR, IDR144101-17.
 
---- Unfortunately, Oracle-Sun’s recent release of Solaris kernel update 9 kernel patch (142909-17) does not include the EFI sd/sdd driver bug fix provided by IDR1444101-17.
---- Therefore IDR1444101-17 should temporarily be applied --- available only upon request from Oracle-Sun support ---  this IDR requires Solaris 10 kernel update 9 patch 1142909-17.
 
---- In addition, the Sun-Oracle changes made to address the OS EFI problems required Symantec collaboration with VxVM DMP “device discovery” code changes to accommodate Solaris 10 OS presentation of EFI LUNs to VxVM. 
 
---- Furthermore, subsequent VxVM tests against the referred IDR (see below) revealed new EFI problems with device discovery creating incidents e2125217 and e207115 leading Engineering to create a VxVM hotfix.  Hotfix 5.0MP3RP4HF5 is now available for proper VxVM support for EFI LUNs with active-passive arrays.

Cause



 

 
Combination of both Solaris 10 kernel update 8 and VxVM version 5.x creates the inability to properly detect and declare EFI label LUNs.

Solution



 

Inter-dependent Requirements for Solution:
 
A)  Solaris 10 update 9 (142909-17) required
# uname -a
SunOS host1.xyz.com 5.10 Generic_142909-17 sun4u sparc SUNW,Sun-Fire-V240
----- Use format –e to label LUNs greater that 1 terabyte for all device paths of the LUN
----- prtvtoc declares partition 8 for all device paths for EFI LUNs; for example
# prtvtoc -s /dev/rdsk/c4t500601603CE02709d2
*                          First     Sector    Last
* Partition  Tag  Flags    Sector     Count    Sector  Mount Directory
       0      4    00         34 2516565949 2516565982
       8     11    00  2516565983     16384 2516582366
# prtvtoc -s /dev/rdsk/c6t500601683CE02709d2
*                          First     Sector    Last
* Partition  Tag  Flags    Sector     Count    Sector  Mount Directory
       0      4    00         34 2516565949 2516565982
       8     11    00  2516565983     16384 2516582366
#
----- Verify all format (echo |format) device paths for each EFI LUN are equivalent in device size declaration (as described above)
----- All EFI LUN device paths must have “wd” declared as the “whole disk” partition; (see example below for # ls -al /dev/dsk )
 
 
 
B)  Oracle-Sun support IDR patch:  IDR144101-17 (kernel hotfix) required
---- Contact Oracle-Sun support for this patch; refer to these Oracle documents
---- Refer to Bug ID:  6912703
ssd lun with EFI label presents SMI minor device :h rather than EFI device :wd
Solaris 10 patches 141444-09/141445-09 May Cause EFI Labeled LUNs to Become Inaccessible Due to Incorrect Device Nodes Being Presented
----- This 7/10 alert says:   “A final resolution is pending completion.”  IDR144101-17 is likely targeted for the “final resolution” late this year. Contact Oracle-Sun support for the estimated time for release.
 
 
 
C)  SAN configuration assumes no Third Party Driver (TPD) multipathing (e.g. EMC PowerPath) is used. 
---- This does not mean TPD multipath managed EFI LUNs are not supported by VxVM.
---- This technote is intended to address VxVM DMP multipath managed LUNs only.
 
 
 
D)  For Active-Passive arrays you must configure controller (or service processor) failover mode where “Unit Attention” messages are suppressed. 
---- For example, EMC Clariion active-passive array must have “mode 2” set for failover mode.
---- See related Symantec articles: TECH73743 and TECH61373
---- Note: ALAU (aka A/A-A) “mode 4” may be supported too. 
 
 
 
E)   Apply VxVM hotfix 5 for 5.0MP3RP4
---- For EFI issues this hotfix requires Solaris 10 kernel update 9 and IDR Patch IDR144101-17
----  VxVM 5.0MP3RP4HF5  README Fixed Incidents:

2159975   Enhance iohvold.so to display DA records.

 
2078115   Maxfli,X64:install product with DMP only then create zpool on dmpnode, the dmpnode show as "auto:error"
2067038   Solaris 10: Correction of EFI detection logic in DMP after fix provided by Sun/Oracle 
1951930   Maxfli DMP: all paths goes to disable state when we create a zpool on native support enabled dmp-device

---- Contact Symantec support to receive this hotfix.

 
 
 
 
F)  Post requirements procedures
---- Confirm prtvtoc for all device paths for the LUN; for example:
# prtvtoc -s /dev/rdsk/c4t500601603CE02709d0s2
*                          First     Sector    Last
* Partition  Tag  Flags    Sector     Count    Sector  Mount Directory
       3     15    01         34     65536     65569
       4     14    01      65570 4613652413 4613717982
       8     11    00  4613717983     16384 4613734366    <<<<<<<<< whole disk  (EFI)
#
# prtvtoc -s /dev/rdsk/c4t500601603CE02709d2s2
*                          First     Sector    Last
* Partition  Tag  Flags    Sector     Count    Sector  Mount Directory
       3     15    01         34     65536     65569
       4     14    01      65570 2516500413 2516565982
       8     11    00  2516565983     16384 2516582366    <<<<<<<<< whole disk  (EFI)
. . .
. . .
 
---- Verify Solaris 10 “wd” (whole disk) declaration for all device paths/ioctl links:
---- For example: 
# ls -al /dev/dsk |egrep c4t500601603CE02709d2
lrwxrwxrwx   1 root     root          72 Nov  1 17:53 c4t500601603CE02709d2 -> ../../devices/pci@1e,600000/SUNW,qlc@3/fp@0,0/ssd@w500601603ce02709,2:wd
lrwxrwxrwx   1 root     root          71 Oct  5 14:54 c4t500601603CE02709d2s0 -> ../../devices/pci@1e,600000/SUNW,qlc@3/fp@0,0/ssd@w500601603ce02709,2:a
lrwxrwxrwx   1 root     root          71 Oct  5 14:54 c4t500601603CE02709d2s1 -> ../../devices/pci@1e,600000/SUNW,qlc@3/fp@0,0/ssd@w500601603ce02709,2:b
lrwxrwxrwx   1 root     root          71 Oct  5 14:54 c4t500601603CE02709d2s2 -> ../../devices/pci@1e,600000/SUNW,qlc@3/fp@0,0/ssd@w500601603ce02709,2:c
lrwxrwxrwx   1 root     root          71 Oct  5 14:54 c4t500601603CE02709d2s3 -> ../../devices/pci@1e,600000/SUNW,qlc@3/fp@0,0/ssd@w500601603ce02709,2:d
lrwxrwxrwx   1 root     root          71 Oct  5 14:54 c4t500601603CE02709d2s4 -> ../../devices/pci@1e,600000/SUNW,qlc@3/fp@0,0/ssd@w500601603ce02709,2:e
lrwxrwxrwx   1 root     root          71 Oct  5 14:54 c4t500601603CE02709d2s5 -> ../../devices/pci@1e,600000/SUNW,qlc@3/fp@0,0/ssd@w500601603ce02709,2:f
lrwxrwxrwx   1 root     root          71 Oct  5 14:54 c4t500601603CE02709d2s6 -> ../../devices/pci@1e,600000/SUNW,qlc@3/fp@0,0/ssd@w500601603ce02709,2:g
#
# ls -al /dev/dsk |egrep c6t500601683CE02709d2
lrwxrwxrwx   1 root     root          72 Nov  1 17:54 c6t500601683CE02709d2 -> ../../devices/pci@1d,700000/SUNW,qlc@1/fp@0,0/ssd@w500601683ce02709,2:wd
lrwxrwxrwx   1 root     root          71 Oct  4 23:47 c6t500601683CE02709d2s0 -> ../../devices/pci@1d,700000/SUNW,qlc@1/fp@0,0/ssd@w500601683ce02709,2:a
lrwxrwxrwx   1 root     root          71 Oct  4 23:47 c6t500601683CE02709d2s1 -> ../../devices/pci@1d,700000/SUNW,qlc@1/fp@0,0/ssd@w500601683ce02709,2:b
lrwxrwxrwx   1 root     root          71 Oct  4 23:47 c6t500601683CE02709d2s2 -> ../../devices/pci@1d,700000/SUNW,qlc@1/fp@0,0/ssd@w500601683ce02709,2:c
lrwxrwxrwx   1 root     root          71 Oct  4 23:47 c6t500601683CE02709d2s3 -> ../../devices/pci@1d,700000/SUNW,qlc@1/fp@0,0/ssd@w500601683ce02709,2:d
lrwxrwxrwx   1 root     root          71 Oct  4 23:47 c6t500601683CE02709d2s4 -> ../../devices/pci@1d,700000/SUNW,qlc@1/fp@0,0/ssd@w500601683ce02709,2:e
lrwxrwxrwx   1 root     root          71 Oct  4 23:47 c6t500601683CE02709d2s5 -> ../../devices/pci@1d,700000/SUNW,qlc@1/fp@0,0/ssd@w500601683ce02709,2:f
lrwxrwxrwx   1 root     root          71 Oct  4 23:47 c6t500601683CE02709d2s6 -> ../../devices/pci@1d,700000/SUNW,qlc@1/fp@0,0/ssd@w500601683ce02709,2:g
 
 
 
---- If EFI label is proper for all device paths, the VxVM vxdisk-list LUN status becomes “online invalid”.
# vxdctl enable
# vxdisk list
DEVICE       TYPE            DISK         GROUP        STATUS
c1t0d0s2     auto:none       -            -            online invalid
c4t500601603CE02709d0s2 auto            -            -            error
c4t500601603CE02709d1s2 auto:none       -            -            online invalid
c4t500601603CE02709d2 auto:none       -            -            online invalid
 
 
 
---- VxVM initialization for EFI LUN requires “sliced” format; 
#  /etc/vx/bin/vxdisksetup –i <LUN_name> format=sliced; 
---- For example:
# /etc/vx/bin/vxdisksetup -i c4t500601603CE02709d2 format=sliced
# vxdisk list
DEVICE       TYPE            DISK         GROUP        STATUS
c1t0d0s2     auto:none       -            -            online invalid
c4t500601603CE02709d0s2 auto            -            -            error
c4t500601603CE02709d1s2 auto:none       -            -            online invalid
c4t500601603CE02709d2 auto:sliced     -            -            online
 
 
----- prtvtoc reveals device paths for VxVM”sliced” initialized LUN (tags 14, 15)
----- For example, prtvtoc for both device paths:
 
# prtvtoc -s /dev/rdsk/c4t500601603CE02709d2
*                          First     Sector    Last
* Partition  Tag  Flags    Sector     Count    Sector  Mount Directory
            3     15    01         34     65536     65569
            4     14    01      65570 2516500413 2516565982
            8     11    00  2516565983     16384 2516582366
 
# prtvtoc -s /dev/rdsk/c6t500601683CE02709d2
*                          First     Sector    Last
* Partition  Tag  Flags    Sector     Count    Sector  Mount Directory
            3     15    01         34     65536     65569
            4     14    01      65570 2516500413 2516565982
            8     11    00  2516565983     16384 2516582366
 


Article URL http://www.symantec.com/docs/TECH143696


Terms of use for this information are found in Legal Notices