Video Screencast Help

VRTSvxfs-6.0.300.000-SLES11 patch install reports "Failed to create /dev/vxportal"

Created: 03 May 2013 • Updated: 03 May 2013 | 4 comments
kwachtler's picture

Hi All.

The VRTSvxfs-6.0.300.000-SLES11 patch install failed while applying the SF 6.0.3 patches with installmr on top of SF 6.0.1. The RPM  verification passes (rpm -V), but the VRTSvxfs patch log shows "Failed to create /dev/vxportal" and  "ERROR: No appropriate VxFS drivers found that can be loaded."  A check of the kernel modules showed that vxportal and vxfs are not loaded. A reboot was performed to load the missing kernel modules, and VxFS now functions without any further issues.

The number of servers affected  was 4 out of ~ 50.

Q: Has anyone else experienced this issue on SLES 11.1 when applying 6.0.3 patches ?

Q: Any pointers on where to look/check on the next time that it occurs ?

 

Here are some specifics...

 

Installmr Error

Performing SF preinstall tasks ........................................................................................ Done
Installing VRTSperl patch ............................................................................................. Done
Installing VRTSspt patch .............................................................................................. Done
Installing VRTSvxvm patch ............................................................................................. Done
Installing VRTSvxfs patch ........................................................................................... Failed
Installing VRTSdbed patch ............................................................................................. Done
Installing VRTSodm patch .............................................................................................. Done
Installing VRTSsfcpi601 patch ......................................................................................... Done
Performing SF postinstall tasks ....................................................................................... Done

Veritas Storage Foundation Install did not complete successfully

VRTSvxfs patch failed to install on HHHHH

Refer to the installer logs for details and correct the settings. Continue with the installer again, or try to install the
packages/patches manually.

 

VRTSvxfs patch log

# cat opt/VRTS/install/logs/installmr-201305011327LbS/install.patch.VRTSvxfs.SVBMN14GLBSM4A

Preparing packages for installation...
VRTSvxfs-6.0.300.000-SLES11
WARNING: No modules found for 2.6.32.59-0.7-default, using compatible modules for 2.6.32.12-0.7-default.
Failed to create /dev/vxportal
ERROR: No appropriate VxFS drivers found that can be loaded. See VxFS documentation for the list of supported platforms.

 

Platform

# rpm -qa *-release
sles-release-11.1-1.206

# uname -r
2.6.32.59-0.7-default

# uname -m
x86_64

 

Full installmr

# ./installmr -responsefile ../installmr_SF603_patchupgrade

Logs are being written to /var/tmp/installmr-201305011327LbS while installmr is in progress

    Verifying systems: 100%                                                                                                     

    Estimated time remaining: (mm:ss) 0:00                                                                                8 of 8

    Checking system communication ......................................................................................... Done
    Checking release compatibility ........................................................................................ Done
    Checking installed product ............................................................................................ Done
    Checking prerequisite patches and rpms ................................................................................ Done
    Checking platform version ............................................................................................. Done
    Checking file system free space ....................................................................................... Done
    Checking product licensing ............................................................................................ Done
    Performing product prechecks .......................................................................................... Done

System verification checks completed successfully

The following notes were discovered on the systems:

CPI NOTE V-9-30-1309 SF version 6.0.100.000 will be upgraded to SF version 6.0.300.000 on HHHHH

 

The following Veritas Storage Foundation patches will be installed on all systems:

Patch                Rpm

VRTSperl-5.14.2.8    VRTSperl
VRTSsfcpi601-6.0.300.0 VRTSsfcpi601
VRTSspt-6.0.300.0    VRTSspt
VRTSvxvm-6.0.300.0   VRTSvxvm
VRTSvxfs-6.0.300.0   VRTSvxfs
VRTSdbed-6.0.300.0   VRTSdbed
VRTSodm-6.0.100.100  VRTSodm

 

Logs are being written to /var/tmp/installmr-201305011327LbS while installmr is in progress

    Stopping SF: 100%                                                                                                           

    Estimated time remaining: (mm:ss) 0:00                                                                              18 of 18

    Performing SF prestop tasks ........................................................................................... Done
    Stopping vxsvc ........................................................................................................ Done
    Stopping vxodm ........................................................................................................ Done
    Stopping vxdbd ........................................................................................................ Done
    Stopping fdd .......................................................................................................... Done
    Stopping vxportal ..................................................................................................... Done
    Stopping vxfs ......................................................................................................... Done
    Stopping sfmh-discovery ............................................................................................... Done
    Stopping vxdclid ...................................................................................................... Done
    Stopping vxconfigbackupd .............................................................................................. Done
    Stopping vxsited ...................................................................................................... Done
    Stopping vxrelocd ..................................................................................................... Done
    Stopping vxnotify ..................................................................................................... Done
    Stopping vxesd ........................................................................................................ Done
    Stopping vxconfigd .................................................................................................... Done
    Stopping vxspec ....................................................................................................... Done
    Stopping vxio ......................................................................................................... Done
    Stopping vxdmp ........................................................................................................ Done

Veritas Storage Foundation Shutdown completed successfully

 

 

Logs are being written to /var/tmp/installmr-201305011327LbS while installmr is in progress

    Installing SF: 100%                                                                                                         

    Estimated time remaining: (mm:ss) 0:00                                                                                9 of 9

    Performing SF preinstall tasks ........................................................................................ Done
    Installing VRTSperl patch ............................................................................................. Done
    Installing VRTSspt patch .............................................................................................. Done
    Installing VRTSvxvm patch ............................................................................................. Done
    Installing VRTSvxfs patch ........................................................................................... Failed
    Installing VRTSdbed patch ............................................................................................. Done
    Installing VRTSodm patch .............................................................................................. Done
    Installing VRTSsfcpi601 patch ......................................................................................... Done
    Performing SF postinstall tasks ....................................................................................... Done

Veritas Storage Foundation Install did not complete successfully

VRTSvxfs patch failed to install on HHHHH

Refer to the installer logs for details and correct the settings. Continue with the installer again, or try to install the
packages/patches manually.

installmr log files and summary file are saved at:

        /opt/VRTS/install/logs/installmr-201305011327LbS

 

RPM List After 6.0.3 patch

# rpm -qa VRTS\* | sort | cat -b
1 VRTSaslapm-6.0.100.000-GA_SLES11
2 VRTSdbed-6.0.300.000-GA_SLES
3 VRTSfsadv-6.0.100.000-GA_SLES11
4 VRTSfssdk-6.0.100.000-GA_SLES11
5 VRTSlvmconv-6.0.100.000-GA_SLES11
6 VRTSob-3.4.553-0
7 VRTSodm-6.0.100.100-SLES11
8 VRTSperl-5.14.2.8-SLES11
9 VRTSsfcpi601-6.0.300.000-GA_GENERIC
10 VRTSsfmh-5.0.196.0-0
11 VRTSspt-6.0.300.000-GA
12 VRTSvlic-3.02.61.004-0
13 VRTSvxfs-6.0.300.000-SLES11
14 VRTSvxvm-6.0.300.000-SLES11

 

 

Kernel Modules Before Reboot

# lsmod | grep vx
vxspec                  3622  2
vxio                 3924994  1 vxspec
vxdmp                 423852  482 vxspec,vxio

 

Kernel Modules After Reboot

 # lsmod | grep vx
vxodm                 216428  1
vxspec                  3622  6
vxio                 3924994  46 vxspec
vxdmp                 423852  482 vxspec,vxio
vxportal                6695  0
fdd                    55690  2 vxodm
vxfs                 3651117  49 vxportal,fdd

~~~~

end

 

 

 

 

 

 

 

 

 

Operating Systems:

Comments 4 CommentsJump to latest comment

stinsong's picture

Hi kwachtler,

It seems you met the same issue with RHEL 6.4.

Pls check on public technote: http://www.symantec.com/docs/TECH203099

Maybe Symantec is working on fix of the issue, but I didn't find available solution currently from public info.

To open a tech support case from Symantec is better to work you out the situation.

rsharma1's picture

Hi kwachtler,

                      By any chance do the nodes with failure have relatively more number of luns/dmp paths?       Incase you're doing more of these upgrades  & encounter this again, instead of a reboot can you try starting vxfs manually using :/etc/vx/vxfs-startup and share the result?

 

kwachtler's picture

Thank you for comments above. In response...

1. Yes, similar to RHEL 6.4 issue, but this may be 1st report of this issue on SLES11, so wanted to raise it here.

2. Symantec support case has been opened, but at a lower priority, since workaround exists (reboot).

3. Yes, all 4 systems affected have a high number of LUN/paths, so that could be a significant clue. These systems are non-clustered, so just basic Storage Foundation.

       2 systems @  200 LUNs,  1600 paths  (8 paths per LUN)

       2  systems @ 240 LUNs ,  960 paths   (4 paths per LUN)

I am waiting for Symantec TS to weigh-in via the support case.

Thank You all.

 

rsharma1's picture

at 6.0.3, /etc/vx/vxfs-startup script should have the implementation to recreate /dev/vxportal if it failed for some reason (you may want to cat the script to doublecheck) . Instead of reboot, retrying to start vxfs by executing this script might be a better w/around than reboot. In some earlier instances it was seen that /dev/udevd (thats reponsible for creating devices under /dev was busy updating dmp devices (in case of large number of devices) ..that is was not able to service the request for creating /dev/vxportal  -probably a timing issue. Good that you already opened a support case for a detailed analysis here.