Storage Foundation (SF) dynamic multipathing driver (vxdmp) may fail to load on IBM AIX machines with large amounts of physical memory

Article:TECH148028  |  Created: 2011-01-12  |  Updated: 2011-10-03  |  Article URL http://www.symantec.com/docs/TECH148028
Article Type
Technical Solution


Environment

Issue



The DMP driver load fails after upgrading to 5.1SP1 on AIX 5.3, AIX 6.1 or 5.1SP1PR1 (Platform Release 1) on AIX 7.1. This issue is noticed on some systems with memory greater than 35GB. This is due to failure in creating memory pool which is utilized to serve I/Os in DMP.

The problem can also cause system panic if VxVM command tries to open DMP device files (e.g. /dev/vx/dmpconfig) after the DMP driver failed to load into the memory.


Error



The following messages are seen in the errpt when the vxdmp module fails to load:

WARNING VxVM vxdmp V-5-0-1442 : Could not create dmppool error code 12 retry 0
WARNING VxVM vxdmp V-5-0-1481:dmp ldata pool creation failed

Following this the Volume Manager configuration daemon (vxconfigd) fails to start due to the vxdmp module not being loaded on the system:

# vxconfigd -k
VxVM vxconfigd FATAL ERROR V-5-1-8379 vxdmp driver not loaded on the system. This version of VxVM mandatorily requires vxdmp to be loaded. Exiting.   

# lsdev -C | grep vx
vxdmp          Defined                 Veritas VxDMP Device Driver   <<<= Defined but not Available
vxfen          Available               N/A
vxfend         Available               N/A
vxglm0         Available               N/A
vxgms0         Available               N/A
vxio           Defined                 Veritas VxIO Device Driver
vxportal0      Available               VERITAS VxPORTAL Device Driver
vxqio0         Available               VERITAS VxQIO Device Driver
vxspec         Defined                 Veritas VxSPEC Device Driver

 

The following is the system panic stack when a VxVM command (vxesd in this case) tried to open the DMP device file (/dev/vx/dmpconfig) after the DMP driver failed to load into the memory.

(0)> th * | grep pvthread+02A200
pvthread+02A200  674*vxesd    RUN   2A2023 03C    0         0               <<< vxesd command

CRASH INFORMATION:
CPU 0 CSA F00000002FF47600 at time of crash, error code for LEDs: 70000000
pvthread+02A200 STACK:
[04CF9760]04CF9760 ()
[003F7380]devcopen+00033C (??, ??, ??, ??, ??)
[003F6CE0]rdevopen+000094 (??, ??, ??, ??, ??)
[0052E970]cdev_open+000054 (F10001005FE8B5F0, 0000000300000003, 0000000000000000)
[004CE018]spec_open@AF69_29+00016C (??, ??, ??, ??, ??, ??)
[0040AD3C]vnop_open+0000E8 (??, ??, ??, ??, ??)
[0044CF98]openpnp+0002FC (??, ??, ??, ??, ??)
[0044D47C]openpath+0000BC (??, ??, ??, ??, ??, ??, ??)
[0044D738]copen+0001F4 (??, ??, ??, ??, ??)
[0044CC38]open+00001C (??, ??, ??)
[04DA5A88]my_open+0003A4 (000000002001D2A0, 0000000200000002, 2002831820028318)
[00003810].svc_instr+000110 ()
[kdb_get_memory] no real storage @ 2FF22980

 

In some cases, we have seen the following panic stack pointing to OS syscall ldata_create().

(4)> f  
pvthread+80EA00 STACK:  
[0052E89C]vm_place_common+00027C (0000000000000000, 0000000000000000,  
 00000000000002A0, 00000000000017A0, 000000000000000E, 0000000000000000,  
 0000000000000002 [??])  
[00532154]vm_place+000034 (??, ??, ??, ??, ??, ??)  
[001F0CE4]ldata_create+000444 (??, ??, ??, ??, ??)          <== NOTE
[05CCA3FC]dmp_ldata_create+000044 (0000000005D40D40, 0000000005D40D60)  
[05CCA530]ldata_pool_init_create+0000E8 (0000000005D40D40, 0000000000000188,  
 00000000000000FA, 00A72F0539436558)  
[05CCA6C0]dmp_ldata_init+0000A4 ()  
[05CD61D8]dmp_init_modules+00002C ()  
[05D242E4]dmp_load+000024 ()  
[05CC1510]vxdmp_config+000224 (800000280000FFFF, 0000000100000001,  
 F00000002FF46C58)  
[005F2430]config_dd+000270 (??, ??, ??)  
[005F2F00]sysconfig+000240 (??, ??, ??)  
[00003850]ovlya_addr_sc_flih_main+000130 ()  
[kdb_get_virtual_memory] no real storage @ 2FF21A20  
[100009AC]100009AC ()  
[kdb_read_mem] no real storage @ FFFFFFFFFFF9900  
(4)>  


 

 


Environment



AIX 5.3/6.1/7.1

Storage Foundation 5.1SP1 for AIX 5.3 and 6.1 or 5.1SP1PR1 (Platform Release 1) for AIX 7.1.

Large amount of physical memory present on the system (i.e. > 35Gb)


Cause



The issue is caused by the way in which the vxdmp module attempts to perform initial memory allocation when loaded on the system. In the 5.1SP1 product the module attempts to reserve a given percentage of physical memory for use by vxdmp memory pools. On systems with large amounts of physical memory this can lead to a request for an extremely large allocation of memory which the system cannot honour, and as such the module fails to load as expected.


Solution



To prevent the issue, changes have been made to vxdmp code to ensure that an upper threshold is placed on the amount of memory requested during module load. This ensures that even on systems with large amounts of physical memory DMP will only request a reasonable amount of memory be allocated for vxdmp memory pools.

This issue is resolved with following patches for Veritas Volume Manager on AIX:

- Veritas Storage Foundation 5.1SP1 RP1 for AIX 5.3 and AIX 6.1  (Rolling Patch 1)
- VxVM 5.1SP1 P2 for AIX 5.3 and AIX 6.1 (P-Patch 2)
- VxVM 5.1SP1 PR1P1 for AIX 7.1 (Platform Release 1 + P-Patch 1)
 

The above patches can be downloaded from SORT site https://sort.symantec.com/patch/matrix
 


Supplemental Materials

SourceETrack
Value2226248
Description

VXDMP driver loading failed after SP1 applied




Article URL http://www.symantec.com/docs/TECH148028


Terms of use for this information are found in Legal Notices