Video Screencast Help
Symantec to Separate Into Two Focused, Industry-Leading Technology Companies. Learn more.

VxVM vxdmp V-5-3-0 memory allocation failed for size =0x1d1000

Created: 18 Feb 2014 | 21 comments

Hi Everyone !

Have a RedHat 6.5 with these kernel messages :

VxVM vxdmp V-5-3-0 memory allocation failed for size =0x1d1000
VxVM vxdmp V-5-3-0 memory allocation failed for size =0x1d1000
VxVM vxdmp V-5-3-0 memory allocation failed for size =0x1d1000
...
...

$ cat /etc/redhat-release
Red Hat Enterprise Linux Server release 6.5 (Santiago)

$ uname -r
2.6.32-431.5.1.el6.x86_64

 $ rpm -qa | grep -i vrt
VRTSfsadv-6.0.100.000-GA_RHEL6.x86_64
VRTSvxfs-6.0.300.300-RHEL6.x86_64
VRTSpbx-1.5.0.6-6.x86_64
VRTSvlic-3.02.61.004-0.x86_64
VRTSob-3.4.553-0.i686
VRTSspt-6.0.300.000-GA.noarch
VRTSsfcpi601-6.0.300.000-GA_GENERIC.noarch
VRTSvxvm-6.0.300.200-RHEL6.x86_64
VRTSaslapm-6.0.100.000-GA_RHEL6.x86_64
VRTSperl-5.14.2.8-RHEL6.1.x86_64
VRTSsfmh-5.0.196.0-0.i686
VRTSdbed-6.0.300.100-GA_RHEL.x86_64
VRTSobgui-3.4.30.0-0.i686
VRTSodm-6.0.300.300-RHEL6.x86_64

Any ideas what is going on ?

Operating Systems:

Comments 21 CommentsJump to latest comment

Gaurav Sangamnerkar's picture

Hello,

What array you are using ? The errors started coming after any recent change/upgrade ?

any other errors in dmpevents.log file ?

G

PS: If you are happy with the answer provided, please mark the post as solution. You can do so by clicking link "Mark as Solution" below the answer provided.
 

sverrewl's picture

Arrays are on SATA and SAS disks in Nexsan E60s.

# vxdisk list
DEVICE       TYPE            DISK         GROUP        STATUS
NexE60_SAS_03_0 auto:cdsdisk    prod8_dg0_0  prod8_dg0    online
NexE60_SAS_03_1 auto:cdsdisk    prod8_dg0_1  prod8_dg0    online
NexE60_03_0  auto:cdsdisk    sdaa         prod8_dg2    online
NexE60_03_1  auto:cdsdisk    prod8_dg3_0  prod8_dg3    online
NexE60_03_2  auto:cdsdisk    prod8_dg4_0  prod8_dg4    online
NexE60_03_3  auto:cdsdisk    sdz          prod8_dg1    online
sda          auto:none       -            -            online invalid

End of curent /etc/vx/dmpevents.log

Tue Feb 18 11:31:50.548: Unmarked as idle Path sdy belonging to Dmpnode NexE60_SAS_03_1
Tue Feb 18 11:31:51.549: Unmarked as idle Path sdm belonging to Dmpnode NexE60_SAS_03_1
Tue Feb 18 11:32:03.550: Unmarked as idle Path sdo belonging to Dmpnode NexE60_03_3
Tue Feb 18 11:32:03.550: Unmarked as idle Path sdc belonging to Dmpnode NexE60_03_3
Tue Feb 18 11:32:03.550: Unmarked as idle Path sdp belonging to Dmpnode NexE60_03_0
Tue Feb 18 11:32:03.550: Unmarked as idle Path sdd belonging to Dmpnode NexE60_03_0
Tue Feb 18 11:32:03.550: Unmarked as idle Path sdn belonging to Dmpnode NexE60_03_2
Tue Feb 18 11:32:03.550: Unmarked as idle Path sdb belonging to Dmpnode NexE60_03_2
Tue Feb 18 11:32:51.558: Marked as idle Path sdm belonging to Dmpnode NexE60_SAS_03_1
Tue Feb 18 11:32:51.558: Marked as idle Path sdy belonging to Dmpnode NexE60_SAS_03_1
Tue Feb 18 11:33:03.561: Marked as idle Path sdo belonging to Dmpnode NexE60_03_3
Tue Feb 18 11:33:03.561: Marked as idle Path sdc belonging to Dmpnode NexE60_03_3
Tue Feb 18 11:33:03.561: Marked as idle Path sdp belonging to Dmpnode NexE60_03_0
Tue Feb 18 11:33:03.561: Marked as idle Path sdd belonging to Dmpnode NexE60_03_0
Tue Feb 18 11:33:03.561: Marked as idle Path sdn belonging to Dmpnode NexE60_03_2
Tue Feb 18 11:33:03.561: Marked as idle Path sdb belonging to Dmpnode NexE60_03_2
Tue Feb 18 11:34:50.564: Unmarked as idle Path sdy belonging to Dmpnode NexE60_SAS_03_1
Tue Feb 18 11:34:54.564: Unmarked as idle Path sdm belonging to Dmpnode NexE60_SAS_03_1
Tue Feb 18 11:35:50.571: Marked as idle Path sdy belonging to Dmpnode NexE60_SAS_03_1
Tue Feb 18 11:35:54.571: Marked as idle Path sdm belonging to Dmpnode NexE60_SAS_03_1
Tue Feb 18 11:36:08.575: Unmarked as idle Path sdm belonging to Dmpnode NexE60_SAS_03_1
Tue Feb 18 11:36:08.575: Unmarked as idle Path sdy belonging to Dmpnode NexE60_SAS_03_1
Tue Feb 18 11:37:47.602: Marked as idle Path sdy belonging to Dmpnode NexE60_SAS_03_1
Tue Feb 18 11:38:50.613: Marked as idle Path sdm belonging to Dmpnode NexE60_SAS_03_1
Tue Feb 18 11:40:53.624: Unmarked as idle Path sdm belonging to Dmpnode NexE60_SAS_03_1
Tue Feb 18 11:40:53.624: Unmarked as idle Path sdy belonging to Dmpnode NexE60_SAS_03_1
Tue Feb 18 11:41:54.629: Marked as idle Path sdm belonging to Dmpnode NexE60_SAS_03_1
Tue Feb 18 11:41:54.629: Marked as idle Path sdy belonging to Dmpnode NexE60_SAS_03_1
Tue Feb 18 11:43:27.639: Unmarked as idle Path sdm belonging to Dmpnode NexE60_SAS_03_1
Tue Feb 18 11:43:27.639: Unmarked as idle Path sdy belonging to Dmpnode NexE60_SAS_03_1

sverrewl's picture

Uh - forgot to mention. Yes, came after upgrade from 6.0.3 on RedHat 6.4.

Did upgrade of SF with the following patches :

dbed-rhel-VRTSdbed-6.0.3.100-patches.tar.gz, fs-rhel6_x86_64-6.0.3.300-rpms.tar.gz and vm-rhel6_x86_64-6.0.3.200-rpms.tar.gz.

As I understand it, the patches above is required to run 6.5 - wich we now are on.

sverrewl's picture

Uh - where did my previous post go ?? Trying again.

Arrays are Nexsan E60'.

# vxdisk list

DEVICE       TYPE            DISK         GROUP        STATUS
NexE60_SAS_03_0 auto:cdsdisk    prod8_dg0_0  prod8_dg0    online
NexE60_SAS_03_1 auto:cdsdisk    prod8_dg0_1  prod8_dg0    online
NexE60_03_0  auto:cdsdisk    sdaa         prod8_dg2    online
NexE60_03_1  auto:cdsdisk    prod8_dg3_0  prod8_dg3    online
NexE60_03_2  auto:cdsdisk    prod8_dg4_0  prod8_dg4    online
NexE60_03_3  auto:cdsdisk    sdz          prod8_dg1    online
sda          auto:none       -            -            online invalid

# tail -20 /etc/vx/dmpevents.log
tail: inotify cannot be used, reverting to polling
Tue Feb 18 11:41:54.629: Marked as idle Path sdm belonging to Dmpnode NexE60_SAS_03_1
Tue Feb 18 11:41:54.629: Marked as idle Path sdy belonging to Dmpnode NexE60_SAS_03_1
Tue Feb 18 11:43:27.639: Unmarked as idle Path sdm belonging to Dmpnode NexE60_SAS_03_1
Tue Feb 18 11:43:27.639: Unmarked as idle Path sdy belonging to Dmpnode NexE60_SAS_03_1
Tue Feb 18 11:55:26.789: Marked as idle Path sdm belonging to Dmpnode NexE60_SAS_03_1
Tue Feb 18 11:55:26.789: Marked as idle Path sdy belonging to Dmpnode NexE60_SAS_03_1
Tue Feb 18 11:57:10.829: Unmarked as idle Path sdy belonging to Dmpnode NexE60_SAS_03_1
Tue Feb 18 11:57:16.833: Unmarked as idle Path sdm belonging to Dmpnode NexE60_SAS_03_1
Tue Feb 18 12:02:32.888: Unmarked as idle Path sdo belonging to Dmpnode NexE60_03_3
Tue Feb 18 12:02:32.888: Unmarked as idle Path sdc belonging to Dmpnode NexE60_03_3
Tue Feb 18 12:02:32.888: Unmarked as idle Path sdp belonging to Dmpnode NexE60_03_0
Tue Feb 18 12:02:32.888: Unmarked as idle Path sdd belonging to Dmpnode NexE60_03_0
Tue Feb 18 12:02:32.888: Unmarked as idle Path sdn belonging to Dmpnode NexE60_03_2
Tue Feb 18 12:02:32.888: Unmarked as idle Path sdb belonging to Dmpnode NexE60_03_2
Tue Feb 18 12:03:32.899: Marked as idle Path sdo belonging to Dmpnode NexE60_03_3
Tue Feb 18 12:03:32.899: Marked as idle Path sdc belonging to Dmpnode NexE60_03_3
Tue Feb 18 12:03:32.899: Marked as idle Path sdp belonging to Dmpnode NexE60_03_0
Tue Feb 18 12:03:32.899: Marked as idle Path sdd belonging to Dmpnode NexE60_03_0
Tue Feb 18 12:03:32.899: Marked as idle Path sdn belonging to Dmpnode NexE60_03_2
Tue Feb 18 12:03:32.899: Marked as idle Path sdb belonging to Dmpnode NexE60_03_2

sverrewl's picture

Tried twice to post output of dmpevents.log - trying by attachment instead.

AttachmentSize
syma1.txt 2.28 KB
Gaurav Sangamnerkar's picture

Because of continuous posting within short span the spam filter came active & hence you were unable to see your posts, anyways have published all your comments ..

From the known errors I don't see anything relevant to above errors, doing some more search to find relevant solutions.

One thing I see though is, RedHat 6.5 is not in supported list even for 6.1

From SF6.1 notes

Operating systems                        Levels             Kernel version
Red Hat Enterprise Linux 6            Update 3        2.6.32-279.el6
                                                   Update 4        2.6.32-358.el6

Red Hat Enterprise Linux 5            Update 5        2.6.18-194.el5
                                                   Update 6        2.6.18-238.el5
                                                   Update 7        2.6.18-274.el5
                                                   Update 8        2.6.18-308.el5
                                                   Update 9        2.6.18-348.el5

G

PS: If you are happy with the answer provided, please mark the post as solution. You can do so by clicking link "Mark as Solution" below the answer provided.
 

Gaurav Sangamnerkar's picture

Hi,

Supportability is not an issue, just saw a LBN article which says

http://www.symantec.com/docs/TECH164885

Support for Red Hat Enterprise Linux 6 Update 5 (RHEL 6.5) on SFHA Solutions
A compatibility issue has been discovered on Red Hat Enterprise Linux 6 update 5 (RHEL 6.5) with Storage Foundation and High Availability Solutions 6.0.1, 6.0.2 & 6.0.3. The symptom is that the VxFS component cannot be loaded or used. Symantec have released a Public patch for SFHA 6.0.3 to resolve this issue. Customers at SFHA 6.0.1 or 6.0.2 are required to upgrade to SFHA 6.0.3 before installing this patch.

So you seem to be on right levels & issue is at DMP layer itself

can u paste below outputs:

# vxdmpadm listctlr all

# vxddladm list devices

G

PS: If you are happy with the answer provided, please mark the post as solution. You can do so by clicking link "Mark as Solution" below the answer provided.
 

sverrewl's picture

# vxdmpadm listctlr all
CTLR_NAME       ENCLR_TYPE      STATE        ENCLR_NAME      PATH_COUNT
=========================================================================
c1              NXESERIES       ENABLED      NexE60_03            8
c2              NXESERIES       ENABLED      NexE60_03            8
c1              NXESERIES       ENABLED      NexE60_SAS_03        4
c2              NXESERIES       ENABLED      NexE60_SAS_03        4
c0              Disk            ENABLED      disk                 1

# vxddladm list devices
DEVICE               TARGET-ID    STATE   DDL-STATUS (ASL)
===============================================================
sdb                  c1_p4_t0     Online  CLAIMED (libvxnexsan.so)
sdd                  c1_p4_t0     Online  CLAIMED (libvxnexsan.so)
sdc                  c1_p4_t0     Online  CLAIMED (libvxnexsan.so)
sde                  c1_p4_t0     Online  CLAIMED (libvxnexsan.so)
sdf                  c1_p3_t0     Online  CLAIMED (libvxnexsan.so)
sdi                  c1_p3_t0     Online  CLAIMED (libvxnexsan.so)
sdg                  c1_p3_t0     Online  CLAIMED (libvxnexsan.so)
sdh                  c1_p3_t0     Online  CLAIMED (libvxnexsan.so)
sdj                  c1_p2_t0     Online  CLAIMED (libvxnexsan.so)
sdk                  c1_p2_t0     Online  CLAIMED (libvxnexsan.so)
sdl                  c1_p1_t0     Online  CLAIMED (libvxnexsan.so)
sdm                  c1_p1_t0     Online  CLAIMED (libvxnexsan.so)
sda                  -            Online  CLAIMED (Disk)
sdn                  c2_p4_t0     Online  CLAIMED (libvxnexsan.so)
sdo                  c2_p4_t0     Online  CLAIMED (libvxnexsan.so)
sdq                  c2_p4_t0     Online  CLAIMED (libvxnexsan.so)
sdp                  c2_p4_t0     Online  CLAIMED (libvxnexsan.so)
sdr                  c2_p3_t0     Online  CLAIMED (libvxnexsan.so)
sds                  c2_p3_t0     Online  CLAIMED (libvxnexsan.so)
sdt                  c2_p3_t0     Online  CLAIMED (libvxnexsan.so)
sdu                  c2_p3_t0     Online  CLAIMED (libvxnexsan.so)
sdv                  c2_p2_t0     Online  CLAIMED (libvxnexsan.so)
sdw                  c2_p2_t0     Online  CLAIMED (libvxnexsan.so)
sdx                  c2_p1_t0     Online  CLAIMED (libvxnexsan.so)
sdy                  c2_p1_t0     Online  CLAIMED (libvxnexsan.so)

sverrewl's picture

Spam-filter takes forever - at least half hour since I updated with the output.

AttachmentSize
syma2.txt 2.27 KB
Gaurav Sangamnerkar's picture

not sure why spam filter is getting active ..

can you also give

# vxdmpadm listenclosure all

G

PS: If you are happy with the answer provided, please mark the post as solution. You can do so by clicking link "Mark as Solution" below the answer provided.
 

sverrewl's picture

#  vxdmpadm listenclosure all
ENCLR_NAME        ENCLR_TYPE     ENCLR_SNO      STATUS       ARRAY_TYPE     LUN_COUNT
=======================================================================================
disk              Disk           DISKS                CONNECTED    Disk        1
NexE60_SAS_03     NXESERIES      00000000402D80007539  CONNECTED    ALUA        2
NexE60_03         NXESERIES      E0000000402D8000417D  CONNECTED    ALUA        4

Gaurav Sangamnerkar's picture

As I said above, I don't see anything from known issue point of view however I did observe one thing that you have array configured with ALUA mode however if we look at hardware compatibility list(HCL) for 6.0 or 6.1, the Nexsan arrays are supported with A/A-A which is Active/Active - Asymmetric mode

HCL can be found here

http://www.symantec.com/docs/TECH170013

You might wanna check tuning the array to A/A-A mode & see if that helps.

Also, one of the recommended settings for 6.0 DMP to work with RHEL is to set a particular parameter called fast_io_fail_tmo to 20, do u have this file populated in server ?

/etc/udev/rules.d/40-rport.rules

Refer

http://www.symantec.com/docs/TECH47728

G

PS: If you are happy with the answer provided, please mark the post as solution. You can do so by clicking link "Mark as Solution" below the answer provided.
 

sverrewl's picture

Hm. You might be on to something here. The Array is already in A/A-A mode. However, we haven't made any tuning outside of this. So, no, we haven't created the 40-rport.rules yet. But, I found this file :

# rpm -qf /etc/udev/rules.d/40-VxVM.rules
file /etc/udev/rules.d/40-VxVM.rules is not owned by any package

... hmm - mystic - not installed by RPM ??

# cat 40-VxVM.rules
ACTION=="remove|add", KERNEL=="[sv]d*[!0-9]", SUBSYSTEM=="block", RUN+="/lib/udev/vxvm-udev.sh $env{ACTION} $env{MAJOR}:$env{MINOR} %k"

KERNEL=="[sv]d*[!0-9]", SUBSYSTEM=="block", RUN+="/lib/udev/vxpath_links %k"
# Do not create nodes for VERITAS VxVM devices
KERNEL=="Vx*", OPTIONS="ignore_device"
KERNEL=="dmpconfig", OPTIONS="ignore_device"

# Raw device permissions and ownership
KERNEL=="raw[0-9]*", RUN+="/etc/vx/bin/vxdmpraw initiate %k"

I guess a reboot will be required to implement the solution you mention ?

Gaurav Sangamnerkar's picture

If the array is already in A/A-A mode I would expect to reflect the same in # vxdmpadm listenclosure output however it is detecting as ALUA

To take the effect of file I mentioned above, yes ,worth to reboot once & check

G

PS: If you are happy with the answer provided, please mark the post as solution. You can do so by clicking link "Mark as Solution" below the answer provided.
 

sverrewl's picture

Currently we have it set to All Ports All LUNs mode.
Do you mean that it should be set to 4- port Active-Active mode ?

4- port Active-Active mode (4 ports active) - When this mode is selected volumes will be presented on both host ports, a controller failure will result in the volumes being presented on one of the ports on the other controller. This mode will present two 'paths' to the configured volumes, your host must have software to cope with a path failure for this mode to be used correctly. 

All Ports All LUNs mode (4 ports active) - When this mode is selected volumes can be mapped on all host ports on both controllers. A controller failure will result in volumes mapped exclusively to ports on that controller becoming inaccessible. This mode can present up to four 'paths' to the configured volumes; your host must have software to cope with a path failure for this mode to be used correctly.

Gaurav Sangamnerkar's picture

Hi,,

I would suggest to have a try with 4-port Active-Active mode, then run a "vxdctl enable" on the host, want to see if DMP detects it as "A/A-A" mode

G

PS: If you are happy with the answer provided, please mark the post as solution. You can do so by clicking link "Mark as Solution" below the answer provided.
 

Hari_G's picture

We have similar issue with 6.0.300 running on RHEL6.4 and apparently its a known issue

What we got back from support:

"These messages are a known issue and here is what they are caused by. To maintain I/O statistics, the dmp_daemon requests 1 MB continuous memory paging, which seems to be too large. The same cannot be granted on heavy I/O load systems and thus these messages get reported in the message log during high I/O loads."

"There is an enhancement request for these messages, but it has not been fixed yet as it is a major code change.

Here is the workaround that engineering has recommneded.

This fix is not so critical, as well as we have provided short term workaround of stopping iostats which is accepted by customer.We don’t have roadmap or exact ETA/ VxVM version in which fix for this incident can be included.

While re-starting iostats, we can set maximum amount of memory which iostat should use. I think this will help customer to better manage per CPU memory.

For ex:

set memory limit (per cpu) by 256k or even smaller, by using vxdmpadm command $ vxdmpadm iostat stop $ vxdmpadm iostat start memory=262144"

al_from_indiana's picture

Set it /etc/rc.local as well so that upon bootup the same parameters are set persistently

Morten Seeberg's picture

I believe that it will require iostat to be stopped before changing? I put this in rc.local:

# Storage Foundation VxDMP "memory allocation failed for size" workaround
vxdmpadm iostat stop
vxdmpadm iostat start memory=262144

Did you restore something today?