Video Screencast Help
Symantec to Separate Into Two Focused, Industry-Leading Technology Companies. Learn more.

DMP I/O error is disabling the NETAPP paths via DMP

Created: 12 Oct 2013 • Updated: 21 Nov 2013 | 4 comments
This issue has been solved. See solution.
For everty roo boot the VXVM volume is going to DEPORT state.
I am importing it manually.
I saw the below intresting logs on "dmpevents.log"
 
 
 
Thu Oct 10 16:53:56.317: Enabled Disk array other_disks
Thu Oct 10 16:53:56.317: Enabled Disk array fas32200
Thu Oct 10 16:53:56.318: Added Dmpnode c0t1d0s2
Thu Oct 10 16:53:56.318: Added Dmpnode fas32200_0
Thu Oct 10 16:53:56.318: Dmpnode c0t0d0s2 has migrated from enclosure - to other_disks
Thu Oct 10 16:53:56.318: Disabled Disk array -
Thu Oct 10 16:53:56.367: I/O error occured on Path c3t60A980004431702F643F44436B446B46d0s2 belonging to Dmpnode fas32200_0
Thu Oct 10 16:53:56.370: I/O analysis done as DMP_PATH_OKAY on Path c3t60A980004431702F643F44436B446B46d0s2 belonging to Dmpnode fas32200_0
Thu Oct 10 16:53:56.370: I/O retry(4) on Path c3t60A980004431702F643F44436B446B46d0s2 belonging to Dmpnode fas32200_0
Thu Oct 10 16:53:56.373: I/O error occured on Path c3t60A980004431702F643F44436B446B46d0s2 belonging to Dmpnode fas32200_0
Thu Oct 10 16:53:56.374: SCSI error occured on Path c3t60A980004431702F643F44436B446B46d0s2: opcode=0x12 reported generic device failure (status=0x0, key=0x0, asc=0x50, ascq=0x50)
Thu Oct 10 16:53:56.374: I/O analysis done as DMP_PATH_FAILURE on Path c3t60A980004431702F643F44436B446B46d0s2 belonging to Dmpnode fas32200_0
Thu Oct 10 16:53:56.374: Disabled Path c3t60A980004431702F643F44436B446B46d0s2 belonging to Dmpnode fas32200_0 due to path failure
Thu Oct 10 16:53:56.374: Disabled Dmpnode fas32200_0
Thu Oct 10 16:53:56.374: Disabled Disk array fas32200
Thu Oct 10 16:53:56.374: I/O error occured (errno=0x6) on Dmpnode fas32200_0
Thu Oct 10 16:53:59.558: Enabled Disk array fas32200
Thu Oct 10 16:53:59.558: Enabled Path c3t60A980004431702F643F44436B446B46d0s2 belonging to Dmpnode fas32200_0
Thu Oct 10 16:53:59.558: Enabled Dmpnode fas32200_0
Thu Oct 10 16:53:59.559: Failover initiated for Dmpnode fas32200_0 without quiescing
Thu Oct 10 16:53:59.559: CURPRI set to primary for Dmpnode fas32200_0 without quiescing
Thu Oct 10 16:54:04.000: Initiated SAN topology discovery
Thu Oct 10 16:54:05.000: Completed SAN topology discovery
Thu Oct 10 22:06:11.460: Marked as idle Path c3t60A980004431702F643F44436B446B46d0s2 belonging to Dmpnode fas32200_0
Thu Oct 10 22:07:22.012: Unmarked as idle Path c3t60A980004431702F643F44436B446B46d0s2 belonging to Dmpnode fas32200_0
Thu Oct 10 22:07:24.012: Lost 5120 DMP I/O statistics records
Thu Oct 10 22:07:25.012: Lost 2071 DMP I/O statistics records
Thu Oct 10 22:08:28.011: Marked as idle Path c3t60A980004431702F643F44436B446B46d0s2 belonging to Dmpnode fas32200_0
 
Operating Systems:

Comments 4 CommentsJump to latest comment

Gaurav Sangamnerkar's picture

Hello,

It is clear from the errors that DMP/VxVM is trying to make IO to the device however is failing intermittently ..

Questions:

1. What version of VxVM is in use ?

2. What Storage array is in use ?

3. attach the output of # iostat -En

4. paste output of # vxdmpadm listenclosure all

5. Any analysis done from Storage / SAN switch side to know if there are any IO errors noticed ? I would expect the SAN switch logs should show some errors.

6. Any other multipathing software underneath DMP ?  mpxio or powerpath ?

 

G

PS: If you are happy with the answer provided, please mark the post as solution. You can do so by clicking link "Mark as Solution" below the answer provided.
 

Ramesh.Kasireddy's picture

1. What version of VxVM is in use ?

             VERSION:  5.1,REV=10.06.2009.21.48

2. What Storage array is in use ?

             NETAPP

3. attach the output of # iostat -En

              c0t0d0           Soft Errors: 0 Hard Errors: 0 Transport Errors: 0

Vendor: ATA      Product: ST31000524NS     Revision: SN12 Serial No:
Size: 1000.20GB <1000204885504 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 49 Predictive Failure Analysis: 0
c0t1d0           Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: ATA      Product: ST31000524NS     Revision: SN12 Serial No:
Size: 1000.20GB <1000204885504 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 29 Predictive Failure Analysis: 0
c3t60A980004431702F643F44436B446B46d0 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: NETAPP   Product: LUN              Revision: 811a Serial No:
Size: 536.95GB <536952700416 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 1 Predictive Failure Analysis: 0
 

4 . paste output of # vxdmpadm listenclosure all

ENCLR_NAME        ENCLR_TYPE     ENCLR_SNO      STATUS       ARRAY_TYPE     LUN_COUNT

=======================================================================================
fas32200          FAS3220        700000265539         CONNECTED    ALUA        1
other_disks       OTHER_DISKS    OTHER_DISKS          CONNECTED    OTHER_DISKS   2
 

5. Any analysis done from Storage / SAN switch side to know if there are any IO errors noticed ? I would expect the SAN switch logs should show some errors.

        A) Contacted SAN admin, every thing look fine.

6 . Any other multipathing software underneath DMP ?  mpxio or powerpath ?

         A) mpxio

 

 

ursi's picture

hmm -- just my two cents:

 

This looks like a SCSI error while trying to write to the disk c3t60A980004431702F643F44436B446B46d0s2 (nice name). From the ASC it says it is an "Write Append Error" but I could not find the additional ASCQ Qualifier "50" for that error code.

If it is a SCSI error it seems unappropriate to disable the path since the path (to the broken (virtual) disk) is just fine.

 

AlexeyL's picture

It's possible that MPxIO and DMP interfere with each other. In my environment I tend to disable MPxIO for controllers/disks managed by DMP. Here is some info from Symantec about interoperability of MPxIO and DMP.

http://www.symantec.com/business/support/index?pag...

Also if you could provide the following:

1) mpathadm list lu

2) vxdmpadm getsubpaths

 

SOLUTION