System panic at the panic string: BAD TRAP: type=31 rp=2a101da7610 addr=148 mmu_fsr=0 occurred in module "vxio" due to a NULL pointer dereference

Article:TECH145070  |  Created: 2010-11-26  |  Updated: 2012-07-28  |  Article URL http://www.symantec.com/docs/TECH145070
NOTE: If you are experiencing this particular known issue, we recommend that you Subscribe to receive email notification each time this article is updated. Subscribers will be the first to learn about any releases, status changes, workarounds or decisions made.
Article Type
Technical Solution

Environment

Issue



System went panic and left the panic strings ( BAD TRAP: type=31 rp=2a101da7610 addr=148 mmu_fsr=0 occurred in module "vxio" due to a NULL pointer dereference )


Error



1. SolarisCAT(./vmcore.1/10U)> panic
panic on CPU 17
panic string:   BAD TRAP: type=31 rp=2a100687610 addr=148 mmu_fsr=0 occurred in module "vxio" due to a NULL pointer dereference
==== panic kernel thread: 0x2a100687ca0  PID: 0  on CPU: 17  affinity CPU: 17 ====
cmd: sched
t_procp: 0x1837fc0(proc_sched)
  p_as: 0x1838ce0(kas)
  zone: global
t_stk: 0x2a100687a90  sp: 0x1845211  t_stkbase: 0x2a100682000
t_pri: 60(SYS)  pctcpu: 0.000000
t_lwp: 0x0  psrset: 0  last CPU: 17 
idle: 1 ticks (0.01 seconds)
start: Wed Nov 24 09:35:35 2010
age: 5071 seconds (1 hours 24 minutes 31 seconds)
tstate: TS_ONPROC - thread is being run on a processor
tflg:   T_TALLOCSTK - thread structure allocated from stk
        T_PANIC - thread initiated a system panic
tpflg:  none set
tsched: TS_LOAD - thread is in memory
        TS_DONT_SWAP - thread/LWP should not be swapped
        TS_SIGNALLED - thread was awakened by cv_signal()
pflag:  SSYS - system resident process
pc:      unix:panicsys+0x48:   call     unix:setjmp
startpc: vxio:voliod_loop+0x0:   sethi  %hi(0x0) %g4    ( set   0x0, %g4 )
unix:panicsys+0x48(0x109b0b0, 0x2a1006873b8, 0x1845be0, 0x1, , , 0x9900001604, , , , , , , , 0x109b0b0, 0x2a1006873b8)
unix:vpanic_common+0x78(0x109b0b0, 0x2a1006873b8, 0x9, 0x1, 0x8, 0x8)
unix:panic+0x1c(0x109b0b0, 0x31, 0x2a100687610, 0x148, 0x0, 0x30000559788, 0x181bef8)
unix:die+0x78(0x31, 0x2a100687610, 0x148, 0x0)
unix:trap+0x9e0(0x2a100687610, 0x148)
unix:ktl0+0x48()
-- trap data  type: 0x31 (data access MMU miss)  rp: 0x2a100687610  --
  addr: 0x148
pc:  0x7be9b0a4 vxio:vol_multistepsio_read_source+0x9c:   ldx   [%o1 + 0x148], %g5
npc: 0x7be9b0a8 vxio:vol_multistepsio_read_source+0xa0:   add   %o0, %g4, %g4
  global:                       %g1                0x1
        %g2                0x6  %g3                0x6
        %g4               0x80  %g5               0x4c
        %g6                  0  %g7      0x2a100687ca0
  out:  %o0        0x1325c6000  %o1                  0
        %o2      0x6006c3df900  %o3          0x264b8c0
        %o4                0x1  %o5      0x60076dfd580
        %sp      0x2a100686eb1  %o7         0x7be1b7f0
  loc:  %l0      0x6006c3df900  %l1                0x3 
        %l2      0x600648615a0  %l3          0x105d8e8 
        %l4                  0  %l5      0x60064861000 
        %l6                0x2  %l7              0x5a0 
  in:   %i0      0x6006cfd91c0  %i1      0x2a100687a38 
        %i2      0x6006cfd91c0  %i3      0x6006c3df9c8 
        %i4      0x2a100687a38  %i5      0x6006c3df900 
        %fp      0x2a100686fa1  %i7         0x7be9a224 
<trap>vxio:vol_multistepsio_read_source+0x9c(0x6006cfd91c0, 0x2a100687a38, , 0x6006c3df9c8, 0x2a100687a38, 0x6006c3df900)
vxio:vol_multistepsio_start+0x4f4(0x6006cfd91c0, 0x2a100687a38, , , , 0x3)
vxio:voliod_iohandle+0x44(0x6006cfd91c0, 0x1, 0x2a100687a38?, , , 0x30003316000)
vxio:voliod_loop+0x4f8(0x0, 0x0)
unix:thread_start+0x4()
-- end of kernel thread's stack --


2. SolarisCAT(./vmcore.1/10U)> msgbuf

..
<snip>
..
NOTICE: VxVM VVR vxio V-5-0-91 Connected from node 192.168.1.136 to node 192.168.1.66
NOTICE: VxVM VVR vxio V-5-0-91 Connected from node 192.168.1.137 to node 192.168.1.77
NOTICE: VxVM VVR vxio V-5-0-265 Rlink rlk_192.168.1.66_nfs_pri_rvg connected to remote
NOTICE: VxVM VVR vxio V-5-0-265 Rlink rlk_192.168.1.77_pgsql1_rvg connected to remote
NOTICE: VxVM VVR vxio V-5-0-91 Connected from node 192.168.1.133 to node 192.168.1.63
NOTICE: VxVM VVR vxio V-5-0-265 Rlink rlk_192.168.1.63_msg2_pri_rvg connected to remote
Notifying cluster that this node is panicking

panic[cpu23]/thread=2a101da7ca0: BAD TRAP: type=31 rp=2a101da7610 addr=148 mmu_fsr=0 occurred in module "vxio" due to a NULL pointer dereference

sched: trap type = 0x31
addr=0x148
pid=0, pc=0x7be9b0a4, sp=0x2a101da6eb1, tstate=0x880001600, context=0x0
g1-g7: 1, 8, 8, 80, 4c, 10, 2a101da7ca0

000002a101da7330 unix:die+78 (31, 2a101da7610, 148, 0, 2a101da73f0, 109b000)
  %l0-3: 0000000000001fff 0000000000000031 0000000001000000 0000000000002000
  %l4-7: 000000000181bef8 000000000181bc00 0000000000009fe8 00000000ca5a6148
000002a101da7410 unix:trap+9e0 (2a101da7610, 0, 1fff, 5, 0, 1)
  %l0-3: 0000000000000000 0000000001837fc0 0000000000000031 0000000000001c00
  %l4-7: 0000000000000000 0000000000000001 ffffffffffffe000 0000000000000005
000002a101da7560 unix:ktl0+48 (132556380, 0, 3003152d9c0, 264aac7, 1, 300222bf1a0)
  %l0-3: 0000000000000001 0000000000001400 0000000880001600 000000000101c030
  %l4-7: 0000000000000000 000003000053fda8 0000000000000000 000002a101da7610
000002a101da76b0 vxio:voldco_inactive+470 (6005b93de00, 2a101da7a38, 6005b93de00, 3003152da88, 2a101da7a38, 3003152d9c0)
  %l0-3: 000003003152d9c0 0000000000000003 00000600703125a0 000000000105d8e8
  %l4-7: 0000000000000000 0000060070312000 0000000000000002 00000000000005a0
000002a101da77a0 vxio:vol_multistepsio_start+4f4 (1, 0, 6005b93de00, 0, 0, 3)
  %l0-3: 000003003152d9c0 000002a101da7a38 0000000000000207 0000000000000003
  %l4-7: 000002a101db7ca0 0000000000000002 0000000001814d20 0000000000000000
000002a101da7890 vxio:voliod_iohandle+44 (1931c40, 2a101da7a38, 2a101da7a38, 6005b93de00, 1, 3000339c000)
  %l0-3: 000000007be99d30 0000000000000006 0000000000001070 0000000000000000
  %l4-7: 000003000339c000 0000000000000107 0000000000649107 0000000000649106
000002a101da7940 vxio:voliod_loop+4f8 (6005b93de00, 1930be0, 1930800, 0, 0, 1930be0)
  %l0-3: 0000000001930968 0000000001924f50 0000000001927650 000002a10001fca0
  %l4-7: 0000000000000001 0000000000000000 0000000001931a30 0000000000000000

syncing file systems... [2] 1953 [2] 1917 [2] 1901 [2] 1901 [2] 1901 [2] 1901 [2] 1901 [2] 1901 [2] 1901 [2] 1901 [2] 1901 [2] 1901 [2] 1901 [2] 1901 [2] 1901 [2] 1901 [2] 1901 [2] 1901 [2] 1901 [2] 1901 [2] 1901 [2] 1901 [2] 1901WARNING: pcisch2: spurious interrupt from ino 0x2d
su-1#0
 done (not all i/o completed)
dumping to /dev/md/dsk/d1, offset 13744275456, content: kernel
WARNING: pcisch2: spurious interrupt from ino 0x2d
 


Environment



1. SunOS sun-system 5.10 Generic_141414-07 sun4u sparc SUNW,Sun-Fire-V890
2. SF4.1MP2(VVR)


Cause



System went panic in vxio:vol_multistepsio_read_source as a result of having invalid DCO objects. (vxio:voldco_inactive+470)


Solution



[ FINDINGS  AND  SUGGESTION ]
According to the known incident referred from the other case,  Panic in vxio:vol_multistepsio_read_source as a result of having invalid DCO objects. (vxio:voldco_inactive+470)

So therefore, please it is required to take prompt measure on the system by aboding by the following suggestion.

0. Summary
- Incident e1428238 ( I/O gets stuck in drl_logbusy queue )
- Resolution: According to the previous case, it was recommended to install either 41mp2rp5hf1 or 41mp2rp6 to resolve e1428238.
- So therefore, please take it into serious account to apply latest 41mp2rp5 hotfix.(or 41mp2rp6)


1. For more details, please refer to the fixed incidents in 4.1MP4RP6.
Date:2009-03-11
OS:Solaris Sparc
OS Version: Solaris 8,9,10
Etrack Incidents: 1246621 1545835 1589881 1594928 1589018 1596811 1634861 1149665 1459831 1590314 1797540 524014  1729558

Errors/Problems Fixed:
(e1246621)  In CVM env,when all the ports on switch are disabled from host(M)side and then enabled,cpp of dmpnode will remain set to NULL.
(e1545835)  vxconfigd core dump during system boot after VxVM4.1RP4 applied.
(e1589881)  Dump device was changed to none (dumps disabled) after encapsulating a boot disk.
(e1594928)  Avoid unnecessary retries on error buffers when disk partition is nullified.
(e1589018)  num_retries field is getting re-initialized to initial value leading to looping and delay in error handling time.
(e1596811)  Slowness re-enabling individual paths on vx4.1mp2rp1
(e1634861)  Potential hang in vxio after SAN issue
(e1149665)  Disabling failed paths gives wrong Error message
(e1459831)  VVR:Replication hangs due to deadlock on Secondary nmcom receive with TCP multiconnection.
(e1590314)  EMC PP/DMP interoperability defect - vxdmpadm getsubpaths dmpnodename=emcpower## details incorrect
(e1797540)  VxVM: vxdisk resize intermittently causes vxconfigd to dump core.
(e524014)    FMR: fast resync does not kick in after force import of dg
(e1729558)  multiple vxplex attach cmds running in parallel on a volume lead to clearing DCO map and subs equently lead to corruption in FMR2


2. To download the RP6, please follow the link
https://sort.symantec.com/patch/detail/2755/0/cGF0Y2gvc2VhcmNobWF0cml4LzMvMS80
 


Supplemental Materials

Description

 

 




Article URL http://www.symantec.com/docs/TECH145070


Terms of use for this information are found in Legal Notices