System panic in VxFS with error message “redzone violation: write past end of buffer”

Article:TECH181263  |  Created: 2012-02-11  |  Updated: 2012-03-12  |  Article URL http://www.symantec.com/docs/TECH181263
Article Type
Technical Solution


Environment

Issue



System panic with redzone violation (write past end of buf) while freeing inodes fiostats structure with following error and stack trace, fiostats structure holds file & subfile I/O statistics.

 


Error



kernel memory allocator:                            
redzone violation: write past end of buffer        
buffer=<buffer_address> bufctl=0  cache: kmem_alloc_80  

unix:panicsys+0x48(0x1295b48, 0x2a1041d5078, 0x18b5150, 0x1, , , 0xe2001606, , ,, , , , , 0x1295b48, 0x2a1041d5078) unix:vpanic_common+0x78(0x1295b48, 0x2a1041d5078, 0x18dc400, 0x4e0af4c6620, 0x1,0x3000016a008) unix:panic+0x1c(0x1295b48, 0x30051eec46f, 0xf, 0xf, 0x1, 0x18dc400) genunix:kmem_error+0x4b4(0x1, 0x3000016a008, 0x4df06a6c658) genunix:kmem_cache_free_debug+0x8c(0x3000016a008, 0x4df06a6c658, 0x7a6dd380) genunix:kmem_cache_free+0xec(, 0x4df06a6c658) vxfs:vx_free(0x4df06a6c658, 0x48, 0x0) - frame recycled vxfs:vx_fiostats_alloc+0x48(0x54d9f7450a8, 0x1, 0x7021f, 0x0) fdd:fdd_common1+0xf8(, 0x4e0e059a440, 0x301839d1968, 0x4, 0x1) fdd:fdd_odm_open+0x1a4(0x2a1041d5628, 0x10000003, 0x301839d1968) odm:odm_vx_open+0x7c(0x2a1041d5628, 0x3, 0x301839d1968) odm:odm_ident_init+0x9c(0x4e0e059a440, 0x0, 0x3ff6a2, 0x0, 0x3, 0x0, 0x3c8b10a28c8, 0x2a1041d5780, 0x2a1041d5788, , 0x2a1041d5720, 0x64e6, , 0x1) odm:odm_identify+0x21c(0x3c7a47f2f80, 0x0) odm:odmioctl+0x2b4(0x4dfa3eeb840, 0x56584f1b, 0xffffffff7ffe9cd8, 0x202003,0x301839d1968) genunix:fop_ioctl+0x20(0x4dfa3eeb840, 0x56584f1b, 0xffffffff7ffe9cd8, 0x202003,, 0x2a1041d5adc) genunix:ioctl+0x184() unix:syscall_trap+0xac()

Environment



Solaris 10
Veritas Filesystem 5.1SP1RP2


Cause



Two types of File-level stats are maintained internally within VxFS, vx_qiostat is recorded during file access in Quick I/O (qio) mode and vx_fiostats during non-qio mode. Common stats are copied internally between vx_qiostat and vx_fiostat structure when file access mode is change between qio mode to non-qio or vice versa E.g while trying to copy an Oracle Data file using cp command will change the access mode for cp from qio to non-qio.

While switching from qio to non-qio mode, vx_qiostats structure is freed and vx_fiostats structure is allocated to maintain File-level stats. Due to a race condition between thread “A” freeing vx_qiostats which also allocate vx_fiostats to copy common values and another thread “B” updating vx_qiostats as file is opened in qio mode, the newly allocated vx_fiostats by thread “A” is corrupted as thread “B” writes to it assuming vx_qiostats is already allocated.


Solution



Temporary Workaround.

One of the following workaround can be used.
1- Do not use cp to copy files that are in use by oracle database.
2- Modify /etc/system to include “set vxfs:vx_fiostats_tunable=0” and reboot the system to avoid the race condition.

Permanent Fix.
A patch vxfs-5.1SP1RP2HF2 is created to resolve the race condition.

Please contact Symantec Technical Support using the method described in http://www.symantec.com/support/contact_techsupp_static.jsp to obtain this patch.
 


Supplemental Materials

SourceETrack
Value2709869


Article URL http://www.symantec.com/docs/TECH181263


Terms of use for this information are found in Legal Notices