System panic in VxFS with error message “redzone violation: write past end of buffer”
| Article:TECH181263 | | | Created: 2012-02-11 | | | Updated: 2012-03-12 | | | Article URL http://www.symantec.com/docs/TECH181263 |
Problem
System panic with redzone violation (write past end of buf) while freeing inodes fiostats structure with following error and stack trace, fiostats structure holds file & subfile I/O statistics.
Error
kernel memory allocator:
unix:panicsys+0x48(0x1295b48, 0x2a1041d5078, 0x18b5150, 0x1, , , 0xe2001606, , ,, , , , , 0x1295b48, 0x2a1041d5078) unix:vpanic_common+0x78(0x1295b48, 0x2a1041d5078, 0x18dc400, 0x4e0af4c6620, 0x1,0x3000016a008) unix:panic+0x1c(0x1295b48, 0x30051eec46f, 0xf, 0xf, 0x1, 0x18dc400) genunix:kmem_error+0x4b4(0x1, 0x3000016a008, 0x4df06a6c658) genunix:kmem_cache_free_debug+0x8c(0x3000016a008, 0x4df06a6c658, 0x7a6dd380) genunix:kmem_cache_free+0xec(, 0x4df06a6c658) vxfs:vx_free(0x4df06a6c658, 0x48, 0x0) - frame recycled vxfs:vx_fiostats_alloc+0x48(0x54d9f7450a8, 0x1, 0x7021f, 0x0) fdd:fdd_common1+0xf8(, 0x4e0e059a440, 0x301839d1968, 0x4, 0x1) fdd:fdd_odm_open+0x1a4(0x2a1041d5628, 0x10000003, 0x301839d1968) odm:odm_vx_open+0x7c(0x2a1041d5628, 0x3, 0x301839d1968) odm:odm_ident_init+0x9c(0x4e0e059a440, 0x0, 0x3ff6a2, 0x0, 0x3, 0x0, 0x3c8b10a28c8, 0x2a1041d5780, 0x2a1041d5788, , 0x2a1041d5720, 0x64e6, , 0x1) odm:odm_identify+0x21c(0x3c7a47f2f80, 0x0) odm:odmioctl+0x2b4(0x4dfa3eeb840, 0x56584f1b, 0xffffffff7ffe9cd8, 0x202003,0x301839d1968) genunix:fop_ioctl+0x20(0x4dfa3eeb840, 0x56584f1b, 0xffffffff7ffe9cd8, 0x202003,, 0x2a1041d5adc) genunix:ioctl+0x184() unix:syscall_trap+0xac()
redzone violation: write past end of buffer
buffer=<buffer_address> bufctl=0 cache: kmem_alloc_80
Environment
Solaris 10
Veritas Filesystem 5.1SP1RP2
Cause
Two types of File-level stats are maintained internally within VxFS, vx_qiostat is recorded during file access in Quick I/O (qio) mode and vx_fiostats during non-qio mode. Common stats are copied internally between vx_qiostat and vx_fiostat structure when file access mode is change between qio mode to non-qio or vice versa E.g while trying to copy an Oracle Data file using cp command will change the access mode for cp from qio to non-qio.
While switching from qio to non-qio mode, vx_qiostats structure is freed and vx_fiostats structure is allocated to maintain File-level stats. Due to a race condition between thread “A” freeing vx_qiostats which also allocate vx_fiostats to copy common values and another thread “B” updating vx_qiostats as file is opened in qio mode, the newly allocated vx_fiostats by thread “A” is corrupted as thread “B” writes to it assuming vx_qiostats is already allocated.
Solution
Temporary Workaround.
One of the following workaround can be used.
1- Do not use cp to copy files that are in use by oracle database.
2- Modify /etc/system to include “set vxfs:vx_fiostats_tunable=0” and reboot the system to avoid the race condition.
Permanent Fix.
A patch vxfs-5.1SP1RP2HF2 is created to resolve the race condition.
Please contact Symantec Technical Support using the method described in http://www.symantec.com/support/contact_techsupp_static.jsp to obtain this patch.
|
|
| Source | ETrack |
| Value | 2709869 |
Article URL http://www.symantec.com/docs/TECH181263
Terms of use for this information are found in Legal Notices
Thank you.