System panic in vxfs code vx_qiostats_update()

Article:TECH196601  |  Created: 2012-09-12  |  Updated: 2012-10-13  |  Article URL
Article Type
Technical Solution




System panic with a redzone violation while releasing vxfs inodes File Input/Output (FIO) statistics structure.


Panic stack shows:

CPU 12 CSA F1000815B0337D00 at time of crash, error code for LEDs: 70000000
pvthread+06E800 STACK:
[0001BF00]abend_trap+000000 ()
[005098A4]slock+000524 (??, ??)
[00009558].simple_lock+000058 ()
[04AD7470].vx_aio_iodone+000200 ()
[00176F54]internal_iodone_offl+000174 (??, ??)
[00176DA0]iodone_offl+000080 ()
[0017A2B4]i_softmod+000294 ()
[001A5DE4]flih_util+000250 ()
[049E3280].vx_qiostats_update+000074 ()
[049E4310].vx_fiostats_update@AF48_34+000020 ()
[049E39E4].vx_fiostats_update+00001C ()
[04C93A70].fdd_rw+000790 ()
[04C91C54].fddstrategy1+0005D4 ()
[04C91E1C].fddstrategy+000100 ()
[04C9064C].fddstrategy_skey+000024 ()
[002D3BA4]std_devstrat+000364 (??)
[002D41D0]devstrat+000050 (??)
[00014F50].kernel_add_gate_cstack+000030 ()
[04D3C9C4]devstrat_lio@AF64_18+0000A4 (??)
[04D37574]devstrat_lio+000014 (??)
[04D3B124]kaio_rdwr+000184 (??, ??, ??, ??)
[00003850]ovlya_addr_sc_flih_main+000130 ()

56)> dr r3
r3 : EEEE0000DA278030            <== ESYSERROR_SL_RECURLOCK
(0)> dd vx_fiostats_tunable
vx_fiostats_tunable+000000: 0000000A          <== Note the value of "0000000A" when enabled


This issue is applicable to systems running:

-AIX, HP-UX, Linux and Solaris
-VxFS 5.1SP1 and above


This issue is tracked via etrack # 2726056.

Different types of statistics are maintained when a file is accessed in Quick Input/Output (QIO) and non-QIO mode. Some common statistics are copied when the file access mode is changed from QIO to non-QIO or vice versa. While switching from QIO mode to non-QIO, the QIO statistics structure is freed and FIO statistics structure is allocated to maintain FIO file-level statistics. There is a race between the thread freeing the QIO statistics which also allocates the FIO statistics and the thread updating the QIO statistics when the file is opened in QIO mode. Thus, the FIO statistics gets corrupted as another thread writes to it assuming that the QIO statistics is allocated.


Symantec has made code changes to protect the allocation/releasing of FIO/QIO statistics using the read-write lock/spin lock for file statistics structure. This change is available from 5.1SP1RP3 patch which is released in October 2012.

Please follow the patch README file for installation details.

Unit the patch is installed, please follow the following workaround to disable FIO stat gathering.

Workaround for AIX

On AIX the vx_fiostats_tunable has to be tuned above 100 to turn off fiostat gathering. Recommendation is this tunable to be tuned to 1000.
So to disable, update /etc/vx/vxfssystem  with the below line. If this file doesn't exist then please create one.

vx_fiostats_tunable 1000  

Please note a reboot is required for this change to take effect. 

Patch links for 5.1SP1RP3


AIX 7.1

Solaris SPARC

solaris x64

RHEL5 x86_64

RHEL6 x86_64

SLES10 x86_64

SLES11 x86_64

Supplemental Materials


System panic with redzone violation when vx_free() tried to free fiostat

Article URL

Terms of use for this information are found in Legal Notices