df command hangs and VCS Mount agent monitor times out, with many VxFS kernel threads stuck in vx_do_vget()

Article:TECH67862  |  Created: 2009-01-13  |  Updated: 2010-01-13  |  Article URL http://www.symantec.com/docs/TECH67862
Article Type
Technical Solution

Product(s)

Environment

Issue



df command hangs and VCS Mount agent monitor times out, with many VxFS kernel threads stuck in vx_do_vget()

Solution



The problem is seen with accessing VxFS filesystem over NFS on Solaris platform.  The problem may exhibit as poor performance or hangs momentarily of VxFS commands.   The problem is caused by an inefficient handling of a VxFS kernel data structure.
The problem can be verified by capturing the kernel threads during the 'VxFS command hang (e.g. df). This should be captured several times at a few mins apart, so that any hung threads can be captured.
   #echo "threadlist -v" | mdb -k
The stacks of the kernel threads may look like the following.
stack pointer for thread 2a1016b1ca0 (TS_FREE): 2a1016b1121
 000002a1016b11e1 cbe_level1+8
 000002a1016b1291 intr_thread+0x168
 000002a102d565e1 default_lock_delay+0x6c
 000002a102d56691 mutex_vector_enter+0x428
 000002a102d56741 rwst_enter_common+0x298
 000002a102d567f1 vx_do_vget+0xc
 000002a102d568d1 nfs3_fhtovp+0x7c

stack pointer for thread 2a101781ca0 (TS_FREE): 2a101780f81
 000002a101781021 xt_one_unchecked+0xd4
 000002a101781121 cyclic_softint+0xac
 000002a1017811e1 cbe_level1+8
 000002a101781291 intr_thread+0x168
 000002a104d145e1 0
 000002a104d14691 mutex_vector_enter+0x428
 000002a104d14741 rwst_enter_common+0x298
 000002a104d147f1 vx_do_vget+0xc
 000002a104d148d1 nfs3_fhtovp+0x7c
 000002a104d14991 rfs3_getattr+8
 000002a104d14ad1 common_dispatch+0x564
 000002a104d14df1 svc_getreq+0x1c8
 000002a104d14f41 svc_run+0x198
 000002a104d14ff1 nfssys+0x1c8
 000002a104d152e1 syscall_trap32+0xcc

This problem is tracked through the following Etrack number.
1293928 (Parent Etrack) - vxfs performance problem

Fixes are provided through the following child Etrack numbers for the corresponding VxFS patch levels.
1322195 - upcoming 5.0MP1 Rolling Patch (The existing RP4 doesn't have the fix yet)
1322196 - 5.0MP1 RP4 HF3 or later
1414174 - upcoming 5.0 Maintenance Patch (The existing MP3 doesn't have the fix yet)
1414175 - 5.0MP3 RP1 or later



Supplemental Materials

SourceETrack
Value1414175
Descriptionvxfs performance problem , remove vfs_lock_wait in vx_vget platform: solaris 10.


Legacy ID



319499


Article URL http://www.symantec.com/docs/TECH67862


Terms of use for this information are found in Legal Notices