Poor disk write performance
We have a server 2003 R2 machine, which primarily is running as our NetBackup Master server.
In addition to this we have a second machine which is/was setup as in a cluster using Veritas Storage Foundations VVR to replicate the data and VCS to handle the application side of things.
About three weeks ago we noticed that the disk write performance was very bad. Using a tool provided by NetBackup to write zero's to the file systems, we noted that it would start off fast, and then performance would drop significantly to the point where it was writing about 0Mb/s or 1Mb/s. This is present on both SAN volumes and local volumes. The same write profile is observed.
We also see disk queue lengths go up to 130, but no higher than this
This is the case on both of the servers running VCS/VSF, but is not present on any other servers running NBU but not running the storage foundation software.
There is no VxCache memory config enabled on the server or any disks.
All the disks are reporting as being misaligned.
Some time ago VVR was disabled/broken and has not been reinstated as we do not have Symantec support on the product, and noone here knows how to use it.
So I am reaching out to anyone who may be able to point me in the right direction. I have read the admin guide and it has given me a bit more understanding of the components in the software, but has not provided much in the way of troubleshooting this write performance problem.
Today I will run IOMeter to test reads/writes and a combination of both.
VSC/VSF version is 5.1 SP1