Recommended steps to optimize performance on Enterprise Vault (EV), Compliance Accelerator (CA), Discovery Accelerator (DA), and SQL Servers in an EV environment

Article:TECH56172  |  Created: 2007-01-14  |  Updated: 2014-07-09  |  Article URL http://www.symantec.com/docs/TECH56172
Article Type
Technical Solution

Product(s)

Issue



Compliance Accelerator (CA) and / or Discovery Accelerator (DA) search and / or export performance is slower than expected.


Environment



Any supported combination of Enterprise Vault, Compliance Accelerator, Discovery Accelerator and Microsoft SQL Server installed in a production or lab environment.


Cause



The Enterprise Vault (EV), Compliance Accelerator (CA) and Discovery Accelerator (DA) installations are configured by default to settings that have been found to be optimized in what is considered to be common environments.  There are instances where the default optimization settings are not adequate and issues arise with CA and DA searching and exporting.  Some adjustments to the EV, CA/DA and SQL servers may be necessary.


Solution



The following settings have been found to be good starting values to help adjust the EV, CA/DA and SQL servers to better optimize performance in individual environments.

 

Warning: Incorrect use of the Windows registry editor may prevent the operating system from functioning properly.  Great care should be taken when making changes to a Windows registry.  Registry modifications should only be carried-out by persons experienced in the use of the registry editor application.  It is also recommended that a complete backup of the registry and workstation / server be made prior to making any registry changes.

1. Optimize Enterprise Vault (EV) 32-bit Indexing (not for Enterprise Vault 10 or greater)

Add the following registry Key and values to the EV Indexing server(s), then restart the EV Indexing Service to become effective.  These values are not needed for Enterprise Vault 10.0.

32-bit Operating System Registry key:  
HKLM\Software\KVS\Enterprise Vault\Indexing
64-bit Operating System Registry key:  HKLM\Software\Wow6432Node\Enterprise Vault\Indexing
 

·               REG_DWord: AVSMaxLoc  Value: 500000000 (Decimal) (500 Million)

*** (Requires rebuild of Indexes to apply to all existing indexes - effective immediately on new indexes)

 

·               REG_DWord: SchemaType  Value: 1

*** (Requires rebuild of existing Indexes - effective immediately on new indexes)

***IMPORTANT - Rebuilding some indexes (i.e., a journal archive index with lots of index volumes) can have a significant impact on a production environment.  As such, these settings may be implemented at any time to affect current and new Index Volumes.  Any index or index volume rebuild should be delayed until the time frame and impact on the production environment has been studied and sufficient time has been allotted.


2. Disable Opportunistic Locking on the EV, CA/DA and SQL servers (for 32-bit and 64-bit Windows Server operating systems)

HKLM\System\CurrentControlSet\Services\MRXSmb\Parameters\  
 

·               REG_DWord: OplocksDisabled  Value: 1


HKLM\System\CurrentControlSet\Services\LanmanServer\Parameters\  
 

·               REG_DWord: EnableOplocks  Value: 0



3. Set memory management registry keys on the EV, CA/DA and SQL servers (for 32-bit Windows Server operating systems only)

HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\Session Manager\Memory Management\
 

·               REG_DWord: PoolUsageMaximum  Value: decimal 60  (Default is 80 and can be set as low as 20 - this represents % used before memory management takes control)

·               REG_DWord: PagedPoolSize  Value 0xFFFFFFFF



4. Set paging file on all EV, CA/DA and SQL servers (for 32-bit and 64-bit Windows Server operating systems)

My Computer | Properties | Advanced | Performance | Advanced | Virtual Memory initial and maximum sizes to an appropriate value (recommended in 4092 MB increments where possible) with multiple paging files across multiple drives as needed to get the total paging file size to approximately 2X the amount of physical memory on the server for 32-bit or 64-bit Windows Server, or set paging file to System managed size for 64-bit Windows Server.  Move the paging file off of the system drive (i.e., C:\-Drive) whenever possible for better system performance.  The system drive, by default, hosts the Windows Registry and the paging file used for memory crash dumps and to swap stale pages out of active memory.  As both of these are heavily used by normal Windows operations, moving the paging file used to swap stale pages out of memory off of the system drive will allow for improved disk I/O, presuming the drive on which the paging file was moved is a physically separate drive from the system drive.

 

Note: Microsoft Knowledge Base Article 2860880 discusses how to determine the appropriate size for a page file on 64-bit Windows.  If the Windows server has sufficient physical memory to process all memory requests without having to swap stale pages to disk, there is no need to have a page file other than under certain conditions.  Symantec acknowledges the possibility of no need for a page file provided that performance monitoring under heavy loads indicates physical memory remains available to be used (i.e., peak memory used is always less than physical memory).  Careful monitoring under multiple periods of heavy loads should be performed before deciding no page file or a reduced page file from the above recommendation is needed.


5. Place the Vault Service Account (VSA) Temporary folders on a physical drive with at least 40 gig of available free space (80 GB free space preferred for large exports or export with large sized messages) and off of the system drive (i.e., C:\-Drive) where possible (for 32-bit and 64-bit Windows Server operating systems)
 
A. Log onto the EV server as the Vault Service Account (VSA)
B. Stop all EV services on the server being changed
C. Navigate to
My Computer | Properties | Advanced | Environmental Variables | User variables for VSA
D. Click the line for the
TEMP variable
E. Click the Edit button
F. Input new folder path location in the Variable value: field (for example:
D:\Temp)
G. Repeat Steps D through F for the
TMP variable
 

Note: using the same destination folder for both the TEMP and TMP folders is permitted and recommended
 

H. Click the OK button to save the changes
I. Restart the server to clear the memory of any cached instances of the old locations and fully implement the new locations

 

Note: The VSA TEMP folder is used heavily by normal EV processes.  The system drive is also used heavily by the Windows Registry.  Moving the VSA TEMP folder off of the system drive allows for improved disk I/O on both drives (presuming both drives are physically separate devices) as the need to access the registry and VSA TEMP folder would be spread across multiple drives.


6. Optimize TCP/IP communications (for 32-bit and 64-bit Windows Server operating systems) 

 

1) Disable any Network Teaming on the Enterprise Vault (EV) servers running the EV Indexing Service and / or EV Storage Service, plus on the Compliance Accelerator (CA) and Discovery Accelerator (DA) server.

 

2) Disable TCP Chimney at the operating system level and all TCP Offload / Receive Side Scaling / TSOEnabled settings on all Network Interface Cards on the EV, CA, DA and SQL Servers per Technical Article TECH55653 (see the Related Articles section below).

 

3) Optimize the TCP Parameters:

 

Windows 2003 Server:


A.  On each of the Accelerator, Enterprise Vault and SQL server(s), locate the following key in the Windows registry:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters

B.  Update the following values, or create them if they do not already exist:
Name                                 Type                Default     Recommended (decimal)
MaxUserPort                      DWORD        5,000              64,512
TCPTimedWaitDelay        DWORD        240                 120
MaxFreeTcbs                     DWORD        2,000              65,536
MaxHashTableSize           DWORD        512                 16,384

EnableTCPChimney         DWORD                                0

EnableRSS                        DWORD                                0

EnableTCPA                      DWORD                                0

C.  For changes to take effect, restart the server.
 

 

Windows 2008 Server:
 

A.  On each of the Accelerator, Enterprise Vault and SQL server(s), locate the following key in the Windows registry:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters

B.  Update the following values, or create them if they do not already exist:
Name                                  Type               Default     Recommended (decimal)
TCPTimedWaitDelay        DWORD        240                 120

C.  To display the current ports for the TCP protocol use the netsh command:

netsh int ipv4 show dynamicport tcp

 

To increase the ports use the net shell command:

netsh int ipv4 set dynamicport tcp start=1500 num=63000
This will provide 63,000 TCP user ports.

 

D.  To display current global TCP settings , use the net shell command:

netsh int tcp show global

 

To disable Chimney Offload (TCP Chimney), use the net shell command:

netsh int tcp set global chimney=disabled

 

To disable Receive-Side Scaling (RSS), use the net shell command:

netsh int tcp set global rss=disabled

 

To disable NetDMA (TCPA), use the net shell command:

netsh int tcp set global netdma=disabled

 

 Alert: Microsoft has identified an issue running the netsh command to set global TCP parameters on Windows Server 2008 and Vista machines.  Some global parameters, such as TCPTimedWaitDelay, can be changed from their default or manually set values to 0xffffffff.  Before running the netsh command to change any global TCP parameter, Symantec recommends reviewing Microsoft KB Article 967224 (support.microsoft.com/kb/967224).  Upon completion of any netsh command to change any global TCP parameter, Symantec also recommends reviewing the TCP parameters noted in the KB article and applying the hotfix from the article if needed.

 

E.  For changes to take effect, restart the server.

 

 

NOTE: 
•When increasing the MaxUserPort, Microsoft recommends that port 1434 be reserved for use by the SQL Server Browser service (sqlbrowser.exe).

•On a Windows 2003 server, add the following registry key
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters
Add a Multi-string Value key.
Rename the new key to ReservedPorts
Add the values: 1434-1434

•On a Windows 2008 server, start the port allocation at 1500 as shown in the previous Windows 2008 netsh command.



7. CA and DA Tuning
I. Versions 6.X, 7.X and 2007.X

A. Open the CA/DA home page (for example,
http://EVDAServer/EVBADiscovery)
B. Navigate to the System Configuration pages listed below

Application Administration | System Configuration | Settings for:

System
 

·               Set a date for Enterprise Vault Oldest Archived Item Date to any date preferred (for example, January 1, 2001) - Does not require a restart of any process, service or task to become effective.

·               Set a date for Enterprise Vault V5 Installation Date to the date of Enterprise Vault 5.0 installation, or any preferred date if EV5 was never installed in the environment (for example, January 1, 2001) - Does not require a restart of any process, service or task to become effective.


Search
 

·               Set Total Number of Search Result Worker Threads to 2 - Requires a restart of the Customer's Background Task or the Enterprise Vault Accelerator Manager Service to become effective.

·               Enable Optimise searches based on oldest and youngest items by placing a check mark in the check box in the middle column of this option - Requires a restart of the Remoting process, performed by restarting the Enterprise Vault Accelerator Manager Service, to become effective.

·               Set Maximum Number of consecutive searches on same index to 1 - This setting is hidden: to view hidden settings, hold down the Ctrl button and select the white System Configuration words in the upper gray bar.  Requires a restart of the Customer's Background Task or the Enterprise Vault Accelerator Manager Service to become effective.

·               In DA, Enable Use sequence numbers for searches.


Legal Hold (DA only)
 

·               Set Placing hold items Batch Size to 1,000 - Requires a restart of the Customer's Background Task or the Enterprise Vault Accelerator Manager Service to become effective.

·               Set Removing hold items Batch Size to 1,000 - Requires a restart of the Customer's Background Task or the Enterprise Vault Accelerator Manager Service to become effective.


II. Version 8.X, 9.X and 10.X

A. Open the CA/DA Client
B. Navigate to the System Configuration pages listed below
 

1. Click on the Configuration button in the top toolbar
 

2. Click on the Settings button in the second toolbar
 


Section Name: System
 

·               Set a date for Enterprise Vault Oldest Archived Item Date to any date preferred (for example, January 1, 2001) - Does not require a restart of any process, service or task to become effective.

·               Set a date for Enterprise Vault V5 Installation Date to the date of Enterprise Vault 5.0 installation, or any preferred date if EV5 was never installed in the environment (for example, January 1, 2001) - Does not require a restart of any process, service or task to become effective.


Section Name: Search
 

·               Set Total Number of Search Result Worker Threads to 2  - Requires a restart of the Customer's Background Task or the Enterprise Vault Accelerator Manager Service to become effective.

·               Enable Optimise searches based on oldest and youngest items by placing a check mark in the check box in the middle column of this option - Requires a restart of the Remoting process, performed by restarting the Enterprise Vault Accelerator Manager Service, to become effective.

·               Set Maximum Number of consecutive searches on same index to 1 - This setting is hidden: to view hidden settings, hold down the Ctrl button and select the Configuration Settings words in the upper blue bar.  Requires a restart of the Customer's Background Task or the Enterprise Vault Accelerator Manager Service to become effective.  Note that this value change is only needed when searching against 32-bit indexes when using CA or DA 10.x.

 

If searching large index volumes tend to cause indexe process to fail or timeout frequently even though there is nothing wrong with indexes, it is also recommended to have the following settings set to the value below:

·               Set Number of Vault Search threads from 10 (default) to 4.

·               Set Search Result Chunk Size (hidden setting) from 50000 (default) to 10000. 

 

DA only Section Name: Legal Holds
 

·               Set Placing hold items Batch Size to 1,000 - Requires a restart of the Customer's Background Task or the Enterprise Vault Accelerator Manager Service to become effective.

·               Set Removing hold items Batch Size to 1,000 - Requires a restart of the Customer's Background Task or the Enterprise Vault Accelerator Manager Service to become effective.

 

 

Note: Microsoft has identified an issue running the netsh command to set global TCP parameters on Windows Server 2008 and Vista machines.  Some global parameters, such as TCPTimedWaitDelay, can be changed from their default or manually set values to 0xffffffff.  Before running the above command, Symantec recommends reviewing Microsoft KB Article 967224 (support.microsoft.com/kb/967224).  Upon completion of the above command's execution, Symantec also recommends reviewing the TCP Parameters noted in the KB Article and applying the hotfix from the article if needed. 

 

 




Legacy ID



294980


Article URL http://www.symantec.com/docs/TECH56172


Terms of use for this information are found in Legal Notices