TCP Chimney, TCPIP Offload Engine (TOE) or TCP Segmentation Offload (TSO) will cause a transport-level error to be logged resulting in inaccurate hit counts for Accelerator searches

Article:TECH55653  |  Created: 2007-01-20  |  Updated: 2014-06-17  |  Article URL http://www.symantec.com/docs/TECH55653
Article Type
Technical Solution

Product(s)

Issue



TCP Chimney, TCPIP Offload Engine (TOE) or TCP Segmentation Offload (TSO) will cause a transport-level error to be logged resulting in inaccurate hit counts for Accelerator searches


Error



A transport-level error has occurred when sending the request to the server (provider: TCP Provider, error 0 - An existing connection was forcibly closed by the remote host.)


Cause



This issue can occur when either TCP Chimney Offload, TCP/IP Offload Engine (TOE) or TCP Segmentation Offload (TSO) are enabled.

TCP Chimney, TCPIP Offload Engine (TOE) and TCP Segmentation Offload (TSO) off loads the TCP protocol stack to a Network Interface Card (NIC).    
 

  • TCP Chimney is Microsoft's software enhancement.
  • TOE is the NIC manufacturer's hardware enhancement.
  • TSO is the equivalent to TOE for some virtual environment configurations.

The TCP Chimney Offload feature is enabled by default in the Windows Server 2003 Scalable Networking Pack.
This update is included in Windows Server 2003 Service Pack 2 and can also be installed on a server running Windows 2003 Service Pack 1.  

The following event errors may also be seen:

Event 63 - Error adding installing/upgrading a Customer. System.Data.SqlClient.SqlException: A transport-level error has occurred when sending the request to the server.

Event 99 - An error occured deleting folders. System.Data.SqlClient.SqlException: Cannot open database "EVTest" requested by the login. The login failed.

Event 116 - An error occurred while The Background SQL tasks. System.Data.SqlClient.SqlException: Cannot open database "EVTest" requested by the login. The login failed.

Event 117 - Error adding Background SQL to the queue. System.Data.SqlClient.SqlException: Cannot open database "EVTest" requested by the login. The login failed.

Event 133 - An error has occured when initializing the upgraded/installed Customers. System.Data.SqlClient.SqlException: A transport-level error has occurred when sending the request to the server.

Event 141 - An error occured in ProcessingQueue::QueueingThread. System.Data.SqlClient.SqlException: Cannot open database "EVTest" requested by the login. The login failed.

Event 148 - GetServerRegistered: Error Getting Server information from the configuration database. System.Data.SqlClient.SqlException: Cannot open database "EVConfiguration" requested by the login. The login failed.

Event 149 - An error has occured when initializing the Servers. System.Data.SqlClient.SqlException: Cannot open database "EVConfiguration" requested by the login. The login failed.

Event 160 - No delegate! System.Data.SqlClient.SqlException: Cannot open database "EVTest" requested by the login. The login failed.

Event 186 - Error starting the AS processes. System.NullReferenceException: Object reference not set to an instance of an object.

Event 187 - Error starting the ATM processes. System.NullReferenceException: Object reference not set to an instance of an object.

Event 223 - An error occurred. System.Data.SqlClient.SqlException: A transport-level error has occurred when sending the request to the server.

Event 223 -  An error occurred. System.Data.SqlClient.SqlException: Cannot open database "EVTest" requested by the login. The login failed.

Event 284 - An Error has occured when getting the search results from the database. KVS.Accelerator.Common.Exceptions.SystemBusy: The system is busy.

Event 285 - An Error has occurred while bulk inserting search items

Event 287 - Error accepting a Search Accepted Thread - worked thread (0)

Event 288 - Error adding abandoning a Search. System.Data.SqlClient.SqlException: Cannot open database "EVTest" requested by the login. The login failed.

Event 289 - Error adding accepting items to queue. System.Data.SqlClient.SqlException: Cannot open database "EVTest" requested by the login. The login failed.

Event 291 - Error adding sampling a Search. System.Data.SqlClient.SqlException: Cannot open database "EVTest" requested by the login. The login failed.

Event 292 - Error adding search results to queue: Unrecoverable error processing Search results

Event 293 - Error adding Vault to Search in the queue. System.Data.SqlClient.SqlException: Cannot open database "EVTest" requested by the login. The login failed.

Event 296 - Error processing bulk search items: Unrecoverable error bulk inserting search items

Event 308 - An Error has occurred accepting a Search

Event 312 - An error has occured when initializing an Search queue. System.Data.SqlClient.SqlException: Cannot open database "EVTest" requested by the login. The login failed.

Event 323 - An error has occured when initializing the Search Vaults. System.Data.SqlClient.SqlException: Cannot open database "EVTest" requested by the login. The login failed.

Event 328 - An error occurred executing usp_SearchResults_Add

Event 341 - Error inserting search results

Event 345 - Error saving a search result

Event 390 - Error adding Pausing a Search. System.Data.SqlClient.SqlException: Cannot open database "EVTest" requested by the login. The login failed.
 

Solution



Implement the following actions and the below Workaround to better ensure resolution of the issue: 
  • Obtain the latest basic input/output system (BIOS) update for the server.
  • Obtain the latest firmware update for the network adapter.
  • Obtain the latest driver update for the network adapter.

WORKAROUND:
The following changes are to be made on the Enterprise Vault, Accelerator and SQL servers.
 
  • Disable TCP Chimney Offload feature

Windows 2003 Server:

 

If the operating system is Microsoft Windows Server 2003, perform the following steps:
 
Warning: Incorrect use of the Windows registry editor may prevent the operating system from functioning properly. Great care should be taken when making changes to a Windows registry.  Registry modifications should only be carried-out by persons experienced in the use of the registry editor application. It is recommended that a complete backup of the registry and workstation be made prior to making any registry changes.
 
1. Click Start, click Run, type Regedit, and then click OK.
 
2. Locate the following registry subkey:
 
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters
 
*If any of the below keys are not present, create them.
 
3. Double-click the EnableTCPChimney registry entry.
 
4. In the Edit DWORD Value dialog box, type 0 in the Value data box, and then click OK.
 
5. Double-click the EnableRSS registry entry.
 
6. In the Edit DWORD Value dialog box, type 0 in the Value data box, and then click OK.
 
7. Double-click the EnableTCPA registry entry.
 
8. In the Edit DWORD Value dialog box, type 0 in the Value data box, and then click OK.
 
9. Restart the server.
 
 
Windows 2008 Server:
If the operating system is Microsoft Windows Server 2008 (any version including R2), run the following from a Command prompt:
1. netsh int tcp set global chimney=disabled
 
2. netsh int tcp set global rss=disabled
 
3. netsh int tcp set global netdma=disabled
 
Note: To display current global TCP settings, use the net shell command:
netsh int tcp show global
 
4. Restart the server.
 
Note: Microsoft has identified an issue running the netsh command to set global TCP parameters on Windows Server 2008 and Vista machines.  Some global parameters, such as TCPTimedWaitDelay, can be changed from their default or manually set values to 0xffffffff.  Before running the above command, Symantec recommends reviewing Microsoft KB Article 967224 (support.microsoft.com/kb/967224).  Upon completion of the above command's execution, Symantec also recommends reviewing the TCP Parameters noted in the KB Article and applying the hotfix from the article if needed.

 

  • Disable TOE and Receive Side Scaling
Caution: The instructions in the following two sets of steps will cause the Network Interface Card (NIC) being changed to reset itself.  Such a reset will cause a momentary loss of communications through that NIC.  If the NIC is the only NIC on the system, or is the primary NIC through which remote connections are established, Symantec recommends scheduling a down time to implement these steps from the machine's console.  Symantec also recommends restarting any Journal Task on any Enterprise Vault Journal Archiving server where the Compliance Accelerator Journal Connector (JC) is installed, and restarting the Enterprise Vault Accelerator Manager Service (EVAMS) on any Compliance Accelerator (CA) or Discovery Accelerator (DA) server after these steps are implemented on a CA or DA server, or on the SQL Server hosting the CA or DA databases.  Such momentary loss of communications through the NIC can cause Event ID 158 errors to be thrown into the Symantec Enterprise Vault Event Log every 10 minutes on the EV, CA or DA servers.
 
1.  From the Control Panel, open Network Connections.
 
2.  Open the properties page for the network connection being used.
 
3.  Click Configure (See Figure 1)
 
Figure 1
 
 
 
4.  On the Advanced tab, change the Value of all 'Offload' and Receive Side Scaling settings to 0Disabled or Off (See Figure 2). This includes all entries for IPV4 and/or IPV6.
 
Figure 2
 
 
 
5.  Click OK to save changes.
 
 
  • Disable TSO
1.  From the Control Panel, open Network Connections.
 
2.  Open the properties page for the network connection being used.
 
3.  Click Configure (See Figure 3)
 
Figure 3
 
  
  
4.  On the Advanced tab, change the Value of TSOEnabled to 0. (See Figure 4)
 
Figure 4
 
 
 
5.  Click OK to save changes.
 

 
 
For more information see the following Microsoft knowledge base article
 
For More information for these setting in Windows 2008 see the following Microsoft knowledge base article

http://support.microsoft.com/kb/951037


Supplemental Materials

SourceEvent ID
Value287
Description

Error accepting a Search Accepted Thread - worked thread (0)


SourceEvent ID
Value308
Description

An Error has occured accepting a Search


SourceEvent ID
Value285
Description

An Error has occured while bulk inserting search items


SourceEvent ID
Value296
Description

Error processing bulk search items: Unrecoverable error bulk inserting search items


SourceEvent ID
Value341
Description

Error inserting search results


SourceEvent ID
Value345
Description

Error saving a search result


SourceEvent ID
Value328
Description

An error occurred executing usp_SearchResults_Add



Legacy ID



294308


Article URL http://www.symantec.com/docs/TECH55653


Terms of use for this information are found in Legal Notices