Slow performance when backing up remote servers with the TCP Offload Engine (TOE) Feature enabled on network cards and/or Windows 2003 SP2 Scalable Networking Pack (SNP) is enabled.

Article:TECH52727  |  Created: 2010-01-27  |  Updated: 2014-05-08  |  Article URL http://www.symantec.com/docs/TECH52727
Article Type
Technical Solution

Product(s)

Issue



Slow performance when backing up remote servers with the TCP Offload Engine (TOE) Feature enabled on network cards and/or Windows 2003 SP2 Scalable Networking Pack (SNP) is enabled.


Solution




Windows 2003 Service Pack 2 introduced a new update called SNP (Scalable Networking Pack) . This utilizes a feature in newer network cards called TCP/IP Offload Engine (TOE ) or TCP Chimney.  Some servers that have this feature enabled experience throughput or connectivity issues when used in conjunction with certain NICs (Network Interface Cards) between the media server and remote machines.  
 
TCP Offload Engine is an emerging technology which is designed to offload TCP stack handling from the main system CPU to a processor built into NIC cards.  This technology is still relatively new, and when engaged, has been known to cause unstable connections.  This results in dropped sockets, dropped packets, packet reordering, packet retransmits and ultimately slow or failed backups.
 
There are two types of TCP offloads - stateful and stateless.
 
Stateful offload:
 
TCP Chimney is a stateful offload. TCP Chimney offload enables TCP/IP processing to be offloaded to network adapters that can handle the TCP/IP processing in hardware.
 
The Windows 2003 TCP Chimney setting and several Network Card (NIC) settings have been known to interfere with successful inter process communication and connectivity over TCP/IP networks.
 
Stateless offload
 
RSS and NetDMA are stateless offloads. Where multiple CPUs reside in a single computer, the Windows networking stack limits "receive" protocol processing to a single CPU. RSS resolves this issue by enabling the packets that are received from a network adapter to be balanced across multiple CPUs.
 
The following has been observed in cases when this issue occurs:
 <1> Backup of Local Media Server to B2D (Backup to Disk) runs at a normal throughput rate.
 <2> Backup of Local Media Server to tape runs at a normal throughput rate.
 <3> Backup of Remote Server (One or All) to B2D runs at a normal throughput rate in most instances, but runs slower in some network configurations.
 <4> Backup of Remote server (One or All) to tape is consistently slow or loses connection all together during a backup or restore.
 
Please note:  If this occurs only while backing up one remote server, TOE is most likely enabled on that Server's NIC.  If this is happening for all remote servers , then TOE Chimney is most likely enabled on the Media Server's NIC.  To determine if the TOE Feature is possibly causing the performance issue, disable the feature per the following and run additional remote backups to test:
 
1) Modify the network adapter (NIC) properties to disable Receive Side Scaling or the TOE Option.  Contact the Hardware Provider for specific instructions.
 
2) Temporarily disable Windows 2003 SP2 Scalable Networking Pack TCP Chimney Offload feature by issuing the following command at a Windows command prompt (Start, Run, CMD). No reboot is required
 
Netsh int ip set chimney DISABLED

To check the current setting, observe this registry key:
 
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters
 
EnableTCPChimney
 
Type: REG_DWORD
 
Values: 1 (enabled) 0 (disabled)
 
Note:  This is a temporary work-around to disable the TOE Feature and will not persist after a reboot.  Also, if the command completes successfully but no improvement is seen in the performance, then it most likely is not being caused by the TOE Feature.

3) Disable the Windows 2003 SP2 Scalable Networking Pack features (TCP Chimney Offload, Receive Side Scaling , Network Direct Memory Access) by applying the update per Microsoft KB 948496  
 
An update to turn off default SNP features is available for Windows Server 2003-based and Small Business Server 2003-based computers
 

 This issue is currently under investigation by Symantec Software. Pending the outcome of the investigation, this issue may be resolved by way of a patch or hotfix in current or future revisions of the software. However, this particular issue is not currently scheduled for any release. If you feel this issue has a direct business impact for you and your continued use of the product, please contact your Symantec Sales Representative or the Symantec Sales group to discuss these concerns. For information on how to contact Symantec Sales, please see:  www.symantec.com
 
For additional information on the TOE Feature, review the following:
 
Windows 2003 Scalable Networking pack and its possible effects on Exchange
 
The Microsoft Windows Server 2003 Scalable Networking Pack release
 
You cannot host TCP connections when Receive Side Scaling is enabled in Windows Server 2003 with Service Pack 2
 
TCP traffic stops after you enable both receive-side scaling and Internet Connection Sharing in Windows Vista or in Windows Server 2003 with Service Pack 1
 
 
How to enable or disable TCP Offload at the hardware/driver level:
 
In addition to what can be done through the Windows OS, individual network cards have attributes associated with their drivers which can toggle TCP offload settings from the main CPU to the processor on the NIC.
 

To list out which TCP offloads are enabled per network card, run this command:
 
Netsh int ip show offload
 

Here are examples of settings from 2 different NIC cards:
 
Example 1:Output from Netsh int ip show offload:
 
Offload Options for interface "Local Area Connection" with index: 10005:
 
TCP Transmit Checksum
 
IP Transmit Checksum
 
TCP Receive Checksum
 
IP Receive Checksum
 
TCP Large Send
 
In order to toggle these settings, go to Windows Device Manager > Network Adapters > (NIC) > Advanced tab
  
Example 1 Advanced tab settings:
 
 
 
Example 1 map :
 
TCP Transmit Checksum = Offload Transmit TCP Checksum
 
IP Transmit Checksum     = Offload Transmit IP Checksum
 
TCP Receive Checksum   = Offload Receive TCP Checksum
 
IP Receive Checksum      = Offload Receive IP Checksum
 
TCP Large Send              = Offload TCP Segmentation
 

 
Example 2 Output from Netsh int ip show offload:
 
Offload Options for interface "Local Area Connection" with index: 20003:
 
TCP Transmit Checksum
 
IP Transmit Checksum
 
TCP Receive Checksum
 
IP Receive Checksum
 
TCP Large Send
 
Example 2 Advanced tab settings:
 
 
 
Example 2 map :
 
TCP Transmit Checksum = Checksum Offload
 
IP Transmit Checksum     = Checksum Offload
 
TCP Receive Checksum   = Checksum Offload
 
IP Receive Checksum      = Checksum Offload
 
TCP Large Send              = Large Send Offload
 

 Symantec Support recommends setting all these values to Off/Disabled/None  (or whatever value means "off")
 
 
NOTE: As can be seen above, different NIC manufacturers and drivers may name these attributes differently.  Experimentation can be done to identify which settings map to which names by changing the value in the Attribute section of the NIC properties and then rerunning Netsh int ip show offload to see what changed.

 
NOTE: Modifying any of these settings will temporarily disconnect the NIC from the network.  Care should be taken when making this adjustment so as not to disrupt any critical activity.  Also, consider directly interfacing with the machine rather then using Remote Desktop Control.
 
 
NOTE: Modifying these settings for most NIC's does not require a reboot to engage the setting.  But if this is a great concern, check with your NIC vendor to see if a reboot is necessary, or experiment on a non-production machine before modifying a production machine.

 
How to observe the Offload State of a TCP connection:
 
The command netstat with the switch "-t"  can be used to show the Offload State of a network connection.
 
Example Output from the netstat -ab -t command:
 
Proto Local Address Foreign Address State      PID Offload State
 
TCP   server:epmap  server:0        LISTENING  816 InHost
 

 
Proto Local Address  Foreign Address  State       Offload State
 
TCP   Server:netbios name:2151        ESTABLISHED Offloaded
 

 
Flow Control Recommendations for NICs:
 
What is Flow Control?
 
"Flow Control" is a parameter that allows the user to enable or disable the receipt or transmission of PAUSE frames. PAUSE frames enable the adapter and the switch to control the transmit rate. The side that is receiving the PAUSE frame temporarily stops transmitting.
 

 
It is recommended to set Flow Control on the NIC to "Generate and Respond".  Flow Control can also be modified within the Network Card Properties > Advanced tab.  See Example 3.
 
  
 

Supplemental Materials

SourceETrack
Value1107687
Description

Slow Remote Agent Backups when the TCP Offload Engine (TOE) Feature is enabled on a network interface card



Legacy ID



290098


Article URL http://www.symantec.com/docs/TECH52727


Terms of use for this information are found in Legal Notices