Stabilize the Network Interface Cards (NIC's)
1. From the Control Panel, open Network Connections.
2. Open the properties page for the network connection being used.
3. Click Configure
4. On the Driver tab, verify the date of the driver is at least no older than 2 years.  If it is, check for updates.
 

Verify NIC TCP/IP Offload Status

TOE and TSO Features Explained:
•TOE is the NIC manufacturer's hardware enhancement.
•TSO is VMWare’s software on vNIC hardware enhancement.
•TCP Offload Engine or TOE is a technology used in network interface cards (NIC) to offload the processing of the entire TCP/IP stack to the network controller. It is primarily used with high-speed network interfaces, such as gigabit Ethernet and 10 Gigabit Ethernet where processing overhead of the network stack becomes significant.
•TOE is often suggested as a way to reduce the overhead associated with IP storage protocols such as iSCSI and NFS.

Known Issues with Offload (TOE and TSO) Engines:
•Limitations of hardware — because connections are buffered and processed on the TOE chip, resource limitations happen more often then they would if processed by the ample CPU and memory resources that are available to the operating system. This limitation of resources on the TOE chip can cause communication issues.
•Complexity — issues such as memory used by open connections are not available with TOE. TOE also requires very large changes to a networking stack in order to be supported properly, and even when that is done, features like Quality of Service and packet filtering typically do not work.
•Proprietary — TOE is implemented differently by each hardware vendor. This means more code must be rewritten to deal with the various TOE implementations, at a cost of the aforementioned complexity and, possibly, security. Furthermore, TOE firmware cannot be easily modified since it is closed-source.
•Performance — Each TOE NIC has a limited lifetime of usefulness, because system hardware rapidly catches up to TOE performance levels, and eventually exceeds TOE performance levels. TOE does not increase bandwidth on the network. In simple terms, TOE removes the responsibility of the protocol stack from the Server’s CPU allowing the server CPU to process information faster. As hardware performance increases, processes can complete their task prior to TOEs acknowledgment of the receipt of transmission; thus causing communication issues.

Disabling TOE:
1. From the Control Panel, open Network Connections.
2. Open the properties page for the network connection being used an click Configure
3. On the Advanced tab, change the Value of all 'Offload' settings to None, Off or Disabled
4. Set Receive Side Scaling to 0
5. Click OK to save changes.

NOTE: About Disabling TOE or TSO
•Once a modification to the NIC card properties is made, the NIC card itself, resets. If these changes are made through a Remote Desktop Connection, the communications will be broken and a new session will need to be established.

NIC TCP Chimney Offload, Receive Side Scaling, and Network Direct Memory Access Explained:
•TCP Chimney is Microsoft's software enhancement. Receive Side Scaling and Network Direct Memory Access fall under the umbrella of the TCP Chimney feature.
•TCP Chimney is software driven to take advantage of the NIC card TOE capability; this feature requires the use of the NIC hardware in order to function. Even though TCP Chimney improves CPU performance by reducing the CPU overhead, the benefits or disadvantages of TCP Chimney, TOE and TSO are continually being debated, discussed and modified. The benefits or disadvantages are dependent on the specific CPU and RAM memory designs, the CPU and BUS speeds, as well as the NIC’s TOE design, all of which will have an effect on the Offload performance.
 

Disabling TCP Chimney features:
•View the current Offload state
(Run the following from a Command prompt)
Windows 2003: Netsh int ip show offload

Windows 2008/7: Netsh int tcp show global
NOTE: Netstat –t Shows a snapshot in time to the state of a TCP port. The idle state of a TCP port will revert to inHost. This IS NOT a reliable method in determining the Offload status of a NIC card.
NOTE: Netsh show command is unreliable in a Window 2003 Virtual server and requires the registry to examined to verify the offload features are disabled.

Disabling TCP Chimney feature (2):
•Verify the Offload features are disabled on a Windows 2003 server
1. Click Start > Run > type Regedit > OK.
2. Locate the following registry subkey: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters
3. Examine the EnableTCPChimney registry entry for the value 0, otherwise edit and modify .
4. Examine the EnableRSS registry entry for the value 0, otherwise edit and modify .
5. Examine the EnableTCPA registry entry for the value 0, otherwise edit and modify .
6. Restart of the server will be necessary for any changes to the TCPIP parameters to take affect.

Disabling TCP Chimney feature (3):
•Disable TCP Chimney
(Run the following from a Command prompt)
For Windows 2003: Netsh int ip set chimney disabled

For Windows 2008: Netsh int tcp set global chimney=disabled

Disabling TCP Chimney feature (4):
•Disable Receive Side Scaling
Windows 2003: (Must be set at the registry level) HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\EnableRSS Value = 0.

Windows 2008: Netsh int tcp set global rss=disabled

 Disabling TCP Chimney feature (5):
•Disable Network Direct Memory Access (NetDMA)
Windows 2003 and 2008: Must be set at the registry level: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\EnableTCPA Value = 0.
NOTE: Windows 2008 the Dword registry key must be added
NOTE: Windows 2003 and 2008 servers need to be rebooted in order for the registry changes to take affect.

Server Session Sockets/Ports Tuning

Server Session Sockets Explained:
Most programming products in commercial use today are multi-threaded. For example, the Accelerator product from Enterprise Vault is able to search indexes, export emails and allow user to review and mark messages simultaneously by sending out several threads per task.

Each thread consumes a session socket or otherwise known as a User port. Also each thread has a time to live or remain idle before the thread is ‘cleaned up’ and the socket is once again available for use.

In this day and age of multi-tasking programs, several thousands of threads establish connections between servers at any given instance. Since limitations are set to the quantity of sockets made available to threads, this limitation needs to be increased to reduce the possibility of communications failure between servers because of the lack of sockets.

Increase TCP User Ports
•Windows 2003
-By default windows 2003 allows socket connections to be established between the ports 1024 – 5000
-To increase the ports add the registry key:HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters

Update the following values, or create them if they do not already exist:

Name                                    Type            Default Value    Recommended Value (decimal)
MaxUserPort                      DWORD     5,000                 64,512
TCPTimedWaitDelay        DWORD     240                    30
MaxFreeTcbs                     DWORD     2,000                 66,536
MaxHashTableSize          DWORD     512                    16,384

Increase TCP User Ports - II
•Windows 2008
-By default windows 2008 allows socket connections to be established between the ports 49152 – 65535
-To display the current ports per protocol: Netsh int ipv4 show dynamicport tcp
-To increase the ports: Netsh int ipv4 set dynamicport tcp start=1500 num=63000

If the MaxUserPort registry key (32-bit) was added to a Windows 2008 server as done in a Windows 2003 server, the same effect would occur, but is not the recommended method per Microsoft.

Increase TCP User Ports - III
•When increasing the MaxUserPort, Microsoft recommends that port 1434 be reserved for use by the SQL Server Browser service (sqlbrowser.exe).
•On a Windows 2003 server add the following registry key

1. Locate and then click the following registry subkey: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters
2. Add a Multi-string Value key.
3. Rename the new key to ReservedPorts
4. Add the values: 1434-1434

•On a Windows 2008 server start the port allocation at 1500 as shown in Increase TCP User Ports - II (Previous section.)

Memory Management
•Multiple programs will compete for memory resources.
•To determine which processes are consuming a server’s memory resources, open the server’s Task Manager and sort by Mem Usage.
 

Task manager – Performance Tab

Task manager – Commit Charge (Pagefile) Defined:
•Total Commit Charge should never be higher then Total Physical memory. Otherwise add memory to the server.
•Peak should not be near Limit. Otherwise in crease the page file size or add additional page files.
 

Microsoft’s Page File Memory Usage
•Controlling Registry Key: HKEY_LOCALMACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management
•PagedPoolSize = 0xffffffff - hex (DWORD)
This sets the amount of the servers memory that can be under the control of Microsoft’s Memory Management process.
•PoolUsageMaximum = 60 – decimal (DWORD)
This sets the percentage of memory that must be consumed before Microsoft’s Memory Management takes control.

 Overview of Server Memory
•A 32-bit server can address up to 4 gigabyte of physical memory
- The first half of memory is reserved for the Kernel or operating System (Normally 2 GB)
-The second half of memory is reserved for the Application software (Normally 2GB)
•A 64-bit server can address up to 4 petabytes of physical memory
 

Boot.ini Memory Management -
 NOTE: This step is ONLY recommended for SQL Servers and NOT for SEPM's.
•SQL servers running on Windows 32-bit platform, should make use of the /3GB and /PAE switches in the boot.ini. These switches should be removed in a Windows 64-bit operating system.
•/3GB: Windows 32-bit operating systems address space up to 4 gigabytes (GB) of virtual memory. The address space is usually split so that 2 GB of address space is directly accessible to the application and the other 2 GB is only accessible to the Windows executive software. The /3GB switch allocates 3 GB of virtual address space to an application that uses IMAGE_FILE_LARGE_ADDRESS_AWARE in the process header. This switch allows applications to address 1 GB of additional virtual address space above 2 GB.
•/PAE (Physical Address Extension) is a feature to allow 32-bit processors to access physical memory higher than the 4 gigabyte limit. The 4GB limit is not changed, so regular application software continues to use instructions with 32-bit address boundary, but the operating system uses page tables to point a process within the 4-GB address space that points to a block of memory within the additional physical memory above the 4GB limit. This allows a 32-bit system to access up to 64GB total physical memory

How the Boot.ini Switches Affect 32 Bit Memory Usage:
No Switches in Boot.ini        /3GB Switch in Boot.ini Only                   /3GB and /PAE Switch in Boot.ini
   32 Bit                                             32 Bit                                                                  32 Bit
12GB Total Memory                12GB Total Memory                                      12GB Total Memory
                                                                                                                   App Mem Moved to Upper Memory
    2GB App Use                          3GB App Use                                                  3GB App Use
    2GB OS Use                           1GB OS Use                                                    1GB OS Use

/3GB and /PAE Switches (continued)
•Each process in a 32-bit server has the ability to address up to 4GB of physical, virtual or a combination of both.
•Each process will consume a portion from each partition, Kernel and Application memory partitions .
•More physical memory is available to the applications when using the /3GB switch which reduces the need to swap memory to or from the page file and results in increased performance.
•When using the /PAE switch, the paging of memory blocks will be first to the faster upper physical memory verses the slower virtual memory on hard drives.
•If using the BOTH the /3GB and /PAE switches, there is a limitation of 16GB of physical memory that can be accessed.

HDD Fragmentation
Excessive hard drive fragmentation will cause performance degradation.
•To resolve hard drive fragmentation use the Disk Defragmenter tool located under The Computer’s Accessories | System Tools.
•Care should be taken before undertaking this task because of the performance impact of running the Disk Defragmenter tool in large and should only be run after hours.
If the hard drive is bad, running the Disk Defragmenter tool might result in the inability to access the programs or the drive itself.  Be made aware of this possibility before performing this utility. 

Server Optimization Summary
•Keep NIC Drivers up to date and matched between servers in a cluster.
•Disable TOE and TSO.
•Disable TCP Chimney.
•Open maximum session sockets by use of the MaxUserPort registry key (Windows 2003) or Net Shell command line (Windows 2008).
•Add additional Server memory if Commit Charge Total exceed Physical memory Total.
•Add or increase Page Files sizes if Commit Charge Peak nears Commit Charge Limit.
•Set the Microsoft Memory Management registry values at full or 0xffffffff and to take control at 60% or sooner.
•Establish page files on separate physical hard drives and set to 1.5x the size of the physical memory.
•Use the both the /3GB and /PAE boot.ini switches on SQL servers running on 32-bit operating systems (with physical memory between 4GB and 16GB) .\

Terms used in this document defined:

TCP
TCP requires the operating system to maintain state information for each connection. As client numbers rise, the number of concurrent TCP connections must be allowed to increase. If you manage more than 5,000 clients, Symantec recommends that you tune the Windows TCP implementation for better scalability and performance.

TCP can be tuned at the operating system level. Most settings are in the Registry at HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\Tcpip\Parameters.
TCP parameters are documented on the Microsoft Web site. Typically, changes to these settings require that you restart the computer before the changes effect the operation of the TCP stack.

MaxUserPort
Registry value: HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\Tcpip\ Parameters\MaxUserPort
Value type: REG_DWORD
Benchmark setting: 50000 (decimal)

Windows servers limit the number of outbound TCP connections. Since the SEPM server initiates outbound connections during a virus definition push, heartbeat, database connection, etc.  This key should be set to a very large number. The maximum allowable is 65535, which is the highest possible port number. However, it is good practice to leave space for inbound connections.

(Excerpt from microsoft : http://support.microsoft.com/kb/328476 )Note: If you adjust the MaxUserPort setting, we recommend that you reserve port 1434 for use by the SQL Server Browser service (sqlbrowser.exe). For more information about how to do this, click the following article number to view the article in the Microsoft Knowledge Base:

MaxFreeTcbs and MaxHashTableSize
Registry value: HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\Tcpip\ Parameters\MaxFreeTcbs
Value type: REG_DWORD
Benchmark setting: 65535 (decimal)

Registry value: HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\Tcpip\ Parameters\MaxHashTableSize
Value type: REG_DWORD
Benchmark setting: 65535 (decimal)

These settings cap memory usage by the TCP stack. Based on memory availability, they should be set to the maximum allowable number. Both settings should be kept in sync.

TcpTimedWaitDelay
Registry value: HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\Tcpip\ Parameters\TcpTimedWaitDelay
Value type: REG_DWORD
Benchmark setting: 30 (decimal)

TCP puts closed connections into the "timed wait" state to prevent stray packets from a connection that may be on the network due to re-transmissions from interfering with any subsequent connections that use the same port. The default is two times the maximum segment lifetime, which is typically four minutes. A common practice In LAN environments is to reduce that setting to the minimum of 30 seconds. This reduction allows Windows to free up TCP ports more quickly for outbound connections, which speeds up a SEPM server's content rollout.  This setting is also appropriate for implementation on most systems hosting the database used by a SEPM system.

Registry usage and paged pool
The size of the Registry limits the maximum number of clients. The size of paged pool memory controls the maximum size of the Registry. These settings only apply to Windows 2000. Windows 2003 Server and Windows XP no longer observe the RegistrySizeLimit setting and do not limit the size of the Registry based on paged pool memory. Windows 2003/XP dynamically sets the size of the Registry.

PagedPoolSize
Registry value: HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\Session Manager\Memory Management\PagedPoolSize
Value type: REG_DWORD
Benchmark setting: 0xFFFFFFFF (hexadecimal)

This setting controls the amount of paged pool memory that Windows is allowed to create. The maximum size of the Registry is 80% of the paged pool size. A setting of 0xFFFFFFF tells Windows to make the Registry as large as possible.

Registry SizeLimit
Registry value: HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\RegistrySizeLimit
Value type: REG_DWORD
Benchmark setting: 0xFFFFFFFF (hexadecimal)

This setting controls the maximum size of the Registry. A setting of 0xFFFFFFF tells Windows to make the Registry as large as possible.