Scalability and performance guidelines for Symantec AntiVirus Corporate Edition 10.x and Symantec Client Security 3.x
|Article:TECH101448|||||Created: 2005-01-29|||||Updated: 2010-01-14|||||Article URL http://www.symantec.com/docs/TECH101448|
You need detailed information about deploying Symantec AntiVirus Corporate Edition 10.x or Symantec Client Security 3.x.
This document applies to the following Symantec products:
- Symantec AntiVirus 10.x Small Business Edition
- Symantec AntiVirus 10.x Corporate Edition
- Symantec Client Security 3.x Small Business Edition
- Symantec Client Security 3.x Corporate Edition
This document describes the performance and scalability characteristics of a Symantec AntiVirus Corporate Edition parent server or a Symantec Client Security parent server running on Microsoft Windows-based server platforms. Novell NetWare is not discussed in this document.
Note: For brevity,when this document refers to Symantec AntiVirus, Symantec Client Security 3.x is included. References to Symantec AntiVirus 8.1 include Symantec Client Security 2.0.
Plan your deployment
The following lists are to help you decide which optimizations and trade-offs fit the specific needs of your network.
Decide on the frequency of virus definition updates
First, identify the goals and priorities affecting the frequency of virus definitions updates. Include exceptions, such as emergency updates that should be anticipated.
- How quickly and how often do you plan to update virus definitions?
- In terms of acceptable performance, how is an emergency virus update different from a normal weekly virus update?
- How much network bandwidth do you want to use in normal and emergency situations?
- How much visibility do you want into the state of the machines that you manage?
- What is the acceptable time frame to get current virus definitions to 90% of your machines during an emergency?
90% coverage usually contains a virus effectively while you update the slow machines. This time frame is usually a factor when you want to limit the amount of network bandwidth that is used for updates.
- How many parent servers do you want?
The number may depend on organizational issues or network infrastructure.
Determine the number of clients per parent server
Consider the four following parameters when you decide how many clients you can or want to manage from a Symantec parent server:
- Network load issues.
These issues include the amount of network traffic that is generated, how frequently that load is generated, how long that load lasts on your network, how that load is distributed on your network, and how much network load you are willing to use for these tasks.
- The urgency with which you want to send virus definitions updates or settings changes.
The urgency may vary, depending on the circumstances.
- The resources that are needed for the server memory, CPU, and network card bandwidth.
- The resources that are needed to run the Symantec System Center console (primarily in terms of memory).
Tune some parameters
Tuning some operating system and application parameters was necessary to achieve the results shown in the Technical Information section of this document
TCP requires the operating system to maintain state information for each connection. As client numbers rise, the number of concurrent TCP connections must be allowed to increase. If you manage more than 5,000 clients, Symantec recommends that you tune the Windows TCP implementation for better scalability and performance.
TCP can be tuned at the operating system level. Most settings are in the Registry at HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\Tcpip\Parameters.
TCP parameters are documented on the Microsoft Web site. Typically, changes to these settings require that you restart the computer.
Registry value: HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\Tcpip\ Parameters\MaxUserPort
Value type: REG_DWORD
Benchmark setting: 50000 (decimal)
Windows servers limit the number of outbound TCP connections. Since the Symantec AntiVirus server initiates outbound connections during a virus definition push, this key should be set to a very large number. The maximum allowable is 65535, which is the highest possible port number. However, it is good practice to leave space for inbound connections.
MaxFreeTcbs and MaxHashTableSize
Registry value: HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\Tcpip\ Parameters\MaxFreeTcbs
Value type: REG_DWORD
Benchmark setting: 65535 (decimal)
Registry value: HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\Tcpip\ Parameters\MaxHashTableSize
Value type: REG_DWORD
Benchmark setting: 65535 (decimal)
These settings cap memory usage by the TCP stack. Based on memory availability, they should be set to the maximum allowable number. Both settings should be kept in sync.
Registry value: HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\Tcpip\ Parameters\TcpTimedWaitDelay
Value type: REG_DWORD
Benchmark setting: 30 (decimal)
TCP puts closed connections into the "timed wait" state to prevent stray packets from a connection that may be on the network due to re-transmissions from interfering with any subsequent connections that use the same port. The default is two times the maximum segment lifetime, which is typically four minutes. A common practice In LAN environments is to reduce that setting to the minimum of 30 seconds. This reduction allows Windows to free up TCP ports more quickly for outbound connections, which speeds up a Symantec AntiVirus server's content rollout.
Symantec AntiVirus server
The Symantec AntiVirus server itself can also be tuned. Settings are in the Registry in either HKEY_LOCAL_MACHINE\Software\Intel\LANDesk\VirusProtect6\CurrentVersion or HKEY_LOCAL_MACHINE\Software\Symantec\Symantec Client Security\SecureComms.
Registry value: HKEY_LOCAL_MACHINE\Software\Symantec\Symantec Client Security\SecureComms\ConnectionMax
Value type: REG_DWORD
Benchmark setting: 50000 (decimal)
This setting controls the maximum number of concurrent connections that a Symantec AntiVirus server can have. This setting is queried when the server opens outbound connections for rollout. Inbound connections are not throttled because the server handles as many connections as the operating system allows.
The default for this setting is 3000. Symantec does not advise that you set this value below 3000. A lower value may prevent virus definitions rollouts when the server handles large number of concurrent client check-ins.
This setting has an important impact on memory usage. For details, see the "Memory usage" section of this document.
Registry value: HKEY_LOCAL_MACHINE\Software\Intel\LANDesk\VirusProtect6\ CurrentVersion\ClientUpdateThreadPool
Value type: REG_DWORD
Benchmark setting: Based on CPU. See the "Effect of CPUs" section of this document.
For information about adjusting the ClientUpdateThreadPool, see document Configuring the number of updates that a Symantec AntiVirus server can push simultaneously
As in Symantec AntiVirus 8.1, the number of rollout threads is configurable. Each thread handles one client at a time, so the number of rollout threads equals the number of clients that can receive virus definitions concurrently. A configuration dialog has been added to the Symantec System Center that allows you to set up this number on a server or server group basis.
Registry usage and paged pool
The size of the Registry limits the maximum number of clients. The size of paged pool memory controls the maximum size of the Registry. These settings only apply to Windows 2000. Windows 2003 Server and Windows XP no longer observe the RegistrySizeLimit setting and do not limit the size of the Registry based on paged pool memory. Windows 2003/XP dynamically sets the size of the Registry.
Registry value: HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\Session Manager\Memory Management\PagedPoolSize
Value type: REG_DWORD
Benchmark setting: 0xFFFFFFFF (hexadecimal)
This setting controls the amount of paged pool memory that Windows is allowed to create. The maximum size of the Registry is 80% of the paged pool size. A setting of 0xFFFFFFF tells Windows to make the Registry as large as possible.
Registry value: HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\RegistrySizeLimit
Value type: REG_DWORD
Benchmark setting: 0xFFFFFFFF (hexadecimal)
This setting controls the maximum size of the Registry. A setting of 0xFFFFFFF tells Windows to make the Registry as large as possible.
Symantec System Center scalability
The functions of the Symantec System Center that affect scalability have not changed significantly. The following graph shows the amount of time that it takes to open a server group and view the list of clients under a particular server by the number of clients that report to that server. The computer that ran the Symantec System Center was a workstation class machine with a P4 2.8 GHz hyperthreaded CPU and 1 GB of RAM.
The refresh time goes to the heart of the visibility of your clients. If you want to efficiently use Symantec System Center to troubleshoot individual clients, you can deploy more servers with fewer clients.
KB of memory used
- Each client requires 4KB of memory on the Symantec System Center computer.
- Symantec System Center itself (hosted by Mmc.exe) uses 3 KB.
- The Symantec System Center Topology Service (nsctop.exe) uses 1 KB.
For information on the scalability of the Reporting feature in Symantec AntiVirus 10.1 and following versions, read the white paper (in .pdf format) Reporting and optimizing reporting performance for your network.
Differences between Symantec AntiVirus 10.x and previous versions
This section describes scalability and architectural differences.
Scalability differences between Symantec AntiVirus 10.x and previous versions
Scalability differences involve definition rollout speed and network and CPU utilization.
Virus definition rollout speed
On comparable hardware, the rollout of equally-sized virus definitions takes approximately 40% more time than in previous versions. However, with the addition of two CPUs, rollout of equally-sized virus definitions takes approximately 35% less time than in previous versions. CPU speed likely has an effect as well, although this speed was not tested.
The virus definitions size increased since the last scalability numbers were published because of the addition of security risk detection, such as Spyware and Adware. This increase affects the rollout speed, as more data must be pushed to each client.
Network bandwidth use
Symantec AntiVirus 10.x sends the same amount of data across the network as Symantec AntiVirus 8.1. However, its characteristics are slightly different. During a virus definition push, a Symantec AntiVirus 10.x server sends approximately 15% more data but receives approximately 15% less data. The increase in outbound data is due to the addition of TCP and SSL overhead. The reduction in inbound data is due to an optimization in the product made possible by the switch to TCP. The UDP-based protocol sent an ACK for every packet before it sent the next packet. This behavior was removed because TCP provides guaranteed delivery.
Symantec recommends that you tune the number of push threads and use a pair of Symantec AntiVirus settings that control thread throttling and outbound connection maximums (described later) to generate the network bandwidth that you want. In Symantec AntiVirus 10.x, the maximum number of push threads is 100, however in most environments a ClientUpdateThreadPool value higher than 30 results in poor system and network performance and is not recommended.
In Symantec AntiVirus 8.1, the number and speed of CPUs is generally not a limiting factor. In Symantec AntiVirus 10.x, CPU count has a significant affect on scalability.
Architectural differences between Symantec AntiVirus 10.x and previous versions
The primary architectural differences revolve around TCP protocol implementation.
What has changed in the architecture
The move to TCP from UDP adds connection establishment overhead and requires the addition of connection management logic to Symantec AntiVirus 10.x. TCP connections are reused as often as possible to minimize the impact of connection establishment on scalability. When a client with out-of-date virus definitions checks in to its parent server, the server reuses that TCP connection when it sends out the new set of virus definitions.
The move to SSL encryption requires more CPU overhead than in previous versions. Symantec chose default encryption algorithms that try to create an optimal balance between performance and security. Symantec AntiVirus 10.x uses the default SSL cipher suite with RSA and 1024-bit keys for certificates, RC4 for bulk data encryption, and SHA-1 for hashing.
We modified the client simulation scheme in the test bed to account for the use of TCP/SSL and to cover elements of real customer environments. Our goal was to provide the most accurate, actionable results possible. The maximum number of push threads is now 100. This parameter can be configured in the Server Tuning options pane in the Symantec System Center.
What has not changed in the architecture
The internal threading and memory architecture that handles check-ins from clients and secondary servers and virus definition updates has not changed.
Client identity and state information is still stored in the Registry. Secondary server identity and state information is also still stored in the Registry. The size of the Registry is still the limiting factor for the upper limit for the number of clients that a single parent server can manage.
Files that are rolled out to clients are still cached in memory by the server, which eliminates the need to make redundant disk accesses.
Symantec AntiVirus 8.1 clients do not have to be upgraded. The server uses the same UDP-based protocol to manage these clients that a Symantec AntiVirus 8.1 server uses. Symantec AntiVirus 10.x servers can simultaneously manage those clients that use the UDP-based protocol and clients that use the TCP/SSL based protocol.
This section describes the simulation that was performed in the test bed.
We performed tests on a server with the following:
- Quad 2.0 GHz, hyper-threaded, Xeon processors
- 4 GB of addressable RAM (8GB physical, but PAE was disabled)
- Windows 2000 Advanced Server, Service Pack 4
During the tests, the number of CPUs was changed to determine the effect of CPU counts on scalability. Except where explicitly noted, the tests were performed with the number of CPUs set to two with hyperthreading disabled. In the scalability tests for Symantec AntiVirus 8.1, the server had dual 1.1 GHz processors and 1 GB of RAM and ran Windows 2000 Server SP4.
The characteristics of the hard disk are usually not relevant to Symantec AntiVirus and Symantec Client Security scalability. Files to be rolled out are cached in memory after they have been read from the disk for the first client rollout.
The server was connected to a 1 GBit network. Client computers were connected to a 100 MBit switch in groups of six. Each client computer hosted between 25 and 100 simulated Symantec AntiVirus clients. In the scalability tests for Symantec AntiVirus 8.1, the server was connected to a 1 GBit fiber ring, and the clients were configured similarly to the Symantec AntiVirus 10.x tests.
Since it is impractical to recreate an environment with tens of thousands of computers that run Symantec AntiVirus client, we used a simulation scheme. The simulation scheme was carefully chosen to accurately emulate actual large environments. The scalability tests of Symantec AntiVirus 8.1 also used a simulation scheme.
Clients were simulated by reconfiguring actual Symantec AntiVirus clients to listen on multiple TCP ports simultaneously. The server then sent the virus definitions to each port multiple times and counted each as a push. The server was modified in two important ways. First, the server never pushed more than one set of virus definitions to the same port simultaneously. Second, the server always allowed a period of time to pass between pushes to the same port to allow for the proper cleanup of the TCP/SSL session. This step allowed for accurate delay representation between the client's virus definition import and its check-in to the server. This delay was set to 30 seconds.
In the Symantec AntiVirus 8.1 scalability tests, the clients listened on two UDP ports. The same effort was made to prevent simultaneous rollout to the same port. The delay to allow clients to process virus definition was not part of that test. The Symantec AntiVirus team added the delay to the simulation for Symantec AntiVirus 10.x because we believe that it creates a more realistic test.
As a consequence of shared hardware for concurrent rollouts, we introduced artificial resource contention into the test. However, Symantec believes that the contention impact to test results represents a responsible underestimation of scalability, rather than an overestimation.
Unless specifically noted, these results were achieved by using a server with dual 2.0 GHz Xeon CPUs and 4 GB of memory on a 1 GBit subnet with switched 100 MBit subnets down to the clients. The server was tuned to optimize virus definitions rollout, as described in the "Tune some parameters" section of this document.
Results are categorized as follows:
- Content rollout speed
- Memory usage
- Network bandwidth utilization
- Maximum number of clients and the registry
Content rollout speed
This section provides content rollout speed test results in the following categories:
- Virus definitions
- Effect of virus definitions sizes
- Settings (GRC)
- Effect of CPUs and push thread pool size
The following graph demonstrates the scalability of Symantec AntiVirus 10.x using a 68 KB virus definitions package as compared to previous versions.
As in previous versions, an increased number of clients has a linear effect. This graph is primarily provided for comparison to previous versions. The average size of the weekly virus definitions set has increased in the last few years.
The following graph illustrates the scalability of Symantec AntiVirus 10.x with a virus definition package that is closer to today's average weekly incremental update virus definitions package.
As in previous versions, an increased number of clients has a linear effect. This linear rise occurs regardless of the size of the virus definition packages. CPU count has a dramatic effect on rollout speed. In previous versions, additional CPUs did not make a difference. With the addition of SSL in Symantec AntiVirus 10.x, CPUs do make a difference. The next chart illustrates the effect of additional CPUs on virus definition rollouts.
Effect of virus definitions sizes
Since version 8.0, Symantec AntiVirus servers included micro-definitions functionality in virus definition rollout. When a client checks in to its parent server, it tells the server which version of the virus definitions it has. The server then sends the smallest incremental virus definitions package (an IDB file) necessary for that client to get up to date. The size of that package has a direct affect on how quickly the server can distribute the virus definitions update.
If a client is significantly out-of-date, the server may not have an incremental package for it and must send the full set of virus definitions (a VDB file). The server pulls a larger set of virus definitions (an XDB file) from Symantec's LiveUpdate server. The .XDB file includes the VDB file and many IDB files.
Since the release of Symantec's integrated Security Risk detection with the March 9, 2005 virus definitions package, the virus definition sizes have increased noticeably. In response to customer feedback on the size of the XDB packages, Symantec's Security Response decided to decrease the IDB cache from 28 days to 10 days, effective May 25, 2005. This cache decrease means that those clients whose virus definition versions are more than 10 days older than the server receive the VDB package.
Virus definitions date
April 15, 2005
April 22, 2005
April 29, 2005
May 6, 2005
May 13, 2005
May 20, 2005
May 27, 2005
June 3, 2005
June 10, 2005
June 17, 2005
June 24, 2005
July 1, 2005
July 8, 2005
The following graph illustrates the effect of virus definition file size on rollout speed for small to medium client counts.
The following graph illustrates the effect of virus definition file size on rollout speed for medium to large client counts.
The two graphs are split only to ensure that the larger client count numbers are visible, due to the scale of file sizes that were tested in the smaller client counts. At every client level, the effect of file size is linear, with some fixed connection establishment overhead.
The following graph illustrates the speed of settings rollout.
While the rollout of settings (GRC files) is faster, important memory considerations should be understood about these numbers. These considerations relate to the number of concurrent connections that you allow a Symantec AntiVirus server to hold. During GRC file rollout, a Symantec AntiVirus server can create thousands of concurrent connections, because the file rollout to each individual client takes a very short time. However, the server holds the connection open until the client checks back in to report that it applied the settings correctly. Typically, this transaction takes a few seconds.
Effect of CPUs and push thread pool size
A Symantec AntiVirus server has a pool of push threads that handle content rollout to clients (and secondary servers). During rollout, one push thread is dedicated to communication with one client until that client has received all applicable content updates. Thus, the number of push threads that you configure dictates the number of clients that a server can roll out to concurrently. The use of multiple threads also allows the server to use more than one CPU. Additional CPUs did not affect scalability much in previous versions. In Symantec AntiVirus 10.x, the use of SSL puts significantly more load on the CPUs.
In general, a Symantec AntiVirus 10.x server uses 100% of the CPUs during rollout, unless an effort is made to throttle back the rollout as described in the Memory Usage and Network Bandwidth sections that follow. In Symantec's scalability tests, CPU usage was under 100% only for scenarios where the number of push threads was configured to five or less on a multiple CPU box.
However, our test environment did not include high latency links, multiple concurrent pushes across low-bandwidth WAN links, or large numbers of off-line clients. These factors may reduce overall CPU load during a rollout and make require a larger number of push threads than is recommended in this document.
The following graph illustrates rollout speed by CPU and push thread count.
On a computer with one CPU, a larger thread pool only gives a small advantage. Each additional thread adds overhead as well as push capacity. On multi-CPU computers the improvement is dramatic, as threads can now do computationally expensive operations like SSL encryption in parallel. Most of the scalability tests were done with a dual-CPU server, since these are more economically viable than quad-CPU servers in small-to-medium business environments.
In the scalability tests, we used the following push thread pool sizes:
Push thread pool size
In Symantec AntiVirus 10.x, you can tune the number of push threads in Symantec System Center in the Server Tuning Options pane. The default value is five threads. Symantec recommends that you increase the count based on the tuning guidelines per CPU. Changes to the push thread pool size are applied dynamically (no restart needed) within three minutes.
In previous versions, the size of the push thread pool had a linear effect on network bandwidth consumption. This effect has changed somewhat. The "Network bandwidth" section of this document provides more information about this effect.
Note that the push thread pool is separate from the thread pools that handle inbound requests from clients and management consoles. Examples of such requests include attempts to forward event logs and virus alerts. Those pools do not need to be tuned to achieve the scalability numbers in this document.
The primary factor that drives memory requirements is connection management. The number of connections that a Symantec AntiVirus server can have open at any given time dictates the maximum amount of memory that is used. This number is configurable by editing the ConnectionMax Registry value. See the "Tune some parameters" section of this document for details.
In all of our tests, we never had more than 10,000 concurrent connections, even during a rollout of a 6KB settings file to 175,000 clients on a dual-CPU server. However, if you use faster hardware, you may see higher numbers of concurrent connections. Remember that the smaller the file rolled out, the more concurrent connections a Symantec AntiVirus server handles. You can monitor connections in the Windows Performance Monitor by using the TCP:Connections Established counter.
The formula for calculating memory usage by a Symantec AntiVirus server is as follows:
Virus definitions and scanning
Client Track feature
1 KB / client
25 KB / connection
All other purposes
For example, a server with 10,000 clients needs 340 MB at peak load during a settings rollout, if it is configured to allow that many concurrent connections. By default, the ConnectionMax setting is 3,000. Without tuning, 10,000 clients require 145 MB. However, the rollout slows as the server throttles back the opening of outbound connections.
Once the rollout to a client completes and the connection is closed, the memory that is used for that connection is released. So, most of the time, the Symantec AntiVirus server does not consume anywhere near the peak memory use that is derived from this formula. The cleanup occurs on a timed basis, every 10 seconds by default, so you may see a step-down effect. In the Windows Performance Monitor, the counter to use is Process:Private Bytes on the Rtvscan process.
Note that this formula applies only to the main Symantec AntiVirus service, Rtvscan.exe. The memory usage of other Symantec AntiVirus services is relatively static - around 20 MB total.
The Client Track feature monitors clients and logs events when clients don't check in for a period of time. Reports that are based on this data can be generated in the Symantec Event Manager. In steady state, this feature uses 1.5 KB per client. When one server manages many thousands of clients, there is steady memory growth over the first few minutes of server execution. This growth is due to the Client Track feature caching client information. To disable the feature, create the following REG_DWORD value in the Registry and set it to 0:
In previous versions, a decrease in the push thread pool size controls network bandwidth. This control is still generally the case, although to a lesser extent, as the following graph illustrates:
Note that any multiple CPU system can be configured to fill a 100 MBit network. In our tests, these numbers were recorded on the server's 1GBit subnet, hence the ability of the quad-CPU server to consume 200 MBits. The virus definition rollout numbers in this document were achieved with a dual-CPU server, so a 100 MBit network is sufficient to reach that high level of scalability.
If less network bandwidth needs to be consumed, you can use the FileCopyPacketThrottle setting to make each push thread pause between any packets that are sent to clients. This setting is applied immediately. See the "Tune some parameters" section of this document for details.
The following graph illustrates network bandwidth utilization throttling.
These network utilization numbers were generated with 100 push threads on a dual-CPU server with 10,000 clients. Based on the size of the push thread pool that you select, your numbers may differ.
The scalability tests did not include any scenarios that may have an important affect on network utilization in an enterprise environment. Specifically, we did not have any off-line clients, low-bandwidth WAN links, or high-latency WAN links. To gauge network use in your environment, use the Network Interface:Byes Total counter in the Windows Performance Monitor.
Maximum number of clients and the registry
The size of the Registry limits the maximum number of clients. The parent server stores data about each client in the Registry. Each client has its own key. On Windows 2000 Server, the maximum number of clients that Symantec AntiVirus 10.x can manage is 40,000. On Windows 2003 Server, the maximum number of clients that Symantec AntiVirus 10.x can manage is 60,000. The amount of data that Symantec AntiVirus 10.x stores is slightly larger than the amount of data that Symantec AntiVirus 8.1 stores, which is why the number is smaller.
To reach the scalability level in this document, the size of the Registry must be set to a maximum. We accomplished this by maximizing the Windows Paged Pool. The maximum Registry size is a fixed percentage of Paged Pool. For more information, see the "Tune some parameters" section of this document.
Article URL http://www.symantec.com/docs/TECH101448