In testing scenarios we found that not all systems would power-up using Intel vPro technology when a Task Server job was pushed to the upper single digits. Depending on the environment the success rate would fall to as much as half the targeted machines. Through the course of troubleshooting the issue we discovered that what appeared to be an issue with Real-Time Console Infrastructure ended up being a default limitation in ASP.NET concerning how many concurrent connections can be made within a single session. This article covers how to configure your Notification Server so that all RTCI functions, especially power-on scenarios, would succeed to all targeted systems.

Introduction

This document addresses the issues seen when using Task Server and Real-Time Console Infrastructure to execute power management functions and other tasks that utilize multiple protocols. The Real-Time Console Infrastructure has a batch-type approach to executing functions, but we've discovered that the number of systems batched at one time (ten) appears to be too many for a default ASP.NET session. As such there are two methods for avoiding this issue, detailed in this article.

What is the problem?

To put it simply, the symptoms are that not all targeted system in a Task Server task executes the task's function. Failure rates vary depending on the environment, but typically the failure rate falls between very low percentages to half. Once you understand the cause of the issue and some of the potential fixes. The reason for the issue becomes quite clear.

The symptoms for this issue are detailed below. This process also illustrates what causes the issue and what type of tasks are most likely affected. The following shows how the issue is easily reproduced:

1. In the Altiris Console browse under Manage > Jobs > browse down through Tasks and Jobs > Jobs > right-click on Jobs and choose New > Task/Job. (if so desired you can choose another location to create the Job in)
2. From the left-hand tree, under the Server Tasks > Real-Time Console Infrastructure, select Power Management Task.
3. Leave the defaults in the Task but provide a unique name, such as AMT power on Test, as shown in this screenshot:
   
   

   Click to view.

4. Click OK to save the Task.
5. On the left you'll see the new Task. On the right click the 'Run Now' button.
6. Provide a Run Name such as Test Power-on.
7. Select a Connection Credential profile that contains correct credentials to use AMT if the one selected by default does not contain the right credentials.
8. Click the 'Select Computers' link and choose 8 to 10 vPro capable systems (why this number is important will be detailed later). See this screenshot for an example:
   
   

   Click to view.

9. Click the Run Now button.

1. Typically 1 to 4 of these systems will fail to wake up. Repeating the task often shows different results, ruling out the possibility that some of the machines are not correctly configured or are in an inoperable state. This problem may also be seen with the following types of tasks:

• Power control, Power-on
• Power control, Power-off
• Power control, Restore State

What Causes the problem?

As previously eluded to, the problem stems directly from the default concurrent connection settings per session enforced by ASP.NET. The defaults allow a low number of concurrent connections, which may be considered restrictive depending on the application and the environment. Because of the nature of RTCI (it being an agent-less solution) we make many connection attempts when moving through the supported protocols. For example, we try to connect to each system using the following protocols by default:

• AMT (Intel vPro Technology)
• ASF (Alert Standard Format)
• SNMP (Simple Network Management Protocol)
• WMI (Windows Management Instrumentation)

Each protocol may take more than one connection per machine. For example AMT uses a number of connections (6 being the standard) to work an AMT power control function. As these connections are made, we begin hitting the max connection limit. When we add in the multiple connections from the other protocols, we easily exceed the limit. Connection attempts made after the limit has been reached are immediately dropped, causing the calls to timeout. As connections to all systems are opened and closed, calls will fail, and not always on the same machines, depending on how long and how many connections are opened at any given time. This accounts for the different results we've seen in retesting the issue.

The reason I've discussed a low number of systems is that Real-Time Console Infrastructure uses a special queue when dealing with all target systems. Each ASP.NET session is limited to 10 or less computers, so if we target 20 computers 2 sessions of 10 are used to execute the Task. This limit was set to avoid problems such as the concurrent connection limit with ASP.NET, but it appears the number we chose is too high.

Resolutions/Fixes

There are two ways to avoid this problem. Both have their merits, but review both to decide which resolution bests fits in your environment. Both fixes focus on the ASP.NET sessions, one increasing the amount of concurrent connections allowed by ASP.NET, while the other limits the number of systems allowed in each queued ASP.NET session during the execution of the task.
NOTE: You can implement both fixes if the success rate is quite low, or if you want to streamline the fix between both implementations.

Limit Computers per ASP.NET Session

This fix is much simpler to implement, but may affect the amount of time a large Task takes to execute. By limiting the number of computers per ASP.NET Session, it will take more sessions to kick off the Task to all targeted systems. The number of concurrent connections is controlled by a configuration file named machine.config. This file, located in several places depending on which Web application you're accessing, can be edited.

To implement this fix, use the following procedure:

1. On the Notification Server, go to Start > Run.
2. Type in regedit and click OK.
3. Browse to the following location in the registry: HKEY_LOCAL_MACHINE > SOFTWARE > Altiris > eXpress > Notification Server > ProductInstallation > {13987439-8929-48d2-aa30-ef4bf0eb26a6}.
4. Right-click on the open space to the right and choose New > DWORD Value.
5. Name the value:
   

   ResourceThreads
   
   

6. Once named, double-click on the new entry.
7. What value you place in here depends on the severity of the issue. If half or more systems fail, try inputting 3, or if less than half, try inputting 5. The default value is 10 when this entry does not exist. See this screenshot for an example:
   
   

   Click to view.

8. Restart IIS by going to Start > Run > type, iisreset > click OK.
9. Restart the Altiris Service using the Services Manager on the Notification Server.

Once everything has reset, when a power management task is initiated via Task Server, the number value you entered into the above registry entry will limit the number of computers per session. Once you've found the magic number each session can reliably handle in your environment, you can schedule how ever many you need to per job.

Increase ASP.NET Concurrent Connection Limit

This fix is recommended for those servers that can handle a larger number of connections efficiently. No benchmarks have been set, but testing a large job with a new limit will reveal whether additional adjustments need to be made. Consider using the previous fix if connection problems occur after increasing the threshold. The following Microsoft KBs were used to create this fix:

• http://support.microsoft.com/?id=821268
• http://msdn.microsoft.com/en-us/library/ms998549.aspx

To implement this fix, use the following procedure:

1. On the Notification Server, browse to the following location on the System drive (typically C:\): Windows\Microsoft.NET\Framework\v1.1.4322\CONFIG\
2. Back up the machine.config file! In case of file corruption, this file should be copied out to a backup location with the file in its current state as a precaution.
3. I recommend using a true editor such as Crimson Editor (www.crimsoneditor.com) for this next step. Locate and open the file Machine.config found in this direction in a true editor.
4. Do a search for the value:
   

   maxconnection
   
   

5. Change the setting to: 12 multiplied by the number of processors on your Notification Server. See this screenshot for an example:
   
   

   Click to view.

6. Save the file.
7. Restart IIS by clicking Start > Run > Type iisreset, click OK.

Once completed, the ASP.NET limit should be increased to allow enough connections for us to use. If you notice IIS performance problems after increasing the limit, try scaling back, or incorporating the previous fix as well to limit how much RTCI hands off to ASP.NET at any given time when the Task is executed.

Conclusion

Once implemented, this should take care of any issue stemming from the ASP.NET concurrent connection limits and the way RTCI queues and executes tasks on targeted systems.