Intel,Altiris Group

Troubleshooting issues with the Intel® AMT setup and configuration process can be a daunting prospect. This series walks through the troubleshooting methods to pinpoint where problems originate and how to fix them.

Part 4: Setup and Configuration Server Troubleshooting

Symptoms
No Update To Intel Amt Systems Node
   Problem
   Cause
   Resolution
No Systems Appearing
   Problem
   Tools:
   Cause
   Resolution
FQDN Not Acquired
   Problem
   Cause
   Resolution
No Systems Configuring
   Problem
   Cause
   Resolution
Properties Script Failed
   Problem
   Cause
   Resolution
   Issue: FQDN
   DNS Lookup
   Out of Band Discovery
Obtaining a Remote Configuration Certificate
   Certificate Authority (CA)
Installing the Certificate
Reinstalling the Certificate
Enabling Remote Configuration
   Other Issues

The server components constitute a lot of 'background' processes that support what is only seen as Symantec Management Console points. Much of what goes on in the background is invisible to the user save as a change in status. If setup correctly, machines simply are setup and configured. It's when they do not get setup and configured that a user should understand the server components so that proper troubleshooting can be accomplished. Note that this covers the symptoms of server-component problems. Some of the symptoms do overlap client-side issues, but in this process we are assuming we've confirmed that the client systems are functioning as expected. If you are unsure, see Part 1.

Symptoms

The following symptoms are seen on the Server. Please note that some of the symptoms may appear to be both client and server related making it difficult to know where the issue lies. Use Part 1 in conjunction with this article if necessary in troubleshooting these issues.

1. No update to Intel AMT Systems Node - At times this node can abruptly appear stagnant with no new systems coming in and no setup and configuration taking place.
2. No Systems Appearing - The Intel AMT Systems node may stay blank even after connecting systems in Setup Mode onto the Network.
3. FQDN Not Acquired - Once the SCS receives a hello message, it needs to acquire the FQDN, and if this fails the machine will remain in an unconfigured state
4. No systems being setup and configured - This can occur where systems show up in the system, but none of them get setup and configured.
5. Properties Script Failed - This is a common error to be covered separately, though many of the above symptoms end up throwing this particular error.
6. Remote Configuration - PKI errors in the AMT Log.

In addition to the symptoms, the following tools were used to troubleshoot the issues to find out which particular issue afflicted the Server:

1. AMT Logs
2. OOB Trace Logging
3. Wireshark

See Part 1 on how to use these. These will be referenced in the following items.

No update to Intel AMT Systems Node

Problem

The typical symptom is an abrupt stop to updates on this node. For example if you have a number of configured systems, with systems added as they are brought up on the network, and abruptly they stop updating or being added, this is indicative of this issue.

Tools:

AMT Logs - No updates to this log occur.

Cause

AMTConfig Service - The AMTConfig service has stopped, crashed, or is in a hung state. This isn't common in version 3.0 of SCS or higher, though it has been seen on rare occasion.

Resolution

Check that the AMTConfig Service is running.

1. Go to Services Manager under Administrative Tools.
2. Check the Service named AMTConfig to make sure it is running.
3. If the service is not running, start it. If the service is running, try restarting it just in case it's in a hung state.
4. Once the service is up and running again (if this is the issue) setup and configuration should start occurring.
5. You can also set the AMFConfig service to automatically restart if it crashes or stops. For intermittent issues this will keep things up and running.

No Systems Appearing

Problem

The symptom is that no machines appear in the Intel AMT Systems list when the page is refreshed over a period of time when new systems are expected. The page ties directly into the IntelAMT database to populate the systems, so if the list isn't updating on the page, the list is also not updating in the database.

Tools:

AMT Logs

1. No entries found
2. No entries found
3. Invalid PID Map error

Wireshark

1. On the client the "Hello" packet is sent, but on the server it never arrives.

Cause

The causes vary. See below for known causes for this issue:

1. AMTConfig Service - The AMTConfig service has stopped, crashed, or is in a hung state. This isn't common in version 3.0 of SCS or higher.
2. "Hello" packets - The routing of "hello" packets is not configured correctly, so clients can't reach the Setup and Configuration Server.
3. PID rejected - The PID provided in the "Hello" packet is not contained as a valid security key in the IntelAMT database. This is only seen in the AMT Log found in the Setup and Configuration Console under Logs, selecting the 'Log' icon.

Resolution

See the steps to follow for the above causes.

1. AMTConfig Service
   1. See the resolution to the section No update to Intel AMT Systems Node.
2. "Hello" Packets
   1. In the Setup and Configuration console go to the DNS Configuration node. Does the 'Test' button allow Provisionserver to resolve back to the IP of the Notification Server?
   2. If yes, go to the segment of the network the client is on and try to ping the name 'Provisionserver'. Does the IP resolve?
   3. If answer to either question above is NO, a CNAME record needs to be created on each DNS Server to route to the IP address of the Notification Server.
3. PID rejected
   1. In the Setup and Configuration Console go to the Security Keys node under the Configuration Service Settings. The list of unused PID and PPS combinations are listed.
   2. In the IntelAMT database, within the csti_pid_map table all used and unused security keys are listed. The ones with a value 'True' in the 'Used' column will not show up in the console.
   3. Either import the keys if the OEM placed the AMT systems in TLS-PSK Setup Mode through the import button in the Security Keys page, or manually enter the PID PPS.

FQDN Not Acquired

Problem

One or more Intel AMT Systems are registering in Intel SCS, but they never show an FQDN and never move out of the 'Unconfigured' status. In the AMT Log often these systems show the error 'Properties Script Failed' (note that the cause of this error can be many, and this issue is but one of them).

NOTE! If no system is configuring, the issue may not be FQDN related. See No Systems Configuring in this section for more information.

Tools:

AMT Logs - Properties Script Failed messages
OOB Trace - Unable to locate FQDN (Fully Qualified Domain Name) entries

Cause

Intel SCS calls the Out of Band Setup and Configuration or Properties script oobprov.exe to do a number of items. The first item it executes is obtain an FQDN for the machine needing setup and configuration. If it fails to obtain an FQDN Setup and Configuration will fail and the computer will remain in an unconfigured state until oobprov.exe can successfully locate the FQDN.

Resolution

To find the FQDN, oobprov.exe runs through a number of checks. The suggested method is to have the Symantec Management Agent installed and have run the OOB Discovery Task (located in the Symantec Management Console under Settings > All Settings > Agents/Plug-ins > Remote Management > Out of Band Management > Out of Band Discovery). This populates the Symantec_CMDB database so it has both an FQDN in the AeX AC Location data class and the UUID in the Inv_OOB_Capability data class. If this data is not available, another option is to check DNS resolution as a method. In the Symantec Management Console look under the Resource Synchronization node, within the Intel AMT Systems folder. As shown below, this option enables oobprov.exe to use DNS IP resolution as a method.

NOTE the warning found directly below the checkbox: Warning! Using DNS for IP to FQDN resolution might lead to incorrect profile mapping. Make sure your DHCP server is configured correctly to give updated and not stale records when the FQDN is sought via the IP Address.

No systems Configuring

Problem

Systems are added regularly to the Intel AMT Systems node, but they never are setup and configured. This includes never getting an FQDN (see the above section for more information), though the cause may not be the inability of oobprove.exe to obtain the FQDN.

Tools:

AMT Logs - Provisioning Script Failed messages
OOB Trace - No references to oobprov.exe

Cause

If it is not the previously covered FQDN mapping issue, this issue stems from a timeout value in the IntelAMT database being set to 0. In the IntelAMT database, in the table csti_configuration, The row with the configuration_name of props_script_timeout should have a valid number, but if the value is 0 IntelSCS will timeout before it even has a chance to call oobprov.exe.

Resolution

The following SQL can update this value if you find it to be set to 0. This updates the specific configuration row that deals with the timeout:

USE IntelAMT
UPDATE csti_configuration
SET configuration_value = 180
WHERE configuration_name = 'props_script_timeout'

Execute the script within SQL Query Analyzer or SQL Enterprise Studio against the Symantec_CMDB_IntelAMT database to update the value.

Properties Script Failed

Problem

This message can mean a number of things, including the symptoms described in the preceding two sections. This message can continually appear into the AMT logs as setup and configuration is attempted over and over.

Cause

The causes of this issue vary. The basic explanation is that when oobprov.exe is called, if it returns anything other than success, the resulting error message in the AMT logs is 'Properties Script Failed'.

Resolution

Referencing the two sections above labeled: No Systems Configuring and FQDN Not Acquired, see this section for direct tips for troubleshooting this common message.

Generic errors are often the hardest to troubleshoot because of the lack of details. With the error 'Properties Script Failed' the Out of Band Setup and Configuration piece (OOBProv.exe) has returned an error when trying to configure an Intel vPro system. The errors returned by OOBProv vary, but the Intel SCS component produces this generic error when configuring isn't successful. The error by OOBProv is not logged by default. See the previous link concerning OOB Trace Logging on how to capture

Issue: FQDN

The most common cause of this error is the inability of the OOBProv process to discover what the FQDN (Fully Qualified Domain Name) is for a given hello packet. The identifier used to link to the FQDN is the UUID. This Universally Unique IDentifier is contained within the hello packet including the IP Address of the system. This IP Address can be used to identify the FQDN via DNS, however this can be a risky method due to the possibility that the IP Address is no longer accurate.

The following methods can be used to rectify this situation so that the FQDN can be obtained by OOBProv.

DNS Lookup

In Out of Band Management, the default install does not try to attempt an FQDN acquisition using the IP Address. This was done being sensitive to the way IP Addresses can be reassigned, especially when a system is going through a build or imaging process. Items such as PXE boots, AMT IP requests, and OS reconfigurations including name changes can change the IP Address assigned to the physical device.

Note: If you are using the Symantec Management Agent in the provisioning and system configuration process, please see the subsequent section entitled Out of Band Discovery for a recommended way of resolving the FQDN. If you are not using the Symantec Management Agent, continue with this section.

The method for DNS lookup to use the reported IP Address in the AMT's hello packet for FQDN resolution is located under the configuration of the Resource Synchronization. In the Symantec Management Console browse under Settings > All Settings > Remote Management > Out of Band Management > Intel AMT Systems > and click on Resource Synchronization. See the screenshot below for details. Check the option entitled:

• Use DNS IP resolution to find FQDN when assigning profiles.

Once this option is checked, OOBProv will use DNS to try and find the IP Address of the system.

Out of Band Discovery

For those systems that are managed within the Symantec Notification Server infrastructure, the best method to allow setup and configuration is to run the Out of Band Discovery Task. This task executes by the Symantec Management Agent and does a check to see if the system is AMT capable. If it is, the following information will be collected and placed within the Symantec_CMDB database relating to AMT:

• _ResourceGuid - Used to tie the data to the specific computer resource in the database
• AMT Capable
• AMT Enabled
• IDE Redirection Enabled
• Serial Over LAN Enabled
• Network Interface Enabled
• UUID

This data correlates by _ResourceGuid to the basic NS Resource tables, namely Inv_AeX_AC_Identification and Inv_AeX_AC_Location. The Location table contains the FQDN captured by the Altiris Agent, and is passed to OOBProv at setup and configuration time.

The Task for Out of Band Discovery is located in the Symantec Management Console under Settings > All Settings > Agents/Plug-ins > Remote Management > Out of Band Management > Out of Band Discovery, as shown in this screenshot:

The default filter accounts for all managed systems that are potentially AMT capable, and is labeled:

• All Potentially ASF or Intel® AMT Capable Computers

This policy executes the file located at: C:\Program Files\Altiris\Notification Server\NSCap\Bin\Win32\X86\Out of Band Management\Discovery (path may vary if Notification Server was installed on a different drive or to a different path). The files in use are:

• ClientUtility.exe
• ClientUtility64.exe
• BMAPI.dll
• BMAPIa.dll

The command line used for Discovery is:

ClientUtility.exe -mode:discovery

Noted in the preceding screenshot the policy is set to: 'Run once ASAP'. This means each system will only ever run the Out of Band Discovery Task once. This also means any new computers receiving the task will also execute it once. This should be sufficient for most environments, but if another discovery needs to be run, a schedule can also be set in additional to the ASAP option. Use the Add Schedule option.

Obtaining a Remote Configuration Certificate

First, the following article covers how to properly obtain a certificate:

• Obtaining and Applying a VeriSign Remote Configuration Certificate

Note that part of obtaining a Remote Configuration is submitting the request from the Server you plan to install the certificate onto. This process creates the private key for the server-side certificate, and this item will not be available until partway through the application of the crt (or cer) file obtained from the vendor. The specific step that provides the full key, both private and public, is when the certificate is exported into a PFX format after the initial import, checking the option to export the private key will give you a complete backup of the full certificate in case it is needed in the future. If something happens, or if the application doesn't go right, we'll need both, so it's essential to export this as soon as possible.

During the steps to install the certificate emphasis will be given on the step where the export should take place.

Certificate Authority (CA)

In order to use Remote Configuration with Out of Band Management the Microsoft Certificate Authority services must be installed on the Notification Server or the OOB Site Server. Use the following steps to install if it is not installed:

1. Go to Start > Administrative Tools > and click on Add or Remove Programs.
2. In the left-side button bar click the button Add/Remove Windows Components.
3. Check the option labeled Certificate Services. See this screenshot for details:
   

   

4. You'll receive the pop-up:
   

   After installation Certificate Services, the machine name and domain membership may not be changed due to the binding of the machine name to CA information stored in the Active Directory. Changing the machine name or domain membership would invalidate the certificates issues from the CA. Please ensure the proper machine name and domain membership are configured before installing Certificate Services. Do you want to continue?

5. Click Yes to continue once your system has the intended identity. Click Next.
6. Choose what type of CA to create. If you are not installing a hierarchy of CAs you can leave the stand-alone root CA option selected. Click Next.
7. Input the name the CA will be known by. This must match what is in the hierarchy or by what the Remote Configuration certificate name will be known by.
8. The Distinguished Name is generated automatically in an AD Environment and will be the suffix of the system.
9. Click through the rest of the options, noting where the services data files are stored.
10. You will be prompted to restart IIS. This is required during the installation.
11. Click Finish to complete the installation.
12. Done! The NS or Site Server is now prepared to handle certificates in the Remote Configuration process.

Installing the Certificate

The recommended application for a Remote Configuration certificate is to let the certificate dictate where to be installed. However this process has sometimes resulted with the certificate installed to an incorrect place. When this occurred we've had headaches trying to clean up the system to properly install the certificate. Why this occurs is unclear. For reference I'm including the process of adding a certificate automatically here:

1. Save the acquired cer or crt file from the vendor onto the Notification Server or the Site Server for Out of Band Management.
2. Right-click on the file and choose Install Certificate.
3. Click next on the Welcome screen.
4. Leave the radial option on 'Automatically select the certificate store based on the type of certificate' and click Next.
5. Click Finish to complete the installation. You'll receive a confirmation pop-up that the certificate installed successfully.

While I won't advise against using this method, the below steps uses the manual installation method to ensure the certificate is installed to the correct place.

NOTE: You must install the certificate while logged onto the server as the Notification Server Account, the Application Identity, or the account used to install Notification Server and Out of Band Management.

I've condensed the steps required into the following list. This process works for all vendors once you've obtained a certificate. Note that these steps are provided to consolidate both recommended steps and documentation into one whole.

1. Go to Start > Run > type mmc > and click OK.
2. In the resulting console click under File and choose Add/Remove Snap-ins...
3. Near the bottom of the resulting window click the Add button.
4. From the list that appears select Certificates and then click the Add button.
5. Leave the radial button selected on 'My user account' and click Finish.
6. From the same list select Certificates again and click the Add button.
7. From the resulting window change the radial select to 'Computer account' and click Next.
8. Leave the selection at 'Local computer: (the computer this console is running on) and click Finish.
9. Click the Close button in the window offering you the list of available snap-ins.
10. At the original add/remove snap-in screen verify that you have two entries:
    1. Certificates - Current User
    2. Certificates (Local Computer)
11. Click OK.
12. Expand both trees in the left-hand pane within the console. You should see the full certificate stores as shown in this screenshot:
    

    

13. Right-click on the Personal folder under the Current User certificate store and highlight 'All Tasks' and click on 'Import' in the pop-out menu.
14. Click Next on the Welcome page of the Certificate Import Wizard and click the Browse button.
15. Browse to the cer or crt file provided by the vendor, highlight it, and click Open.
16. Click Next, and leave the radial option on 'Place all certificates in the following store', which should be set to 'Personal'. Click Next.
17. Under the Completing section of the wizard, Click Finish. You should receive a pop-up indicating the certificate was successfully installed.
18. NOTE! This is the vital step mentioned previously in the article. We will now export the certificate with both public and private keys, which will give us the full set and allow us to remove and reapply if necessary. In the MMC select the newly imported certificate > right-click > and choose All Tasks > Export...
19. Click Next on the Welcome screen. In the resulting list you should have an active option for 'Personal Information Exchange - PKCS #12 (.PFX)'. If this option is not available (grayed out as shown in this screenshot), there is a problem with the certificate and the private key is not accessible:
    

    

    If this occurs please note the following items:

    1. The application of the public key, or cer/crt file, must be done on the server where the key was requested.
    2. If this is not your Setup and Configuration Server you'll need to contact the Vendor of the certificate to resolve the discrepancy.
    3. If you did request this certificate from the server you are operating on, you'll also need to contact the vendor to explain that the private key is not found when exporting the certificate after initial application.
20. Follow the wizard, and ensure you select the option 'Yes, export the private key'. When saving the file, it will prompt you to set a password to protect the private key (this is recommended for security reasons). The export should leave you a PFX file. Keep this in a safe place, preferably in line with your company's encryption certificate backup policy.
21. Next we need to import the full key into the Computer store. Start back in the MMC > under the Local Computer certificate store > right-click on the Personal folder > select All Tasks > Import...
22. Click Next on the Welcome screen and click the Browse button on the subsequent screen.
23. Browse to the newly exported PFX file. Note that you will need to change the 'Files of type' to include the PFX format. Click Next.
24. The Password screen prompts for the password you set when you exported the key in step #20, as shown in the following screenshot. Enter the password and click Next.
    

    

25. Choose or leave the select to 'Place all certificates in the following store'. The value should be Personal. Click Next.
26. Click Finish on the end details page to complete the import.
27. Done!

NOTE: In Out of Band Management 6.x, with Intel SCS 3.x or earlier, a separate utility was required to load certificates into Intel SCS so the Setup and Configuration Server was aware of them. This is no longer required as Intel SCS 5.x possesses intelligence to automatically acquire all installed Intel vPro Remote Configuration encryption certificates.

Reinstalling the Certificate

If you need to reinstall the certificate and you have a PFX file, you can do so by opening both certificate stores (User and Local Computer) as outlined in the previous steps. Browse through the certificate stores and delete any instance of the vendor certificate. This will remove any associations and allow a clean application of the certificate to occur. Look for the following:

• The name matching the name of the cer or crt file obtained from the vendor
• The vendor's certificate (the entry will contain the vendor name).

NOTE: Be careful when removing vendor certificates as they may not be part of the Remote Configuration. The best example is Verisign, which may have many entries. If unsure, leave the certificate in place, or export it before deleting it so you can restore it if necessary.

Enabling Remote Configuration

To ensure that Out of Band Management is setup to use Remote Configuration as a valid setup and configuration method, follow these steps:

1. In the Symantec Management Console browse under Home > Remote Management > and click on Out of Band Management.
2. In the left-hand tree browse under Configuration > Configuration Service Settings > and select General.
3. In the resulting page ensure that the option labeled Allow Remote Configuration is checked. If it is not, check it. See this screenshot for an example:
   

   

4. If you needed to check the option, be sure to click Save Changes to register the change.

That should do it for the certificates. You've now completed the steps required to install and enable Remote Configuration in the Out of Band Management Environment. However you are not done yet! Certain infrastructure components are required to make this process seamless. Proceed to the next section for details.

Other Issues

When discovering the issue using the trace log, the issue may not be caused by the inability to find the FQDN of the system. In these cases the logs should provide details on what's causing the error. Search the Symantec Knowledgebase, found here:

• http://www.symantec.com/business/support/index?page=home

Use the search in a manner that will constrain the results to Out of Band Management Component. This screenshot demonstrates this:

It's possible the trace logs will not reveal where the problem is coming from. If this is the case, we must rely on the AMT Logs to see if the issue is detailed by Intel SCS directly. Look for errors surrounding the 'Properties Script failed' entries to see if they are paired with another error message. Also check the Action Log to see what actions were attempted when Setup and Configuration failed.

Return to Part 1/Index

Read Part 5: Supporting Components