Patch Management Best Practices

Article:HOWTO3124  |  Created: 2006-04-10  |  Updated: 2010-10-16  |  Article URL http://www.symantec.com/docs/HOWTO3124
Article Type
How To



Question
Are there any Patch Management Solution Best Practices?

Answer

Patch Management Solution Best Practices

Contents

Overview

In pursuit of increased worker productivity, a variety of IT devices, operating systems, and applications entered organizations during the boom years of the 1990's, many IT groups now support a range of older, proprietary, packaged, and leading-edge investments. The resulting portfolio of systems will therefore vary in vintage, configurations, and characteristics.

Patch Management is a practice designed to proactively prevent the exploitation of vulnerabilities on IT devices. The expected result is to reduce the time and money spent dealing with vulnerabilities and their exploitation. Taking a proactive approach to patch management will reduce or eliminate the potential for exploitation and involve considerably less time and effort than responding after vulnerability has been exploited.

This document details industry best practices for patch management and the Altiris® Patch Management Solution™ best practices as observed by Altiris professional services and technical support. Where applicable, the frequently seen requirements of various industries and customer environments are provided.

The first portion of this document is geared towards individuals responsible for setting IT policies and/or architecting Patch Management Solution. The latter half of the document drills deeper into the aspects of system administration and provides advanced administrative techniques.

The Patch Management Solution Process

At a high level, the Patch Management Solution process works as follows:

  1. The necessary Altiris agents are deployed on each managed node.
  2. On a repeating schedule, the agents gather and upload current vulnerability data to the central server.
  3. On a scheduled interval, the central server retrieves updated patch management data from Altiris.
  4. Administrator enables a “patch bulletin” for staging on the central server.
  5. The NS retrieves the related patch installation files form the Vendor’s web site
  1. The central server advertises the patch to the managed nodes through intermediary package distribution servers.
  2. The managed nodes check in with the central server and download the patch from the appropriate location.
  3. The agent installs the patches at a predetermined time and reboots the node if necessary.
  4. On a repeating schedule, the agents gather and upload current vulnerability data to the central server.
  1. Web based reports reflect the current patch compliance levels and rollout status.

Process design recommendations

Defining Service Levels

When defining service levels, it is first necessary to understand that not all patches are created to resolve security vulnerabilities and that the severity of the vulnerability will vary. Service levels should be defined as a combination of a time frame based upon Severity Level to reach the minimum compliance level. For example:

  • The Service Level for remediation of applicable Critical severity vulnerabilities is 3 business days at 87 percent compliance.

Microsoft’s Severity Rating System

Because this white paper focuses on the Altiris Patch Management Solution for Windows, Microsoft’s vulnerability rating definition is provided below. Microsoft’s severity rating system provides a single rating for each vulnerability. The definitions of the ratings are:

Rating

Definition

Critical

A vulnerability whose exploitation could allow the propagation of an Internet worm without user action.

Important

A vulnerability whose exploitation could result in compromise of the confidentiality, integrity, or availability of user’s data, or of the integrity or availability of processing resources.

Moderate

Exploitability is mitigated to a significant degree by factors such as default configuration, auditing, or difficulty of exploitation.

Low

A vulnerability whose exploitation is extremely difficult, or whose impact is minimal.

(Reference: http://www.microsoft.com/technet/security/bulletin/rating.mspx .)

Note: Microsoft bases this rating upon atypical usage of their products and the level of user interaction required to trigger the exposure; this rating should be weighted against the usage patterns of the organization. For example, if the vulnerability only exists in MS Outlook, and the organization is a Lotus Notes shop, then the risk of exposure is reduced for that particular vulnerability.

Critical vulnerabilities that have published exploit code should be given the highest severity weighting. Additionally, most organizations will accelerate the distribution of these patches.

Most organizations with an automated patch distribution mechanism establish a short time frame (the unofficial average is 48 hours to 1 week) for the testing and distribution of critical patches. Patches that resolve Important to Moderate severity vulnerabilities are commonly given additional time for testing and deployment.

The management of Low severity vulnerabilities varies frequently from company to company. Some organizations do not track Low severity items, while others treat them as equivalent to Moderate severity for purposes of testing and deployment. The tradeoff is between the minor increase in risk and the potential loss of software functionality.

Compliance Levels

A compliance level refers to the percentage of computer devices that have been successfully patched or otherwise remediated such that they are no longer vulnerable. Setting a reasonable goal for compliance levels is often a difficult concept. At first glance, a completely patched environment (100%) would appear to be a realistic goal.

However, this does not take into account the reality of:

  • Users that own multiple computers (which aren’t always connected and powered on).
  • Traveling employees with notebook computers that don’t log into the corporate VPN every night.
  • Computers being repaired or replaced by a hardware vendor.
  • Contractor’s computers that may or may not even still exist on the corporate network, yet still show up on recent network inventory reports.
  • Failed or over saturated WAN connections.
  • Computers registered in Active Directory that have been repurposed and/or re-imaged.
  • Virtual computer sessions that remain powered off for weeks or months at a time.

The most commonly observed compliance levels are between 85 and 97 percent. Some organizations go the additional step of setting a multi-phased compliance goal, such as 91 percent within 72 hours, 97 percent within four weeks, and 99 percent within 6 months.

Common Organizational Roles

The assignment of responsibility for patch management will vary, depending upon the size of the organization, its network environment, and the type of industry. The following are the most common roles for patch management activities.

  • Network Security Administrator
    • This role is responsible for the tracking of newly discovered vulnerabilities and the associated patch compliance, as they apply to the network and computing devices in the organization. This role is also responsible for defining and publishing the organization’s patch distribution policies, disaster recovery guidelines, and target service levels. This role often reports to the CIO. In very large organizations, this individual may hold the title of CSO (Chief Security Officer) and report to the CEO.
    • This role may not officially exist in smaller organizations. This role is usually found in medium-sized or larger organizations, and/or organizations managing sensitive data such as those in the defense, financial, pharmaceutical, or data-processing industries. Because of requirements from the Sarbanes-Oxley Act, it can be argued that all publicly held companies should staff this role. Some organizations opt to establish a small committee to fulfill this role.
    • Altiris-related activities include enabling patch bulletins (but not configuring the bulletin’s distribution), reviewing patch compliance reports, and following up with Regional IT Managers when compliance is outside the established boundaries.
  • Change Management committee member
    • Members of this role are responsible for approving the scheduled rollout of patches, as they pertain to changes made to the production network environment.
    • This role does not exist in all organizations but shares many of the same existence criteria as the Security Administrator role.
    • Direct Altiris activities do not exist. Altiris-related information is communicated by the Security Administrator, Regional IT managers, and/or the IT administrators.
  • Patch testing team
    • Members of this role are responsible for the testing of patches against the most commonly used workstation and server configurations in the environment. This role is also responsible for testing each patch’s compatibility with business critical applications.
    • This role should exist in all organizations, but frequently (and unfortunately) does not.
    • Altiris-related activities include distributing patches to test lab workstations and servers, observing the results, and checking for post-patch functionality. The Altiris Wise Package Studio™ can be used to perform virtual patch testing against known operating system configurations and software packages to expedite the testing process.
  • IT administrators
    • Members of this role are responsible for the patch testing, installation, and restart management of their assigned workstations/servers.
    • This role will exist in all organizations.
    • Altiris-related activities include creating software update policies for each enabled bulletin, monitoring the patch rollout, and troubleshooting of failed patch applications.
  • Regional IT manager
    • Members of this role are responsible for ensuring that their region’s IT administrators are adhering to the established policies for testing and distribution of patches.
    • This role will exist in all organizations.
    • Altiris-related activities include reviewing patch compliance reports and following up with IT administrators when compliance is outside the established boundaries.
  • Altiris Notification Server administrator
    • This role is responsible for the health of the Altiris infrastructure (Notification Servers, Package Servers, and any related custom file storage solutions).
    • In smaller organizations, the individual filling this role will often also fill the role of Security Administrator.
    • Altiris-related activities include reviewing Notification Server health reports, monitoring the health of the Notification Server Infrastructure, ensuring that the PMIMPORT.cab file was properly downloaded and configuring Notification Server security to grant appropriate Altiris Console access to the other roles.

Testing process

The patch testing process is composed of two stages: Preparation and Testing

Preparation

The goal of the preparation phase is to:

  • Understand a patch’s impact on the environment before deployment.
  • Identify possible file version conflicts between the contents of a given patch and pre-existing software in the environment.
  • Identify which applications might be impacted because of their dependence upon updated system-level files.

Having this knowledge will greatly reduce the number of test cases to be performed before a patch is considered to be safe for deployment.It will also help to define the current vulnerability of the environment.

The Wise Package Studio Impact and Risk Assessment tool identifies when a patch will overwrite dependent system files used by other packages.

Next, the Wise Package Studio ConflictManager® can identify situations in which a patch will overwrite the files and/or registry keys associated with another package. It can also identify packages that depend upon a system-level file, and where a patch will not update all the potential copies of a given file (such as a private copy of a .DLL file). The tool can then be used to repackage software to avoid reintroducing the vulnerability in a future software installation.

Testing

Test cases are the documentation of the procedures, targets, and expected results for each individual test to be performed. When building a list of test cases for a patch, include each of the specified test-case types.

Types of tests cases

  • Installation Tests—To validate that the patch installs without error, and that any launch conditions contained in Windows Installer patches are working properly.
  • Verification Tests—To verify that shortcuts, help files, and file associations set or modified by the patch are working properly.
  • Execution Tests—To verify whether the files and registry keys created or modified by the patch can be read and updated when the application is executed by typical users who do not have administrator-level privileges.
  • Standard Tests—To verify that the installation of a patch does not negatively impact the ability to execute another application found on the desktop or the ability to connect to a URL, network share, or database.
  • Rollback Tests—To verify the safe method of uninstalling the patch and/or restoring the target computer to the pre-patch state in the event of a conflict.
Lab Testing

Initial patch testing should always be performed on non-production computers. To facilitate adequate testing of automatically deployed patches, organizations should use standardized configurations for IT devices as much as possible. A popular technique is to leverage virtual machine technology that provides a rollback capability to facilitate rapid testing and retesting against common workstation and server configurations.

Primary testing candidates are those configurations that exist in the production environment, such as:

  • Each workstation OS per service pack level (Windows 98, Windows 2000 SP4, Windows XP gold, Windows XP SP2, and so on)
    • Each of the above with core workstation applications installed (MS Office, WinZip, antivirus client, Acrobat, and so on)
    • Each of the above with any custom/proprietary applications in the environment
  • Each Server OS per service pack level (Windows 2000 Server SP4, Windows 2003, Windows 20003 SP1, and so on)
    • Each of the above with core server applications (IIS, primary Application Server software, Antivirus client)
    • Each of the above with any custom/proprietary applications in the environment

More focused testing should be performed on replicas of business critical servers (customer facing servers, HR and ERP applications, mail servers, database servers, and so on)

Pilot Testing

Roll out the patches to one or more pilot groups. As previously discussed, pilot groups should focus on commonly used configurations in the production environment.

Good pilot groups are those that are readily accessible by IT staff in the event of a failure, computers assigned to the more technically savvy users, and computers that are the least likely to impact the primary business functions of the organization in the event of failure.

After comparing the differences between the pilot testing and the earlier testing phases, develop a rollout plan that addresses any observed issues. The recommended technique of a multi-phased rollout is discussed under the Software Update rollout strategies section of this document.

Disaster Recovery planning

In the context of patch management, disaster recovery is limited to resolving problems caused by the installation of a patch. This section provides the most common techniques. As previously discussed, part of proper patch testing includes verifying that a patch can be safely removed, and having a viable fallback plan in the event the patch can’t be correctly uninstalled.

Traditional desktop recovery

  1. Uninstall the patch through Add or Remove Programs—this method is not scalable for widespread problems.
  2. Uninstall the patch through a command line—this method can be replicated to all affected computers through Altiris® Software Delivery Solution™ or Altiris® Deployment Solution™.
  3. Restore the desktop to an official corporate image—this is a much more drastic method and will cause productivity losses. Prior to re-imaging, consider using Altiris® PC Transplant Solution™ to capture and restore the user’s application settings and data to the newly imaged computer.

Traditional server recovery

  1. Backup the server prior to deploying the patch.
  2. Uninstall the patch through Add or Remove Programs—this method is not scalable for widespread problems.
  3. Uninstall the patch through a command line.
  4. Restore the server from a backup.

Desktop recovery process with Altiris® Recovery Solution™

  1. Enable the Recovery Solution pre-patch snapshot option.
  2. Uninstall the patch through Add or Remove Programs—this method is not scalable for widespread problems.
  3. Uninstall the patch through a command line.
  4. Initiate a Recovery Solution rollback to the snapshot.

Future: Altiris® Software Virtualization Solution™ (SVS)

A future version of Software Virtualization Solution will provide the ability to implement patches as SVS layers. Removing the problem patch will be a simple step of disabling the associated layer for affected computers through the Altiris Console.

Software Update rollout strategies

A successful rollout strategy should include multiple phases. A phased approach provides checkpoints at which problems can be identified and constrained. Below is the foundation for any desktop rollout strategy.

Targeting Technique

Collections are the key to this technique.

  • Create an empty master collection for each new bulletin.
  • Add collection inclusions to the master collection during each phase.
    • Provides granular, repeatable rollout control.

Phase 0—Smoke testing (2–6 hours)
  • Only target personal test computers.
  • Validate proper command line parameters for the patches.
    • Patch installed correctly.
    • Proper level of user interaction.
      • Completely silent?
      • DOS window flashing open and then closed?
      • User response required?
    • Reboot required?
  • Check for blue screens (fatal OS errors).
  • Check for obvious performance issues.
  • Check for dependency prompting (such as MS Office source files).
  • Core applications still execute as expected (MS Word, WinZip, IE, and so on).
  • Intranet portal accessible?
  • Schedule policy should be ASAP.
Phase 1—Pilot groups (2–5% of total environment) (24–72 hours)
  • Should be easily recoverable computers.
  • Target users that are more tolerant of IT related problems.
    • IT staff
    • Non-business critical computers.
  • Some customers will also target the secondary computers assigned to application owners.
  • Other customers maintain a pilot lab in which each primary business application is installed. Application owners are then responsible for approving the patch(es) after performing unit and integration testing.
  • Schedule policy should be ASAP.
Phase 2—Majority rollout (40–75% of total environment) (1–14 days)
  • Breakout the rollout targets by utilizing collections representing major sections of the environment, such as:
    • By AD organizational unit (OU).
    • By type/purpose of computer.
    • By business unit.
      • In this case, some customers assign responsibility to business unit leaders for managing which computers are in their respective collections.
      • This approach also allows each business unit to select a default installation time and rebooting behavior.
  • Schedule the patch to install several hours in the future to allow sufficient time for it to be cached on the clients.
  • Schedule should match what makes sense in your environment.
    • Do users leave computers on the network at night?
      • If not, then you should probably schedule installation during working hours or enable Wake on LAN functionality.
    • Do you have 24/7 operations?
      • Establish a maintenance window.
      • See the Rebooting section below for dealing with shared, multiple-shift computers.
Phase 3—Final rollout (Remainder of environment)

Invariably, each customer will have computers that require special attention during Patch Management operations. In many of these cases, the impact of a patch-related fault is considered a greater risk than that of the vulnerability being exploited. After observing the results of the prior phases, a determination can be made about automating the patch rollout, manually installing the patch, removing the vulnerable applications, configuring a personal firewall, or removing the at-risk computer from the network.

Sample computers include:

  • Factory floor automation.
  • Vendor maintained systems.
  • Traveling users with VPN access.
  • Financial staff computers during end-of-quarter or end-of-year operations.

Rebooting

Patch Management Solution provides a wide variety of options for reboot scheduling. This section discusses commonly used configurations and the thought process behind the configuration. It is also important to note that this section only refers to the default reboot policy and may be overridden as needed.

Altiris provides a very flexible method of setting reboot policies by utilizing collections of computers. If desired, the Patch Management Agent settings policy can be cloned, and a different collection can be assigned to each copy of the policy.

Using Windows Installer 3.x

The number of reboots required to install patches will be significantly reduced for computers that have version 3 of the Windows Installer installed. All workstations with Windows XP SP2 already have version 3 installed. All other current Windows operating systems require a standalone package to be executed.

Other benefits for patching include:

  • You can double-click a patch file to apply it.
  • Patches are smaller and more reliable.
  • Delta compression patches no longer require the source media.
  • You can install multiple patches in one transaction.
  • You can install patches that are targeted to different products in a single transaction.

The package can be found at:

http://www.microsoft.com/downloads/details.aspx?FamilyId=5FBC5470-B259-4733-A914-A956122E08E8&displaylang=en

More details about Windows Installer v3 can be found at:

http://support.microsoft.com/?kbid=884016

Because of the new functionality, Altiris recommends upgrading existing computers to Windows Installer v3.

Reboot Options

The following basic reboot options are available

  • Do not reboot after installing patches.
  • Reboot immediately after installing patches.
    • And if necessary, you may allow multiple reboots during the patch window (this is rarely needed with MS patches released in the last two years).
  • Reboot on a schedule (This is only triggered if a patch was applied since the last reboot).
    • Daily, Weekly, multiple times, and so on.
    • Reboot at the next login.
  • Reboot at system startup regardless of user logon.
    • Custom technique, not present in the UI.
    • Useful for multiple shifts with shared desktops such as call center operations.
      • See the Advanced Techniques section of this document for sample code.

Reboot Notification

The following notification options are available:

  • Never
  • Once (after applying the patch).
  • On a schedule (with repeating warning messages if desired).
  • Warn the user about a pending reboot in X time.
  • User can defer reboot for X time.
  • Custom notification messages—We recommend that customers consider modifying the default reboot messages to remind users about adherence to their IT policy on patch installation and published patch reboot schedules.


 

Workstations

Traditional

For computers in a traditional 9–5 working hour environment, a common approach is:

  • Reboot on a schedule—5 a.m. every day
  • After installation, don’t notify the user that a reboot is required.
  • Scheduled reboot after hours, allow user to defer for 30 minutes.

     
Legacy Windows SUS

Customers that have used Windows SUS in the past will often model Altiris reboot options to match the prior user experience. This translates to:

  • Do not reboot after installation.
  • After installation, notify the user that a reboot is required every 30 minutes.

Servers

Traditional Enterprise Servers

These computers provide services to users, primarily during traditional business hours. Patching and rebooting is scheduled to occur during non-peak production hours. An enterprise-level policy can be utilized for these computers

    • Reboot on a schedule – 4 a.m. every day.
      • Alternative: Reboot on a schedule – 3 p.m. Saturdays.
    • After installation, never notify the user that a reboot is required.
    • Schedule after backup jobs are completed (11 p.m. – 3 a.m.).
      • Goal is to apply the patch and reboot after performing a backup
Local Application Servers

These computers provide services to one or more business units and have maintenance windows that suit the needs of a particular group of users. Collections and a software update agent policy should be created for each grouping of application servers.

    • Apply patches at the appropriate time.
    • Reboot ASAP after patching.
    • After installation, never notify the user that a reboot is required.


 

Business Critical Servers

These computers will often have tight change control rules and no regularly scheduled maintenance windows. To meet these needs, some customers use the following technique:

  • Schedule patches to be applied in 10 years. This will provide local caching of the patch but will not trigger the installation. Disable rebooting, or also schedule to occur in 10 years.
  • During the maintenance window, the administrator manually triggers the patch with the following steps:
    • Open the Altiris Agent on the target server
      • For servers with high uptimes (>21 days), consider manually rebooting before applying the patch.
    • Select the Software Update tab
    • Click the Start Software Update button
    • After installation, the administrator reboots the server

Reporting

The focal point of Microsoft patch management reporting is theMicrosoft Compliance by Bulletinreport.

By right-clicking on a row, the compliance report provides a drill down to view the affected computers.

View Applicable Computers returns the computers that are relevant to the bulletin. It is purely based upon the perquisites for the patch.

View Installed Computers provides a list of computers that currently have the patch installed. This does not indicate that Altiris installed the patch, just its presence.

View Vulnerable Computers provides a list of computers that don’t have the patch installed, as of the last time they posted their patch inventory data. To analyze these results, see the Patch Remediation section of this document.

Remediation

In the context of patch management, remediation refers to the steps taken when a fault occurs as result of the patch process.

Patch Installation Causes Application or Operating System Failure

In this situation, the patch must be removed until the application or hardware vendor can supply a corresponding fix. To prevent re-installation of the patch, exclusion for the affected computers should be added to the corresponding collection for the patch policy.

Patch Fails to Install

In this situation, the patch was scheduled to be installed, but it still isn’t present on one or more computers after sufficient time has elapsed.

Symptoms
  • Patch not listed in Add or Remove programs (not a consistent detection method).
  • Computer continues to show up on the Compliance by Bulletin report as being out of compliance.
  • Secondary vulnerability scanner indicates patch is not installed.
 
Diagnostic techniques
  • Review the software update summary data by opening the Resource Manager on the problem computer > Summarytab > Software Update Summary> Return Codecolumn for the corresponding software delivery task.
    • Lookup the returned error code.
      • Google for “msi error code 12345”.
      • Google for “error code 12345”.
    • Review the TechNet article for the bulletin.

 

  • Manually execute the patch (without switches) on a problem computer.
    • Identify and resolve the computer-specific problem. Replicate the fix through Software Delivery Solution or through a Deployment Server job to other affected computers
    • Some patches trigger the application to prompt for MSI source files—invalid MSI source paths for MS Office installations can be fixed through a registry key change. See Appendix A : Office source MSI registry keys.
    • Check for any running instances of msiexec.exe, setup.exe, or install.exe. This is a strong indication of a hung installation job from another activity.
    • A prior package installation may have set the Windows installer reboot flag. Additional production installations will refuse to execute until the computer has been rebooted.
    • On some occasions, the manual installation process will succeed without providing an obvious reason for the previous failures. This is generally caused by the patch requiring the target application to be closed prior to installation. Because of this situation, the default mode for the software update agent is to retry installation three times.
      • In some environments, such as those where users infrequently close MS office applications, it may be necessary to increase the number of attempts to four or five.

Patch inventory rules

In some situations, it is necessary to review the inventory rules utilized by the software update agent. Inventory rules can be used to collect information about the operating system, installed hardware, current applications, and installed patches. To expose the inventory rules, the following steps can be performed if using Patch Management Solution 6.1

These steps illustrate the process of viewing the inventory rules associated with Microsoft bulletin MS06-001. The key to this process is to alternate between Resource Manager and the Associations Tab as you drill down through the bulletin’s child items.


 

  1. On the Notification Server virtual machine, open the Altiris Console.
  2. Click the Tasktab.
  3. In the left pane, select Software Management> Patch Management> Manage Software Updates.
  4. Right-click on MS06-001 and select Resource Manager.

 

 

  1. In the Resource Manager window, click the Associationstab.
  2. Select the Association Type Name that corresponds to the computer’s operating system in question, such as Windows Server 2003.
  3. Right-click on the selected row from the previous step and select Resource Manager.

 

 

  1. In the new window, click the Associations tab.

  2. Right-click on the Resource Is Installed row, and select Resource Manager.

Note: The Resource is Applicablerule is also worth reviewing.

 

 

  1. In the new window, click the Inventorytab.

  2. In the left pane, open the Inventory Rule Management folder and click on the inventory rule.

  3. On the right pane, click the InventoryRuleXMLand click the Copy icon in the toolbar.

 

 

  1. Open Notepad and paste the inventory rule into it.

  2. At the beginning of the file, delete everything before the <detection>tag.

  3. Save this to a file called MS06-001.xml and then open it in Internet Explorer or another XML capable viewer.

 

 

The topmost <or>tag indicates that one of the child expressions must be true in order for the agent to report that the patch is installed.

 

Arrow 1 refers to a registry key that must be present, and has a specific value of 1.

 

Arrow set 2 identifies an alternative combination of a registry key that should not be present and file that should exist and be a minimum version of 5.2.3790.462

 

Arrow set 3 provides the last alternative combination of registry key that should be present and a file that should be present and have a minimum version of 5.2.3790.2606

Configuration Recommendations

Disable unnecessary languages

This will reduce:

  • Inventory Rule cache size (aka the IAD file) on the client.
  • Defragmentation time for the IAD file at startup.
    • Note:Defragmentation schedule and trigger level will be controllable in Patch Management Solution 6.2.
  • Processing load on the Notification Server for each node.
  • Number of patch-related packages (disk space and network traffic).

Use appropriate scanning intervals
  • In production, don’t schedule scans more frequently than every 2 hours.
    • Scales best at 4–6 hour intervals.
    • Tip: OS service pack data is very static. Set “Default Windows OS Inventory policy” to scan once per day.
  • In lab environments, don’t scan more frequently than 10–15 minutes.
    • The scan is approximately 9 minutes on a typical computer.
Use the default settings for reapplying patches
  • Don’t enable “Reapply all scheduled software updates” on Post Windows 2000 SP2 computers.
  • Changes where made to the Windows platform (upon release of Windows 2000 SP2) to avoid a problem with installation of older patches being applied after newer patches. Therefore, this option is only needed on Windows platforms running Windows 9x and less than Windows 2000 SP2

(Screen shot represents correct setting)


 

Only send changed patch inventory data
  • Don’t leave “Only report inventory if changed” unchecked on a permanent basis.
  • The setting “Only report inventory if changed” is provided as a temporary troubleshooting aid.
  • The most common need for this option is in the event the Notification Server is rebuilt and a discrepancy exists between the inventory data the agent believes exists on the Notification Server, and what actually exists on the replacement Notification Server.

(Screen shot represents correct setting)

Patch Management Import cab file options
  • Don’t leave “Only download if modified” unchecked on a permanent basis.
  • This is another troubleshooting aid, in the event the Notification Server believes it has a full set of data from the current version of the PMImport.cab file, but some external action has removed portions of the Patch Management Resource data.
  • Each time this event occurs it will cause the Notification Server to spend approximately 45 minutes processing the results of the cab file, and merging them into the SQL database.

(Screen shot represents correct setting)

Considerations for large environments

Rolling out the Software Update Agent.

    • Initially, the Software Update Agent will need to retrieve about 2 MB of compressed inventory rule data from the Notification Server through a web-service request to the Notification Server.
  • Frequent area of confusion: The inventory rule data is currently 7+ MB per language uncompressed, but compression is at a 1:10 ratio. The actual amount of network traffic is much lower due to compression.
    • Multiply times 10,000 nodes = heavy load on the NOTIFICATION SERVER during the initial rollout (and rebuilds of the NOTIFICATION SERVER).
    • Same concern exists for upgrades of the Software Update Agent from version 6.0 to 6.1.
  • Solution: Use the master collection technique to stagger the rollout over time.

Advanced techniques

Forcing a reboot to occur at next logon

Any entry under the “RunOnce” registry key will be executed at the beginning of the next user logon session. By utilizing Altiris Software Delivery Solution, a package to inject a reboot command into this key can be delivered after rolling out a patch.

Place an entry in the HKLM\Software\Microsoft\Windows\runonce registry to call the shutdown command.

A sample registry merge file for rebooting a Windows 2003 server is listed below. There are many more commands available for the shutdown command, including messages presented to the end user, and the length of time before the reboot occurs. The /r switch is used to reboot after shutting down. The /d is used to describe the reason for the shutdown. In this situation, it was a planned outage (/P), with reason code of 0:0. Also please note, the extra quotation marks at the end of the line are not a typo.

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\RunOnce]
"RebootAtLogon"="cmd /c \"shutdown /r /d P:0:0\""

By default, the user will see this dialog box for 30 seconds as it counts down.

 

Not all Windows versions support the shutdown command. A more flexible alternative for all Windows NT based platforms is to utilize Sysinternal’s PsShutdown utility. PsShutdown is available for download at: http://www.sysinternals.com/Utilities/PsShutdown.html

  1. Create a package containing psshutdown.
  2. Create a RunOnce registry merge file that calls psshutdown with the appropriate parameters.
    1. At a minimum, use the –r and –e parameters.
  3. Use Altiris Software Delivery Solution to deploy the package after each cycle of patch installations.


 

Using custom inventory to populate dynamic collections

Many customers find it useful to allow their local IT staff to classify how their assigned computers will be treated during patch deployments.

Altiris Inventory Solution® for Windows provides a function known as custom inventory. Custom inventory, among other features, is capable of scanning for values in the registry. With that in mind, the following technique is provided:

Create a series of registry merge files (.REG) corresponding to each level of desired patch management. Any alpha-numeric value for the level can be safely used. All highlighted text should be modified to match the actual company name.

Note:If copying these samples from a .PDF file, quotation marks and question marks are frequently converted to non ASCII characters. Visually confirm that the copied text looks the same when viewed in Notepad. It is also crucial to develop and test this process in a lab environment prior to implementing on the production Notification Server. Removal of accidental dataclasses is beyond the scope of this document (and will occur during development of custom inventory templates).

Paste the contents of the box below into a file named level1.reg. Execute on the target computers to assign them to level 1.
Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SOFTWARE\ACME ]
"Level"="1"

The following custom inventory template can be utilized to capture the managed level into a custom dataclass. Don’t forget to modify all highlighted text before placing the script into the production. Place the file on the Notification Server at:

C:\Program Files\Altiris\Notification Server\NSCap\Bin\Win32\X86\Inventory Solution\MangedLevel.xml

<?xml version="1.0" encoding="windows-1252"?>
<InventoryClasses>
<InventoryClass name="Acme Managed Level" manufacturer="Acme" description="Defines the managed level of the computer" platform="Win32" version="1.0">
<xml xmlns:s="uuid:BDC6E3F0-6DA3-11d1-A2A3-00AA00C14882" xmlns:dt="uuid:C2F41010-65B3-11d1-A29F-00AA00C14882" xmlns:rs="urn:schemas-microsoft-com:rowset" xmlns:z="#RowsetSchema">
<s:Schema id="RowsetSchema">
<s:ElementType name="row" content="eltOnly" rs:updatable="true">
<s:AttributeType name="c0" rs:name="ACME Managed Level" rs:number="1" rs:basecolumn="ACME_Managed_Level" rs:keycolumn="true" mifAttrId="1">
<s:datatype dt:type="string" dt:maxLength="10"/>
</s:AttributeType>
</s:ElementType>
</s:Schema>
<rs:data>
<%set path="HKEY_LOCAL_MACHINE\SOFTWARE\ACME "%>
<z:row c0="<%writexml "reg:%path%\Level"%>" />
</rs:data>
</xml>
</InventoryClass>
</InventoryClasses>

The first line below (aexcustinv) may be added to the existing AeXInvSolnAdm2.inion the Notification Server. Alternatively, place both lines below in a new ML.ini file, and create a new Inventory Solution task to gather the information on a daily or weekly schedule.

The ini files referenced above are located on the Notification Server under:

\Program Files\Altiris\Notification Server\NSCap\Bin\Win32\X86\Inventory Solution

Note: When modifying inventory scanning items, the aexnsinvcollector line must ALWAYS be last in the .INI file.

aexcustinv.exe /in .\ManagedLevel.xml /out ManagedLevel.nsi
aexnsinvcollector.exe /nsctransport /v default /useguid

Once the custom inventory data has been posted by the Altiris Agent, the new data can be seen within the Resource Manager on the target computer > Inventory Tab > Default Folder > Acme Managed Level > Value.

img SRC="21677_html_m15625c65.png" NAME="graphics21" ALIGN=BOTTOM BORDER=0>

Once the above steps are complete, the last step is to create a dynamic collection that leverages the new inventory data class.

  1. Create and name a new collection.
  2. Select the Query icon.
  3. Select the Resource type radio button > Computer.
  4. Click the Filter link.
  5. Add the corresponding data class Table and Field for the level as defined in the custom inventory template. Use the operator =and enter the appropriate value for the level.

img SRC="21677_html_1d26783f.png" NAME="graphics22" ALIGN=BOTTOM WIDTH=389 HEIGHT=292 BORDER=0>

 


 

After clicking OK, you should see the new filter defined.

img SRC="21677_html_m45f8d7ae.png" NAME="graphics23" ALIGN=BOTTOM WIDTH=512 HEIGHT=101 BORDER=0>

Click OKto save the filter, and then OKagain on the window below to save the changes to the collection.

This collection can now be used for Software Update Agent policies to control when patches are installed, when and if reboots occur, and the level of interactivity with a user. If desired, it may also be used as the target for individual patch bulletin policies.


 

Reducing the initial vulnerability window for new computers

Newly imaged computers will often contain the latest service pack but none of the post service pack patches. Typically, companies will not re-create their gold desktop image from scratch for each new patch, so pre-applying all the patches in the image is not an effective technique. Several alternatives exist:

Solution 1:

Inject post Service Pack patches into the image, and execute each of them as part of the sysprep process in the command.txt file.

Solution 2:

Use a Deployment Server script, after the image restore, to apply all post SP patches.

Solution 3:

Setup:

  • Use a private Lab Notification Server
  • Configure the agents to check in every 5 minutes
  • Configure the inventory scan rules to run every 15 minutes


 

Usage:

  • With Deployment Server, push down the Altiris Agent
  • Schedule patches to be applied immediately, with reboot as necessary
  • Finally, change the Notification Server pointer on the agent to the production server with the following command:
    aexagentutil /server ProductionNS.domain.com

Server Optimizations

Patch Management Solution 6.1 is heavily dependent upon the processing activities performed on the host Notification Server. Because of this dependency, this section focuses on optimizing the Notification Server’s processing capability.

Symptoms of poor server configuration include:

  • Slowness or timeout of console pages. This is observed on an intermittent basis, and may be temporarily corrected by restarting the Altiris Service (AeXSvc.exe). Alternatively, increasing the Altiris Agent configuration intervals will sometimes alleviate the issue.

Note: Significant processing optimizations (for client configuration requests) have been made in the core Notification Server product with Hotfix 22 for Notification Server 6 SP2. Please apply the hotfix prior to making any changes to the default settings (as described below).

Certain configuration changes will relieve this issue without negative impacts on the Notification Server or other Notification Server Solution operations. Methods of monitoring for this issue are also provided.

Diagnose and Monitor

This section describes the reports and performance monitoring counters that can be used for assessment. It is important to collect this data before and after, making any of the suggested configuration changes in the next section.

Reports

From the Reports tab > Reports folder > Notification Server Infrastructure > Agent > Configuration request > Configuration Request statistics by time. Initially, run this report with the following settings.

  • Time units for from-to date range and time {Hour, Day, Week, Month, Quarter, Year}—Week
  • Time units for grouping {Hour, Day, Week, Month, Quarter, Year}—Week
  • Limit report to last N time units—12

The most important column is Avg CPU(MSecs). This value is the average length of time the Notification Server is consuming to create the client configuration responses for the agents which report to it. Increasing values may indicate a problem because it means the server is spending longer to do the job than it did before, and this leaves the server less time for other processes which it must run. There is expected to be some increase in this value when more policies, advertisements, and packages are being managed by the Notification Server.

The sample report below shows this value increasing to approximately double the previous value and staying high, starting in week 28. This indicates an issue in creating the client configuration requests. This report also shows the effects of the changes detailed in this document. With same processing load the Avg. CPU (Msecs)has decreased almost to the pre-Patch levels.

 

img SRC="21677_html_meff4199.png" NAME="graphics24" ALIGN=BOTTOM WIDTH=686 HEIGHT=336 BORDER=0>

Performance Monitor counters

The other item to monitor on the SQL server is the performance monitor counter for Physical Disk: % Idle Time. If the disks which SQL is using are consistently showing low percentages for idle time (as in the yellow circled area) then SQL is probably too busy, to adequately process requests from the Notification Server.

Note:in the overall view below the % Idle Time is not an issue as it is only a spike and those are to be expected when certain operations, like collections update, big reports, and so on, are occurring.

img SRC="21677_html_mca56ca0.png" NAME="graphics25" ALIGN=BOTTOM WIDTH=637 HEIGHT=450 BORDER=0>

An additional item which can be monitored is the counter Web Service:Current Connections. If a sustained level for several hours near or above the Max concurrent configuration requests setting is observed, then additional adjustments may be needed to allow the configuration requests to process faster.

After the suggested adjustments are made, continue to monitor these items for improvement. Additional adjustments (up or down) may be useful in the future. Please note that increasing some of the items to values out of the suggested ranges will likely have negative impacts of their own.

Settings Adjustments

The table below represents the recommended adjustments. Prior to implementing any changes, review the reports and the performance monitoring counters described in the previous section "How to Diagnose and Monitor" for an hour of typical usage to obtain a baseline for comparison. Then, after initial adjustments are made, the impact can be measured and further adjustments made as necessary.

Settings adjustment table

Use NSConfigurator to make these changes. It can be installed by launching '\Program Files\Altiris\Diagnostics\NSConfigurator.msi'

Setting

Default value

Initial Adjustment

Appropriate Range

ItemCacheSize

500

3000

For any value above 3,000, consider the memory impact as each additional cached item can consume 1-10KB of memory

ClientPolicyCacheTimeoutSecs 600 1800

600 – 3600

Values above 3,600 should only be used if monitoring shows improvement

MaxConcurrentConfigRequests

50

5 to 10
(lower is better)

5 – 50

Values above 30 should only be used if monitoring shows improvement

Note the product ships with a default value of 50.

Parameter descriptions

Item cache: The Notification Server internal memory cache for items frequently used in processing.

Having an item in cache reduces the time to access it. This is because items are stored in the database and when the call for an item is not served by the cache then SQL must provide the information. This could mean waiting for access to the disks where the Notification Server database is stored, which is much slower than memory. And if the number of items which are being called frequently are larger than the cache size there will always be waits for SQL.

Note: Items in the Item cache originally come from the DB but a subsequent call for them is served from the memory cache. This is until such time as the item is changed and then it is flushed from the cache and a new copy is retrieved from the DB. The receiving of the copy from the DB takes the normal amount of time, but each subsequent call for that item will be many times faster.

Client policy cache refresh interval: The DB table cache of the individual policies which are used to construct any client configuration policies.

This table holds a copy of the policies which have been recently called and used to create a client policy configuration. These are not flushed when updated but when the time interval specified by this setting is passed. The time they are entered in the cache is kept in the ModifiedDate column.

Note: This table can be flushed manually by issuing the SQL command (without quotes) 'delete clientpolicycache' in SQL Query Analyzer. Flushing the cache is generally only used in testing scenarios where waiting for the cached item to expire and be replaced by the updated item is not a good use of time. In normal operation a delay is normal since agents only check to get new polices on intervals as well. And since this interval is usually longer than the cache flushing interval, agents should get the new policy at their first or at most second client policy check-in after the policy is updated.

Max concurrent config requests: The setting controls the number of computers allowed to connect at one time to receive configuration. Having this value too high will cause query contention and thus slowness in generating the configurations. Testing has indicated that lower values can be more efficient in reducing the likelihood of incurring SQL deadlocks on Notification Servers under heavy loads.

Scenarios

This section highlights the practices in use for selected customers. They are classified by industry to callout the varying priorities and weighting in use.

Conglomerate (multiple lines of business)

  • 35,000 workstations, 5,000 servers.
  • Facilities in North America, Latin America, EMEA, and Asia-Pacific.
  • Target SLA for compliance:
    • Urgent patches within 30 days.
    • 30 days at 90–95 percent.
    • 60 days at 97 percent.
    • 3 months at 98–99 percent.
  • Patch testing and rollout phases:
    • Primary Altiris administrators—Stages bulletin, checks for blue screens, command lines, and inventory rules.
    • Released to all business unit administrators for initial testing.
    • Bulletin enabled for pilot collections.
    • Full enablement per business unit.
    • Uses the master collection technique to allow business units to enable patch rollout at their own pace.
    • Each business unit sets one or more software update configuration policies for rebooting, notification, and patch installation schedule.
  • Disaster Recovery
    • Uses Altiris® Recovery Solution™ for most situations.
    • Altiris® PC Transplant Solution™ with a re-image of the computer as a fallback.

Semiconductor manufacturer and distributor

  • 17k workstations, 4k servers.
  • Facilities in North America, Europe, and Asia-Pacific.
  • 24 primary facilities plus satellite offices.
  • MS severity level is only part of internal weighting.
  • All patches are weighted and initially tested within 36 hours of release.
  • Target compliance SLA for critical patches within 7 days.
    • Level 1 computers at 98 percent.
    • Level 2 & 3 computers at 75–85 percent.
    • Level 4 computers are N/A, protected through other mechanisms.
  • Non-critical patches are deployed after 2 weeks of testing, during the next scheduled maintenance window.
  • Workstation patches are silently installed, with multiple reboot requests each day. Forced reboot during the weekend.
  • Implementing a Security Operations Group to follow-up with non-compliant business units.

Financial (stock management services)

Piloting Patch Management Solution on 10 percent of their managed nodes.

  • 4,000 workstations and 500 servers
  • International presence
  • Has a Risk Management Group
  • Researches vulnerability announcements prior to bulletin’s release
  • Higher weighting given if a an exploit has been published
  • Has the capacity to assign an Urgent severity, bypassing most of the rigorous patch testing process
  • Target SLA for compliance:
    • 10 days to most workstations
    • 12 days to notebooks
    • 14 days for servers
    • Urgent patches 100% within 30 days
    • Average compliance at 93%
  • Patch testing and rollout phases:
    • Primary Altiris administrators—Stages bulletin, checks for blue screens, command lines, and inventory rules.
    • Within 32 hours of release, deployed to integration testing lab which contains the 50 primary applications in use by the customer
    • Application owners perform rigorous testing. Testing is partially automated but depends upon the application
    • Application testing should be completed within 48 hours
    • On day 4, the bulletin is enabled for a large pilot collection (10 percent)
    • Over the next 6–11 days
      • Bulletin enabled for the next 40 percent of the environment.
      • Bulletin enabled for the remaining 50 percent of the environment.
  • Workstation scheduling and rebooting policies.
    • Patch installations scheduled overnight
    • Large collections of desktop computers are assigned different nights to apply enabled patches.
    • Traveling notebooks are configured to apply patches ASAP
    • External tool used to force a reboot on non-compliant computers
  • Server patch management
    • Patch installations scheduled to occur far in the future (to force caching of patch files).
    • A privately created Deployment Server script is initiated by the server administrator to ensure patching occurs during the requested maintenance window. Average rollout size is 50 servers per night.
    • The Deployment Server script sends a status e-mail to the customer during each step.
    • Prior to patching, the script reboots the server.
    • Standardized preparation tasks performed prior to any modifications to the server.
    • Software Update event triggered.
    • Post patch cleanup scripts.
    • Reboot.

Appendix A: MS Office source MSI registry keys

Many customers have installed MS Office through a CD, a temporary mapped network drive, or from a UNC share that no longer exists. The most recent version of MS Office (2003) provides the ability to cache the source files, but this option is not always utilized due to the large amount of disk space taken by the cached files.

The following registry keys are used to define locations for the MS office installation files. Access to these files is required when adding product features, applying MS Office service packs, and when installing some MS Office patches. Altiris® Software Delivery Solution™ or Altiris® Deployment Solution™ can be utilized to overwrite or add to these values.

For customers not utilizing cached source files, a best practice is to define one or more network UNC shares containing the office source files, such as \\fileserver\Office2k3.

Note! It is possible to define multiple search locations under the registry keys indicated below. Any future installation of MS Office should either be done from the networks share (don’t map a drive, use the UNC path), or embedded into the installation as part of the MS Office Deployment Wizard’s MSI transformation file.

For Office 2003:

HKEY_LOCAL_MACHINE\SOFTWARE\Classes\Installer\Products\9040110900063D11C8EF10054038389C\SourceList\Net

For Office 2000:

HKEY_LOCAL_MACHINE\SOFTWARE\Classes\Installer\Products\904040001E872D116BF00006799C897E\SourceList\Net

For Office XP:

HKEY_LOCAL_MACHINE\SOFTWARE\Classes\Installer\Products\9040820900063D11C8EF00054038389C\SourceList\Net

Revision History

Revision Date Description
1a 12/14/2005 Shell document created

1b

1/9/2006 First aggregation of content
1c 2/12/2006 Addition of Notification Server tuning, rewrite of PM overview, rebooting of workstations
1e 4/10/2006 Major additions incorporated from development of best practice presentation and additional customer interviews
1f 4/11/06 First published release
1g 4/11/06 Added How to section of forcing a reboot at next logon
1h 4/13/06 Minor terminology update, added a clarification in the section on Remediation
1i 4/21/06 Added item to explain where to place the custom inventory ini file
1j 5/23/06 Added recommendation for Windows Installer 3.x
1k 12/7/06 Added descriptive text for step #5 on Patch Management Process diagram
1l 9/7/08 Rebased the document to an html format

Legacy ID



21677


Article URL http://www.symantec.com/docs/HOWTO3124


Terms of use for this information are found in Legal Notices