Resynchronizing Package servers with invalid or stale codebases for Notification Server 6.x

Article:HOWTO1108  |  Created: 2005-09-02  |  Updated: 2013-03-08  |  Article URL http://www.symantec.com/docs/HOWTO1108
Article Type
How To

Product(s)


Problem
There are several conditions that this article attempts to resolve with a resynchronization of the Package Servers but generally it is an issue where the Package Server agent shows that all packages are ready when within the Notification Server, its package status shows that one or more packages are not ready on one or more Package Servers.

Symptoms could be but may not be limited to the following:

  • Stale Codebases
  • Invalid Packages
  • Packages won't download to clients
  • Packages Not Ready
  • Packages Invalid - Error while downloading package: Server is busy.  Package sources request is backing off (-214746725)

If this is an issue that the number of Available Packages in the Package Servers status are being duplicated please refer to article 38246.

Environment
Notification Server 6.x
Package Servers

For Symantec Management Platform 7.x Please refer to HOWTO83906

Cause
 

There may be several causes for the problem but the reasons for this issue is that there is either a package version mismatch between the Notification Server and the Package Server, or the Package Server may not have sent the status for the package in question.

For example, it could be one, but not limited to the following:

  • During the process of building the snapshot, if a transaction rollback occurs in the SQL, then this can cause a version mismatch.
  • A large number of NSE's in the queue and the package status is still waiting to be processed. Perhaps the NSE processing is having difficulty
  • An error could have occurred while processing the NSE and the package status is in the Bad folder.
  • The package server ligitimately has not downloaded the package yet.
  • The Package Server is having client/server communication problems.
  • It could be Site Maintenance problems.
  • Permissions on the NSCap and NSCap Event Queue subfolders.
  • Again, there could be other causes for this.

Resolution
 

Be aware that improper Site Maintenance can cause similar behavior. If sites and subnets are not correctly created with the Package Server assigned to that site, then invalid code bases can be given to the clients within that site, causing the situation where clients won't download packages.

As mentioned in the Problem/Symptoms section there are several situations that this article tries to remedy:

  • Stale Codebases
  • Invalid Packages
  • Packages won't download to clients
  • Packages showing Not Ready (in the Notification Server console)
  • Packages showing Invalid (in the Notifidation Server console and on the Package Server tab of the Package Server's Altiris Agent properties)
  • Permissions on the NSCap and NSCap Event Queue subfolders on the Notification Server.

At a fairly high level, these issues can be attributed to synchronization problems between the Notification Server and client (Package Server), specifically dealing with the snapshot.xml. This ultimately causes the client an inability to download a package. The purpose of this information is to give several aspects of synchronization to assist in making a decision on how best to handle the current situation. This will be done in several steps.

There are six basic sections in this article. They are purposefully ordered in this fashion from the most likely to the least likely causes. It isn't necessary to follow all sections, start with Section 1 first, then if it doesn't work, proceed to Section 2, and so on.

  1. CheckSnapshots
  2. Manual Synchronization
  3. Dealing with the Database
  4. Share Permissions
  5. Agent Version
  6. NSCap Permissions

Section 1 (CheckSnapshots)

Download and run the attached tool (CheckSnapshots.zip) onto the Notification Server. This tool checks the contents of the current snapshots in an attempt to synchronize the package information kept within the database.

  1. Download the attached file CheckSnapshots.zip (by clicking a link on the right side of this article).
  2. Unzip the files into a folder. The location doesn't matter but when run the executable must be able to load the .config file included in the .zip file.
  3. Double-click on the executable to run it. There is nothing else that needs to be done.
  4. It is suggested that the CheckSnapshots.exe be run as a Windows scheduled task. Please see the section of notes just below.
  5. Check to see if you can now send packages.

If this did not correct the problem  then continue  following instructions below.

Note 1! CheckSnapshots.exe will checks if the snapshot version is greater on the xml files that on the database SWDPackage table. If the xml is at a newer version than the DB it will remediate the issue by updating the DB [Package Version] field for the package.

Note 2! To find out if the problem is  because a package is not ready run the following SQL query and then open some of the corresponding snapshot.xml files that are located in %ProgramFiles%\Altiris\Notification Server\Snapshots.

SELECT
        vi.[Name] AS 'Package Server'
       ,ps.[Status] AS 'Status'
       ,pack.[Package Version]
       ,ps.[PackageId] AS 'Package Guid'
       ,pack.[Name] AS 'Package Name'
  FROM SWDPackageServer ps
  JOIN vItem vi
    ON vi.[guid] = ps.[PkgSvrId]
  JOIN SWDPackage pack
    ON pack.[PackageId] = ps.[PackageId]
   AND pack.[_Latest] = 1
 WHERE ps.[Status] != 'Ready'
 ORDER BY vi.[Name], ps.[Status]

If  packages not being ready is the problem, the CheckSnapshots.exe should be set to run on a schedule about 5 minutes past the amount of time required to run the NS.PackageRefresh schedule. For example, if it takes 15 minutes to run the NS.PackageRefresh schedule then the CheckSnapshots.exe should be scheduled about 20 minutes after.

 

 

If Section 1 does not resolve the issue, then it may be required to manually synchronize the snapshots. The following steps illustrate how to do this.

Section 2:(Manual Synchronization)

Important: There are implications when taking this approach. When deleting the snapshots on the Notification Server, it does create a new version for that snapshot, therefore all the clients within the Notification Server infrastructure will download the new version after their next configuration update check. This will potentially generating a lot of IIS traffic.

From the Notification Server:

  1. Delete the snapshots ({GUID.EN_US}.xml) found under the “%ProgramFiles%\Altiris\Notification Server\Snapshots” folder. On 7.1 this is located: C:\ProgramData\Symantec\SMP\Snapshots. (ProgramData is a hidden directory) 
  2. Then go to Control Panel > Scheduled Tasks and run the task called NS.Package Refresh.{GUID.EN_US}. This will re-create the package snapshots on the server. Make sure that the scheduled task does in fact start. If it does not, see article 32038, "Scheduled Tasks on the Notification Server show a status of 'Could not start'".

     

 

From the Package Servers:

  1. Delete the snapshots (snapshot.xml) found under the %ProgramFiles%\Altiris\Altiris Agent\Package Delivery\{GUID.EN_US} folders (see Notes below).
  2. Delete the .xml file (<NS name>.xml) under the %ProgramFiles%\Altiris\Altiris Agent\Client Policies\ folder.
  3. Right-click on the Altiris Agent icon > Altiris Agent Details in the system tray and click on Update Configuration. This will cause the newly created snapshots to be downloaded from the Notification Server.
  4. Right-click on the Altiris Agent icon > Altiris Agent. This will open the Package Server window. Go to the Package Server tab and click on the Refresh Packages button and then Resend Package Status. This will cause the Package Delivery snapshots to be re-created and update the Notification Server of the Package Server status.

     

From the Client (see Notes below, but deleting the snapshots from the clients should not normally be done):

  1. Delete the .xml file (<NS name>.xml) under %ProgramFiles%\Altiris\Altiris Agent\Client Policies.
  2. Delete the snapshots (snapshot.xml) found under the %ProgramFiles%\Altiris\Altiris Agent\Software Delivery\{GUID.EN_US} folders
  3. Right-click on the Altiris Agent icon > Altiris Agent Details in the system tray and click on Update Configuration. This will re-create the client policy and client snapshots.

NotesHow to delete snapshots quickly on the Client and/or Package Server:

 

  • Open %ProgramFiles%\Altiris\Altiris Agent. If the original agent was installed from earlier Notification Server (5.5), the path on the Client\Package Server will be different from the above mentioned path, and all upgrades will use the original installation path. This old path should be under the %ProgramFiles%\Altiris\eXpress\NS Client folder.
  • Search for "snapshot.xml".
  • Once you have the results, select all and delete. After this is done, there is potential for the clients within the infrastructure to download all the snapshots. This would occur if the snapshots on all the clients were out of sync with the information contained on the Notification Server. As the intent for this article is primarily to synchronize the Package Servers with the Notification Server, there generally shouldn't be a problem with the rest of the clients. To alleviate some of the potential for this, rather than removing all snapshots on the Notification Server, only remove the snapshots for the packages that are having difficulty.

 

 A more extensive approach is to takes the steps outlined in Section 2 then remove the codebases from the database and have them regenerated from scratch. Since the codebases are stored in the SWDPackageCodebase table that table must be cleaned.

Section 3: (Dealing with the database)

***VERIFY THE PACKAGE FILE SETTINGS BEFORE DOING THIS STEP! Change or verify the setting "Delete package files if they are unused for" is set to 1 week.*****

The following SQL is used to truncate the SWDPackageCodebase table. While it does copy the information into a temporary table, this generally isn't necessary.

-- Select each item distinctly into a temporary location
select * into #tmp_SWDPackageCodebase1
from SWDPackageCodebase
truncate table SWDPackageCodebase

-- Finally display the newly built table to verify the data
select * from SWDPackageCodebase
-- This final piece of SQL will remove the temporarily created table from the database
-- once the data has been recreated and verified. Uncomment this and run it alone.
-- drop table #tmp_SWDPackageCodebase1

If the SWDPackageCodebase table is truncated and these scheduled tasks are run, the package codebases will be re-created but only those codebases for the Notification Server. The Package Server codebases are not repopulated in the table until they have communicated the "Resend Package Status" event. This event sent from the Package Server will resend the status of each package on the Package Server and repopulate the codebase information for each Package Server.

  • NS.Package Distribution Point Update Schedule
  • NS.Package Refresh

To complete the codebase refresh, the Package Servers themselves must either send an Update Configuration request or "Resend Package Status". Once this occurs, the SWDPackageCodebase table will be completely rebuilt. The NS.Package Refresh schedule normally runs once a day at 3:30 a.m.  Verify that it runs correctly (Check the date it last ran). If it does not, see article 32038, "Scheduled Tasks on the Notification Server show a status of 'Could not start'."


In addition to truncating the SWDPackageCodebase table, a similar approach can be done with the SWDPackageServer table. Although this is normally not needed for the synchronization process, it can be done to assure that the Package Server table is clean when the information is regenerated.

Make a backup of your database and do the following, run each separately (Section 2, Section 3):

/* Step 1:
Copy the current contents into a temporary table for backup purposes if
needed */
select * into #tmp_SWDPackageServer
from SWDPackageServer

/* Step 2:
Remove the contents from the table completely */
truncate table SWDPackageServer

/* Step 3: */
Run the following tasks.


 

NS.Package Distribution Point Update Schedule
NS.Package Refresh
NS.Package Server Status Event Capture Item

/* Step 4:
Display the contents of the table. */
select * from SWDPackageServer

Verify the contents of the SWDPackageServer table. You should be able to do this visually to make sure that there is only one package per Package Server. When you have displayed the contents of the table, just take a look to see if the duplicates have been removed. They should be gone.

/* Step 5:
Uncomment the SQL statement below and run it to remove the
Temporary table once you have verified that the data is intact */
/*drop table #tmp_SWDPackageServer*/

 

Section 4 (Share permissions)


 

There could be a permission issue if the packages are located on a share. The share permissions for the local directory on the Package Server (that is, \\%COMPUTERNAME%\Sharename\Package) are set to "Everyone—Read". This is the default on Windows Server* 2003 for a manually created share. Once the permission "SYSTEM—Full Control" was added, then the package was accessible.

Additional Queries and Information

Here is an additional SQL query that can help do some basic analysis of the situation with the Package Servers and Notification Server:

-- This query shows the packages that are not ready
-- along with the Package Server that owns the particular
-- package.
select
[Package Server]= vi.[Name]
,[Status] = ps.[Status]
,[Package Guid] = ps.[PackageId]
,[Package Name] = vi0.[Name]
from SWDPackageServer ps
join vItem vi on vi.[guid] = ps.[PkgSvrId]
join SWDPackage pack on pack.[PackageId] = ps.[PackageId]
and pack.[_Latest] = 1
join vItem vi0 on vi0.[guid] = ps.[PackageId]
where ps.[Status] not like 'Ready'

 

The following SQL cleans up duplicates in the Notification Server console under the Package Server status

select distinct * into #swdpackageserver
from swdpackageserver
 
truncate table swdpackageserver
 
insert into swdpackageserver
select * from #swdpackageserver
 
drop table #swdpackageserver

 

 

 

Section 5  (Agent versions)

Verify that the Altiris Agent and the Package Server sub-agent versions are current or close to current.  It has been seen where problems were created with the Altiris or Package Server Agents and where they weren't communicating consistently with each other or the Notification Server, and where a reinstallation of the agent(s) may be considered, but if the agents aren't current an upgrade to the current agent resolved has resolved the following:

Invalid packages - Error while downloading package: Server is busy.  Package sources request is backing off (-214746725)
 


 

Section 6:  Permissions on the NSCap and NSCap Event Queue subfolders.

When the package server tries to send a package status, it does this using an NSE file.  This NSE file is by default compressed at 200 KB.  Therefore, when this file goes to the Notification Server it will be placed into the NSCap\Temp directory for decompression.  Then it would be moved into the proper EvtQueue folder.  However, if the permissions are not correct on the NSCap\EventQueue folders, this process will not complete.  Unfortunately no errors are displayed in the Notification Server log file, or in IIS.  Using a network trace tool like Wireshark will show an error when this event is posted.  For further information on this and for information on the access rights needed for the NSCap and subdirectories, please view KB 4396.

 

 

 

 

 

Note: If the the number of available packages on the package server are not changing, the package server is shows 'Server is busy', and the trace data in the Altiris agent logs are showing MaxConcurrentPackageInfoRequest, then you may need to up the value in the NSConfigurator. Reference KB #22646

 

 


 


 


Section 3 (Dealing with the Database)

 


Section 2 (Manual Synchronization)


Attachments

CheckSnapshots.zip (4 kBytes)

Legacy ID



1492


Article URL http://www.symantec.com/docs/HOWTO1108


Terms of use for this information are found in Legal Notices