Client Management Suite

I'm looking for some insight into how the new package server works in 7.5 SP1 compared to 7.1 (with the same package server agent settings). I have several different scenarios, so I'll separate them out:

1. What to do to make a new software resource package available as quickly as possible.
   • In 7.1 all I had to do was go to the package server, update configuration and it would download and be available almost immediately. I could then schedule quick deploy jobs for that package on clients in that site and the installs would happen within a few minutes.
   • In 7.5 I try the same thing and while the configuration does update, the package doesn't start to download. It takes up to an hour to actually download the package. And then after the package shows ready, it takes at least ten to fifteen minutes before clients can download the package without the 'No package sources returned by server' error.
2. What to do to make a changed UNC-based software resource package available as quickly as possible.
   • In 7.1 when files where changed in the base UNC path for the package, all I had to do was go to do 'update distribution points' on the software package resource, then update configuration on the package server. It would download the updated package right away and be available to clients almost immediately.
   • In 7.5 'update distribution points' seems to do nothing. Updating the configuration on the package server doesn't receive a new config until some random length of time later (up to an hour). I have tried 'Resource Membership Update' but that doesn't seem to hurry it along. Once it finally downloads to the package server, I have the same issues with slow availability to the client as in #1.
3. What to do to force a client to re-download a software resource package.
   1. In 7.1 I didn't have to worry about this at all. If the package had changed, it would re-download automatically and all was well. At most in a very few cases all I had to do additionally was delete the cached GUID folder in Program Files>Altiris>Altiris Agents>Agents>Software Management>Software Delivery for the package.
   2. Now when a package changes, and I verify it works fine on a client without a cache, it will not always re-download on clients with old caches. It just runs the old cached version. And if I delete the cached GUID folder, sometimes it starts getting errors about package IDs not matching, with the client somehow having a newer package ID than the server and refuses to download the package again at all. It's gotten so bad at some points that I've had to completely remove the agent and re-install, or re-image the client, to get it going again.

These issues have combined to make it extremely frustrating to develop and test new software installs in 7.5 SP1. Even if they are copied verbatim from 7.1 and I know they work, it takes at least an hour from start to finish on each package to get it installing on clients and tested. God forbid if it's a new package that isn't quite working correctly yet.. then it's hours to baby it along while I troubleshoot different changes. I am guessing that these issues stem partly from the new validation/security in the new package servers. Is there some way to turn this off, temporarily preferably but permanently if necessary and it clears up these issues?

Hi sfaucher!

You are right with your guess that 7.5 security initiative eats part of the resources and could increase time for some operations. Unfortunately security does not go cheap and this is a price we need to pay to be secure, but this is not so dramatic in your case. Let me explain.
Part of the security initiative in 7.5 was a package validation and encryption, in order to protect clients from MitM and spoofing attacks. Therefore all packages in 7.5 are being hash validated and this is done on both sides: server and client. This means that when you create a package through the Notification Server console, the NS machine start to calculate the hash of all the files in the package. Later this hash is securely transferred to the interested clients, and when clients download the package, they also do the same: calculate hashes of the received files, in order to be sure no one is spoofed them and they received the correct binaries. The speed of this process could be a less than a second, but could be ten minutes and more, depending on the machine hardware and the size of the content to be hashed. Although those security measures is not possible to turn off for 7.5 SP1 (since a lot of other infrastructure is based on it), it is possible to set-up the appropriate settings in order to have reasonably fast environment.

Now about your case in detail.

1.    >>What to do to make a new software resource package available as quickly as possible.
Actually the logic did not changes so dramatically and all processes are remain the same. The only difference is when you finish creating the software resource, the files are start to be hashed by the system on the background thread, so it will not be possible to download such package, until the hashing finishes. I very doubt your timeout of 1 hour is caused by enourmously big package or very slow system. It more look like the standard “Configuration Refresh timeout” on the Package Server (later PS). I guess you need to refresh the configuration on the PS manually, when package gets finish calculating hashes on the NS (not right away after creation of the big package in the console). Also you could either set up smaller amount to the configuration refresh of the PS-es or use “Emergency Policy update” when creating the Managed Software Delivery (emergency policy update will force PS-es and clients to refresh own policies in order to receive the package description immediately and start to download this package).

>> And then after the package shows ready, it takes at least ten to fifteen minutes before clients can download the package without the 'No package sources returned by server' error.
In case there is at least one PS is configured on the NS, the NS will not return own codebases and will return only PS codebases. The message “No package sources returned by server” means that NS still don’t have the information from PS about package being downloaded by PS. This could happen by several reasons: whether PS have not sent the status, or status was not yet processed by NS. Although PS send statuses not later than a minute after getting them to know, the NS could be busy with NSE processing depending on the current load and environment situation. In order to be sure you could check the packages statuses of your PS on “Site Server Settings” page, and if required resend the status manually, by clicking the “Resend status” link on the PS UI when desired package is selected, and then check whether the status is updated on NS.

2.    The “Update distribution points” task does not force hashes recalculation of the packages. In your case when you change the files of the package, and want it to be immediately available, you need to run the “NS.Package Refresh” task. This task does the hash re-calculation, validation of the package content, re-generation of package source files. When it finishes, you need just to hit the “Update” configuration on PS. By default this task is scheduled to run once a day. Exact scheduling time could be checked in Windows Task Scheduler.

3.    >> What to do to force a client to re-download a software resource package.
The algorithms of this process actually didn’t changed at all. If package is changed, and recalculated on NS (actions from step 2 above are done), the very same “Update” configuration on the Package Server will force it to download a new version of a package. Even more, since the files are now hashed, the only delta changes will be downloaded for the package, and no need to delete manually the package itself.
If you want manually force PS to re-check own packages you could click the “Refresh all packages” link in the PS UI. Although it is not a supported scenario to delete cached files by hands, the SMF still will re-download the content of the package, if it is not found locally.
Here you should be aware of the next: the clients SMF or Package Server, both are working with the packages descriptions sent by NS. They do not check the real files on NS each time they require something. It is an NS responsibility to provide package descriptions which correspond to the actual files. Thus if you need to speed up the processes, you need to be sure your modified packages are properly updated (step 2 above as example).
As for GUID’s: the GUID is unique for each package. You could even create a several Software Resources with one package which will have the same GUID. If you see the different GIUD on the client, this only means that you created a new package on the NS. This case probably need to be investigated with the Symantec Support.

Regards,
Roman.

Thanks Roman, that sheds a little more light on the process, although there still seems to be some more going on behind the scenes. Currently my 7.5 SP1 environment only has a couple of VM clients for test so I went ahead and set my site server to update configuration every minute and send inventory every five. This effectively takes checkin lag out of the picture. There is still some random amount of time that is happening between when the package server receives an updated configuration which I assume is the notification that there is a new package (it gets it within a minute after I save the new package on the NS) and when it starts downloading the package. It's like some hidden schedule. If I rapid fire create several packages on the NS the package server updates its configuration after every package save, but the actual downloads end up taking place some random amount of time later all at once. It's like some hidden schedule is in place somewhere that has it start downloading all new packages it has been notified about recently. This is the schedule that I'd like to find out how to "hurry up", because it seems like it can take up to an hour if I happen to save a package just after it was scheduled to fire off.

On the client end things are fine as long as I leave things alone and just wait. It takes longer than it used to, but as long as the package is ready on the package server, it works with some additional time that I assume is because of the hash checks. I can live with that. I think the problems I was getting on that end was when I was trying to hurry things along and deleting the GUID folders in the middle of a hash check.

>> There is still some random amount of time that is happening between when the package server receives an updated configuration which I assume is the notification that there is a new package (it gets it within a minute after I save the new package on the NS) and when it starts downloading the package. It's like some hidden schedule.

I don't know in very detail the NS side logic of how the codebases become ready after new package creation and when exactly package is 100% ready for downloading, so I will forward this question to the proper guys for analysis.

As for hidden schedule on the PS - there is no such ;) When PS receives the package description in the policy, while the configuration update, it just tries to download the package until it gets it. Actually, you could check if there any warnings or errors in the PS log stating any download problems or something like "backing off". All clients (PS is acting like a client while downloading the package also) has the logic of download retries and in some cases do a back-off, increasing the interval between retries. This could be seen from the client side logs.

Meanwhile I hope the NS side guys could be able to answer on Monday, regarding the package creation logic on the NS.

Regards,
Roman.

The NS guys confirmed that Package will be sent to the PS while PS policy request, immediately after the creation of the software resource. In case if there is slow connection or a big package the background hash process could be not yet finished, and then PS will have some errors in the logs and retry package download several times. That could be the reasons of your "laggs" of getting a new package to PS.

Also they suggested to use the "'Update distribution points" on the software package resource menu (not the task) which do re-create the the missing hashing results (signatures) for specific package only, instead of "Refresh packages" (which does this for all packages).

Additionally, if the package is UNC based on the NS, this also slow down the hash creation, since the NS will need to "download" the files into the memory anyway, in order to calculate the hashes.

If you are interested in the further investigation, we will need to dig into more details and get your logs (preferrably from both sides and trace level).

Regards,
Roman.