Patch Management Solution

We are testing patch management at the moment currently with 20 machines looking to roll out to 1500 in the future.  We are getting some interesting issues in testing that I would like your thoughts on if I may.  Of the 20 machines most of them were showing as 30 bulletins behind but one was 86 so I used the 86 patches as the baseline to deploy to the pilot group.  It downloaded a ton of patches for the 86 updates and was then scheduled for 2am the next morning.  Next morning and not much had happened - a few patches had been installed so I used the machines with 30 missing as a baseline and deployed immediately with restarts allowed.  Then the fun started.

Over the next week we would come in and users would say patches are installing during the week and that a few machines could not be used as the C:\ drive space was full up - C:\Windows\Temp seemed to be the culprit.  Its not a couple of hundred MB's either it is GB's of space. 

From the pilot group there is not much faith in the product at the moment so Im looking for some best practices on how we move this forward and what enterprise deployments of this solution are doing to use this solution to install patches to devices.

I can see a setting where you can move the patch location for downloading them from the site server to the D:\ drive for example instead of where the NS client is located on the C:\ drive.  Is there a way around this C:\Windows\Temp issue or is this hard coded from the patch vendors?

We also started seeing clients dissapearing when we checked for compliance to see how many patches are installed\not installed - any ideas why this happens?

Is there a way to not have to download so many patches from these bulletins?

Whats the max amount of bulletins we should do in any one time?

Which settings should we fine tune to make this work correctly?  The next group we deploy to is 10% of the estate and if they have the issues above we will be in for pain.  After 10% its 30% so even more pain.

What install window do you configure - at the moment it seems to be most of the time but ideally we only want to allow this to be Friday mornings between 2am to 6am but at this rate I cant see much getting installed at all?

Do the patches get removed at all or do we as the administrator need to go in and remove them? Above example 900 patches that will be on the NS server and then site servers as well and then on the clients as well.

If we deploy on Saturday 2am to install to 300 machines. 50 machines are switched off and get turned on Monday morning 9am. Do the patches then get streamed to the clients and installed in the next window so 9am we have lots of patch downloads happening across the LAN and then local installs on the clients? If so is there a way to stop this?

What throttling settings do you enable for your agents for network consumption or do you not bother?

Any issues etc its good to know about with Altiris Patch management that are good to share?

Is there a recommended amount of patches\bulletins to select\use in each patch session? We need to catch up a bit and have a new system but I don't want to kill the NS or the clients.

If you are downloading the patches to the NS, then to the site servers and then to the clients if there are issues somewhere external (e.g. LAN being swamped with traffic and high utilisation affecting normal application access) how can you stop the process from happening at each stage?

If you have downloaded a patch already and you then select it again will it download again or is their logic for it to know it's already available?

I know there is a lot here but many thanks for any assistance you can provide me.  I have a call open with Symantec but Im not getting that far with them.

 

When you download and stage a Bulletin it will download all the patches that apply to the Products you originally set up. So if you said you want to patch IE8, 9, 10 and 11 it will download the patches for all those versions. The PC that only installed a few patches, did the other patches fail or just not run? Had the ones that didn't run download? Or maybe they hadn't had enough time to get to the package server then download? And the PC that filled up it's C:\windows\temp directory, what files were they? Patches, particularly Microsoft ones, will log there, you could change it for each patch as you stage them but that won't stop the other temporary files from going there. If your network is up to it and you allow enough time for staging, 80 patches in one go should be manageable. Patches should only install at the time stated on the Policy, however, depending on when the PCs are rebooted, users may see the "Windows is being updated" type messages when they are rebooted or started up in the morning. The trick with staging in weak networks is to make sure your agent throttling settings are tuned to the requirements and capacity of your specific network and apply the patches in the morning so the PCs are mostly on and have time to download the patches. If your network is weak make sure you don't apply the patches to too many PCs at once and split them using some sort of random split, like the last digit of PC name rather than doing it by site. The trick with patching is to set it up to work with the limitation of your network and not try and interfere with it because of other factors.

Thanks Andy for your comments.

Where is the best place to view what has and hasnt been installed?

C:\Windows\Temp seems to be CAB files from the patches.

Is there a way to stop these random restarts during the day or is it as long as its not in schedule it will not happen?

Look in the Software Updates tab of the Symantec Management Agent in the Notification Area on the client. If the CAB files from the patches are ending up in c:\windows\temp has this been configured on any of your patch settings to use this location? Provided the Software Update Plug-in Policy applied to the clients doesn't have any reboots in it then the PCs shouldn't reboot; again look in log and event files for any that do to find exactly what is triggering the reboot.

While that's a nice theory, I have found out that isn't exactly how it happens.
The best example I can think of just off the top of my head, and I'm still trying to figure out how to make this thing behave, is that I had a JAVA update setup to go after 6pm of any day.
2 weeks passed, nothing, then I came in yesterday AM and found my computer had rebooted.
I set that policy for the JAVA up, like I do all others, NO reboot, I did not check allow reboot and the schedules I have created do not include reboot. Basically I'm relying on natural reboots done by users or other things.
I checked my logs - the MSI installer triggered the reboot but the Patch Management logs on the server when I checked the "installed" report says it was user triggered!

Yeah, right-  I was at home when it rebooted. The reboot happened Saturday at 8:45 pm. That's days after I set the policy up and over 2 hours after 6pm, the supposed scheduled time.

So even though it was not set to reboot, there was no scheduled reboot, no maintenance window, no policy, no schedule was configured to allow reboot - the computer rebooted after the JAVA install and the local Windows application log says the MSI installer triggered a required reboot.
The only thing that was possibly involved was that I did a setting somewhere I can't find again as there are so many places to look, is that I allowed for a user to delay things. But it wans't clear and I assumed that was to allow the user to delay the install........... because with some things it's the INSTALL that chokes a computer, not the reboot. If you are using Word or IE and the patch is for Word or IE, the user can't work if the install starts. If it's JAVA, you have to close all browsers for a JAVA update - so you can't have JAVA patch or install during work days as it interferes with user work.

Apparently the JAVA install itself called for a reboot and so the MSI installer did the reboot, the agent reported back to the server and it's in the server logs that it was user initiated which is bunk as no one was even here at the time.

Doesn't matter what you do, if the patch or update says a reboot is needed, apparently a reboot will happen.

You can change the default install command for a patch, add /norestart to any msiexec command lines. Not sure what the command lines for Java updates are. I tend to install patches at 22:00 every night and then shut down PCs anyway around midnight. I'll wake up PCs once a week around 20:00.

C:\WINDOWS\TEMP I'm being told from Symantec is a Microsoft setting.  If anyone knows a way to move this to the D:\ drive for example I'd love to hear.  Patch management has a setting where you can store the patches for download from the site server etc which is different from the default agent settings which for us is C:\.

Indeed we do as well have some patches sitting waiting to be installed in the Software Updates Tab of the agent even with the allow restart option for the policy enabled.  Not sure why these havent been installed but I will do a manual restart to see what happens.

Id be interested to hear what settings etc you configure in your policies\agents for patch management and when you do actually patch.  At the moment I have changed the schedule to just 1am Sunday morning as we are testing the solution and will look when we are happy to move to 1am Fridays as is the normal schedule at the moment BUT only when all issues have been worked out.

Thanks as always for your help.

Our issues are still ongoing - weve found that the C:\windows\temp directory filling up is a Microsoft bug which we can get around.  What we have found is that weve deployed patches from August 1st to November 28th and on some machines - not all of them they are loosing their DNS settings and we have to visit each machine with issues and run an Ipconfig /renew on them.

Has anyone else seen this before?  This is a bit of a show stopper for patch deployment as we cant run anything on them to resolve it once the issue occurs so any help would be appreciated.  Trying to identify which patch or patches is a bit of a looking for a pin in a haystack.

Thanks.