Video Screencast Help
Symantec Appoints Michael A. Brown CEO. Learn more.

patch installation jump schedule date

Created: 12 Dec 2013 • Updated: 11 Feb 2014 | 7 comments
This issue has been solved. See solution.

Hi there,

I have scheduled some windows servers to have July - August 2013 patches installed on Feb 1st, 2014 (2/1/2014) at 00:30 AM - 3:30 AM. I just found out to day that these subject windows servers had patches installed from 00:30 AM - 3:30 AM on 12/4/2013 (December 4th 2013) which was the date I set up the update plug-in policy and software update policy.

Would you please advise if ever there was similar incident. It'd really cause impact/trouble for production windows server since there's been patches required reboot (3010).

Thank you,

Charlie.

 

Operating Systems:

Comments 7 CommentsJump to latest comment

HighTower's picture

Can you post pictures of how your Software Update Plug-In Policy is configured/scheduled?

Also, can you ensure that the servers in question are only targeted by one Software Update Plug-in Policy?  If you have an policy that is scheduled for 12/4 like you described I'd start by looking at the filter memberships to make sure these servers don't exist in multiple filters.

Charlie D Tran's picture

Good evening, Hight Tower,

Thanks for your reply.

I attached here a snapshot of the plug-in policy. I'll continue to add servers into this policy and pay attention to the dates to see if they are really jump the schedule. I also will make sure that there is no other plug-in policy (that had subject servers) turned on. This might be the case, because there are other filters that have same subject servers, but I am not 100% confident that other plug-in policy were turned off.

I'll keep you posted on the issue.

Thanks again very much for your advice,

Charlie.

 

 

Software Update Plug-in Policy.jpg
HighTower's picture

Hey Charlie, did you ever figure this one out?

Charlie D Tran's picture

Hello High Tower,

 

It is disastrous and shocking now. I should have taken your question more seriously when I reported Altiris suspected to jump deployment schedule at our Lab environment.

It was just because I trusted Altiris Patching system (currently at version 7.1) so much that I took time to investigate the issue and took no caution at all to my company production environment.

I set up 2 sets of policies and schedules to patch 40 production windows servers (both win2K8 and win2K3) with September 2013 patches. 20 were scheduled at 00:30 AM on Saturday 02/08/2014, 20 were scheduled at 00:30 AM on Sunday 02/09/2014.

Somehow, at 00:30 AM on today Friday 02/07/2014, total of 8 production windows servers belong to both Saturday 02/08 and Sunday 02/09 policies fired off installation ofpatch September 2013 patches and rebooted (3010 exit code) which set off a ton of alert to our company security and command center.

These production windows servers are being accessed by world-wilde customer from European and Asian union, etc.System owners and customers jumped to the roof now.

I am freaked and in trouble. I am trying to log the problem to Symantec Business Critical Support now. Man, this is multi-million dollars product with hundred thousands of dollars for annual support.  t

In the meanwhile, please advise if you or anyone ever went thru with this. Attached are screenshots of policies set up and 1 Altiris SMA client log for your review to see if I made any mistake in setting up these policies.

Thank you,

Charlie Tran

 

 

 

Saturday020814 Deployment Policy.jpg Sunday020914 Deployment Policy.jpg
AttachmentSize
uspto-a-efs-3.zip 98.38 KB
HighTower's picture

You have a global implementation?  See how your policy is configured to "Use Agent Time"?  Could this be because your SMP is in one time zone (Friday) and your patched servers are already Saturday?

Also, I never ever use the schedule windows like you have configured as the outcomes are too unpredictable.  My company insists on knowing when something is going to happen and we also have systems in different patch groups orchestrated to reboot at specific times.

So, I use a Schedule Time instead of a Schedule Window.  My patching starts at a specific time and the servers reboot exactly 8 hours later, unless they had no work to do.  My company is regional and all in the same time zone... maybe you'd want to consider changing yours to "Coordinate using UTC" instead?

Attached is a screenshot of one of my Software Update Policy configs.  I hope this helps.

PatchSched.PNG
Roman Vassiljev's picture
Hi Charlie,
 
I am sorry to hear about your troubles.
 
I reviewed Agent logs you provided and found that SMA Maintenance Window was opened at 00:30 AM on Friday 02/07/2014. Patch Management Solution is designed so that If SMA maintenance windows are defined the Software Update Installation schedule will be ignored and updates will install during the first available maintenance window unless checkbox 'Override Maintenance Windows settings' is checked in SWU Policy schedule settings.
 
Could you please check SMA Maintenance window schedules and state of checkbox 'Override Maintenance Windows settings' for affected policies?
- SMA Maintenance Windows may be found at Settings > Agents/Plug-ins > Symantec Management Agent > Settings > Maintenance Windows.
- Checkbox 'Override Maintenance Windows settings' can be defined in SWU plugin policies and in each individual SWU policy - please see my screenshots for details.
 
Thank you,
Roman 
 
  
SWUPolicyMW.jpg SWUPluginPolicyMW.jpg
SOLUTION
Charlie D Tran's picture

Yes, we are so dumb.

I used the maintenance window to patch window servers for over a year. Another Windows engineer group also used Altiris to push and install/upgrade monitoring software.  The recently changed the target of maintenance window to broader population. Once mainteance window opened, as you advised, whatever server including in the targets of maintenance window will be impacted.

Per Symantec tech support advice, I turned off the maintenance window. Everything has its own schedule to use for their deployment/work.

Big and hard lesson learn for us.

 

Thanks again very much for your advice, Roman

Charlie Tran