Symantec Management Platform (Notification Server)

Altiris 7.1 SP1

This isn't really a patch management issue, it is more of a package server issue I guess.  I managed to fix the problem, but I'd like to have someone tell me why this occurred and if it is likely to occur again (and if it is something that can be prevented).

On January 11th, the January patches came out.  As per normal procedure, I downloaded the patches and set up policies for them.  It wasn't until this week that we actually applied the policies to a filter to push out our first phase of patches.  I go to check a test machine and it looks like the machine is having issues downloading the patches from the package server.  I knew right away not to blame the client because I spent 2-3 days after the SP1 upgrade troubleshooting patch management issues that ended up being a server side problem.

IIS was not running on the package server and would not start.  I saw the event log error:

The worker process for application pool 'DefaultAppPool' encountered an error 'Configuration file is not well-formed XML
' trying to read global module configuration data from file '\\?\C:\inetpub\temp\apppools\DefaultAppPool\DefaultAppPool.config', line number '3'.  Worker process startup aborted.

Ok, fair enough... so I go to the DefaultAppPool.config and see this:

<!-- ERROR: There's been an error reading or processing the applicationhost.config file.  Line number: 318111  Error message: Config section 'system.webServer/handlers' already defined. Sections must only appear once per config file. See the help topic <location> for exceptions
 -->

Ok, I guess that is straightforward enough.  I go into the applicationhost.config file and find that at line 318111, there is indeed a second-time defined handlers section for one virtual directory.  See the snippet form the applicationhost file at the end of this article.  I bolded the two handlers sections for you.  I remove the 2nd instance of the handlers defintion and everything works fine.

I begin to wonder what the heck happened here since I have not really done much with the server for quite a while.  I had noted the last time the applicationhost.config file had been modified was January 11.  Patching time.  That virtual directory is undoubtedly for a patch package that was created for the package server.  So something in the process where Altiris creates a virtual directory for each package (a really messed up away to do things in the first place in my humble opinion) went awry and it defined the handlers twice??? How does this happen?  Why did this happen?  Can I get my 2 hours of troubleshooting back?  Thankfully we have 7.1 in a pilot phase so this didn't really affect a lot of clients (just IT) but it does not really bode well for 7.1 being the magical fix for how terrible 7.0 is.  Is there some weird rain dance I should be doing every time I create new packages to prevent this from happening or will I just need to know that when IIS crashes on the server, it is probably because Altiris hosed the applicationhost.config file?

applicationhost.config:

    </location>
    <location path="Default Web Site/Altiris/PS/{98ce4a68-0fc1-9b55-1f0c-f4d007458d72}">
        <system.webServer>
 <handlers accessPolicy="Read, Script">
                <clear />
                <add name="rules-Integrated" path="*.rules" verb="*" type="System.ServiceModel.Activation.HttpHandler, System.ServiceModel, Version=3.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089" preCondition="integratedMode" />
                <add name="xoml-Integrated" path="*.xoml" verb="*" type="System.ServiceModel.Activation.HttpHandler, System.ServiceModel, Version=3.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089" preCondition="integratedMode" />
                <add name="svc-Integrated" path="*.svc" verb="*" type="System.ServiceModel.Activation.HttpHandler, System.ServiceModel, Version=3.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089" preCondition="integratedMode" />
                <add name="ISAPI-dll" path="*.dll" verb="*" modules="IsapiModule" resourceType="File" requireAccess="Execute" allowPathInfo="true" />
                <add name="TraceHandler-Integrated" path="trace.axd" verb="GET,HEAD,POST,DEBUG" type="System.Web.Handlers.TraceHandler" preCondition="integratedMode" />
                <add name="WebAdminHandler-Integrated" path="WebAdmin.axd" verb="GET,DEBUG" type="System.Web.Handlers.WebAdminHandler" preCondition="integratedMode" />
                <add name="AssemblyResourceLoader-Integrated" path="WebResource.axd" verb="GET,DEBUG" type="System.Web.Handlers.AssemblyResourceLoader" preCondition="integratedMode" />
                <add name="PageHandlerFactory-Integrated" path="*.aspx" verb="GET,HEAD,POST,DEBUG" type="System.Web.UI.PageHandlerFactory" preCondition="integratedMode" />
                <add name="SimpleHandlerFactory-Integrated" path="*.ashx" verb="GET,HEAD,POST,DEBUG" type="System.Web.UI.SimpleHandlerFactory" preCondition="integratedMode" />
                <add name="WebServiceHandlerFactory-Integrated" path="*.asmx" verb="GET,HEAD,POST,DEBUG" type="System.Web.Services.Protocols.WebServiceHandlerFactory, System.Web.Services, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" preCondition="integratedMode" />
                <add name="HttpRemotingHandlerFactory-rem-Integrated" path="*.rem" verb="GET,HEAD,POST,DEBUG" type="System.Runtime.Remoting.Channels.Http.HttpRemotingHandlerFactory, System.Runtime.Remoting, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089" preCondition="integratedMode" />
                <add name="HttpRemotingHandlerFactory-soap-Integrated" path="*.soap" verb="GET,HEAD,POST,DEBUG" type="System.Runtime.Remoting.Channels.Http.HttpRemotingHandlerFactory, System.Runtime.Remoting, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089" preCondition="integratedMode" />
                <add name="CGI-exe" path="*.exe" verb="*" modules="CgiModule" resourceType="File" requireAccess="Execute" allowPathInfo="true" />
                <add name="TRACEVerbHandler" path="*" verb="TRACE" modules="ProtocolSupportModule" requireAccess="None" />
                <add name="OPTIONSVerbHandler" path="*" verb="OPTIONS" modules="ProtocolSupportModule" requireAccess="None" />
                <add name="StaticFile" path="*" verb="*" modules="StaticFileModule,DefaultDocumentModule,DirectoryListingModule" resourceType="Either" requireAccess="Read" />
            </handlers>
            <security>
                <authentication>
                    <windowsAuthentication enabled="true">
                        <providers>
                            <clear />
                            <add value="Negotiate" />
                            <add value="NTLM" />
                        </providers>
                    </windowsAuthentication>
                    <anonymousAuthentication enabled="true" />
                    <digestAuthentication enabled="false" />
                    <basicAuthentication enabled="false" />
                </authentication>
            </security>
            <defaultDocument enabled="false">
                <files>
                    <clear />
                    <add value="Default.htm" />
                    <add value="Default.asp" />
                    <add value="index.htm" />
                    <add value="index.html" />
                    <add value="iisstart.htm" />
                    <add value="default.aspx" />
                </files>
            </defaultDocument>
            <handlers accessPolicy="None" />
        </system.webServer>
    </location>

Now most of the patch packages are working, however a couple of them still will not download on the clients.

I have tried doing a recreate patch package test as well as an update distribution points task but the clients still are unable to find the appropriate patch package (they are stuck on retrying download).

Any advice would be helpful.  I also intend to schedule a long vacation after all of this.

I deleted the policy XML on the package server but that did not resolve the issue.  The packages in question ARE in the xml file however there are not files for them in the "package deliver" folder - the folder w/the corresponding guid does not exist. 

I disabled the origianl patch policies and created new ones after recreating patch packages, but it looks like the issue is that there are a couple of patches that aren't getting from the NS to the package server even though the policy is getting to the package server.  No errors in the log for the package server at this point so it is not trying and failing to download it from the NS, it just doesn't seem to acknowledge that it is in the policy xml.