Symantec Management Platform (Notification Server)

I am getting lots of Request timed out errors in the logs after upgarding to 7.5 SP1 HF3

------------------------------------------------------------------------------------------------------------------------------------------------------------

"2/18/2015 6:09:30 PM","Request timed out.
   [System.Web.HttpException @ ]

Exception logged from:
   at Altiris.Diagnostics.Logging.EventLog.ReportException(Int32 severity, String strMessage, String category, Exception exception, String footer)
   at Altiris.NS.Logging.EventLog.ReportException(Int32 severity, String strMessage, String category, Exception exception)
   at Altiris.Web.NS.Agent.Global.Application_Error(Object sender, EventArgs e)
   at System.Web.HttpApplication.RaiseOnError()
   at System.Web.HttpApplication.PipelineStepManager.ResumeSteps(Exception error)
   at System.Web.HttpApplication.BeginProcessRequestNotification(HttpContext context, AsyncCallback cb)
   at System.Web.HttpRuntime.ProcessRequestNotificationPrivate(IIS7WorkerRequest wr, HttpContext context)
   at System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr managedHttpContext, IntPtr nativeRequestContext, IntPtr moduleData, Int32 flags)
   at System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr managedHttpContext, IntPtr nativeRequestContext, IntPtr moduleData, Int32 flags)
   at System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr managedHttpContext, IntPtr nativeRequestContext, IntPtr moduleData, Int32 flags)
   at System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr managedHttpContext, IntPtr nativeRequestContext, IntPtr moduleData, Int32 flags)

**CEDUrlStart** :http://entced.symantec.com/entt?product=SMP&version=7.5.3153.0&language=en&module=BED4YbVj2VVWOQ6qhsktBbQXcs9e2yKi/SKgk4h/AWaZ7Z89oisMOsqahut3Wi5v&error=-2098933893&build=**CEDUrlEnd**
","Altiris.Web.NS.Agent.Global.Application_Error","w3wp","6"

------------------------------------------------------------------------------------------------------------------------------------------------------------

Hi Emil-Jose,

This KB is the same case of your problem?

http://www.symantec.com/docs/TECH211781

• Web Site and Altiris Services are running on upgraded NS?

Thanks,

IP.

Hi Igor,

Thank you for your reply.

Web Site and Altiris Services are well running, So I doubt TECH211781 is applicable in this case.

Regards,

Emil Jose

1. Otherwise, your Symantec Management Agent on NS and on managed endpoints aren't able to receive policies?
2. Or what impact you're seeing in environment, due these errors on NS side?
3. Does this error message still appears in log?

Hi Igor,

Thank you for your reply.

Yes I can confirm the Agents are able to receive policies.

Impact wise..

1) Server performance is very bad as there are many connection attempts. Refer the attachment

2) Clients are taking more time to redirect this SMP. And most of the time it is failing with error “HttpRequest::ReadHeaders error. Bad SMP server version” and after many attempts it may success.

3) Request timed out errors still appearing, Avg 8 per Minute.

Regards,

Emil Jose

Hi Emil,

could you please provide other details, such as:

1. How did you perform upgrade from 7.1.x to 7.5 SP1 HF3?
2. How much managed endpoints you have?
3. What products/suite you're using?
4. There is a lot of Patch/Managed Delivery policies assigned to endpoints?
5. What is Windows OS, where upgraded NS is running?

Also here is a some links about problems on NS side after upgrade:

• https://www-secure.symantec.com/connect/forums/cpu-usage-high-post-sp1-upgrade

Check here latest comments about CoreSettings.config:

• https://www-secure.symantec.com/connect/blogs/it-management-suite-75-sp1-hf5-now-available

Please open SIM and check list of installed products there, whether there are properly installed, or there are some products, which have message "Not properly configured"?

Thanks,

IP.

Hi Igor,

Thank you for your reply.

• How did you perform upgrade from 7.1.x to 7.5 SP1 HF3?

7.1 SP2 MP 1.1 RU8 to 7.5 SP1 + HF3

• How much managed endpoints you have?

20K per SMP

• What products/suite you're using? 

Only CMS (We are using SWD + PMS + INV + pcA Solutions)

• There is a lot of Patch/Managed Delivery policies assigned to endpoints?

Yes we do have many

• What is Windows OS, where upgraded NS is running? 

2K8 R2 EE SP1

I did go through SIM all looks fine.

SingletonPolicyRequest vlaue is set to False now...

 <!-- "1" to allow only single cacheable policy (of each unique key) to be generated in parallel, "0" for legacy behavior  -->
  <customSetting key="SingletonPolicyRequest" type="local" value="False" />

I have also applied..
http://blogs.msdn.com/b/tmarq/archive/2007/07/21/asp-net-thread-usage-on-iis-7-0-and-6-0.aspx

Regards,

Emil Jose

Hi Emil,

make sure that your CoreSettings.config has these values as they are shown here (0 or 1 but not a False).

1. According to http://www.symantec.com/docs/TECH222804  "SingletonPolicyRequest" should be = "1"

• <customSetting key="SingletonPolicyRequest" type="local" value="1" />

2. According to http://www.symantec.com/docs/TECH225352 "StubbedPolicyCache" should be = "0"

• <customSetting key="StubbedPolicyCache" type="local" value="0" />

3. Default value of "ClientPolicyCacheMaxSize" in 7.5 SP1 HF5 is "20000":

• "ClientPolicyCacheMaxSize" type="local" value="20000" />

Open and manually change these values (If they aren't like they are shown above) in CoreSettings.config on upgraded 7.5 SP1 HF3 server ⇒ save changes.

• <customSetting key="SingletonPolicyRequest" type="local" value="1" />
• <customSetting key="StubbedPolicyCache" type="local" value="0" />

Thanks,

IP.

Hi Igor,

Thank you for your reply.

I have now chaged the values, Refer the below

Before
<customSetting key="SingletonPolicyRequest" type="local" value="False" />
<customSetting key="StubbedPolicyCache" type="local" value="0" />

Now
<customSetting key="SingletonPolicyRequest" type="local" value="1" />
<customSetting key="StubbedPolicyCache" type="local" value="0" />

Regards,

Emil Jose

Hi Emil,

I've compared some core settings value from your CoreSettings.config with CoreSettings.config after clean installation of SMP 7.5 SP1 HF5 and there are differences.

Here is compared list so you can analyze them and decide whether you will use the same values r not.

Customer CoreSettings after on-box upgrade of NS from 7.1.x to 7.5 SP1 HF3:
<customSetting key="DBMaxPoolSize" type="local" value="400" />
<customSetting key="SlowQueueThreshold" type="local" value="3000000" />
<customSetting key="LargeQueueThreshold" type="local" value="10000000" />
<customSetting key="MaxConcurrentPriorityMsgsThreadPoolSize" type="local" value="5" />
<customSetting key="MaxConcurrentFastMsgsThreadPoolSize" type="local" value="10" />
<customSetting key="MaxConcurrentDefaultMsgsThreadPoolSize" type="local" value="5" />
<customSetting key="MaxConcurrentSlowMsgsThreadPoolSize" type="local" value="1" upgradeIf="gt" />
<customSetting key="MaxConcurrentPackageInfoRequests" type="local" value="10" />
<customSetting key="NSMessageFilterSubscriberCacheSize" type="local" value="1000" />
<customSetting key="NSMessageFilterSubscriberCacheAbsoluteExpiry" type="local" value="300000" />

Default CoreSettings.config values from clean installed "SMP 7.5 SP1 HF5"
<customSetting key="DBMaxPoolSize" type="local" value="200" />
<customSetting key="SlowQueueThreshold" type="local" value="5000000" />
<customSetting key="LargeQueueThreshold" type="local" value="20000000" />
<customSetting key="MaxConcurrentPriorityMsgsThreadPoolSize" type="local" value="16" />
<customSetting key="MaxConcurrentFastMsgsThreadPoolSize" type="local" value="16" />
<customSetting key="MaxConcurrentDefaultMsgsThreadPoolSize" type="local" value="8" />
<customSetting key="MaxConcurrentSlowMsgsThreadPoolSize" type="local" value="4" upgradeIf="gt" />
<customSetting key="MaxConcurrentPackageInfoRequests" type="local" value="20" />
<customSetting key="NSMessageFilterSubscriberCacheSize" type="local" value="5000" />
<customSetting key="NSMessageFilterSubscriberCacheAbsoluteExpiry" type="local" value="900000" />

Do you have any update for this error "Request timed out. [System.Web.HttpException @ ]" ? Now it works correctly and no such errors in log?

Thanks,

IP.

I am getting the same type of error messages in my CMS 7.5 SP1 HF5 environment.  Unfortunately the items identified previously in this string have been checked, but the errors continue every few mins.

Request timed out.

[System.Web.HttpException @ ]

Exception logged from:

at Altiris.Diagnostics.Logging.EventLog.ReportException(Int32 severity, String strMessage, String category, Exception exception, String footer)

at Altiris.NS.Logging.EventLog.ReportException(Int32 severity, String strMessage, String category, Exception exception)

at Altiris.Web.NS.Agent.Global.Application_Error(Object sender, EventArgs e)

at System.Web.HttpApplication.RaiseOnError()

at System.Web.HttpApplication.PipelineStepManager.ResumeSteps(Exception error)

at System.Web.HttpApplication.BeginProcessRequestNotification(HttpContext context, AsyncCallback cb)

at System.Web.HttpRuntime.ProcessRequestNotificationPrivate(IIS7WorkerRequest wr, HttpContext context)

at System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr managedHttpContext, IntPtr nativeRequestContext, IntPtr moduleData, Int32 flags)

at System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr managedHttpContext, IntPtr nativeRequestContext, IntPtr moduleData, Int32 flags)

at System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr managedHttpContext, IntPtr nativeRequestContext, IntPtr moduleData, Int32 flags)

at System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr managedHttpContext, IntPtr nativeRequestContext, IntPtr moduleData, Int32 flags)

**CEDUrlStart** :http://entced.symantec.com/entt?product=SMP&version=7.5.3153.0&language=en&module=BED4YbVj2VVWOQ6qhsktBbQXcs9e2yKi/SKgk4h/AWaZ7Z89oisMOsqahut3Wi5v&error=-2098933893&build=**CEDUrlEnd**

Hi smccue,

How many days this problem still occurs, when you've changed CoreSettings values, mentioned above?

If possible, then could you please provide some other information:

• How much managed endpoints you have in your environment?
• Mostly you're using Patch Management or Software Management Solution?
• How much Patch Update policies and Managed Delivery policies you have?

Thanks,

IP.