Video Screencast Help
Protect Your POS Environment Against Retail Data Breaches. Learn More.

You have SMP 7.1, and you think you console is slow, please get in touch!

Created: 02 Aug 2012 • Updated: 15 Nov 2012 | 25 comments
Ludovic Ferre's picture
This issue has been solved. See solution.

Please get in touch with me (via messages or on this post) as we are trying to collect a hard data to instrument console timings and see where the problems come from (I suspect there are many different possible root, but we need the widest possible sample to make the data set as conclusive as possible).

PS: Be aware that I will most likely comeback to you asking for a profiler trace capturing the console opertions. If you are not familiar with the profiler trace please check the following articles.

Getting Started with the Altiris Profiler to Troubleshoot the Altiris Web Console Issues
Using the Altiris Profiler to Research "Blackholes" (Where Time Disappears Between Events)

Comments 25 CommentsJump to latest comment

Ian_C.'s picture

Mike from our team will be in contact shortly to help out.

Please mark the post that best solves your problem as the answer to this thread.
Ludovic Ferre's picture

We are 6 days in and I have received a fair few profiler traces and even had a couple of Webex to have a snoop around.

So, anything interesting to report here? Yes,of course.

On the reporting side

I've seen quite a few slow AltirisWebApplication requests that look strange, reporting a significant start delay (5 to 8 seconds) before anything happened on the Altiris side. Now I'll have to look into the profiler traces again because it occured to me (as I write this comment) that the delay could be in our code somehow (given we must do something to start the profiler timer at t).

On interesting point about these 5~8 second delay is that they are all happening in the /Altiris/Reporting web-application. And this is quite significant. Actually I looked at IIS log files and report loading happens in a "funky" manner, when the browser hits the Report.aspx page it does not received a response but is redirected via http 302 (or 303?) to the same page with an extra parameter in the uri-query. The redirect does not result to a response again but to another redirect that points the browser to the same page again, with an extra parameter appended. Finally the browser hits the page and receives a result, which allow it to move to the final stage, of popsting data to the server to get the result set back from the SMP (if I'm not mistaken),

Here's a quick drawing of all the bouncing, with the initial http 401 if a user access the report interface without having opened a session on the server before that:

Now if you are running a high latency network (with 100millisecond+ delay between the report users and the SMP) you'll notice that the redirect is going to cost, if not an horrible lot of time, at least 4 round trips before any data is sent back to the broswer to display.

On the application pool

I saw a customer with a server being hit very badly and with the w3wp.exe process running at 1.1 GiB of memory and 30~50% cpu utilisation.

Now the performance of the console is bad there, from what was reported to me but this seem to be the by-product of other problems. Namely the Task Servers are hitting the ExecSqlCommand interface but failing to authenticate, so they (almost all of the 40+ task servers) are receiving a http 401 and immediately they try again "ad vitam æternam"...

It looks like a site server problem and we'll tend to that one later this week.

So, that's it for the first interim report. I'll come back to this space or to my blog to add more detailed information soon.

I am currently off-net, on a retreat of some kind. I'll be back real soon, and you sure will hear from me then ;-).

Ludovic FERRÉ
Principal Remote Product Specialist
Symantec

Zac H's picture

It's good to see you guys reaching out to us common folk to somewhat get in front of problems.  Thank you Ludovic.

stevericks's picture

Mine seems a bit better since I set up a database maintenance daily task.

I am continuing to monitor.

 

Steve.

Ian_C.'s picture

@stevericks

Is that a custom maintenance? Or have you followed some Symantec Best Practice? Care to share?

Please mark the post that best solves your problem as the answer to this thread.
carlsson's picture

Hi we have a case open for this, happy to provide logs, in the meantime I'll check our DB maintenance as above

 

Regards  C

Hojiblanca

shudman's picture

We have placed at least a handful of cases, and had backline on for many many hours looking at our slow console performance which was as bad as 3-4 mins to load a page.  This is a new install, although we did upgrade from SP1 to SP2, and then eventually gave in due to replication issues and just re-installed our estate with SP2. The console certainly is better....but very random.  Staying within the Activity Center isn't too bad, but when you hear the devs say "we aim to have every page loaded in 10 seconds" we are a little away from that :(

Switching from Manage/Computers(which takes >20 seconds to load a scope 1000/30000 items), to say, PatchManagement takes in the order of 30-40 seconds

Switching from PatchManagement to Push Agent takes 45 seconds

Now....it is somewhat sporadic and frustrating for Symantec and us as it depends on a) time of day, b) RDP or IE to the NS, c) whether it is a child or the parent, d) some other reason (java version?).  Just annoying that some Ops say "I get good performance" and other say "It is actually worse than 6.x" when both can be hitting the same server.

 

We have done all of the SQL maintenance, and running RAID10 SSDs for the database. We have the Activity Center hotfixes, we have sent more profile traces and logs than I care to mention :(

 

conroyd's picture

Some of our console slowness was solved by clearing the IE cache, and also clearing the Silverlight cache (in Windows 7, you can load the silverlight control panel by searching in the start menu for Silverlight).

This definitely improves page load speeds, when switching between "Jobs and tasks" and "computers".

 

However, there's clearly some back-end issue when it comes to performing console functions such as cloning tasks, moving tasks etc. I just had to clone 4 tasks, and then move them to a new folder. Since you can't select multiple tasks, I have to do them individually, and there is a 5-10 second delay on every action I take. Over all, cloning 4 tasks and moving them to a new folder took over a minute - something that should be as simple as "ctrl-select all the tasks, and drag them to the new folder". This is partially missing console functionality, and partially slowness in performing actions via the right click menu.

 

I have noticed something interesting, which may help you out in your investigation - if I right click a task and rename, it takes MUCH longer to respond than if I just select the task, and rename it in the right-hand panel at the top of the task (that is immediate at all times).

 

 

 

 

Ian_C.'s picture

Has any progress been made with this?

Nobody here has yet mentioned IIS Application pools. I don't know the details, but apparently creating additional pools to split the load does help.

Please mark the post that best solves your problem as the answer to this thread.
Ludovic Ferre's picture

Thanks for pointing this out, it is indeed someething I usually advise and try - having said that it isn't the magic bullet in 7.1 as it was in many cases for 7.0...

For anyone else foloowing this thread, I have not forgotten about it, and I intend to come back to this next week when I'll be back in the office, or as soon as time really permit!

I am currently off-net, on a retreat of some kind. I'll be back real soon, and you sure will hear from me then ;-).

Ludovic FERRÉ
Principal Remote Product Specialist
Symantec

stevericks's picture

I also had a problem where the console would keep popping up an authentication box. I would have to put in my credentials again. This was very annoying and the other day I stumbled accross a HOWTO document which told me to open IIS, go to Default Web Site > Altiris > Console. open Authentication in the right panel, right click on Windows authentication and select providers. Then make sure NTLM is above Negotiate. Also you can put it into trusted sites, but I have not done that yet.

Anyway, it solved my annoying authentication pop up box and (this is my main point) speeded up the console.

Steve.

DanC@BYU's picture

We had some significant slowness on our console and finally found some cache settings we could modify when we ran the SSE Tools and did a core settings config scan. Here were the settings we changed and the default values. We are running ITMS 7.1.2 with Rollup 4.

C:/ProgramData/Symantec/SMP/Settings/CoreSettings.config

 

- ItemCacheSize (at line number 195) - currently has value: 3000, the recommended value is: 10000

- StringCacheSize (at line number 313) - currently has value: 10000, the recommended value is: 50000

- SecurityCacheSize (at line number 416) - currently has value: 50000, the recommended value is: 100000

 

We stumbled upon the solution while trying to deal with NSE issues.

http://www.symantec.com/business/support/index?page=content&id=TECH183347

shudman's picture

Dan - we already have those settings across all of our NSes, and that unfortunately has not improved things by any margin.

Ludovic - anything you have found ?

RufusS's picture

Hi,

we also have a slow console some days, I can see w3wp.exe uuses the most processing and it is cuasing the server to consume 100% CPU at times. Our server has 16 GB RAM but only uses about 3 -- 6 GB. the w3wp.exe process is constantly sitting at 1.1 GB memory usage is there a way to optimize the IIS worker proccess?

Ludovic Ferre's picture

Hello everyone,

I have been taken with my own customer problems and could not report back, but with this last update I decided to try summarise some of the things I have seen, and what remains to be seen.

First, and foremost, there is a base line that everyone should be running - in short a profiler trace on the AltirisWebApplication requests and SQL taking more than 3,000 millisecond.

This, should tell us if there is room for improvment on the SMP server or on the SQL.

Now, I Can't devellop this idea right now _but_ I will finish the Profiling article I have in draft and add information directly related to the slow console "problems".

I am currently off-net, on a retreat of some kind. I'll be back real soon, and you sure will hear from me then ;-).

Ludovic FERRÉ
Principal Remote Product Specialist
Symantec

Ludovic Ferre's picture

I was at the French User group in Paris yesterday and received some interesting feedback from a customer (Francois P) that was later repeated by Carsten Giese, our Subject Matter Expert.

If you have a slow SMP console make asnd only one site server (on the SMP) make sure the  server is offloaded to another server, as anything above 500 clients on the SMP Task Server will hinder the console and IIS operation dramatically.

The best option to do this is to clone the "Task Service Settings" under (I'm sure you guessed ;-) "Site Management > Settings > Task Service > Settings" :D.

The cloned policy will be used for the SMP only, so you can rename it to something meaningfull - like "SMP Task Service Settings". Then change the target to make sure the policy applies to the SMP alone, and set the "Maximum number of computers to manage per Task Server" to 500 or 100 (lowest value available on the drop-down list).

Finally, change the target on the "Task Service Settings" to exclude the SMP from it, so it doesn't get 2 conflicting policies.

With that done, you should be one step closer to have a faster console.

I am currently off-net, on a retreat of some kind. I'll be back real soon, and you sure will hear from me then ;-).

Ludovic FERRÉ
Principal Remote Product Specialist
Symantec

Ludovic Ferre's picture

A final comment on this?

Well, it looks to me like 2 elements alone will greatly help most users with a slow console:

  1. Make sure your SMP site server is loaded to no more than 100 computers
  2. Implement split IIS application pools to extend the memory consumption and avoid .net memory issues

Let's hope this closes the subject :D.

PS: For users who's SMP do not have Internet access remember that there the Certificate Revocation List issue that causes the CLR to hang on some slow timouts (many seconds) before loading assemblies due to this. So implement the .Net config changes that are well documented here on connect and the Symantec KB.

I am currently off-net, on a retreat of some kind. I'll be back real soon, and you sure will hear from me then ;-).

Ludovic FERRÉ
Principal Remote Product Specialist
Symantec

Ian_C.'s picture

Symantec KB #82344 article posted. (I'll thank Ludovic):

How to implement custom application pools on the SMP to improve web-console performance

 

Please mark the post that best solves your problem as the answer to this thread.
Ludovic Ferre's picture

Did you recognize the style (or typoes ;-) or you smartly guessed - given I'm so onto the subject?

I am currently off-net, on a retreat of some kind. I'll be back real soon, and you sure will hear from me then ;-).

Ludovic FERRÉ
Principal Remote Product Specialist
Symantec

Ian_C.'s picture

- given I'm so onto the subject?

I smartly guessed.

 

(or typoes ;-)

Didn't look at the style. Typo's does not have an E. cool

Please mark the post that best solves your problem as the answer to this thread.
Ludovic Ferre's picture

And there's the same here on Connect, with nicer formatting:

https://www-secure.symantec.com/connect/articles/how-implement-custom-application-pools-smp-improve-web-console-performance

 

I am currently off-net, on a retreat of some kind. I'll be back real soon, and you sure will hear from me then ;-).

Ludovic FERRÉ
Principal Remote Product Specialist
Symantec

Ludovic Ferre's picture

Here's anohter KB that points to the 3 other artciles, from the main issues discussed here:

Symantec Management platform slow console README

 

 

I am currently off-net, on a retreat of some kind. I'll be back real soon, and you sure will hear from me then ;-).

Ludovic FERRÉ
Principal Remote Product Specialist
Symantec

Stefan S.'s picture

Hi Ludovic,

I have set up a cloned SMP Task Service Settings for all the notficiation servers and exclude the NSs from the default setting. However the client count has not changed since changing the setting. What is the related setting for the "Task Agent Settings" -->

When multiple Task Servers are available at a site:

Choose the Task Server relative to the remaining capacity of each server

Choose the Task Server to which the agent has the fastest connection

Choose the Task Server with the fewest computers currently connected

?

I guess it should be the first one? I have currently the 2nd one selected.

Or does it not matter because I have the setting ticked in SMP Task Service Settings to "Allow maximum computers to be exceeded if a computer cannot connect to any other server"?

Stefan

Ludovic Ferre's picture

Hello Stefan,

Make sure the option to "Allow max... exceeded" is not checked on the SMP policy, as this will nill the speration procedure (and 100 computers limit).

The first 3 options should not cause an issue with the split configurations.

I am currently off-net, on a retreat of some kind. I'll be back real soon, and you sure will hear from me then ;-).

Ludovic FERRÉ
Principal Remote Product Specialist
Symantec