Client Management Suite

We are currently running 7.1 MP1 and all of our clients are currently getting this error message.  Things were fine with around 3000 agents, but we recently expanded to 6000 this week and that is when the problem started.  We increased the check in times on everything and disabled many of our policies to see if that would help.  We also tried increasing the max concurrent config requests but that hasn't helped.  I have verified after recycling the app pool for the NS agent that I can manually retrieve the client policies via a web browser on one of our clients, albeit it is VERY slow.  Right now our database is hosted on a separate server but resource usage there looks fine, as does the resource usage on our NS.  This seems to be an IIS performance issue, any assistance would be greatly appreciated!

Sometimes we will also get the "maximum number of client policy requests has been reached"

Hi Admobandy,

The default request interval to download new configuration settings is one hour. For example, 5,000 managed computers make 120,000 total requests to Notification Server each day. However, by adjusting the setting to every two hours, you reduce the number of requests to 60,000.

The more frequently your managed computers request a new configuration, the more total load is placed on the Notification Server computer’s resources and "maximum number of client policy requests has been reached" will occur.

Could you please verify how frequently the managed computers are requesting configuration? By adjusting this setting to every three or four hours can help you to decrease this error message.

By default 10 is the value of maximum concurrent configuration requests. Could you please verify it using NSConfigurator tool? The best practice is to use the default settings for maximun concurrent configuration requests.

Regards,

Fábio Sanches

Our configuration update policy is currently set to update every 4 hours and our max concurrent requests is currently set at 50.

If I restart IIS I can manually update a few agents by hand, after about 20 minutes though it is back to the state of "the maximum number of client policy requests has been reached."

Hi admobandy,

Do you know if any configation change was made in your IIS?

Also verify the IIS logs when "the maximun number of client policy requests has been reached" started: http://www.symantec.com/docs/TECH12706

If you found errors message on IIS logs, please let us know.

Also debug-mode compiling can cause performance problems for any ASP.NET application. In particular, the following debug-mode behaviors impact Notification Server performance. Execute the DisableDebug utiity to improve IIS performance: http://www.symantec.com/docs/HOWTO6940

Regards,

Fábio Sanches

Looks like we already have debug mode set to false for the web config.

In the IIS log file I see a large number of GETS and POSTS, but no error messages in regards to clientconfigpolicies.aspx failing for anything... but clients show the "max number of policy requests has been reached."

It looks like some of the connecting devices are being returned a 401 error

So we removed 3000 machines from our environment by disabling the symantec management agent on them and the remaining 3000 machines seem to be able to check in and get policy updates without issue now.  The intersting thing is I don't see any difference in CPU / Memory or Network utilization on the database server or the NS.

This really seems like some sort of IIS issue...

It seems to me we are piling up a queue for configuration requests... our environment will eventually need to handle 12,000 clients...  The interesting thing is that everything else seems to work fine, it's just the configuration requests.

I added more worker process' to the Classic.netAppPool and it looks like we may be getting through this now... We've turned our agents back up to over 6000 active clients and had all 4 cores pegged for about 30 minutes here.  Thing have greatly diminished now and agents seem to be checking in fine.

Another posible solution is configure a separate App Pools to configure more worker process only to critical services: https://www-secure.symantec.com/connect/articles/how-implement-custom-application-pools-smp-improve-web-console-performance

Regards,

Did that already some time ago... we didn't notice much of a difference in console performance, and also noticed that a couple things no longer functioned correctly within the console so we moved most of it back to a single APP Pool...

We're going to fill the other cpu socket on this server and follow this from Microsoft and re-test performance... http://msdn.microsoft.com/en-us/library/ms998549.aspx

We reduced the number of active agents again and this still appears to be an issue even with a greatly reduced number of active agents.

So, more information on this.  I've taken a look at the IIS logs, I used a powershell script to just pull out the client config update requests and I'm seeing something quite interesting.

Even though the maxconcurrentconfigrequests is set to 10 I am seeing IIS only service a single request at a time.

An entire days worth of client config request time stamps looks like this...

00:03:08
00:03:13
00:03:40
00:03:43
00:03:47
00:04:32
00:04:37
00:05:05
00:05:18
00:05:28
00:05:40
00:05:41
00:05:53
00:06:04

So in a 3 hour time period the NS only serviced 14 total client config update requests.  Within a 4 hour period we need ALL of our machines serviced...  CPU utlization on the NS and database are low, so this seems to be a serialized vs parallelized request issue...  Any thoughts?

This seems to be better now that we've rebuilt our NS, however I did find one more intersting thing with this.  If the client data/time is off/wrong the agent will not update its client config.  Something to be mindful of if you have power users and staff that like to change system dates and times and you are a novell/microsoft mixed shop like we are right now and novell logins do not seem to care as much as microsoft logins.

If your client is off by more than 5 minutes Kerberos and Windows will definitely care.  Good catch.

• Ensure that Integrated Windows authentication has been enabled for the web service in IIS if applicable
• Ensure that ASP.NET, Authenticated Users, and Internet Guest Account all have Read permissions to the parent folder and containing files of the associated web service
• If the above items appear to be in order, then there may be a problem with the Internet Guest Account. Steps may need to be performed to reconfigure and re-sycnhronize the IUSR and IWAM accounts and their passwords

To Reset IUSR password:

Steps from 3rd party website to resolve IUSR account issues:

1. Open AD Users & Computers. Expand the Users OU, right-click on the user account (iusr_servername/iwam_servername) and select 'Reset password' to reset the password. Note: the password can't be blank.

2. Open this User Account's properties and make sure that 'Password never expires' and 'User cannot change password' are selected.

3. Open IIS from Administrative Tools.

4. Expand servername>Web Sites

5. Right-click on 'Default Web Site' and select Properties.

6. Go to the 'Directory Security' tab and click the Edit button under 'Authentication & Access Control'

7. Enter the new password you just reset for the IUSR_servername account and click OK.

8. Enter the password again to confirm and click OK.

9. Click OK.

10. Open a command prompt and run IISRESET

11. At the command prompt, enter the following commands:

1) cd

2) c:\inetpub\adminscripts adsutil SET w3svc/WAMUserPass “password” (Where password = the password you entered for the IWAM_servername account in AD Users & Computers)

3) c:\windows\system32\cscript.exe "c:\inetpub\adminscripts\synciwam.vbs" –v

4) IISRESET