Client Management Suite

Problem

After recent Hierarchy migration several mac laptops are running very slow, fans running constantly and processor utilization beyond 100%. Laptop battery depletes quickly and machine overheats, often shutting down. Activity Monitor shows aexpluginmanager.bin process using 100% CPU. If you force quit the process it resumes and runs at 100% CPU again. So far the only thing that has resolved the issue is uninstalling and reinstalling Altiris agent.

For the Hierarchy migration we used Targeted Agent Settings with the Child servers addresses and not https. The majority of clients are functioning normally.

Has anyone else seen this on Macs?

How best to troubleshoot Client Task Agent errors and hangs?

Log Sample

/opt/altiris/notification/nsagent/var/aex-client.log
ERRO 2013-04-19 11:47:19.946      9535    2957635584  ClientTaskScheduleAgentPlugin > Failed to execute task 'com.altiris.cta-sched.{8918C4B8-F6D0-45C3-BCB9-4628D264DA20}': 2182086757, The Client Task Agent is currently shutting down
ERRO 2013-04-19 11:59:13.599      9535    2956554240  SMF-Task > Failed to execute client task {AEACAAC3-2415-4EB0-897F-C196C5256B5E}, {B42314A9-DCF7-459D-8B97-00F7892A0430} The Client Task Agent is currently shutting down
ERRO 2013-04-19 11:59:13.994      9535    2956554240  SMF-Task > Failed to execute client task {715D33EC-CEF7-460A-90A7-AFEDB4F733DE}, {B78E6171-5E51-431F-9E2F-58B9E5EAD071} The Client Task Agent is currently shutting down
ERRO 2013-04-19 11:59:14.481      9535    2955472896  EventQueue > Error: EventQueue(TrySendEvent): Cannot Create Object: Altiris.AeXNSManager
ERRO 2013-04-19 11:59:14.481      9535    2955472896  EventQueue > Error: EventQueue(TrySendEvent): Cannot Create Object: Altiris.AeXNSManager
ERRO 2013-04-19 11:59:14.485      9535    2955472896  EventQueue > Error: EventQueue(TrySendEvent): Cannot Create Object: Altiris.AeXNSManager
ERRO 2013-04-19 11:59:14.485      9535    2955472896  EventQueue > Error: EventQueue(TrySendEvent): Cannot Create Object: Altiris.AeXNSManager
ERRO 2013-04-19 11:59:14.486      9535    2955472896  EventQueue > Error: EventQueue(TrySendEvent): Cannot Create Object: Altiris.AeXNSManager
ERRO 2013-04-19 11:59:14.487      9535    2955472896  EventQueue > Error: EventQueue(TrySendEvent): Cannot Create Object: Altiris.AeXNSManager
ERRO 2013-04-19 11:59:14.487      9535    2955472896  EventQueue > Error: EventQueue(TrySendEvent): Cannot Create Object: Altiris.AeXNSManager
ERRO 2013-04-19 11:59:14.490      9535    2955472896  EventQueue > Error: EventQueue(TrySendEvent): Cannot Create Object: Altiris.AeXNSManager
ERRO 2013-04-19 11:59:14.490      9535    2955472896  EventQueue > Error: EventQueue(TrySendEvent): Cannot Create Object: Altiris.AeXNSManager
ERRO 2013-04-19 11:59:14.490      9535    2955472896  EventQueue > Error: EventQueue(TrySendEvent): Cannot Create Object: Altiris.AeXNSManager
ERRO 2013-04-19 11:59:14.491      9535    2955472896  EventQueue > Error: EventQueue(TrySendEvent): Cannot Create Object: Altiris.AeXNSManager
ERRO 2013-04-19 11:59:14.754      9535    2955472896  EventQueue > Error: EventQueue(TrySendEvent): Cannot Create Object: Altiris.AeXNSManager
ERRO 2013-04-19 11:59:14.754      9535    2955472896  EventQueue > Error: EventQueue(TrySendEvent): Cannot Create Object: Altiris.AeXNSManager
ERRO 2013-04-19 11:59:14.831      9535    2955472896  EventQueue > Error: EventQueue(TrySendEvent): Cannot Create Object: Altiris.AeXNSManager
ERRO 2013-04-19 11:59:14.843      9535    2955472896  EventQueue > Error: EventQueue(TrySendEvent): Cannot Create Object: Altiris.AeXNSManager
ERRO 2013-04-19 11:59:14.845      9535    2955472896  EventQueue > Error: EventQueue(TrySendEvent): Cannot Create Object: Altiris.AeXNSManager
ERRO 2013-04-19 11:59:14.884      9535    2955472896  EventQueue > Error: EventQueue(TrySendEvent): Cannot Create Object: Altiris.AeXNSManager
ERRO 2013-04-19 11:59:14.885      9535    2955472896  EventQueue > Error: EventQueue(TrySendEvent): Cannot Create Object: Altiris.AeXNSManager
ERRO 2013-04-19 11:59:15.216      9535    2956554240  ClientScheduler > Failed to connect to the Client Scheduler plug-in: 2164785203, The agent is currently shutting down
ERRO 2013-04-19 11:59:15.216      9535    2956554240  ClientScheduler > Failed to connect to the Client Scheduler plug-in: 2164785203, The agent is currently shutting down
ERRO 2013-04-19 11:59:15.282      9535    2956554240  ClientScheduler > Failed to connect to the Client Scheduler plug-in: 2164785203, The agent is currently shutting down
ERRO 2013-04-19 11:59:15.346      9535    2956554240  ClientScheduler > Failed to connect to the Client Scheduler plug-in: 2164785203, The agent is currently shutting down
ERRO 2013-04-19 11:59:15.360      9535    2956554240  ClientScheduler > Failed to connect to the Client Scheduler plug-in: 2164785203, The agent is currently shutting down
ERRO 2013-04-19 11:59:15.360      9535    2956554240  ClientScheduler > Failed to connect to the Client Scheduler plug-in: 2164785203, The agent is currently shutting down
ERRO 2013-04-19 11:59:15.374      9535    2956554240  ClientScheduler > Failed to connect to the Client Scheduler plug-in: 2164785203, The agent is currently shutting down
ERRO 2013-04-19 11:59:15.374      9535    2956554240  ClientScheduler > Failed to connect to the Client Scheduler plug-in: 2164785203, The agent is currently shutting down
ERRO 2013-04-19 11:59:15.377      9535    2956554240  ClientScheduler > Failed to connect to the Client Scheduler plug-in: 2164785203, The agent is currently shutting down
ERRO 2013-04-19 11:59:15.377      9535    2956554240  ClientScheduler > Failed to connect to the Client Scheduler plug-in: 2164785203, The agent is currently shutting down
ERRO 2013-04-19 11:59:15.380      9535    2956554240  ClientScheduler > Failed to connect to the Client Scheduler plug-in: 2164785203, The agent is currently shutting down
ERRO 2013-04-19 11:59:15.380      9535    2956554240  ClientScheduler > Failed to connect to the Client Scheduler plug-in: 2164785203, The agent is currently shutting down
ERRO 2013-04-19 11:59:15.383      9535    2956554240  ClientScheduler > Failed to connect to the Client Scheduler plug-in: 2164785203, The agent is currently shutting down
ERRO 2013-04-19 11:59:15.383      9535    2956554240  ClientScheduler > Failed to connect to the Client Scheduler plug-in: 2164785203, The agent is currently shutting down
ERRO 2013-04-19 11:59:15.389      9535    2956554240  ClientScheduler > Failed to connect to the Client Scheduler plug-in: 2164785203, The agent is currently shutting down
ERRO 2013-04-19 11:59:15.648      9535    2956554240  ClientScheduler > Failed to connect to the Client Scheduler plug-in: 2164785203, The agent is currently shutting down
ERRO 2013-04-19 11:59:16.161      9535    2956554240  ClientScheduler > Failed to connect to the Client Scheduler plug-in: 2164785203, The agent is currently shutting down
ERRO 2013-04-19 12:06:14.123       261    2956546048  SMF-Task > Failed to execute client task {5A237012-708F-4532-A630-B3600752FB62}, {A098C1DE-3DB9-4003-AA0B-D88A37691394} The Client Task Agent is currently shutting down
ERRO 2013-04-20 16:20:02.239       261    2957897728  PolicyManager > Server URLs are not valid.
ERRO 2013-04-21 16:21:02.927       261    2955202560  PolicyManager > Server URLs are not valid.
ERRO 2013-04-22 10:10:36.209       292    2955472896  SMF-Task > Failed to execute client task {7BFD471D-8E45-4FAA-A36B-970843AE337E}, {27046E2E-BE3B-4F6D-A4C8-F148E9DDC3DB} The Client Task Agent is currently shutting down
ERRO 2013-04-24 09:26:04.245       329    2956554240  SMF-Task > Failed to execute client task {902DEECC-B4FC-4A28-BA42-73A5ADF5C4B3}, {089B8FFE-DC30-4D07-A7E7-704943519B50} The Client Task Agent is currently shutting down
ERRO 2013-04-25 10:56:04.129       267    2956275712  PolicyManager > Server URLs are not valid.
ERRO 2013-04-26 09:23:11.879       269    2953588736  SMF-Task > Failed to execute client task {8C164B08-1BF7-4D0D-B9B4-8A44B52D7885}, {F0B801BD-3224-482A-8B24-AD2395143DE2} The Client Task Agent is currently shutting down
ERRO 2013-04-26 09:23:12.475       269    2953588736  SMF-Task > Failed to execute client task {902DEECC-B4FC-4A28-BA42-73A5ADF5C4B3}, {089B8FFE-DC30-4D07-A7E7-704943519B50} The Client Task Agent is currently shutting down

Hi Luke,

Seems that root cause is that agent cannot resolv server it is targeted to. As  a result agent cannot run scheduled tasks and smf jobs it has in cache.But in any case it should not use 100% cpu.

Can you confirm if agent is able to resolve server?

Full agent log(in devnote mode) and core dump may help to investigate 100% cpu usage deeper.

Thanks,

Andrei

Thanks Andrei,

In all the cases my group has had so far, the agent has been reinstalled and is again functioning normally. But I have requested that next instances be presented in a broken state so I will try pinging and uploading basic inventory when I get the next one.

I have switched the logging to devnote, so that piece is in place. 

I was curious what you make of the log line "The Client Task Agent is currently shutting down" This is leading me to question the CTA and try and restart it or take steps short of a full client reinstall.

I appreciate your insights.

Luke

Luke,

Be carefull with devnote - as it is used only for debugging purposes( it generates a lot of log entries and log file can be increased to several GBs during 1-2 days).

Also can you please specify agent version?

Thanks,

Andrei

With devnote set, wont the log still be limited in size by the debug_file_size=1024 ?

Oh yes it is limited... Forgot about this since I use devnote in conjunction with unlimited debug file size...it produces a lot of entries and can overwrite log files quickly loosing needed info. It will be more usefull to set devnote only on affected machine  with unlimited debug file size. 

As a step toward figuring out what exactly is failing I created a script that will change the logging level and gather a sample then revert the logging level back. The logs then get emailed to my team.

One idea I had was to have this script live on the laptops and be triggered by aexpluginmanager.bin hitting 100%, but that may happen periodically, giving a false positive...

Update:

To cut down on the high call volume we were getting about this issue, I automated a "Health Check" with a launchd item, and a script.

The launchd that samples the aexpluginmanager's CPU use. If the average over 3 consecutive samples is above 100% a script stops the client, clears the /policies folder in /opt, sends me zipped logs, and restarts the client or reinstalls as needed.

This has been working to catch the runaway aexpluginmanager process and correct it before the customer realizes there is an issue. So my help desk wont get as many calls. We have had over a hundred so far about this issue.

Root cause? Engineers do not know but 7.5 is supposed to fix it, sigh

It is believed that the pcanywhere plugin is installed on these clientn macs. That plugin is not supported on the current platform and OS version. Uninstalling that plugin has been shown to resolve the cpu utilization issue on macs. Remove the mac clients from any pcanywhere install policies and either enable the pcanywhere uninstall policy for these macs or run the following command on the macs: "/private/var/root/Desktop/pcAMacAgent/rollout_Mac.sh -uninstall".