Client Management Suite

Hi there, we have a situation that's holding up our Windows 7 deployment. Fairly clean Windows 7 Pro x64 image, ds7.1 sp1. After the client boots into prod, after sysprep does its thing, managed delivery policies hang on the detection rules... They just sit there. I have a mdp that installs all tier 1 apps in a single policy, but it takes a reboot or two and about 10 minutes or so, before it works. Basically the first detection rule sits there never finishing, which stalls the policy. The logs don't indicate anything earth shatteringly obvious. I have also noticed this occurring on a policy with just 1 package as well. Once again, a reboot or two and a bit of a delay usually works around it, but it's not 100% predictable. This is holding up my windows 7 deployment so any input would be appreciated.

Hi,

What ITMS (or CMS) version do you have installed on the environment where the problem occurs? What events appear in the SM Agent log when detection checks are hanging?

Thank you,

Andrew Shishkov.

Hi Andrew, it's 7.1 SP1. Running CMS. Logs don't show anything strange. I will check them tomorrow though, when I do another test deployment.

Oh, and Andrew... If you have anything else you think I can try in the meantime please advise... Time zones and all that stuff makes it difficult to get things done quickly. I was thinking about pushing out the swmgmt agent via a quick delivery task after the SOE rolls out... But I can't think of anything else.

Can you check if System Temp folder exists on the systems where the problem with hanging detection checks appears?

Thanks,

Andrew.

OK here goes Andrew,

There is no C:\TEMP folder on the client machine at the time that the agent attempts to do run the detection check.  I then notice that the agent appears to restart itself, the server it points to changes (same hostname, just with a :80 on the end).  It then changes the server it points to back to the original name (ie. without :80) and then it appears to run OK.

I have captured the following log entries when the detection rules run

<event date='Apr 27 11:06:37' severity='4' hostName='BLAH-RI44OJO0HP' source='Client Task Agent' module='client task agent.dll' process='AeXNSAgent.exe' pid='1616' thread='2780' tickCount='1510682' >
  <![CDATA[Task: Detection check for PCAnywhere Hot Fix Tech182142-4 has started]]>
</event>
<event date='Apr 27 11:06:37' severity='4' hostName='BLAH-RI44OJO0HP' source='Client Task Agent' module='client task agent.dll' process='AeXNSAgent.exe' pid='1616' thread='2788' tickCount='1510682' >
  <![CDATA[Task: Detection check for PCAnywhere Hot Fix Tech182142-4 has finished]]>
</event>
<event date='Apr 27 11:06:38' severity='1' hostName='BLAH-RI44OJO0HP' source='InventoryRuleCache' module='InventoryRuleAgent.dll' process='AeXNSAgent.exe' pid='1616' thread='3180' tickCount='1511712' >
  <![CDATA[loadResponseAndCheckForError()- Failed to load Inventory Rules Response into XML DOM: Error Code = 0xc00ce014 , Source = Line : 6; Char : 77 Error Description = Element content is invalid according to the DTD/Schema.
Expecting: script, style, meta, link, object, isindex, base.
]]>
</event>
<event date='Apr 27 11:06:38' severity='2' hostName='BLAH-RI44OJO0HP' source='CInventoryRuleAgent' module='InventoryRuleAgent.dll' process='AeXNSAgent.exe' pid='1616' thread='3180' tickCount='1511712' >
  <![CDATA[EvaluateRuleNoFrills() - Rule Guid={8C022E19-A1DF-492C-83CC-D48A78F01CE4} was not found in the cache]]>
</event>
<event date='Apr 27 11:06:38' severity='2' hostName='BLAH-RI44OJO0HP' source='CInventoryRuleAgent' module='InventoryRuleAgent.dll' process='AeXNSAgent.exe' pid='1616' thread='3916' tickCount='1511712' >
  <![CDATA[GetRule() - Rule Guid={4F0E9B30-2E0B-481A-9D64-E56438CBBB7B} was not found in the cache]]>
</event>
<event date='Apr 27 11:06:38' severity='2' hostName='BLAH-RI44OJO0HP' source='CInventoryRuleAgent' module='InventoryRuleAgent.dll' process='AeXNSAgent.exe' pid='1616' thread='3180' tickCount='1511712' >
  <![CDATA[GetRule() - Rule Guid={8C022E19-A1DF-492C-83CC-D48A78F01CE4} was not found in the cache]]>
</event>
<event date='Apr 27 11:06:39' severity='1' hostName='BLAH-RI44OJO0HP' source='InventoryRuleCache' module='InventoryRuleAgent.dll' process='AeXNSAgent.exe' pid='1616' thread='3916' tickCount='1512741' >
  <![CDATA[loadResponseAndCheckForError()- Failed to load Inventory Rules Response into XML DOM: Error Code = 0xc00ce014 , Source = Line : 6; Char : 77 Error Description = Element content is invalid according to the DTD/Schema.
Expecting: script, style, meta, link, object, isindex, base.
]]>
</event>

 

 

Can you check if your NS server is available via HTTP\HTTPs from the client when the error occurs? The error may be caused by problems with connectivity to server or to server overloading (check opening

 http://<NS server>/Altiris/InventoryRuleManagement/Agent/AgentRuleData.ashx page from the client, it should write something like "move along, please - nothing to see here").

The error also may occur in case there is some detection check rule is incorrect (contains some invalid symbols), but this is unlikely the root cause since in that case the problem would not dissappear after computer restart.

Also just in case check if C:\Windows\Temp folder exists on the client (this is default value for System Temp environment variable).

Thanks,

Andrew.

Andrew, Yes there is definitely a c:\windows\temp folder on the machine at all times.

The server is not overloaded. The clients get the policies in the first place so there's no communication issue with the server - the agents get the policies and attempt to execute the tasks associated with the policies, so it can't be communication related.  The issue occurs at a different location as well - this other location has its own task server, whereas the location where I've done most of the diagnosis has no task server so it's all handled by the NS.

But the same problems occurs at both locations.  Same image.

There are only 320 or so agents total.  The NS is dedicated to the task.  So it's not load.

And the system temp environment variable is C:\Windows\Temp as well.

Could it be the version of the agent within the image? 

Should I maybe pull apart the image (it's in Ghost, captured via DS7.1) and inject a new version of the agent?  It's 7.1 SP1. 

When the machine first starts up, it is pointing to SERVERNAME.FQDN:80.  When it gets it first client config, it changes to SERVERNAME.FQDN and it usually takes about 15-30 minutes before it starts executing.  I suspect that it's something to do with the agent possibly needing to update itself before it can execute things?  Maybe?  I don't want to have to re-create the image because it's at a client site and that would necessitate costly changes.

Any other ideas?

What version has Notification Server and what version have SM Agent in the image? Are these versions the same?

>>When the machine first starts up, it is pointing to SERVERNAME.FQDN:80.  When it gets it first client config, it changes to SERVERNAME.FQDN

Does you NS server uses 80 port to communicate with clients or some other port?

Thanks,

Andrew.

How was this resolved?  I am guessing it was indeed a performance or communication issue on the NS, but would like to know for sure how it ended.

It certainly wasn't load (there's only 320 clients, SQL is on a separate box)

And it wasn't communication (using http, the clients are in the same site as the server, clients could resolve server by hostname and IP at any time).

It wasn't the temp folder.

There were no issues with the NS itself.

SP2 and rollup 4 seems to have helped though - it doesn't happen as often now.

But it was definitely a BUG, not a server issue at the customer end.  VERY frustrating and costly in terms of time, but after I spoke to one of the Symantec guys they appeared to be aware of the issue.