Client Management Suite

Where do I find filters that have been applied to Agent Settings, policies, etc.? I'm trying to find a filter that has a wrong server name in it somewhere. The filter is called "All Desktop Computers (excluding Site Servers)", and the SMA Settings policy points to it.

I'm trying to troubleshoot all the heavy traffic coming from my agents to the server. We recently migrated from NS 6.x  to SMP 7.1. Once we got the last agents over to the new server, I started noticing a big performance hit on the server. I checked the EventQueue folders, and there were about 200k files waiting to be processed. There were at least that many sitting in all of the "bad" folders. The server guys weren't too thrilled that these NSIs had filled up their C: drive, but that's a different issue alltogether.

I've gone through all the policies and cut back all inventory/task reporting to once a week (some are still set to once a month), but I'm still getting thousands of NSIs being processed. I sent out a DS job that removed all servers, and then set the server to the new server to wipe out the server list/queue. I also sent out a DS job that deletes the contents of the C:\Program Files\Altiris\Inventory\Outbox and NSI folders. Even with those folders being empty, I'm still getting NSIs that appear to be from those computers.

If I knew how to convert this - <TimeGenerated>20110726141613.783000+300</TimeGenerated> - into something legible, I'd know if the files were from today or some other time, although some do have an readable time - DateTime: Tuesday, July 26, 2011 - 12:28:07 PM.

I'm getting NSIs with delta inventories up to full inventory, sometimes both from the same computers. Some of the NSIs are empty - 1kb and 2kb files.

We have SMP 7.1 MP1 on Server 2008 R2 on a VM. The database is on a separate box with 64-bit SQL Server 2008 on Server 2008 R2 w/32GB RAM.

SMP Agents are either 7.1.6797.13797, or NS 6.0.0.2416. About 100 agents didn't upgrade even though they are in the filters just like all the other 1200 that did.

We did a server consolidation as part of the migration. We had Client Management Suite 6.x on one server and Service and Asset Management Suite 6.x on another server. We had Inventory Forwarding setup from CMS to AMS, and I'm pretty sure it was turned off prior to export. Politics and performance are why they were split out. We still use the HelpDesk 6.x on the old server.

Because we have Asset Management, we exported (probably too much) from both servers, and then imported (again, probably too much) from both servers. We did CMS first, then AMS.

I checked the agent settings of at least one of the clients, and it had the proper agent configuration - delta inventory once a week, full inventory once a month, but it was still sending NSIs to the server.

Anybody have any suggestions on where to find filters? I've looked all over the filter page and can't find this particular filter.

In lieu of the filter, does anybody have any suggestions on how to fix my problem? I've turned off the Client Message Dispatcher and File Receiver services until I can get this resolved. NSEs are not being processed, so I can upload any of those that might be helpful.

First, I'm not sure what you mean by a DS job that removed all servers and set the server to a new server.  Can you clarify this?

You are going to see a lot of traffic if you migrate 1300 agents in a short period of time.  Be sure your tuning is correct.  I'd set agent policy interval to 4 hours rather than the default of 1 hour and see how that goes. 

You have at least one package server running task service and package service provisioned, serving the 1300 agents, right?  The NS can only support 500 agents on its own before a site server is necessary.

Are all inventory policies set to weekly for deltas and monthly for fulls?

I would start up services and make these changes and see if the number of NSE's is reduced.  However, at that quantity, the NS may be incapable of processing anything.  You may want to delete all NSE's when you make these changes so that NSE's can process normally.

If you still have issues after making all of these corrections (if they were needed), I would investigate a performance bottleneck.  If you're getting deadlocks, SQL could be the cause -- is it dedicated to Altiris, or sharing juice with other DBs?

The filter you mention initially is a list of computers; it has no server name specified in a settings file, because it contains no settings.  If you want to modify the settings file that points to this filter/target, go to Settings > Agents/Plug-ins > Symantec Management Agent.

The DS job runs AeXNSCUtils.exe /hidden RemoveAllServers . Then it runs AeXNSCUtils.exe /hidden AddServer ServerName. Handy little tool.

The migration was over a period of four weeks - roughly 300 per week. Trying to reduce the number of NSEs, I set send basic inventory to 1 week and configuration update requests to 1 day on the SMA Target settings. Still no real reduction.

The primary NS/SMP and one other Site Server, setup with Task Server and Package Server, are located at the main office, and one Site Server at each branch office. At the present time, I don't have any "packages", other than NS/SMP agent packages.

Inventory policies are set to deltas once a week and full is monthly.

There's nothing to change, so I don't see what starting up the services will help at this point. I've emptied the EvtQueue folders multiple times and they keep filling up.

Not getting deadlocks with the above changes, just getting way too many NSEs for policies that aren't set.

When I look at the XML in the Client Policies folder on a PC, I see a server name listed. It points to a server that shouldn't be there.

<clientpolicy agentclsid="Altiris.AeXNSClientConfigUpdate"><communications><nsurl enabled="False"><serverweb><Policy version="7.1.6797.0" userPolicy="" hash="B492B05D13F9052BEA79CAD70BFB8487" name="All Desktop computers (excluding 'Site Servers')" guid="{142F2372-E64D-43C0-A207-17DB2C0552C4}"><clientpolicy agentclsid="Altiris.AeXNSClientConfigUpdate"><ClientPolicy agentClsid="Altiris.AeXNSClientConfigUpdate"><communications><Communications><compression enabled="True" eventssize="200"><Compression EventsSize="200" Enabled="True"/></compression><nsurl enabled="False"><NSUrl enabled="False"><servername><ServerName>servername.domain.com</ServerName></servername><serverweb><ServerWeb>http://servername.domain.com/Altiris/</ServerWeb></serverweb></nsurl></communications></clientpolicy></serverweb></nsurl></communications></clientpolicy>

Does that particular client have the correct info if you open the agent and view the Symantec Management Agent Settings?  What is the Altiris Server listed as (FQDN)?  What is the task server listed as (FQDN)?

Your log excerpt above seems to say that the Targeted Agent Settings policy is telling clients to go to the old NS name, but that these NSE's are being reported to the new NS name.  Have you verified in your Targeted Agent Settings policies (in this case the one with 'All Desktop computers (excluding 'Site Servers')' as the target) that the NS server name and NS server web are your new NS under the 'Advanced' tab, and that the box is checked?

Basically I'm wondering how the 1300 clients were redirected when you decided to kill your other NS.

The agent settings on the client are correct. The server listed is the correct server by FQDN. Client and server settings match exactly.

The server listed in the Client Policies XML above was not even the main NS. It was the Service and Asset Management NS. All clients reported directly to the Client Management NS. Inventory Forwarding was turned on from the old CMS to the SAM.

The "All Desktop Computers (excluding 'Site Servers') is a new filter and shouldn't have anything about the SAM server in it.

BTW, I've deleted the client policy file and had it re-created, and the same information populates back into the file.

The other two NSs are still active. I'm still trying to track the few remaining clients down to get them moved over to the new server.

Clients were moved as mentioned on page 41 of the 6.x to 7.1 migration document, by having them report to the new SMP using the Alternate server in the old v6.x Agent settings. Once they began reporting in to the new server, they were upgraded and deleted off the old CMS server.

Client settings on the old server have not changed since the migration. The alternate URL still references the new server. All but a handful of clients migrated to the new server and upgraded without any problems, with the exception of this NSE issue...

So you have multiple 7.1 Notification Servers in your environment, but only 1300 nodes?

Sorry for the small book here, but maybe all this will help.

Only one v7.1 NS, and 6 site servers with task server and package server services. One of the site servers is here at the home office to take the load off the NS. The other 5 are at some of our remote locations.

We have two v6 NSs that we migrated from and consolidated into one v7.1 NS. We have a v6 Client Management Suite server that computers reported in to, and it forwarded inventory over to the v6 Service and Asset Management Suite server. The two v6 NSs are still in place because we have a few remaining PCs that are reporting in to the CMS server that have to be moved over to the new v7.1 NS. And, the v6 SAMS server is still running because we still use the HelpDesk. As soon as I get this issue resolved, we'll move on to installing and migrating to ServiceDesk v7.1.

The migration process went something like this:

• I disabled all of the agent installation policies and enabled all the uninstall policies - for everything except the NS Agent.
• I created a new "migration" collection to use to migrate computers.
• Other collections were added to the migration collection when it was their turn to migrate.
• This "migration" collection was excluded from the normal NS Agent settings policy. 
• A new NS Agent settings policy was created to apply to this migration collection and was changed to add the alternate URL in the advanced settings, just like the documentation said to do. I did this by location as well, because the package servers had to be done at the same time for each location.
• I exported the v6 NS data, and then imported them into the v7 NS, using the data migration tool supplied by Symantec. There's a lot of "stuff" that gets exported, so maybe I exported/imported something I shouldn't have. The documentation wasn't very clear on what to export/import.
• Once v6 data was imported and a random check of asset associations verified for Asset Management, a small pilot group of about 20 PCs was migrated.
• Migrated PCs were checked for agent upgrades and the new additional agents.
• Again, asset associations were checked in Asset Management.
• Once the information was verified, the migration colletion/policy was enabled on the v6 NS. Another 50+/- PCs were migrated.
• Agent upgrades checked and asset associations checked.
• We waited a couple of weeks to ensure everything was working properly and then began the full migration of about 300 PCs per week.

The 'New Computers' and 'Installed Agents' filters were worthless for a migration, so I created a new one to see which agents had migrated and upgraded their agents. As PCs were reporting in to the new v7.1 NS, and agents upgraded, they were deleted off the v6 CMS, but left on the v6 SAMS (mostly for historical purposes, and for reporting until all were on the new NS).

Once 'all computers' had been migrated, cleanup began for those that were remaining on the v6 CMS. It turns out that some agents had migrated to the new server, but didn't upgrade, so my filter missed them. Those PCs were removed from the v6 CMS. We still have agent upgrade cleanup to do, but we have tracked down all but a few remaining PCs on the v6 CMS that just won't migrate. But that's a different problem.

Everything seemed to be running fine for about two weeks, and then I got the email from the server administrator. The C: drive was full. That's when I found out that the server wasn't processing NSEs, and had filled up the C: drive. I backed off the agent inventory policies and configuration requests. I also backed off the amount of data agents report - events, application stops/starts, etc. This was all the normal information we collected with our v6 CMS server. The next day the C: drive was filling up again. NSEs were being received for policies that were not set - to my knowledge anyway. I went through every policy and setting that I could find to see if any changes could be made to reduce the amount of NSEs, but didn't find anything.

I started looking at the NSEs to see what policies/settings were being reported. Turns out it is everything - all inventory, application stop/starts, events, client logons, etc. I checked the client policy XML of one of the computers that was continually uploading NSEs, and that's when I saw the v6 SAMS server is listed in a v7.1 collection. I checked my computer, and it is listed there as well. Agents never reported to that server, so I don't know why that one would be listed in a new v7.1 collection.

I've been going through some of the NSEs in the 'bad' folders, and some are empty - 2kb and empty. Some agents are reporting the same inventory/events every 30 minutes.

I used DS to run NSAgentUtil.exe to remove all servers and then set the server to the new v7.1 server and that didn't help.

I deleted all NSIs and BAK files from two computers, and after checking in with the server, they started reporting inventory/events again.

Right now my best guess is that something went wrong with the import of data, or I imported something that I shouldn't have imported. Another guess is that the agent upgrade didn't go well and I might have to wipe all agents off every computer, and then re-install. One option would be to re-install the NS and start over from scratch, only importing computers. We'd have to go through and re-associate computers to owners again, but that may be the only way. Historical data would be lost as well.

If you build a new PC and push the agent to it from your 7.1 server, does it behave properly or does it get policies referencing your SAM server and reporting duplicate data every 30 minutes?

No, it doesn't behave properly. Does it get the proper settings, yes it does. The a.log only shows basic inventory being sent twice. However, when I pull up the resource manager for that PC, it shows inventory being received about every 2 minutes.

Inventory policies and  Event Status policies have been turned off - at least all those that I could find.

One thing I have noticed is that the Outbox on the client never clears out. Shouldn't it empty once inventory has been sent?

Is there a task assigned on a recurring schedule telling them to send a basic inventory every minute or something?  You can open the agent on the client and view the task history to tell what tasks are occurring on what schedule.

We haven't begun using Task Server yet, so task history is empty. Our main software distribution method is, and will probably remain, the Win32 Deployment Console.

What's the path to this outbox folder?

C:\Program Files\Altiris\Inventory\Outbox

Hi, dsmith1954:

I share with you some ideas, just in case any of them can help you.

.- There is a registry key that keeps a copy of the processed NSE, instead of removing it

http://www.symantec.com/business/support/index?page=content&id=HOWTO2908. Look at the registry on your SMP, just in case you have this key.

.- You say that you exported data from your NS6 and that you imported it into your SMP 7.1 with the Symantec Migration Wizard. May be you are exporting collections/packages that reference your NS6. Look at them carefully. Also, be awware that you may have exported/imported the client policies from your NS6 into the SMP 7.1. IF this is the case, disable (or remove) your NS6 client policies.

Another comment, is that it seems that the SMP server is processing the NSEs (or at least, some of them) as the SMP is showing that it got some inventory and events: Could it be that it cannot delete the NSEs from the c: drive? May be the account the SMP is using has no rights to remove files?

Good luck:

       Falquian

• The registry key is there and it was pointing to the C: drive until I changed it to a drive with more space available.
• New v7.1 filters should not point to any v6.x NS. None of my v6 collections pointed to the Asset Management server, that I am aware of. The new v7.1 filter referenced above does not point to the v6.x Asset Management server as far as I've been able to tell. I haven't checked the SQL yet though.
• Because I was merging two separate NS servers into one, I tried to make sure I policies, collections, packages, reports, etc. didn't overlap. I didn't export/import any reports from the Client Management server, and I didn't export/import any collections, policies, packages from the Asset Management server. I imported Client Management data first and Asset Management data second.
• No policies were enabled on the Asset Management server, and I'm fairly certain that I didn't export/import any policies/collections from the Asset Management server. There were no software packages enabled on it either. The Asset Management server was just HelpDesk and a reporting server for Asset Management. We had a separate Client Management server.
• Yes, SMP is processing NSEs. The problem is that it is processing NSEs for policies that are not set - at least not for any policies that are visible to me in the console. If you know where the v6.x policies might be on the SMP server, I'd be more than happy to delete them. I can't find them anywhere, but then again, I haven't looked at the SQL tables yet. I'm guessing I'll have to search through the SQL to find the solution to the problem.
• I believe the issue with deleting NSEs is a client issue. The NSEs that are not being deleted are on the clients.

Having the registry key is neccesary for troubleshooting the NSEs, but I would suggest you to remove this registry key. I see no need to keep all the NSEs, if the health of the SMP is good. When the registry key is deleted, the NS will begin deleting the processed NSEs (I don't remember if you'll have to stop and restart the SMP services or not)

You can examine the policies a machine is executing in the Resource Manager: On the SMP console, go to Manage menu, and select Computers. A list with all the computers in the SMP database appears. For troubleshooting, I would suggest to focus on a small set of machines (may be just one or two). Select one computer, and in the left part of the screen, you can check all the jobs/tasks that are applied to that machine, as well as all the policies afecting it. You can click on any of them to evaluate the policy/job/task.

Good luck:

          Falquian

Those NSEs get processed and deleted. There are also Evtxxxx folders on the C: drive that get processed. My guess is that those I moved to the D: drive with the registry key are processed and passed over to the Evtxxxx folders on the C: drive. The reason I say that is because when I shutdown the two receiver services, the D: drive folder continues to receive NSEs, but doesn't process them. The C: drive also continues to receive files, but they are processed. Data loader service still on. With both receiver services shutdown, NSEs are still received and processed, but not as many.

Once I renamed the registry entry and restarted the services, the number of NSEs received reduced to what I would expect for the way policies are set. That folder that was set in the policy is still being processed. When I copy some of the old NSEs that I kept into that folder, they get moved over to the Evtxxxx folders on the C: drive.

Looks like we're halfway there. That registry entry was put there by the installation, or some configuration setting. This is a new installation, and I have no idea how that entry got there. It isn't on either of the v6.x NSs. What I'd like to know is, why did that entry cause the NSEs to constantly appear? Was that some kind of circular reference, such that files were being copied and processed over and over again by being copied from one place to the other and back again?

As far as I understand the process, the NSEs in your D: drive are a copy of the NSEs received by the NS. (I do not know if they are copied there just when they are received from the agent, or when they are processed by the NS).

A for my expirence, the registry key can be safely deleted: I use it only for troubleshooting. When I wnt to examine the NSEs the NS is processing (normally because of some errors appearing in the logs), I create the key, examine some NSEs, and then I delete the key (until the next troubleshooting session ;) )

Regards:

       Falquian