Video Screencast Help
Symantec to Separate Into Two Focused, Industry-Leading Technology Companies. Learn more.

Data collection takes too long

Created: 08 Jan 2013 • Updated: 01 Sep 2013 | 10 comments
This issue has been solved. See solution.

Hi,

I also have an issue when I try to collect data on windows servers (data collection can take fro, 3 to 11hours instead of few minutes).

I discovered my CCS servers where tried to connect to a domain controller who was not reachable, so they tried to reach another domain controller, but this one is not in the same zone.

I moved my CCS server to the right Site (with the right domain controllers), but when I try to collect data on windows servers, my CCS managers always try to reach the problematic domain controller. (I restarted the netlogon services and even rebooted the servers)

when I sniff my CCS Manager data collector server traffic, I see thousands of Blade.workerprocess traffic to the old domain controller. is there a way to change it ? is it hard coded in the domain cache from CCS ? if yes, can we flush the domain cache ?

thanks for your help

Discussion Filed Under:

Comments 10 CommentsJump to latest comment

Matt Plourde's picture

Several questions:

- What version of CCS?
- What is doing the data collection? (CCS managers, ESM or RMS?)
- What kinds of data collection? (predefined standards, custom standards, queries ?)
- you didn't change the CCS App server's domain membership, did you? (you can't)

That should be good enough to get us started :)

Chaitali's picture

Sebastien,

It depends on what query you are running.
In my experience, if you are running a windows users query, it is best to scope it to only one domain controller instead of scoping it to the entire domain.

If the query is scoped to the entire domain, it is going to query each and every machine in that domain - thus taking longer.
If the same query is scoped to only one DC - it finishes in one to 5 mins max.

Thanks,
Chaitali

SebastienB's picture

Hi,

and thanks for your time.

I use CCS11 with CCS Managers for data collection. I try to collect data for only 1 windows asset in the same  domain as my servers, and use the predefined windows standard (also tried with custom standards)

yesterday, one collection took 3hours to be completed... (for information, I have the same type of installation in my Dev environment, and don't have any issue)

the only thing who was my servers were not using the right domain controller for login at the first stage, but it has been corrected.

I cleared the domain cache by removing the files  for the windows assets in symantec\xxx\control\windows\cache\* and rebooted the servers, but apparently I still see thousands of blade.workerprocess packets during data collection to the wrong domain controller. is it hardcoded somewhere else ??

I also have a case opened to symantec, but I really get crazy with this performance issue and time is running :(

Matt Plourde's picture

When you say "my servers were not using the right domain controller for login at the first stage", do you mean your LDAP settings in Settings>General? Or, something else in CCS? On the Windows box?

Just answer that and we can go from there.

SebastienB's picture

My servers were using a wrong Domain controller site when you logged in to it. (so windows box)

but when you collect data, and build the cache, ccs needs some credentials who tries to reach a domain controller, which is not the right one (even after having  modified the preferred domain controller to use in windows, and cleared the cache)

when I sniff the traffic from the CCS Manager data collector server I see :

process name / source / dest / payload

blade.workerprocess.exe / ccs-manager-hostname / wrong-ldap-server-hostname / ldap(389)-xxxxxxx

this wrong-ldap-server-hostname is a ldap(domain controller) server my ccs manager must not try to reach as it is not in the same subnet, but I'm not able to change it. (even if I remove the cache files)

thanks.

Chaitali's picture

Do you have a query engine installed on this wrong LDAP domain controller?

Matt Plourde's picture

did you change the domain membership of the CCS application server?

you can always edit the hosts file and point "wrong-ldap-server-hostname" to "right-ldap-server-hostname"...

cmccoy2's picture

How large is your domain? Are the Cache files topping out at 2 GB?  if so, there is a known issue for larger domains that if the domain.MDB file that exists in the \DPS\Control\Windows\Cache folder reaches 2 GB files, then it causes the data collections to take a long time because it relies on that cache to be in place to be able to resolve the target machines.   This caching process also attempts to cache Users and Groups.  If this is the scenario that you are running into, then open up a tech support call and ask for the private fix.  Also you may want to check your routing rules to make sure that the appropriate sites/subnets are sent to the correct CCS Manager machine.  I believe that the fix for this domain caching issue is coming out soon, but I don't think it has been put into the Content or program updates yet.

SebastienB's picture

Sorry I didn't answer before, but we had other issues in parallel, and this issue disappeared, but apparently, the problem is again here...

Chaitali  : no query engine on these wrong ldap domain controllers

Matt Plourde  : no we didn't change the domain membership.  we also tried to fake the system modifying the hosts file, but it didn't work. (errors in CCS)

cmccoy2 : yes we've heard about this issue (2gb), and we've received this hotfix. it works for a while, but we now see thousands of ldap requests to another domain controller (which is also not the one we are supposed to see...). so data collection is still low.

thanks!

cmccoy2's picture

The latest SCU (2013-1) should resolve the domain cache issue with the 2GB limit if you were running into that.   I am not sure about it pointing to a particular Domain Controller, but hopefully that would be resolved as well.

SOLUTION