Symantec IT Risk and Compliance Product Group

Has anyone been able to test to see if perfomance is better when doing patch assessment data collection using CCS standards after the 10_5_PCU_2011-2 update?

I installed the update and it seems to perform as it did before the update (badly).  I was at 14 hrs and still running to collect data on 2000 servers which usually takes 1.5 hrs using just the RMS Console.  There were 5 simultaneous patch assessment jobs spawned to the query engines from my 1 patch assessment data collection job which basically brings them to their knees.  I had many other jobs backed up in the queue as a result.

Supoposedly this was to be fixed with the 10_5_PCU_2011-2 update but I am not seeing any improvement. 

Is anyone else seeing better results?

-Mike

I've got 3 asset groups, Critical, Essential, & Supporting.  I collect/evaluate the Patch Assessment Checks against these in 3/3 jobs.  I've got the 2011-2 installed, and speed for me seems improved.  I've only got results for 1 group now, but was 1.25 hrs to collect on 88 assets, and now is 18 min.  Evaluation on the same 88 had been 50 min. and is now 20 min.

A few questions for you:

Assuming you are just running the Patch Assessment in the RMS Console, right? 

What kind of infrastructure configuration do you have?  Single server (all-in-one) where all CCS components are installed on one server?  Distributed setup?

Assuming you are using SQL Server, are the databases on the same box as well?

Did you update all SQE's as well with the PCU update?

Aaron

When I run a PA in the RMS Console it takes 1.5 hrs to complete on roughtly 2000 servers located worldwide.  My QE infrastructure is pretty well tuned utilizing distribution groups and 160+ query engines.

When I run a PA using the CCS Stds module, it takes over 15hrs just for the data collection alone.

I have a distributed CCS infrastructure.  Separate Directory Server, App Server, DPS, BVIS and SQL Server.

All of my MQEs/SQEs/BVIS/ECS are patched to 10_5_PCU_2011-2  level

One thing I did just read on the 10_5_PCU_2011-2 update  is that the data collection piece needed to be updated before the Reporting/Analytics piece.  I did not do in that order so it might be my problem.  I am going to rerun the R/A update again now that the D/C portion is current and see if that helps.  Following the intall path of Directory server first, then App Server, followed by DPS.

I will know more next week as to whether or not it helps.

I use Reporting & Analytics for Patch Assessment based on a modified version of the Windows Patch Assessment Standard that comes with R&A.  I copied the std to a separate folder, and modified it, as it can't be modified in the default folder.  Each time a new Patch Standard comes out, I copy the NEW MS Bulletins to my modified standard.  The new content update just came out today, so Im running again with my updated standard.

Two parts to my infrastructure:  Data Collection, a 2k3sp2 with SQL2k5 local. I have 3ea SQE's on other HOT servers in the environment.  So my atomic jobs are split between 4SQE's against 425 servers.  Reporting and Analytics is an Application server with the Directory Support and CCS_Web iis.  DPS service as a load balancer and report evaluations on another server, DPS is also installed on the RMS server, and SQL for CCS R&A is standalone SQL server.  So there are 7 servers in my infrastructure for this product in a distributed setup.

The 2011-2 update was for all components.  I restarted all of the servers after updates except for the SQE & SQL servers.

The speed of course comes from the standard and just how may checks it has to evaluate.  You may be taking too big of a bite without a distributed setup.

Where the problem lies is that the CCS stds PA job dices the collection into multiple jobs on the BVIS (up to 5 at once I have seen).  The query engines cannot handle running 5 concurrent PA jobs.  When you run a PA job for all missing patches using just the RMS console, it submits just ONE job to the BVIS which the QEs can handle.

I even set up a job scanning for just 6 hotfixes.  It still created 2 concurrent jobs which starts to task the QEs pretty hard.  One PA job will use pretty much all of the cpus on the QE by itself. Mine are running 4 CPUs by the way.

This last update was supposed to quit creating multiple jobs and put them into one.  Like I said in my other post, it might be that I applied the update in the wrong order so once I correct that I will know more.

Apparently I needed to run the May Windows Patch Assessment Check Library update for the speed improvement to work.

Here are the result stats for the job.  Total time for job was 3hrs 37min.  The standard contained checks for 370+ patches:

Data Collection Summary:
Result Files Received: 1983
Asset Error Messages Received: 1
Number of Assets collected against: 1983
Number of Standards collected for: 1
Standards Collected for:
Microsoft Windows Patch Assessment

Data Collection completed in 1h:21m:57s

Evaluation Summary:
Result Files Received: 1983
Asset Error Messages Received: 7
Number of Assets evaluated: 1983
Number of Standards evaluated against: 1
Standards Evaluated:
Microsoft Windows Patch Assessment

Compliance Score: 91.90%

Evaluation completed in 1h:42m:37s

Evaluation synchronization completed in 33m:13s.

The CCS data collection job only spawned 1 machines query and 1 patch assessment query.  The patch assessment query took the same amount of time as when I run a normal patch assessment query using just the RMS console.