Symantec Management Platform (Notification Server)

We upgraded one customer to 7.5 SP1 HF3 with 3k PC and customer have lot of calls about performance after upgrade. We found out that hashing is triggering for example after computer startup (can be changed by policy) and took lot of CPU - about 50 % (Intel E8400) and HDD - about 80 MB/s. Customer have some big packages (one package have 10GB, 8000 files) which they are checking every day by policy to have latest sources on PC. In 7.1 it was ok but with 7.5 and hashing we can see lot of performance impact. I have told that any other Symantec's customer did not complain about hashing and performce but based on Resource monitor I can see such behaviour and don't believe that any other customer doesn't have the performance problem. Because of that we have to turn on LAC and other customers are on hold with upgrade to 7.5 SP1.

Is here anybody having similar problem with hashing and performance impact to system?

Btw: During this we found that registry key Delay at System Boot (secs) is not working on 7.5 SP1 HF3.

Hello,

May I start from the bottom. Starting from 7.5  "Delay at System Boot" registry key is gone for most supported OS, SMA service has automatic delayed start instead, exceptions are Windows2003Server and XPs, that key still presents there and is functional. What were the OSs you had problem with? 

Can you please clarify where do you have performance problems on NS or package server side?

Hi Kada!

Probably this thread will provide some useful information about Package Server 7.5 SP1:

• https://www-secure.symantec.com/connect/forums/75-sp1-package-server-workflow-question

Hello,

customer have Vista and Win 7. We upgraded clients from 7.1 and registry key is there, it is not gone. We can see that start of the service is delayed but for example service starts one minute after PC startup and then start immediatelly do hashing. And that time user is logged on, logging takes more time and during this mapping drives sometimes don't work. When LAC on, logging and mapping drives is OK.

In this case if the registry key work we can setup there 8 minutes which could be lot of time to do logging of user and mapping drives. Then after 8 minutes SMA can start hashing.

No NS, no PS. End clients have performance problem during downloading sources.

Hello Igor,

on PS we don't have any problem. We have problem on 3000 end clients during time when SMA downloads and checks sources.

This registry key should be gone after upgrade, we'll addess this one. I don't believe  delayed start would solve your problem. Can you please provide verbose level logs of the affected SMA?  

Attached. You can see our policy for synchronization needed data for end client. There you can see how long it took. Before triggering policy I deleted one file from WIM and one application from MSI package.

Attachment(s)

Logs_40.zip   462 KB 1 version

Really there is not any company having Altiris 7.5 SP1 and having performance impact during hashing?

No, there were no complains, it is valid behaviour that after SMA service restart package hash is being re-calculated once related policy is being executed.

Sure, but how much CPU and HDD is used during hashing?

Hi Kada,

If you give me something more specific about how you are measuring the hashing overhead I can provide comparable figures from a similar node-count.

Kind Regards,
Ian./

Currently the CPU usage for hashing is not limited, so it will take 100% of the CPU core where hashing thread is run. In your case overall 50% means that one of your cores was 100% loaded.

According to Kada's logs there were two packages being triggred one by one. The packages were not downloaded fully, means they were existing on the disk, and only delta files which were changed were downloaded. But since those packages were addressed first time after agent restart, the hashes of the whole packages were recalculated. Subsequent addressing of those packages should only recalculate the delta files (downloaded or changed).

Package 1 – 15 GB, 8000+ files. (downloaded 17 MB)
  Time spent: 13:37:34 – 13:55:42 (18 minutes)

Package 2 – 17 GB, 10+ msi files (downloaded 159)
  Time spent: 13:56:00 - 13:59:14 (3 minutes)

As it is expected, the big amount of files take more time to process, since the narrow case here become the File System and HDD usage.

Roman,

many thanks for reviewing logs. Those packages are there long time (not first time after restart Agent) and when the is something new in the package Agent ALWAYS do hashing for all files in the package, NOT only for delta.

You mentioned Subsequent addressing of those packages should only recalculate the delta files (downloaded or changed).. Does this mean Agent should not recalculate whole package after download delta but only delta files? If it should work like this I have to say that it is not working like that and can be a bug.

Yes, we have big amount of files in the 2 packages. It cause Windows scheduled task is failing during logon to PC becuase Agent is doing hashing for all files in those 2 packages. There is not any control for delay Agent do hashing after pc startup (registry key is not working). How we can handle this?

Regarding recalculations: I should admit I was speaking about package downloading process (which is done by Package Delivery).  While checking & downloading such (delta changed) packages the hashing is done only for delta (if it wasn't first time after restart).

SMF plugin also has some functionality which could trigger force hash recalculation before package execution. Unfortunately I do not know the conditions when SMF do that and need to ask responsible team members regarding this issue.

P.S. In your logs I didn't found any SMF force recalculation calls, so I assumed this was the first package addressing after SMA restart (logs are not full and the SMA start is not present there either).
You could use DWORD "MaxSize" value in SMA logging registry in order to enlarge log file size and avoid rotation override so often.

>>There is not any control for delay Agent do hashing after pc startup (registry key is not working). How we can handle this?

Directly in the Package Delivery there is no such control. Maybe it is for SMF execution schedule somewhere.

I will point this thread to SMF guys, maybe they could comment this.

Hi Ian,

well, we can see it in Resource monitor. As I mentioned it has impact to Windows Scheduled task which are setup after startup. These are most of the times failing. In the picture you can see impact.

I provide you new log - one file. Please check mainly Software Delivery where you can see the recalculation is done for whole package, not only for delta. You can see that after dowloading and hashing agent is doing hashing for whole package before run command. We can understand that before execution of the command agent needs to validate the package if it is ok. But it means there is second hashing for whole package when we have setup remmediation immediatelly.

It would be nice to have some feedback from your colleagues regarding SMF.

Thanks in advance for checking the log

Attachment(s)

Agent_3.zip   409 KB 1 version

Roman,

have you had time to check new log? Do you have any input from SMF guys?

Thank you

No, I didn't, since in either way this should be addressed by SMF specialists first. I haven't receive any feedback yet, will bump them once more.

Just as a note on this one. In our environment, one thing our client machines have is lots of disk space.

For some of our large packages (which had lots and lots of small files) we delivere the package source as a cluster of files with a zip archive of the most 'troublesome' source tree path. The install script then performs a file copy and unpacks the archive before initiating the install. 

This does come at the cost of client CPU, but overall we found that this increases markedly the package delivery time to the client.

For the future, we are looking at virtualisation to help solve this large package problem.

D.Salnikov

did you check whether the registry key should be gone after upgrade?

I did - it was there after the upgrade, key is not functional anyway, bug was reported and already fixed, now upgrade cleans old keys, fix will be available in next release

Hi Kada,

We have checked the new logs with SMF guys. Unfortunately, I didn't found the Agent starting routine there either, so I only assume this SMF job was launched first time after restart.

What is seen from the log exactly:

1. SMF order to download the package (7ECFA873-B1C5-4160-8899-17ACACB373BE).
2. Package Delivery check the package before downloading and find that it is not hashed yet so it hashes it (the whole package hashing is performed 8K+ files). It is OK if it is a first time after restart. I assume it is since there are not starting routine in the log before that moment.
3. After hashing Package Delivery found that some files are changed (51 files).
4. Those files are downloaded
5. Those 51 downloaded filed are checked for hash in order to be sure there was not MiTM wile the download.
6. The package is passed to SMF, which do it's job with the package.

Later in the log, several validity checks are requested for that package (by SMF) but no extra hashing is done. So, actually there is no "double" hash check for that package. If you do not restart the PC or SMA, the later checks while download will(should) only have the "delta" hashing, like in step 5.

Please note that in previous logs I didn't find "double hashing" either. There were two packages hashed, each was hashed once. If you are sure you have "double hashing" for a single package while single SMA session run (no restarts), then please provide the full logs starting from SMA starting routine. Increase "MaxSize" and/or "MaxFiles" in the SMA logging registry and collect all logs from system start until double hashing is encountered.

Regards,
Roman.

Hi Roman,

thank you for reviewing the logs. You are right that it was first time for agent to trigger the policy - because I change scheduled.

I did some testing and till SMA is not restarted the Agent is hashing only delta files. But after restart it is doing hashing for all files in the package.

Another point is validation - you mentioned validity checks. This always happen before execution command. And when you see resource monitor, during this validation SMA is checking all files in the package and it takes almost same time like hashing during download after restart of SMA. So from our point of view this is some kind of hashing again (before command execution) even if SMA is not restarted and no new files is needed to download.

We are using hibernation of the pc so SMA have control under packages but still we can see lot of cpu and hdd usage during validity check before execution command and we can still see that scheduled task planned after pc startup is not triggered sometimes because of SMA. Can this be somehow improve? Put low process priotiy during validity and hashing? Or any other idea?