SMP Issues since SP2 Upgrade
Updated: 30 Jan 2012 | 15 comments
This issue has been solved. See solution.
I upgraded to SMP 7.1 SP2 last week and starting on Monday have had one problem after another. My Event Queue is filling up with NSE's that aren't reporting back to the server. Several computers are showing up as duplicates in our inventory. But to make a long story short I have run the repair on the SMP but now I try to run a repair on the Client Management Suite and it dies at task 18 of 45 pcA x64. I get an error that the MSI can not be acceessed because it requires write access. Checking the file permissions, it has the same as every other msi that runs prior to it in the repair process. If anyone has any ideas, I'd love to hear them. Thanks.
Group Ownership:
Comments
Are you running the repair as the Service Account?
That might help.
Some things to verify
1. Right-click on SIM and choose "Run as Administrator". We had some bad mojo on a test server when we ran an install without this step.
2. As Andy mentioned, log into the server and run as a service account.
3. You may try a repair on the database first, then a repair on the solutions. This is accessed via the Settings>All Settings>Notification Server>Database Settings menu.
Launch SIM and hold down ctrl+Shift and press "P" (Partial installs). The pcA module might not be the only sketchy item.
The install files are in your installation directory under "Symantec Installation Manager\Installs\Altiris". You may be able to right-click on the .msi file in question and choose "repair" or even "uninstall". The pcA files are at the bottom of the folder and start with symantec_pcanywhere...
Check for any additional errors in the Altiris or Windows event logs.
Wish you luck - 2011 was pure salted wounds for us dealing with 7.1 RTM and forward...
I did not try to run as the
I did not try to run as the service account but I did try to run as the admin. I will try to log on to the server with the service account and try again. The ctrl+shift P trick is nice too. Thanks.
I went to the install directory and every MSI came back with an error that it couldn't be opened. So something is hosed. But I'll try those things and see what happens. We haven't had too many problems with 7.1 until this. We also had something odd happen the other night that I believe triggered this whole thing. Somehow the service account and my admin accound both had our default database stripped on our SQL server. The accounts were there but without a db associated, we couldn't hit the database. Once I figured that out the server was slammed with requests. It's not been the same since. I'm half temped to uninstall/reinstall the server. With the DB intact and daily backups, it's not going to hurt anything. But that's a last resort. Thanks for the quick help.
I think that's Windows permissions
Have you installed to a D: or E: drive? You probably need to add NTFS permissions.
Use NSConfigurator to
Use NSConfigurator to BlockClientMessages. Restart the Client Message Dispatcher service. The queue should process. When the queue is empty or very low (<200 messages), use NSConfigurator again to unblock BlockClientMessages. If queue gets high (>5000), use NSConfigurator again to block, rinse/repeat until all events are processed and queues are normal.
Mike Clemson, Senior Systems Engineer
Intuitive Technology Group -- Symantec Platinum Partner
That is a handy tool for
That is a handy tool for sure. However no matter what I do the event queue will not process. I currently have 10K+ .nse files in C:\ProgramData\Symantec\SMP\EventQueue\EvtQueue. It seems that no matter what I do those will not process.
Move all but 1,000 into a temp folder.
Then see if they'll process.
But make sure all your other services are started first.
Maybe check your "process" folder for nse's stuck there.
Having the same issue
I am having the same issue with events not processing. I as well ran into the repair issue you mentioned and found the fix. It turns out it was a bug with the upgrade of PCAnywhere from SP1 -> SP2 and left some registry keys behind.
The msi in question which is failing is "pcans.msi". The previous version did not remove its meta data from the windows installer registry hive so it is "confusing" the current installer when a custom action queries for a "SourcePath". To resolve this go into the registry and remove the following key:
HKEY_CLASSES_ROOT\Installer\Products\3818E521A6683D1179FD00008F8D2F9E
you can verify this before you delete the key by doing a text search in: "HKEY_CLASSES_ROOT\Installer\Products" for pcans.msi and two results should appear: The one mentioned above and: HKEY_CLASSES_ROOT\Installer\Products\BB26595A3B6F3724784200EA418A895A (This one is ok as it is looking after the new MSI)
At this point you should be able to run a repair on CMS successfully.
With the events not processing it looks like my SQL Server is slammed. I have a constant stream of data flowing from my SQL box to my NS (4-6 MB/s) I have a feeling this is related to the events not processing. If you get any information on this please keep us posted! :)
I worked with Symantec
I worked with Symantec support yesterday. There was a sever bottleneck in our SQL environment. We moved the DB to another SQL server a little earlier than planned and that helped but I still have a ton of issues showing up but it looks like it's processing better. They had me run a query on our SQL server that did a snapshot of the stall times on the server. They said anything over 50ms would show a severe bottleneck. We were at 350 for read and 300 for write. After moving I'm down to 29ms. Obviously moving databases isn't going to work for everyone. It just so happened that we had just built a new server for this purpose anyway. I still have a lot to resolve and once I have some free time I'll put together a better explanation.
I made a bit of a break through.
On your recomendation I went looking at my SQL box to try and determine a bottle neck and I couldn't find one. My SMP and SQL servers each have a dedicated servers in the same enclosure, and the HW specs are just stupid on each.
I did notice something. . .Due to the network topology and number of clients conencted, I have bandwidth throttling enabled for all my clients and it is very restrictive (1 KB/s). I've been watching the # of NSEs grow in the EvtQueue without dropping by one. Any NSE that is under 15K I have been able to manually copy to EvtQFast and have it processed successfully.
Testing something else I did notice something strange. . .A minute or two after disabling bandwidth throttling for the clients (in the All Desktop Computers targeted policy) the NSEs started processing. After I re-enabled bandwidth throttling the NSEs stopped processing and started building up again. I was able to reproduce this a few times.
Possibly the event processor is tied somehow to the throttling of the agents, beyond them uploading the NSEs to the server.
Thoughts?!?
Issue has been resolved.
Just as an FYI ended up working with Symantec on this issue of events not processing. It turned out there was corrupt data in the ResourceMerge Table.
Logging into the SQL server and executing: truncate table ResourceMerge
Then restarting the Altiris Services on the NS solved the problem. It is processing all events as fast as ever!
I hope this helps!
Thank you
I dicided to give this a shot on Friday and haven't touched a thing since. Checked my server this moring, no NSEs in the queue. And now my duplicate entries in my inventory are gone and the computers have been associated with their propers OUs in the Active Directory Domains view. Lifesaver. Or sanity saver anyway. Thanks a ton for the advice.
worked like a charm
worked like a charm
Did I miss something?
Hello,
I am experiencing the issue mentioned in the initial post ... namely that running a repair of ITMS 7.1 SP2 halts when it gets to pcans.msi with an error stating that I do not have write access. The solution mentioned here seems to resolve an issue related to nse files not clearing from the queues. So does this resolution resolve the original problem?
Thanks
Alex
My Solution
Well I worked on the problem where the repair of ITMS 7.1 SP2 was failing because of not enough write access to pcans.msi. My solution was to copy pcans.msi from another server. The size of the bad file was 49,472 KB while the size of the good file was 49,458 KB. Other than the size both files were the same as far as right mouse click properties goes.
Alex
Would you like to reply?
Login or Register to post your comment.