Symantec Management Platform (Notification Server)

CEM has been an awful nightmare for us. We've been struggling for several weeks and it still does not work...even with an accredited partner's help. The published documentation has been a confusing mess.

It is so bad (and now time-critical) that management has started discussions to ditch Altiris as a whole and move to another product...some of which seem to handle remote (non-VPN) machines far better than CEM. This is the first company I have been at that wants to remove this product.

What's been the general consensus on that solution? Any success stories? What were your struggles? When it finally functions...does it actually "perform"? We have low hopes for it.

Well, I dont think your experience is typical. Personally, I have had almost no issues with CEM. We had a small issue during setup with something omitted, or not clearly stated, in the documentation but we have sincebeen been up and running since. I have had CEM running, in our environment for over 2 months now and have not seen any issues. We are patching and doing inventory with ease.

May I ask some more specifics on what issues you are seeing? Maybe the community here can help walk you through some of the issues.

I think one of the last hurdles (?)...when we attempt to connect the Gateway Manager in the DMZ to the SMP...we get a dialogue that says "Failed to contact the server and retreive the certificate details". I also see it hit IIS on the SMP, but it never accepts credentials, which are domain admin level at this point. We also get in the logs. Name and key changed for security.

Unable to get the client certificate associated with the specified request (Request: <resource typeGuid="2C3CB3BB-FEE9-48DF-804F-90856198B600" name="gateway_server" policyKey="AAAAAQAB1gdc1fHZ39lhs0AcRYGeoQa03QcSREnObHFykbYbAxa/gDG5IlDq8LXifUAZfV45uaR52rPlKHDUZU6ePSu++W5021/bfZhsIfLfbP5x6rUtQB8AI11C6IHc3NnqrA4aN6SQxgnWuhePqXDVRPXSNLGZrHlTWN5sqkya6Dm9Zi+mpEuhM64ig4jZkwaVJqaw55GmMOBZKc3folKteSUgd7Y+pLZa+xdO1fFK4W1iU47Zl9rd9HYgh7xgVfcCVfyHwO8Nkc4cdTVK+Q2zKqS1V+9+9Fj2IoT7HdLTtvkpXxuHzNeb3m+pw==">
                                <key name="name.domain" value="gateway_server"/>
                                <key name="fqdn" value="gateway_server"/>
                                <regRequest fqdn="gateway_server" 

We have a purchased third party cert. Port 443 is open to the outside and 4726 from gateway to the internal SMP.

Hi!

@tloenhorst:

1. What version of SMP you're using?

2. That was a clean installation of SMP and CEM envrionment or you've upgraded it?

3. About "Failed to contact the server and retreive the certificate details"

• Did you try to add a "SMP Server:4726" in CEM Gateway manager, using SMP server's IPv4, Hostname or FQDN?
• You're using SMP Server's administrator account from "Symantec Administrators" role, when adding it in CEM Gateway?

4. What configuration you're using:

• What type of certificate is used for SMP Web Site?
• What type of certificate is used for "Symantec Agent" Web Site? Does it running?
• You're using a Windows 2012 R2 as SMP Server or Windows 2008 R2?
• All environment is joined in single Active Directory, where same GPO is used for these CEM and SMP Servers?
• What is shown in Altiris Log on CEM Gateway side, in case if you will restart CEM Gateway service?
• All required port(s) are available and aren't firewalled?

Thanks,

IP.

Hi Igor,

SMP is 7.5 SP1 HF2.

Clean install with one upgrade from HF1 to HF2.

Already added the SMP on port 4726 in the CEM gateway manager using the only account used as the Symantec Administrator.

Single AD and same GPO for all servers. Very simple network.

Gateway Manager shows:

"RSA server certificate CommonName (CN) `prxalt01w.ptbcorp.com' does NOT match server name!?"

In the SSL certificate info field on the gateway manager we have the FQDN of the SMP for the common name.

Thanks for your help.

Hi tloenhorst,

⇒"RSA server certificate CommonName (CN) `prxalt01w.ptbcorp.com' does NOT match server name!?"

IP: Seems like, this is because CEM Gateway server machine CN doesn't match to specified CN in self-signed certificate.

Main question for current case:

So now your problem with adding a SMP Server in CEM gateway is gone? You don't see anymore this message "Failed to contact the server and retreive the certificate details"?

• What is current problem remains for CEM side? Please provide details.

Other questions:

• What CEM gateway address you're using as listening address on Gateway side?

• What CEM gateway address is specified in "CEM" policy on SMP Server side?

• You're using a Windows 2012 R2 Server x64, where SMP is running?

Thanks,

IP.

Hi Igor,

We were never sure what to use for the CN for the SSL certificate information. Not even my Symantec partner's engineer knew what really went there for sure, the SMP or the GW machine name. Even with the gateway machine name there it gives the same error and also complains that the cer CommonName does not match the server name.

(Also note, all of our machines in the DMZ are not joined to any domain so Workgroup only)

We use port 443 and the gaetway server's address for "IP addresses and ports" section.

The CEM gateway address is the Gateway machine name with the matching thumbprint.

The SMP is 2012 R2 x64.

Hi tloenhorst,

1) About "RSA server certificate CommonName (CN) `prxalt01w.ptbcorp.com' does NOT match server name!?"

• On server, where CEM Gateway is running, open httpd.conf and change ServerName field to your current CN of CEM gateway Server (By default it is localhost, therefore it throws such warning mesage in log) ⇒ restart CEM Gateway service and check log output ⇒ no there will be no such warning message amout mismatch of CN with server name.

2. If you're using a Windows 2012 R2 as SMP Server side and there is a 3rd party non-self-signed certificate set for SMP Web Site, then please backup it and then try to remove this 3rd party certificate from "Trusted Root Certification Authorities" (it should has name as hostname of your SMP Server and should be as "Server Authorization | Client Authorization) ⇒ after that, restart Altiris Service and CEM gateway service as well ⇒ try to establish CEM client connection via CEM Gateway with SMP Server

Thanks,

IP.

Igor, thanks for the info. Item #1 worked to remove the errors of the name not matching.

I am confused by item #2. Why remove this certificate? Is this not the certificate that  is used here in this screenshot? I did try to do this, but it still failed to connect.

About #2

#2-1. You don't need to touch certificates on "Symantec Management Agent IIS Web Site" page.

What says log output at this moment on CEM Gateway machine, if you will restart Gateway service? Could you please send me logs via private message.

When I'll check logs, then if will be required you can do this #2-2

#2-2. I meant about certificate, which is used for Web Site where SMP is installed (Not "Symantec Agent" Web Site).

Then you will check which certificate is used for "Default Web Site" for SSL binding in IIS

When you will know what certificate is used for HTTPs binding of SMP Web Site, then open MMC on SMP Server machine and:

• Run ⇒ type MMC  ⇒ click "File" ⇒ click "Add/Remove Snap-in" ⇒ click on "Certificates" ⇒ click "Add >" button ⇒choose "Computer Account" and click "Finish" button ⇒ then "OK". Open "Trusted Root Certification Authorities" ⇒ "Certificates" ⇒ find there this non-self-signed certificate which is used by SMP Web Site (save backup of this certificate before removing it from there) and remove it from there.

⇒ After that restart CEM Gateway service ⇒ try to use CEM connection from Client PC with SMP via CEM Gateway.

Thanks,

IP.

Hi Igor,

I sent you the CEM logs in private message.

On the SMP:

Default Web Site has HTTPS 443 binding on "SMP <FQDN SERVER NAME> Server CA"

Symantec Agent web site has HTTPS 4726 binding on the third party purchased certificate.

When I remove the third party certificate from TRCA...then the Symantec Agent site has no certificate any longer. Which one should it have???

Hi Todd,

Current summary of this problem, after discussion:

1) Warning message "RSA server certificate CommonName (CN) `prxalt01w.ptbcorp.com' does NOT match server name!?" doesn't appear in CEM Gateway log - Fixed in httpd.conf on CEM Gateway server.

2) Non-self-signed certificate which is issued by 3rd party for your SMP Server is removed from "Trusted Root Certification Authorities"

https://social.technet.microsoft.com/Forums/en-US/fae724e8-628e-45a5-bf39-6e812d8a1a70/40316-problem-in-iss8-on-mp-in-dmz?forum=configmanagerdeployment

http://support.microsoft.com/kb/2802568

3) Remaining problem is that CEM Gateway is unable to add SMP Server with enabled reporting, due:

"Failed to contact the server and retreive the certificate details"

Todd, is there any changes for #3 on your CEM Gateway, after possible workarounds applying?

Thanks,

IP.

Hello,

>>3) Remaining problem is that CEM Gateway is unable to add SMP Server with enabled reporting, due:

"Failed to contact the server and retreive the certificate details"

What message would browser return if you would type to address bar https://accessPathTo.SMP:4726

Error 403 is expected. 

Just trying to figure out whether Gateway can actually contact CEM site.

Hi Todd,

About "Failed to contact the server and retreive the certificate details" problem on CEM gateway, while adding a SMP Server with enabled reporting:

Please do following steps:

#1. Login to SMP Console ⇒ "Settings" ⇒ expand "Notification Server" folder ⇒ click on "Notification Server Settings" page

• Check which account is specified there as "Application Identity"

#2. Go to "CEM Gateway" server machine ⇒ open "Symantec Internet Gateway Manager" ⇒ add SMP Server ⇒ specify there credentials from #1 "Application Identity" ⇒ SMP Server should be successfully added and reporting should be enabled as well.

Please let me know, what results you have after these steps execution.

Thanks,

IP.

Hi Igor, sent Wireshark dump to your private inbox.

Verified Applicaton Identity being used it the same on SMP and on Gateway when adding SMP to gateway manager interface. Still will not connect, but as before I do see errors in SMP log when the gateway attempts to get a certificate.

Yes, CEM is hitting that path on port 4726 and getting Error 403.

Still seeing this error in the SMP log when the gateway tries to connect:

Unable to get the client certificate associated with the specified request (Request: <resource typeGuid="2C3CB3BB-FEE9-48DF-804F-90856198B600" name="GATEWAY_SERVER" policyKey="AAAAAQABvw1TEWvkiZRMs0L+SxiLGxkatQqsK8KN4hY7nWCFK0GRHoEeBoMTCllAkH24OzASWFxBKTvMJ3bdmj5ZVQdJUMurPQKKIHlkzF5O3OvcNhy07kK+wnDbl1nd6OlpfLv9A4qC0ndTtwvXeV7+rv3roaaBvewQjgNjASRGubmfOesHHfdJwxQ8poFNw/KCuJOJU5qwSx5bLV/fPdF7PK3tDaA9TLWEZWw0eeiASXfeUzII07H/DaFl71OwgjDDzeALMnRvoZ5YKxGid7x81ltdXij+OV0dbIm9kWqfnL7Wg/eo2mjBcvlYLQRpNfGIWQ==">
                                <key name="name.domain" value="GATEWAY_SERVER"/>
                                <key name="fqdn" value="GATEWAY_SERVER"/>
                                <regRequest fqdn="GATEWAY_SERVER" publicKey="AAAAAQABvw1TEWvkiZRMs0L+SxiLGxkatQqsK8KN4hY7nWCFK0GRHoEeBoMTCllAkH24OzASWFxBKTvMJ3bdmj5ZVQdJUMurPQKKIHlkzF5O3OvcNhy07kK+wnDbl1nd6OlpfLv9A4qC0ndTtwvXeV7+rv3roaaBvegJ46qFCcL+kr3mykTcUL85SgwQjgNjASRGubmfOesHHfdJwxQ8poFNw/KCuJOJU5qwSx5bLV/fPdF7PK3tDaA9TLWEZWw0eeiASXfeUzII07H/DaFl71OwgjDDzeALMnRvoZ5YKxGid7x81ltdXij+OV0dbIm9kWqfnL7Wg/eo2mjBcvlYLQRpNfGIWQ==" certificateType="nsagent"/>
                        </resource>, Exception: System.NullReferenceException: Object reference not set to an instance of an object.
   at Altiris.NS.Security.Cryptography.CertificateManager.ConvertPrivateKey(AsymmetricAlgorithm asymmetricAlgorithm)
   at Altiris.NS.Security.Cryptography.CertificateManager.IssueCertificate(Guid id, Guid resourceID, Guid parentID, X500DistinguishedName subject, String scope, CertificateUsageFlags certificateUsage, AsymmetricAlgorithm publicKey, String caName, DateTime expiryTime, Boolean storePrivateKey)
   at Altiris.NS.Security.Cryptography.CertificateManager.IssueCertificate(Guid id, Guid resourceID, Guid parentID, X500DistinguishedName subject, String scope, CertificateUsageFlags certificateUsage, AsymmetricAlgorithm publicKey, String caName, TimeSpan issuingPeriod, Boolean storePrivateKey)
   at Altiris.NS.Security.Cryptography.CertificateManager.IssueCertificate(Guid id, Guid resourceID, Guid parentID, X500DistinguishedName subject, String scope, CertificateUsageFlags certificateUsage, AsymmetricAlgorithm publicKey, String caName)
   at Altiris.NS.Security.Cryptography.AgentCertificateManager.IssueClientCertificate(Guid certID, Guid ResourceID, Guid parentID, String sScope, X500DistinguishedName subject, AsymmetricAlgorithm publicKey)
   at Altiris.NS.AgentManagement.AgentCertificateDistributer.DistributePermanentCertificateByTemporary(CertificateRequestData& requestData)
   at Altiris.NS.AgentManagement.NegotiateCertificateRequest.GetClientCertificate(CertificateRequestData& requestData)
   at Altiris.NS.AgentManagement.NegotiateCertificateRequest.Process(String requestXml, Guid certID, Boolean bEncryptResponse, Boolean bAdminCall, Byte[]& encryptedData))
**CEDUrlStart** :http://entced.symantec.com/entt?product=SMP&version=7.5.3153.0&language=en&module=n/VHGfYhVq3+uaqg4g94f9BZ1/Db7PAMK09Umt97LUTXlFHsjXESHvvraWzyl8s2&error=1184117990&build=**CEDUrlEnd**

Our setup of CEM was pretty smooth as I recall.  I had two issues.  1) I didn't realize it was necessary to add both the SMP and the site server to the config on the gateway, I only added the SMP.  But my symptom was that CEM agents couldn't download packages or register with a task server.  2) My agents use an alias for the SMP name and the tool that generates the CEM offline install package woudl hard-code in the FQDN of the server.  I found a registry hack to change the KnownAs name of the SMP to the alias to solve this issue.  Note to anyone using an alias, I found I needed both the FQDN and Alias name in the certificate Subject Alternative Name (SAN).

With the "Failed to contact server" error you are seeing, I seems like there are only a few things to check:

1) Confirm the firewall between your DMZ and Internal network is allowing TCP port 4726 inbound.

2) Ensure CEM Agent website is setup on the proper port (4726) and with a valid certificate.

If you put the CEM URL into a browser on your gateway and got a 403 that would seem to indicate your FW is open and the agent website is listening on the right port.  I just tried this and confirmed getting a 403 on a functional CEM.  Does the browser complain about the SMP cert at all if you click the lock icon and vew the cert?  I'm wondernig if something is wrong with the cert or the publisher is not trusted by the gateway.

Hi Joe,

We confirmed that the firewall is open on that port bi-directionally even though a successful one-way connection automatically authorizes the return response.

The browser loads the cert successfully when connecting from the gateway to the SMP.

I wanted to update this thread with some findings. Igor has been an excellent help in private message with general troubleshooting.

We discovered that at some point the SMP (servername) Agent CA certificate had been exported WITHOUT the private key and then re-imported. Who did it and when...not sure. But, it was discovered by our partner and then fixed by running: aexconfig.exe /configure /coresettings.config which created a new Agent CA with private key.

THIS was the issue the entire time! We are kicking oursleves for not noticing it to begin with. You know, when you stare at some problem for too long...

Unfortunately, we still feel there is a lack of CEM knowledge in general (except for Igor!) and Symantec's documentation is really vague and needs to be updated.

Todd, thank you for feedback!

After new Agent CA certificate generation, please check this:

1. Make sure that on SMP Server, thumbprint of your Server and Agent CA certificate in Trusted Root matches with

• HKLM\Software\Altiris\express\Notification Server\CA\Agent\Thumbprint
• HKLM\Software\Altiris\express\Notification Server\CA\Server\Thumbprint

2. Generate "CEM Offline" package via SMP Server console⇒ same errors in log as it was before or?

3. Seems like now you will need to re-install SMA on all managed endpoints, since Server Agent CA is changed. (Probably you will need to re-create an image for DS, where Symantec Management Agent is already included).

What says Log of SMA on Site Server or managed endpoint, on attempt to refresh policy, etc?

• C:\ProgramData\Symantec\Symantec Agent\Logs\

Or everything is OK now and there is no problems in communication between clients and SMP server?

Thanks,

IP.

Glad to see you got this working, going to try and tackle this soon!

Igor has built some really good documentation for CEM. 

https://www-secure.symantec.com/connect/articles/how-install-cem-functionality-smp-75-sp1

https://www-secure.symantec.com/connect/articles/about-different-cases-troubleshooting-cem-functionality