Endpoint Protection

I have 4 sites. Each site has a management server and the replicate in a hub and spoke configuration. I have configured location awareness based on IP and it seems to work fine. But I have an ongoing issue going back to SEPM 11 where the Symantec Endpoint Protection Manager service occasionally stops on the management servers (although this is not what I want help with here). My issues is that when the local management server stops, all of the clients move to a management server across the WAN, despite the fact that none of the MSL's have remote servers listed. Then when I bring up the local server, none of the clients switch back. What I would like is for clients to either not update until the local server comes back or use live update as the secondary. And what I really need is help getting them all to switch back to the local management server once it is back up. I have 12.1.2015

I would suggest enabling sylink logging on one of the affected clients and letting it run through a few heartbeats in order to see what it is doing:

How to enable Sylink debugging for the Symantec Endpoint Protection 11.x and 12.1 client in the Windows Registry

Article:TECH104758

|

Created: 2008-01-18

|

Updated: 2013-02-26

|

Article URL http://www.symantec.com/docs/TECH104758

 

Hmmm, I have a quick few questions for you:

1. Are you using custom MSLs on each of the "Spoke" sites to point clients in those sites to only the "Spoke"  SEPM?
2. In the event of a SEPM becoming unavalable, do the clients revert to the same SEPM each time or does it vary?
3. If it's the same one each time, is it the one that you exported the SEP Client install packages from originally?

Essentially what I'm getting at is:

When first creating/exporting a SEP Client install package, it contains the sylink communications file which (depending on what MSL was assigned to the group you used for the export template listed at the time of the export) may list all of your SEPMs.
In the event a SEP client loses connectivity with it's SEPM:

• it will first onnect to the last known good SEPM in the registry,
• if it cannot connect to any of these, it will go through the MSL it picked up from the SEPM via policy,
• if it cannot connect to any of these, it will connect to the SEPMs listed in the sylink communications file

Sooooo it sounds to me that the SEP client package included a sylink file that lists multiple SEPMs, and that you are experiencing the below issue:

http://www.symantec.com/docs/TECH92879

Which means you either need to start messing with regkeys, or ask your users to leave their machines on for a day or two...

I have done this and have a log file. What am I looking for in it? I am not sure I feel comfortable posting the full thing online.

When I load machines, 9 times out of 10 I do it at the hub site where I push the AV to the machine (from the hub site management server) and place it in the appropriate group. Then I transport the machine to the remote site and set it up. Starting from boot all machines seem to connect to their local management server as expected. All of the MSL's list only their local management server, no secondaries. So far this has happened twice where a server at a spoke site goes down (the service not the whole server) and all of the machines switch to the hub site management server. I did discover that stopping the smc.exe process will cause the machine to revert back to the local server. Since the process restarts itself instantly it seems a good way to switch the servers back without rebooting them. The desktop/laptop clients on the other hand are another matter. I was able to send a reboot command to the group and switch them all back but it caused me a few phone calls. The issue described in the link sounds like what I have happening except that I have no failover management servers listed on any of my MSL's. I was looking at the sylink log that Brian had me create above and I did see an entry early in the log saying something like "last server" and then listing the IP of the hub site server (not the local server). However when I examine the reg key now it shows the correct management server IP. I looked on another machine and it too now shows the local management server IP in that last server reg key so I don't know.

This all sounds consistent with what I expected.  I reckon it's doing the following (in the event the local "spoke" SEPM is unavailable):

1. Try to connect to the last good SEPM (the local spoke SEPM) and fail
2. Try to connect to any other SEPM in the MSL (in this this only the local spoke SEPM) and fail
3. Try to connect to any SEPMs listed in the sylink file (in this case the HUB SEPM) and succeed (which becomes the last known good SEPM)
4. Pick up MSL policy pointing it back at the spoke SEPM (fail as it's still down), and continue using last known good SEPM (the HUB one)

Now assuming you get the local spoke SEPM up at this point, you should then get the following behaviour (according to the article I linked):

1. SEP Client connects to last known good SEPM (the HUB one)
2. After 24 hours, check for for connectivity to the SEPM defined in the MSL (local spoke SEPM)
3. Successfully connect to the local spoke SEPM (which becomes the last known good SEPM)

The only difference is that because your MSL lists only the local/spoke SEPM, the SEP clients are switching back to it when you stop the smc process.

Another option you may consider, to force all clients back to the local SEPM after it's up again, would be to temporarily block the SEP Client communications port (8014 by default) between your spoke site and the HUB.  This should only need to be for as long as two heartbeats, then you can open it up again.

Just pick whichever method is easier for pointing clients back at the local SEPM 

Try this

 

Please double sheck your MSL and applied groups

Comapare the policy number of each group with the corresponding clients in the same group on problematic clients..

As it turns out I had occasion to test some things this morning as I was having electrical work done at a spoke site and had to turn off the servers but not the desktops. As expected the desktops switched over to the hub site management server. I blocked port 8014 on the hub site server for the remote subnets and this did disconnect the machines that had moved to it. However the machines are not reconnecting with their local site management server. This is strange to me because they switch to the hub site server with just an hour or so of downtime on their local site management server but even with more than an hour of not being able to reach the hub site they sit unconnected to any server.

Well the problem appears to be that the clients never switch back to their local management server. I did some more testing and with the ports blocked so that they cannot switch to the hub site server they go into unmanaged state and when the local server comes back they never reconnect. I don't think it is the issue outlined in the tech article you posted SMLatCST because the last server reg key has the correct server listed (because the port block is keeping them from failing over to the other server at all). Location awareness is working because at a reboot they connect to the server they should (and roaming laptops connect to the local server for wherever they happen to be). Location awareness is configured to recheck location every 30 minutes (the default) but that does not seem to be happening as there is never a switch unless they are rebooted or the SMC.exe process is restarted.