Endpoint Protection

hi guys!

we ran into a pretty strange problem after the migration to mr6. we have 2 sepm servers (sepm001 higher priority than sepm002) and updated the manager from mr5 to mr6. so far so good, the migration ran without problems or failures.

after some days i realized that the client of sepm001 wasn't updated anymore (sepm002 and about 3500 clients are up2date) and does not receive policy updates or issued command from sepm. the sylink log shows the following:

08/16 07:31:11 [2032] <DoHeartbeat> Set heartbeat event
08/16 07:31:11 [2032] Use new configuration
08/16 07:31:11 [2032] <CSyLink::IndexHeartbeatProc()>
08/16 07:31:11 [2032] <IndexHeartbeatProc> Got ConfigObject to proceed the operation.. pSylinkConfig: 0000000003C0E990
08/16 07:31:11 [2032] <IndexHeartbeatProc>====== Reg Heartbeat loop starts at 07:31:11 ======
08/16 07:31:11 [2032] HEARTBEAT: Check Point 1
08/16 07:31:11 [2032] Get First Server!
08/16 07:31:11 [2032] HEARTBEAT: Check Point 2
08/16 07:31:11 [2032] <PostEvent>going to post event=EVENT_SERVER_CONNECTING
08/16 07:31:11 [2032] <PostEvent>done post event=EVENT_SERVER_CONNECTING, return=0
08/16 07:31:11 [2032] HEARTBEAT: Check Point 3
08/16 07:31:11 [2032] <IndexHeartbeatProc>Setting the session timeout on Profile Session to 30000
08/16 07:31:11 [2032] HEARTBEAT: Check Point 4
08/16 07:31:11 [2032] <IndexHeartbeatProc>===Get Index STAGE===
08/16 07:31:11 [2032] ************CSN=172
08/16 07:31:11 [2032] <mfn_MakeGetIndexUrl:>Request is: action=12&hostid=5CDDB8DF0A0B0A29005539D8FAF1E859&chk=430554F9733BC86746AB422DF6E51274&ck=315657C9EE705352A974E8BC840CAE99&uchk=92A952B2E7B321AB284D3571ABB50304&uck=72AB0578E26C21B1EB2D76F4D5EB7260&hid=08962AEE571C8E5D0CF42F73C40063D9&groupid=DE2A0F630A0B0A290169EAAA73215882&mode=0&hbt=1800&as=172&cn=[hex]5345504D303031&lun=[hex]4141553037393639&udn=[hex]4E45542E424C554D
08/16 07:31:11 [2032] <GetIndexFileRequest:>http://sepm001.domainname:8014/secars/secars.dll?h=50AF2BCD87E1DA2E8A.....
08/16 07:31:12 [2032] <GetIndexFileRequest:>SMS return=200
08/16 07:31:12 [2032] <ParseHTTPStatusCode:>200=>200 OK
08/16 07:31:12 [2032] <mfn_DoGetIndexFile200>Content Lenght => 0
08/16 07:31:12 [2032] <GetIndexFileRequest:>RECEIVE STAGE COMPLETED
08/16 07:31:12 [2032] <GetIndexFileRequest:>COMPLETED
08/16 07:31:12 [2032] <IndexHeartbeatProc>GetIndexFile handling status: 1
08/16 07:31:12 [2032] <IndexHeartbeatProc>Switch Server flag=0
08/16 07:31:12 [2032] HEARTBEAT: Check Point 5.1
08/16 07:31:12 [2032] Index File Error!
08/16 07:31:12 [2032] <PostEvent>going to post event=EVENT_SERVER_ONLINE
08/16 07:31:12 [2032] <PostEvent>done post event=EVENT_SERVER_ONLINE, return=0
08/16 07:31:12 [2032] <ScheduleNextUpdate>new scheduled heartbeat=1024 seconds
08/16 07:31:12 [2032] HEARTBEAT: Check Point 8
08/16 07:31:12 [2032] <PostEvent>going to post event=EVENT_SERVER_DISCONNECTED
08/16 07:31:12 [2032] <PostEvent>done post event=EVENT_SERVER_DISCONNECTED, return=0
08/16 07:31:12 [2032] <IndexHeartbeatProc>====== IndexHeartbeat Procedure stops at 07:31:12 ======
08/16 07:31:12 [2032] <IndexHeartbeatProc>Set Heartbeat Result= 3
08/16 07:31:12 [2032] <IndexHeartbeatProc>Sylink Comm.Flags: 'Connection Failed' = 1, 'Using Backup Sylink' = 1, 'Using Location Config' = 0
08/16 07:31:12 [2032] <IndexHeartbeatProc>Connection Failed! No. of tries = 3

i tried the following to fix the issue:
- telnet sepm -> ok
- http://sepm001:8014/secars/secars?hello,secars -> ok
- export communication settings and import them in the client (with and without sylinkdrop)
- reinstall client
- uninstall mr5 and install mr6 client
- created new client settings with the "delete all logs and old communication settings" and reinstalled the client with these settings

does anyone know how to fix this?

thanks in advance!

try this document

http://service1.symantec.com/SUPPORT/ent-security.nsf/docid/2008100811312148

hi rafeeq - thanks for the fast response!

oops i forgot to mention that i checked the proxy settings already.

the proxy settings of the machine aren't set in the registry and no proxy is configured elsewhere

Are you using Windows 2008 R2 as on OS on the machine in which the SEP client is not able to communicate with the SEPM. If yes then either disable IPv6 or Create a new Management Server List that only specifies IPv4 addresses.

To disable ipv6 please check the Microsoft article below.

http://support.microsoft.com/kb/929852

try by disabling windows firewall....

no its only windows 2008 - no R2. and it worked for nearly a year before we updated to mr6

firewall is disabled!

If all above does not work; try rebooting the box once: i see that it communicating successfully with return code 200
delete the client if its in SEPM ; update policy on the small icon on the task bar.

While you are taking the sylink log at the same time on the client update the policy and then paste as the sylink.log

 

@rafeeq: after i did the reboot, deleted the client from sepm and hit the update policy button the client showed up again in the console, but only for a few minutes. now it's marked as offline (no yellow dot in the console). the patters weren't updated. the only change is, that the client changed the status in the console from "not reporting status" to "normal" but offline. the last checked in date is also set to the time after i hit the update policy button.

@prachand: the sylink log during a update policy shows exactly the same as the one i posted in my first post. could you please clarify if you want me to do something other?

and btw. ipv6 is disabled since i installed the server!

now it seems to be a certificate issue; it authenticated the client an then disconnected
click on the group the client reports to
click on policies on the right hand side
click on connections ( something like that i dont remember)
uncheck require certificate;
replace sylink from any group
update the policy again
see if that comes with constant green dot

maybe it's a stupid question, but how can it be a certificate issue when about 3500 other clients are working properly?

i'm already at home now - i will try your your suggestion tomorrow morning...

now it seems to be a certificate issue; it authenticated the client an then disconnected
click on the group the client reports to
click on policies on the right hand side
click on connections ( something like that i dont remember)
uncheck require certificate;
replace sylink from any group
update the policy again
see if that comes with constant green dot

Is the server using the SEP firewall?  If not, for Server 2008, the Windows firewall is enabled by default.  If the Windows firewall does not have an exception for 8014 it won't be able to talk to itself, i.e. communicate with the SEPM.

If the clients are using NTP that would expain why they can communicate.

sandra

i did the following:

• 
  changed the certificate option over client-group -> general settings -> security settings -> "Enable secure communications between the management server and clients by using digital certificates for authentication"
• export communication settings from the group
• delete the client from the group
• smc -stop
• use sylinkdrop to replace the new sylink
• smc -start
• update policy
• green dot and the client did the pattern updates

i'm now trying to change the policy back, that the clients use the certificate und check if everything is working. i post as soon as i know more and mark the solution.

do you know why this client (and only this client) wasn't able to authenticate with the certificate?

ok i was too enthusiastic

the policy update is only working when the certificate authentication is disabled. AND i sometimes need to reboot the client, that the new policy is active. if i don't reboot the policy stays old.

is there a way to get the client working with the certificate again?

Try by repairing the SEP client from add/remove programs now.

If above suggestion not helps run SEP support tool and see the error.
The Symantec Endpoint Protection Support Tool

hi AravinkKM,

first thanks for your respone, but i don't get how this should help since the sep client is working properly? only the communication with sepm in association with the certificate fails.

i did the repair with no error. also the support tool showed nothing irregular...

Check for the registry entries.
 
1. HKCU\SOFTWARE\Microsoft\Windows\Currentversion\Internet settings Click on the internet setting key check for the keys called "ProxyEnable" if it is set to 1 then change it to 0 also check if there is a registry value called "GlobalUserOffline" if it is present delete it
2. Now expand Internet settings key and take a backup of the "Connections" key, Delete the entire key
 
3. Check HKU\.Default\SOFTWARE\Microsoft\Windows\Currentversion\Internet settings Click on the internet setting key check for the keys called "ProxyEnable" if it is set to 1 then change it to 0 also check if there is a registry value called "GlobalUserOffline" if it is present delete it
 
4. expand "Internet settings" key in the above said location and take a backup of the "Connections" key, Delete the entire key
 
5. Reboot the machine.

i checked the proxy settings already - nothing set.

if there were wrong proxy settings or a problem with the proxy the sylink would show a proxy error, but there is nothing.

the client is working as supposed when the "use certificate to authenticate" is NOT set. so i only need to get the client up and running with certificate authentication...

While you are replacing the sylink file SEP client will get the certificates.Can you create a new sylink file and replace in the client and try.You cal also try the repair option,it can restore any missing registry/file ..

does it make a difference if i import the sylink manually or do an update policy after i set the checkbox to use the certificate again? i think it's the same, isn't it?

Does the sylink debug log actually show "Signature verification FAILED for Index File Content." when certificate authenitcation is on?  Is the sylink.xml one that was freshly exported from the SEPM?

sandra

no, i got the "index file error!" but i will try to export the new settings and import them....

i just exported the communication settings from sepm and load them in the client but the policy serial and date is still the old one. this client drives me crazy *?*

the sylink log shows the same as in the first post:

08/17 17:01:56 [3556] <DoHeartbeat> Set heartbeat event
08/17 17:01:56 [3556] Use new configuration
08/17 17:01:56 [3556] <CSyLink::IndexHeartbeatProc()>
08/17 17:01:56 [3556] <IndexHeartbeatProc> Got ConfigObject to proceed the operation.. pSylinkConfig: 0000000003CFDC90
08/17 17:01:56 [3556] <IndexHeartbeatProc>====== Reg Heartbeat loop starts at 17:01:56 ======
08/17 17:01:57 [3556] HEARTBEAT: Check Point 1
08/17 17:01:57 [3556] Get First Server!
08/17 17:01:57 [3556] HEARTBEAT: Check Point 2
08/17 17:01:57 [3556] <PostEvent>going to post event=EVENT_SERVER_CONNECTING
08/17 17:01:57 [3556] <PostEvent>done post event=EVENT_SERVER_CONNECTING, return=0
08/17 17:01:57 [3556] HEARTBEAT: Check Point 3
08/17 17:01:57 [3556] <IndexHeartbeatProc>Setting the session timeout on Profile Session to 30000
08/17 17:01:57 [3556] HEARTBEAT: Check Point 4
08/17 17:01:57 [3556] <IndexHeartbeatProc>===Get Index STAGE===
08/17 17:01:57 [3556] ************CSN=270
08/17 17:01:57 [3556] <mfn_MakeGetIndexUrl:>Request is: action=12&hostid=F00567240A0B0A29009A1411BA9C1B35&chk=430554F9733BC86746AB422DF6E51274&ck=315657C9EE705352A974E8BC840CAE99&uchk=92A952B2E7B321AB284D3571ABB50304&uck=72AB0578E26C21B1EB2D76F4D5EB7260&hid=08962AEE571C8E5D0CF42F73C40063D9&groupid=DE2A0F630A0B0A290169EAAA73215882&mode=0&hbt=1800&as=270&cn=[hex]5345504D303031&lun=[hex]4141553037393639&udn=[hex]4E45542E424C554D
08/17 17:01:57 [3556] <GetIndexFileRequest:>http://sepm001.localdomain:8014/secars/secars.dll?h=50AF2BCD87E1DA2E8A3ECEBCA566CAC8270459CB287B5FA0DE7EF54769375B01F1C5AFD3909EC3CEE760F5DF7D38863CE0CAB7FAFAF0E4A9AE55EE90B4E43456366EA18C80E756FED40FE1AD76BD80703D8A1EBD53CD8AE9AB015DC2509537357C8A815DE2C1E9D79EFB844AF7960DB14FFB25922166FFD14A0ACACFACD87B0E8A75EB9B4F2C2C8F8A71B855B89D2E142B291EAF9C5A4B32D5CD9F6F483790391AB18F1F7198771D713DB38D3141583AA78E55FFD84AAD79C2FE349B311C1539A84F4CC6C7C420490D99681E0B345859E1FBC6638776F1A69D0ACFBBD38BD515056CC5FCEDAE8463CC8A91C99AE01FC12ACD0F8D8A8CDB53B1B6022FF4A1884C4C8397489D0B25F8123758236FF6B1F6A876E8E02A6123A50A6DB8DFA56900D0F079393995C40ADFFFEE56EE7DB0142333C39461F031F54634595A73EF0DC2F2D7D4ACC86B38DB12B3723DFF23F480842F9B3D0D7C125ADF26E0D6180DDA835825BFFA853DF3E4197324083AAE5E34FFC78763F18C2EE2447F3DDFFA096763C4
08/17 17:01:57 [3556] <GetIndexFileRequest:>SMS return=200
08/17 17:01:57 [3556] <ParseHTTPStatusCode:>200=>200 OK
08/17 17:01:57 [3556] <mfn_DoGetIndexFile200>Content Lenght => 0
08/17 17:01:57 [3556] <GetIndexFileRequest:>RECEIVE STAGE COMPLETED
08/17 17:01:57 [3556] <GetIndexFileRequest:>COMPLETED
08/17 17:01:57 [3556] <IndexHeartbeatProc>GetIndexFile handling status: 1
08/17 17:01:57 [3556] <IndexHeartbeatProc>Switch Server flag=0
08/17 17:01:57 [3556] HEARTBEAT: Check Point 5.1
08/17 17:01:57 [3556] Index File Error!
08/17 17:01:57 [3556] <PostEvent>going to post event=EVENT_SERVER_ONLINE
08/17 17:01:57 [3556] <PostEvent>done post event=EVENT_SERVER_ONLINE, return=0
08/17 17:01:57 [3556] <ScheduleNextUpdate>new scheduled heartbeat=256 seconds
08/17 17:01:57 [3556] HEARTBEAT: Check Point 8
08/17 17:01:57 [3556] <PostEvent>going to post event=EVENT_SERVER_DISCONNECTED
08/17 17:01:57 [3556] <PostEvent>done post event=EVENT_SERVER_DISCONNECTED, return=0
08/17 17:01:57 [3556] <IndexHeartbeatProc>====== IndexHeartbeat Procedure stops at 17:01:57 ======
08/17 17:01:57 [3556] <IndexHeartbeatProc>Set Heartbeat Result= 3
08/17 17:01:57 [3556] <IndexHeartbeatProc>Sylink Comm.Flags: 'Connection Failed' = 1, 'Using Backup Sylink' = 1, 'Using Location Config' = 0
08/17 17:01:57 [3556] <IndexHeartbeatProc>Connection Failed! No. of tries = 1
08/17 17:01:57 [3556] Use new configuration
08/17 17:01:57 [3556] HEARTBEAT: Check Point Complete
08/17 17:01:57 [3556] <IndexHeartbeatProc>Done, Heartbeat=256seconds
08/17 17:01:57 [3556] </CSyLink::IndexHeartbeatProc()>
08/17 17:01:57 [3556] <CheckHeartbeatTimer>====== Heartbeat loop stops at 17:01:57 ======
08/17 17:02:03 [3264] <CSyLink::mfn_DownloadNow()>
08/17 17:02:03 [3264] </CSyLink::mfn_DownloadNow()>
08/17 17:03:06 [3264] <CSyLink::mfn_DownloadNow()>
08/17 17:03:06 [3264] </CSyLink::mfn_DownloadNow()>

 

Turn off IE ESC and try....

IE ESC was already disabled

Do you tried by repairing SEP now?

yes, but no change. the communication with sepm is working when i uncheck the use certificate checkbox and import the sylink. but as soon as i change the setting to use the certificate the communication is dead and the sylink log shows the index file error...

Put the check mark,export sylink replace it in the client.It should be able to communicate...

i can do whatever i want, the policy stays the same even when i import a sylink from another group. the client showed in the client is wrong. when i delete the client from the console and add the sylink from another group the client gets registered to the correct group but the client has the old policy from the wrong group.

Whether the client is showing online in other group?
If yes try by moving it to the proper group and restarting smc service in the client...

helped nothing! the client has a green dot, was displayed as online in the console and the last check-in date was also maintained but no policy and pattern updates are sent to the client.

is it possible, that the last check-in date is set after the server sends the 200 OK to the client?

try this also
Delete %programfiles%\Common Files\Symantec Shared\HWID\sephwid.xml
Open the registry and navigate to HKLM\Software\Symantec\Symantec Endpoint Protection\SMC\Sylink\Sylnk
Edit the "HardwareID" value data to be blank
Restart the Symantec Management Client (SMC) service in the services snap-in.

no success either :(

i deleted the xml file and blanked the regkey and restarted smc. after that i changed the policy back to use the certificate -> update policy -> new policy arrieved, but now the index file error is back...

Can you paste/attach the sylink file here?

there you go...

Your sylink file looks ok.Replace the sylibnk file with _cer and try.Before copting remove any sylink present in sep folder along with its backups......... 

maybe this is also helpful for you: i just installed updates on my second sepm (sepm002) and realized that all clients (i just took some samples from desktops and servers) were communicating with sepm002, which has a higher priority and should only be the backup sepm. when i hit update policy on a client during the reboot of the box, the sep client showed itself as offline -> no sepm available.

Take a copy of SerDef.dat and sylink.xml from a working PC.It will be under C:\Program Files\Symantec\Symantec Endpoint Protection folder.Move these original files along with its backup files from the problematic client(You have to stop smc service by using smc -stop command before doing this.)Put the new files in problematic pc in C:\Program Files\Symantec\Symantec Endpoint Protection then start SMC service(smc -start command).

still no change

i tried both of your suggestions - the index file error persists :(

any other suggestions?

i just did an update to ru6 mp1, but the problem isn't fixed

sylink log is still showing the Index File Error.

does someone know if the cleanwipe tool touches the sepm? if it only affects the client i want to use it to completly erase the sep client installation...

Cleanwipe tool will remove SEPM also.....If you want to remove only SEP you can try by using manual uninstalltion KB
How to manually uninstall Symantec Endpoint Protection client from Windows Vista, Windows 7, and Windows 2008 32-bit

Have you've tried to check if Simple File Sharing is disabled? Sometimes it affects the updating of clients to server.

yes simple filesharing is disabled... and 3500 other clients are able to update so i don't thing the problem is the smb :)

@AravindKM: i tried it - but it also affected the sepm :( i will cleanwipe the system and reinstall sepm, add it with the same name to the site and load the backed up certificate. i hope after that i will be able to update the servers client

Yes you can try.
Keep this Kb in your mind
Best Practices for Disaster Recovery with Symantec Endpoint Protection

thx, i know this chapter :)

does it make any difference if i have two sepm servers (higher and lower prio) connected to a sql server. isn't it possible to just join the newly installed sepm (same ip and dns name like the old sepm) to the site? when i then enter the correct name of the sepm it should place itself as the "old" server, right? after that i replace the cert and the manager should be back?!

I think that should do.Anyway for as a best practice take a backup as per that KB and keep....

open sepm
click on domain
select add new domain
right click  and select administer this domain
create a new group
take this sylink file
replace this sylink file to the client
check if the green dot is constant..

this solved the problem:

- cleanwipe the whole system
- reinstall sepm and join to site (same ip and name than the old sepm had)
- update certificate from backup
- create install settings where the old communication settings are ignored (in case there is something on the system)
- install client and check the sylink log if everything is working properly

thanks everyone for your suggestions!