Endpoint Protection

I have a test version of MR4 MP1 that looks good, but I have to delay deploying it while a project freeze is in place.  In the meantime, I'm having problems with our production version, which is MR2 MP2.

What I can see is that on test where things work, clients connect, stay connected and look okay. On prod where things dont, I really dont see much in the way of visiting clients.  Only if someone does an 'Update Policy' from the client side do I see a connect attempt.

I ran a sniffer and I can see the client do a GET request, 200 OK reply, where its passing a 'secars' kind of thing a big hash string.

I compared a Sylink.xml file from a newly generated test package and it matches exactly whats out in the clients Sylink.xml

I ran the SylinkMonitor tool and got this output.  It seems like the problem is in an index file?

 

03/18 10:04:50 [3812] Stored HostGUID=120957BA80D0A9380198DC193316E7C3; outlen=16
03/18 10:04:50 [3812] <RestoreSettings>Stored UserGuid=0; outlen=2
03/18 10:04:50 [3812] <mfn_DecodeSSN>Sygate-SSN=11496
03/18 10:04:50 [3812] <mfn_DecodeSSN>Read CSN=11497
03/18 10:04:50 [3812] Product Type=3,Major Ver=5,Minor Ver=2,Platform ID=2,OSType=50659842
03/18 10:04:50 [3812] OS=Windows Server 2003 family Standard Edition; number=5.2.3790
03/18 10:04:50 [3812] SyLinkCreateInstance => Instance created: 01C72008 Registry path: SOFTWARE\Symantec\Symantec Endpoint Protection\SMC\SYLINK
03/18 10:04:50 [3812] <GetOnlineNicInfo>:Netport Count=1
03/18 10:04:50 [3812] <GetOnlineNicInfo>:NicInfo<SSANICs><SSANIC Ip="128.208.169.56" Mac="00-17-08-4c-be-5a" Gateway="128.208.169.100" SubnetMask="0.0.0.0"/></SSANICs>
03/18 10:04:50 [3812] SyLinkCreateConfig => Created instance: 01C6A008
03/18 10:04:50 [3812] UseNewConfig => Created m_hNewConfig: 01C6A008
03/18 10:04:50 [3812] Importing ConfigObject: 01C66008 into: 01C6A008
03/18 10:04:50 [3812] Importing ConfigObject: 01C66008 into: 01C682F8
03/18 10:04:50 [3812] SSA packageType is set as 105
03/18 10:04:50 [3812] SyLinkDeleteConfig => Deleting instance: 01C66008
03/18 10:04:50 [3812] <SetHiStatus>HI status is changed to=3; reason=0; rule=Host Integrity check is disabled.
 Host Integrity policy has been disabled by the administrator.
03/18 10:04:52 [3812] SyLinkCreateConfig => Created instance: 01B626D8
03/18 10:04:52 [3812] Set current location=FCEFEC1D8C8EC0C9009F1A7724A1DF1B
03/18 10:04:52 [3812] SyLinkDeleteConfig => Deleting instance: 01B626D8
03/18 10:04:52 [1064] <ScheduleNextUpdate>Manually assigned heartbeat=1 seconds
03/18 10:04:52 [5320] <ScheduleNextUpdate>Manually assigned heartbeat=1 seconds
03/18 10:04:52 [3812] <CSyLink::Start()>
03/18 10:04:52 [3812] <CSyLink::ImportConfigFile()>
03/18 10:04:52 [3812] </CSyLink::ImportConfigFile()>
03/18 10:04:52 [3812] <GetDomainHostName>msz_DomainName is taken from szDomainName
03/18 10:04:52 [3812] <GetDomainHostName>DomainName (Final)=amc.uwmedicine.org
03/18 10:04:52 [3812] *********Netport Count=1
03/18 10:04:52 [3812] Physical: Local Area Connection::00-17-08-4c-be-5a::hp nc7782 gigabit server adapter
03/18 10:04:52 [3812] MAC=00-17-08-4c-be-5a# Wireless=
03/18 10:04:52 [3812] Hardwire String=00-17-08-4c-be-5a#
03/18 10:04:52 [3812] <Start>Unable to create Session with 'User Proxy' settings - Proxy Server: Error Code: 87
03/18 10:04:52 [3812] <Start>Unable to create Session with 'No Proxies' settings - Error Code: 87
03/18 10:04:52 [1292] <HeartbeatThreadProc:>Thread is about to begin..
03/18 10:04:52 [3812] <Start>Started, contact SMS every 300 seconds
03/18 10:04:52 [1292] <GetOnlineNicInfo>:Netport Count=1
03/18 10:04:52 [1292] <GetOnlineNicInfo>:NicInfo<SSANICs><SSANIC Ip="128.208.169.56" Mac="00-17-08-4c-be-5a" Gateway="128.208.169.100" SubnetMask="0.0.0.0"/></SSANICs>
03/18 10:04:52 [4304] <CExpBackoff::CExpBackoff()>
03/18 10:04:52 [4304] </CExpBackoff::CExpBackoff()>
03/18 10:04:52 [3812] </CSyLink::Start()>
03/18 10:04:54 [3812] <SetClientAuth>Received new User/Domain from SMC..  User: mcis User Domain: ITS-SIT-SFWWEB2
03/18 10:04:54 [3812] <SetClientAuth>Getting RDNS Domain Name (user domain in AD setup)..
03/18 10:04:54 [3812] <GetLoginRdnsDomain>DNS domain=
03/18 10:04:54 [3812] <SetClientAuth>Checking if domain name is the same as Local Host ..
03/18 10:04:54 [3812] <SetClientAuth>HostName=its-sit-sfwweb2
03/18 10:04:54 [3812] <SetClientAuth>DomainName=its-sit-sfwweb2
03/18 10:04:54 [3812] <SetClientAuth>User Domain name is same as Host name .. Setting 'Login Domain' to =LocalComputer
03/18 10:04:54 [3812] <SetClientAuth>Logged in user info set to: LocalComputer/mcis
03/18 10:04:54 [3812] <SetClientAuth>Marking User Change Notify to redo registration..
03/18 10:04:54 [5320] SyLinkCreateConfig => Created instance: 01C03670
03/18 10:04:54 [5320] Importing ConfigObject: 01C682F8 into: 01C03670
03/18 10:04:54 [5320] SyLinkDeleteConfig => Deleting instance: 01C03670
03/18 10:04:58 [1292] <CalcAgentHashKey>:CH=BBFB77BD80D0A938000F597CD4CF11731its-sit-sfwweb2amc.uwmedicine.org5F094E7698A4D8C7F807454453AAA7E4
03/18 10:04:58 [1292] <CalcAgentHashKey>:CHKey=80529A05E5FC17D82D8D0290CE24C409
03/18 10:04:58 [1292] <CalcAgentHashKey>:C=BBFB77BD80D0A938000F597CD4CF11731its-sit-sfwweb2amc.uwmedicine.org
03/18 10:04:58 [1292] <CalcAgentHashKey>:CKey=4606C62A8489D7798FEEAD86D43C715E
03/18 10:04:58 [1292] <CalcAgentHashKey>:UCH=BBFB77BD80D0A938000F597CD4CF11730mcisLocalComputerits-sit-sfwweb2amc.uwmedicine.org5F094E7698A4D8C7F807454453AAA7E4
03/18 10:04:58 [1292] <CalcAgentHashKey>:UCHKey=3DA251D82AFB324A651933510E11862A
03/18 10:04:58 [1292] <CalcAgentHashKey>:UC=BBFB77BD80D0A938000F597CD4CF11730mcisLocalComputerits-sit-sfwweb2amc.uwmedicine.org
03/18 10:04:58 [1292] <CalcAgentHashKey>:UCKey=62171FAEB1AE032FF7C220F338089D56
03/18 10:04:58 [1292] <DoHeartbeat>HardwareID=5F094E7698A4D8C7F807454453AAA7E4
03/18 10:04:58 [1292] <DoHeartbeat>CHKey=80529A05E5FC17D82D8D0290CE24C409
03/18 10:04:58 [1292] <DoHeartbeat>CKey=4606C62A8489D7798FEEAD86D43C715E
03/18 10:04:58 [1292] <DoHeartbeat>UCHKey=3DA251D82AFB324A651933510E11862A
03/18 10:04:58 [1292] <DoHeartbeat>UCKey=62171FAEB1AE032FF7C220F338089D56
03/18 10:04:58 [1292] <DoHeartbeat> Set heartbeat event
03/18 10:04:58 [1292] Use new configuration
03/18 10:04:58 [1292] <RegHeartbeatProc>====== Reg Heartbeat loop starts at 10:04:58 ======
03/18 10:04:59 [1292] HEARTBEAT: Check Point 1
03/18 10:04:59 [1292] HEARTBEAT: Check Point 2
03/18 10:04:59 [1292] <PostEvent>going to post event=EVENT_SERVER_CONNECTING
03/18 10:04:59 [1292] <PostEvent>done post event=EVENT_SERVER_CONNECTING, return=0
03/18 10:04:59 [1292] HEARTBEAT: Check Point 3
03/18 10:04:59 [1292] <RegHeartbeatProc>Setting the session timeout on Profile Session (Registration) to 30000
03/18 10:04:59 [1292] HEARTBEAT: Check Point 4
03/18 10:04:59 [1292] <RegHeartbeatProc>===Registration STAGE===
03/18 10:04:59 [1292] <MakeRegisterData:>logon id (domain/user)=LocalComputer/mcis

read error, exit
03/18 10:04:59 [1292] <SendRegistrationRequest:>SMS return=468
03/18 10:04:59 [1292] <ParseHTTPStatusCode:>468=>468 Request not allowed
03/18 10:04:59 [1292] <SendRegistrationRequest:>Content Lenght => 48
03/18 10:04:59 [1292] <mfn_DecodeSSN>Sygate-SSN=13
03/18 10:04:59 [1292] <mfn_DecodeSSN>Read CSN=14
03/18 10:04:59 [1292] HTTP returns status code=468
03/18 10:04:59 [1292] <SendRegistrationRequest:>RECEIVE STAGE COMPLETED
03/18 10:04:59 [1292] <SendRegistrationRequest:>COMPLETED
03/18 10:04:59 [1292] HEARTBEAT: Check Point 5.1
03/18 10:04:59 [1292] <RegHeartbeatProc>switch to another server
03/18 10:04:59 [1292] HEARTBEAT: Check Point 9
03/18 10:04:59 [1292] HEARTBEAT: Check Point 8
03/18 10:04:59 [1292] <PostEvent>going to post event=EVENT_SERVER_DISCONNECTED
03/18 10:04:59 [1292] <PostEvent>done post event=EVENT_SERVER_DISCONNECTED, return=0
03/18 10:04:59 [1292] HEARTBEAT: Check Point 1
03/18 10:04:59 [1292] HEARTBEAT: Check Point 2
03/18 10:04:59 [1292] <PostEvent>going to post event=EVENT_SERVER_CONNECTING
03/18 10:04:59 [1292] <PostEvent>done post event=EVENT_SERVER_CONNECTING, return=0
03/18 10:04:59 [1292] HEARTBEAT: Check Point 3
03/18 10:04:59 [1292] <RegHeartbeatProc>Setting the session timeout on Profile Session (Registration) to 30000
03/18 10:04:59 [1292] HEARTBEAT: Check Point 4
03/18 10:04:59 [1292] <RegHeartbeatProc>===Registration STAGE===
03/18 10:04:59 [1292] <MakeRegisterData:>logon id (domain/user)=LocalComputer/mcis

read error, exit
03/18 10:05:00 [1292] <SendRegistrationRequest:>SMS return=200
03/18 10:05:00 [1292] <ParseHTTPStatusCode:>200=>200 OK
03/18 10:05:00 [1292] <SendRegistrationRequest:>Content Lenght => 350
03/18 10:05:00 [1292] <SendRegistrationRequest:>RECEIVE STAGE COMPLETED
03/18 10:05:00 [1292] <SendRegistrationRequest:>COMPLETED
03/18 10:05:00 [1292] HEARTBEAT: Check Point 5.1
03/18 10:05:00 [1292] <ScheduleNextUpdate>Manually assigned heartbeat=3 seconds
03/18 10:05:00 [1292] <PostEvent>going to post event=EVENT_SERVER_ONLINE
03/18 10:05:00 [1292] <PostEvent>done post event=EVENT_SERVER_ONLINE, return=0
03/18 10:05:00 [1292] HEARTBEAT: Check Point 8
03/18 10:05:00 [1292] <PostEvent>going to post event=EVENT_SERVER_DISCONNECTED
03/18 10:05:00 [1292] <PostEvent>done post event=EVENT_SERVER_DISCONNECTED, return=0
03/18 10:05:00 [1292] <RegHeartbeatProc>====== Registration Procedure stops at 10:05:00 ======
03/18 10:05:00 [1292] HEARTBEAT: Check Point 10
03/18 10:05:00 [1292] HEARTBEAT: Check Point Complete
03/18 10:05:00 [1292] <RegHeartbeatProc>Done, Heartbeat=3seconds
03/18 10:05:01 [1292] <CheckHeartbeatTimer>====== Heartbeat loop starts at 10:05:01 ======
03/18 10:05:01 [1292] <GetOnlineNicInfo>:Netport Count=1
03/18 10:05:01 [1292] <GetOnlineNicInfo>:NicInfo<SSANICs><SSANIC Ip="128.208.169.56" Mac="00-17-08-4c-be-5a" Gateway="128.208.169.100" SubnetMask="0.0.0.0"/></SSANICs>
03/18 10:05:01 [1292] <CalcAgentHashKey>:CH=BBFB77BD80D0A938000F597CD4CF11731its-sit-sfwweb2amc.uwmedicine.org5F094E7698A4D8C7F807454453AAA7E4
03/18 10:05:01 [1292] <CalcAgentHashKey>:CHKey=80529A05E5FC17D82D8D0290CE24C409
03/18 10:05:01 [1292] <CalcAgentHashKey>:C=BBFB77BD80D0A938000F597CD4CF11731its-sit-sfwweb2amc.uwmedicine.org
03/18 10:05:01 [1292] <CalcAgentHashKey>:CKey=4606C62A8489D7798FEEAD86D43C715E
03/18 10:05:01 [1292] <CalcAgentHashKey>:UCH=BBFB77BD80D0A938000F597CD4CF11730mcisLocalComputerits-sit-sfwweb2amc.uwmedicine.org5F094E7698A4D8C7F807454453AAA7E4
03/18 10:05:01 [1292] <CalcAgentHashKey>:UCHKey=3DA251D82AFB324A651933510E11862A
03/18 10:05:01 [1292] <CalcAgentHashKey>:UC=BBFB77BD80D0A938000F597CD4CF11730mcisLocalComputerits-sit-sfwweb2amc.uwmedicine.org
03/18 10:05:01 [1292] <CalcAgentHashKey>:UCKey=62171FAEB1AE032FF7C220F338089D56
03/18 10:05:01 [1292] <DoHeartbeat>HardwareID=5F094E7698A4D8C7F807454453AAA7E4
03/18 10:05:01 [1292] <DoHeartbeat>CHKey=80529A05E5FC17D82D8D0290CE24C409
03/18 10:05:01 [1292] <DoHeartbeat>CKey=4606C62A8489D7798FEEAD86D43C715E
03/18 10:05:01 [1292] <DoHeartbeat>UCHKey=3DA251D82AFB324A651933510E11862A
03/18 10:05:01 [1292] <DoHeartbeat>UCKey=62171FAEB1AE032FF7C220F338089D56
03/18 10:05:01 [1292] <DoHeartbeat> Set heartbeat event
03/18 10:05:01 [1292] Use new configuration
03/18 10:05:01 [1292] <CSyLink::IndexHeartbeatProc()>
03/18 10:05:01 [1292] <IndexHeartbeatProc> Got ConfigObject to proceed the operation.. pSylinkConfig: 01C682F8
03/18 10:05:01 [1292] <IndexHeartbeatProc>====== Reg Heartbeat loop starts at 10:05:01 ======
03/18 10:05:02 [1292] HEARTBEAT: Check Point 1
03/18 10:05:02 [1292] Get First Server!
03/18 10:05:02 [1292] HEARTBEAT: Check Point 2
03/18 10:05:02 [1292] <PostEvent>going to post event=EVENT_SERVER_CONNECTING
03/18 10:05:02 [1292] <PostEvent>done post event=EVENT_SERVER_CONNECTING, return=0
03/18 10:05:02 [1292] HEARTBEAT: Check Point 3
03/18 10:05:02 [1292] <IndexHeartbeatProc>Setting the session timeout on Profile Session to 30000
03/18 10:05:02 [1292] HEARTBEAT: Check Point 4
03/18 10:05:02 [1292] <IndexHeartbeatProc>===Get Index STAGE===
03/18 10:05:02 [1292] ************CSN=11498
03/18 10:05:02 [1292] <mfn_MakeGetIndexUrl:>Request is: action=12&hostid=120957BA80D0A9380198DC193316E7C3&chk=80529A05E5FC17D82D8D0290CE24C409&ck=4606C62A8489D7798FEEAD86D43C715E&uchk=3DA251D82AFB324A651933510E11862A&uck=62171FAEB1AE032FF7C220F338089D56&hid=5F094E7698A4D8C7F807454453AAA7E4&groupid=BBFB77BD80D0A938000F597CD4CF1173&mode=0&hbt=900&as=11498&cn=[hex]6974732D7369742D73667777656232&lun=[hex]6D636973&udn=[hex]4C6F63616C436F6D7075746572
03/18 10:05:02 [1292] <GetIndexFileRequest:>http://128.208.169.56:80/secars/secars.dll?h=DDBB205A64A53ADC6B3FFE73EFF5EA980702538FCD81EC567D146CB5F4D097461C657689E6001DADC12F410C6E32533F4525241CD909508F8638744DA63F70CA64B429BAD998C01B3C5F97978E0BB32D08079CC845A052B5521E44221737D287374B9E141B90D59B224698588EC0402077902CB239D6C254650EEAFFEF11A03A1D88114D019645C64B01E985DA3630C18213CC6B771EA5A7390603AA70F6C812AD6F0AE6266BEBDC67E6EFE50507ED77372A8FC727E31A61FBDC48E1972304A0F278CB35F01BC4109BAB3762C09284B9F11A5F0BFF0CF62E7A2FD7F29D7AE6035C61E1EB5BC1349911D90E8B70B12B56B52C71090FB98B084B97EC2F638969B2F087B66062EFE5229E7845164541E4CC353052C9A0AD39EF7BCC071D83231F5E170B74030EB5B9B059227948FEDC67AF15BEBC89569274B2F626095C9048735C894EE0EE90CFBDD80E796A40170745D2EB92937612BA3DD2441E8E7FE5A6C62B983D44DDDD94BA8E47991C0D7DC925F950567D36B1ADE4EBDED390CCB3CA992E564827D78592DA26CAF735EB94DA9CEB
03/18 10:05:02 [1292] <GetIndexFileRequest:>SMS return=200
03/18 10:05:02 [1292] <ParseHTTPStatusCode:>200=>200 OK
03/18 10:05:02 [1292] <FindHeader>Sem-HashKey:=>80529A05E5FC17D82D8D0290CE24C409
03/18 10:05:02 [1292] <FindHeader>Sem-LANSensor:=>0
03/18 10:05:02 [1292] <FindHeader>Sem-Signatue:=>30E0F102D1E0236BEF01137C4987A0DF8CE1E99944182D735ABE1F560C433B850AB117F17906A66A2BC5958F6AEE57ACE7CFBEB4E0CF7EC1F76D4D8115D09E156C0EB7DB1EBF809149799E8C67C8D13047006CF9E67E11A63AB3975236FA97DE8A03B1BC466B54DAC509B42092DBEDB6FE9744792F337EA2AF429E93380D1FF9
03/18 10:05:02 [1292] <mfn_DoGetIndexFile200>Content Lenght => 1183
03/18 10:05:02 [1292] <mfn_DoGetIndexFile200>Signature verification FAILED for Index File Content..
03/18 10:05:02 [1292] <GetIndexFileRequest:>RECEIVE STAGE COMPLETED
03/18 10:05:02 [1292] <GetIndexFileRequest:>COMPLETED
03/18 10:05:02 [1292] <IndexHeartbeatProc>GetIndexFile handling status: 101
03/18 10:05:02 [1292] <IndexHeartbeatProc>Switch Server flag=0
03/18 10:05:02 [1292] HEARTBEAT: Check Point 5.1
03/18 10:05:02 [1292] <ScheduleNextUpdate>new scheduled heartbeat=32 seconds
03/18 10:05:02 [1292] HEARTBEAT: Check Point 8
03/18 10:05:02 [1292] Notify Server down!
03/18 10:05:02 [1292] <PostEvent>going to post event=EVENT_SERVER_DISCONNECTED
03/18 10:05:02 [1292] <PostEvent>done post event=EVENT_SERVER_DISCONNECTED, return=0
03/18 10:05:02 [1292] Get Next Server!
03/18 10:05:02 [1292] <IndexHeartbeatProc>switch to another server
03/18 10:05:02 [1292] HEARTBEAT: Check Point 1
03/18 10:05:02 [1292] HEARTBEAT: Check Point 2
03/18 10:05:02 [1292] <PostEvent>going to post event=EVENT_SERVER_CONNECTING
03/18 10:05:02 [1292] <PostEvent>done post event=EVENT_SERVER_CONNECTING, return=0
03/18 10:05:02 [1292] HEARTBEAT: Check Point 3
03/18 10:05:02 [1292] <IndexHeartbeatProc>Setting the session timeout on Profile Session to 30000
03/18 10:05:02 [1292] HEARTBEAT: Check Point 4
03/18 10:05:02 [1292] <IndexHeartbeatProc>===Get Index STAGE===
03/18 10:05:02 [1292] ************CSN=11499
03/18 10:05:02 [1292] <mfn_MakeGetIndexUrl:>Request is: action=12&hostid=120957BA80D0A9380198DC193316E7C3&chk=80529A05E5FC17D82D8D0290CE24C409&ck=4606C62A8489D7798FEEAD86D43C715E&uchk=3DA251D82AFB324A651933510E11862A&uck=62171FAEB1AE032FF7C220F338089D56&hid=5F094E7698A4D8C7F807454453AAA7E4&groupid=BBFB77BD80D0A938000F597CD4CF1173&mode=0&hbt=900&as=11499&cn=[hex]6974732D7369742D73667777656232&lun=[hex]6D636973&udn=[hex]4C6F63616C436F6D7075746572
03/18 10:05:02 [1292] <GetIndexFileRequest:>http://its-sit-sfwweb2:80/secars/secars.dll?h=DDBB205A64A53ADC6B3FFE73EFF5EA980702538FCD81EC567D146CB5F4D097461C657689E6001DADC12F410C6E32533F4525241CD909508F8638744DA63F70CA64B429BAD998C01B3C5F97978E0BB32D08079CC845A052B5521E44221737D287374B9E141B90D59B224698588EC0402077902CB239D6C254650EEAFFEF11A03A1D88114D019645C64B01E985DA3630C18213CC6B771EA5A7390603AA70F6C812AD6F0AE6266BEBDC67E6EFE50507ED77372A8FC727E31A61FBDC48E1972304A0F278CB35F01BC4109BAB3762C09284B9F11A5F0BFF0CF62E7A2FD7F29D7AE6035C61E1EB5BC1349911D90E8B70B12B56B52C71090FB98B084B97EC2F638969B2F087B66062EFE5229E7845164541E4CC353052C9A0AD39EF7BCC071D83231F5E170B74030EB5B9B059227948FEDC67AF552027470CC0097ABB0B5F678AF11410894EE0EE90CFBDD80E796A40170745D2EB92937612BA3DD2441E8E7FE5A6C62B983D44DDDD94BA8E47991C0D7DC925F950567D36B1ADE4EBDED390CCB3CA992E564827D78592DA26CAF735EB94DA9CEB
03/18 10:05:02 [1292] <GetIndexFileRequest:>SMS return=200
03/18 10:05:02 [1292] <ParseHTTPStatusCode:>200=>200 OK
03/18 10:05:02 [1292] <FindHeader>Sem-HashKey:=>80529A05E5FC17D82D8D0290CE24C409
03/18 10:05:02 [1292] <FindHeader>Sem-LANSensor:=>0
03/18 10:05:02 [1292] <FindHeader>Sem-Signatue:=>30E0F102D1E0236BEF01137C4987A0DF8CE1E99944182D735ABE1F560C433B850AB117F17906A66A2BC5958F6AEE57ACE7CFBEB4E0CF7EC1F76D4D8115D09E156C0EB7DB1EBF809149799E8C67C8D13047006CF9E67E11A63AB3975236FA97DE8A03B1BC466B54DAC509B42092DBEDB6FE9744792F337EA2AF429E93380D1FF9
03/18 10:05:02 [1292] <mfn_DoGetIndexFile200>Content Lenght => 1183
03/18 10:05:02 [1292] <mfn_DoGetIndexFile200>Signature verification FAILED for Index File Content..
03/18 10:05:02 [1292] <GetIndexFileRequest:>RECEIVE STAGE COMPLETED
03/18 10:05:02 [1292] <GetIndexFileRequest:>COMPLETED
03/18 10:05:02 [1292] <IndexHeartbeatProc>GetIndexFile handling status: 101
03/18 10:05:02 [1292] <IndexHeartbeatProc>Switch Server flag=0
03/18 10:05:02 [1292] HEARTBEAT: Check Point 5.1
03/18 10:05:02 [1292] <ScheduleNextUpdate>new scheduled heartbeat=64 seconds
03/18 10:05:02 [1292] HEARTBEAT: Check Point 8
03/18 10:05:02 [1292] Get Next Server!
03/18 10:05:02 [1292] <PostEvent>going to post event=EVENT_SERVER_DISCONNECTED
03/18 10:05:02 [1292] <PostEvent>done post event=EVENT_SERVER_DISCONNECTED, return=0
03/18 10:05:02 [1292] <IndexHeartbeatProc>====== IndexHeartbeat Procedure stops at 10:05:02 ======
03/18 10:05:02 [1292] <IndexHeartbeatProc>Set Heartbeat Result= 1
03/18 10:05:02 [1292] <IndexHeartbeatProc>Sylink Comm.Flags: 'Connection Failed' = 0, 'Using Backup Sylink' = 0, 'Using Location Config' = 0
03/18 10:05:02 [1292] Use new configuration
03/18 10:05:02 [1292] HEARTBEAT: Check Point Complete
03/18 10:05:02 [1292] <IndexHeartbeatProc>Done, Heartbeat=64seconds
03/18 10:05:02 [1292] </CSyLink::IndexHeartbeatProc()>
03/18 10:05:02 [1292] <CheckHeartbeatTimer>====== Heartbeat loop stops at 10:05:02 ======
03/18 10:05:53 [1872] <CSyLink::mfn_DownloadNow()>
03/18 10:05:53 [1872] </CSyLink::mfn_DownloadNow()>
03/18 10:06:07 [1292] <CheckHeartbeatTimer>====== Heartbeat loop starts at 10:06:07 ======
03/18 10:06:08 [1292] <GetOnlineNicInfo>:Netport Count=1
03/18 10:06:08 [1292] <GetOnlineNicInfo>:NicInfo<SSANICs><SSANIC Ip="128.208.169.56" Mac="00-17-08-4c-be-5a" Gateway="128.208.169.100" SubnetMask="0.0.0.0"/></SSANICs>
03/18 10:06:08 [1292] <CalcAgentHashKey>:CH=BBFB77BD80D0A938000F597CD4CF11731its-sit-sfwweb2amc.uwmedicine.org5F094E7698A4D8C7F807454453AAA7E4
03/18 10:06:08 [1292] <CalcAgentHashKey>:CHKey=80529A05E5FC17D82D8D0290CE24C409
03/18 10:06:08 [1292] <CalcAgentHashKey>:C=BBFB77BD80D0A938000F597CD4CF11731its-sit-sfwweb2amc.uwmedicine.org
03/18 10:06:08 [1292] <CalcAgentHashKey>:CKey=4606C62A8489D7798FEEAD86D43C715E
03/18 10:06:08 [1292] <CalcAgentHashKey>:UCH=BBFB77BD80D0A938000F597CD4CF11730mcisLocalComputerits-sit-sfwweb2amc.uwmedicine.org5F094E7698A4D8C7F807454453AAA7E4
03/18 10:06:08 [1292] <CalcAgentHashKey>:UCHKey=3DA251D82AFB324A651933510E11862A
03/18 10:06:08 [1292] <CalcAgentHashKey>:UC=BBFB77BD80D0A938000F597CD4CF11730mcisLocalComputerits-sit-sfwweb2amc.uwmedicine.org
03/18 10:06:08 [1292] <CalcAgentHashKey>:UCKey=62171FAEB1AE032FF7C220F338089D56
03/18 10:06:08 [1292] <DoHeartbeat>HardwareID=5F094E7698A4D8C7F807454453AAA7E4
03/18 10:06:08 [1292] <DoHeartbeat>CHKey=80529A05E5FC17D82D8D0290CE24C409
03/18 10:06:08 [1292] <DoHeartbeat>CKey=4606C62A8489D7798FEEAD86D43C715E
03/18 10:06:08 [1292] <DoHeartbeat>UCHKey=3DA251D82AFB324A651933510E11862A
03/18 10:06:08 [1292] <DoHeartbeat>UCKey=62171FAEB1AE032FF7C220F338089D56
03/18 10:06:08 [1292] <DoHeartbeat> Set heartbeat event
03/18 10:06:08 [1292] Use new configuration
03/18 10:06:08 [1292] <CSyLink::IndexHeartbeatProc()>
03/18 10:06:08 [1292] <IndexHeartbeatProc> Got ConfigObject to proceed the operation.. pSylinkConfig: 01C682F8
03/18 10:06:08 [1292] <IndexHeartbeatProc>====== Reg Heartbeat loop starts at 10:06:08 ======
03/18 10:06:08 [1292] HEARTBEAT: Check Point 1
03/18 10:06:08 [1292] Get First Server!
03/18 10:06:08 [1292] HEARTBEAT: Check Point 2
03/18 10:06:08 [1292] <PostEvent>going to post event=EVENT_SERVER_CONNECTING
03/18 10:06:08 [1292] <PostEvent>done post event=EVENT_SERVER_CONNECTING, return=0
03/18 10:06:08 [1292] HEARTBEAT: Check Point 3
03/18 10:06:08 [1292] <IndexHeartbeatProc>Setting the session timeout on Profile Session to 30000
03/18 10:06:08 [1292] HEARTBEAT: Check Point 4
03/18 10:06:08 [1292] <IndexHeartbeatProc>===Get Index STAGE===
03/18 10:06:08 [1292] ************CSN=11500
03/18 10:06:08 [1292] <mfn_MakeGetIndexUrl:>Request is: action=12&hostid=120957BA80D0A9380198DC193316E7C3&chk=80529A05E5FC17D82D8D0290CE24C409&ck=4606C62A8489D7798FEEAD86D43C715E&uchk=3DA251D82AFB324A651933510E11862A&uck=62171FAEB1AE032FF7C220F338089D56&hid=5F094E7698A4D8C7F807454453AAA7E4&groupid=BBFB77BD80D0A938000F597CD4CF1173&mode=0&hbt=900&as=11500&cn=[hex]6974732D7369742D73667777656232&lun=[hex]6D636973&udn=[hex]4C6F63616C436F6D7075746572
03/18 10:06:08 [1292] <GetIndexFileRequest:>http://its-sit-sfwweb2:80/secars/secars.dll?h=DDBB205A64A53ADC6B3FFE73EFF5EA980702538FCD81EC567D146CB5F4D097461C657689E6001DADC12F410C6E32533F4525241CD909508F8638744DA63F70CA64B429BAD998C01B3C5F97978E0BB32D08079CC845A052B5521E44221737D287374B9E141B90D59B224698588EC0402077902CB239D6C254650EEAFFEF11A03A1D88114D019645C64B01E985DA3630C18213CC6B771EA5A7390603AA70F6C812AD6F0AE6266BEBDC67E6EFE50507ED77372A8FC727E31A61FBDC48E1972304A0F278CB35F01BC4109BAB3762C09284B9F11A5F0BFF0CF62E7A2FD7F29D7AE6035C61E1EB5BC1349911D90E8B70B12B56B52C71090FB98B084B97EC2F638969B2F087B66062EFE5229E7845164541E4CC353052C9A0AD39EF7BCC071D83231F5EFF24AB47C8828A0DB6285E5ED6CC657BFE8BD84C9DDA2FDCCABEC0CC1CEFF7E8894EE0EE90CFBDD80E796A40170745D2EB92937612BA3DD2441E8E7FE5A6C62B983D44DDDD94BA8E47991C0D7DC925F950567D36B1ADE4EBDED390CCB3CA992E564827D78592DA26CAF735EB94DA9CEB
03/18 10:06:08 [1292] <GetIndexFileRequest:>SMS return=200
03/18 10:06:08 [1292] <ParseHTTPStatusCode:>200=>200 OK
03/18 10:06:08 [1292] <FindHeader>Sem-HashKey:=>80529A05E5FC17D82D8D0290CE24C409
03/18 10:06:08 [1292] <FindHeader>Sem-LANSensor:=>0
03/18 10:06:08 [1292] <FindHeader>Sem-Signatue:=>30E0F102D1E0236BEF01137C4987A0DF8CE1E99944182D735ABE1F560C433B850AB117F17906A66A2BC5958F6AEE57ACE7CFBEB4E0CF7EC1F76D4D8115D09E156C0EB7DB1EBF809149799E8C67C8D13047006CF9E67E11A63AB3975236FA97DE8A03B1BC466B54DAC509B42092DBEDB6FE9744792F337EA2AF429E93380D1FF9
03/18 10:06:08 [1292] <mfn_DoGetIndexFile200>Content Lenght => 1183
03/18 10:06:08 [1292] <mfn_DoGetIndexFile200>Signature verification FAILED for Index File Content..
03/18 10:06:08 [1292] <GetIndexFileRequest:>RECEIVE STAGE COMPLETED
03/18 10:06:08 [1292] <GetIndexFileRequest:>COMPLETED
03/18 10:06:08 [1292] <IndexHeartbeatProc>GetIndexFile handling status: 101
03/18 10:06:08 [1292] <IndexHeartbeatProc>Switch Server flag=0
03/18 10:06:08 [1292] HEARTBEAT: Check Point 5.1
03/18 10:06:08 [1292] <ScheduleNextUpdate>new scheduled heartbeat=128 seconds
03/18 10:06:08 [1292] HEARTBEAT: Check Point 8
03/18 10:06:08 [1292] Get Next Server!
03/18 10:06:08 [1292] <IndexHeartbeatProc>switch to another server
03/18 10:06:08 [1292] <DecrementScheduleTime:>New scheduled heartbeat=64 seconds
03/18 10:06:09 [1292] HEARTBEAT: Check Point 1
03/18 10:06:09 [1292] HEARTBEAT: Check Point 2
03/18 10:06:09 [1292] <PostEvent>going to post event=EVENT_SERVER_CONNECTING
03/18 10:06:09 [1292] <PostEvent>done post event=EVENT_SERVER_CONNECTING, return=0
03/18 10:06:09 [1292] HEARTBEAT: Check Point 3
03/18 10:06:09 [1292] <IndexHeartbeatProc>Setting the session timeout on Profile Session to 30000
03/18 10:06:09 [1292] HEARTBEAT: Check Point 4
03/18 10:06:09 [1292] <IndexHeartbeatProc>===Get Index STAGE===
03/18 10:06:09 [1292] ************CSN=11501
03/18 10:06:09 [1292] <mfn_MakeGetIndexUrl:>Request is: action=12&hostid=120957BA80D0A9380198DC193316E7C3&chk=80529A05E5FC17D82D8D0290CE24C409&ck=4606C62A8489D7798FEEAD86D43C715E&uchk=3DA251D82AFB324A651933510E11862A&uck=62171FAEB1AE032FF7C220F338089D56&hid=5F094E7698A4D8C7F807454453AAA7E4&groupid=BBFB77BD80D0A938000F597CD4CF1173&mode=0&hbt=900&as=11501&cn=[hex]6974732D7369742D73667777656232&lun=[hex]6D636973&udn=[hex]4C6F63616C436F6D7075746572
03/18 10:06:09 [1292] <GetIndexFileRequest:>http://128.208.169.56:80/secars/secars.dll?h=DDBB205A64A53ADC6B3FFE73EFF5EA980702538FCD81EC567D146CB5F4D097461C657689E6001DADC12F410C6E32533F4525241CD909508F8638744DA63F70CA64B429BAD998C01B3C5F97978E0BB32D08079CC845A052B5521E44221737D287374B9E141B90D59B224698588EC0402077902CB239D6C254650EEAFFEF11A03A1D88114D019645C64B01E985DA3630C18213CC6B771EA5A7390603AA70F6C812AD6F0AE6266BEBDC67E6EFE50507ED77372A8FC727E31A61FBDC48E1972304A0F278CB35F01BC4109BAB3762C09284B9F11A5F0BFF0CF62E7A2FD7F29D7AE6035C61E1EB5BC1349911D90E8B70B12B56B52C71090FB98B084B97EC2F638969B2F087B66062EFE5229E7845164541E4CC353052C9A0AD39EF7BCC071D83231F5EFF24AB47C8828A0DB6285E5ED6CC657BDF5BFCBD0EBB80F25D916973D820554F894EE0EE90CFBDD80E796A40170745D2EB92937612BA3DD2441E8E7FE5A6C62B983D44DDDD94BA8E47991C0D7DC925F950567D36B1ADE4EBDED390CCB3CA992E564827D78592DA26CAF735EB94DA9CEB
03/18 10:06:09 [1292] <GetIndexFileRequest:>SMS return=200
03/18 10:06:09 [1292] <ParseHTTPStatusCode:>200=>200 OK
03/18 10:06:09 [1292] <FindHeader>Sem-HashKey:=>80529A05E5FC17D82D8D0290CE24C409
03/18 10:06:09 [1292] <FindHeader>Sem-LANSensor:=>0
03/18 10:06:09 [1292] <FindHeader>Sem-Signatue:=>30E0F102D1E0236BEF01137C4987A0DF8CE1E99944182D735ABE1F560C433B850AB117F17906A66A2BC5958F6AEE57ACE7CFBEB4E0CF7EC1F76D4D8115D09E156C0EB7DB1EBF809149799E8C67C8D13047006CF9E67E11A63AB3975236FA97DE8A03B1BC466B54DAC509B42092DBEDB6FE9744792F337EA2AF429E93380D1FF9
03/18 10:06:09 [1292] <mfn_DoGetIndexFile200>Content Lenght => 1183
03/18 10:06:09 [1292] <mfn_DoGetIndexFile200>Signature verification FAILED for Index File Content..
03/18 10:06:09 [1292] <GetIndexFileRequest:>RECEIVE STAGE COMPLETED
03/18 10:06:09 [1292] <GetIndexFileRequest:>COMPLETED
03/18 10:06:09 [1292] <IndexHeartbeatProc>GetIndexFile handling status: 101
03/18 10:06:09 [1292] <IndexHeartbeatProc>Switch Server flag=0
03/18 10:06:09 [1292] HEARTBEAT: Check Point 5.1
03/18 10:06:09 [1292] <ScheduleNextUpdate>new scheduled heartbeat=128 seconds
03/18 10:06:09 [1292] HEARTBEAT: Check Point 8
03/18 10:06:09 [1292] Get Next Server!
03/18 10:06:09 [1292] <PostEvent>going to post event=EVENT_SERVER_DISCONNECTED
03/18 10:06:09 [1292] <PostEvent>done post event=EVENT_SERVER_DISCONNECTED, return=0
03/18 10:06:09 [1292] <IndexHeartbeatProc>====== IndexHeartbeat Procedure stops at 10:06:09 ======
03/18 10:06:09 [1292] <IndexHeartbeatProc>Set Heartbeat Result= 1
03/18 10:06:09 [1292] <IndexHeartbeatProc>Sylink Comm.Flags: 'Connection Failed' = 0, 'Using Backup Sylink' = 0, 'Using Location Config' = 0
03/18 10:06:09 [1292] Use new configuration
03/18 10:06:09 [1292] HEARTBEAT: Check Point Complete
03/18 10:06:09 [1292] <IndexHeartbeatProc>Done, Heartbeat=128seconds
03/18 10:06:09 [1292] </CSyLink::IndexHeartbeatProc()>
03/18 10:06:09 [1292] <CheckHeartbeatTimer>====== Heartbeat loop stops at 10:06:09 ======

***[0x568]:[2009-03-18 10:06:12:468]***SylinkMonitor Stopped
 

By the way, we only use Network Threat Protection.  We use McAfee for antivirus issues (long story).

We dont use proxy servers and have none of those kinds of settings.

The firewalls function fine, they just dont get policy updates.  We're working around this right now by leaving machines in the field alone unless the firewall change is urgent.  If it's truly urgent, we have to uninstall prod, install test, and then make the rule changes on the test server.

I'm not sure what I can do though besides that.

 

http://localhost/secars/secars.dll?hello,secars   works returns OK

 

but in the tomcat log (scm-0-server.log) I have:

 

2009-03-17 09:17:33.437 SEVERE: ================== Server Environment ===================
2009-03-17 09:17:33.437 SEVERE: os.name = Windows 2003
2009-03-17 09:17:33.437 SEVERE: os.version = 5.2
2009-03-17 09:17:33.437 SEVERE: os.arch = x86
2009-03-17 09:17:33.437 SEVERE: java.version = 1.5.0_14
2009-03-17 09:17:33.437 SEVERE: java.vendor = Sun Microsystems Inc.
2009-03-17 09:17:33.437 SEVERE: java.vm.name = Java HotSpot(TM) Server VM
2009-03-17 09:17:33.437 SEVERE: java.vm.version = 1.5.0_14-b03
2009-03-17 09:17:33.437 SEVERE: java.home = C:\Program Files\Symantec\Symantec Endpoint Protection Manager\jdk\jre
2009-03-17 09:17:33.437 SEVERE: catalina.home = C:\Program Files\Symantec\Symantec Endpoint Protection Manager\tomcat
2009-03-17 09:17:33.437 SEVERE: java.user = null
2009-03-17 09:17:33.437 SEVERE: user.language = en
2009-03-17 09:17:33.437 SEVERE: user.country = US
2009-03-17 09:17:33.437 SEVERE: scm.server.version = 11.0.2020.56
2009-03-17 09:17:37.515 SEVERE: ================== StartClientTransport ===================
2009-03-17 09:17:38.062 SEVERE: Schedule is started!
2009-03-17 09:17:39.078 SEVERE: StateCheckpointTask connect to secars failed: SERVICE NOT AVAILABLE
2009-03-17 09:17:39.796 SEVERE: IISCacheTask connect to secars failed: SERVICE NOT AVAILABLE
2009-03-17 13:32:19.777 SEVERE: Broken content link detected! Skipping content: {1CD85198-26C6-4bac-8C72-5D34B025DE35} Revision: 90305002 Reference Type: ObjReference ID: 859C0F82B1993B0F8B26C0F2F92EC744
2009-03-17 13:34:31.917 SEVERE: Broken content link detected! Skipping content: {C60DC234-65F9-4674-94AE-62158EFCA433} Revision: 90305002 Reference Type: ObjReference ID: 9CA82D68126AF724C5A92C56F9FBF636
2009-03-17 17:37:55.354 SEVERE: Broken content link detected! Skipping content: {CC40C428-1830-44ef-B8B2-920A0B761793} Revision: 90202019 Reference Type: ObjReference ID: 5FE5DB30773DEE5F855D9AD809E3D127
2009-03-17 21:43:37.180 SEVERE: Broken content link detected! Skipping content: {1CD85198-26C6-4bac-8C72-5D34B025DE35} Revision: 90305038 Reference Type: ObjReference ID: 8C2BE890BF553296D336C127F6A39E21
2009-03-17 21:45:54.570 SEVERE: Broken content link detected! Skipping content: {C60DC234-65F9-4674-94AE-62158EFCA433} Revision: 90305038 Reference Type: ObjReference ID: DD5974B6E258602A5ED449AC8625A4FB
2009-03-18 01:50:02.382 SEVERE: Broken content link detected! Skipping content: {1CD85198-26C6-4bac-8C72-5D34B025DE35} Revision: 90305050 Reference Type: ObjReference ID: 8BD93916D152AC2736A9CA9BCF05E013
2009-03-18 01:52:08.632 SEVERE: Broken content link detected! Skipping content: {C60DC234-65F9-4674-94AE-62158EFCA433} Revision: 90305050 Reference Type: ObjReference ID: 2ED14D77690B801D769415098071C802
2009-03-18 05:55:33.179 SEVERE: Broken content link detected! Skipping content: {812CD25E-1049-4086-9DDD-A4FAE649FBDF} Revision: 90202019 Reference Type: ObjReference ID: 64BF63E465D0E570D65A22FF77775FBA
2009-03-18 10:01:00.549 SEVERE: Broken content link detected! Skipping content: {1CD85198-26C6-4bac-8C72-5D34B025DE35} Revision: 90306004 Reference Type: ObjReference ID: F58995FF7FBEC45A3475FA11C45978A9
2009-03-18 10:03:06.345 SEVERE: Broken content link detected! Skipping content: {C60DC234-65F9-4674-94AE-62158EFCA433} Revision: 90306004 Reference Type: ObjReference ID: 2C7E94D465485BFD1EDA082B31713CDB
2009-03-18 11:42:17.432 SEVERE: Schedule is stopped!
 

Okay, going with the error message EVENT_SERVER_DISCONNECTED, it looks like the service 'Symantec Event Manager' is somehow implicated.

I used Microsoft's autoruns tool, and this tells me that that service is actually:

ccEvtMgrEvent propagation and logging service Symantec Corporation c:\program files\common files\symantec shared\ccsvchst.exe 

and in the Services window:

"C:\Program Files\Common Files\Symantec Shared\ccSvcHst.exe" /h ccCommon

So it looks like something I could chase down.  I went to the directory C:\Program Files\Common Files\Symantec Shared

and then explored, and one more folder down is:

C:\Program Files\Common Files\Symantec Shared\SymcData

and in there are a bunch of directories:

cndcipsdefs

sesmipsdef32

sesmipsdef64

sesmvirdef32

sesmvirdef64

 

Exploring around some more, the files under sesmvirdef# look fine, with todays dates, but sesmipsdef# look like they have the dates that things last broke - 3/12. 

I suspect that this is where the bad index or bad content is.

Right now, it is so damn tempting to erase the directory made on 3/11

 

 

Hi Tyoud, Thanks for posting all the logs, it makes it easier to give a correct answer the first time.

The logs show you have 2 problems, perhaps related.
First, your SEPM database has some corruption which needs to be fixed before moving forward. In this state your Manager most likely cannot communicate properly with any of it's clients. When you upgrade your server to a newer SEPM server, there are tools that run during the upgrade process that fix known database issues that previous versions, such as MR2, can cause. So here is what I suggest you do.

1) Upgrade to the latest version of SEPM -- RU5 was just released.
2) After you upgrade, open a command prompt and navigate to the SEPM install directory.
3) You'll see a subdirectory called "Tools", open that.
4) Run the file, DBValidator.bat
If this says there are NO errors, then check the scm-server-0.log file. If there are no "Broken Content Link" errors then the problem has been resolved.
If the DBValidator BAT says there is still an error, or you still have the "Broken Content Link" errors, open a support case. Tell them you have a corrupt database. The database file itself isn't physically corrupt, but some of the data inside is not valid.

If you don't want to migrate this machine (producting environment) then export the database and import it into a test machine and try the upgrade there. If you absolutely must have this machine fixed before you upgrade, then open a support case and be ready to give Support a backup of the database. Sometimes they will give you a tool to fix just your DB
I really suggest migrating the server to RU5, it will probably be a lot faster than calling support. If it fixes the problem, your done! (and you have an updated server). If not, you'll have to call support anyways.


Now before you address the second issue, first address the first issue. The second issue may be related to the first, and you even if you fix the second issue, you really can't do anything until the first issue has been addressed.

The second issue is that the client is rejecting the servers communication because the policy files have not been signed correctly. This is because there is a certificate mis-match between the SEPM server and the client. The index file itself is not bad, but the client cannot verify it came from the correct source.
This may happen if you un-install and re-install the SEPM server. It may also happen if run the Configuration Wizard multiple times. We are trying to prevent these issues from happening in the next major version of SEPM.

The client stores a certificate in the file called Sylink.xml. If this certificate does not match the servers certificate, the client rejects the servers communication. This is a security feature to prevent man-in-the-middle type attacks.
To correct this issue you can either:
1) Manually copy an updated sylink.xml file onto all your clients client.
See this KB article: http://service1.symantec.com/support/ent-security.nsf/854fa02b4f5013678825731a007d06af/13669c4f8319b89e882574e5004e7328?OpenDocument
or
2) Restore the old SEPM servers certificate file.
http://seer.entsupport.symantec.com/docs/330611.htm

Lastly, I'm not sure what your final post was about, but I'm sure if you address the two issues above (in order) that you problems will go away.