Endpoint Protection

I suspect I’m getting bad advice from Symantec Support, therefore asking for suggestions of theproblem I outline below.

Problem: Symantec service Symantec Endpoint Protection (SepMasterService) randomly doesn’t start on a reboot or booting. A reboot or running SymDiag with Debug Logging option fixes the problem temporally. The Debud Logging option from SymDiag runs a command to stop and then start the Symantec Endpoint Protection (SepMasterService) service at which point this service starts normally. Windows Event Logs indicate that the Symantec Endpoint Protection (SepMasterService) service stops on shutdown, but on boot there is no call to start this necessary service.

Affected Operating Systems: Windows 10 Professional x64 1709 (not observed with Windows 10 1703 or our Windows 7 computers)

Affected Version of SEP Client: 14.0.3897.1101 (14 RU1 MP1b) and 14.0.3892.1101 (14 RU1 MP1a)

Diagnostics Provide to Symantec Support: Numerous SymDiags from several different computers with and without Debug Logging option have been provided while the problem was occurring

Initial Proposed Solutions from Support: I was initially told this was a known bug with 14.0.3892.1101 (14 RU1 MP1a) and therefore I upgraded SEPMs to 14.0.3897.1101 (14 RU1 MP1b), upgrade affected clients, and additional computers with 1709 version of Windows 10. However this problem seems to be more wide spread with 14.0.3897.1101 (14 RU1 MP1b) and latest update for Windows 10 1709, KB4088776 (https://support.microsoft.com/en-hk/help/4088776), released a week ago.

Current Proposed Solutions from Support (as I understand): Install KB4073290 per TECH248552 (https://support.symantec.com/en_US/article.TECH248552.html) because this problem has been seen by Symantec Support and only solution from Symantec Support is to install the corresponding KB listed in TECH248552 (https://support.symantec.com/en_US/article.TECH248552.html) while 14.0.3897.1101 (14 RU1 MP1b) is installed.

Problems with Current Proposed Solution from Support: Checking Microsoft’s KB article for KB4073290 (https://support.microsoft.com/en-hk/help/4073290/unbootable-state-for-amd-devices-in-windows-10-version-1709) stated that is a cumulative update for Windows 10 Version 1709 with AMD processors only. The affected systems are running Intel processors including the system (Intel Core i5-6500) looked at by support this afternoon via WebEx. Also this cumulative update has been superseded by sequential cumulative updates, including KB4088776 (https://support.microsoft.com/en-hk/help/4088776) released by Microsoft March 13, 2018, and installed on the affected systems.

Background History of these 1709 Systems: These systems were originally Windows 10 x64 Professional 1703 systems which had both KB4057144 (listed in TECH248552 (https://support.symantec.com/en_US/article.TECH248552.html) and 14.0.3892.1101 (14 RU1 MP1a), had one or more cumulative updates for 1703 installed (KB4074592, KB4077528, KB4092077), upgraded to 1709, then latest cumulative update(s) for 1709 installed, and then SEP upgraded to 14.0.3897.1101 (14 RU1 MP1b).

Any ideas or suggestions would be greatly appreciated.

Can you upload one of the Symdiags you sent into support or provide a case number so I can look into it for you?

Thanks,

John Owens

Thanks John. The SymDiags are attached to Case 14209612 (I tried uploading SymDiag in this form, but received error “A file could not be uploaded. Please file sizes and extensions meet the criteria above.” Open Cases 14242716 and 14248348 could be related.

John, I'm trying to reattach the SymDiag logs by zipping them...

Attachment(s)

Case 14209612 - Event Logs for Krafty.zip   1.02 MB 1 version

Reviewing.  This does not appear to be related to the issues with the January 3rd Microsoft updates.

Thanks for reviewing John. How do you recommend I proceed with Symantec Support to have this problem troubleshooted and resolved since they want me to install a Microsoft update from January for AMD processors when I have an Intel CPU?

So it looks like the service does not start after a reboot, correct?  Is it able to start up manually?  I am also noticing the Apple Mobile Device service is timingout as well.

3/15/2018 6:30:07 PM    System    Information    User32    krafty.systemsbiology.net    ISB\skraft    1074    "The process C:\Windows\System32\RuntimeBroker.exe (KRAFTY) has initiated the restart of computer KRAFTY on behalf of user ISB\skraft for the following reason: Other (Unplanned)
 Reason Code: 0x0
 Shutdown Type: restart
 Comment:"

3/15/2018 6:31:44 PM    System    Information    EventLog    krafty.systemsbiology.net        6013    The system uptime is 35 seconds.

3/15/2018 6:32:29 PM    System    Error    Service Control Manager    krafty.systemsbiology.net        7009    A timeout was reached (30000 milliseconds) while waiting for the Apple Mobile Device Service service to connect.

3/15/2018 6:32:29 PM    System    Error    Service Control Manager    krafty.systemsbiology.net        7000    "The Apple Mobile Device Service service failed to start due to the following error:
The operation completed successfully. (0x0000041D)"

3/15/2018 6:32:31 PM    System    Error    Service Control Manager    krafty.systemsbiology.net        7009    A timeout was reached (30000 milliseconds) while waiting for the SepMasterService service to connect.

3/15/2018 6:32:31 PM    System    Error    Service Control Manager    krafty.systemsbiology.net        7000    "The SepMasterService service failed to start due to the following error:
The operation completed successfully. (0x0000041D)"

Yes, the service (Symantec Endpoint Protection/SepMasterService) isn’t starting randomly on boot or reboot. I haven’t tried to manually starting this service. I would highly suspect it would start because when Debug Logging option from SymDiag runs (which runs a command to stop and then start the service) this service starts normally.

Are you suspecting the Apple Mobile Device service not starting might be related? The affected computers do have this service/application installed. I could change the Apple Mobile Device service from automatic to automatic with Delayed Start if you think that would be a good troubleshooting step.

I am not sure it is related, but could be.

If you increase the timeout does it still happen randomly?  Maybe do this on one system and monitor?

1. Click Start, click Run, type regedit, and then click OK.
2. Locate and then click the following registry subkey:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control

3. In the right pane, locate the ServicesPipeTimeout entry.
   
   Note If the ServicesPipeTimeout entry does not exist, you must create it. To do this, follow these steps:

a.                  On the Edit menu, point to New, and then click DWORD Value.

b.                  Type ServicesPipeTimeout, and then press ENTER.

4. Right-click ServicesPipeTimeout, and then click Modify.
5. Click Decimal, type 60000, and then click OK.
   
   This value represents the time in milliseconds before a service times out.
6. Restart the computer.

Ideally we would want to get some boot logging of the issue happening.

Thanks John. I’ll make the registry change on a couple systems.

I don’t have any experience with boot logging. Do you have an article you can point me to enable?

Hey Scott,

I don't do this normally, but I am going to go ahead and take ownership of your case and work with you there.  Please add the registry change to a couple of systems and I will email you from the case once I have it in my name.

John

Thanks John. I appreciate it.

Logging needed for anyone else having this issue:

Here is the logging I would like to have if possible during an issue reproduction.  You may have to enable this a few times until you notice the service is stopped after a reboot.

1. Open the SymDiag tool, and then accept the EULA.
2. In the Symantec Diagnostics Tool window, click Collect Data for Support.
3. In the Select Products dialog, check the Endpoint Protection Client box, and then click Next.
4. In the Select Data Type dialog, ensure that All data is selected, and then, under Debug Logging, set the timer to 0 and check the Endpoint Protection Client box, and click Advanced.
5. In the resulting dialog box, click WPP reboot only, ensure that the Trace level is Verbose, and then click OK.
6. In the main window, click Next.
   You will be prompted to reboot the system.
7. Click Enable and Reboot, and then allow the system to reboot.

Once restarted the Symdiag tool will come back up asking you to collect the data.  Verify the SEPMasterService did not start and then collect the logging.  If it did start please continue these steps until the issue is reproduced.