Endpoint Protection

Hello Symantec Community,

I manage an environment of 750 SEP clients around the world pointing back to the one SEPM. Have virtually no problems with client updates, one here and there but are easily resolvable by doing the usual things - except for this one stubborn client.

Now i could probably just update from LiveUpdate initially and it would resolve the issue, but i'd like to understand specifically why this client is posing a problem.

Troubleshooting steps undertaken:

• Client has been uninstalled, restart, installed, restarted again
• Access to the SEPM on port 8014 is open
• Machine has been restarted
• SEP client is a GUP
• SEPM can see machine
• I've enabled and run a Sylink log overnight to give it ample time to update but still has not done so.
• Run through this how to clear corrupted definitions tech note. - http://www.symantec.com/business/support/index?page=content&id=TECH103176

I would very much appreciate if someone with a deeper understand of the function of SEP please review the attached logs and advise why this client is not updating?

Many thanks

Keir Doubas

Attachment(s)

symlog110314.zip   109 KB 1 version

how many clients have such issue? all of them? is your db on sql or embedded?

I checked the logs and it seems like the clients are hung at downloading and it times out after the try

I think you have two sepms with same priority..

SEPM DTC-STJ-SEP-01:8014

SEPM2  SEPMDTC-STJ-SEP-01

client does not know which client to take it from....

Simillar to this one

https://www-secure.symantec.com/connect/forums/gup-group-update-priver-running-crazy-has-anyone-seen-something

I'd be curious to find out if this machine (which is using itself as a GUP) has any content in the below location:

C:\Program Files\Symantec\Symantec Endpoint Protection\SharedUpdates

If so, then it would suggest the GUP component is correctly downloading content from the SEPM via 8014, and that the issue might actually be with it communicating with itself over 2967 (which I suspect is the problem)

On a vaguely related note, the logs also indicate that the client is lokoing for defs that the SEPM does not hold:

http://www.symantec.com/docs/TECH148419

None of the identified ones (below) are for the VirusDefs, so it's not directly causing the problem.  Just worth noting if other bits are failing to update:

"03/11 10:08:17 [1136] <mfn_LiveUpdate> EVENT_LU_REQUIRE_STATUS returned ERROR_SYSTEM_UNKNOWN - Ignore LU content. Moniker: {D3769926-05B7-4ad1-9DCF-23051EEE78E3} Seq:140310001

03/11 10:08:17 [1136] <mfn_LiveUpdate> EVENT_LU_REQUIRE_STATUS returned ERROR_SYSTEM_UNKNOWN - Ignore LU content. Moniker: {C25CEA47-63E5-447b-8D95-C79CAE13FF79} Seq:80929016

Oh yeah, to aid in your troubleshooting the GUP process, it's also recommended that you enable debug logging in addition to sylink logging:

http://www.symantec.com/docs/TECH104539

The debug logs will show if the GUP component is seeing this connection request from the SEP Client (on the same box), and how it handles that connection (e.g. if it's refusing the connection).

BTW Rafeeq, that's the same SEPM name, it's just got "SEPM" appended to the front is all.  It's a missed space in the log generation, not a different SEPM.

hello SMLatCST and thank you very much for your knowledge on this one!

There is indeed a load of content in the C:\Program Files (x86)\Symantec\Symantec Endpoint Protection\SharedUpdates directory indicating its downloading the defs but not updating itself.

I've confirmed port 2967 is open on the client via netstat and can access it from another machine.

  TCP    0.0.0.0:2967           0.0.0.0:0              LISTENING       3596

debug.log has been enabled and policy pulled from the SEPM. it's currently downloading a bunch of defs down over a slow link so ill send the debug.log through to you tomorrow once it's finished.

cheers

Keir

You can open the content.info file which is in <install drive>:\Program Files (x86)\Symantec\Symantec Endpoint Protection Manager\Inetpub\content folder

if a client has only AV/AS, it would say ignore moniker for the rest of the contents ( for ex, ntp, sonar, etc)

for me its timeout issue may be you need to check your proxy settings.

debug.log attached from affected GUP.  Any advice is highly appreciated as to why this client will not update.

Attachment(s)

debug_2.zip   446 KB 1 version

does this client use proxy ?

A proxy server is configured in internet options yes.

Took the backup of the registry
Delete the following registry keys:
 HKEY_USER\.DEFAULT\SOFTWARE\Microsoft\Windows\CurrentVersion\Internet Settings\Connections\DefaultConnectionSettings
 HKEY_USER\.DEFAULT\SOFTWARE\Microsoft\Windows\CurrentVersion\Internet Settings\Connections\SavedLegacySettings
Set the proxy enable in the registry to 0 under HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Internet Settings
Uncheck the proxy setting option in the internet options settings.
Reboot the system.

==================

Because of this log file

03/11 21:00:37 [3596:5212] Using proxy type 1

Bit of a long post here so please bear with me  (btw if you're interested, you can verify each of my findings by comparing with the GUP Troubleshooting article I posted earlier):

We can see teverthing appears to be hunky dory for the most part.  If you search for the below string, you can see the various log events for the GUP accepting connections for lots of various clients, and subsequently providing defs to them:

GUProxy: accepted socket

The one we're focussing on is the one below and the log events that follow it (this event shows the GUP reciving a connection from itself):

03/11 17:38:02 [3596:10508] GUProxy: accepted socket 3236 for 146.77.55.160 port 50092

Following this event, we see the below logs which state the file requested by the client, the fact that it's not present in the GUP's cache, and the GUP adding the file to its list of things to grab:

As you can tell, it appears this client (itself) has requested the full defs (called Full.zip).

The next log entry shows the download starting and how far it gets:

These entries essentially combine to say that the GUP managed to download bytes 0 to 8000 of 353470041 bytes (about 337MB).

The thing is, at no point do the logs ever show it completing the download.  The last logs for this particular file (Full.zip of definition revision 140310033) in the debug log show it got fairly far, but had a further 82MB to go when it errored out as below:

All in all, it looks like the SEP client is asking for the full defs, the GUP tries to download the full defs from the SEPM and fails, and so is never able to provide the defs to the SEP Client.

The connection from the SEP Client to the GUP (itself) appears fine, as the GUP seccessfully receives the request.  It's unclear why the download errors though.  Are there bandwidth issues between this GUP and the SEPM?

My recommendation is to update this particluar machine using the Intelligent Updater (grab it from the below link), so that it will update using deltas going forward, which appear to work going by the logs of the other connecting IP addresses.

http://www.symantec.com/security_response/definitions/download/detail.jsp?gid=savce

SMLatCST, thank you VERY MUCH for your analysis of the logs and extensive reply to my issue.

Turns out I logged on this morning to the client and it had updated it's defs. Upon reading your reply I had a major Aha! moment and have realised what the issue was.

We have 2meg MPLS links between our remote sites and the SEPM, so in my wisdom in a drive to conserve bandwidth over the link, i set a bandwidth restriction of 64Kbps. With it needing to download 300+ meg of defintions it must have just timed out due to the restriction. Leaving it over the weekend and the definition has finally downloaded. So i've lifted the bandwidth restriction back to 'unlimited' now, so the GUP can get the full.zip or delta definition as fast as possible to send to the rest of the clients.

Once again thank you for your insight into my issue, I will take this forward with me in my day to day SEPM duties.

Kind regards

Keir Doubas

Good Morning Keir Doubas,

You're very welcome, and thanks for updating this thread 

I'm glad to be of help.  As I see it, the more of us with this info the better for the community.

It might be worth noting as well, now that the client has updated it's defs, it's probably (hopefully) requesting the much smaller delta definitions and not the full fat def file any more, so you might be alright to place some bandwidth throttling back in if needed.  You can always verify what kind of defs the client is requesting by looking in the sylink logs.

If you're happy that's it's all resolved though, then don't forget to turn off the logging as it can take up a fair amount of disk space over time.