Endpoint Protection

 View Only
Expand all | Collapse all

Client / Server Communications - can you help verify what's happening?

  • 1.  Client / Server Communications - can you help verify what's happening?

    Posted Jul 25, 2014 07:33 AM

    Hi dear members of the forum

    We've encountered an issue which I'm currently working with tech support on ;  since upgrading our SEPM to 12.4 RU4 MP1 we've had massive spikes on our WAN data links;  some remote sites are running at 100% network utilisation where as they usually operate at 20-40%.   If i shut down the apacahe web server service on the SEPM, these network spikes go down and the site runs normally.   Its so bad the remote sites are unable to operate as their WAN link is completely saturated. 

    While working on this problem, I have a question about how clients get updates when you use GUP's:

    My understanding is that when you have a GUP configured at a remote site, the client will perform heartbeat communications with the SEPM but get content and policy from the GUP. Is this correct ?

    Take this scenario:

    SEPM: location UK

    Client with GUP : location Czech Republic    (GUP server name: SECZESSSLUT001)

    Here is a sample of the client log:    (SEPM server name : SEUKESSSCRA001)

    Remote file path: http://SECZESSSLUT001:2967/content/{810D5A61-809F-49c2-BD75-177F0647D2BA}/140711017/xdelta140710008.dax SYLINK 
    00000106 01cf9f293fcb1e9d 1207030c 00000000 00000002 00000000 Downloaded new content update from Group Update Provider failed.

    ile path: http://SECZESSSLUT001:2967/content/{810D5A61-809F-49c2-BD75-177F0647D2BA}/140711017/xdelta140710008.dax SYLINK 
    00000106 01cf9f2a590ff465 1207030c 00000000 00000002 00000000 Downloaded new content update from Group Update Provider failed.

    Remote file path: http://SECZESSSLUT001:2967/content/{810D5A61-809F-49c2-BD75-177F0647D2BA}/140711017/xdelta140710008.dax SYLINK 
    0000008c 01cf9f2ab74813c0 12130002 00000002 00000001 00000000 [File reputation submission] Submitting information to Symantec failed. REP 
    00000106 01cf9f2b72893eb9 1207030c 00000000 00000002 00000000 Downloaded new content update from Group Update Provider failed.

    Remote file path: http://SECZESSSLUT001:2967/content/{810D5A61-809F-49c2-BD75-177F0647D2BA}/140711017/xdelta140710008.dax SYLINK 
    00000081 01cf9f2d6fa56a5c 120b0007 00000000 00000000 00000000 Failed to connect to all GUPs, now trying to connect SEPM SYLINK 
    000000cb 01cf9f2d701038be 12070800 00000002 00000000 00000000 An update for {810D5A61-809F-49c2-BD75-177F0647D2BA} was successfully installed.  The new sequence number is 140711017. LiveUpdate Manager 
    0000010c 01cf9f2d701038be 1207030c 00000000 00000000 00000000 Downloaded new content update from the management server successfully.

    Remote file path: http://SEUKESSSCRA001:8014/content/{810D5A61-809F-49c2-BD75-177F0647D2BA}/140711017/xdelta140710008.dax SYLINK 
    000000cb 01cf9f2d89a1ea34 12070800 00000002 00000000 00000000 An update for {07B590B3-9282-482f-BBAA-6D515D385869} was successfully installed.  The new sequence number is 140711018. LiveUpdate Manager 
    0000010c 01cf9f2d89a1ea34 1207030c 00000000 00000000 00000000 Downloaded new content update from the management server successfully.

    Remote file path: http://SEUKESSSCRA001:8014/content/{07B590B3-9282-482f-BBAA-6D515D385869}/140711018/xdelta140710024.dax SYLINK 
    000000cb 01cf9f2d8c5fc8b8 12070800 00000002 00000000 00000000 An update for {55DE35DC-862A-44c9-8A2B-3EF451665D0A} was successfully installed.  The new sequence number is 140711011. LiveUpdate Manager 
    0000010c 01cf9f2d8c5fc8b8 1207030c 00000000 00000000 00000000 Downloaded new content update from the management server successfully.

    You can see the client is communicating with the GUP (SECZESSSLUT001) but the download failed for whatever reason.

    Can someone tell me why the client talks to SEUKESSSCRA001. Is this because it was unable to get the required updates from the GUP ?

    Thanks

     

     



  • 2.  RE: Client / Server Communications - can you help verify what's happening?
    Best Answer

    Posted Jul 25, 2014 07:55 AM

    My understanding is that when you have a GUP configured at a remote site, the client will perform heartbeat communications with the SEPM but get content and policy from the GUP. Is this correct ?

    only content policy will received from SEPM.as per your sep client failed to download content  from GUP.

    have you configure if sep client not update GUP after update SEPM ?

    What setting do you have done ?

    may be gup defination are corrupted please run symhelp tool and check  virus defination corrupted or not.

    Download the Symantec Help (SymHelp) diagnostic tool to detect Symantec product issues

    Article:TECH170752  | Created: 2011-09-29  | Updated: 2014-07-14  | Article URL http://www.symantec.com/docs/TECH170752

    have you enable (Maximum time that clients try to download updates from a Group Update Provider before trying the default management server) if yes please select never,if you have select never (Check Never if clients only get updates from the Group Update Provider and never from the server. For example, you might use this option if you do not want client traffic to run over a wide area connection to the server )

    2009092901593448_1.jpg



  • 3.  RE: Client / Server Communications - can you help verify what's happening?

    Posted Jul 25, 2014 08:22 AM

    Do your clients come back to the SEPM iof the GUPs are not avialable? Did you verify your GUPs are up and acting as GUPs?



  • 4.  RE: Client / Server Communications - can you help verify what's happening?

    Posted Jul 25, 2014 08:54 AM

    Currently the GUP settings are set so that "after 30 minutes" to try the SEP manager.

    Based upon your recommendation I will change this so that it is set to NEVER.

    Can you tell me something:

    When do clients fail over to the Manager?

    the setting says "after 30 minutes"

    But what if the manager defs are 24 July 2014 r35 but the GUP has 24 July 2014 r23. 
    The client performs a heartbeat to the manager and knows there is new content, so talks to the GUP. However the GUP defs are older than the manager.  What happens here?  will the client pull "old" defs from the GUP or will it go to the manager?

    Thanks !



  • 5.  RE: Client / Server Communications - can you help verify what's happening?

    Posted Jul 25, 2014 09:09 AM

    They will try the GUP for 30 minutes and after that will go to the SEPM on next heartbeat.

    They will not "rollback" to older defs, they would ignore the old ones



  • 6.  RE: Client / Server Communications - can you help verify what's happening?

    Broadcom Employee
    Posted Jul 25, 2014 10:11 AM

    Hi DannyUK,

    Could you share the support case number?

     



  • 7.  RE: Client / Server Communications - can you help verify what's happening?

    Posted Jul 25, 2014 10:30 AM

    As you say "manager defs are 24 July 2014 r35 but the GUP has 24 July 2014 r23. 
    The client performs a heartbeat to the manager and knows there is new content, so talks to the GUP. However the GUP defs are older than the manager.  What happens here?  will the client pull "old" defs from the GUP or will it go to the manager?"

     

    If client reach GUP server then it will not download the old it will request for the latest defition and GUP will request the manager for the same.



  • 8.  RE: Client / Server Communications - can you help verify what's happening?

    Posted Jul 25, 2014 10:36 AM

    Hi,

     

    I have few question What version you have install on SEPM and in GUP and the client?

    Do you have available disk space in the GUP server?

    All clients failed or any perticular list of clients failed to get defintion from GUP server?

    Are they in same subnet where the GUP configure?

    What type of GUP?



  • 9.  RE: Client / Server Communications - can you help verify what's happening?

    Posted Jul 25, 2014 12:19 PM
    The version of SEPM is current 12.1.RU4 MP1a and the GUP has client 12.1.4100. Disk space is fine 200gb+ I don't know what clients are talking to GUP and what clients are bypassing GUP and going to manager. All I know is that when the SEPM is running the site WAN link is saturated and maxed at 100% utilisation and I can see active tcp connection on my manager talking to Clients at that remote site. So I assume those clients are getting content from SEPM instead of GUP. I need to know why. They are not in same subnet. What type of GUP? What do you mean? It's a windows 2008 r2 x64 server.


  • 10.  RE: Client / Server Communications - can you help verify what's happening?

    Posted Jul 25, 2014 12:34 PM

    With no indication that it's been fixed, it could just be that SEP Services on the GUP machine are crashing (as per the below article) and causing clients to go to the SEPM.

    http://www.symantec.com/docs/TECH213461

    If the clients are going direct to the SEPM, then this would explain the increased network utilisation.  Might want to try reverting to 12.1RU3 on the GUPs to see if this resolves the issue.  This rollback should only need to be performed on the GUP itself.



  • 11.  RE: Client / Server Communications - can you help verify what's happening?

    Posted Jul 26, 2014 10:31 AM

    Thanks for the reply - is there an easy way to verify if the GUPs are working as they should be?

     

     



  • 12.  RE: Client / Server Communications - can you help verify what's happening?
    Best Answer

    Posted Jul 26, 2014 10:58 AM

    Try the GUP content distribution monitor found here:

    https://www-secure.symantec.com/connect/downloads/new-sep-content-distribution-monitor-gup-health-checking



  • 13.  RE: Client / Server Communications - can you help verify what's happening?

    Posted Jul 26, 2014 11:10 AM

    How to analyze Debug logs from GUP to determine which clients are taking definitions from GUP

    https://www-secure.symantec.com/connect/articles/how-analyze-debug-logs-gup-determine-which-clients-are-taking-definitions-gup

    How to confirm if SEP Clients are receiving LiveUpdate content from Group Update Providers (GUPs)

    http://www.symantec.com/docs/TECH97190

     

     



  • 14.  RE: Client / Server Communications - can you help verify what's happening?

    Posted Jul 26, 2014 02:12 PM
    As in log do search with you GUP IP address Or with port no 2967 you can find your result.


  • 15.  RE: Client / Server Communications - can you help verify what's happening?

    Broadcom Employee
    Posted Jul 26, 2014 03:12 PM
    Collect Slink logs from 2-3 machines by restarting Sep client service. Use smc -stop and smc -start to restart service. Logs can reveal the source for content update. It seems GUP is culprit in this case.


  • 16.  RE: Client / Server Communications - can you help verify what's happening?

    Broadcom Employee
    Posted Jul 26, 2014 03:12 PM
    Collect Slink logs from 2-3 machines by restarting Sep client service. Use smc -stop and smc -start to restart service. Logs can reveal the source for content update. It seems GUP are culprit in this case.


  • 17.  RE: Client / Server Communications - can you help verify what's happening?

    Posted Jul 28, 2014 03:38 AM

    It kinda depends on which way you want to investigate it tbh.

    If you're looking at the GUPs themselves, then the article suggests the smc.exe process will actually crash, meaning you should see abnormal stops of the Symantec Management Client service on the GUP machines.  This way should show you if you are affected by the issue I linked.

    Another way of checking what's going on is using the GUP monitor linked by _Brian.  This works off of the SEPM's logs and can also be gathered/viewed from the System:Client-Activity Logs.  You can see events where clients are saying they are unable to talk to a particular GUP (by IP address).  This method focusses on identifying if/when Clients aren't using the GUP when they should be, rather than definitively saying if you're affected by the issue.

    I'd go with the first option if I were you, as it should be fairly easy to spot.  As I mentioned, if you are affected, then reverting back to 12.1RU3 (on the GUPs only) might be the way forward until the issue gets resolved.



  • 18.  RE: Client / Server Communications - can you help verify what's happening?

    Posted Jul 28, 2014 03:53 AM

    OK thanks gents for your replies; I'll continue troubleshooting this morning and update the post later today.

    I can already see lots of clients getting updates from my manager this morning even though they have a local GUP configured at their local site, so something is not right.

     



  • 19.  RE: Client / Server Communications - can you help verify what's happening?

    Posted Jul 29, 2014 04:14 AM
      |   view attached

    Just wanted to update the post;

    After making several changes to our Symantec Endpoint manager to optimise the performance, I am still seeing some clients ignore local GUPs (which i am still trying to troubleshoot using the info from posts above).

    The original post was about higher than usual network utilisation and I will not be able to confirm until later this afternoon when our USA sites come online if these changes have made a positive impact, so will update the thread later tdoay.

    In the mean while here is an example of network utilisation on our SEPM, showing clients pulling data via the apache web service:  The clients highlighted are at a different location to the manager and have local GUPs setup, however they look to be pulling data from my manager. This behaviour continues all morning while various clients start up and are requesting / getting updates ( i assume ).  So something still looks to be incorrect with the GUPs.

    I've checked the smc.exe process and cannot find any instances of this crashing/ stopping as described in the post above.  Therefore i need to try and catch a client and get sylink logging setup before it finishes its update - easier said than done but i will persevere.



  • 20.  RE: Client / Server Communications - can you help verify what's happening?

    Posted Jul 29, 2014 05:25 AM

    Thanks for the update.

    I'd be careful with the screenshot you posted however.  All clients must communicate with the SEPM directly for normal heartbeats anyway, the GUPs only proxy the definition transfer side of things.  So you will always see clients connecting directly for policy updates, log uploads, etc.

    Therefore, I'd recommend checking out the GUP Monitor Brian mentioned earlier, and using it to identify clients that are not using GUPs for def downloads, and focussing on these alone.

    If it turns out that the GUPs are being correctly used for def downloads, then you may want to consider extending the hearbeat interval, and/or escalate to the network guys to see if they can pin down any specific IP addresses consuming the most bandwidth.



  • 21.  RE: Client / Server Communications - can you help verify what's happening?

    Posted Jul 29, 2014 06:08 AM

    Thanks for the reply

    Understand that all client communicate with SEPM for heartbeats, however I would have expected this process to be very small in terms of data transferred (and in most cases the bytes sent is high - suggesting data from manager to client).  But point taken, I will use Brians GUP monitor to see if i can get some info.

    Can you tell me something.  I understand Symantec typically release 3 definition revisions per day ; can you tell me what the GUPs will be downloading from the manager during each of these daily updates?

    i/e Do GUPs pull a full zip ~ 400mb three times a day, or do they only pull delta definitions?

    The reason i ask is because I do see a lot of communications between manager and GUP's via the httpd.exe service (like my screen shot above) and I assume these are definition updates throughout the day.

    Thanks

     



  • 22.  RE: Client / Server Communications - can you help verify what's happening?

    Posted Jul 29, 2014 06:19 AM

    The GUPs only ever proxy and cache the content that the clients ask them for.

    Therefore, in an ideal world (where the clients are always up to date), the GUPs will only ever download a single delta file.  In reality however, this means the GUPs usually hold a variety of deltas, plus a full def if there's a client out there that needs it.

    If there's a real problem with the GUP <-> SEPM link load, then don't forget that you can limit this link with GUP throttling options in the LU Policy.



  • 23.  RE: Client / Server Communications - can you help verify what's happening?

    Posted Jul 29, 2014 07:49 AM
    Yes it will download delta only as long out of date client request GUP for full zip. Note if client out of date for 10 days and if u have setup revisions for 5 in sepm then client will request gup for full zip


  • 24.  RE: Client / Server Communications - can you help verify what's happening?

    Posted Jul 29, 2014 08:02 AM

    Just to come at it from another direction, the worst case scenario is also very rare.

    The only way a GUP would download all three full defs in a day is if the below happens:

    1. SEPM Updates and grabs first release of the day
    2. SEPClient-1 is switched on, and has defs older than the SEPM can provide deltas for
    3. SEPClient-1 asks GUP for full defs
    4. SEPClient-1 updates
    5. SEPM Updates and grabs second release of the day
    6. SEPClient-2 is switched on, and has defs older than the SEPM can provide deltas for
    7. SEPClient-2 asks GUP for full defs
    8. SEPClient-2 updates
    9. SEPM Updates and grabs last release of the day
    10. SEPClient-3 is switched on, and has defs older than the SEPM can provide deltas for
    11. SEPClient-3 asks GUP for full defs
    12. SEPClient-3 updates

    Essentially in order for a GUP to request the full defs for all three revisions released in a day, you'd have to have a out-of-date client check-in after the SEPM has downloaded each release.  Not to metion it's also dependent upon the combination of your SEPM's LU schedule and Symantec's variable release schedule.

    As I said, it's quite rare but can happen...

    Given then you'd usually see the clients being switched on at the same time though, any that need the full defs would usually nab the first released revision, then update using the deltas from there on out.



  • 25.  RE: Client / Server Communications - can you help verify what's happening?

    Posted Jul 29, 2014 08:19 AM

    OK guys thanks for the info. We've increased the content revisions on the SEPM from 15 to 30 (this was done end of last week) so i am hoping this will reduce the number of clients requesting full zip.

    Also I have since found a lot of clients pulling updates from liveupdate.symantec.com (at locations where network utilisation is high) - and have therefore unticked the liveupdate scheduling check box within their liveupdate policy.  It was set to every 4 hours.

    Why clients at a remote site would hit liveupdate instead of updating from their GUP is also another anomaly but this change should stop this from happening in the future.

    Am going to monitor netowrk utilisation and will update the thread.  Thanks again.



  • 26.  RE: Client / Server Communications - can you help verify what's happening?
    Best Answer

    Posted Jul 29, 2014 08:40 AM

    Sounds like a plan.

    Just a quick note regarding the LU Schedule.  I'd recommend reading up on the "Options for Skipping LiveUpdate" if you have a SEP12.1 envrionment.  If these skipping options are disabled (or you're working with SEP11), then the clients will always run a LU session on their schedule, regardless of whether or not they are up to date or have other working update mechanisms.

    As the schedules will always run, it's then pot luck as to if Symantec released a new def between the client's last heartbeat and it running a LU.  Don't forget they are loking at different sources here, so one (Symantec) might have newer defs than the other (SEPM/GUP), in which case an update will occur.



  • 27.  RE: Client / Server Communications - can you help verify what's happening?

    Posted Jul 29, 2014 10:55 AM

    I just double checked our liveupdate policy settings, and it was set to "LiveUpdate runs only if the client is disconnected from Symantec EndPoint protection for more than 1 hour".

    This would suggest clients should NOT have been running liveupdate because they were all connected to the manager (green dot visible).  So again, begs the question why were clients running liveupdate (other than because users run it manually, and i dont think this is the case).

     



  • 28.  RE: Client / Server Communications - can you help verify what's happening?

    Posted Aug 05, 2014 04:03 AM

    Hi Guys

    After monitoring for almost a week now, things look somewhat better so I think a combination of "tweaks" to our environment have improved performance.   For purposes of the forum, here are the changes I've made:

    • SEPM upgraded to latest version 12.1 RU4 MP1a  (not directly related but good practice)

    • Client heartbeat interval changed from 45 minutes to 2 hours

    • SEPM set to retain 30 content revision (increase from 15)

    • All GUPs upgraded to latest version of endpoint client

    • GUP cache sizes increased from 500mb to 2gb

    • Bandwidth throttling set on small WAN links (~2mb) GUP to 512kb

    • Sites with small WAN links set never to fail over to SEPM (i/e only get updates from local server)

    • Liveupdate scheduling disabled (to prevent clients downloading updates from internet)

     

    One side effect of increasing our content revisions is that the SEM_5 content database has grown to 30GB and hit a limit.  i've had to increase this to 40gb for the time being.

    Does anyone have any thoughts on this database size. For an environment of 2000 client, is a 30GB + database expected?

    Thanks

     

     



  • 29.  RE: Client / Server Communications - can you help verify what's happening?

    Posted Aug 05, 2014 04:23 AM

    It's perfectly normal for the DB to hit that sort of size when retaining 30 def revisions.  Don't forget, the vdefs alone when zipped are about ~400MB now, and you're keeping 30 copies of both the 32bit and 64bit versions, plus all the others defs required.

    Unfortunately, without knowing your requirements (log retention, client packages, traffic/packet log uploads, etc), we can't really say how much you should expect.



  • 30.  RE: Client / Server Communications - can you help verify what's happening?

    Posted Aug 05, 2014 06:35 AM

    OK if you think its perfectly normal for a DB to reach around 30gb in size then thats fine - understand you dont know all the additional settings regarding log retention etc.   Do client packages make a difference also?  I could probably remove a few old pakcages from the manager.

    I'll mark the useful posts as the solutions - thanks to all for your help.



  • 31.  RE: Client / Server Communications - can you help verify what's happening?

    Posted Aug 05, 2014 06:58 AM

    Yup, client installer packages are also stored in the DB, and therefore contribute to the size.

    If you no longer need the older ones, then removing them will help you recover some space from the DB.

    Note however, that you won't see the DB files decrease in size themselves, but the amount of free space inside the DB will increase (as seen from the SQL Management Studio).



  • 32.  RE: Client / Server Communications - can you help verify what's happening?

    Posted Aug 05, 2014 10:22 AM

    OK gotcha, thanks for your help with this!  very impressive wealth of knowledge on the forums.



  • 33.  RE: Client / Server Communications - can you help verify what's happening?

    Posted Aug 05, 2014 12:05 PM

    Things I know of GUPs and SEPM communication.

    FYI...40k+ clients with split GUP/LiveUpdate servers, 5 SEPM, SQL installed.

    Our SQL DB is 21GB with 21 days of defs and about 6 installation packages. Make sure application learning is turned off globally and on all groups. 30GB sounds about right for how many defs you have.

    GUP's need time to relay and pull defs down. If you have many clients attaching to one GUP then you can have multiple copies of files being overwritten. Large updates like full.zips can be from corrupt defs on clients and at 400Mb a piece they will clog the GUP and force clients to go to the SEPM.

    What we have seen is when the GUP becomes overloaded the clients go to the SEPM and creates a cascade affect. Clients are already connecting to the SEPM for policy and log file sending so adding to this with updates...delta or full creates more traffic clogging links. We have seen 1GB switch port saturated and bring down a site.

    Increasing space on the GUP and more GUP's helps drastically. Setting the GUP policy to never go to the SEPM will help but then you have no failover. GUP's are supposed to be able to support 10k clients at a time but we are limiting it to about 1k per GUP.

    Make sure all clients are on 12 RU4 MP1. RU3 and RU4 have issues with network spamming.

    We increased GUP to 2GB at small sites and 5GB at large sites. Each has 10 revisions and 4 hours before attempting SEPM. All clients have at least 2 GUP's to choose from. Some have 5...at large sites. In addition LiveUpdate is enabled for manual updates and as a tertiary update location.

    All our GUP servers use Windows QoS for port 2967 with 20 to 40% utilization of the link as necessary.

    I would check the apache logs for how many connections your SEPM is getting. We have 30 minutes for heartbeat at some sites and 1 hour at others depending on the links between sites and SEPM's. We had one SEPM that was constantly logging errors due to over utilization. Heartbeat increase and moving some clients to another SEPM as primary seemed to help.

    SEP is very chatty.