Video Screencast Help
Symantec to Separate Into Two Focused, Industry-Leading Technology Companies. Learn more.

What did SEPMs push out?

Created: 20 Sep 2013 • Updated: 16 Jan 2014 | 12 comments
This issue has been solved. See solution.

Guys,

I need to find out the best way to determine what was pushed out from the SEPMs to clients on

9/19/13 2:00:00 PM CDT - 9/19/13 3:00:00 PM CDT. Networking is telling me that there was a spike in traffic coming from the SEPMs. I need report back on what it was. Can anyone point me in the right direction? Running SEPMs 12.1.2015 on server 2008.

 

Thanks in advance.

 

Operating Systems:

Comments 12 CommentsJump to latest comment

GP Rob V's picture

Did you retain logs for the day?  I would go to Monitor, the logs tab, and take a look at the server and client-server activity logs to see if anything shows up during that time frame.

Will V's picture

Fine tune the settings in Time Range to narrow down your search to the date and time desired.

Capture.JPG

 

 

Please mark posts as the solution if they solve your problem!

.Brian's picture

What port was the spike in traffic? 8014?

Please click the "Mark as solution" link at bottom left on the post that best answers your question. This will benefit admins looking for a solution to the same problem.

Seyad's picture

you may also check the reporting.log. If you still have the entries for the particular date then you would be able to see what was downloaded from server to clients.

%Program Files%\Symantec\Symantec Endpoint Protection Manager\apache\logs\reporting.log

Do let us know, if you have any questions.

Beppe's picture

Hello,

I doubt you will be able to identify the reason of that spike by only looking at the logs in the SEPM console, you may see there were content updates, client upgrades or something else not the size of the transferred files. More details about what has been transferred between clients and manager, including the size of the files, would be in the Apache access logs which are disabled by default.

Common reasons are:

+ SEPM => clients:
- SEP client package upgrade (did you schedule any client upgrade?)
- definitions (they are bigger when clients are offline for longer time)
- policies (changing a policy for large amount of clients may cause a spike)

+ clients => SEPM:
higher client logging (firewall, applications, outbreak, etc.)

 

Regards,

Giuseppe

RichBagurdes's picture

We had a very similar problem happen a few times.   Our network team was monitoring bandwith and found that at clients were counsuming 10-100x the normal bandwith during random times.  SEPM logs, and client logs were useless.  The best tools we had to monitor and track down workstations were bandwith reports from the network team, wireshark packet captures on known bad clients, and/or the SEPM server, and Symantec Critical System protection policies applied to the SEPM's to track clients using up more then one connection.

 

After diligent work we discovered that some clients were stuck in a loop downloading definitions.  The problem stemmed from a resource problem on the client machine.  Something on the client was consuming all of the available paged pool memory.  On the client when this resource was exausted the SEP client tended to get into a loop.  When observing a "bad" client in wireshark one would see the client connect to the SEPM, then start to download definitions.  During a certain point in the download the SEP client would reset the connection and try over again.  On the SEPM if one were to check the number of half open connections on the SEPM one would see hundreds, from the same client.

 

So after working with Symantec an issue was found with the way the client did some error handling and this was resolved in 12.1.3.  See http://www.symantec.com/docs/TECH206828 See fixid 3160786.

 

Untill 12.1.3 clients could be distibuted we found that by agressive use of the GUP's could at least keep the bandwith consumption local to the site and didnt affect our WAN anymore.

 

Let me know if you have any other questions.  I hope 12.1.3. resolves your problem. 

Rigo's picture

RichBagurdes,

Where on the SEPM would you be able to ..."check the number of half open connections on the SEPM...?"

GP Rob V's picture

Where on the SEPM would you be able to ..."check the number of half open connections on the SEPM...?"

I'm not sure I understand your question.  If you're using a pull communication to update clients, the SEPM would see "half open" connections.

Are you using any GUP's, or is all communication right from the SEPM to the clients?
 

.Brian's picture

Another thing you can do is use wireshark on the SEPM to detect full.zip download to the clients. This could be indicative of corrupt defs on the client if they continuously ask to download over and over even though they show as up to date. See article here:

https://www-secure.symantec.com/connect/articles/u...

Please click the "Mark as solution" link at bottom left on the post that best answers your question. This will benefit admins looking for a solution to the same problem.

RichBagurdes's picture

Before our bandwith problems all our clients were set to pull updates from the SEPM.  The command we used to search for half open connections on the SEPM was:

netstat -anbo > netstat.log

RichBagurdes's picture

Also - once we observed the half open connections via the netstat command we pushed a CSP policy out to the SEPM's that logged all connections to the SEPM port.  Then I created a query to create a list of "problem" clients.    When we were deep into the problem we found that some clients had thousands of connections to the SEPM in the three hour period we were reporting on.

If you plan to use this query - update the Policy with the name of the policy you decided to use. In my example I removed the policy name so  below it says POLICY_NAME = 'redacted'

 

Select
 Sum (event_cnt)as "Events",
 max(value1)                       as "Process Set",
 max(Process_name)                 as Process,
 dbo.FmtEventType(Event_Category,'','')   as "Event Type",
 max(CASE event_type WHEN 'PNET' THEN CASE OPERATION
 WHEN 'Accept'  THEN CASE DISPOSITION WHEN 'D' THEN 'Inbound ' + target_info + ' Connect from ' ELSE 'Inbound ' + target_info + ' Connect from ' END + VALUE5 + ':' + VALUE6 + ' to local IP ' + VALUE4 + ':' + VALUE3 + REPLACE(' ('+ VALUE10 + ')','()','')
 WHEN 'Connect' THEN CASE DISPOSITION WHEN 'D' THEN 'Outbound ' + target_info + ' Connect to ' ELSE 'Outbound ' + target_info + 'Connect to '   END + VALUE5 + ':' + VALUE6 + REPLACE(' ('+ VALUE10 + ')','()','') + ' from local IP ' + VALUE4 + ':' + VALUE3
 ELSE 'Unknown network operation (' + COALESCE(OPERATION,'') + ')' END
 ELSE  dbo.getpathleaf(target_info) END) as "Resource"
 From CSPEVENT_VW WITH (NOLOCK) WHERE EVENT_TYPE LIKE 'P%' AND event_dt > dateadd(hh,-3, getutcdate()) AND POLICY_NAME = 'redacted'  AND Process_name = 'Smc.exe'
 GROUP BY
 SUBSTRING(PROCESS_NAME,1,50),SUBSTRING (dbo.getpathleaf(target_info),1,75),EVENT_CATEGORY,SUBSTRING(VALUE1,1,25),
 OPERATION, VALUE2, DISPOSITION,EVENT_SEVERITY,
 SUBSTRING (HOSTNAME,1,15),HOSTADDR,VALUE5,OSTYPE
 Order by 1 DESC

Beppe's picture

Would not be easier to enable the Apache Access log to trace all clients' connections?
Once you get it, you may import it in Excel and apply filters and counters as you wish.

Regards,

Giuseppe

SOLUTION