Data Loss Prevention

Hi,

I am looking for an explanations on the system reaction, including Monitor, Enforce and any other components, when a Network Monitor comes under heavy load? For example, the span sport send a large number or network packets, the CPU utilization on monitor goes up, etc?

Are we start seeing longer wait times, longer incidents queuing, wait times, or there is potential packets drop? Is there any related KB articles on the subject?

We are running ver. 10.5, preparing to migrate to 12.0

Thanks!

Alex

Alex,

There are a couple of things that will happen on the Network Monitor.

1. You will see a ton of dropped or currupted packets. This will tell you that either you have a "dirty" connection that might have duplicate or incomplete packets. The other is that there is too much traffic and the server cannot keep up.
2. You can begin to work with WireShark to analyize the traffic and see what the issue is.

As far as what will happen on the Enforce Server or the Detection servers when under load. The following will happen.

1. If you have a badly designed policy it will overload the servers for it will trigger too many violations. This will cause the detetion servers to QUEUE up the incidents on the local hard drive as it tries to send them to the Enforce server to be processed. Where it will queue up until it is put in the DB. So this is where the long wait times will come into play. it's a matter of being able to process the violations into the servers.

You can find more information on the KB site.

Article ID: 50068 - What is the maximum bandwidth a network monitor can process?

Article ID: 43758 - Primary Causes for Long Message Wait Times

Article ID: 42000 - Why Are Packets Discarded?

Hope this makes sense.

If this solves your questions please marked as solved.

Ronak

How'd we see/identfy dropped packest on Windows server? Any logs reflect that, or some nwtwork analyser (what to look for?)

Thanks!

Alex

Alex,

When you look at the Network Monitor or Under the Traffic Stats you will see the number of dropped packets or corrupted .

System > Traffic

Look at the Unprocessable Components and Discarded Packets.

If this number is very high for a single day then you might have some issues. There will always be a samll amount of these, but if it is in the hundreds or thousands in a SINGLE day you should start to investigate the issue. This is where you will need to use WireShark to analyze the packets.

Hope this makes sense.

If this solves your questions please marked as solved.

Ronak

OK, got it. Just to clarify, what would be an "acceptable" number of Discarded Packets per a single monitor, before start investigating? For example we have on a single monitor for "Today":

Data: 26.35 GB
Messages: 136,783
Incidents: 354
Encrypted Attachments: 498
Unprocessable Components: 692
Discarded Packets: 148

Does this number looks negligible? How we see the total number of processes packet to get the percentage of discarded? 

Thanks for your help and advice!

Alex

Alex,

I would start to diagnose this more..

That is more than I woud be comfortable with. Also keep in mind that this alldepends on the following:

1. What Traffic is on the Span Port (inbound and outbound) or is it configured porperly
2. Do you have any IP filters in place.
3. How Much traffic is on the pipe.

If you contact SYMC support there is a Traffic Analyzer application that can help you too. YOu need to give it a packet capture file from WIreShark. (attached)

Hope this makes sense.

If this solves your questions please marked as solved.

Ronak

Attachment(s)

AnalyzePackets.zip   362 KB 1 version

Alex,

Here is the path data follows on a Network Monitor from induction to database:

PacketCapture > /drop_pcap folder > FileReader > /incidents folder > Incident Writer > Enforce (Monitor Controller) > /incidents folder > Incident Persister > Oracle database

With that in mind you can fairly quickly check the common 'choke' points for data and know what's not performing as expected based on where you find queued data.

On the Detection Server

Packet Capture
This is responsible for ingesting raw packets and re-assembling them into TCP streams. Because it's only getting a copy of traffic from a SPAN/TAP port, it can't re-request any missed packets. If it misses a single packet the whole stream must be discarded. This can happen if the source traffic is missing information, if the SPAN/TAP are overloaded (and not able to send all traffic to DLP) or if there is too much traffic for PacketCapture. In the Enforce console, go to System > Traffic and check for Discarded Packets to see if there are streams that PacketCapture can't reassemble.

/drop_pcap folder
Successfully captured TCP streams are written as .vpcap files to the Network Monitor. By default this folder is located at /var/SymantecDLP/drop_pcap (C:\drop_pcap on Windows). It's OK to have a number of directories here at any given time, but if you have files queued here - especially if the number of files is growing - it indicates that either FileReader is down (check System > Overview > [server name]) or not able to keep up with the volume of traffic captured by PacketCapture.

FileReader
The scanning engine for the detection server. If it's not able to keep up with data captured by PacketCapture, you'll need to consider adding additional servers and dividing the traffic. If you want more information about how to do that, please let me know.

/incidents folder
If FileReader finds a message that violates a policy it writes an .idc file to /var/SymantecDLP/incidents (<DLP root>/Protect/incidents on Windows). From here it should be passed to Enforce. If an .idc file is unable to process repeatedly, it is renamed to .bad. Renaming a .bad file back to .idc will cause it to attempt to re-process (in the event of a processing blockage being cleared).

Incident Writer
This process takes files from the /incidents folder and passes them up to Enforce. There's rarely issues here unless there is no connection to Enforce.

On Enforce

MonitorController
Manages all connections to detection servers and is the process that recieves .idc files sent by IncidentWriter on the detection servers. All incidents are written to the /incidents/ folder on Enforce (same default locations as above).

IncidentPersister
Picks up .idc files from the /incidents/ folder and inserts them into the database. It is also responsible for executing Response Rules on Enforce (things like Send an Email and Send to Syslog). If .idc files are queuing on Enforce it could be due to Enforce not having a connection to the database (in which case try restarting Vontu Manager service and checking the Tomcat logs for database connection errors).

Oracle database
Beyond the Oracle server being down, the next most common reason this would be a bottleneck is if one of the tablespaces fills up. Check for errors in System > Overview > Enforce about tablespace being full and then add/extend datafiles to the tablespace listed in the error. The KB details this process well.

I hope that helps give you a better understanding of your DLP system as a whole and also a series of checkpoints to see how well it's handling load.

If you have any questions, please let me know. If this answers your question, please mark it as a solution!

Thanks!

To avoid this situation it is best to develop a strategy of what your looking for before deploying this passive monitoring device. I usually remove encrypted traffic SSH, SSL from the list of monitored channels and try to cut out all other channels that may not be useful for a given application. (eDonkey)

I would be curious to know how the protocols that are not selected in the Network Monitor get processed when they reach the Network Monitor. I would assume it has to process the packet up to atleast layer 4 to read the protocol and port. Since this is extra work I like working with a device like a Gigamon to filter traffic before it hits the Network Monitor thus reducing the amount of traffic it must process.

There is also the option of increasing resources and using a high end capture card like a Naptech or Endace to meet your traffic requirements.

Tim,

Thanks so much for the detailed answer! Apology, I was traveling, just got back to the office last week. We will be looking to streamline the process, this information is extremely useful, hope there would be a support article covering this topic in details.

Thanks!

Alex

Hi Ronak,

Thanks for the reply! Apology, I was traveling, just get back to the office last week.

This What Traffic is on the Span Port (inbound and outbound) or is it configured properly

[AT]. We have outbound Span

Do you have any IP filters in place.

[AT]. yes, we use IP filters. On the monitor I see some dropped packets we have IP filer for 5 MTAs: +,xx.xx.xxx.xxx/32,*;+,xx.xx.xxx.xxx/32,*;+,xx.xx.xxx.xxx/32,*;+,xx.xx.xxx.xxx/32,*;+,xx.xx.xxx.xxx/32,*;-,*,*

How Much traffic is on the pipe.

[AT]. This is 1GB span port. Are you asking for the line utilization? For the monitor in question there is 7 days stats:

Data: 175.73 GB
Messages: 859,549
Incidents: 4,130
Encrypted Attachments: 3,065
Unprocessable Components: 3,657
Discarded Packets: 12,776
 

Unfortunately, we cannot filter traffic on the load balancer device, it is just a simple traffic splitter, we have to filter for each MTAs IP. We are using IP filter in *;+,xx.xx.xxx.xxx/32,*;-,*,* format. What would be most efficient way to filtering? IP or L7 filter?
 

Thanks, Derek! I was traveling, could not reply earlier...

In our configuration, we use direct feed from the span port (1GB). The span is multiplexed by a simple splitter device (no load balancing!) to every DLP monitor. We are using IP filter in the format +,xxx.xx.xx.xxx/32,*;+, to filter selected MTAs.

We cannot filter ANYTHING on the splitter appliance. Not the best design, I know, I have inherited it...

We do not use any high capture cards either...

Thanks!

Alex