Video Screencast Help

Integrating More Intelligence into Your IDS, Part 1

Created: 05 Mar 2008 • Updated: 02 Nov 2010
Language Translations
Admin's picture
0 0 Votes
Login to vote

by Don Parker and Ryan Wegner

The more an intrusion detection system (IDS) knows about the network it is trying to protect, the better it will be able to protect the network. This is the fundamental principle behind target-based intrusion detection, where an IDS knows about the hosts on the network.

This article explores how artificial intelligence (AI) is influencing IDS development, and what capabilities a popular IDS has with respect to intelligent intrusion detection. Snort is the IDS in question and this article describes some of its features that users might not be taking advantage of that would allow the IDS to adapt to networks and detect anomalies. AI alleviates some of the security professionals' work load by first learning about a network and gauging reactions from a security professional to reduce false positives, and second, by adapting to changes in the network to identify new attacks.

Such knowledge is important, for example, in identifying packet fragmentation attacks, where the hosts on a network have different policies for reassembling fragments. The packet fragmentation issue was discussed by Thomas Ptacek and Timothy Newsham roughly eight years ago in their paper Insertion, Evasion, and Denial of Service: Eluding Network Intrusion Detection. Since then, Snort developers have implemented a preprocessor for Snort that attempts to address the fragmentation issue. Judy Novak of Sourcefire published a paper in 2005 titled Target-Based Fragmentation Reassembly (pdf). Readers interested in more detail in this topic should read both papers.

Target-based intrusion detection

The fragmentation attack problem can be summarized as follows: if the IDS reassembles packets differently then the host machine, it will miss attacks against the host machine. The fragmentation problem and subsequent attempts to deal with it demonstrate the evolution of intrusion detection and in particular the agility of Snort.

As an example of a fragmentation attack, consider the following Snort rule in the exploit.rules file that comes with a stock Snort install:

Alert tcp $EXTERNAL_NET any -> $HOME_NET 22 (msg: 'EXPLOIT ssh CRC32 overflow /bin/sh'; flow:to_server,established; content:'/bin/sh'; reference: bugtraq,2347 reference: cve,2001-0144; reference: cve,2001-0572; classtype: shellcode-detect; sid:1324; rev: 6;)

One of the characteristics of this rule is that it looks for the content '/bin/sh'. Now consider that an attacker fragments his packets such that the packets sent have the following format. Assume for simplicity sake that each packet is engineered to only send one byte of data:

The IDS manages to reassemble the packets and gets the following content:

However, it turns out that several of the packets are source routed. The target host does not accept source routed packets and drops those packets. As a result the host machine reassembles the content to:

Do you see the problem? The IDS and the target host have different packet fragment reassembly policies. The target host does not accept source routed packets while the IDS does, so they end up with different payloads. The payload at the IDS appears harmless; however, the payload at the target host is not. The IDS is unsure how the packets will be reassembled at the host and therefore misses an alarm. This is called an insertion attack, where the IDS accepts a packet that the target host rejects. A similar attack, called an evasion attack, occurs when the IDS rejects a packet that the target host accepts.

When solving the fragmentation problem the solution is basically rule based. The IDS maps each host on the target network to a specific packet reassembly policy. For example, Windows has a different reassembly policy then FreeBSD, which is the result of interpretations made by the developers of the protocol stack in regards to the RFCs. When packets destined for a specific host are fragmented, the IDS is able to reassemble them in the same fashion as the host. The problem described above is simplified from the typical real world example. It should be clear that the IDS requires more knowledge about the systems it's protecting -- information that could potentially be learned.

Anomaly detection

Consider a more complex issue where intelligence is crucial to security and traffic flows. Typically a noticeable change in network traffic, an anomaly, indicates a security threat. Your FTP server has an outgoing session on port 3164 or your Web server is accepting very large client requests. How does your IDS learn to distinguish between normal traffic and anomalies?

Large client requests are normal for some Web servers and people do run FTP servers on port 3164. Security professionals can hard code the expected network activity; however, this is tedious and not practical for dynamic environments. Instead, your IDS learns what the typical network flow is like and then when the flow changes the IDS alerts you. Then you can give the system feedback as to whether this anomaly is interesting or not. Interesting anomalies will encourage the IDS to identify future interesting anomalies, while allowing the IDS to ignore non-interesting ones. This idea is not new in the AI or the intrusion detection development community and is typically referred to as anomaly detection.

Example of intelligence at work

Here we attempt to demystify the concept of AI by writing a Snort preprocessor that can adapt (that is, a learning preprocessor). It will be a proof of concept to begin with, showing some benefits of intelligence in an IDS, and it will also highlight features of Snort that some people may not have had a reason to tinker with. What is Snort? Below is an explanation that was taken directly from the Snort project Web site:

Snort is an open source network intrusion prevention and detection system utilizing a rule-driven language, which combines the benefits of signature, protocol and anomaly based inspection methods.

There are two key elements that make Snort an ideal choice in this article. For one, it's open source. Also, it's, well, it's open source. So maybe that's just one, but itís a big one! Even if you are not a developer it allows you to leverage developers to produce customizations to Snort.

Snort Traffic flow

Snort passively captures all traffic from a given source using a packet capture module based typically on either WinPcap or LibPcap depending on the given operating system. The traffic then passes through a decoding module responsible for identifying the link level protocols and working its way up the network layers (e.g. Ethernet->IP-> TCP) all the while organizing the packets into data structures for further processing.

At this point Snort may generate alerts for malformed packets. Next, the traffic passes through the preprocessors. For now it suffices to say that the preprocessors identify suspicious traffic or traffic that requires special processing, but we will come back to them later. For example the Frag3 preprocessor identifies fragmented packets and reassembles them according to the destination host (discussed earlier).

Next, the traffic is forwarded through the detection engine, which is the portion of Snort that most people identify as the IDS. The detection engine applies predefined rules to the traffic and generates alerts based on those rules. The rules file is the portion that most administrators update on a regular basis as new vulnerabilities are discovered.

Lastly, output plug-ins dictate how alerts and logs are written. For example, a flat file with an associated tcpdump log. By default there are two customizable portions of Snort -- the rules file and the preprocessors. Here we are interested in preprocessors.

Snort Preprocessors

A Snort preprocessor is essentially some C code that is applied to every packet that is decoded before it is passed on to the detection engine. If Snort is not working the way you like it, or if there is missing functionality, developers or security professionals can take it upon themselves to write in that missing functionality. An example of a Snort preprocessor is spp_fnord.c by Dragos Ruiu.

When exploit code is sent to a machine in the form of shellcode, it typically contains a long series of "no operation, commands" (NOPs). This is called a NOP sled. In a buffer overflow attack the NOP sled supplies the exploit developer with a fudge factor when they figure out what address they should be jumping to in order to execute their shellcode. If the jump instruction lands anywhere in the NOP section of the buffer containing the shellcode the machine will execute the NOPs one after the other, which does nothing, until it reaches and executes the shellcode.

The intention of the preprocessor is to identify NOP sleds in shellcode, since it provides an indication of an exploit. Depending on the underlying architecture of a host, several different assembler commands in shellcode can be used to create the same effect as your typical NOP. Dragos wrote a Snort preprocessor that examines every packet to see if a NOP sled exists. There are a finite number of possible NOP commands per architecture, so it is possible to step through a packet payload byte by byte looking for the commands. Here is a snippet of the code from spp_fnord.c:

if( CMP3(0x20,0xBF,0xBF) || /* bn -random */ CMP3(0x81,0xD0,0x20) || /* tn random */ CMP4(0x89,0xA5,0x08,0x22) || /* fadds %f20,%f2,%f4*/
CMP3(0xBA,0x56,0xA0) /* umul %i2,0x42,%i5 */ )

This code is an if statement that uses a CMPx macro to compare a number of bytes in a packet (here 3 or 4) to an assembler command in hex. The comment on each line indicates which NOP assembly command it is being compared to. In this code snippet the target architecture is SPARC. This is another rule-based approach for solving a problem. There are a finite number of NOP instructions being tested for. If an exploit developer uses a NOP instruction that is not tested by the preprocessor it will fail. However, assuming assembly languages are half decently static and the list of NOP instructions looked for is complete; the preprocessor will be able to identify NOP sleds.

Setting up the preprocessor

A FreeBSD system was used for this article, but setting up the preprocessor in other environments is quite similar and the configuration is basically the same. Differences in the path structures of other UNIX flavors are likely to exist, but those differences should be relatively minor. In order to include a new snort preprocessor you have to get the source code for snort and compile it yourself. The snort source is available on the project's main Web site. If you are running FreeBSD and you have installed the ports collection, go to the directory /usr/ports/security/snort and execute ìmake installî, this will install a stock version of snort and at the same time it will uncompress the source code and stick it under:


Once you have the source code and it has been uncompressed, look for a directory called preprocessors. It should be directly under the src directory. On FreeBSD you can find the preprocessors under:


Just replace the x.x with the specific version you have installed. In that directory are a series of spp_ files. For each preprocessor there is a header file, a source file and an object file (e.g. spp_flow.h, spp_flow.c and spp_flow.o). Remember to back up any files you make changes to. You can re-make snort so that your changes take effect by executing make install again under /usr/local/ports/security/snort/. An interesting one to look at to learn more about writing snort preprocessors is spp_telnet_negotiation. It is a relatively short preprocessor that can server as an example for writing others.

Each snort preprocessor you write should be put into this directory and compiled so that it can be referenced by the snort.conf file. The snort.conf file is separated into four steps. The second step is the preprocessor configuration and the snort preprocessor configuration looks like this:

preprocessor :

For example:

Preprocessor stream4: disable_evasion_alerts

The name of the preprocessor is contained in the source code in the setup function of the preprocessor. If you do a quick search on the file you should find a line:

RegisterPreprocessor(ìî, );

For example:

RegisterPreprocessor(ìstream4î, Stream4Init);

Typically the snort.conf file contains instructions specific to the standard preprocessors bundled with the default snort download.

This article originally appeared on -- reproduction in whole or in part is not allowed without expressed written consent.