by David Elson
Network security has been an issue almost since computers have been networked together. Since the evolution of the internet, there has been an increasing need for security systems. One important type of security software that has emerged since the evolution of the internet is intrusion detection systems.
This article gives an overview of several types of intrusion detection systems, and introduces the reader to some of the concepts and practices involved in intrusion detection. Be aware that this article is only introductory, and while I have suggested a number of possible systems, further research should always be undertaken before trusting in the strength of your intrusion detection system.What is Intrusion Detection?
Intrusion detection, is the attempt to monitor and possibly prevent attempts to intrude into or otherwise compromise your system and network resources.
Simply put, it works like this: You have a computer system. It is attached to a network, and perhaps even to the internet. You are willing to allow access to that computer system from the network, by authorised people, for acceptable reasons. For example, you have a web server, attached to the internet, and you are willing to allow your clients, staff, and potential clients, to access the web pages stored on that web server.
You are not, however, willing to allow unauthorised access to that system by anyone, be that staff, customers, or unknown third parties. For example, you do not want people (other than the web designers that your company has employed) to be able to change the web pages on that computer. Typically, a firewall or authentication system of some kind will be employed to prevent unauthorised access.
Sometimes, however, simple firewalling or authentication systems can be broken. Intrusion detection is the set of mechanisms that you put in place to warn of attempted unauthorised access to the computer. Intrusion detection systems can also take some steps to deny access to would-be intruders.Why use Intrusion Detection?
The underlying reasons why you might use intrusion detection systems are relatively straightforward: You want to protect your data and systems integrity. The fact that you cannot always protect that data integrity from outside intruders in today's internet environment using mechanisms such as ordinary password and file security, leads to a range of issues.
Adequate system security is of course the first step in ensuring data protection. For example, it is pointless to attach a system directly to the internet and hope that nobody breaks into it, if it has no administrator password! Similarly, it is important that the system prevents access to critical files or authentication databases (such as the NT SAM or the Unix /etc/passwd or /etc/shadow files) except by authorised systems administrators.
Further measures beyond those normally expected of an intranet system should always be made on any system connected to the internet. Firewalling and other access prevention mechanisms should always be put in place. While it may be acceptable to allow NT logon, file sharing, or telnet access to a system that is entirely internal, an internet server should always use more secure mechanisms, such as firewalling off the NT file sharing (SMB protocol) ports such as TCP/UDP ports 137 - 139, and using secure shell (SSH) instead of telnet for access to Unix systems.
Intrusion detection takes that one step further. Placed between the firewall and the system being secured, a network based intrusion detection system can provide an extra layer of protection to that system. For example, monitoring access from the internet to the sensitive data ports of the secured system can determine whether the firewall has perhaps been compromised, or whether an unknown mechanism has been used to bypass the security mechanisms of the firewall to access the network being protected.What types of Intrusion Detection systems are there?
Intrusion Detection systems fall into two broad categories. These are:
I will also discuss briefly a more recent type of intrusion detection system: Those that reside in the operating system kernel and monitor activity at the lowest level of the system. These systems have recently started becoming available for a few platforms, and are relatively platform specific.
Network Based Intrusion DetectionIntroduction
Network based intrusion detection systems are those that monitor traffic on the entire network segment. A network interface card (NIC) can operate in one of two modes, these being:
A network card can normally be switched from normal mode to promiscuous mode, and vice-versa, by using a low-level function of the operating system to talk directly to the network card to make that change. Network based intrusion detection systems normally require that a network interface card is in promiscuous mode.Packet Sniffers and Network Monitors
Packet Sniffers and Network Monitors were originally designed to aid in the process of monitoring the traffic on an Ethernet network. The first of these were two products; Novell LANalyser and Microsoft Network Monitor.
These products basically capture all packets that they see on the network. Once the packets are captured, a number of possibilities arise:
Packet sniffing products have become more sophisticated in recent years. Programs such as Ethereal and more recent versions of Network Monitor can disassemble the insides of various types of packets to show what type of communication is happening inside that packet.
One final word about packet sniffers: These tools can be used to do evil as well as good. For example, packet sniffing can be used to find out someone's Unix password by sniffing telnet packets to the machine that they connect to. Once an attacker has compromised your network, one of the first things they might install is a packet sniffer of some kind.Packet Sniffing and Promiscuous Mode
All packet sniffers will require that a network interface is in promiscuous mode. Only in promiscuous mode will every packet received by the NIC be passed up to the packet sniffer itself. The packet sniffer normally requires administrative privileges on the machine being used as a packet sniffer, so that the hardware of the network card can be manipulated to be in promiscuous mode.
Another point to consider is the use of switches, rather than hubs, in a network. Note that packets received on one interface of a switch are not always sent to other interfaces of the switch. For this reason, a heavily switched environment, rather than an all-hubs (single segment) environment, will often defeat the use of packet sniffers.Network Based Intrusion Detection: The Evolution of the Packet Sniffer
Unfortunately, from a security point of view, a packet sniffer is of limited benefit. The task of capturing every packet on the network, disassembling it, and manually taking action based on the contents of the packet is far too time-consuming, even for a horde of specially trained network gnomes. What if we were to have some software that automated the process for us (after all, that is what computers are for in the first place, is it not?).
This is basically exactly what a network based intrusion detection package does. Two packages that are available that perform this type of intrusion detection are the ISS RealSecure Engine and Network Flight Recorder.
Here is an example of some of the types of intrusion detection that the RealSecure Engine can perform:
In this manner, RealSecure can perform an effective second layer defence for a target system when hosted behind a firewall. Some implementations have also used RealSecure in place of a firewall, which is, however, something that I do not recommend.
Network based intrusion detection can perform a few other tasks, for example:
When unwanted activity is detected, network based intrusion detection can take action, including interfering with future traffic from the intruder, or reconfiguring a nearby firewall to block all traffic coming from the intruder's computer or network.
Host Based Intrusion DetectionIntroduction
Once a network packet has arrived at the host that it was intended for, there is still available a third line of defence behind the firewall and network monitor. This is called "host based intrusion detection", and comes in several flavours.
The two main types of host based intrusion detection are:
It is possible on most hosts to monitor packets that attempt to access the host before those packets are passed onto the networking layer of the host itself. This mechanism attempts to protect a host by intercepting packets that arrive for the host before they can do any damage.
Some of the actions that can be taken include:
Two software products that perform this type of monitoring include the RealSecure Agent by ISS, and PortSentry.Monitoring Login Activity
Despite the best efforts of the network administrator, and the most recently deployed and monitored intrusion detection software, occasionally an intruder will slip by and manage to log on to a system using a heretofore unknown type of attack. Perhaps an attacker will have obtained a network password by some means (packet sniffing or otherwise) and now has the ability to log on to the system remotely.
Looking for unusual activity on a system is a job for a product such as HostSentry. This type of package monitors log-in and log-out attempts, and alerts the system administrator to activity that is unusual or unexpected.Monitoring Root Activity
The aim of all intruders is to obtain super-user (root) or administrator access on the system that they have compromised. Well-maintained and reliable systems that are used as web servers and databases will usually have little or no activity by the super-user, except at particular times of the day or night for scheduled system maintenance. Fortunately, crackers do not believe in system maintenance. They rarely stick to scheduled downtime windows and often work at odd hours of the day. They perform activities on the system that are unusual for even the most propeller-headed system administrator.
There is one more line of defence: Monitoring any actions performed by the root user or system administrator. Many unix systems allow logging or other monitoring of all activity by the root user, and packages such as Logcheck can then scan these logs for unusual activity and notify others about it.
Administrators of open-source operating systems have one final option: alter the kernel to log specific types of activity. How to do this is outside of the scope of this article, however there are a number of sites on the internet that detail how this can be done.Monitoring the File Systems
Once an intruder has compromised a system (and despite your best hopes, and the best efforts of the intrusion detection systems that you have laid down, one cannot discount entirely the possibility that one day an intruder will compromise a system), then they will start to change files on the system. For example, a successful intruder might want to install a packet sniffer or portscan detector, or modify some of the system files or programs to disable some of the intrusion detection methods that they have worked around.
Installing software on a system usually involves modifying some part of that system. These modifications will usually take the form of modifying files or libraries on the system.
For example, the following approaches can be taken:
The methods used by Tripwire, Fcheck, and AIDE vary somewhat, but they are all based on the above mechanisms. These programs also take care to make sure that the database of known cryptographic checksums itself isn't compromised in any way. For example, it would be possible for an intruder with a knowledge of the operating system and the intrusion detection software to modify the system files and also modify the cryptographic checksum database so that it would appear that nothing had changed.
Kernel Based Intrusion Detection
Kernel based intrusion detection is a relatively new art form, and one that is starting to become prevalent, especially within Linux.
There are two main kernel based intrusion detection systems currently available for Linux. These are OpenWall and LIDS. These systems take the approach of preventing buffer overflows, increasing file system protection, blocking signals, and generally making it difficult for an attacker to compromise a system. LIDS also takes steps to prevent certain actions by the root user, such as installing a packet sniffer or changing firewall rules.Kernel Protection vs File System Monitoring
Obviously, systems like LIDS and systems such as Tripwire take a rather different approach to attempting to achieve the same thing. Both of these packages attempt to prevent a cracker from using the system for unauthorised purposes.
At first glance, one might think that a system such as Tripwire is less than perfectly useful. While it is a Good Thing to monitor file systems for signs of abuse, it is readily accepted that once your system has been compromised by an external intruder, it is time to shut down and rebuild. The damage has been done, and the system integrity cannot be guaranteed so it is best to re-build your operating system from the pristine version supplied on CD from your vendor. The approach offered by LIDS seems much more attractive - to protect the system from damage rather than to note that the gate was left open after the horse has bolted, so to speak.
Although I tend to agree with this analysis in principle, there is a stronger level of security offered by running both LIDS and Tripwire together. While LIDS is exceptional in its capability to protect the file system, it is worth using a file system monitoring package such as Tripwire as an "independent" auditor, in the event that an exceptionally knowledgeable hacker should manage to somehow defeat LIDS.
It is possible, using the most up to date tools that are available, to protect against virtually every type of threat that is currently known about. Unfortunately, new threats and security holes in some software package or another are being discovered on a daily basis.
It is important in any environment to know what types of threats you might be facing. Be aware of any potential security holes in your system, and take care to prevent attacks against these. For example, a web server that is connected to the internet and placed behind a firewall may be reasonably secure against most packet based attacks, but a CGI program on the server might expose a vulnerability. Pay special attention to ensuring that CGI programs correctly bounds check all arrays and validates input data before processing. An intrusion detection program between the firewall and the web server might configured to throw out any accesses that are suspicious.Staying Up To Date
Most of the intrusion detection tools that I have mentioned are regularly updated to include information about new threats as they are discovered, however it is important to keep up to date with the latest version of these tools.
Watching certain mailing lists (such as BUGTRAQ) and security web sites can help you stay informed about the latest security issues affecting software that you have installed. If you are alerted of a vulnerability in a software package that you are using, or in a firewall product, or perhaps even in an intrusion detection software package itself, then don't be shy about contacting the vendor for a fix.Which Tools?
I have discussed a number of different types of tools in this paper, all with different functions.
To keep your environments as secure as possible, it is important to choose tools from across the range of functions. Each of the tools forms an extra layer of protection in case the others are defeated. So, your first layer of protection should be a firewall. Behind that, a network based intrusion detection system will catch any breaches of the firewall. Behind that again, a set of tools that monitor connection attempts, such as PortSentry and HostSentry form an extra layer of protection. Finally, if all else fails, tools to catch an actual break-in, such as LogCheck or Tripwire, form the final layer of defense.
David Elson (Del) is a security and technology consultant working for Wang New Zealand in Christchurch, on the South Island of New Zealand. With 15 years IT experience, he consults to various clients on security and networking issues. He also maintains a set of web pages on Linux and other related security topics, and has given talks on various security and networking issues at conferences in Australia and New Zealand.
This article originally appeared on SecurityFocus.com -- reproduction in whole or in part is not allowed without expressed written consent.