Detecting Rootkits And Kernel-level Compromises In Linux
by Mariusz Burdach
This article is intended to outline useful ways of detecting hidden modifications to a Linux kernel. Often known as a rootkit, this stealthy type of malware gets installed in the kernel of an operating system and requires special techniques by Incident handlers and Linux system administrators to be detected.
In this article we will make use of just one tool,
This focus on detecting kernel modifications is important because it is the most stealthy of all methods for an intruder to install malicious code in an operating system. Once this malicious code is in place, intruders can defeat most commercial and free host intrusion detection systems (IDSs) which monitor the integrity of the operating system's files.
Introducing the threat of rootkits
More and more user-mode malicious programs, such as Trojan horses, backdoors or rootkits, modify existing operating system software. To install one of these tools on a victim's machine, an attacker must replace or modify the normal programs that are associated with the operating system. For example, let's consider a replacement of the ubiquitous
Let's suppose that an attacker doesn't replace or modify any existing programs, such as
Now, let's consider the another example where an attacker modifies both the sys_open and the sys_read system calls to block access to a set of selected files. These same system calls are usually used by file integrity checkers to verify the integrity of important system files, like the kernel image or loadable kernel modules. When these tools try to compare the hashes of files to their previous values, they will remain the same even when these files have, in fact, been modified. In other words, critical files could be shown to be intact by file integrity checkers when they are not, just by hooking two system calls. It should be quite obvious, then, that when kernel components are modified by an attacker, users and administrators cannot trust any of the results received from the kernel or from any security related tools that run as a user.
Linux kernel-mode rootkits, or other kinds of malicious code, are installed directly in a memory area reserved for the kernel code and they are really powerful. They modify kernel structures in order to filter data which will be hidden from system administrators. To filter this data it is necessary to take control over some kernel components like system calls, interrupt handlers, internal functions of the netfilter, and more. It is easy to imagine several places in the kernel of an operating system where such control can be manipulated. At this point, it is now important to understand some of the most popular attack vectors.
Understanding the attack vector
The most common targets of a compromise are system calls. This vector is chosen by intruders for two reasons: because it is the easiest way to take control over a compromised machine, and also because system calls are very powerful. System calls are basic functions used by an operating system. For example, they are used to read and write data to and from files, they are used to get an access to various devices, to run executables, and so on.
There are about 230 system calls in the current stable version of the Linux kernel, 2.4.27, and about 290 system calls in the Linux kernel 2.6.9. Note that the number of system calls changes depending on the version of the kernel. The full list of system calls in your kernel is always available in the file
Table 1. Important Linux system calls, their description, and their system call IDs.
For the above table, note that the ID is the number of an entry in the system call table. For the purposes of this article, the IDs are used for Linux kernel 2.4.18-3.
While all of the examples presented in this article were tested on the Red Hat 7.3 with kernel 2.4.18-3, similar steps can be done during an investigation of other versions, including the latest Linux kernel 2.6.x. Some differences can appear in internal structures of the Linux kernel 2.6.x, however. For example, the address of system call table is kept inside of the function syscall_call instead of system call handler named system_call .
Modifiying the system call table
Current addresses of system calls are kept in the system call table, in a memory area reserved for the kernel of an operating system. Addresses there are kept in the same order as their functions, and are presented in the
Let's start with an example. When the sys_write system call is invoked, its ID of 4 is placed into the eax register and a software interrupt is generated (int 0x80). There is a special interrupt handler which keeps this address in its interrupt descriptor table and is responsible for handling the interrupt (again, int 0x80). Next, the system call handler system_call is invoked. This handler can locate the direct address of the requested system call by knowing the address of the system call table and the ID of the system call (which is kept by the eax register). There is also a longer way to invoke this system call handler, but I have omitted some details to simplify this article.
The first method an intruder uses when taking control over a desired system call is to overwrite the address of the original system call in the system call table. When the system call is requested, the handler calls a replacement function. We can easily watch these addresses, kept in the system call table, by using the
Of course, another problem arises. We have to be sure that the current addresses in the system call table are not modified -- that we are not already compromised. How can we verify this? The addresses of system calls are always permanent and do not change after a reboot of the operating system. These addresses are set during kernel compilation, so knowing the original addresses we can compare them to addresses currently placed in the system call table. This information on the original addresses is written into two files on the file system during compilation time. The first one is the
Sometimes only a compressed version of the kernel may be available (named
We could also use a simple loadable kernel module, presented in the references section of this article, to print virtual addresses of each system call. To do this we compile the source code as follows:
In most cases, the kernel is modified by rootkits after a system initialization. It is done by loading a malicious kernel module or by an injecting some malicious code directly into the
The first step is to find an address of the system call table. This should be an easy task, as shown below, because the symbol sys_call_table is presented in the
Now, using the
We can also print the address of each system call by providing its name, as shown below:
Now, by using the
To print out the current state of the kernel, we must run the
As we can see from the above output, one of the addresses of the system call has indeed been changed. This is entry number 3 in the system call table (counting entries from 0), and is bolded in the above output for clarity. In the
The another sign of a system compromise is that the new virtual address of this function (sys_read) is above 0xc8xxxxxx. It is, by nature, quite suspicious. A Linux operating system by default can address up to 4 GB. Virtual addresses range from 0x00000000 to 0xffffffff in hexadecimal notation. An upper part of this virtual memory area is reserved for kernel code (their values range from 0xc0000000 to 0xffffffff). When a new loadable kernel module is loaded, the vmalloc function allocates a part of this memory for the code of the module. It allocates a memory region -- usually starting from 0xc8800000. So, whenever the reference to the address of the system call is above this address, as we saw in this example, it indicates that our kernel could be compromised. At this point, it becomes necessary to look closer at this system call.
System call hooking
Now we will examine a method of detecting system call hooking. None of entries in the system call table are modified with this method. Instead, the first few instructions of the original function are overwritten with a jump to a replacement function (called a detour function). Let's imagine that an intruder wants to hook the sys_read system call. He must first load the replacement function into memory and then place the address of this function in the first few bites of the original function. In doing so, the intruder must redirect an execution flow of the original function to the replacement one. Assembly instructions, like call or jmp, are usually used.
To detect whether any system call has been hooked, we must print out all the instructions of the target function. We start by running the
From the above output, we can see that the first instruction moves the value (the address of the replacement function) to the ecx register. The second instruction does an indirect jump to this virtual address - 0xc88ab0a6.
To make sure that the sys_read system call was hooked, we must disassemble the original function. The original function is presented in the kernel image
The output confirms that the sys_read system call has indeed been modified. To investigate what this new function really does, we can disassemble the function by using the
A modification of the system call handler
Using the method as described above, we can also disassemble other critical functions in the kernel memory. One of them is the system call handler, named system_call, that is used to locate and call requested system calls. The handler uses the system call table to find an address of the requested system call. By disassembling this handler, we can check if the right address of the system call table is used or if the handler is hooked. In an attack scenario, an intruder can create his own system call table using replacement system calls. Then, he can place a new address for the system call table in the system call handler.
Please note that this disassembled handler contains the address of the original system call table.
Of course, it would be even better to automate all tasks described in this article. One of the ways to use this is with host based intrusion detection systems which monitor critical kernel structures in real time. For example, the Samhain tool can be used. This tool was briefly described in a previous Infocus article, "Host Integrity Monitoring: Best Practices for Deployment". The Samhain tool is able to monitor the system call table, the first few instructions of every system call including some handlers, the interrupt description table, and much more.
When considering an IDS to monitor the integrity of our kernel, we must remember one basic rule -- every kind of monitoring tool such as this must be installed on a clean install of the operating system, as this is the only way to know that we have not already been compromised. If we don't install any host based IDSes we should at least create a hash sum of the kernel image.
As we have seen, the
The detection of a kernel level compromise can be very complicated without confirmation that at least one source of knowledge is trusted. In examples used in this article, the kernel image is the trusted source of knowledge.
 Daniel P. Bovet, Marco Cesati. "Understanding the Linux Kernel", 2nd Edition. O'Reilly; 2002.
 "Host Integrity Monitoring: Best Practices for Deployment", http://www.securityfocus.com/infocus/1771.
 "Linux on-the-fly kernel patching without LKM", http://www.phrack.org/phrack/58/p58-0x07
 System manual (2), "Linux Programmer's Manual".
 The GNU Project Debugger, http://www.gnu.org/software/gdb/gdb.html .
 scprint.c is the loadable kernel module that allows one to print system calls from the kernel memory. This is available for download from SecurityFocus.
 The samhain file integrity / intrusion detection system, http://la-samhna.de/samhain/
View more articles by Mariusz Burdach on SecurityFocus.
This article originally appeared on SecurityFocus.com -- reproduction in whole or in part is not allowed without expressed written consent.