by Mariusz Burdach
During the incident response process we often come across a situation where a compromised system wasn't powered off by a user or administrator. This is a great opportunity to acquire much valuable information, which is irretrievably lost after powering off. I'm referring to things such as: running processes, open TCP/UDP ports, program images which are deleted but still running in main memory, the contents of buffers, queues of connection requests, established connections and modules loaded into part of the virtual memory that is reserved for the Linux kernel. All of this data can help the investigator in offline examination to find forensic evidence. Moreover, when an incident is still relatively new we can recover almost all data used by and activities performed by an intruder.
Sometimes the live procedure described here is the only way to acquire incident data because certain types of malicious code, such as LKM based rootkits, are loaded only to memory and don't modify any file or directory. A similar situation exists in Windows operating systems -- the Code Red worm is a good example of this, where the malicious code was not saved as a file, but was inserted into and then run directory from memory.
On the other hand, methods presented below also have serious limitations and violate the primary requirement of the collection procedure for digital investigation -- a requirement which can not be easily fulfilled. That is: every user and kernel space tool used to collect data by nature changes the state of the target system. By running any tools on a live system we load them into memory and create at least one process which can overwrite possible evidence. By creating a new process, the memory management system of the operating system allocates data in main memory and then can overwrite other unallocated data in main memory or in the swap file system.
Other problems arise when we plan to take legal actions and need to comply with local laws. The signs of intrusions found in images of main memory can be untrusted, because they could be created by our acquisition tools. So before taking any action we must decide whether to acquire some data from a live compromised system or not. It is very often worth it to collect such information. In the main memory image we can find passwords or decrypted files. Using /proc pseudo file system we can also recover programs that have been deleted but are still allocated in memory.
In an ideal world, I could imagine a kind of hardware based solution for Intel-based computers, which would allow us to dump the whole memory to an external storage device without assistance of operating system. Such a solution exits on Sparc machines, whereby we can dump the whole physical memory by using the OpenBoot firmware. Unfortunately, no similar solution exists for Intel- or AMD-based computers.
Despite the above problem, software based methods also have advantages for forensic purposes, and I'll try to show them in this paper. The main goal of this article is a presentation of methods used during an evidence collection procedure. All collected data can be used later to perform offline forensic analysis. Some of presented tasks can be also be performed in the preparation and identification phases of the incident response cycle -- these are two of the six phases defined in a guide called "Incident Handling Step by step", published by the SANS Institute.
2. Forensic Analysis
This article is divided into four related sections:
Sections 2.1, 2.2, and part of 2.3 will be discussed in this article; the remaining steps and some offline procedures will be discussed next month in part two of this article series.
2.1 Fitting to the environment
Before gathering data from a live system we have to fit ourselves into the environment. First of all we have to run a network sniffer and it must "see" communication flows to and from a compromised system. This condition is mandatory. We can detect some types of malicious activities just by recording and analyzing, in real time, this communication. The utility tcpdump is excellent tool for this purpose. My advice is to record packets in a raw format because of performance issues that may result otherwise.
Before taking any activities on the compromised system we have to create a paper copy of our data collection procedure. An example procedure can be found in chapter three of this article. This procedure helps us to avoid any mistakes during the forensics of an incident. We must make additional notes after every finished step as well as if something goes wrong. Documentation is important, and is something to keep in mind if we plan to take our forensic case to court.
Our next step is to record the results of commands run during our phase of data gathering. From there, we connect a destination host to the same local area network on which we will be sending information from the compromised host. Remember, we are not allowed to write any results on the compromised system. Recording data locally on the compromised host can delete signs of an intrusion. To make less of an impact on a compromised system we have to send all our digital data to a remote, or destination, host. This is one of the most important rules in the forensic analysis process. And once again, as described earlier this is a requirement that is not always easily to fulfill.
If we don't have a forensic toolkit available for install on removable media, now is a good time to prepare it for our compromised system. Using tools from this toolkit we will collect all important data, beginning from the volatile to the less volatile.
The following methods describe a method how to prepare our media into a forensic toolkit.
2.2 Preparing the forensic toolkit media
It is important to remember that during a data collection process we have to fulfill following criteria:
Table 1: Requirements for a forensic toolkit on removable media.
When we build all above tools successfully, we can copy all of them to our removable media (such as a CD-RW disc).
2.3 Data collecting from a live system - a step by step procedure
The next requirement, and a very important one, is that we have to start collecting data in proper order, from the most volatile to the least volatile data. We have to remember about this during data gathering.
Step 1: Take a photograph of a compromised system's screen
Before moving on to step two, mounting our media, let's first think about the impact this next step will have on a compromised system. What will be an effect of our activity? For the moment let's ignore the impact it will have on the compromised system's memory.
It is clear that we have to mount external media into the compromised system. We must use the untrusted mount command to perform this task. This will be probably the sole situation when an untrusted system command is used. If everything will go according to the plan, we will run the rest of the command from the mounted media using tools that we trust.
We also have to check to see what the impact of the mount command will have on the system. I have done some research on a computer, and Table 2 lists the relevant files and directories that are modified.
Table 2: Files accessed by the mount command.
*We can avoid access to this file by using a "-n" switch.
We can imagine a situation when an intruder modifies the mount command. When someone tries to run this command perhaps a special process, which removes all evidence from the compromised system, is initiated instead of allowing the media to be mounted. Such a process is called a "deadman switch". But let's assume this is not the case, and now go back to the process of data collection.
I suggest that one verify every command that is going to be put on the forensic toolkit media, which later will be used on the compromised system to collect evidence.
We also have to stop and think about potential problems met during the mounting process:
Step 2: Media mounting
If the mounting process is successful we can start with the most important phase of data collection. Remember, all results generated by trusted commands have to be sent to the remote host. I use the netcat tool and the pipe method to do this. To better differentiate which tasks are performed on which host, all commands run on the compromised host will be prefixed with a (compromised) word in brackets. Commands run on the remote host will be prefixed with a (remote) word in brackets. Consider the following example.
To send information about an actual date of the compromised system into the remote location (the IP address of remote host in this case is 192.168.1.100) we have to open TCP port on the remote host as it follows:
Next, on the compromised host we do the following:
To maintain the integrity of digital evidence we calculate the hash value of the collected file and clearly document every step on our paper copy, to document this procedure.
Sometimes we can generate checksums on the compromised system and send the result to the remote host. A bit more about some of the problems this can cause has been discussed elsewhere in this article.
Step 3: Current date
The result is presented in the UTC format (Coordinated Universal Time)
Step 4: Cache tables
Mac address cache table:
Kernel route cache table:
Step 5: Current, pending connections and open TCP/UDP ports.
We can use the cat command instead of the netstat one in this case. Information about open ports is kept in the /proc pseudo file system (/proc/net/tcp and /proc/net/udp files). Information about current connections is placed in the /proc/net/netstat file. All data in those files are represented in the hex format.
For example: 0100007F:0401 in decimal is 127.0.0.1:1025.
As mentioned before, current connections can be detected by analyzing of the recorded traffic. It is important to note: an easy method of detecting a rootkit, loaded into kernel memory, is when one of its tasks is hiding an open port. We have to scan the compromised host remotely and compare the detected open ports with our result from the netstat command. But this causes a lot of harm and we once again change the state of the compromised system, in step seven I will present an alternate method of detecting hidden LKM based rootkits.
Concluding part one
Now that we have the date and the network status logged, we're ready to take some additional steps on the compromised machine before we power it off. Next month, in part two of this article series, we will focus on the search for malicious code by collecting more data to be sent to our remote host. We'll also discuss some of the searching that can be done with the data once we're able to go through it in a safe environment.
About the author
View more articles by Mariusz Burdach on SecurityFocus.
This article originally appeared on SecurityFocus.com -- reproduction in whole or in part is not allowed without expressed written consent.