by H. Carvey
|No Stone Unturned: Part One
by H. Carvey
last updated February 27, 2002
Eliot sat before the glow of his screen. It was early Monday morning, too early for most people to be in the office and still quiet enough for him to indulge in the ritual that burned away the pleasant and comforting fog of the weekend...strong coffee, e-mail, and a little Web surfing. Subscribing to several lists and having a bookmarked list of pertinent sites kept him in the loop on developments in the computing industry that might impact his day-to-day life. Add to that a liberal dose of humor, such as the UserFriendly Web site, and he'd developed a routine that he followed every Monday morning before the other employees of the telecomm company he worked for began trickling in and logging on to the network. After all, in any given week, it was this time that offered the only single, contiguous period of quiet.
As a system administrator (sysadmin or just admin for short) Eliot knew that as the other employees began arriving, things would quickly pick up. That was what the coffee was for, anyway…to wake him up, but also to prime his system to get ready to respond to the calls that would inevitably come. He could expect the usual calls for help as they began trying to log onto the network. They'd forget that they'd turned off their monitors on Friday when they left, or they'd left their laptops with the power cord unplugged but running over the weekend. Eliot handled all of these events in stride and with good humor, but he knew that he had to be awake to do so.
Every now and again, Eliot used this time to reflect back upon his short career, to see where he'd been and how far he'd come. Eliot had started out as a junior network administrator responsible for a couple of Solaris Web servers. He'd had the usual Windows 95 desktop provided by the company, but he much preferred the Solaris systems. Many times while researching an issue for one of his "babies", he'd strike upon something interesting, and couldn't wait to get home to his various Linux systems to see how the same issue applied there. Whenever possible, Eliot had attempted to automate any task he had by using shell scripts or Perl. He'd found that by adapting scripts he'd found on the Internet to meet his needs, he'd greatly reduced the amount of time he had to spend on repetitive tasks, such as log file collection, review, and archiving. Sometimes he did have to create something from scratch, but that was no problem. Besides, it was fun to pass his new creation around the office, and then step back into that creative haze when one of the senior admins said something like, "yeah, but can it do *this*?"
He hadn't given the Windows systems much notice, because after all, weren't they basically the old DOS systems with pretty pictures? Because of the initiative and persistence he’d shown working with the Solaris systems, he'd been ripped out of his comfortable and secluded little server womb and thrust up closer to the corporate level. Now he was responsible for managing and administering Windows NT and 2000 systems, not only as Web servers, but also as domain controllers and servers on the corporate LAN. Since he had a lot of interaction with the helpdesk when troubleshooting a user's issue, Eliot was often called upon to lend a hand in the desktop arena, as well.
Even though he'd switched platforms and the learning curve had been initially steep, Eliot had found, to his astonishment, that all the time he'd put into learning Perl programming would actually be of use. In it's own way, he found Perl on Windows systems to be equally as powerful as it's Unix counterparts. And with the addition of various command line tools he’d developed a pretty powerful toolkit that rivaled anything he’d had on his Solaris systems.
Even with the new challenges, all in all, things weren't bad. He continued to automate repetitive tasks, such as collecting and reducing EventLog data from the Windows servers. For example, he'd written a script that could go through the IIS Web logs, produce reports on a myriad of metrics (unique IP addresses, numbers of errors versus successful page requests, etc.). He'd also written scripts that combed through the logs looking for malicious activity, such as worms, or attempts to run 'cmd.exe'. He'd done similar things for the EventLogs on many of the key servers. He'd found that he could dump the output into a comma-delimited format that could be read by Excel (dumpevt.pl), and then save information culled from the reports in an Excel spreadsheet so that graphs could easily be produced. The Perl distribution for the Windows platform provided a rich set of functionality, essentially by providing wrappers around many of the Win32 API calls.
Eliot had also found that he had an ability to work and communicate with the people who used the systems. He felt equally at ease whether assisting a user, working with a customer to resolve an issue with their Web server, or explaining the effect of the latest bit of malware to the VPs. His boss liked that, and called him "a nerd who doesn't sound like a nerd."
At 8:52am, Eliot noticed that the contents of this coffee cup had become desperately low, and had gone from tepid to just plain cold. He considered actually getting up and fetching a fresh cup of this steaming elixir of the gods, but he knew the gods well, and they could be cruel. He'd leave the relative calm and safety of his little virtual world, and either find an empty glass carafe in the coffee machine growing progressively hotter, or a problem requiring his immediate attention would crop up the instant he was out of the room. As the thought of installing a video camera trained on the coffee pot flashed through his now-alert brain for the thousandth time, in one swift motion he locked his workstation, spun out of his chair, grabbed his coffee cup, and headed down the hall.
By now, many employees were at their desks, moving through their own Monday morning rituals. Even from across the room, he could see e-mail clients being opened, as several folks simultaneously sipped a hot cup of coffee and checked their voice mail. Such is the pulse of corporate America, as the giant awakes. As Eliot approached the break area where the coffeepot shared space with snack and soda machines, he saw Cynthia from HR chatting with a manager, Bob somebody. Wow, thought Eliot, this conversation must be important. Cynthia was always friendly and chatty, offering a cheery good morning or good afternoon as she passed by folks. Eliot was usually up and at work much earlier than she was, so he was able to make it to the safety of his second cup of coffee by the time she showed up in the halls of the building. Caught unprepared and unprotected by that warm, comforting cloak of caffeine, many bristled at her pleasantries. But this time, Cynthia was serious...her arms were crossed high up on her rib cage with her shoulders drawn up, and her expression was stern and intense, almost grim. Eliot couldn't see Bob's face, as Bob was facing Cynthia. From what he saw, this was something pretty important, so as he approached to retrieve his coffee, he tried to remain unobtrusive. Yet as he started to step away from the break area, he heard Cynthia call his name.
He responded. "Hey, Cynthia, good morning. What's up?"
She said, "Eliot, Bob and I need to talk to you. Do you have a couple of minutes?"
"Sure. No problem," he said.
Cynthia motioned Bob and Eliot into one of the many "team rooms" located throughout the building. These rooms had phones, tables, and white boards in them, and even network drops. They were smaller than conference rooms, and used by small teams of employees to get together and hammer things out. Eliot and the other administrators had used rooms like this when they'd worked out how to update the EventLog configurations on remote servers, and then to collect the resulting audit data.
Once inside the team room, Cynthia shut the door and took a seat. Eliot put his cup down and did the same, and Bob leaned against a wall, obviously agitated. Since he'd been brought into this meeting, Eliot decided that it was best to just see what was going on, to let either Cynthia or Bob talk first. Fortunately, Cynthia didn't let the awkward silence go on for too long.
"Eliot, we've got a situation and we'd like to ask for your advice with something."
"Sure, Cynthia. What’s up?"
Cynthia let out a slight sigh. “Bob’s seen some unusual things on the network lately.”
Eliot turned to Bob, who was still leaning against the wall, with his arms folded across his chest. Eliot blinked and held the silence for one very pregnant second, waiting for Bob to say something. “What kind of things, Bob?”
Bob cleared his throat. “We’ve been upgrading some of the network devices…routers, switches, and even some of the firewalls. One of the things we’ve started doing that we hadn’t been in the past is logging. Each of our network engineers now uses a unique name and password combination to log on, so we can track activity. Well, lately we’ve noticed that there are a lot of failed log-in attempts to these devices, using our old account names.”
Eliot was somewhat incredulous. “This sort of thing is bound to happen. Maybe some bored kid on the Internet found some IP addresses that belong to us, and is just trying to see what he can find.”
“The IP address that the attempts are coming from are from right here on our own network,” said Bob.
With that, the conversation took a hairpin turn, and Eliot’s mood became more somber, matching Bob’s. Eliot had dealt with mistakes made by users and administrators alike in the past, but this was something he hadn’t seen before…someone within the company purposefully trying to gain access to devices on the network. If this sort of thing had been going on in the past, no one was the wiser…there hadn’t been any auditing and logging performed on the network devices. Now that is was being done, the network engineering staff was getting a whole new view of what happens on the network.
Eliot thought for a moment, and then turned to Cynthia. “So what can I do?”
“We’d like you to take the IP address from Bob, and take a look at the computer that’s using that IP address. We don’t know if this is some kind of virus or something, or if it’s the employee doing these things themselves. Can you find out for us, without causing a scene?”
“Sure,” responded Eliot. He asked Cynthia a few more questions, and determined that going directly to the employee’s computer was out of the question. Cynthia was concerned about getting any rumors or speculation started, particularly when they didn’t really have a clear idea of what was going on.
Before the trio broke up, Eliot asked Bob about the nature of the traffic he was seeing. All of it was telnet traffic, meaning that the failed login attempts were being made against the telnet server running on the various network devices. Bob told Eliot that some of the firewalls had already been upgraded, and didn’t have telnet servers running on them because they were being managed through another means. However, these devices were showing the initial TCP communications, the SYN packets, being dropped, as if someone were trying to login. Armed with that information, Eliot felt that he had enough to get started, and promised to keep Cynthia and Bob informed of this progress.
Once he got back to his office, Eliot set about the task of trying to find out what was going happening. There wasn’t really any clear idea of how long this sort of activity had been going on, so he couldn’t rely on searches of file times. All he knew was that these failed attempts to access these devices were coming from on system in particular, and that Bob was going to have logs of this activity available later in the day, should Eliot need it. All Eliot really had at his disposal at this point was the fact that the corporate infrastructure was primarily a flat NT domain with a smattering of Windows 2000 systems, and he had a domain admin account.
Armed with the IP address, Eliot started by using the ‘nbtstat’ command native to NT and 2K to determine the system’s NetBIOS name and the logged on user. Armed with that, he then enumerated the available shares and found the ubiquitous hidden administrative shares he was expecting. He then mapped the C: drive from the remote system to his own, and began to look for clues by examining files on the remote system, via Windows Explorer.
After almost a full hour of looking through folders and not finding anything more suspect than a couple of “hacker” text files and a back issues of Phrack on the system, Eliot was just about to give up and tell Cynthia that he hadn’t found anything. In particular, he hadn’t found any files associated with some of the trojans and viruses he’d been reading about lately. The last things he opted to do were collect the EventLogs from the system, and use a free tool he’d found on the Internet to enumerate all of the currently running processes on the system. He did find a process called ‘telnet.exe’ running, so he figured he’d found something to take to Cynthia. Eliot’s final act for Monday was to send Cynthia and Bob an e-mail, explaining what he’d found.
Later that week, Eliot was attending an NT users group meeting with his friend Steve. Steve hadn’t had the benefit of an education steeped in the mysteries of Unix, as he’d been using Windows systems from a very early age. Eliot wasn’t a regular to these meetings, but he’d go along with Steve whenever there was something interesting planned, such as a product demo or someone from Microsoft speaking.
During a break in the meeting, Eliot started to tell Steve about his conversation with Cynthia and Bob on Monday, and what he’d done. Steve listened intently for a few minutes and started to ask questions.
“Did you check the contents of the ‘Run’ key?”
Eliot paused. “Uh…no. Why would I do that?”
Steve took a sip of his Coke. “You said that Cynthia had mentioned a virus or something. It sounds more like a trojan or a worm, but either way, those things need a way to be persistent, right? You know, to be running when the box is bounced? So what they do is create an entry in a Registry key so they get started when the box is started. It just happens that the ‘Run’ key is pretty popular.”
Eliot was bemused. He’d read a lot of the reports from anti-virus companies on various bits of malware, but hadn’t really picked up on the Registry issue.
“You might also want to check other places were programs get started automatically,” said Steve.
“Like the StartUp directories,” Steve said with a devilish grin. That made sense, Eliot thought, so why hadn’t he thought of it.
“You said this was telnet activity, right?” asked Steve.
“Yes, that’s what the network manager said. I did see that telnet was running on the system.”
“Did you check the running services?” asked Steve.
“No, I didn’t do that. What’s there?”
“Maybe nothing,” said Steve. “But you’d want to check, just in case that telnet you found is a server and not a client. Since the system is NT, did you check the contents and LastWrite time of the Telnet Registry key?”
Confronted with Eliot’s blank stare, Steve chuckled, and continued. “The telnet client on NT is a GUI, and a list of the last 10 clients and ports connected to are maintained in the Registry. Whenever a new entry is added, or the order is changed, the LastWrite time of the Registry key is updated, much like the last modification time on files. If this time corresponds with one of the attempts to access a router or switch, then you’ll know that the telnet client was used.”
This was all good information that Eliot wished he’d had available before looking at the system. As it was, Eliot felt as if he’d approached the situation with too little knowledge, and no plan whatsoever. As it was, he hadn’t been able to give Cynthia and Bob any real information about what was happening. He’d thought that by not finding any of the files that various viruses use, he’d done a pretty thorough search. As it was, he really hadn’t done a very thorough examination at all. Eliot decided that he’d better get with Steve and compile a list of tools and utilities he could use. And he’d better put together a plan for using them.
To read Part Two of this series, please click here.
No Stone Unturned: Part Two
This article originally appeared on SecurityFocus.com -- reproduction in whole or in part is not allowed without expressed written consent.