|
This story is true; only the names have been omitted to protect the (sort of) innocent.
Monday, 7:15 AM: I log onto my Solaris box and start the day's regimen. After scanning my 245 email messages for anything that might require immediate attention, I settle in to do some log surfing. As I pull down the first log, I notice that the Perfmeter session I had running to monitor a remote Sun enterprise server has suddenly coughed up little "RIP" icons for each of the system parameters I was tracking. This can't be a Good Thing (TM).
7:20 AM: My rlogin session to the box is hosed, also. Fortunately, telnet still works. I check the procs and discover that rstatd itself is RIP. Red light, red light. I decide to restart inetd and see what happens. In retrospect, this probably wasn't the best diagnostic approach. The box heads south, taking my telnet connection with it. Houston, we have a problem.
7:35 AM: I manage to track down my backup sysadmin on site with the Sun box. She heads obediently down to the computer room to take a look in the patient's mouth for me. I start feeding her commands and she relays the results. We are a team. Go, team.
8:00 AM: By now I'm rather puzzled. The box came back up, but a slew of nonstandard warnings in /var/adm/messages tells me that something ain't right with RPC. This, and the fact that none of the remote services are working. I take a look at the rpcbind binary. It's way too small. In fact, it's 0 bytes. Warning, we have reached DefCon 4.
8:15 AM: There's something rotten in Denmark, and it isn't two week old cod. Scanning through some files I don't recognize, I see a reference to /dev/ttyp. My trusty teammate scoots on over to /dev, and does a 'file ttyp' for me. Budda bing, budda boom; it's a text file. Not your run of the mill tty device, is it? The contents of this file are a veritable who's who of Words You Don't Want To See on Your Computer, including references to 'eggdrop,' 'smurf,' and 'zap3.' I'm appalled. I tell my assistant to yank the twisted pair cable out of the NIC and guard the console with her life, or at least with a baseball bat. I'll be there as soon as I can.
4:10 PM: I'm sitting on a flight headed for ground zero, wishing I'd gone into real estate or office supplies instead of computer security. I hate flying, but what I hate more is flying to the scene of a cracking. I have an inkling of what the NTSB investigators must feel on their way to some horrific air crash. I jam the brake on this line of thought before it sends me into a convulsion and order some tomato juice and a Prozac. They're out of Prozac. Just my luck. Next flight I think I'll bring a mallet and just club myself unconscious.
7:15 PM: I check into the motel and head for the site. Being the Information Systems Security Officer, I only have to bribe 2 of the 4 guards to let me in the building. Fortunately, as a seasoned road warrior I'm never without a bag of soft-baked chocolate chip cookies.
11:30 PM: With a laptop full of Files that Shouldn't Have Been on My System and a notebook brimming with questionably decipherable scribbles, I head tiredly back to the motel to get a little sleep before the Big Day. It's hard for me to sleep in motels anyway, and the knowledge of what I've got to face tomorrow doesn't exactly give me a dedicated line to the sandman.
Tuesday, 6:30 AM: Sitting at the server console with a *very* large mug of coffee, I try to unravel just what has befallen this hapless machine. The first thing to do, I guess, is to look for all files created/modified in the past 5 days or so, just to be on the safe side. This spits out quite a slew, which have to be picked through carefully, looking for the bad guy's fingerprints.
8:15 AM: By now I've shuffled the recent files into harmless/routine and suspicious piles. The crown jewel of my collection is 'setup.sh,' which I found in a hidden directory called '/dev/...'. Looks like my inadvertent reboot while trying to restart inetd kicked the widdy biddy scwipt kiddie off before he could clean up. What a shame. I thank whatever higher powers there may be that I was running something that monitored rstatd.
The setup script is a veritable roadmap to sysadmin hell: unset history files, nuke logs, replace netstat, ls, ps, rpcbind, lpsched, and login with custom versions, install pico, zap the user IDs used to install this mess, and finally kill inetd and sadmind. A pretty thorough hosing. Fortunately, having the install instructions will make cleanup a little easier.
4:00 PM: After a total of 16 hours sitting at this #$@! console, I've finally cleaned out every last foreign file, replaced all broken binaries from the install disk, reconstructed the logs as well as I can manage, and restored the system to the point it was just before the breakin. Now I've got a plane to catch.
Wednesday, 7:00 AM: Sitting back at my desk, I start the painful and tedious post-mortem. Best I can figure, entry was gained through an rpcbind vulnerability that hasn't even been officially patched yet. Aren't I the trendsetter. Also looks like I was rooted automatically. Entry was gained at 5:14 AM local time, but the first human activity doesn't seem to have taken place until about two hours before I closed the door. Even script kiddies need their beauty rest, I suppose.
Found a whoooole bunch of IP addresses in a couple of files that supposedly run IRC servers. Scanning them, I see several subnets I recognize that should definitely *not* be doing the chat thing. Guess I'd better run 'em down, one at a time, and see what there is to see. That oughta keep me busy for about a week.
1:00 PM: Back from lunch, I sit in front of a daunting blank screen and try to screw myself up mentally for the onerous task of generating a formal incident report. As I sit there feeling for the moment unequal to the task, it suddenly occurs to me that I should do my civic duty and alert the folks on the Incidents mailing list. Pleased at having found a way to put off writing the formal report while still getting something useful done, I throw myself enthusiastically into the job. By a judicious application of slow typing, meticulous attention to the mechanics of the written form of my native language, and a generous dollop of tangential Web searching, I manage to pass the balance of the afternoon in this fashion.
Thursday, 7:00 AM: Having posted my alert late yesterday afternoon, I am now of course obligated to review and reply to any and all comments it elicited from the other list members, even the ones that seem to have almost no bearing on the gist of my post. This is good for another couple of carefree, not-writing-the-formal-report hours. Eventually, of course, I have to face the bitter facts and crank the thing out. The problem with this report-writing business is that I have to admit in hard, cold, black and white print that someone with potentially the computer knowledge of a moderately intelligent chimpanzee compromised my system. This is a bit of an ego-cruncher, and it does absolutely nothing to improve my disposition or in fact my opinion of the human race in general (although it does give me a healthy respect for chimpanzees).
3:15 PM: There, I've done it. Written and sent off the report. I have bared my inadequate soul to the world. Bereft of any shred of remaining professional dignity, I flounder helplessly in a sea of...oh, never mind. Let's just say that I was majorly bummed. Home is looking mighty good right about now.
Summary for busy computer security practitioners: Exploits bad. Kiddies bad. Patches good.
Robert G. Ferrell, CISSP, is the Information Systems Security Officer for the National Business Center of the U.S. Dept. of the Interior. He is also active as a Perl Monger, an Internet Technologist, and a humor columnist. He has been involved with (primarily Unix) systems programming, administration, and security on and off since 1977.
|