Say “Aloha” to Hawaii

I had big plans to spend Spring Break with the family. We had reserved a townhouse on the western beach of the "Big Island" of Hawaii, and I was looking forward to seven days of some very well-deserved time with Eugene, Jr. (Junior) and my wife. My wife had plans to go shopping and to the spa while I spent some time playing several rounds of golf with Junior. We also had a day of snorkeling and two days hiking through some of the interior of the island.

Hawaii Detour from "Berkeley"

The night before our flight to Honolulu, however, I received a call from "Berkeley". ("Berkeley" is one of my security specialists. We call him "Berkeley" because he looks like a 60's Berkeley hippie—hair down to his waist, drives an electric car that he built himself with nearly 100 bumper stickers plastered across it.) Anyway, "Berkeley" indicated ("Hey dude, you'd better come down here. There are some really wild things happening with some of our systems. It's like some of them are behaving like Dewey Cox—they're really freaking out!) that the systems were behaving oddly; certain backups were failing and select systems were slowing to a crawl.

The worst news he gave me: among the "freaked out" systems were the servers running our Oracle Financials applications. With payroll scheduled for Monday morning, this had the potential of becoming a big tsunami. Not only would the "Warlord" (our fearless CEO) go on the warpath if payroll didn't happen on time, but the entire company would join him; the ensuing "hand-to-hand combat" would leave a path of destruction directly through my department.

The Dewey Cox "Trip"

When I arrived at the data center, I discovered "Berkeley" was—unfortunately—not simply on a "Dewey Cox trip" (which was my hope). A quick diagnosis showed we had at least 50 infected machines, and another 250 or so desktops and laptops were exhibiting varying degrees of problems.

I thus made two phone calls. The first was an all-hands-on-deck call to my team. The second was to my wife. Neither was very pretty. Several members of my team were out enjoying the weekend with spouses, boyfriends, and girlfriends—and spending the weekend in the data center was not on the top of their list of things to do. My wife's reaction paralleled the data center environment; cancellation or postponement of our Hawaii vacation was not a word in her vocabulary—as well as aloha—and the "Dewey Cox" experience with her was not something I plan to replicate again.

The "Patman" Revisited

After several hours of diagnosis, we discovered the cause of the outbreak. While we had cleaned up the mess caused by the "Patman's" Patriot's screen saver fumble after the Super Bowl, the virus had gotten backed up and then replicated when the "Patman" requested a restore of his Exchange mailbox after he accidentally deleted his PST files.

The team did some heavy-lifting over the weekend, and we somehow had the system virus-free and, in particular, ready for an 8:00 a.m. payroll run on Monday. I made it home around 9:00 a.m. and while my wife was still livid, she did admit a good laugh when I presented her with one of the Patriot's hula dolls from the "Patman".

Say aloha to Hawaii. Later…

Hey Eugene, for all of the IT professionals who have postponed or cancelled their vacations due to last-minute IT failures, we feel your pain. However, you could have prevented the incident and saved your vacation if you had upgraded your backup-and-restore environment to Symantec Backup Exec 12. With its integration with Symantec Endpoint Protection 11.0, you have proactive Symantec ThreatCon backups.

Message Edited by Turlas on 04-14-2008 09:49 PM