It was supposed to be an uneventful—yet productive—weekend. My NY Giants had earned a bye—and home-field advantage—for the playoffs. Many employees had taken the week off between Christmas and New Year’s Day.
The one exception was the Patman who was in a state of melancholy as a result of the Patriots’ failure to make the playoffs. After his 11-5 Patriots missed the playoffs—the first 11-5 team to do so since the 1985 Denver Broncos—the Patman spent the weekend blogging on the iniquities of the current playoff system. His venting included a 10-page tome—stored on a special USB drive with Patriots insignia—to NFL Commissioner Roger Goodell. I’ll get back to the Patman in a moment, since he is the reason for this blog entry.
Server migration, virtualized blade architecture
As part of our effort to reduce our data center footprint and power consumption, we had embarked upon a three-month server consolidation initiative in October focused on migrating select UNIX-based systems over to a virtualized Microsoft Windows-based blade architecture. The projected results were exciting: a 60 percent reduction in our server footprint, a 50 percent improvement in CPU utilization, and a 65 percent decrease in hardware, software, and maintenance costs. The C-Man had assigned the project to my team and given me the mandate to complete the migration by January 1.
Nearly everyone on the team was excited about the project, despite the crimp it put in everyone’s holiday plans. In addition to job security, something particularly compelling in today’s current economic environment, the team welcomed the chance to learn a new technology and to participate in an initiative that would directly contribute to the bottom line. We had made good progress by the time Christmas arrived, and I was able to give the entire team Christmas Eve and Christmas Day off. When we returned to the office on the 26th, we had an estimated two days of work left—and many on the team were making plans to turn the coming post-New Year’s Day weekend into a five-day weekend.
Patman gets free lunch, syndicates tome
The one non-IT employee in the office on the 26th was the Patman. He should have been able to take the week between Christmas and New Year’s Day off, but he had taken three days off the week before Christmas to work on his Patriots blog and tome to Roger Goodell. And as his boss had given him the task of completing a pile of paper work before the end of the year, he had no choice but to come into the office. He was pleased to discover that he could get a free lunch; wanting to keep the team focused on completing the project, I had arranged to have lunch catered every day.
It was over lunch that the Patman proudly told the team that he had loaded his tome to Roger Goodell onto his Microsoft Office SharePoint site. He encouraged the entire team to download and check it out. I wasn’t at the luncheon roundtable, as I had to attend a status meeting with the C-Man, or I may have had some cause for concern—not only a violation of corporate policies but the potential security risk.
“Houston, we have a problem!”
Berkeley alerted me around 5 PM that some on the team were experiencing system issues. The project documentation on the SharePoint site had disappeared and the performance of certain systems was quickly deteriorating. After a quick look, I ascertained that we had a serious problem and pulled everyone into a 6 PM meeting. Priorities had shifted; the task was to pinpoint the cause of the problems, confine it, and remediate it—no further work on the migration project could take place until this occurred.
Five days later we were reimaging the final machine affected by the malicious code outbreak. We finally identified the cause late Sunday evening; the tome the Patman had loaded—using his personal USB drive—contained a new WORM. The Patman arrived Monday morning in good spirits, having enjoyed a weekend watching the NFL Wildcard games with his mother. When he arrived for his “free” lunch, he found a vacant room; my team had relocated the luncheon to another building. When he returned to his office space, he discovered a pile of dirty dishes strewn across his desk.
New Year’s Eve and Day—and weekend—cancelled
With the threat contained and remediated, it was time to recommence work on the migration project. With flights and hotel reservations cancelled and angry spouses, girlfriends, and boyfriends, the team reconvened at 10 PM New Year’s Eve to plot the completion of the project. We put together a detailed plan that demarcated a release Sunday evening at 6 PM—and we succeeded in achieving the target.
It’s now 6:01 PM, and I just sent an email to the executive staff, including the C-Man, to let them know we are live and fully operational. Perhaps, if I hurry, I can watch the highlights of the Wildcard Games on ESPN and find out which team the Giants will be playing next week.
I wonder if Roger Goodell has network access control for USB drives on his desktop and laptop…?
Eugene, a comprehensive endpoint security and management solution, including network access control, would have preventing this malicious code intrusion. Symantec Endpoint Protection combined with Symantec Network Access Control includes antivirus, antispyware, firewall, intrusion prevention, and application and device (including USB drives) control.