"Change” was ubiquitous as a campaign theme in last year’s election cycle in the United States. The concept filled the stump speeches and policy papers of members of both major political parties—for incumbents and challengers alike. For those who were elected on the platform of change, the challenge will be not simply supporting change, but rather implementing beneficial change.

Perhaps more than most states, Michigan’s government has exper-ienced frequent change. Hit hard by the decline of the domestic automobile industry and other types of manufacturing, state budget cuts have occurred several times in recent years, prompting agencies to find innovative ways to do more with less.

Information technology has been a part of these mandates, and the key strategy has been consolidation. In 2001, Gov. John Engler issued an executive order creating the Michigan Department of Information Technology (DIT), a cabinet-level agency devoted to serving the technology needs of each of the state’s departments.

Merging IT processes statewide

“The mandate was to consolidate 19 disparate IT organizations,” recalls Ken Theis, who is now the state’s CIO. “The result was that about 2,300 employees were brought together under one organization—plus the HR functions associated with those workers, all IT contracts, the ownership of the hardware and software, procurement processes, and information security.”

The governor had already experimented with a piece of the IT consolidation puzzle with his 2000 initiative to centralize all state Web services under a revamped Michigan.gov portal. “At the time, every agency had its own Web site with its own look and feel,” says Dan Lohrmann, who led this effort before going on to become the state’s CISO. “The idea was to bring everything together and launch a single portal for the state for e-government.”

The creation of the DIT was a similar but much larger undertaking. “The governor’s strategic objective was twofold,” Theis asserts. “One was to bring efficiencies in doing technology across the 19 agencies, but another was effectiveness. He thought that bringing things together would not only formalize our standard architecture and standardize our processes, but ultimately would result in technology that better supported the goals and objectives of his administration.”

Early challenges

By the time DIT was launched in early 2002, Gov. Engler was less than a year from the end of his final term in office and campaigning was beginning in earnest for the fall election. “There was a significant effort to ensure that we were far enough along that the agency would survive the transition to a new administration,” remembers Patrick Hale, who is now the state’s CTO.

This urgency meant that “time was not taken to properly plan the organizational structure, the key methodologies, and processes,” Theis relates. “This created great anxiety—not only with our 2,300 employees, but most importantly, with our 19 client agencies. Many probably hoped that the new governor would go back to the old model.”

The quick transition also exacerbated problems in supporting the agency’s newly consolidated but very heterogeneous infrastructure. “We centralized before we standardized, and that became a true challenge,” Theis states. “For example, technicians were suddenly fixing PCs in other agencies where they had no understanding of the technical environment and little documentation that they could consult.”

About the same time, the state initiated an early retirement incentive, and more employees than expected took advantage of it. “We lost 320 of our employees—almost 20 percent of our workforce—and were not able to replace them,” Theis recalls.

Getting strategic

Jennifer Granholm won the 2002 election, and despite the change in political parties in the governor’s mansion, “she expressed full support for the overall model very early in her administration, though she also made the commitment to address the real concerns that had been raised by our employees and our client agencies,” says Theis.

Once the new administration signaled its support, the DIT team began assessing how to move forward. “We got a little bit of room to breathe,” Theis recalls, “and we asked, ‘Strategically, how are we going to deal with this?’”

As a result, the change management and strategic planning processes that had been deferred were now able to proceed. These discussions resulted in several key initiatives, including the Secure Michigan initiative and the stand-ardization and consolidation effort known as Michigan One.

Securing Michigan

Lohrmann, who worked for six years at the National Security Agency and has written a book and many articles and blog posts on IT security, led Secure Michigan. He worked with Hale and his team to build security into the IT infrastructure as it was being designed.

“We had to look at things much more as an enterprise, and we had to do a lot of very basic things to get there,” Lohrmann explains. “For example, we had 19 different security policies around acceptable use. We had to formulate to an enterprise-wide PC policy. And we had to make sure that the policy, and the technologies that support it, would still work after everything was consolidated.”

The business needs of the state agencies also had to be considered. “I had a big challenge to be an ‘enabler’ rather than a ‘disabler’ from a security perspective—not just turning things off, but figuring out how to allow them and secure them, Lohrmann says.”

Standardizing security software

One of the first solutions Lohrmann standardized on was Symantec AntiVirus. “It’s been a phenomenal product,” Lohrmann asserts, “and it’s not just the product. When we have state-wide security issues, we need people on the other end of the phone who not only have a global view, but also can zero in on our issues. Symantec has both.”

The state is currently in the process of upgrading to Symantec Endpoint Protection. “With so many people connecting at restaurants and airports, we know that the comprehensive security technologies of Endpoint Protection are the way to go,” Lohrmann says.

The state relies on Symantec Critical System Protection to protect against intrusions on critical servers, and Symantec Security Information Manager to correlate log data from across the state’s systems to provide custom alerts and reports on the state’s security landscape. As part of this implementation, Symantec Consulting Services designed custom data collectors for some of the state’s infrastructure.

For monitoring and reporting on compliance with regulations and standards, Lohrmann’s team has relied on Symantec Enterprise Security Manager for several years, which is now integrated with Symantec Control Compliance Suite 9.0. “It would be impossible for my staff to keep up with reporting and compliance remediation if we didn’t have Symantec’s automation tools,” Lohrmann contends, “and I’m looking forward to using the enhanced features of the integrated product.”

Consolidating the infrastructure

While Lohrmann was standardizing security operations, Hale was busy helping to plan and implement Michigan One—first as a consultant, and later as a state employee. “The first year or two,” he remembers, “Michigan One was heavily focused on securing our network. But even in that first year, there was work underway to re-architect things. We wanted to make sure that our infrastructure would support our enterprise vision.”

“We’ve got 1,400 remote offices,” Hale explains, “and some are located in rural geography with very limited connection capabilities. We had to deal with that infrastructure before we could lay on top of that a standardized solution.”

In 2004, Hale’s team began moving systems into consolidated data centers. “As we did so,” Hale relates, “we began to find architectural details that we didn’t like, and we had to shut down some systems. This made some things inconvenient for state users. However, the risk that was unknowingly being taken outweighed the benefits.”

Nurturing relationships

At the time, Theis was DIT’s agency services deputy director, responsible for maintaining lines of communication with client agencies. As Hale and Lohrmann were making these difficult transitions, “my focus was to repair these fractured relationships. I needed to make sure our organization was responsive to the needs of the business.”

As functions were disabled or changed as a part of the consolidation process, Theis met with affected agency directors to find appropriate workarounds or process changes. “In retrospect, Ken’s role was key at the time,” Lohrmann asserts. “We could not have pulled things off as quickly or smoothly without the buy-in that he was negotiating.”

An emergency with backups

In 2005, Hale’s team accelerated its consolidation efforts, closing 32 data centers in the Lansing area alone and consolidating them into three centralized centers. “At the time, we also brought in a number of legacy backup solutions,” Hale recalls. “As a result, in late 2005, we started to see backup success percentages getting into the low 70s. That’s a whole lot of backups failing every night, literally into the hundreds.”

Due to its reliability and compatibility with a variety of systems, Veritas NetBackup had already been selected as the state’s backup standard. With failed backups putting the state’s two petabytes of data in jeopardy, Hale’s team engaged Symantec Consulting Services to expedite implementation of NetBackup across the enterprise. In addition, a new SAN solution from Symantec Partner EMC was deployed concurrently.

“Today, we’re successfully executing 21,000 backup jobs a week,” Hale reports. “We have now gone almost a year since we have had any backup fail for multiple nights, and our backup success rate is now at 98 percent. Further, our restore rate with NetBackup is 100 percent. The product’s scalability and ease of use allowed us to save about $250,000 annually in backup administrator time.”

Well-deserved recognition

While a number of states have undertaken IT consolidation projects in recent years, Michigan was among the first, and the team has received many awards over the years for its efforts. The state received three awards for excellence in information technology at the National Association of State Chief Information Officers (NASCIO) last September—with awards for the Michigan.gov Web site, the data center migration project, and the information security and privacy project.

Regarding efficiency, the numbers speak for themselves. DIT’s workforce today is just over half of what it was in 2002. “When we consolidated,” Theis explains, “we had around 2,300 employees and 2,300 contractors. Today, we have 1,700 employees and 800 contractors.”

Yet everyone involved would assert that these efficiencies were accompanied by a significant increase in the breadth and quality of IT services over the past seven years. “Ironically, IT is tied more strategically to the business needs of each agency today than when each department had its own IT shop,” Hale quips.

A place at the table

“When governors put their strategic plans together,” Theis explains, “IT organizations usually struggle to even get into the room. Our organization actually helps the governor facilitate that process through our 19 client agencies. It gives us tremendous insight into the critical capabilities, goals, and objectives of the organization—and it helps us align our organization toward helping them achieve those objectives.”

“I think we were very fortunate,” Theis concludes. “Both governors really saw the vision of how can we better utilize technology to transform state government. I think that’s why Gov. Engler was so passionate about putting it in place before he left. Then, Gov. Granholm understood the true value of IT as a strategic tool to help her accomplish the things that are important to her administration.”

And those are changes that you can believe in.

Mark L.S. Mullins is a managing editor of CIO Digest and manager of Symantec’s Global Customer Reference Program team.