How much faith do you have in your Disaster Recovery plan?

If your answer is “not very much,” join the club. For the most part, IT professionals aren’t completely confident in their ability to recover in an emergency. Further, they view DR testing as costly, disruptive, time-consuming, and rarely comprehensive.

That’s according to Symantec’s latest Disaster Recovery Survey, released in June. The survey polled more than 1,650 IT managers in large organizations around the world.

This article explores the key HA/DR testing issues facing IT organizations today and then examines a new approach to overcoming these problems.

Data center changes impact DR testing

Perhaps more than anything else, the latest Disaster Recovery Survey demonstrates the rising pressures on IT departments caused by soaring downtime costs and more stringent service-level requirements.

The survey found that the average cost of implementing DR plans for each downtime incident worldwide was $287,600. In North America, the median cost can climb to as much as $900,000. That’s disturbing given that one in four DR tests fail, and 93% of organizations had to execute on their DR plans in the past year.

Asked how frequently they carry out full-scenario testing of their disaster recovery plan, 26% of respondents said they do so every six months, while 20% said they do so just once a year.

Not surprisingly, the survey found that DR testing, while considered important, continues to have a negative impact. Asked about barriers to running full-scenario tests on their disaster recovery plan, 40% of respondents indicated a fear of disruption to customers, while 27% indicated a fear of disruption to sales and the revenue stream.

The survey shows that while there are several reasons why DR tests fail, one certainty is that constant change is common to every data center. The production environment changes daily, as storage, servers, databases, and data layouts are added, removed, and reconfigured. However, the DR infrastructure cannot always be “turned on” to check if a change was implemented correctly. Over the course of a year, thousands of changes are applied, but the DR infrastructure isn’t necessarily checked.

To remedy this problem, IT departments conduct annual DR tests. After the test, IT finds that nothing works according to plan because changes at the primary site or infrastructure weren’t implemented at the secondary site or on the target devices. So IT will attempt to remedy all of the problems, elongating the testing cycle, and, when everything is apparently fixed, announce that “The DR Test Was Successful.” In reality, however, IT simply has managed to make the test work.

Generally speaking, IT will take one or more of the following testing approaches: annual DR tests, manual audits, custom scripts, or they will try to follow a set of outdated best practice documents. But given the nature of the ever changing data center, these approaches will fail, since no one can keep maintaining and updating these manual processes forever and account for the thousands of possible gaps that may occur over time.

The need to automate

Symantec believes that disaster recovery testing shouldn’t disrupt business operations. Given that survey respondents pointed to people and processes as two of the main reasons that DR tests fail, it’s also very likely that more automated tools will help address this challenge.

“Too many IT departments are trying to test manually, going from server to server with a clipboard and a checklist,” says Jiwon Youm, Product Manager for Disaster Recovery Advisor at Symantec. “But this is clearly impractical given the growing complexity and scope of today’s data centers. You have to automate testing and monitoring.”

Youm says more organizations today are looking to augment their DR testing methods with solutions that can automatically check for vulnerabilities and identify problems such as configuration “drift” due to routine, day-to-day changes before they impact business operations.

One such solution, Symantec’s Disaster Recovery Advisor, has a knowledgebase of more than 3,000 HA/DR vulnerabilities—nearly anything that can prevent a data center from recovering from an outage. Disaster Recovery Advisor scans heterogeneous environments including storage, databases, servers, and replication configurations for vulnerabilities such as unprotected databases or database partitions, non-compliant replication configurations, and data that can’t be recovered to a valid consistency point.

“Disaster Recovery Advisor will run a scan, analyze the configuration drift issues against the internal gap signature database, and then it will provide a report,” says Youm. “The report includes suggestions for remediation of any issues detected, so we’re not just telling customers they have a problem. We’re also enabling them with information to make the necessary adjustments and corrections to their infrastructure.”

Youm adds that installation of Disaster Recovery Advisor is quick and seamless, as the solution is agentless and requires zero configuration.

Conclusion

Successful Disaster Recovery requires automated, end-to-end scanning, monitoring, and reporting of application, server, and storage configurations, and a flexible way to test DR. With Disaster Recovery Advisor, IT organizations can automatically detect thousands of issues, understand the risk to their applications, servers, and storage, and take action while leveraging their existing IT resources.

Symantec offers a free, two-day DR Assessment. To request an appointment, call your local Symantec sales representative or 1-800-745-6054.

Related Links

• Disaster Recovery Advisor
• White Paper: Five Critical Recovery Flaws Your Last Disaster Recovery Test Missed