Nick Leake, Director of Operations and Infrastructure, ITV plc

Measuring Up

Finding the Right Security Metrics

Every worthwhile experiment relies on an unaltered "control" subject or process for a reality check. So it is with security software. To know how effectively a combination of technologies and policies are guarding against threats, companies need a subgroup of unprotected systems for comparison.

ITV plc, Britain's popular commercial television broadcaster, runs such a control group, albeit not by choice. The vendor of the custom software ITV uses to edit and manage news presentations grew up in the broadcasting industry and remains insensitive to the security fears that haunt most IT organizations. The company advises its customers to tune the application for performance rather than harden it against security threats.

Nick Leake, ITV director of operations and infrastructure, thinks this approach is shortsighted. "We have undertaken a test of installing antivirus software on some of the machines, and we've recognized that the machines function perfectly adequately," he says. "The vendor would be better off to support security measures."

This situation offers one unintended benefit: It demonstrates the virus threats ITV faces and unveils the consequences the company would face if its current strategies weren't in place. It has been demonstrated that when problems arise, they almost always occur on the unprotected PCs. "It's not a control I'm happy with," Leake adds.

Leake isn't alone with his concerns. Security executives crave universally accepted ways to measure and document the effectiveness of their strategies. The right data, after all, can serve to generate answers to such recurring fundamental questions as: "What additional measures should my company be taking to harden its operations?" and "Where should future spending be targeted for hardware, software, consulting labor, and external services?"

Despite their value, valid security metrics remain elusive for many companies like ITV, which often rely on ad-hoc measures.

"The issue for any commercial organization," says Leake, "is how much effort you want to put in to gathering any of these metrics, and what's the value you get back if you do."

Stumbling blocks

Companies that take the time to develop ongoing security statistics can use the results to justify ongoing funding.

"As the chief security officer, you want a tool that is able to gather data quickly and generate decent-looking reports that a manager can then take to the boss and say, 'Look at how effective our security operations are, or alternatively, look at all the problems we've got,'" says Andrew Jaquith, program manager for security research at the Yankee Group, a technology research company based in Boston, Massachusetts. "The classic technique for gathering this information is to work with the security software vendor, or to roll up your sleeves and use your own tools to gather the data information about the problem you are trying to solve."

The incentives for doing this work continue to grow. In the United States alone, 2.8 million organizations suffered at least one security assault resulting in average individual losses of US$24,000, according to the 2005 FBI Computer Crime Survey. The agency calls the statistics "very conservative." Even so, these losses represent US$67.2 billion a year (or US$7.6 million per hour), which equals one-half percent of all the goods and services produced annually by the U.S. economy. The FBI estimates account for only a fraction of the staff, technology, time, and software that organizations commit to preventing security incidents, the agency says.

On the surface, implementing and gathering security metrics seems simple-just track the numbers of incoming and outgoing emails, document the percentage of virus attachments, and record trends in the number of network intrusions attempted by outsiders in recent years. Solid metrics aren't so easy to quantify, however, for individual companies or for industries. This is what makes developing reliable benchmarks so challenging, Jaquith says.