Measuring Up (cont.)

Finding the Right Security Metrics

Companies would be best served by collecting accurate data not only on security incidents but also on the existing investment in security technology. Focal points should include how long it takes companies to respond to individual assaults and how much labor they had to devote to the threat.

"That is your real cost," says Jaquith, who goes on to say that time and cost are measures of agility and organizational effectiveness. "And those," he says, "are arguably the best metrics."

Jaquith warns, however, that three challenges traditionally conspire against the rise of universally accepted security metrics: disagreement among security professionals about how to properly model and quantify risks, a lack of consensus on which security attributes to measure, and insufficient data sharing among companies in vertical markets in regard to documenting the numbers of incidents and ongoing trends in terms of the volume and character of attacks.

Short of industry-wide standards, companies can begin to measure security strategy effectiveness by comparing departments across large organizations that don't enforce consistent policies. Variations could build a case for both security spending and policy consistency. "Companies might see, for instance, that a North American business unit has, on average, 11 missing operating system patches on its workstations, whereas the European business unit might have five," Jaquith says. "If you can also show that, lo and behold, one of these units had a lot more security incidents than the other, you could start to draw some connections."

Coping with the unknown

Leake explains the difficulties in devising security metrics by saying, "there are known knowns, known unknowns, and unknown unknowns." The third group is the hardest to cope with, he says. "Once you put a measurement in place, that's only telling you the status of the things that you're measuring. If there are problems that you don't know exist, you can't really put a measurement in place around them," he points out.

Because IT security focuses on prevention, companies consider themselves successful in the absence of incidents. "But when you've got no activity, you're never quite sure whether it's because you've been extremely successful in what you've done or whether you're not being subjected to any attacks," Leake says. "It's very difficult to measure exactly how successful you've been. Having said that, you need to take reasonable precautions to protect yourself from being attacked in any particular area."

For ITV, measuring success comes down to logging downtime or other performance problems resulting from attacks. ITV combines Symantec AntiVirus with an overall security approach that configures most of ITV's PCs with similar processors, software, memory, and related system components for maintenance ease and fewer complications. ITV also uses software that automatically downloads patches from the source sites, which it then tests and rolls out to its PCs. "Within this 'lock down' environment, security is very effective," Leake says. "We get a very low level of incidents."