It’s a bad day when you’ve got a severe security incident to respond to. But the difference between a bad day and a disastrous one can be the quality of the response plan you’ve built. You did build a plan, didn’t you? Here are some key points you may have overlooked.
1. Have a Workable Plan
Surely, most large organizations have a well-thought out incident response plan in place, right? You’d think so, but the Ponemon Institute surveyed 623 companies in 2015, two-thirds of which had headcounts of more than 1,000 people. Of those organizations, 60% say they either have no incident response plan or an “ad hoc” plan; only 17 percent said they had a well-laid-out plan across their entire enterprise. That’s a heck of a thing, and downright scary when you think about it.
“Sort of having an idea” of how your organization will respond to a serious incident is simply not enough. If your organization doesn’t currently have a solid, formalized plan for how to respond to critical incidents, the first step is to put a good one together.
2. Define an “Incident”
As strange as it may sound, the first step in building an effective incident response plan is recognizing what actually constitutes an “incident”, then categorizing incidents by type and severity. For instance, you might have random scanning against your firewalls for open ports. Or you might have someone actively attempting to get into your network. Or maybe they've managed to get access to a system, and now they're attempting to access a repository of PII. Or perhaps you wake up to find ransomware has taken key data hostage. Just as each situation here is different, each requires a different level of response.
As part of your response plan, you need to define and categorize incident types. These definitions directly affect what your planned response will be. What is the severity and type of incident you are looking at? Once you’ve put some definition around what it is you’re dealing with, you can then determine the appropriate level of response. That's stuff that should be inside an incident response plan so that whenever people are using the plan, your organization has guidance as to how to appropriately escalate incidents, and at what point you need to activate the incident response team.
3. Keep the Plan (and Supporting Documentation) Up to Date
Whenever an organization hasn't really run through their plan in a while, they'll often find basic items like the phone lists are out of date, as people have left, or moved, or been promoted. Without regular updates, you may think you've got all that information at your fingertips, but when it comes time to activate your plan, you may find an absolutely outdated mess.
And it’s not true about just people. Some organizations are a disaster at asset management, documenting their networks, and standardizing policy among different units. That's especially common with M&A activity. Whenever you see new units come in through mergers and acquisitions, usually it takes a good long while (sometimes years!) for the network and the network security policies of the parent organization to get aligned with those of the company they’ve bought. Regular updates to your network documentation and incident response plan can go a long way to minimizing confusion when it’s time to use it.
4. Don’t Just Have a Plan. Test it
Of the organizations that did have an incident response plan, over a third don't actually do anything with the plan after they have it; it's basically done as a “check-the-box” exercise to meet a requirement, then sits and gathers dust. As a result, you end up with a plan that really hasn't been tested, and that’s never adapted to operational realities and organizational changes.
In some ways, that’s more dangerous than having no plan at all. If you have no plan at all, at least you know you have no plan. But if you have a plan that hasn't been tested, and isn't reiterated and refined, you may have a false sense of security thinking you’ve got a good working plan, when the truth is you probably don't.
One way to know if your plan is any good is to actually experience a breach, which is a fantastic way to learn, but a really costly, painful way to do so. A much less painful way to do it is to do tabletop exercises. What's great about tabletop exercises is they let you test how your organization responds to a major incident, and how well the various components in the organization are working together, all without the costs and associated panic of an actual breach.
5. Have the Right People Testing the Plan
When you’re doing your drills, you want your core information security team members as part of it, of course, but it needs to be much larger than that. There is a role for senior leadership, and public relations or corporate communications play a massive role. Legal should be also represented. Additionally, the information technology folks (distinct from the information security types) definitely have a role in those tabletop exercises.
So, do any third parties or any partners that are going to be important to an actual incident or an actual breach scenario. Sometimes, some folks will work in law enforcement contacts. If these are people you're going to engage if you have an actual significant event, then it's probably good to have them as part of the tabletop exercise in order to test those lines of communication.
There's also value in just getting to know some of the people that you would be dealing with in a crisis that you may not deal with on a daily basis. For instance, information security generally doesn’t have daily touchpoints with legal or corporate communications. When something does hit and you're dealing with relative strangers, it’s harder to work together quickly. It's one thing if I have to go find a point of contact with Legal in an emergency, as opposed to picking up the phone and calling the exact person I worked with on a drill six months ago. I know who that person is, and she knows me. It makes for swifter communications and a better working relationship.
6. Test Often, but Keep it in Perspective
You should be running a drill at least annually, and probably more often in large organizations.
That’s a big commitment, but there are ways to make it less painful. The way we do them is typically two consecutive days of tabletop exercises, but only for four-hour blocks each day. Any longer, and people just get burned out, and attention spans are strained. It's also much easier to get a block of time on someone's calendar than it is to get full days of their time, especially with executives and leadership. By doing this, you make it possible for people to actually be able to take the time to participate and to be fully engaged.
In a scenario, you may need a few decisions made at the executive level, and you can have a designated point person who's going to handle that decision-making for the purposes of the exercise. That actually gives people practice in briefing up the chain and succinctly explaining to leadership what they need to know so they can get the ball rolling.
7. Have a Post-Incident Action Plan
You’ve put out the fire? Great. Now take some time, document what went right and what went wrong, and incorporate that hard-won knowledge into the plan for next time. I don't know anybody who doesn't think lessons learned are critical things to do, but unless you formalize it as part of your process, it tends to become a lower priority than all the stuff that’s backed up due to having to deal with the incident. Everybody was busy before. Now, they're doubly busy, because everything's been knocked off of schedule.
“Lessons learned” sounds like a nice idea you'll get around to eventually, but unless you make it a priority, it often falls by the wayside. Instead, it needs to be something done in very short order after recovery is complete. It should be part of your post-incident process every single time.
Building and testing a well-oiled incident response plan is hard work, no question about it, but it’s cheaper and easier than panicking when something goes wrong. You won’t be able to prevent every incident, but having a response plan you can count on will allow you to minimize the negative impact of those incidents, and ultimately save your organization’s time, money, and reputation.
[Editor's note: This blog was originally published on 11/07/2017]