9 Ways to Avoid an Incident Response Disaster

Police departments sometimes have an unusual term for EMTs who rush to an accident scene to help a crime victim – “evidence mangling teams.” No one disputes the need for EMTs to do their critical, life-saving duties. At the same time, they can trample over evidence that could help investigators solve a case. Even an EMT leaving behind a rubber glove can delay investigators, who can never be sure if it was there before the incident.

The same thing can happen in cyber incident response. IT personnel looking to “stop the bleeding” of a cyber attack can destroy critical information that might otherwise help find the intruder or reduce the damage. The way to minimize this problem is to prepare for evidence collection well in advance of an incident. Here are nine steps that companies should take before, during and after a malware attack or other incident.

1. Develop a Playbook. When companies suffer an intrusion, people shouldn’t have to run to someone in the IT department to ask what to do. The procedures and policies for handling an intrusion should clearly lay out the steps that everyone needs to take, so there is no delay or confusion. The plan should be checked at least once a quarter, and preferably once a month. These should be considered living documents and updated regularly to maintain relevance to current practices. Technology changes quickly, and new iterations of technology could require revisions to a response plan.
2. Find a Trusted Partner. Before an attack, you should develop a relationship with a trusted Incident Response partner, who can be ready to help your IT and security personnel deal with an intrusion. Make sure your Trusted Partner is approved under your cyber insurance policy. Some are like health insurance policies that will only reimburse if you go to an approved doctor.
3. Run Mock Exercises. Your Incident Response partner can run you through cyber attack exercises to prepare your team to handle the real thing. In one case, I presented mock exercises of a ransomware attack for a company, even though they insisted “it could never happen” to them. Five minutes into the exercise, one of the IT people received a phone call that told him the company was suffering a ransomware attack at that very moment. Remember: anything can happen to anyone at any time. When it comes to incident response, practice really does make perfect … or at least better.
4. Capture All the Right Information.  When an intrusion occurs, it’s essential to immediately access web logs, proxy logs, authentication logs, and other information that can determine the cause and consequences of the attack. First make sure you are capturing all the information you need. For example, some IT systems have a default setting to collect failed log-ons but not successful log-ons. This can lose vital information if a hacker logs in using the stolen credentials of an actual employee.
5. Centralize Logging. Ideally, all the log information you might need during an incident response should be sent to a central location. Time is of the essence during an incident. You don’t want to search in eight different places where 10 critical pieces of information are stored.
6. Keep Information Accessible. Capturing log information is of little comfort if you can’t access it. Consider a healthcare company that stored its logs at a vendor’s facility. When the company was attacked during the holidays, everyone at the external facility was on vacation. The logs were unavailable for a week. That’s an eternity – a costly eternity -- in incident response. Make sure that you can always access your own logs through a console.
7. Document Your Response Steps. As your IT and security team deal with the intrusion, they should document every step they take and whatever changes they make to company systems. At the same time, while dealing with the intrusion, they should be careful not to cause more harm.  For example, a lot of important information is kept in volatile memory, computing storage like RAM that only maintains its data when the device is powered on.  Something as simple as turning a computer off and back on can destroy valuable data. It’s the IT version of the “evidence mangling team.” This is where having established policies and running the mock exercises pays off by keeping everyone on the same page.
8. Preserve the Chain of Custody. You never know if an intrusion will result in legal action, so evidence needs to be safeguarded.  If you pull an infected server offline, store it in a locked cabinet so you can affirm no one has touched it.   In addition, utilize a Chain of Custody document to capture key information such as a detailed description of the evidence, location and time/date of Seizure, along with relevant custody transfer information between individuals since time of seizure.  In criminal proceedings, the forensic process itself is rarely questioned. However, attorneys will often try to punch holes in the “chain of custody”, which can get the entire case thrown out of court.
9. Track Your Lessons Learned. After you’ve handled an incident, the temptation might be to relax. After all, this is a tense and draining experience. Instead, you should immediately document what you did right and wrong while everything is still fresh in your mind. That way, the inevitable next incident can be less tense and handled even more effectively.