Clear and Hold: The 4 Stages of Incident Remediation

As security practitioners, we often spend time and energy motivating the organization to act.

On the rare occasion an organization is faced with widespread compromise from a targeted attacker, that thinking needs to be flipped on its head. Act too fast, and a persistent and skilled attacker can re-compromise an environment shortly after they were detected and expelled.  Or it might help them avoid being expelled to begin with.

When you’re the guy accountable for leading a remediation – as I’ve been across several recent roles – you need to take a more cautious approach.

Clear and Hold

Remediation is the final stage of an incident response process. It can involve everything from an enterprise-wide password reset to pulling a network cable and rebuilding an infected box.

The military term “clear and hold” is a good analogy for understanding remediation and its importance.  A counter-insurgency tactic, “clear and hold” involves clearing an area of enemies and holding it to prevent those enemies from reoccupying. Successful execution of this phase is considered central to long-term strategic objectives.

In the same way, when we’re remediating in a security context, we’re trying to remove an attacker’s presence from our environment and neutralize any mechanisms they can use to re-compromise.

This typically follows these four phases:

• Prepare
  When it comes to remediation, the saying could not be truer: ‘If you fail to prepare then prepare to fail’. Successful remediation is grounded in preparation, where we consolidate everything we understand about the attacker, as well as prioritizing activities to be performed immediately versus later. Sometimes, this a good place for a reality check – based on the information gathered about the extent of compromise, we may need to acknowledge (and communicate) that large gaps still exist in what we understand about the attacker, and that the event cannot be fully contained.

• Execute
  In this phase we execute activities to expel the attacker from the environment – such as isolating infected machines, resetting accounts and blacklisting domains. Most often, a “shock and awe” approach gives an organization the greatest chance of successfully ejecting an attacker from the environment. The attacker unexpectedly loses network connectivity, credentials and tools in an aggressive, coordinated event. These activities are often performed at times when the attacker has demonstrated that they are less likely to be active. 
  
  There may be situations where the activities needed to remove an attacker’s access to the environment have to be staggered over a longer period of time. This gives attackers a larger window of opportunity to circumvent remediation, placing greater emphasis on your organization’s detection capabilities and understanding of their Tactics, Techniques and Procedures (TTPs).
   
• Verify
  In this phase we make sure all activities carried out during containment and eradication were successful. As we’ll discuss later, this is often overlooked.
   
• Future enhancements
  This final phase is where you implement measures to improve the organization’s longer-term security posture. While preparing for remediation, we identify security gaps exposed by an attacker and opportunities for improvement. Activities that cannot be carried out in the narrow timeframe before or during the eradication of the attacker are addressed longer term as strategic projects. For example, implementing a new access control or multi-factor authentication.

Common Challenges

These four steps are reasonably well-known among remediation teams in the industry. But we still see many organizations stall or fail to successfully remediate widespread compromise by a targeted, persistent attacker. In our experience, this can be traced back to some common problems:

Misunderstanding the threat:
Organizations can underestimate the severity of a threat, and subsequently put inadequate time and energy towards remediation efforts. NotPetya was good example – it was an attack aimed at just one nation but ultimately affected multiple industries in countries across the globe. You pay dearly for underestimating the threat and accepting a risk you haven’t fully understood.

Mis-timing:
The most common mistake we see is poor timing of remediation activities, particularly when the execution phase commences before an organization has successfully prepared.

Sure, when you discover attackers have been on your network for years, it’s hard to suppress the temptation to act. But without sufficient planning first, you won’t understand the full extent of the attacker’s presence in your environment, nor will you be prepared to execute all the necessary remediation activities in a deliberate, coordinated manner. Either way, attackers will become aware of your efforts, leading them to become destructive, “go dark” by burrowing deeper into your network, or leave previously unseen backdoors to enable re-compromise.

Mis-timing can also occur when remediation activities are deferred to a later date as strategic enhancements, especially where they relate to gaps that are being actively exploited.

Top-level support:
Getting the timing right is that much harder without support and understanding from those above.

In the wake of a detection, boards and senior executives will naturally press for fast containment action. In response, security leaders need to be able to gather and communicate a full understanding of the threat and explain the risks and consequences of premature mitigation.

We’ve found regularly running table-top exercises – with senior executives and other teams involved in incident response – to be invaluable. These drills can iron out incorrect assumptions and create a common understanding of key factors ahead of time.

Exercises are also useful in getting support from other teams that manage applications and processes relevant to the remediation process, such as network and IT teams.

Failure to verify:
Failure to adequately test and verify is common in organizations that haven’t had much experience with widespread compromise. Before closing an incident, ask yourself:

• Have all compromised accounts been successfully reset?
• Are all indicators of compromise being detected and blocked?

By asking questions such as these, we not only limit the likelihood of re-compromise, but put ourselves in a position to have a more confident conversation with our board and executives.

As an example, we might place a benign file in different locations across our network and then create detection logic for the file’s hash in security tools such as Symantec Advanced Threat Protection and Cloud Workload Protection. By testing and confirming that all instances of the files have been detected, we can better communicate confidence in our visibility across the network and our ability to detect the attackers should they return.

Staying Connected

Many of the positive outcomes we get at Symantec come from the way both our products and our teams work together.

A key to our success as a remediation team is being well informed about the tradecraft of attackers targeting our industry and organization. Our internal threat intelligence teams help us understand this with regular briefings, while we also make use of intelligence feeds including DeepSight and MATI. Frameworks like the MITRE ATT&CK matrix are also great at mapping out the techniques used by attackers and shine a light not only on how they seek to compromise but how they might respond to discovery.

Connecting to our staff across the globe also helps. They can be our eyes and ears in areas where we might have technical visibility gaps. Our security awareness team has worked hard to encourage staff to report strange files or phishing emails that could represent a targeted attacker – which becomes invaluable information during any response.