As the dust starts to settle after the impact of the Crowdstrike incident on Friday, July 19 2024, understanding how similar situations can be avoided or mitigated is forefront of mind for organizations large and small.
But with the competing priorities of ensuring robust security while committing to operational resilience, the pressures on IT teams are increasing every day. How can IT teams manage the requirements to deploy security updates quickly while ensuring that processes and systems are robust and reliable?
Security has long been at the forefront of IT decision-making, particularly with regulations around data privacy and ever-increasing cyber threats in the world. But as the failures caused by the roll-out of the Crowdstrike update proved, prioritizing security can lead to operational issues. Crowdstrike described this as a content update, likely leading to clients assessing the update as low-risk and automatically rolling out of the change across their estate. This wave of updates occurred across supply chains and around the globe impacting companies directly and indirectly through their suppliers.
Crowdstrike’s Falcon Sensor and similar security solutions are key in ensuring an IT estate is protected, mitigating cyber-threat risks and securing critical systems and data. The events of July 19 showed us is that when such a system is compromised, the impact is felt across an organization with lasting consequences; this will have led to investigations and evaluations by every organization affected and close monitoring from those not impacted.
Resilience and Stability are key
The key findings that have been reported seem to revolve around how updates are applied and insufficient planning for recovery in case of an incident. While risks from external cyber threats are monitored, tracked and mitigated, the risk of complications from poorly validated updates is of much lower visibility.
The Digital Operational Resilience Act (DORA), which comes into force in January 2025, is focused on ensuring the resilience of financial services organizations and targets internal and third-party risk management, detailed testing of operational resilience and oversight for critical third parties. This regulation only targets larger financial services companies, but the objectives are now critical considerations for all organizations.
So what steps can be taken:
- 1. Use a “Zero Trust” approach to content provided by suppliers.
More vigilant processes can be put in place to ensure that, if any problem arises during an update, issues are captured before release or early in the roll out to minimize the impact. A phased update process adds a level of security risk as it could take longer to deploy a patch, but it ensures that a repeat of the Crowdstrike incident does not bring down your entire Windows infrastructure.
- 2. Understand and validate how quickly operations can be restored in the event of a failure.
I visited an automotive parts store on Saturday, July 20 and was greeted by staff using pen and paper to track their customer interactions and they had no visibility of their inventory and asked me to phone their head office as that was the only mechanism they had to check if they had the required parts in stock! Ensuring that you have a detailed, validated and tested business continuity plan (BCP) in place is critical and having the ability to roll back or restore over an issue is potentially more important.
- 3. Have the right tools in place to properly understand the IT estate and detect a problem with minimal outages.
Monitoring tools can give early warnings of issues and give clear audit around what and when changes occurred. This can be hugely valuable in diagnosing an incident and reducing the time to resolution and restoration of services.
Microsoft have put together a blog with best practices for resiliency which discusses these points further (https://techcommunity.microsoft.com/t5/windows-it-pro-blog/windows-resiliency-best-practices-and-the-path-forward/ba-p/4201550) but planning needs to take into account all areas of the IT landscape and ensure the right monitoring and protection is in place.
Conclusion
The Crowdstrike incident has impacted 8.5 million Windows machines around the world and caused significant knock-on effects in diverse industries in many verticals. Both Crowdstrike and Microsoft have made announcements on how they are going to reduce the risks of this type of incident happening again but IT teams are working hard to put additional measures in place to ensure operations are not impacted in a similar way again.
Rolling out changes, updates and patches is going to become more tightly controlled and monitored and there will be a need for more stringent recovery testing and infrastructure resilience validation but there are some clear measures that can be taken to help mitigate the risks of these changes.
This will mean IT solutions are going to become more stable and secure but the pressures on IT teams to monitor and control their estates are going to be much higher.
