Navigating the Storm: Lessons from the CrowdStrike Outage
(Cary, NC) The recent CrowdStrike outage serves as a critical lesson for the cybersecurity and IT community at large. CrowdStrike, a leader in cloud-delivered endpoint and cloud-based workload protection, faced a significant challenge when a problematic update led to widespread system disruptions for its users. This incident underlines the complexities and potential risks inherent in maintaining robust cybersecurity defenses, and provides a roadmap for how businesses manage outages going forward.
Overview of the Outage
The trouble began with a routine software update intended to enhance security features. However, this update contained flawed code that inadvertently caused Windows to experience a Blue Screen of Death (BSOD) on any system that received the update. As systems went down, companies found themselves unable to access vital data and services, leading to operational disruptions and increased vulnerability.
The immediate impact of the outage was profound. Businesses of all sizes experienced downtime, and the ripple effects on productivity and security were felt widely. All said and done, approximately 8.5 million devices were affected by the bad code, and Fortune 500 companies are estimated to have suffered upwards of $5.4 billion in losses. The outage, while well-intentioned and not malicious in origin, spurred opportunistic attacks. Cybercriminals swiftly exploited the situation, deploying deceptive websites to distribute harmful “updates” and causing even more chaos, severely disrupting the supply chain and causing widespread concern and even panic in some cases.
"In recent times, we've witnessed the profound impact of supply chain vulnerabilities in cybersecurity incidents, reminding us of the critical need for comprehensive cybersecurity training,” says Brian Olliff, Cybersecurity Instructor at INE Security, a global leader in cybersecurity training and certifications. Olliff points to the SolarWinds incident as another stark and costly example of seemingly benign software updates turning malicious and compromising countless systems. “These incidents underscore a crucial point: proper cybersecurity training can empower both administrators and end users to detect and thwart such attacks effectively. Cyber teams must be equipped with the knowledge to navigate and secure against these intricate threats, ensuring they're not just participants in cybersecurity, but proactive defenders within their organizations."
The Importance of Swift Deployment and Vigilance
Despite the fallout, the decision by companies to deploy the CrowdStrike update was fundamentally correct. In the fast-evolving landscape of cyber threats, keeping software up-to-date is crucial. Updates often include patches for vulnerabilities that, if left unaddressed, could expose organizations to data breaches and attacks. The issue in this instance was not the policy of regular updates but rather the unforeseen errors within the update itself.
Mitigating the Damage Through Manual Resets
When the systems went down, the immediate solution involved manually rebooting Windows into Safe Mode, deleting a file, and rebooting to restore functionality. One of the complications from this process involved systems with BitLocker enabled, as fixing the CrowdStrike issue triggered BitLocker's recovery mode, requiring users to enter a lengthy recovery key. For many businesses, these keys were stored on servers that were also affected by the outage, making it difficult to access them and prolonging the recovery process.
“This outage highlights the critical need for robust incident response strategies,” says Brian McGahan, who is a CCIE in Security and Director of Networking Content at INE Security and 4 X CCIE. “Manual resets are a stopgap measure, not a solution. They involve significant labor and can lead to further issues if not performed correctly. Organizations that had clearly defined procedures and well-trained IT staff were able to minimize downtime more effectively compared to those less prepared.”
The emphasis going forward, McGahan says, must be on both preventing such outages and preparing to manage them efficiently when they occur, ensuring strategies are in place for procedures to follow for network and system outages.
Training as a Critical Defense
One of the key takeaways from the CrowdStrike outage is the importance of proper training to respond to incidents like this. IT teams with advanced training in network management and incident response were better equipped to handle the challenges posed by the outage. They could quickly diagnose the problem, understand the implications of the flawed update, and implement manual resets and other corrective measures effectively, ensuring business continuity in the face of massive security challenges.
Key strategies for ensuring business continuity include:
Rapid Assessment and Response: Immediate identification of affected systems and deployment of manual resets or patches to restore functionality.
Cross-Departmental Communication: Keeping all parts of the organization informed and coordinated in their response efforts.
Leveraging Redundancy: Using backup systems and protocols that can operate when primary systems fail.
Training, however, should not be limited to technical staff alone. All employees can benefit from understanding the basics of cybersecurity. This knowledge can empower them to act swiftly and appropriately in times of crises like this, reducing the potential for human error, which often exacerbates the issue. Additionally, it helps everyday users prevent cybersecurity attacks by improving their overall awareness and response to potential threats.
Ongoing Monitoring and Security Post-Outage
The restoration of systems is just the beginning; ensuring they remain secure against further issues is crucial. Teams must remain vigilant, continuously monitoring systems to detect and address vulnerabilities. Some effective measures include:
Enhanced Surveillance: Implementing real-time monitoring tools to detect unusual activity that could indicate a breach or new vulnerabilities.
Regular Audits: Conducting security audits to ensure all systems are secure and any detected anomalies are swiftly addressed.
Update Rollout Oversight: Managing the deployment of updates carefully, including staged rollouts and thorough testing, to avoid incidents similar to the CrowdStrike debacle incidents.
Long-term Implications and Lessons
The CrowdStrike outage is a potent reminder of the complexities involved in cybersecurity management. It highlights the need for:
Robust Testing Protocols: Before deploying updates, especially those that are widespread and critical, robust testing must be a standard procedure to catch potentially disruptive bugs.
Incident Response Planning: A well-defined and practiced incident response plan can drastically reduce the time and impact of system outages.
Continual Learning and Adaptation: Cybersecurity is not static. Continuous learning, updating, and testing are essential to adapt to new challenges.
Comprehensive Training: Training should be comprehensive and ongoing to prepare all levels of an organization to respond effectively to cybersecurity incidents.
Conclusion
The CrowdStrike outage was undoubtedly a setback, but it also provides a valuable learning opportunity. By understanding what went wrong and how different companies responded, the cybersecurity community can improve its practices and strategies. Most importantly, it underscores the critical importance of balancing proactive security measures with rigorous testing and training to safeguard against the unpredictable nature of cyber threats.
About INE Security:
INE Security is the premier provider of online networking and cybersecurity training and certification. Harnessing the world’s most powerful hands-on lab platform, cutting-edge technology, global video distribution network, and world-class instructors, INE Security is the top training choice for Fortune 500 companies worldwide for cybersecurity training in business, and for IT professionals looking to advance their careers. INE Security’s suite of learning paths offers an incomparable depth of expertise across cybersecurity and is committed to delivering advanced technical training while also lowering the barriers worldwide for those looking to enter and excel in an IT career.
