On Friday 19 July 2024, the UK woke up to news of a widespread IT outage that seemed to be global, affecting hundreds, if not thousands, of organizations.
The disruption began in the early hours of Friday morning in Australia and quickly spread across Asia, Europe, and the Americas, with the travel industry being among the hardest hit.
The outage was traced back to cyber security firm CrowdStrike, which was already working on incident response amidst the chaos. Stay updated on this evolving incident in the days and weeks to come with our Essential Guide.
What is CrowdStrike’s role?
CrowdStrike is a leading global cyber security company with thousands of customers worldwide. Headquartered in Texas, it has over 8,000 employees and generates around $3 billion in revenue annually. The company was established in 2011.
The company describes itself as follows: “CrowdStrike has redefined security with the world’s most advanced cloud-native platform that protects and enables the people, processes, and technologies driving modern enterprise. CrowdStrike secures critical areas of risk – endpoints and cloud workloads, identity, and data – to stay ahead of today’s adversaries and prevent breaches.”
CrowdStrike may not be familiar to most individuals outside the tech industry, but Formula 1 fans are likely aware of it due to its sponsorship of the Mercedes AMG Petronas team. The company’s branding is visible on the halo safety device and in onboard footage from Lewis Hamilton’s car.
Security professionals are familiar with CrowdStrike due to its involvement in major incident investigations, such as the Sony Pictures hack, WannaCry crisis, and the 2016 hack of the Democratic National Committee by Russia.
What occurred during the CrowdStrike outage?
The initial sign of the disruption was the appearance of the blue screen of death on Windows PCs, indicating a fatal system error.
Initially, Microsoft responded to what seemed like a Microsoft issue, confirming before 8 am BST that they were investigating problems affecting cloud services in the US.
It was soon discovered that the problem stemmed from a faulty channel file in CrowdStrike’s Falcon sensor product.
Falcon is designed to prevent cyber attacks by integrating next-gen antivirus, endpoint detection and response (EDR), threat intelligence, threat hunting, and security hygiene. The issue seems to have originated from the cloud-managed sensor.
The faulty update caused a boot loop, where Windows devices restart during startup, preventing them from completing a stable boot cycle and turning on.
As of now, the full details of the incident are still being investigated and may take some time to fully understand.
Is there a cyber security threat from the CrowdStrike outage?
While similar to a supply chain attack in its impact and origin, it’s important to note that the CrowdStrike outage is not a cyber security incident and no one is currently under attack as a result.
However, since it affects a cyber security product, there is a possibility that threat actors may exploit the downtime and any coverage gaps that arise.
In the coming days and weeks, threat actors may leverage the incident in phishing and social engineering attacks to lure new victims. Potential tactics could include offering fake technical support or CrowdStrike updates, leading to data theft, ransomware, or extortion.
Security and IT leaders are advised to inform their users about the potential risks.
Who was impacted by the CrowdStrike outage?
The full extent of organizations affected by the outage is currently unknown. However, those known to have been impacted or have confirmed experiencing some effects include:
- Airlines such as American Airlines, Delta, KLM, Lufthansa, Ryanair, SAS, and United;
- Airports including Gatwick, Luton, Stansted, and Schiphol;
- Financial institutions like the London Stock Exchange, Lloyds Bank, and Visa;
- Healthcare facilities including most GP surgeries and many independent pharmacies;
- Media outlets such as MTV, VH1, Sky, and some BBC channels;
- Retailers, leisure, and hospitality businesses like Gail’s Bakery, Ladbrokes, Morrisons, Tesco, and Sainsbury’s;
- Sports organizations like F1 teams Aston Martin Aramco, Mercedes AMG Petronas, and Williams Racing, competing in the Hungarian Grand Prix on 20 and 21 July, and the Paris 2024 Organizing Committee for the Olympic and Paralympic Games starting on 26 July;
- Train operating companies (TOCs) including Avanti West Coast, Merseyrail, Southern, and Transport for Wales.
CrowdStrike’s response to the outage
In an initial statement, CrowdStrike CEO George Kurtz stated: “CrowdStrike is actively assisting customers affected by a flaw found in a single content update for Windows hosts. Mac and Linux hosts are unaffected. This is not a security incident or cyber attack.
“The issue has been identified, isolated, and a fix has been implemented. Customers can refer to the support portal for the latest updates, and we will continue to provide comprehensive and continuous updates on our website.
“We also advise organizations to ensure they are communicating with CrowdStrike representatives through official channels. Our team is fully committed to ensuring the security and stability of CrowdStrike customers.”
In an interview with NBC on US breakfast TV, Kurtz expressed his regret for the impact caused to customers, travelers, and all those affected, including other companies.
Microsoft’s statement, provided to the BBC by a spokesperson, reads: “We are aware of a problem affecting Windows devices due to an update from a third-party software platform. We expect a resolution soon.”
Can I resolve the CrowdStrike issue on my own?
CrowdStrike has automatically rolled back the changes to the affected product, but some hosts may continue to crash or be unable to stay online for the remedial update.
The short answer is yes, but resolving such issues can be challenging and may require significant effort from IT teams. It could take days or longer to reach all affected devices.
System administrators should take the following steps:
- Boot Windows into safe mode or the Windows Recovery Environment;
- Navigate to C:\\Windows\\System32\\drivers\\CrowdStrike directory;
- Locate the file matching “C-00000291*.sys” and delete it;
- Reboot normally.
CrowdStrike customers can find more information by logging into the support portal.
How can I prevent similar issues in the future?
Security companies like CrowdStrike face significant pressure in developing and updating products to protect customers from new threats like zero-days and ransomware. This pressure extends to customers who often want their security tools to update automatically.
To avoid future problems like this, IT teams should consider a phased approach to software updates, especially for security solutions, and test them in a sandbox environment or on a limited set of devices before full deployment.
Having some level of system redundancy to isolate and manage fault domains, particularly in critical infrastructure, can also help prevent similar incidents.
