This page is being updated as the situation develops.
Summary
While this has yet to be confirmed, it looks like the current global IT outages are a perfect storm caused by a Crowdstrike Falcon update resulting in Windows computers, VMs, and endpoints to loop, alongside a Microsoft 365 outage caused by an Azure configuration issue. It may transpire that the two issues are linked.
Initial report
A ‘content deployment’ issue linked to the Crowdstrike Falcon cyber security solution has resulted in Microsoft outages around the world, with subsequent huge business continuity issues for many organizations.
On a support update, Crowdstrike stated that it has identified the issue and has now reverted the change. Crowdstrike users can find more information on the Crowdstrike support site here.
Update 1
It appears that although the content update has been reversed the fact that many systems are in a reboot loop means that the necessary Channel File Changes to resolve the issue cannot be received because the hosts are not remaining online long enough to do so.
Workaround steps for individual computers or work stations have been published as follows. Resilience Forward cannot verify this so please use with caution and consider any cyber security issues that may result from this action.
Workaround Steps:
- Boot Windows into Safe Mode or the Windows Recovery Environment
- Navigate to the C:\Windows\System32\drivers\CrowdStrike directory
- Locate the file matching “C-00000291*.sys”, and delete it.
- Boot the host normally.
Latest: The above workaround has been confirmed by CrowdStrike and additional methods have been outlined here. Microsoft has also confirmed that Virtual Machines running Windows Client and Windows Server and using the CrowdStrike Falcon agent have also experienced the same issues. Recovery methods for these are available at https://azure.status.microsoft/en-gb/status
Update 2
Microsoft has published a status report on its cloud status site which seems to indicate that a separate issue with Microsoft 365 may have added to the Crowdstrike issue. The status reads as follows:
Scope of impact
This issue may be impacting any user attempting to use various Microsoft 365 apps and services.
Preliminary root cause
A configuration change in a portion of our Azure backend workloads, caused interruption between storage and compute resources which resulted in connectivity failures that affected downstream Microsoft 365 services dependent on these connections.
10.30am UTC:
Microsoft has issued a new status update on the above. It reads as follows:
The underlying cause of the issue has been fixed and several Microsoft 365 apps and services have been restored to full functionality. Residual impact is still affecting some Microsoft 365 apps and services, and Microsoft 365 engineering are continuing to conduct additional mitigation actions to provide relief. We’re continuing to observe an increase in functionality and availability for the remaining impacted scenarios and we’re monitoring this closely to ensure we’re progressing towards full recovery. Microsoft is continuing to treat this event with the highest possible priority.
12.37pm UTC:
A new update from Microsoft:
We’re continuing to apply mitigation actions to provide relief from the residual impact affecting the remaining impacted Microsoft 365 apps and services. Our telemetry is indicating that the remaining impacted scenarios are progressing towards a full recovery and we’re closely monitoring to ensure this progress continues.
Update 3
George Kurtz, President and CEO of CrowdStrike, has posted the following update on X:
CrowdStrike is actively working with customers impacted by a defect found in a single content update for Windows hosts. Mac and Linux hosts are not impacted. This is not a security incident or cyberattack. The issue has been identified, isolated and a fix has been deployed. We refer customers to the support portal for the latest updates and will continue to provide complete and continuous updates on our website. We further recommend organizations ensure they’re communicating with CrowdStrike representatives through official channels. Our team is fully mobilized to ensure the security and stability of CrowdStrike customers.
What is Crowdstrike Falcon?
According to Crowdstrike, the CrowdStrike Falcon platform is a widely-used cyber security solution that ‘leverages real-time indicators of attack, threat intelligence on evolving adversary tradecraft and enriched telemetry from across the enterprise to deliver hyper-accurate detections, automated protection and remediation, elite threat hunting and prioritized observability of vulnerabilities – all through a single, lightweight agent’.
Expert comments
The following comments have been provided to Resilience Forward:
Simon Pardo, director of technology specialists Computer Care:
“An IT failure double whammy has brought the world to its knees this morning, and the ongoing chaos will be costing businesses billions of pounds every hour.
“Two separate problems have struck at the same time, with issues hitting Microsoft’s 365 services, and an update from anti-malware product Crowdstrike that is pushing computers into a blue screen of death.
“Updates should always be thoroughly tested before they are pushed out to users, and it’s hard to know whether this is human error or a failure of process.
“Microsoft services are slowly coming back on line, but those users affected by the Crowdstrike update could face days or weeks of severe disruption.
“Unfortunately, there is not a huge amount you can do if your computer has the blue screen of death. Computer users will need to wait for their IT team to resolve the issue, and many people may need to work from their phone or use a pen and paper in the meantime.
“This is a wake-up call for all the companies that have been floored by this attack. Organizations need to urgently review their disaster recovery plans to make sure they can deal with such problems.
“For all business leaders my recommendation is to challenge your IT team – ask them how do we roll-out updates and patches, how would we recover from a Crowdstrike type failure and make sure you run a test and prove your disaster recovery plan works.”
Mark Grindey, CEO of Zeus Cloud:
“It’s clear that adequate testing for updates should be done in a safe environment before issuing them company-wide. Companies should never have auto-updates set in a live environment and always test an update in a safe environment before releasing it live to minimise potential risks. This global outage highlights the need for businesses to not blindly trust their suppliers when it comes to updates before testing first.
“The only fix now is to reboot in safe mode and remove the erroneous file; unfortunately, this can’t be done remotely. It could so easily have been a security incident or cyber-attack and this manual intervention required to get back up and running opens the door for other potential security risks and vulnerabilities. The only course of action now is to manually and safely reboot the thousands of computers affected – a task that will undoubtedly be challenging and time-consuming.”
Ryan Thornley, Security Practice Lead, Appsbroker CTS:
“For organizations around the world, this incident underscores the crucial importance of having a clear and thorough disaster recovery plan. In an era where we are so completely dependent upon interconnected devices, organizations need to plan for the modern risks that come with modern IT practices, and as with this incident, around third-party providers and services.
“We don’t yet know the full extent of this incident, but what’s clear at this point is that the recovery phase for businesses and public organizations is going to be huge and very costly. Potentially millions of machines around the globe, from hospital computers to supermarket checkouts have received a defective content update to Crowdstrike and in many situations will need to be physically accessed to make them bootable again. This will be a mammoth undertaking and a real stress test for companies’ disaster recovery plans. In cases like hospitals and emergency services, this could be life-threatening.”
Kevin Reed, Chief Information Security Officer, Acronis:
“The recent CrowdStrike outage appears to stem from a bug in their EDR agent, which was unfortunately not thoroughly tested. This resulted in widespread disruption as many installations were affected globally.
“The flawed update necessitates manual intervention to resolve, specifically rebooting systems in “safe mode” and deleting the faulty driver file. This process is cumbersome and leaves systems vulnerable in the interim, potentially inviting opportunistic attacks.
“This incident highlights the importance of rigorous testing and staged updates for EDR agents. Normally, testing is done with every release and can take days to weeks, depending on the size of the update or changes.
“For our Acronis customers, those with recent backups can restore their systems to a stable state, minimizing downtime and exposure. Moving forward, we recommend all businesses ensure robust backup solutions and advocate for better testing protocols from their security vendors.
“This issue reminds us how fragile IT infrastructure is and why cybersecurity should be natively integrated with backup. An integrated solution is the only way to provide complete protection that would enable fast roll-back to the working state.”
