What happened with CrowdStrike should be a wake-up call for many EMS and NEMT organizations out there.
Boston EMS reverted to a system used 30 years ago because of the CrowdStrike outage.
Non-urgent medical visits were cancelled because of the outage at Massachusetts General Hospital.
Chatham EMS and Savannah Fire paramedics in Savannah, Georgia, had to switch from digital incident reports to pen and paper.
For life-saving services like EMS where every second counts, CrowdStrike’s incident highlights the importance of mitigating risks and single points of failure.
In this blog post, we’ll explore the key lessons learned from the CrowdStrike outage, focusing on how EMS and NEMT organizations can better prepare themselves and prevent similar unplanned downtime in the future.
A Quick Glance at the CrowdStrike Incident
In July 2024, CrowdStrike had a major outage that affected millions of computers around the world. The problem started when a software update (CrowdStrike Falcon) meant to improve security was automatically patched, causing many Windows systems to crash repeatedly. This led to what’s commonly known as the “blue screen of death”, where the computer shuts down suddenly and won’t start up properly.
The issue impacted critical services from hospitals, airlines, government agencies, and EMS/NEMT. Although a patch was released almost immediately to fix the error, many affected systems required manual repairs, making it a slow process to bring everything back to normal.
3 Lessons Learned for EMS and NEMT Organizations
Lesson 1: Try Not to Over-Rely on a Single Software or Platform for Your IT Infrastructure
The recent CrowdStrike outage highlighted the risks of over-reliance on a single platform in an IT infrastructure. When a critical issue has such widespread IT and operational exposure, it can create a single point of failure that disrupts entire operations. In CrowdStrike’s case, the single point of failure was the software patch that took down servers and endpoints.
This incident highlighted the need for organizations, especially those in essential services like healthcare and emergency response, to diversify their technology stack. By relying on multiple platforms and not putting all your eggs in one basket, you can better manage risks and maintain continuity even when one system goes down. While it may be challenging to switch endpoints away from Microsoft, migrating servers to alternative platforms should be considered.
Alternatively, if you have no choice but to rely on a single platform for your IT infrastructure, then it is best NOT to automatically apply patches to your system. Faulty patches manifest more frequently than what is released in the news. It’s always best process to test new patches in a pre-production, controlled environment before rolling them out to your IT infrastructure.
Lesson 2: Protecting Against Other Single Points of Failure
What happened in the CrowdStrike incident was a single point of failure caused by software. There are many other potential single points of failures that can cause more damaging outages, such as hardware failures, natural disasters, employee sabotage, or ransomware attacks. A few notable ransomware attacks include the MGM and Caesars ransomware attack and the CDK outage that affected more than 10,000 auto dealers. Ransomware hackers are increasingly targeting the healthcare industry because they are easy targets and house the most comprehensive set of Personal identifiable Information (PII).
To protect against other single points of failure mentioned earlier, you need well-planned disaster recovery strategies and solutions, including remote backups and cloud-based solutions that can reduce downtime. However, having a disaster recovery solution is not enough. You also need to have a disaster recovery plan and an incident response plan so your team knows what to do when an incident occurs. At the very least, both plans should outline the immediate step-by-step on what to do, including whom to communicate with and how to recover from each unique type of incident.
What’s more important is to rigorously test your disaster recovery and incident response plans. All too often, we see organizations letting their plans gather dust. When an incident does occur, the listed contacts in both plans have already left the company. The lack of testing is often what leads to an extended recovery time that results in significant losses.
Lesson 3: Evaluate Third-Party Risk Management
Many organizations rely on third-party vendors to operate, just as organizations today use CrowdStrike Falcon for cybersecurity. Evaluating third-party risk management is essential to ensure that your organization can maintain operations during disruptions, even when those disruptions originate outside your direct control.
Third-party risk management involves identifying, assessing, and mitigating risks associated with external vendors. These risks can include everything from financial instability and cybersecurity vulnerabilities to operational failures. When one of your critical vendors experiences an issue, it can have a ripple effect on your business, potentially leading to downtime, data breaches, or compliance violations.
When evaluating third-party risks, you should ask them questions like: Do you have redundancy and failover systems in place? How quickly can you recover from a disruption? You will also need to review contractual obligations such as service level agreements (SLAs) that specify acceptable level of downtime. To prevent over-reliance on a third party, develop a contingency plan that outlines how your organization will respond if a critical vendor experiences a disruption.
How to Mitigate: Using a Cloud-Based SaaS Solution
A well-established cloud-based SaaS solution like Traumasoft can mitigate the risks mentioned above.
Traumasoft is an all-in-one solution for EMS and NEMT organizations to manage their entire operations – from dispatch and ePCR to billing and reporting. Our solution is hosted in the cloud and can be accessed by simply launching a web browser.
At this point, you might be asking:
But… isn’t using Traumasoft’s all-in-one software relying on a single platform?
No, it is not because we rely on a diverse technology stack and multiple vendors so the risks are dispersed and mitigated. Traumasoft was NOT impacted by the CrowdStrike outage.
In essence, when you implement Traumasoft, you’re also getting the below technology stack:
A Diverse Technology Stack
Cloud-based SaaS solutions have been around since the early 2000’s, and Traumasoft has 16 years of experience and constant improvements to ensure that our diverse technology stack and infrastructure use proven best practices. We’re constantly evaluating third-party risk to prevent over-reliance on any third party or platform.
Disaster Recovery
Our platform is our bread and butter. That is why we have invested in disaster recovery, remote backups and recovery, and cloud-based backups and recovery. This is how we protect against single points of failure, including faulty software patches, hardware failures, employee sabotage, natural disasters, and other threats.
Cybersecurity and Data Security
We are well aware of the increasing cyberattacks on the healthcare industry, which is why we’ve implemented the zero-trust security model to protect against both internal and external threats.
Mobile, Tablet and Desktop Access via Web Browser
If you have a web browser, you can access Traumasoft for dispatch, ePCR, billing, and more. That means if your office PCs couldn’t boot because of CrowdStrike, your employees can improvise by using their home computers, tablets, or smartphone. You wouldn’t need to revert back to paper and pen like what happened to Boston EMS in the CrowdStrike outage.
Other Benefits of Using a Cloud-Based EMS and NEMT Solution
Virtual Offices and Remote Dispatchers
When work can be completed simply by using an internet browser, the employee can work anywhere, at any time. This makes virtual offices possible for office workers, such as dispatchers and billers. Several Traumasoft’s customers take advantage of this by employing remote dispatchers and/or billers to save on cost and improve employee retention.
Employee Retention
With a cloud-based SaaS solution, you can offer your employees the option to work from home and work more flexibly. Traumasoft takes employee retention further by lightening employee workloads through automation and by fostering camaraderie and collaboration with both web and mobile communication tools.
Lower Cost to Maintain
SaaS vendors like Traumasoft are responsible for monitoring, maintaining, and updating the infrastructure, ensuring that the software is always running at peak performance. The SaaS vendor carries the cost of hardware, software, and IT personnel, which is why this is can be such a cost-efficient option.
Scalability
As you grow, you only need to license for expanded usage. You do not have to worry about hardware capacity, IT workloads, or any other software costs.
Explore Traumasoft’s Technology Stack
At Traumasoft, we boast 99.9% uptime for our clients. Get a demo to explore our technology stack and see how we maintain this standard.
About Traumasoft
Traumasoft is one truly integrated product for Emergency Medical Services (EMS) and Non-Emergency Medical Transportation (NEMT) that drives timely efficiencies. We are an all-in-one system capable of managing every aspect of your EMS operations. One system comprised of interconnected solutions handles staff, integrates processes, manages assets, and grows revenue. Schedule a demo today to see how we can help you increase operation efficiency and profitability.