Over the past few years as we have assisted organizations in containing, remediating and recovering from major breaches, as with the Security Information and Event Management (SIEM) boon we now see a common theme re-occurring with Endpoint Protection Platforms (EPP), Next Generation Antivirus (NGAV), and Endpoint Detection & Response (EDR) across enterprises that kind of feels like DeJa’Vu for us.
Now, in the modern-day business landscape, organizations looking to secure their systems, devices and data, often believe the claims of security vendors who tout the supposedly ‘magical’ qualities of SIEM and now EDR solutions. By the time the firms realize that they’ve been taken in by a snake-oil salesman, it’s too late. Instead of a Wizard, they got more smoke and mirrors.
That’s why enterprises need to be more pragmatic about the pitfalls of EDR solutions before making expensive investments (that they may end up regretting).
First a quick history on SIEMs
If you know me or if you have worked in security then you know that central to every SOC is the SIEM that has proved expensive to own, complex to operate and being demoted in most enterprises to a glorified syslog server. The SIEM is also the root cause of alert fatigue that has delivered nothing more than a false sense of security to enterprises for more than 20 years. And organizations continue to get breached, while their SIEM sits in the corner quietly eating the enterprise IT budget, year after year.
Aside from basic log management and compliance use cases, SIEM’s have been such a failure that vendors (over the past few years) have resorted to add many other features in hopes of making it do something useful. These includes adding User and Entity Behavioral Analytics (UEBA), Security Orchestration Automation and Response (SOAR), Threat Intelligence, Network Threat Analytics (NTA), Artificial Intelligence and Machine Learning (AI and ML), or more recently completely renaming it to X Detection and Response (XDR) and trying to distance themselves from the legacy SIEM – the only thing they haven’t thrown at the SIEM domain is the kitchen sink.
Unless you’re a large enterprise with a large security savvy team, then you are most likely going to fail to gain any benefit from a SIEM investment. You can read more on this here and here.
And now a quick history on EPP, NGAV and EDR
EDR solutions record system activities and events taking place on endpoints and provide security teams with the visibility they need to uncover incidents that would otherwise remain invisible. EDR solution arise from and are the evolution of whitelisting solutions. A whitelisting solution essentially monitors and prevents the execution of events. In doing so, whitelisting solutions were able to prevent the execution of software that was unrecognized or untrusted. Its strength was based on the fact that there were no known ways to evade execution events fired by the OS.
EDR solutions are predicated on the idea that a defensive tool can monitor a series of events along the kill chain and capture enough event information that the likelihood of something getting by undetected is significantly reduced. A fundamental underlying assumption to this approach is that there is no way to evade all of the different events along the kill chain that are being monitored. In reality, EDR vendors openly acknowledge that they sometimes miss the primary events associated with malware.
Several years ago, security researchers disclosed findings that showed how to execute software on Windows OS without triggering any events; this discovery, and the widespread sharing of it, triggered the wave of fileless malware we experience today. This single event can be pointed to as the reason whitelisting was significantly weakened as an effective defense.
EDR solutions evolved whitelisting by adding more events and applying analytics engines to the data to uncover instances of things that don’t belong and enrich the discoveries with intelligence gathered and maintained about files and events. As it stands with security technologies, early adopters reaped the greatest rewards. Today, the limits of EDR solutions are becoming well understood as vulnerabilities are exploited and attack methodologies multiply.
NGAV is the next step in endpoint protection, using a signature-less approach to deliver complete and more effective endpoint security than is possible with legacy AV. Antivirus is the prevention component of endpoint security, which aims to stop threats from entering a network.
EPP saw the merging of NGAV and EDR into a single agent to prevent file-based malware attacks, detect malicious activity, and provide the investigation and remediation capabilities needed to respond to dynamic security incidents and alerts. As you can read from this production client they don’t fare too much better than your legacy AV.
Point of Reference: EDR
Over the years we have come across dozens of organizations that have invested in EDR (or EPP with EDR capability) solutions without the skills to use these tools. They get breached all the time, but they don’t have the skills to use their EDR to conduct any investigations. As a matter of fact their ability to detect & respond to threats was still near zero post investment in such tools.
As a point of reference, a recent client with 1500 endpoints that had invested in not one but two types of EPP solutions, was hit by ransomware that bypassed both vendors and completely locked up their servers and workstations. You can read all the technical details on our response and ultimate attribution here. If they didn’t have a solid backup and restore process there was no way we could have assisted them in getting back up and running within 48hrs. Is this client large enough or have the skills to warrant an investment in EDR? Clearly not, but yet their reseller sold them one and then another vendor convinced them to buy another.
In another case, the client with 800 endpoints replaced their perfectly good EPP with another EPP solution following a ransomware attack that stopped their business for over 5 weeks. The board made available USD $800K for new security spending and the IT team went to town buying every type of security solution you can imagine, from encryption to the new EPP solution. It was unfortunate but just like the case with Toll Group, in less than two months they had another ransomware attack which again locked up their computers. Lucky for them this time they had established a new backup regime that saved them and allowed them to recover within a week.
EDR, just like SIEM, over-rated, over-hyped, underperforming
A lot of folks still believe that EDR can address many of the problems inherent in traditional antivirus (AV) solutions – solutions that are inadequate in dealing with the emerging wave of malicious cyber threat tactics, techniques, procedures and of course, criminals. This signature-less solution supposedly provides enterprises with greater visibility into what’s occurring on their network so they can detect, investigate and even remediate security incidents on endpoints.
But a lot of these happy campers don’t realize that EDR is not, and never will be enough, no matter how many bells and whistles you put on it.
You can read about a production client and how EPP and NGAV are not much better than legacy AV.
First, the increased visibility of EDR means increased data, which needs more analysis, and increases complexity that requires more bandwidth, time, money, and a trained workforce to deal with. These additional expenses are usually not feasible for most small to mid-size firms, but they have now all got either an EPP with EDR or a dedicated EDR which will never see the light of day.
Second, EDR does not provide 100% network visibility. Threats can sneak through the network, move laterally across it and clandestinely ‘talk’ to a remote C&C server controlled by a cybercriminal to carry out DDoS attacks against enterprises, steal their data or even to hold their data for a huge ransom.
Third, EDR focuses only on endpoints to the detriment of other probable security event locations like network data, log data and the cloud.
Fourth, significant capability differences exist between EDR agents for different operating systems. This can result in an inconsistent, patchy security solution that can cause more issues than it solves.
Fifth, talk about cutting-edge AI and ML technologies in modern EDR solutions are just terms thrown out by slick marketing teams, not actual capabilities. You will note that how most EDR solutions turned EPP overnight simply OEM a cheap AV such as Bitdefender into their product to be able to have a competitive offering in the market and on the flipside NGAV vendors added terrible EDR capabilities to their solutions.
Moreover, contrary to many organizations’ expectations, these technologies do not resolve the underlying security or operational issues. You need people with knowledge and experience to do that. If you don’t, your EDR deployment will fail right from the outset.
Finally, many organizations end up using only the average antivirus capabilities of their EDR even though they have paid for a EPP solution for prevention with EDR for high-visibility threat detection. Result: huge investment, very small return.
Thinking of pairing your EDR solution with a SIEM? Boy, now you’re asking for trouble. Like EDR, implementing SIEM can also be costly, complicated and time-consuming. In the Netwrix SIEM Efficiency Survey, 85% of respondents experienced significant price hikes due to their adoption of SIEM. 81% also said that their SIEM’s reports were ‘noisy’ and failed to provide useful, actionable insights. Therefore, assuming that you can simply buy a SIEM and EDR and then get immediate value from them can be a very expensive mistake.
I have seen these issues in many organizations firsthand so I urge you to hire a red team to test your SOC once you make your SIEM and EDR investments and you will quickly realize that your ability to detect & respond to threats is still pretty much close to zero.
Claims vs Reality
There is no dispute that EDR solutions are highly effective and much needed innovation from legacy signature-based endpoint solutions of yesteryears. However, the market positioning and vendor claims made have led many enterprises to labour under false impressions. Let’s examine the market claims and contrast them with the realities.
| Claim | Reality |
| We collect all events along the kill chain. We don’t miss a thing. | Limited Event CollectionHow do you collect tens of thousands of events per minute from operating systems without bringing the endpoint to a halt? It’s simple, EDR doesn’t collect everything and EDR does not collect alternate channel events. |
| Easy to operate and delivers outcomes in minutes not months | Limited Value Out of the BoxNeeds to be wielded by highly trained forensic experts, analysts and hunters with many years’ experience who have extensive knowledge of OS internals. Requires significant investment into building automated hunts to scale any operation. |
| Very low upkeep and rapid efficacy | Time & TuningEDR requires 1-3 months of endpoint data before they reach peak efficiency, and then require ongoing tuning |
| You can very effectively hunt for malware using the data set available in our EDR | Alert Fatigue and Data OverloadEvery EDR solution we have seen produces tens of thousand to millions of events per day. Good luck in finding the needle (things EDR misses) amongst this haystack. |
| EDR can hunt in memory | Memory ProtectionsEDR solutions generally do not go into memory, nor do any of them forensically reconstruct discoveries. They do have SOME memory based defenses – however they are focused on monitoring for events and protecting files that can be used in memory-based attacks. |
The disparity between the claims being made by EDR vendors and the real-world experiences of customers using the products clearly demonstrates that there are gaps that EDR cannot address. To achieve strong protection, what’s required is a solution based on the understanding that it’s impossible to defend against all threats, some will bypass even the strongest defences. When threats do evade EDR defenses, the EDR solutions deployed are not built to hunt the threats they missed and are residing in the estate undetected.
Prepare before you go forth and conquer!
Still considering investing in EDR? Be aware of these proven best practices to derive the most value from your investment.
1. Estimate the time and resources required to build EDR capability
Estimate the average volume of alerts likely to be generated daily, weekly, monthly, etc. Understand how much time you’ll need to analyze and investigate potential threats. Outline the investigation and response process.
If you don’t have the bandwidth and human resources to manage these tasks, consider investing in a Managed Detection and Response (MDR) solution like LMNTRIX. Download a copy of our MDR Buyers Guide: How to Select and MDR Provider? Questions to Ask and Pitfalls to Avoid
I cannot stress the importance of this point. Unless you are a security savvy enterprise with a minimum 10 experienced people in your security team then you most likely don’t have what it takes to benefit from an EDR investment without significant investment.
The skills you will need include malware analysis, binary triage and analysis, Windows internals, Windows file system, Windows registry, intelligence research, open source research, programming/scripting, Intrusion Detection System (IDS) signature writing, netflow analysis, protocol analysis and digital forensics expertise.
2. Understand the infrastructure requirements
Based on your preferred deployment method (on-premise, appliance or cloud), understand the infrastructure requirements and costs: hardware, software, upgrades and maintenance. Make sure the solution supports your organization’s operating systems, platforms and scalability goals. Some of these EDR tools record everything in/out of each machine and so they are very heavy on memory, disk and network bandwidth, so test them before you make a buy decision.
3. Avoid Managed Security Service Provider (MSSP) where possible
EDR requires a specific skillset and additional technologies that an MSSP’s team and infrastructure is not designed to support.
Conduct due diligence to dig into the provider’s expertise and staffing capabilities. If you’re not convinced, don’t hire them to manage your EDR.
At LMNTRIX our structure includes teams specializing in Threat Detection, Threat Response, Threat Hunting and Threat Intelligence in addition to the Engineering, Development and Service Delivery teams. Unlike MSSPs we don’t manage any client security controls and we don’t rely on logs or a SIEM to detect threats.
4. Get an insight into the solution’s visibility level
What are the endpoint activities your EDR collects information about? Does it monitor registry changes and file modifications? Keep an eye on process information and network connections? Your EDR is only as valuable as the information it provides.
Contact us for a full list of the NGAV and EDR features and functions used in the LMNTRIX Respond agent.
5. Do assign metrics to measure and improve efficiency
Metrics will show how well your EDR is performing at detecting and/or preventing threats – or not. Some commonly used metrics are:
- Occurrence to detection
- Severity of detected threats (high/medium/low)
- Resolution time
- False positive and negative rates
A final word
Is this article going to make any difference to the security technology selection or buying habits of enterprises? Absolutely not! How can we possibly be expected to compete with the hundreds of millions in Venture Capital dollars spent annually by vendors marketing teams who consistently manage to swindle organizations out of their security budgets by selling them yet one more silver bullet.
Email us at [email protected] to find out more.