I’ve been pretty tame so far in this series (see part one and part two). Let’s get spicy.
This article puts a bow on the previous two installments we’ve shared on our safari excursion on defense evasion.
So... how do you catch defensive evasion? Seems like a paradoxical thing to ask.
Well, bad guys have to get IN somehow, they have to RUN stuff somehow, and there has to be some kind of IMPACT to what they run. The advanced ones are able to make the above sneakier but not invisible.
To catch this kind of advanced intrusion, you need proactive security. This is a huge, messy, loaded term and one I’d really like to unpack properly another time. But I’d like to note: you cannot ‘product’ your way to proactive security. Anyone trying to convince you otherwise is likely trying to sell you something (something probably crap too).
Proactive security needs some internal skin in the game. This doesn’t always mean you’re spending crazy money. It just means you’re thinking ahead of time about the possibility of intrusion and how you can best thwart and recover from varying severities of attack, not simply relying on security product X to do all the work for you.
In this blog, we’re going to...
- Have a brief, realistic conversation about your risk landscape, risk appetite and threat management
- Offer granular advice for monitoring and detecting defense evasion
- Share some insight about hunting and distracting adversaries before they have momentum in their campaign
- End with some thoughts on testing the efficacy of your security strategy
Security Maturity: "We Accept the Risk" 🤡
Let’s not get caught up in the hype on defense evasion. There’s nuance to be had in the conversation about evasion taxonomies, but we aren’t here for that.
We at Huntress see some really advanced adversaries….but the vast majority of what we observe does not qualify as sophisticated or advanced. In fact, just hardening your Active Directory would probably mitigate 80% of the attacks we see (see Trimarc Security’s June 2022 webinar).
Anyway, how are you appropriately considering your risk?
I have heard from too many the famous last words, ‘we accept the risk,’ but how exactly has that decision-making process been informed?
Is it a belief that by purchasing security product x for $$$ that the risk has somehow been tamed? Did a shady salesperson come crashing into your orbit like an interloper asteroid right after an incident, claiming their product would have stopped [insert incident]’? Or does ‘accepting the risk’ translate to “security is a cost center, and we’re rather f**k around and find out than invest?” ?
When you think about your security risks (your threat model), are you focusing on flashy advanced persistent threats (APTs) more than the disgruntled finance employee exfiltrating important documents as we speak? Or that employee in logistics who is lovely but would open and trust anything an email said?
Planning your security around the more mundane threat scenarios is not just better for your wallet, but is also a more attainable and realistic threat model. Which MSPs, SMBs, or contractors are going to achieve zero trust? And why should they bother? The juice isn’t worth the squeeze for that kind of junk.
A Realistic Plan
If you plan for mundane security, you’ll reap security dividends when later working on ‘advanced’ security strategies. A solid base to lock down includes
- Appropriate patch management
- An awareness of your internet-exposed machines
- An inventory of your machines
- Suitable ways to monitor data exfiltration
- Standardly rolling out MFA
- Road-tested backup solutions
And just a note to the people who like to wantonly purchase very expensive security products that promise machine-learning and AI-driven remediations: Are you mature enough for this conversation?
I’m only slightly tongue-in-cheek here.
In your network, I see way too many `ransomware.exes` and mimikatz binaries pulled straight from Github for any of you to be worried about the polymorphic quantum C3 malware that your security product says it can thwart.
Security product X just told you there’s lateral movement with the humble PsExec. What you doing now, chief?
And another thing: If you spend $10000000000000 for machine learning security solutions but won’t invest $0.00 to close Remote Desktop Protocol (RDP) from the internet, I’m confused why you’re confused you got ransomed.
I’m not bashing security product X. It’s probably a good investment. But no matter what it’s sold as, no purchase has the same impact as human beings in your company making good security-conscious choices.
You will save yourself money upfront by deploying human capital for your security posture, and then recoup your money later when you’ve successfully contained a ransomware attack and the subsequent cost of rebuilding.
Monitor for Impact
Okay, that’s the spice on hold…for now. Let’s talk shop.
By doing the basics in layers, you will catch threat actors. Yes, anything can be evaded, but not everything can be evaded quietly.
Defense evasion is a fascinating conversation we’re having as a community. There are some incredible innovative red team techniques that allow defense evasion to be ‘trivial.’ But something that makes me chuckle is that a significant number of these evasion techniques rely on bringing a malicious file on the target machine, and moreover, that these techniques have a noisy impact.
The adversary may get into your network quietly, and they maybe are even running commands sneakily, but from the telemetry we see at Huntress, I can assure you that they are not evading the impact of their tactics.
So to be simple, there are two opportunities here:
- Monitor for malicious files being dropped on disk
- Monitor for and detect system and account changes that do not make sense
The latter is actually easier, as your organization will have files flooding in and out every day, whereas the number of accounts, privileged users and systems with disabled defenses are less animated in comparison. The latter seeks to focus on the behaviours and techniques of the adversary that do not align with the routine and expected activity of your environment.
To monitor and detect impact, you will need appropriate teams and appropriate tools too. I’m not a hypocrite who just wrote an essay bashing security products; appropriate tools don’t mean expensive.
There are plenty of fantastic affordable or open-source technologies that will allow you to gather, query and alert on security telemetry in your network (like Sysmon, Wazuh, Elastic’s stack, and I’mma include Huntress Neighbourhood Watch because it’s dope).
How We Roll at Huntress
To share how Huntress’ tools consider defense evasion, I asked one of the architects of Huntress’ Process Insights, Principal Software Engineer Rob Noeth:
We firmly hold to the three principles: persistence, processes, network, and we've build that into Rio.
Instead of playing the never-ending games of binary signatures or trying to sample every API call happening on a system, we look for behaviors and techniques used by attackers.
We know they're going to have to run something. It could be malware and/or lolbas stuff. This is usually easy to find and alert on. But what if they're using more sophisticated techniques? That's when network/persistence comes in. The attackers must talk on the wire to either remotely control the host, or have the beaconing implant phone home.
I truly believe using these techniques, we could catch any intrusion. The tricky part is doing it at scale and collecting just the right amount of information. Too little and we miss key indicators, too much and our storage costs get too high.
Being able to do just-in-time forensics is appealing, but it's wayyyy too much data. Some EDRs try to do this and they end up dumping all of that data on their customers to make sense of. Our approach is to build the story for our customers and present it to them.
At the end of the day, folks just want to get the malware off of their systems and get back to work, so that's what we're aiming for.
<end rant>
The Specifics of Monitoring
If you’ve got some of the basics right, you’re ready to deploy a mature security apparatus. It will take some trial and error, some testing of configurations, and maybe you’ll get a false positive or two…but if you deploy your own security stack, you, my friend, could have security autarky.
Imagine being free from security vendors’ pitches??! I’ll give you some suggestions, which you’re free to ignore, but you have nothing to lose but your chains.
Now, Sysmon isn’t infallible, but it ain’t bad when it costs a whole $0.00. And application allow/deny listing isn't too shabby when you can deploy via Group Policy. And what about giving PowerShell a lil shackling down, too, while you’re here? Whilst you’re at it, would you give your Event Logs some love from me? [1, 2 ,3]. Switch some of these bad boys on and you’ve just imposed some cost on an adversary by making them sweat in a network hostile to their presence.
The below is an amalgamation of different documentation on PIM, PAM, tiering architecture, professional experience, and a good amount of ‘licking my finger and raising it in the air. Don’t think of this as architecture advice, just more of a conceptual way of thinking about the machine’s importance relative to the defense evasion implementations worth making.
Tier 0 (nuclear level protection)
DCs
ADFS servers
ADCS servers
Internet-facing servers
Backup servers
Sysmon
AV alerts
Allow-listing (hostile*)
WEC and WEF
Proactive hunting
Intrusion detection systems
Tier 1 (important)
File servers**
Print servers
Legacy tech***
Remaining servers
Sysmon
Expand log retention
AV alerts
Allow-listing (strict)
Proactive hunting
Tier 2 (every day)Users' workstations
AV alerts
Allow-listing (permissive)
* By hostile, we mean HOSTILE. Lock down PwSh with as many different restrictions as the Good Lord will provide.
** If you’re able to segregate your file servers, your super important files (intellectual property, sensitive data) could be on a totally different file server that gets Tier 0 protection. Meanwhile, other more mundane files can exist in a Tier 1 file server.
*** Legacy machines are hard to offer cutting-edge security protections. But you can still hack workarounds, so if you only have a Windows XP machine because it helps run your manufacturing factory floor, then nerf and neutralize the machine so it will only run the prescribed software. Even better if you can ensure it isn’t domain and/or network joined, too.
All of the above can be subverted and evaded, but the point of having these layers of security is that you’re raising the ‘cost’ an adversary must spend to evade and persist.
With the defense in depth suggested here, you’re forcing the threat actor to flawlessly execute a herculean gymnastic routine without once tripping over the constellation of security obstacles you have laid in wait for them.
Proactive Security
My colleague Anthony Smith already schooled us on proactive security through the vehicle of threat hunting, where you go beyond needing alerts and leverage your telemetry to identify malicious activity not yet detected in your environment.
This falls under the philosophy of assuming compromise. Assume it has happened, is happening, and will happen. Now it’s not me trying to scaremonger you. Rather, by assuming there is some kind of malicious activity reverberating in the corners of your network somewhere, you can begin to deploy some seriously advanced proactive security measures, which will foil 99% of adversaries who enter your network.
Now Anthony gave us the skinny on threat hunting, and if you’re in need of a philosopher for threat hunting, Chris Sanders drops some serious threat hunting knowledge via his Twitter account. Having a strong conceptual foundation that drives your threat hunting regime is more important than anything.
And whilst you’re thinking about what Chris Sanders has to say about threat hunting, have a little gander at what he has to say about laying traps for your enemies. These traps will alert the defenders to an adversary trying to pull something sneaky off. ‘Honey’-based proactive security has really taken off in the last couple years, with fine gentlemen like Chris Sanders, Casey Smith, and Ippsec [1,2,3,4] giving us an edumacation about how to deploy our own honey-based security projects. Even I have been swept up in all the fun.
We’re going beyond the remit of most security implementations that are reactive—i.e., wait for an alert from a product to highlight something malicious - and instead moving to a proactive posture for security, where you’re continuously hunting and trawling for the more advanced bad guys who have managed to persist.
Human capital is paramount here. A great security investigator can deploy solutions for free, whereas expensive products cannot augment a non-security person into an investigator, no matter the promise behind the product. I’m repeating myself, but it bears repeating really: your security maturity is reliant on people, not products.
Testing Your Strategy Ahead of Time
Whilst hunting and honeypots are great, those are what you do when times are good. But when times are good, you also need to kick the tires and sharpen the swords for when times are bad. Businesses have their contingency and disaster plans for floods, fires, earthquakes….so y u no have cyberattack recovery plan?
I cannot emphasize it enough: have a plan for when you catch evil:
- Practice containing a threat
- Isolating a machine
- Manually and granularly eject a threat actor controlling an account,
- Restore from a backup every now and again on a machine to check those things actually work and how long they take, PLEASE
- Test security product X, or outsourced SOC Y
I wouldn’t suggest doing it all the time every day, but a healthy test of your brake pedal as you’re pulling out of your driveway is better than finding out your brakes aren’t working when you’re on the highway.
What did you learn when you did this? Are there some servers that you cannot isolate as it will have a huge operational impact, a particular legacy machine too old for a security solution? Learning these things now means you have time to think of an answer.
The aim of this objective is not to identify flaws in product X or team Y. The aim is to iteratively emulate the adversary in your own environment, to identify the gaps in your layers of defence where malicious evasion could thrive.
Don’t do red-team vs. blue-team either—it’s weird and anyone selling you that is weird.
Red teams are skilled operators who deserve more than just aiming for the goal to ‘own’ your network. Instead, blue and red teams should work in collaboration, firing off offensive security tooling and iteratively working with the blue team to work out: why did something not get detected, how can we tune this detection better to tune out false positives? What gaps do we have to accept due to limitations (in cost, time, storage, personnel)?
Most companies don’t need to threat model for APTs; they need to threat model for standard adversaries and insider threats. Many red team operators tell me that X or Y tool will evade a defense if it was used in a certain way [1, 2, 3]. From the huge telemetry source we have at Huntress from everyday networks, adversaries are deploying tools in the less sophisticated, easier-to-detect way—and they still win, even deploying it out of the box. My red team friends spend more time exploring the nuances of a tool, and they do not consider that most adversaries don’t use the full flexibility of offensive security tools. Catching cyber criminal bad guys isn’t hard; it just requires upfront planning, preparation and investment.
By the way, a real security incident isn’t just the responsibility and remit of IT and security folk. Domain intrusions can and will impact an entire organization, and representatives from the entire organization need to buy in.
Use tabletop incident simulation. Don’t go for worst-case scenario, but instead have different tiers of attack: the insider threat may be one tier, the standard threat actor, and then the sophisticated APT. Tabletop scenarios allow you to test your cyber resilience and recovery plan, and let you have some fun whilst doing it! See Backdoors & Breaches, from Black Hills Information Security (Huntress expansion pack coming soon).
Proactively addressing your security strategies and flaws means that you can fix and mitigate them. Don’t let a ransomware campaign be the most expensive security consultancy fee you’ve ever paid.
Achieving Defense in Depth
Things can be bypassed and evaded. The point is not to construct a single impenetrable obstacle—that is a quest doomed to fail. Instead, it’s about building multiple security obstacles of varying complexities and textures. The constellation of components that comprise your security apparatus are investments that allow the security team more opportunities to catch, halt and eject an adversarial campaign before they can exact malicious impact.
But you cannot purchase your way into defense in depth. Neither can you ‘tech’ your way into it.
The core layers of your defence in depth may be purely technical—backups, for example. But as you traverse from the core to the outer layers, you need to ensure that the human context is the first lens with which you’re analyzing the efficacy of a security layer.
To briefly illustrate the importance of the human context in your outer layers of defence, take Uber. Uber was not compromised with a super leet zero day, but from user interaction; the MFA solution technically worked so there was no flaw or exploit there. Rather, the adversary deployed MFA exhaustion until the user slipped up.
If you do not think about your users when you’re layering your security, you’re setting yourself up for failure.
***
I hope this defense evasion series has given you a crash course in what defense evasion is, what advanced adversarial techniques look like, and has offered actionable suggestions of what ‘good security' looks like.
Now we’re all a bit more comfortable with how bad guys do their thing, I hope you can pick up some of the threads of conversation here and continue them in your own organization to deploy security.
If you’re keen on more Huntress thoughts on implementing and doing security real good, check out our Tradecraft Tuesday webinar series. We’ve an archive of great material, with a great 2023 programme lined up to spoil you rotten with all things shady and secure.
