The Verizon DBIR found that about half of all breaches involved legitimate credentials. It’s a huge attack surface that we’re only starting to get a handle of. So where are we in terms of monitoring anomalous behavior of our users? And how are new AI-based tools helping us to scale efforts?
Check out this post for the discussion that is the basis of our conversation on this week’s episode co-hosted by me, David Spark (@dspark), the producer of CISO Series, and Steve Zalewski. Joining me is our sponsored guest, Adam Koblentz, field CTO, Reveal Security.
Got feedback? Join the conversation on LinkedIn.
Huge thanks to our sponsor, Reveal Security
Full Transcript
[David Spark] Threat actors abusing legitimate credentials is old news. We’ve heard it before – threat actors don’t break in, they log in. But we’re still struggling to understand what happens after they’re in our networks. What’s working to track them, and where do we need to improve?
[Voiceover] You’re listening to Defense in Depth.
[David Spark] Welcome to Defense in Depth. My name is David Spark. I am the producer of the CISO Series, and joining me as my cohost for this wonderful episode… We were talking about how we don’t know how to make bad episodes. It’s Steve Zalewski. And Steve doesn’t know how to do this either, do you, Steve?
[Steve Zalewski] Absolutely. Hello, audience.
[David Spark] That’s Steve talking to you, the audience. Our sponsor for today’s episode…brand new sponsor of the CISO Series. We’re thrilled to have them onboard. It is Reveal Security – detect and stop identity attacks in your enterprise applications. And in fact, that’s what we’re going to talk about today.
And in fact, our guest is from our sponsor, and he has some very unique insight on that. But let’s talk about this topic first, Steve, before I introduce our guest. The Verizon DBIR, or the Data Breach Investigation Report, found that about half of all breaches involved legitimate credentials. This is, by the way, probably one of the most quoted stats, and people kind of all know this one.
Malicious actors rooting around our networks with legitimate credentials is an enormous attack surface. It’s like known, and everyone realizing it’s happening. So, the answer is often tracking anomalous behavior. This is an area that’s constantly evolving. My hope is, given the jump in AI, we’re seeing sort of a hockey stick curve in improvement here.
Where are we with these issues, and do you think we have improved as of recent with the rise of AI?
[Steve Zalewski] Yeah, this is going to be an interesting conversation. What I would say to lead off is I think the bad guys are being able to leverage AI way better than we are at this point to create the anomalous identity than we are to be able to find the anomalous identity. The state of the industry for us is still very nascent.
User based analytics and everything for the last five to seven years have been coming along, but gen AI has really thrown a wrench into the speed with which we need to innovate.
[David Spark] Very good point. Well, very excited to have our guest. It’s actually someone from our sponsor, Reveal Security. This person is the field CTO… And, by the way, prior to this, he’s been making a lot of jokes about doing crops. All right? So, we’ll find out if he actually does any farming himself.
But he is the field CTO…
[Steve Zalewski] That’s right. It was kind of corny when he was doing it.
[David Spark] Oh, that one hurt.
[Laughter]
[David Spark] He is the field CTO over Reveal Security. None other than Adam Koblentz. Adam, thank you so much for joining us.
[Adam Koblentz] Thank you for having me. I’m really excited to be here, and I do have many tractors, and hos, and things I do in the field.
What are the elements that make a great solution?
3:17.406
[David Spark] Joshua Neil said, “All rare events are NOT malicious. They’re just rare. The first wave of anomaly detection suffered terribly from false positives as a result. This led to anomaly detection having better RECALL, but worse PRECISION, than rule-based detectors. Anomalies can be resolved as either malicious or benign through CONTEXT…
I believe the way we unlock anomaly detection is via the automation of the gaining of that context.” Gene McGowan Jr. of SANS Institute said, “Context matters, yes. But quality context matters even more. When you merge your anomaly (noisy-noise) with poor quality context (more noisy-noise) you get more noise.
When you merge quality context with your noise, you improve your signal to noise ratio.” And Jason Keirstead of Cyware said, “The key with anomaly detection is finding normal for a user. It is not impossible to create user profiles that have low false positive rates. Context is king.” So, I just heard context.
I heard context. And then you know what after that I heard, Steve? I heard context. So, we hear that, but that word, context, has an extraordinarily broad definition as to levels of context, isn’t it?
[Steve Zalewski] Yes. And the other thing you saw is we know how to fix the problem. Three people just said, “This is what you have to do.” But clearly knowing that we have to do it and our ability to do it, there seems to be a gap.
[David Spark] Yes. I’m just going to double down on this context thing. Do you think we’re getting better understanding of context with new tools, or is it kind of the same thing we’re still struggling?
[Steve Zalewski] What we’re doing is understanding the breadth of the context so that we’re not just thinking about, “How can I be sure that the authentication is factual?” But that the context we need has to move to authorization as well and that that context historically might have been just geographic.
Some primitive things. And now we’re realizing that it has to be much more comprehensive to understand how the identity operates to get to a point in time, not just at a point in time, what do I know about you.
[David Spark] All right, I am throwing this to you, our farmer, Mr. Adam Koblentz. What is the context story today that we didn’t have just two years ago?
[Adam Koblentz] The reality is these guys are all right. Everyone’s comment on this is accurate in terms of, you know what, if I knew everything that was bad in every application, and I knew everything that was wrong, and I could write a rule for everything that I thought was actually a problem, yeah, that’s totally right.
You will have more precision if you know exactly what you’re looking for than you would otherwise. Contextually, though, we have a lot of challenges because now we have work from home, more contractors, more etc. that are possibly going to be in different places. But [Inaudible 00:06:52] just because someone is somewhere doesn’t mean that they’re bad.
How much do I travel for work? A ton. Should my company be alerted every time that I log in from not my house? No. And that’s the real challenge. Context is changing.
[David Spark] Let me ask you – are there different signals that we’re able to measure than before? I know there have been tools out there to follow people’s typing pattern, if you will, and that people’s typing pattern actually has a signature to it. So, I’m just using that as an example, but are there other things that have a clear signature that even if a criminal tried their best, they’re just not going to be able to match this fingerprint?
[Adam Koblentz] Yeah, absolutely. Let’s take, for example, one of our customers where we found a locally sourced crime group had leveraged somebody, and they were doing things differently than people normally do things. What they were doing was every day after the person would go home, they would log in.
They’d get the MFA code forwarded to them. Then they would read emails sequentially to try to look for customer financial information in the inbox. Have you ever done that? Have you ever sat there and read your inbox for an entire day, item by item? Like, no. That’s not how humans do things.
What’s the best tool for the job?
8:22.057
[David Spark] Erik Bloch of Atlassian said, “LLMs can add context but can’t yet reason. Today when this happens, almost always there is an alert, but it’s after the fact, and the investigation turns up it was a ‘log in’ versus a ‘break in.’ It’s not vice versa, as every human is unique and is an anomaly when compared to anyone else using LLM models, etc.” Yeah, so for example… Yeah, that’s a perfect example.
Steve and I are just humans. And so far, I can tell Adam is also a human. And yet we all have unique fingerprints in our usage. I don’t behave just like Steve or Adam does, but yet we’re all legitimate users. Then when you think about it, the criminal is not a legitimate user but a user of some sort that behaves like a human, assuming they’re not using a bot.
Going on with the quotes, Chris Bates of SentinelOne said, “The best way is to use automation and a trusted channel to verify if they knowingly did the behavior with the user. An automated Slack message to them, ‘Hey, Bob. We got alerted that another factor was added to your MFA. If you did add the factor, please click yes.
If you didn’t or don’t know, click no.’ This helps us identify possibly attackers.” So, I kind of think this is an interesting thing, is like you can track all this behavior and be really, really sure that something fishy is going on, but it’d be nice if you had a second factor as a sort of SOC analyst to know, “I’d like to know from another human.
Could I get another set of eyes on my work here?” By the way, I’m going to ask you a little bit about Reveal Security. Could Reveal Security allow me to do something like that?
[Adam Koblentz] Yeah. So, it’s really important… You brought up a good point here, right? Like the how do you leverage AI and ML to do what you want to do from a defensive standpoint. Identifying anomalous behaviors, possibly malicious, hopefully not, and find those. I got to tell you, I was sitting at RSA last year when all the ChatGPT stuff started hitting the market.
And I remember, there was a blog post from a vendor I’m not going to name who said, “You can now ask us, ‘Am I safe from China?’” And I was sitting there with a bunch of CISOs, and we were just cracking up because it’s like, “How do you possibly ask anyone that? That’s unreasonable. That’s ridiculous.” To think that threat actors don’t have VPNs…
That’s a legacy way of thinking about any of this stuff. We got to look at this from the perspective of I have a credentialed user who is doing things that they’re permissed to do. Is that okay? I don’t know. Let’s find out. Are they doing their jobs normally, or are they not? I have no idea. But I’m trying to find people leveraging Steve’s account, your account who have permissions that are much grander than they should possibly and find them doing weird things.
Weird could be bad. It could be a compliance issue. It could be, “I was lazy today.” But the point is that we’re trying to find the weirdness. We’re trying to find the anomalies. That’s where you’re going to pull the strings and be the real meat.
[David Spark] And you can’t pre-predict human behavior, can you, Steve? This is why we so have to lean on these models. Because there’s just an infinite number of possibilities of what someone can do. And trying to guess ahead of time will only get you so far. There’s just not enough time in the day to figure all those out.
Steve?
[Steve Zalewski] So, David, what I would say is what we have is a digital twin problem here, which is it’s not that I need to know more about you to know it’s you. It’s what happens is I have a digital twin that can be exactly like you because I don’t see you. So, visually, knowing what we know about you, our challenge is the fact that the digital twins have gotten so good they are indistinguishable from you.
And so when we determine that the digital twin isn’t quite right, one way that we do that is we look further back in time for the behavior that the human is doing and seeing if the digital twin followed the same behavior over time. Then once we do that, we end up with the challenge of, “But then how do we actually do verification to the physical version of that entity when I only have a digital connection?”
[David Spark] You know what I’m envisioning here? And we’ve seen many bad television shows do this where they have the evil twin. I’m thinking like an episode of “Knight Rider,” if you will.
[Adam Koblentz] [Laughs]
[David Spark] And you have the good David Hasselhoff, and you have the bad David Hasselhoff. And the two David Hasslehoffs are trying to say, “No, I’m Michael Knight,” I believe is the character’s name. “I’m really Michael Knight.” “No, I’m really Michael Knight.” And you have to figure out which one is.
And they both look very similar, but this is where the specific questions can come in. So, I’m going to throw to you, Adam. You’re like, “All right, I know one of these is wrong. It’s not acting. How do I verify? What is my…? So, what would be my next action? I’m learning this. What’s my next action here?”
[Adam Koblentz] That’s a great question. You think about this… The way I think about this, the worst example from a CISO’s perspective or a SOC [Inaudible 00:14:18] perspective of the evil twin problem is an admin account or an admin identity that’s not being leveraged. The challenge here is you have to compare them to others and their peers.
It’s not only what did this admin do, but what did their peers do. Because you have things like business processes that are going to be driving a lot of the behavior that a threat actor wouldn’t do or know about. It could be something as simple as ticketing work that is not done on the other side of the activity that would drive that.
[Steve Zalewski] See, and that’s the extension of context. See, now we’re getting at when we talk about context and know where you are, this is what I mean by looking back in time is to say, “What is the business process?” Another way to look at it is, “Hey, how do I know it’s really David Spark at nine o’clock in the morning?” So, what I’m going to do is look back to 8 o’clock last night and see if he was watching his favorite TV show and then turned the lights off at 11 o’clock when he normally went to bed to know that this is really David at nine o’clock the next morning.
Sponsor – Reveal Security
15:29.322
[David Spark] Before we go on any further, let me tell you about our absolutely awesome sponsor, Reveal Security. Reveal Security detects identity threats in and across your SaaS applications and cloud surfaces. That’s where they operate. So, in the case where an attacker leverages stolen credentials… We’ve seen it before.
It’s what we’re talking about. They bypass your preventative identity controls like MFA or PAM and they enter your SaaS apps or cloud surfaces environment. Would you be able to detect and stop them? Do you know what your employees or your admins are doing inside your SaaS apps and cloud services after the point of login?
We have this classic metaphor of like the candy bar. It’s hard on the inside but gooey on the inside, so we’re talking about the gooey on the inside place, which we want to make a little bit more hard, I guess. Chances are you don’t. And therein lies the risk. Reveal Security uses a patented unsupervised machine learning algorithm to continuously monitor and validate the behavior of human and machine identities to quickly and accurately alert on suspicious activity.
You can check them out. The website is really easy to find. It’s reveal.security. That’s it. Go check it out.
What’s our visibility into this problem?
16:54.032
[David Spark] Randall Hettinger of Permiso Security said, “Security teams don’t have visibility into attacks against the identity control plane. Advanced threat actors excel at bypassing security measures. Organizations need a comprehensive threat detection approach supported by a robust library of detection signals starting at the identity provider level.” And Ryan Franklin over at Amazon said, “Credential theft is a serious problem, and strong multifactor remains a successful mechanism to deter these attacks.
That still leaves service exploitation as a viable entry point.” So, I mean, these guys are just summarizing what we have been talking about all through this episode. I want to, though, open this up to you, Adam, to explain like what I have seen is this story of what are they doing once they’re inside…this is not a new story.
And us trying to figure this out is not new. What I’m interested to know is what is Reveal doing different today that was not being done before?
[Adam Koblentz] Let’s take, for example… You and I go to the same grocery store, the one I go to every week. Same shopping list. Okay? We’re going to have different journeys. Now, I know I park in roughly the same part of the parking lot. I can have four or five different main journeys through the store.
I have minor deviations here and there. And the store, the company, has spent a lot of money on trying to make sure they optimize getting money out of me. So, there is external things and internal things that are driving what I do in my journey. In application, it’s the same. Whether it’s the 365 Suite, or Okta, and OneDrive, and Box, and whatever.
The point is that I have some things that I do, how I use my applications and how I use my permissions, and then the companies that I’m a member of have their own processes that are the external guardrails that help drive some of my journeys.
[David Spark] So, you’re looking at the matching between the two?
[Adam Koblentz] Correct.
[David Spark] And not only that, you’re looking at all the employees. Because all employees behave in the same model. Even though Steve and I say we work for the same company, we don’t do the same things. We have sort of similar goals, if you will.
[Adam Koblentz] Correct, yeah. If you were to put a camera looking down from the ceiling of the grocery store that we’re all in, you’re going to find that over a period of time there are a handful of normal journeys that happen through this process. And the reality is we’re not saying that, “Oh, man, David sent too many emails today or downloaded too much data today.
That’s a problem.” What we’re saying is that, “This is not David. This is not David at all, because we can tell that he is not doing any of the things that David normally does. This is a net new situation post auth that we don’t want.”
[David Spark] Okay, so let me just ask, in terms of just mechanics here, let’s say you do identify then. Is there a situation where you’re 100% certain, “That’s not David. Shutting this off immediately. We’re kicking this person out.” Or is there a situation of, “I’m 90 to 95% sure it’s not David, but I don’t want to kick this person out.
I want to make sure. Do we send the legitimate David a message going, ‘Are you currently in the app,’” kind of a thing? Can you choose those two options, I guess is my question.
[Adam Koblentz] Yeah, absolutely. And it’s a good point, because you can imagine locking someone out, especially if they’re an executive or something, it would be a real big problem. But maybe you want to de-privilege them. Maybe you want to say that, “Okay, look, you’re an admin, or you’re a C-level.
But maybe today, right now, for the next couple hours, we have some more handholding, and we want to cut you down from your ten level of access down to a five until we can figure out what’s going on with this because there is a lot of risk.”
[David Spark] That’s a good point. All right, Steve, how…? Because it’s such a ludicrously complicated problem, and having this kind of information to educate you at these times is kind of critical. Because before, it seems like you probably were shooting a lot from the hip, yes?
[Steve Zalewski] So, I’m going to go Maverick thinking here for a minute. So, I think this problem is actually the malware problem just being solved again. Here’s why I say that – which is the way we’ve tackled identity and access management is we have hard authentication, and then we do least privilege.
When we looked at malware way back when, how did we handle malicious malware detonation? Well, somebody had to be first. Then we would analyze it, and then we would send out to everybody else, “This is what that bad file looked like.” That’s where we are today.
That’s in essence what we’re doing. And how did we solve the malicious problem? We used machine learning. We looked at a file not to detonate it but to understand enough context about the contents of that file to determine that it was likely malicious and block it. Well, what we’re doing here is the same thing.
We’re now understanding that a human is made up not of a password to do hard authentication and least privilege, but he’s made or she is made up of a whole bunch of pieces of data that taken together to correlate through machine learning allows us to solve a problem. And we know we can because it’s very similar to malware as long as you look at it in that light.
Where does this effort fall flat?
22:48.537
[David Spark] David Movshovitz of Reveal Security, in fact one of the founders, said, “Detection solutions are mostly focused on malicious activities at the accessor or CASB, network infrastructure, NDR, and operating systems layers, EDR/XDR. UEBA have been particularly effective due to major commonalities in the network, device, and user access layers.
When it comes to the business applications, models have been developed only for a limited set of application layer scenarios.” So, it sounds like what your colleague is saying, Adam, is that we have actually some solutions but not in the cloud at these applications, and I think it’s because, correct me if I’m wrong, there’s a lot of obfuscation that’s going on in the cloud, and this has been kind of a number one of concern of what can we see is happening in the cloud.
Do you think this is a concern, or is it just the cloud is newer, and that’s why we have less insight into it? What’s your take?
[Adam Koblentz] Cloud is not new. It’s just that it’s varied. I think that one of the big challenges that security teams are focused on is how do we not impact the business but also still protect the business. And as more and more applications and more and more data, business critical functions are moving from on prem to other applications, or the cloud, or whatever you want to say, the challenge really is how do you know what’s okay.
When I was in the EDR space or in the IR space, I could tell you with certainty from an OS perspective what’s okay and what’s not okay with rules.
But the challenge is how does someone like Steve tell his SOC analysts, “No, no, you need these 500 rules that we have to now write for these business processes and these applications that you guys don’t even know anything about and don’t understand, and we weren’t involved in the purchasing of.” And that’s a big challenge.
Because everyone understand the EDR. Everyone understands the NDR, or next gen firewalls, or whatever. But we are still in a serious education phase when it comes to understanding that application audit logs are no less valuable than things like firewall logs because of that. It’s a huge problem. And so far, you look at some of the UVA type solutions.
They have on their website… They’re really proud of themselves. “We have thousands of rules.” You know what that says to me? I have thousands of things to now maintain, turn off, edit, etc., and I have five people. Tell me more about how you’re helping me. That’s my perspective.
[David Spark] Well, this, I think, is where the big context thing… Like it’s great that you have thousands of rules. Can you boil it down to one piece of context or explain to me what the heck is going on? Steve?
[Steve Zalewski] So, I can’t resist. What we’re doing, historically, is we’re hunting for information. And what we are now realizing with XDR and what we were just talking about is let’s start farming all of the information that we have available instead of hunting for specific information. Because it’s in farming everything that we get what we need to know if this is good or not.
And as long as we continue to hunt, we’re never going to get there.
[Adam Koblentz] How dare you, sir. How dare you.
[Laughter]
[Steve Zalewski] I thought it was pretty good. It brought the whole farming thing home, okay?
[David Spark] What is the one way that says, “Hey, we’re getting ahead of this problem.” Because for years, we felt behind. And, again, we’re isolating it to the very problem of they got into the hard candy shell. They’re not in the gooey stuff. They’re swimming around. We know better, and we can act better.
That’s the key thing is we can act better than before. Where do you think that we have gained ground?
[Adam Koblentz] So, what I will say is… And I think Steve mentioned this earlier is the way that AI, generative AI, etc. is being leveraged, whether it’s by the threat actors or the defensive strategy blue teamers, the idea that we can leverage what we know about our own environments is really important.
Because if you’ve ever red teamed, ever been a hacker, ever been on the opposite of what we’re talking about right now, it’s always a black box. And the idea that we, as defenders, could take all of our data and know what’s normal to identify the needles in a stack of needles, that’s where we really are going to shine.
And we’re going to see more value in our ability to defend accurately with a high fidelity.
[Steve Zalewski] And I’ll say from my perspective, historically we can’t be wrong. We’ve been afraid to be wrong because of the consequence of not being able to login, for example. Well, we’ve moved on. Now we know it’s okay to be wrong because we’ve found ways to have incremental friction in the business process.
[David Spark] This is like going from a ten to five.
[Steve Zalewski] Right. And because we found ways to do incremental friction and we know it’s okay to be wrong, because we can’t be right all the time, that’s why we’re making progress.
[David Spark] We’re just pulling people off the button to set off the bomb essentially it what it is. It’s like when you get that close, I’m going to pull you back a little bit. That’s a really, really point. Incremental friction and taking it from a ten down to a five. A great summary of our discussion.
Closing
29:10.977
[David Spark] Now we come to the point of the discussion… And you’re going to go first, Adam. Tell me which quote was your favorite and why.
[Adam Koblentz] I’m partial to David’s quote, just because…
[David Spark] Well, because he pays your salary. Go ahead.
[Adam Koblentz] Yeah, exactly.
[Laughter]
[Adam Koblentz] But I will also say that Joshua Neil had a great quote as well. Anomalies are not necessarily malicious, but they are important. And I think that Joshua did a great job of explaining that precision is definitely worse from an anomaly perspective versus rules. But I think that what he’s kind of getting at and I think he did in some other parts of the conversation was explaining the challenge of the precision for the rules.
Because, again, going back to the, okay, what are the 500 things my guys can do, or not do, or what’s allowed, that’s impossible. Because you’re not going to get the time from the business leaders to sit down with somebody and build those rules.
I think that Joshua is right about a lot of things, and he’s lived through a lot of the historical UEBA problems that people have had. Like Steve…UEBA for ten years has been promising the world and falling flat, and a lot of that is because of this kind of thing that Joshua is talking about. I think that anomalies post auth are much more interesting because now you know that either it’s account takeover, it’s a bad actor internally, or a compliance issue where the actual legitimate user just didn’t do the right thing today.
And those are all interesting to you and important to you but are not necessarily “malicious.” Like, I can’t guarantee a nation state is doing those things, but you should look into them.
[David Spark] All right, Steve, your favorite quote and why.
[Steve Zalewski] I’m going to go with Jason Keirstead of Cyware, and he said two things. He goes first, “The key with anomaly detection is finding normal for a user.” Well, we faced that problem now, and we are doing it. That was the first part of what he said. The second part of what he said is, “It’s not impossible to create user profiles that have low false positive rates.” And the key is, we can accept low false positive rates.
It doesn’t have to be no false positives rates. And so those two things, to me, are the key for the takeaways today, which is we don’t have to always be right, and we understand that creating normal for a user is the way to solve the problem.
[David Spark] Excellent. Well, that brings us to the very end of this very show, which has been a little supersized, but it was damn good. And that’s all I care about. I want a give huge thanks to our sponsor. That’s Reveal Security. Remember, they can help you with this very problem that we’ve been discussing right here.
reveal.secuity. That’s the easiest way you can find them. But, Adam, I’m going to let you have the very last word, if there’s any sort of callouts you’d like to make to the audience, how they can get in touch with you, any opportunities they can see with Reveal Security. Let us know.
[Adam Koblentz] Yeah, absolutely. You can find me on LinkedIn. You can find me on X, formerly Twitter, as I’m reminded constantly on Slack. Reach out to me, [email protected]. Our goal is to help you find these issues post auth that seem intractable otherwise.
[David Spark] Excellent. Well, thank you so much, Adam. Thank you very much, Steve. We’ll have a link to Adam’s…also his LinkedIn account on the blog post for today’s episode. And to our audience, as always, I greatly appreciate your contributions and for listening to Defense in Depth.
[Voiceover] We’ve reached the end of Defense in Depth. Make sure to subscribe so you don’t miss yet another hot topic in cyber security. This show thrives on your contributions. Please, write a review. Leave a comment on LinkedIn or on our site, cisoseries.com, where you’ll also see plenty of ways to participate including recording a question or a comment for the show.
If you’re interested in sponsoring the podcast, contact David Spark directly at [email protected]. Thank you for listening to Defense in Depth.