For four years in a row, Verizon’s DBIR, has touted compromised credentials as the top cause of data breaches. That means bad people are getting in yet appearing to be legitimate users. What are these malignant users doing inside our network? What are the techniques to both understand and allow for good yet thwart bad lateral movement?

Check out this post for the basis for our conversation on this week’s episode which features me, David Spark (@dspark), producer of CISO Series, co-host Steve Zalewski, and our sponsored guest Sandy Wenzel (@malwaremama), cybersecurity transformation engineer, VMware.

Got feedback? Join the conversation on LinkedIn.

Our sponsor for this week’s podcast is VMware

Full transcript

David Spark

For four years in a row, Verizon’s Data Breach Investigations Report has touted compromised credentials as the top cause of data breaches. That means bad people are getting in yet appearing to be legitimate users. What are these malignant users doing inside our network? What are the techniques to both understand and allow for this good movement in our network, yet thwart the bad lateral movement?

Voiceover

You are listening to Defense In Depth.

David Spark

Welcome to Defense In Depth. My name is David Spark. I am the producer of the CISO series and joining me for this episode of Steve Zalewski. Steve, let’s the hear the sound of your voice.

Steve Zalewski

Hello David.

David Spark

That’s the sound of Steve’s voice. You’ll hear it far more often during the show. Our sponsor for today’s episode is VMware. We are thrilled to have VMware on board. They’re been a phenomenal sponsor. Now, today’s topic, I am actually shocked that it’s taken us this long to get to this topic, because it is one of the most common problems that lots of different technologies are dealing with. And it is the subject of lateral movement. Now, I posted this question and I got a lot of extensive answers and everyone agreed there is no one answer. There are multiple tools and you have to attack it from the angles of both prevention and detection and response. And all the technologies were brought up in the discussion– segmentation, microsegmentation. PAM, IAM, honeypots, SSO, MFA, password managers, UEBA, NDR, SIEM, SOAR, EDR and a lot more. We’re not gonna focus too much on the technologies, but this issue in general and about what’s the most effective means to attack. Let me ask you, what is your sort of take when you saw this discussion Steve?

Steve Zalewski

So, I also right, posted a question on this and I posted it in a slightly different way and was interesting. And what I said was “Look, I’m getting frustrated, like a lot of people on this topic” and I tried to say “Folks, do we believe that this is a technology innovation issue?” Which is we have a laundry list of acronyms. Is there more necessary? Or is this a process issue that what we really have is there’s some foundational gap in our process, in our maturity or on how we’re using things or is it just unsolvable. And to be able to, again, tease out for this episode, what is really happening and what’s the acceptable middle ground right now.

David Spark

To help us go through this very thorny issue, I’m gonna say, and I think that’s kind of a good way to put it is our sponsor guest today. We’ve had her on the other show and she was phenomenal. I’m thrilled that she’s coming on our show at well. It is the Cybersecurity Transformation Engineer for VMware Sandy Wenzel. Sandy, thank you so much for joining us.

Sandy Wenzel

Yes, thank you gentlemen for inviting me back and Steve, I’m surprised you didn’t get the answer to Layer Eight problem or PEPCAQ. You know, the problem exists between the keyboard and the chair.

David Spark

A classic answer.

How do I start?

00:03:19:00

David Spark

Jonathan Waldrop of Insight Global said “Knowing what is normal behavior, so you can know when something isn’t working and enforcing the Principle of Least Privilege. ProTip: It’s not just for user identities. It goes for systems also.” Which is, by the way, I’m surprised he was the only one who mentioned that in this discussion. Michael Manrod, CISO over at Grand Canyon Education said “we need to see how users and accounts are behaving. These technologies typically employ a blend of Machine Learning to identify baselines and deviations as well as signatures or pre-defined bad things to look for.” So, this whole idea of what is normal behavior, that alone seems really difficult to determine doesn’t it Steve?

Steve Zalewski

Yes. And I think part of this is we still live in a world where we try to implement a hard edge to all of our soft middle.

David Spark

Explain to me what you mean by that?

Steve Zalewski

So, we still look on authentication versus authorization as our primary means of determining if we trust you. And at that point the trust is forever forward.

David Spark

Like the candy bar metaphor “Hard on the outside, chewy on the inside.”

Steve Zalewski

Chewy on the inside, meaning once you’re into our infrastructure, then we don’t have a whole lot of ways of determining if you’ve decided to go from good to bad, either accidentally or maliciously. And so, I think we’re still living with that paradigm, even though we’re talking about the transition from authentication to continuous authorization as the way forward.

David Spark

That’s a good point. We are still living in that paradigm, even though we don’t want to be. Do you feel the same way? I kind of think that’s happening, isn’t it Sandy?

Sandy Wenzel

It is, yeah. I mean, if you think about it, trust is a human emotion and it’s very, very exploitable, right? Like Steve said, once a user is authenticated in, no matter how, whether2FA, MFA or all the technologies that were mentioned before, once they’re in they’re in, right? Sometimes they’re asked to re-authenticate, sometimes keys are rotated, but that’s the issue today and I think all the points that were made are really, really good and I’m glad someone actually focused on, not only just users and human interactions, but house systems are talking to each other as well. But we need to take a huge step back and start with the basics. You know, what are you actually trying to protect and how do those assets fall into security paradigm in terms of priority? Do you under the risks in blast radius. So, understanding not only authentication, but how are the machines configured and being able to harden those configurations, then we can start to understand what are actual normal behaviors and what isn’t. You know, I see talk in there of ML, but if you drop ML right into a comprised environment because the industry is trying to tell us to assume breach in compromise, then those decision tree based bottles are already poisoned when it comes to establishing that baseline of what we consider normalcy.

David Spark

A good point. Are there certain parameters we should look at when determining normal behavior, or does what you describe as, you know, if you drop a machine learning algorithm into something that’s been compromised, well then you’ve just kind of messed yourself up. (A) How do you know you have a clean environment to determine normal behavior and when do you know you’ve got normal behavior? I mean, there’s so many sort of questions of what parameters am I watching? I mean, where does one even start here?

Steve Zalewski

The first thing I’m gonna say there, and this is what a lot of practitioners I think are thinking through correctly is, just start, okay? There is no place where we start with all known good data. It’s just not possible, because we never have a green field. So, what you want to do is simply let the tool start to work. You may have more false positives or false negatives than you want, okay, but what you’re doing immediately is you’re starting to clean it up on a go forward basis, and you’re starting to establish knowledge on subsets of identities and you keep going. So, don’t think about how hard it’s going to be to try to stop the world, to get it to a known good state to start, just jump right in and have low expectations on the outcome and then build the efficiency or the effectiveness of the outcome matrix.

David Spark

So, it’s better to start on bad than on nothing.

Steve Zalewski

Yes, right.

David Spark

Sandy, do you agree?

Sandy Wenzel

Oh, I absolutely agree and this is where you start your retrospective in looking at your processes as well. It’s do we have a gap in our process? Are there users out there or accounts that shouldn’t be because they’ve left the company on either bad terms or they’ve just left the company and moved on. The same thing with system accounts. You know, do the system accounts still need to exist? So, it’s starting to also take a look at your internal processes and making sure that they align up, or if they need to be updated because they’re old and you guys are following something else or some other guardrails, that you update them.

Steve Zalewski

Yes. And I’m gonna jump on that too. And if you are still of the mindset that you allow your users to have admin access because they think they need it, shame on you. Get rid of that and 90% of your problems are gonna go away that day and time has passed.

How would you handle the situation?

00:08:42:24

David Spark

Matt Black at Total Expert said “With so many SaaS platforms in use, it feels hard to separate lateral movement from credential stuffing. How do I reduce the likelihood of a compromised credential in system X leading to an issue in system Y? Especially if the user has a higher privilege in system Y.” And Paul Lanzi of Remediant, who works in the privileged access management space said “you’ve got to strip away all privileged access rights that the user isn’t using right now. This “just in time” and “zero standing privilege” approach mitigates the risk of lateral movement. Standing privileged access rights are like oxygen for attackers.” Sandy, you are nodding your head there. When you have standing privilege, like the beginning this candy bar metaphor when it’s like, oh, I’ve got admin rights forever? Wahoo! They don’t have much to worry about, right?

Sandy Wenzel

Right. They didn’t have much to worry about and we see that with, like, credential dumps that you find on the dark web or even just data dumps where those credentials still exist and they can literally use things like credential stuffing and gain access. So, we’re seen even numerous times over the years that adversaries are able to gain access with those stolen and still valid credentials. So, solutions and controls like with C2FA, tokens, bypass, it’s all done, so I could literally pay $16 to incept 2FA as mass requests and that’s not even needing the direct telephone number of the target.

David Spark

Wait! Where does the $16 go to?

Sandy Wenzel

Oh, it’s a service. It’s a service that incepts these SMS requests and basically duplicates them and sends them to my burner phone. That’s it. 16 bucks, that’s all I need.

David Spark

And you have $16, as I understand it?

Sandy Wenzel

Oh, of course, of course.

David Spark

As much as $17.

Sandy Wenzel

Yeah, $16.50. Sometimes you can negotiate with those folks. Maybe I can bring them down to $10.

David Spark

Oh well, there you go. Steve, what do you think of the issue of privileged access as the just in time issue?

Steve Zalewski

That is what continuous authorization means, right? Authentication versus continuous authorization. I have a slightly different way of describing this to somebody who is having a hard time about why this is right. I said for the last 15 or 20 years we’ve lived where every company. Sandra, myself, our companies are like a giant household. It’s one house with a front door and a back door and a whole bunch of bedrooms and living rooms and everything else, and what we try to do is make sure you’re allowed in the front door, right, and when you’re allowed the back door as a teenager. And inside we hide all the complexity about whether you’re allowed into certain rooms or not, okay, within the family. That’s what we do. You allow a boyfriend or a girlfriend in, right, we handle it inside and there’s the risk. Well, what we just did, whether you like it or not, with digital transformation and where we’re going is all of a sudden I’ve bought an apartment complex and everybody has their own room with their own door and now I have to allow everybody to move independently between all those rooms, with a lock on every door. How do I do that, because I can’t vet you into the house to then let you look at the second set of doors? So, every authentication is an authorization and it’s right on the front end and I can’t hide it anymore inside, right? That’s what we’re doing. But the fact that we’ve been forced to do that, it’s really hard now, right? And we are all struggling with what’s appropriate technologies given we’re not gonna go back to one big house per family.

What are we going to do now?

00:12:34:07

David Spark

Dan Desko of Echelon Risk + Cyber said “Do some threat modeling to understand how a threat actor might take advantage of the situation and then start to segment, ring fence, etc., at the port, protocol and service level.” And 

David Davis of COLSA said “What plagues many organizations is a fundamental lack of understanding of which systems / processes need to communicate and the level of accesses required to function. Always assume at some point you will be breached and explore the different paths available for lateral movement within your organization.” Sandy, good old fashioned threat modeling. Just understanding if your attackers are essentially doing this, shouldn’t you be doing it first to understand the paths they’re taking?

Sandy Wenzel

Oh yes, absolutely. Start being offensive and moving towards those offensive type strategies and being proactive rather than traditional reactions, right? It’s better for us to find those gaps than the actual attacker. So, understanding your threat landscape and doing that modeling, that’s fundamental. And all the comments I read about the technologies to use were great and very accurate, but they all have their relevance in point and time and this will be strongly based off of the maturity and direction of your security program. So, there is no single technology or vendor that will deploy all of the security things– like I saw Zero Trust in there a couple of times– into your network automatically and not need some sort of human intervention. So, absolutely we need to fix or strengthen the foundations and address the people and process problem before adding these more complex technologies that end up producing more logs, more data, more alerts for someone to sort through and prioritize and still need to be tuned.

David Spark

Steve, at the beginning you talked about if you got some admin rights out there that are unnecessary, lose them. This kind of speaks to it right there, I mean. But even at a more finite level, I mean, start to break it down to an even more refined level than just that. But, like what you said, at the very beginning, let’s just start there. That’ll solve a lot of problems. But what are we doing at defined levels?

Steve Zalewski

So, I agree with the threat modeling with Sandra and here’s why. You have to ask yourself, forget everything I know, right? So what, now what? Knowing that I now am in an apartment complex, how do I reevaluate my strategy to do the most effective job I can? Is it time to decommit some of these technologies? Is it time to re-prioritize based on your threat modeling? Like, up your security awareness campaign twice to what it is right now. Realizing, even if you can get it 2% better, it’s better than many of these technologies that you can’t figure out how to actually get to work effectively. Not efficiently, effectively. And so, when I look at this I say where it led me is active containment. So, what we really have got to get good at is the prevent is as good as it’s gonna get and it’s hard to get more prevent. Detect is what we want to do, but active contain is where we’ve really got to automate on this so that we can allow more bad things to happen in a way that minimizes the impact to the company. So get on the resiliency, because we’re all realizing this is gonna be a multi-year journey to completely reinvent from authentication to continuous authorization, so get on the containment and automated containment, so that we can move forward.

David Spark

But I want to focus on that journey comment that Steve just made Sandy. You have dealt with this journey yourself, correct?

Sandy Wenzel

Correct, yes.

David Spark

Let me ask you to just look back. Think about where you were at the time. Let’s just say the first time you were struggling dealing with this. Just think about where you were mentally and what you understood and think now, what is it you wish you knew back then that you know now in terms of beginning that journey? Because I’m sure you took a lot of missed steps at the beginning.

Sandy Wenzel

Yeah, absolutely. I mean, going back to Steve and his apartment complex building. If one of us acts up and needs to be grounded, we need to be quickly isolated so it doesn’t contaminate other people in the building. So, I think, for me personally, it’s been, you know, the journey of automation and being able to do this and trust in automation and that it’s doing the right things because I programed it too and I’m having it too, and not be so afraid of it.

David Spark

Did you have fear of automation when you started?

Sandy Wenzel

Oh, absolutely. It was more of, at the time, the Wild Wild West days were over. If you made a mistake and you isolated something or you did something or routes were lost, that was money lost to the company and people were, you now, that’s RGE résumé generating event. So, it was a way to do it or isolate and / or contain it, where business can still be conducted, yet in a safe space where if we feel the user or an entity was doing something malicious or has turned.

There must be a better solution.

00:17:28:19

David Spark

Christian Taillon of Grand Canyon Education said “While there are many options.” as I mentioned at the very beginning of the show. He goes on to say “Some solutions are more effective at combating this problem than others.”

And lastly, Mathew Biby of Satcom Direct, “until we get past the traditionally antiquated password use requirement and move towards passwordless technologies we will continue to struggle with lateral movement via credential compromise.” So, I’m gonna start with you Steve. Both of these just have to isolate on the technologies now as we’re coming to. Matthew thinks, if we could sort of in unison get onto passwordless or better authentication scheme we could solve a lot of these problems, because we wouldn’t have to worry nearly as much about lateral movement and then Christian just says well, there’s better technologies than others. So, let me ask you, to Matthew’s comment and Christian’s comment, what say you?

Steve Zalewski

I say I wish we were back ten years, when this conversation really was around human identities and identity and access management in its first phase. But what we’re really calling out here is we’re now onto generation two, where systems and processes are all identities and more and more, those are the things that are going wrong or that were having to be able to identify and passwords and system admin accounts are still the primary way to be able to create and enable those things. So, passwordless is the right idea, but stop thinking about passwordless for humans and let’s realize that identity and access management is now on generation two, where everything is an identity and so it’s not a passwordless problem. We’re actually having to go back and rethink some of the core capabilities to identify all those things, discover them and then go forward on the journey and apply some new technology.

David Spark

Sandy, I take it you can reference the passwordless discussion but also, I mean, what are the technologies that give you the biggest bang for the buck in this situation because, again, a lot were listed?

Sandy Wenzel

Yeah, a lot were listed and I feel these days we are no longer gatekeepers. We’re actually toolsmiths and advisors. So, again, just be forward. Technology is not going to solve a process problem and the more inundated and cumbersome you make it for a user by cobbling these technologies together, it actually starts to compound that people and process problem. People will seek other avenues of less constraint or the path of least resistance to accomplish their tasks. So, if you understand at the end of life of a developer and align it with tactics and techniques of an attacker, you will actually see a parallel or a trend that they actually have the same behaviors, just different motives. So this is when focusing on behaviors and relationships and looking at UEBA and ML, for example, are imperative when combating lateral movement.

David Spark

Have you gone through this exercise with your team? Like, in advertising you create these sort of, like, vision boards of, like, this is the target we’re trying to reach, but it was, like, this is the person who’s trying to attack us and this is what they’re trying to do? Do you go, like, let’s now walk through their process. Have you done this as an exercise?

Sandy Wenzel

Oh, so you’re thinking of, like, red teaming and how an adversary looks?

David Spark

Even before you go to the red teaming, like, let’s just walk through. What would they do? Because we referenced this earlier in the show. What would they do? What are the kinds of avenues they’d take advantage of here?

Sandy Wenzel

Yeah, absolutely. So, spirit fishing. That’s the very very common one, especially on mobile devices with people being remote. They’re not checking links or really being able to safeguard links. They’re just gonna click if they’re looking at it in their phone. So, that’s gonna be the biggest one is fishing and making sure users are aware of what that looks like and how that’s going to look. Even attachments or downloading those attachments. And then again, like I mentioned before, there’s data dumps on line that you can buy of compromised or data that’s been exfiltrated from previous campaigns higher and you can just buy a whole lot of credentials to try and do and try and us.

David Spark

Alright. I’m gonna throw this to both of you, because we did mention a ton of technologies and, by the way, I know this is not a technology problem. It’s important. It’s a people in process problem. But I’m still throwing this out. You’ve got three technologies you can use to do the most effective work on this problem of lateral movement. What are the first three you start with, Steve?

Steve Zalewski

Identity and access management program first and foremost. Start there. I can’t protect what I can’t see, right? I mean, if I don’t know who has access to what. So, let me get all over identity discovery. Second thing. Correlate it all with the UBA. What I want to know is how they’re having. Not whether they’re behaving well or not, but let’s get the identities and the behavior and then let’s automate with SOAR to figure out what to do when they misbehave. Those are the three.

David Spark

SOAR being the third? Okay. Sandy, you’ve got three technologies to use to get the most bang for you buck, what are you using?

Sandy Wenzel

Well, the first one I’m gonna use is SOUL decryption. I want to know exactly what those handshakes look like, what they’re doing. I want that visibility east west. I want it everywhere. So, that would be the first I am doing. Then I would probably move to a SIM product, because those actually have UEBA and SOAR capabilities built in, so I’m cheating there Steve. And then I’d start to look at XDR, which is gonna stitch together both EDR and my NDR. So, those are the three I would go with as my Swiss army knife of security things.

David Spark

You actually picked smartly, because you put two and three, because you picked the ones that sort of group a ton together.

Sandy Wenzel

Yes, like bind. Yes.

David Spark

There you go. Yes, she was thinking a lot smarter than you Steve.

Steve Zalewski

And that’s why we bring these folks, right? And so what we always say is, the best security is simple security, right? So what she did was she picked the right tools so that I didn’t have to manage more tools, because simple is better in security.

Close

00:23:31:00

David Spark

It always is. Alright, that brings us to the end of the show and this is what I always like to do at the end of the show is ask you what was your favorite quote and why? And Sandy I’ll begin with you. Which quote was your favorite and why?

Sandy Wenzel

I’m gonna go back to Dan Desko as my favorite quote because, not only does Dan address and acknowledge segmentation, but Dan is also providing practive advice in how you see yourself through an adversary’s eyes. So addressing what you need to know and deploying these countermeasures on how to better you security posture. So again, stop letting the adversaries be the ones to show you where the gaps are. It’s just too dang expensive. Gas and meat prices need to calm down.

David Spark

Alright. Good answer. I like it. Steve, what say you, your favorite quote and why?

Steve Zalewski

I will give it to Dan and Sandra. I think Dan Desko is an excellent idea, but I have to go with Paul Lanzi today, because I started with it from the beginning and I hammered right on through. Get rid of admin privileges everywhere, every way every how, because that is the lowest hanging fruit that most companies can get to is the more you hammer that down to zero that, to me, is the quickest so what, now what link.

David Spark

We asked you, do you think it’s possible that an organization can operate where no-one has admin privileges in the sense that it’s kind of like, to get anything done you might need a two key turn for two different people to operate? Do you think that’s even possible? Steve or Sandy, could that be pulled off where there’s sort of this tiered I’ve got access to that, you’ve got access to that. Like, no one person has the sort of God mode, if you will. Is it conceivable?

Sandy Wenzel

So, I’ve actually worked for a company that was close and it was more of on the security side. If I needed admin rights to any machine for any reason, it was a break glass situation where I had to submit all the appropriate change control requests. But once I used that password, it was a one time password. Meaning that once I used it, I logged in, it was done. It was deleted, it was recycled.

David Spark

So kind of like the just in time example that Paul Lanzi mentioned.

Sandy Wenzel

Exactly, the just in time. So, once I logged out, my administrative session was kaput. That same password would not work again. Although it was very cumbersome, I’m sure– again, this was ten plus years ago– so I’m sure there is a very automated way and a better way of doing this. But that’s probably the closest that I’ve seen and worked with.

David Spark

What about you Steve? Do you think this is conceivably possible?

Steve Zalewski

It is conceivably possible. Now break the glass, like Sandra said. There’s always got to be a way to get out, right, because you can’t think everything through and break the glass and one time passwords, right, this is the way we do it. There’s some maverick research that’s been around for a whole bunch of years that says what we do is we do a form of federated approvals. Which is we, if all the admins together agree that two admins have to say yes to get administrative privileges, then it can be any two admins, but that’s the defense in depth that way. So we don’t have break the glass, we have a delegated way of realizing who is able to make that decision and support that as the way of moving forward. Very maverick, but it would get us past the problem that we have.

David Spark

Good answer. I want to thank both of you very much and both of you get the last word here. But first I want to thank your company VMware for sponsoring this very episode of the podcast. Thank you so much VMware and VMware, again, have been phenomenal supporters of the CISO series, so we greatly appreciate that. Sandy, I’ll let you have the last word and, as always ask, please be able to answer the question, are you hiring? Steve, any last words?

Steve Zalewski

I would say, this episode, I think we really did a good job with the talk.

David Spark

So, you’re patting yourself on the back?

Steve Zalewski

No, I’m patting Sandy on the back because I think she had the best solution, so I’m patting my other co-host this time and simply saying it was really liked the way that she was able to succinctly bring the problem out, where we can do this so what now what? Thank you Sandra.

Sandy Wenzel

Yes. Thank you gentlemen.

David Spark

Alright Sandy. You get the last word and I want to also know if you’re hiring?

Sandy Wenzel

Absolutely. Always hiring good people.

David Spark

And also, by the way, any plugs for VMware you’d like to make?

Sandy Wenzel

Yes, absolutely. So, definitely for the NSX Security Business Unit, that’s the unit I work out of so, again, if you’re trying to social engineer and find me, that’s where you’ll find me. Again, I will be attending virtually for Defcon this year and all the different cons, so I wish well and good wishes and good luck to anyone doing any competitions and also to my black hat doc family who will be down in Vegas in the Mandalay Bay. Please give them my best David if you do go. And that’s it.

David Spark

Alright. Thank you very much. And thank you to our audience. As always, we greatly appreciate you contributions and listening to Defense In Depth.

Voiceover

We’ve reached the end of Defense in Depth. Make sure to subscribe so you don’t miss yet another hot topic in cybersecurity. This show thrives on your contributions. Please write a review, leave a comment on LinkedIn or on our site: CISOSeries.com where you’ll also see plenty of ways to participate, including recording a question or a comment for the show. If you’re interested in sponsoring the podcast, contact David Spark directly at David@Cisoseries.com. Thank you for listening to Defense in Depth.