Defense in Depth: Virtual Patching

What if you didn’t spend all your time patching vulnerabilities but instead created a security policy that prevented known vulnerabilities from being exploited. How doable is this solution of virtual patching?

Check out this post for the discussion that is the basis of our conversation on this week’s episode co-hosted by me, David Spark (@dspark), the producer of CISO Series, and Steve Zalewski. Our guest is Ody Lupescu, CISO, Ethos Life.

Got feedback? Join the conversation on LinkedIn.

Huge thanks to our sponsor, Araali Networks


Managing vulnerabilities at the speed and scale of the cloud is challenging, especially when the implications of a single mistake gives attackers an asymmetric advantage over defenders. Araali allows your security teams to resilient patch and monitor the most valuable apps and services so they cannot be exploited even if they are vulnerable.  To learn more, visit araali.

Full transcript

[David Spark] What if you didn’t spend all your time patching vulnerabilities but instead created a security policy that prevented known vulnerabilities from being exploited? How doable is this solution of virtual patching?

[Voiceover] You’re listening to Defense in Depth.

[David Spark] Welcome to Defense in Depth. My name is David Spark, I am the producer of the CISO Series. And joining me for this very episode is Steve Zalewski, who I don’t know if you know this, audience, used to be the CISO over at Levi Strauss and has been in cybersecurity for quite some time. He’s in a retired advisory-type position now and co-hosts this very show. Steve, let’s hear the sound of your voice.

[Steve Zalewski] Hello, everybody.

[David Spark] Everybody is listening, and they’re listening to this discussion today on virtual patching. And our sponsor, Araali Networks, lo and behold, does this thing of virtual patching of which we are going to discuss at great lengths on this episode. It is also known as “resilient patching.” They seem to be synonymous. If there is a difference, please let me know during the show, but that’s what I got from the online discussion which will hopefully be our discussion today. So, let’s talk about this for a second. Virtual patching is somewhat of a misnomer in that no code is actually beingpatchedor modified. But rather, there’s a layer that’s preventing the exploitation to ever reach the web application itself. It sounds really interesting in theory. My question to you, Steve, and which will be our discussion, your headline, can this actually be pulled off?

[Steve Zalewski] Yes. And when I heard about resilient patching from Araali Networks, that’s what caused me to talk about this, is really it’s are we doing mitigating controls or compensating controls, but I think what it keeps driving at is patchingas an underlying capability to just remove vulnerabilities. At scale, it’s very difficult, and that’s putting it nicely. It’s almost impossible in practicality. So, I think there’s this constant push now to try to look further up the stack, to be able to see if mitigating your compensation controls and the underlying technology are actually at a stage that we can consider it.

[David Spark] Thanks a lot. And this is going to be our discussion today. And the person that we have invited, in fact a good friend of yours, Steve, is Ody Lupescu who is a CISO over at Ethos Life. Ody, thank you so much for joining us.

[Ody Lupescu] Thanks for having me.

What’s the situation?

2:37.499

[David Spark] Harris Schwartz who’s the field CISO over at Elevate Security said, “Virtual patching or resilient patching is also known as just-in-time patching–an action taken, usually urgent or emergency-wise to block an exploit from being attacked but doesn’t mean it’s a solution over the actual patching of the issue.” And Abhishek Singh who’s with Araali Networks said, “Virtual patching is a mitigation for not being able to patch. The beauty of resilient patching is that it is permanent and forward-looking. It takes care of zero day and yet-to-be-disclosed vulnerabilities.”

So, this is very interesting. Both Harris and Abhishek’s comments are really kind of setting the stage for what virtual patchingis, and we’re going to talk about how far it does go and doesn’t go. Steve, do you think this is sort of a good opening explanation of it?

[Steve Zalewski] I do. And I want to actually argue that between Harris and Abhishek, they’re actually different, they’re not agreeing. What Harris is showing is what I would call the traditional view of where we were going with virtual patching. And what Abhishek is talking about is what the next generation looks like if you think about virtual patching as something done at the CASB layer. So, I think those two statements are actually excellent because it calls out a lot of the questions and discussions that we had on the post.

[David Spark] And Ody, is this how you’ve been viewing virtual patching and does it have a place in a security program?

[Ody Lupescu] My take is this–virtual patching is actually a new term to me. In the past 20-some-odd years, 24 years I’ve seen, it’s funny, it reminds me of George Carlin, the Euphemisms. It’s really vulnerability management or exposure management. What you’re doing with this patching, you’re essentially addressing a vulnerability. You’re not patching. Patching–the act of modifying the code to solve a potential defect. In this case, you’re not doing any of that so this isa vulnerability mitigation. And this has been around since, what, 2000? Even Symantec used to have, there was the whole idea of HIPS, Host Intrusion Prevention. This was why Palo Alto bought Cyvera back in, I don’t know, maybe seven, eight years ago. The idea was I can prevent exploitation of a known defect by keeping an eye on what’s happening on the system. Now they want to call it virtual patching but to me it’s not virtual patching. I mean, the term, I understand it’s used. It’s used the same way as Attack Surface Managementand a lot of others. So, it’s a great concept, it’s been around for quite some time.

[David Spark] Yeah, I think the wordpatchingis kind of the issue. As I said, it’s kind of a misnomer. It’s a mechanism, it’s a tool, but I just think the wordpatching is just the wrong word to use, right?

[Ody Lupescu] Absolutely. Because you’re not actually changing the code, you’re not patching the code. You’re actually putting something in front of it to prevent someone from exploiting it so it’s exploit prevention, if nothing else.

[David Spark] Let me throw this–this could be damaging. And by the way, I’ve seen this happen. This could be damaging to the category itself. If you can’t name things appropriately and they’re confusing, then people won’t use it or won’t know how to use it. But then again–and I’m throwing this out again as well–people like the concept of patching, of “fixing things” and maybe this is the right way to name it. Steve?

[Steve Zalewski] And that’s just it. Which was this may be a way–and I’m thinking about this out loud with you–is to be able to have the right conversation with your leadership team. Because when you talk about patching in its traditional sense, what I call physical patching, we are so sick of having those conversations and knowing why we can’t solve the problem, can’t solve it. So, therefore, somehow you have to be able to move the conversation into a direction where you can offer some risk management, some relief. And I think to a certain extent, yes, we are abusing the definition of patching but what we’re really trying to say is if you’re asked, “Are we safe? Are we secure?” that is the opening of aconversation.

And so vulnerability management is really enterprise vulnerability management, and that’s where we’re having this opportunity to introduce these concepts of virtual patching or resilient patching that actually provide mitigation. So, I completely agree and I think some of that conversation is absolutely true, but I would argue that’s why we’re talking about this, is Defense in Depth is is it a complex conversation made simple or is it a reality that the market has to move, when we have to adopt either this language or change the language to reflect the fact that we just can’t solve the underlying problem.

What aspects haven’t been considered?

7:36.457

[David Spark] Tony M. said, “Our IPS vendor does this automatically by creating rule sets as new vulnerabilities come out.” And Abhishek Singh of Araali Networks said, “Well, signature scanning with IPS has a upper bound on the number of patches it can handle. It can’t accommodate an unbounded pile of vulnerabilities–they do pile up over time!” Ofer Shaked said, “You must also be certain that you’re aware of all data flow paths into the target system to prevent someone bypassing your control.” That’s not easy–that’s me adding that. And Christophe Foulon of CapitalOne said, “Unfortunately, more organizations can’t define all the functions of their own applications, much less 3rd party applications to wall off their applications in that manner.”

So, this whole discussion we’re having of virtual resilient patching, whatever, it sounds really good in theory, but wow, it’s not so easy to pull off because you don’t know where everything’s coming in from. Yes, Ody?

[Ody Lupescu] Yes. That’s part of the problem, indeed. I think the problem is broader. Here’s the situation. So, you potentially know enough about your internal applications–potentially. And I’ll give you an example where this kind of resilient patching may run into problems. So, an example–I have an application, I developed it, it’s internal, I ran all the tools on earth against it, like your SaaS, your DaaS, and everything. I even have an API security in front of it. But guess what? At the end of the day, someone comes and is creative and finds a way around it. Can you potentially close all the gates basically to a garden? Can you wall off your garden completely? And I don’t think you can ever do that. I think you can help a lot but I don’t think you’ll ever be 100%. So, the garden is too complex, there’s too many land features, you’re not going to block all the way in.

So, I think the idea is, yes, this has a role into preventing. So, this virtual resilient patching, you can virtually prevent a certain type of attack, like an IDOR with your API security solution. You can do that, but don’t rely on it 100%. Nothing is infallible and this is just one of the tools in the toolset. So, yes, I think it helps. I think the whole idea of signatures, it helps to some degree, but none of them is going to be even 60 or 70% effective. So, at that point, you’ve got to go further down and see what else can you do in addition to it. So, it’s not one solution–I have a signature or I have some heuristic or I have some anomaly and therefore I’m going to be all good. That buys you something but not everything. So, that’s my two cents.

[David Spark] So, Ody, what you’re saying I think is kind of the theme of this discussion which was it’s definitely a tool in the toolset, like it is an example of defense in depth. But going back to what we were talking about in the previous segment, Steve, of the way it’s named, it sounds like it’s supposed to fix your problem but we’re seeing it’s really justa security layer and that’sit.

[Steve Zalewski] Yes. Bottom line–yes. And so like I said earlier, we’re using the names because we’re trying to move the conversation. And when you do that, you have to start in an area where people understand the current problem. And so when I look at this, right now what we’re coming to is virtual patching, resilient patching, it’s not layer three and four in the stack, it’s not port and protocol anymore. It’s now layer seven and layer eight. It’s now application vulnerability management and business application. Right?

So, what we’re really saying is we’re not trying to patch an underlying system or application. Now what we’re trying to do is patch the business application from a usage perspective. And that allows us to be able to use machine learning and a lot of the newer generation of capability to be able to do that real-time mitigation of zero day attacks, as opposed to simply looking at existing what I call physical vulnerabilities, and then doing one-time policy management at layer seven or layer eight. Now that’s kind of complex and I apologize for that, it’s harder to make it a little simpler. But for the security practitioners out there, they’ll understand that is the transition in the discussion that we’re actually having. Because the underlying vulnerability management problem is just getting worse with IoT and OT and business applications to the point where I have to find a way of managing the risk, when I can’t just do the underlying patching of all the systems to make the problems legitimately go away.

[Ody Lupescu] But Steve, I would point out this is something that while it helps in general, to me this is the fundamental aspect was buying time. I don’t want the fire drills. Like I remember back 20 years ago, we had the fire drills. The fire drills should go away. But ultimately, you have to go fix the problem. Patching doesn’t go away, you don’t want to leave applications outdated. You’re going to run into[Inaudible 00:13:04] problem in your resilience so you ultimately have to fix the patching, it doesn’t go away. You still have to patch, you just patch differently if you have some solutions in place. So, that’s my two cents.

[Steve Zalewski] And we’re going to stay on this a little longer, which was let’s talk about this, it really is the key–buying time. What this is doing really is buying you time so that you don’t have to use all your resources on an inefficient process. And buying time could buy you weeks or it could buy you months or it could buy you years. Because if you have legacy applications that you just cannot change, then this may be the way that it buys you two years until the legacy application is eventually retired. And so this buying of time has a pretty long tail.

[David Spark] And think about that. That’s a huge cost savings if you can buy time.

[Steve Zalewski] You got it.

[David Spark] It can be enormous.

[Steve Zalewski] It can be enormous, right? It’s saving time, it’s saving money. Think about this as an efficiency play, right? The resources that you have available, not just for the security team, but for your operations team, for everybody else that’s involved, testing to do this. So, buying time, the fact that you can do that from a risk perspective for security, has all these second-order positive effects which is why you want to pursue this.

Sponsor – Araali Networks

14:26.697

[Steve Prentice] Existing cloud infrastructure is rife with implicit trust that attackers abuse, making it less suitable for modern threats and modern work environments. Abhishek Singh is founder and CEO of Araali Networks. His company’s product focuses on resilient patching in which every workload can be individually segmented and made self-defending. He describes this asa form of offensive defense and he says, “This is something that cannot wait.”

[Abhishek Singh] Some people postpone this kind of maturity for later. But perfection isa thing that never happens so you have to start thinking resiliency day one. It’s not somethingyou wait until it becomes perfect and then you worry about what happens after your thing is coming in. So, if you think of how the zero trust as foundation was born, it was to change the mindset from trying to prevent what comes in to be resilient to attacks. So, if you try to prevent what comes in, you’reas weak as your weakest link and that’s a losing game for defenders. If as defenders you want to win that game, you have to be resilient, you have to play that polymorphic defense, the offensive defense game, and be resilient to an attacker in your environment. And that’s what it allows you to do–be resilient to making mistakes, being resilient to the fast pace of cloud. It allows you to be resilient to an intruder coming in and the impact is that it is dead on arrival. You are resilient in spite of all the complexities of operating in the cloud and that’s what we enable with a single click.

[Steve Prentice] For more information visit Araali Networks at araalinetworks.com.

What else is required?

16:05.250

[David Spark] Tony M. once again said, “I hesitate to say this is THE answer to physical patching but it’s definitely a really good compensating control that gives the organization some breathing room on planning emergency patch installation.” Literally, what you just said, Steve. And Ofer Shaked said, “Sometimes it can completely eliminate the risk, most of the time it’s just buying time.” Again, what you said. So, we’re just echoing what was said previously so since you already said it, I want your take on this, Ody. What does this buying time do?

[Ody Lupescu] So, what it does is essentially allows me to prioritize my work and it allows me to have some level of assurance that when something comes up, when I look at what happened, instead of going on this, as I mentioned the fire drill, I need to do something like Log4j. If I know that I basically can mitigate my Log4j exposure, then I’m not going to go on this crazy hunt. Basically, I’m going to prioritize this and I’m going to go fix it but I’m going to do it on my accord, and I have other things that maybe I need to worry about that are more important. So, that’s one aspect of things. So, yes. So, the timing is very important.

I wanted to point out that, by the way–and I meant to say this, not to digress–but we talked aboutIPS in a previous segment. What I find somewhat amusing is that in a way, that’s what IPS’s were supposed to do. It’s just that no one likes the term anymore, like DLP, IPS, those three-letter abbreviations. They fell out of favor so we’re searching for new terms and now it’s virtual patching.

[David Spark] And now we’re arguing the term itself. Steve, what were you about to say?

[Steve Zalewski] I was going to say Ody brings up a really good point and I want to riff on this. Which is, hey, he said Log4j, it’s in front of everybody right now, right? Log4j consumed almost every security organization for a minimum of three days, and some of them for two to three weeks, and yet while they’re doing that, other vulnerabilities are happening. Patch Tuesdays are happening. Depending upon your environments and what you’re getting from your testing teams. And so it’s to Ody’s point–I only have so many resources and the problem is getting worse every day on the number of vulnerabilities that are coming at me, that I have to be able to address.

And that gets back to where the virtual patching at the layer seven application layer buys me time. And then what we’re talking about here with resiliency, which is the ability to withstand attacks to my business processes. And vulnerabilities are underlying gaps that are exploited, right, that attacks happen. And I either learn from them through an attack or I learn through them because somebody else was attacked and I get to do this. But what Ody is saying is the number of vulnerabilities that we have to address are just getting larger and larger on a daily basis, and that’s why we’re being pushed into this corner to have these conversations.So, that buying of time, again, it is a force multiplier in the conversation for all of the reasons why we are struggling so hard with enterprise risk management.

Does anyone understand what’s going on?

19:25.113

[David Spark] Mathew Biby, CISO over at Satcom Direct said, “We saw a number of years ago the promise of WAFs and RASP technologies in this space but neither really became sustainable or scalable solutions to address the core issue of vulnerability management.” And Alex J. A. of Under Armour said, “It feels like most of us are doing this generically one way or another today.” Pretty much what you were just saying earlier, Ody. “When we realized we are waiting on our vendors to provide patches, compensating controls were tightened as stopgap measures.” Rob Batey of Layline Automation said, “It’s always going to be a race between patching preemptively versus patching reactively.” And lastly, Malcolm Harkins of Epiphany Systems said, “While in some cases we need to do it, patching also adds risks and costs. So, pinpointing accurately the proper approach to mitigate risk, which in some cases could be to patch.”

So, I want to go with this last quote because we actually referenced the things that Matthew, Alex, and Rob said throughout the show. But Malcolm puts up an interesting point which is whether you should or shouldn’t patch, which we’ve discussed previously on the show. But having a mitigation control like virtual patching could allow that decision of patch or not patch tobe a lot easier. Yes or no, Ody?

[Ody Lupescu] Yes. However, yes and. So,I learned from my time with PeopleOps is it’s best not to say “yes but” but say “yes and.” Here’s the idea. You get some assurance but I can’t think of too many cases where, at least from my experience, there was enough certainty that this mitigation will actually work and it will be somewhat infallible. So, that would be one thing. So, I don’t know that I could just rely on it. The other thing that these things are pointing out, and we didn’t talk about this, but to me it’s one of the most significant problems that we haven’t learned in the past 20 years–we still don’t know how to prioritize what’s important to patch and what’s not.

I’ll give you an example. When I do Pentest, I give the external Pentester access to our source code and I say, “By the way, we have some dependency vulnerabilities here. We internally tried to exploit them and we couldn’t, and we know the source code, but they said they’re exploitable. Like there are exploits out there and we couldn’t and we went to others and we went to our bug bounty program and no one could exploit them.” And the problem is you don’t know what’s exploitable and what’s not. There are tools out nowadays that even Snyk was trying at some point to say, “Hey, by the way, this is loaded and this may not be loaded,” or, “This function that’s vulnerable may or may not be used.” This we still don’t do today. So, when we go after it, we go the whack-a-mole game and ultimately we don’t know what’s important and what’s not, what’s the most important. So, yes, this helps, but still in many ways I’m looking at the overall attack surface instead of focusing on one, perhaps it’s the most critical. So, those ought to be the one that I should go try to fix and patch right away, I just don’t know what those are yet.

[David Spark] Steve?

[Steve Zalewski] I would say I think Ody just summarized the whole reason for this conversation. Vulnerable does not necessarily mean exploitable. And what all of this conversation really is is us understanding that exploitability really is the new measurement or metric that we want to be looking at, to get us out from the weeds of simply trying to address every vulnerability that is identified, so that we have a way of doing the a thousand problems but I only have to address a hundred. And so focusing on exploitability as really the core of every conversation we have. I think that’s the key takeaway.

Closing

23:25.008

[David Spark] And with that said, we’re going to wrap us this very episode. Thank you very much, Steve. Thank you very much, Ody. Now it comes to the point of the show where I ask you, first our guest Ody, what was your favorite quote and why? And let’s hear it.

[Ody Lupescu] So, I would have to say Malcolm was probably…

[David Spark] The very last quote?

[Ody Lupescu] The very last quote.

[David Spark] So, just to remind everybody, Malcolm made the quote about saying you have to decide whether you want to patch or not, because sometimes patching will add more risk than you want to deal with. And so this just becomes, well, essentially another wrinkle in our long discussion here. Yes, Ody?

[Ody Lupescu] Yes. Absolutely.

[David Spark] I didn’t want to say it, why did you like it so much? Is it more than what I just said?

[Ody Lupescu] No,I agree with it. I’ve seen where the whack-a-mole games–and this is not just in this case. To me, this reminds me so much of a long time ago when I was in advisory services. I had this client, large manufacturing, probably one of the largest. And they had the largest P-CADdeployment in the world. And they ran into problems because they, like everyone else, had said, “Well, certificates have toexpire every two years, and intermediates have to expire every five years.” And then they start expiring and the production line stopped and millions of dollars were lost.

And when I went there, I asked, “So, why do you expire this every five years or every two years or every year?” “Well, because something could go wrong.” I said, “If something goes wrong, you have to expire it right there.” So, no one had any reason why they kept doing it, and it was more of an operational risk to make a very short duration for certificates rather than make them a little longer. Same thing with patching. It’s probably more concerning at some point to keep patching and patching. You may take more operational risk, outages and so forth, versus not patching. Sometimes it’s not necessary. So, that’s why I like the quote because actually, I subscribe to that thinking.

[David Spark] Steve, your favorite quote and why?

[Steve Zalewski] So, I’m going to start with I really appreciate Malcolm Harkins quote as well, I know Malcolm, a really smart guy. And the reason why is because I think it really pushed us into the conversation of exploitability versus vulnerability. And that’s what he’s really starting to do, right? Is open us up to what is the true exploitability. Excellent. But I need to dovetail on that with Abhishek Singh from Araali Networks, where he said, “Virtual patching is a mitigation for not being able to patch. The beauty of resilient patching is that it is permanent and forward-looking. It takes care of zero day and yet to be discovered vulnerabilities.” And that is the forward-looking conversation that dovetails with exploitability. So, when looking at exploitability, let’s look forward to where patching has a role, but what we have to do is look at the resiliency and a way to be able to address zero days by mitigation, not necessarily by removal. And I think those two together really highlight the outcome of this conversation.

[David Spark] Excellent, Steve. Thank you very much, Ody. And thank you to our sponsor, Araali Networks, who is in the space of virtual patching. And you heard Abhishek earlier in the show and during our sponsor segment talking about this. So, for more about them, go to their site. They are araalinetworks.com for more. Now, we’ll let you each have closing comments. Ody, you get the last comment, but one question I always ask our guests–if they’re hiring. So, make sure you have an answer for that. Steve, any last thoughts?

[Steve Zalewski] Yes. I want to thank our audience. I think this is the first time I’ve put a post out where there have been more comments than likes.

[David Spark] Which is the way it should be.

[Steve Zalewski] Which really is. Okay?

[David Spark] Would hope it to be.

[Steve Zalewski] Hope it to be. And so I really want to thank our audience because this was not a simple topic. This was not just a like, it’s a good idea. This was really asking for thought and to look at how we as an industry are raising the security child. So, I found it to be very interesting with all the comments and I thought this episode, really for me, was a lot of fun and I think really forward-looking.

[David Spark] Excellent. Ody, any last thoughts on this topic? And are you hiring?

[Ody Lupescu] So, let’s start with the thoughts on the topic. So, first of all, I want to start by thanking you and Steve for having me. I do think this is a topic that is not a settled thing and you have different perspectives. And it’s always an interesting debate as people have different opinions, it makes for interesting conversation. It also makes for good learning, kind of seeing different perspectives. So, absolutely enjoyed this. As far as hiring, I’m looking for builders. We’re actually building. The security time at Ethos is building. We’re not just pointing problems, we’re actually building components, and looking for anything from lead engineers to senior engineers and compliance analysts. Please reach out to me, would love to tell you more about it.

[David Spark] Awesome. Well, thank you very much again, Ody. Thank you, Steve. And as Steve said, thank you to our audience. We greatly appreciate your contributions and for listening to Defense in Depth.

[Voiceover] That wraps up another episode. If you haven’t subscribed to the podcast, please do. We have lots more shows on our website CISOseries.com. Please join us on Fridays for our live shows, Super Cyber Friday, our virtual meetup, and Cyber Security Headlines – Week in Review. We’re always looking for fascinating discussions for Defense in Depth. If you’ve seen one or started one yourself, send us the link. We’d love to see it. And when any of our hosts posts a discussion on LinkedIn, participate. Your comment could be heard in a future episode. If you’re interested in sponsoring the podcast, contact David Spark directly at David@CISOseries.com. Thanks for listening to Defense in Depth. 

David Spark
David Spark is the founder of CISO Series where he produces and co-hosts many of the shows. Spark is a veteran tech journalist having appeared in dozens of media outlets for almost three decades.