Developers and security professionals have been heavily sold on the concept of “shift left” or deal with security issues early in development rather bolting it on at the end. It all made logical sense, but now we’ve been doing it for a few years and has shift-left actually reduced application security concerns?
Check out this post, this post, and this post for the discussions that are the basis of our conversation on this week’s episode co-hosted by me, David Spark (@dspark), the producer of CISO Series, and Steve Zalewski. Our sponsored guest is Mike Gorman (@gormamic), head of security and compliance, NetFoundry.
Got feedback? Join the conversation on LinkedIn.
Huge thanks to our sponsor, NetFoundry
[David Spark] Developers and security professionals have been heavily sold on the concept of “shift left” or deal with security issues early in development rather than bolting it on at the end. It all made logical sense. But now we’ve been doing it for a few years, and has “shift left” actually reduced application security concerns?
[Voiceover] You’re listening to Defense in Depth.
[David Spark] Welcome to Defense in Depth. My name is David Spark. I am the producer of the CISO Series. Joining me on this very episode is a gentleman you’ve heard many times before, and I’ll introduce him yet again. His name is Steve Zalewski. Steve, what do you sound like?
[Steve Zalewski] Hello, audience.
[David Spark] He sounds just like that, and he’s consistent. That’s what we like in a cohost. Our sponsor for today’s episode is NetFoundry. You’re going to hear a lot from NetFoundry because they brought our guest today for this really interesting discussion. And I’m actually pretty excited about this. Steve, you asked this question on LinkedIn about “shift left” as to whether or not application developers and security professionals were actually reducing security issues with “shift left”. I mean, let’s just start with when the concept came along, everyone is like, “Yeah, I get it. That makes sense. Of course we should do that.” So, did anything actually reduce or even eliminate the need for other security controls? What do you think was the general tone of the response?
[Steve Zalewski] I would summarize it as one, I was surprised. And two, I was a little disappointed.
[David Spark] Do not disappoint Steve, by the way.
[Steve Zalewski] Don’t disappoint me, team. Come on. My audiences out there, we got to fix this.
[David Spark] He’ll come down like a hammer.
[Steve Zalewski] There you go.
[David Spark] Go ahead, Steve.
[Steve Zalewski] So, what I would say is…what we’re going to talk about here was I was surprised at the vehemence of the people saying “shift left” was the right thing to do. And I was very…
[David Spark] Without explanation
[Steve Zalewski] Without explanation and was disappointed because I was asking pretty succinctly, “Give me some examples of how you’re succeeding and how this is helping the general security program.” And all of a sudden we got crickets.
[David Spark] Let me just say, I’ve had a history of interviewing security professionals who have this, “Yeah, this is the right thing to do. Just listen to me,” kind of attitude where it doesn’t go over too well. Now, I don’t hear that nearly to the level now that I used to hear it, but it’s still around.
[Steve Zalewski] Yeah, and I think today’s conversation is going to be great because that was it. Which is it’s okay for crickets. It’s understanding why we’re seeing crickets, and I think that’s why today’s conversation and with our guest is going to be so exciting.
[David Spark] Yes, and I’m very excited to introduce our guest because this is an issue they’ve been specifically addressing, NetFoundry. And they may actually have an answer that our audience will be interested in listening to. So, excited to introduce them. It is our sponsored guest, Mike Gorman, who is the head of security and compliance over at NetFoundry. Mike, thank you so much for joining us.
[Mike Gorman] Thank you so much for having me, David.
What are they doing right? What are they doing wrong?
[David Spark] Andreas Schneider, group CISO of the TX Group AG said, “Shift left works. It comes with a mind change on how to do security. As a CISO, you guide them and try to enable them, and you search for new solutions that even might be better than the existing ones.” And Jonathan Waldrop of Insight Global said, “The purpose of “shift left” isn’t to avoid work later. It’s to make work smarter so there’s less rework later.” I really like Jonathan Waldrop’s comment there, Steve, because I think that’s really what “shift left” is, but I don’t think it was sold that way, was it.
[Steve Zalewski] So, what I’m going to say is, “So what? Now what?” Both of these conversations or both of these comments are actually real. They represent the thinking of “shift left” over times has matured in different ways of looking at it. But when we look at it from the, “So what? Now what?” Which is how are you attacking this as a CISO and rethinking what your security controls might look like in addressing the philosophy of “shift left”. That’s where I think it was so interesting to see some of the philosophies that were proposed and in some cases some of the technologies. But it’s still primarily driven by secure software development life cycle, and getting in as early in the design process as you can is that it will somehow result in a better outcome. And these are two examples of how that is represented.
[David Spark] Mike, let me ask – the difference between how “shift left” was pitched to us and where we are now with it, it seems like people are still sort of singing its praises but without, like Steve said, “So now what?” attitude. It’s just kind of, “Listen to me,” attitude. Isn’t it?
[Mike Gorman] Yeah, I think so. I think, like you said, everybody understands the idea. Everybody gets the concept. We’re going to get problems addressed earlier. We’re going to reduce rework. We’re going to reduce the load on the QA team, the security team, the production team. But mostly I think if you really talk to developers and you really talk to software people in general they’ll say “shift left” is really just doing it right the first time, the way we should have been doing it to start with. There’s a lot more about we have to look at the tooling. We have to look at the efficiency. Listening to some of your previous podcasts, there’s a great conversation on speed versus efficiency, versus effectiveness. And I think that has to come into play – that we have to build up this whole thing. My other fundamental argument is I don’t know that we go far enough. Usually when we say “shift left”, we kind of stop at the development cycle, and the development tools, and how we manage people that write code for a living. And we don’t back up the architecture.
[David Spark] You’re saying we’re not even thinking about it right.
[Mike Gorman] I don’t think we are. I don’t think we go back far enough to architecture to say, “What do we have to put in the code as functionality that is fundamentally more secure than we did yesterday?” Any good developer knows how to stop a SQL injection attack. That’s not the zero days that are found these days. It’s there are more fundamental things that we have to get ahold of and get into the software development life cycle.
[David Spark] So, it’s an architecture issue, which would save a lot of headaches. What do you think, Steve?
[Steve Zalewski] I’m going to take some maverick thinking here – which was I think “shift left”… Too many people think of “shift left” into the design process for building secure apps. I’m going to say another way to look at this is we’re not shifting right. We have to shift very far right and look at incident response, and think about what we’re doing to be able to better provide resiliency and contain the incidences, knowing that “shift left” as a philosophy is just that.
[David Spark] And do you mean by shift right meaning once this application is done, what are the ways this kind of thing could be attacked? That we should understand that at the far end before we start dealing with “shift left”?
[Steve Zalewski] Meaning that when you do the testing for the application… And this is an example of the maverick thinking is have your incident response teams be the ones to do the testing so that what they’ll understand is where are the gaps, and therefore what do I have to do to be resilient. How do I do defense in depth to limit the access to the gaps rather than necessarily then sending it back to square zero with the developers? Because often times the business doesn’t have the luxury of that time. And so now what you’re doing is building in “shift left”, immediately going way to shift right to see what’s likely to happen and have your defense controls ready for resiliency.
What should we be measuring?
[David Spark] Parker Brissette of the Colorado Judicial Branch said, “With the “shift left” and towards automation we wouldn’t have to rely on legacy knowledge that isn’t documented just to discover our exposure. Remediation would happen faster as well due to consistent code cases.” And Sharon Besser of Seemplicity said, “In my opinion, people ‘get it’ that security must be an integrated part of the service production solution they provide. So, how would you know that “shift left” works? I suggest the following – measure the time that it takes to prepare and pass an ISO 27001 or SOC2 audit or a similar regulation. In my experience you would find that the time and cost reductions are enormous.” Mike, what do you think about that in terms of seeing whether “shift left” has actually worked? Like to see if your audit goes quicker.
[Mike Gorman] I’ll address the second point first, the audit point, because while I understand the person’s interest and I understand their point of view, my personal experience in compliance is that the software development lifecycle is a fairly small piece. Shift left isn’t going to make sure that your acceptable use policy was updated. It’s not going to make sure that your business analysts all have their antimalware updated. The kind of certification things are so much larger than the life cycle, unfortunately, to an extent. On the how we measure, I’m going to pick up on what Steve just said because I think the best measure of is “shift left” working is how hard is the SOC or whatever your incident event response team…whatever they take, whatever they become, working on that software. As you have done all of these things – as you’ve implemented testing, as you’ve implemented scanning, as you’ve engaged them hopefully, are you actually getting less events out of the system on the far end? Because SOC personnel, IT personnel, whatever they are, that’s really money. Their time is real money to the business. And with so much difficulty in hiring security talent these days, if you can lower the resulting impact on those teams, that’s a big payoff to the business. You’ve obviously reduced the risk posture, and now you have this metric that says, “Our security teams are doing less. They’re doing less garbage work. They have to less to actually go in and look at, so they can look at it better. And we just don’t need an army of analysts. We can do with a platoon.”
[Steve Zalewski] So, I’m going to continue my maverick thinking for today a little bit more and say what should we be measuring. And SDLC, you measure vulnerabilities. I would say what we should be measuring is exploitability. That’s why I say you shift hard right because what you do is you go through all whatever you’ve got – your current SDLC processes – you measure against that. But my metric wants to be against exploitability, and I want my incident teams and my red teams to look at the app not to see the vulnerabilities but to understand what the exploitabilities are. Because as long as I can mange the likelihood of exploitability, I can accept a lot more vulnerability. And now we’re moving towards a real conversation in the value of how we measure “shift left” beyond just how we build the app.
[David Spark] Steve, we’re going to be doing a video chat on extreme vulnerabilities soon. Is that the same thing as exploitability? Like a vulnerability… Because to say a vulnerability means a lot of things. It could be something very benign that nobody really cares about to, “Oh my God, the whole house will come crashing down if this is exploited.”
[Steve Zalewski] Right.
[David Spark] So, exploitability is essentially extreme vulnerability – something that would cause the house to come crashing down.
[Steve Zalewski] No. No, it is two different ways of thinking in my mind. An extreme vulnerability is go ahead and look at the one through ten when a vulnerability is announced, and it’s given a class one through ten. Ten is the highest. Solar winds was a ten. That means it’s highly vulnerable. That you can execute that vulnerability. But what’s the exploitability for me? If I don’t use solar winds, the exploitability is zero.
[David Spark] I see.
[Steve Zalewski] And so therefore let’s not concentrate on vulnerability and going ten means, “Oh my God.” Move it towards what’s the exploitability and the material weakness that you have in your company from the business perspective so that you can start to put vulnerabilities into perspective and change your risk profile and not have the, “I can’t patch my 10,000 systems. Therefore I’m in trouble.” To, “Wait a minute. I only have four systems and these business processes.”
[David Spark] Mike, I want you to respond to that exploitability, quick.
[Mike Gorman] I think Steve hit it on the nose, which is exploitability varies wildly in architecture. It varies wildly in risk. And as we’ve seen in the security space over the last several years and people trying to move much more to a risk based analysis, that is the difference between exploitability and vulnerability. And I would add in there of course you have the value of the data asset you’re protecting. If you have a very high vulnerability on a test box that the worst thing that could happen is someone gets in and drops a crypto miner on it, and they’re eating up cycles on your $20 a month AWS instance, that’s not a huge risk. How much are you going to spend to stop that? If you have that same vulnerability that’s attached to a database with your customers’ information in it, that’s a very different thing. That’s a risk analysis question. It’s a risk management question and vastly more important than just counting CVSS scores.
Does anyone have a better solution?
[David Spark] Abhishek Singh of Araali Networks, who we’ve quoted many times before, said, “We have self-defending apps, which takes care of not having to trust or worry about endpoint, network, api, credentials, and infrastructure. When you have end to end security built into the app, the hop by hops don’t matter.” So, this is going to an architecture issue, also, Mike, in that if you think intrinsically about what you’re dealing with then this “shift left” concept becomes less of a concern. Yes? What do you believe, Mike?
[Mike Gorman] Yeah, I really like the terms elf defending apps. I’m stealing that, Abhishek. Thank you very much.
[Mike Gorman] And this is near and dear to our hearts at NetFoundry. That’s where we are, and we talk more about that. But he’s absolutely right.
[David Spark] Well, actually… I want you to. And I want to set you up in that one of the things that was made clear to me about what you guys are doing is securing the communications between apps so other concerns start to essentially fall off. Can you explain?
[Mike Gorman] Yeah, absolutely. So, what we do is we offer SDKs in multiple languages where you can embed secure networking directly into your application. If you’re a developer then this is replacing socket.io, or socket.h, or the various languages. Literally the network communication from the application outbound goes directly into a secure network. It’s not a client on the host, etc. It can be. Crawl, walk, run. We can help with a lot of solutions. But what that does when you look at today’s world and you look at the old tower and moat kind of thing, now a days we have virtualization. We have Cloud. We have hybrid Cloud. We have multi cloud. We have mobile edge compute, which is Cloud distributed all around the world. It’s very difficult to manage the old security infrastructure that would sit in your DMZ, and you have your HIDs or your IDs, and your IPS, and your firewall, and your UEBA, and your DLP, and all of these things. If you want to follow zero trust, if you want to micro segment, if you want to parcel this stuff out then the application needs to protect itself. Self-defending applications. That’s where we are. We just take that network interface. You can’t see that network interface unless you’ve been previously authenticated and authorized, and that authentication is X.509 certificates. It’s very difficult to get it, and it just takes it off the internet. You can’t see it.
[David Spark] Okay, so here’s what I want to know… And, Steve, I want your response to Mike’s answer here is… And what we were driving at with this question is what falls away. What are we now no longer doing, no longer worrying about… Or maybe not completely not worrying about. We’re just not worrying about it at the extreme level that we normally do. What we are not doing if we have this kind of secure communications that you’re offering through NetFoundry?
[Mike Gorman] The easiest way to explain it is it simplifies the security configuration. So, I [Inaudible 00:17:29] or any other Cloud scaling or Cloud process. Security group, no inbound. Period. No inbound. All the connections from the application are established outbound to the secure network, so you just clip it off. No more punching holes in your firewall. No more managing the rules. The rules are very simple. Can anything get in? No. And when you do that, you take away the functionality that we’ve built in firewalls, in WAFs in all of these other tools because we had to manage the exception. We just don’t make exceptions. And that mean that you can spin applications up. You can deploy them anywhere in the world, and they are secure by default. Not even by design but actually secure by default. When that application comes up in and initializes, the only thing it can talk to is the network, and the only thing that can talk to it is a previously authenticated and authorized user or identity. That turns the whole security program on its head that we’ve used in the past and getting rid of all these just massive tool chains that are just so hard to admin, and audit, and operate.
[David Spark] And this is where I want your take, Steve, because this is what we’ve been frustrated with, actually our audience a little bit, as you expressed is that we want to talk at this level. So, what’s your response to how Mike sort of played out NetFoundry?
[Steve Zalewski] So, I’m going to be a little bit more technical than I usually am here.
[David Spark] Go for it.
[Steve Zalewski] To Mike. And it was one of the things I was thinking when I posted it. David, you heard me this… Because I said is really “shift left” an opportunity to understand that all we have to do is secure the data in flight, in use, and at rest? And if I can put independent controls around that then every application, no matter how weak, I have affectively implemented a “shift left” that gives me relief downstream on some of the other things that I have security controls for, and I don’t have to with regards to protecting the data. Because ultimately it really just comes down to how good am I at managing the entitlements to the data. And so when I look at Mike, I go what Mike is really saying is if we look at it as what are we doing to simply protect the data against everything. And “shift left” is applications are everything. Then are we doing the right level of entitlement management there, and do we want to rethink what our architectures look like to be able to protect us from our developers?
Why are they behaving this way?
[David Spark] Cy Khormaee of Google said, “More than any other time I’m seeing developers work to build in security to everyday features. The vast majority of reCAPTCHA Enterprise adoption has been developer led. It’s been great to see developers across all areas take on security challenges are a core part of their platform reliability and success.” Now, we have heard for years that developers do want to do good work. And yet at the same time, we’ve heard complaints from security professionals that they don’t think about security early on. And it’s good to hear what Cy is saying in this place here. So, what I’m hearing and hopefully the two of you are seeing is there is a mutual desire to do this right and do it securely. Mike, what have you seen in terms of…? And I want to talk more in trend like what is it now as opposed to maybe six years ago.
[Mike Gorman] I think it’s absolutely right, but we are seeing the shift towards developers doing a better job of securing function and functionality in their personal development cycle and the change going forward. But part of that is because of the “shift left” mentality. There used to be kind of a mentality of a develop an operational spec. Here’s an interface. Here’s a function. Code this up, throw it over the wall. Somebody will figure out how it fits in. And as we’ve put pressure on people that you have to do this securely. Okay, now that’s part of my core requirement. Kind of a sideways compliment. Developers are lazy. They will find the best way, the most efficient way to meet those goals. And they love to talk to each other, and they love to look around. And they love to shift around in the industry. And so when you have something like reCAPTCHA that catches someone’s attention and says, “You know what? If we implement this, we stop 4,000 bots a day from crawling the application.” So, can I go back to my earlier comment about taking pressure of the SOC and all of that? And so if somebody says, “Hey, this is great. Here’s 4,000 bots a day blocked by picking what looks like a traffic light. Here’s the library on GitHub. Go.” And that will spread across like wildfire through developer communities. And that’s where I think we get into this – that if we give the tools, we produce the tools, we give the requirements, developers will find the way. That’s what open-source is about – making it easy for people to do it. That’s why we participate in an open-core model. Developing communities are just amazing. They will find the way to do it as long as you make it part of their job. If the business holds them accountable to do things securely they will do things securely, and they will very rapidly do things securely and efficiently.
[David Spark] Steve?
[Steve Zalewski] I’m going to be a little maverick here again, Mike, on you.
[David Spark] Three segments in a row you’re being a little maverick.
[Steve Zalewski] Maverick, right. So, part of it is we’re picking some hard topics, which is what I really enjoy here. We’re talking about reality and perception, and what we have to do – the, “So what? Now what?” Which is in my mind… And we talk about this. There are three types of developers. There are malicious – the ones that are going to go in and try to change my code for the bad thing. There are the ignorant – the ones that haven’t been trained in all the SDLC processes that many of us have because we have lots of contractors and offshore. And so it’s just the way that we do business introduces a lot of people that aren’t trained, and so therefore we have the ignorant making mistakes. Okay? And then we have my favorite ones, which are the rebellious, which are the majority that know better but believe that they have the authority to not follow the security rules because the business tells them, or they’re above the law. And it’s the rebellious ones that are doing the damage in reality. And all of that “shift left” to be able to give them the tools, and the processes, and the knowledge doesn’t help when they rebel. Which is why when I made the comment about move from vulnerability to exploitability and acknowledge that go right to get your incident response teams to go left with you to look at exploitability as the metric is the way that we can understand why the behavior is the way it is. What are we going to do about it to be able to manage the true risk rather than continue to just berate ourselves that if people would just do what we tell them everybody would be secure.
[Mike Gorman] I have to comment. First, I’m going to add a fourth category – people that really love their job and want to do it. I think you missed them.
[Steve Zalewski] Fair. Fair.
[Mike Gorman] But I think you’re right. And we can’t forget, “shift left”…a big part of that tooling is coming from security teams. That mentality offers security teams the opportunity to go in and instrument that SDLC with their own instruments. You’ve got pipelines, whatever pipelining system you’re using, Jenkins or whatever. And you’ve got all these things. And it’s very easy for me as a security professional to go in and go, “Hey, you know what? I’d really like to run my vulnerability scanner against your staging environment. And how do we script that.” And we’ll script it all, and so we just give you a big, red, easy start button, and you start it. And I’ll get the report, and I’ll come back to you later. We can instrument that to help mitigate those issues of the rebellious and the ignorant early on. So, we know these are the requirements, and these are what we’re going to do. And by the way, we take on the role of analysis. To your point of vulnerability versus exploitability. If you run any vulnerability scanner, you can get a log of a thousand problems. They’re going to be there. But a lot of them are mitigated by design or whatever, and we take that responsibility and feed it back. And you create a virtuous cycle where you’re feeding back early enough the developer can repair. We can get that back out. And in the end, we can both maintain business agility and improve the security posture. That, to me, in the end, that’s what “shift left” is really about is to maintain or improve your security posture and maintain or improve your business agility at the same time so that you have better business outcomes. And I think we do that through what’s commonly known as “shift left”, what we’ve talked about today where you’re addressing it at an architectural level and the fundamental data protection, information protections that you can include in your code with features that are open-source available, licensed available, whatever. That’s how we take this to the next level. This is how we make the application the edge, and we simplify security, and we improve security all at the same time.
[David Spark] And that is a good spot to close today’s episode. Now we come to the portion of the show where I ask both of you, tell me which is your favorite quote and why. And I will start with our guest, Mike. Mike, which quote was your favorite, and why?
[Mike Gorman] I have to say Abhishek, and we have self-defending apps. It is core to our being and my personal belief, and I think he nailed it on the head in a way I wish I had. So, it’s my favorite.
[David Spark] Right. Well, you’re stealing self-defending apps. When he posted I didn’t see a TM next to you, so you’re good to go. Steve, your favorite quote and why.
[Steve Zalewski] I’m going to go with Abhishek as well, but I read that statement in a completely different way.
[David Spark] Backwards?
[Steve Zalewski] Well, backwards. Okay. I interpreted his conversation be if we don’t have good SDLC “shift left” and self-defending apps then I have to put controls in to not trust the apps and do things to be able to manage the damage as it’s done. And that is defense in depth. So, when we talk about the behavior and we talk about what we’re doing, I like Abhishek Singh’s conversation because it’s acknowledging that we believe everybody is going to do the right thing, to what Mike said. Most of the developers are trying to do the right thing. But there are always some that don’t. And therefore what’s my defense in depth? And so shift right to “shift left”, as I talked about, and to get to exploitability and realize that we still have to have downstream capabilities to be able to do what I call the offensive defense – continuous authorization to be able to look at the movement of data, to your point, as a way of looking there and getting to where we are. So, I found it very interesting that you read his comment one way. I read it another. One is kind of the I won’t say proactive but good developers, and if we get it right, this is why it’s going to save us money downstream. And me saying, “But we always have to have that downstream view, and it may allow me to be less prescriptive further left in the development process to be able to account for the types of developers that are not the family trusted ones.”
[David Spark] Very good. Thank you very much, Steve. Thank you very much, Mike. Now, Mike, I’m going to let you have the last word. And what I always ask my guests is are you hiring? So, please have an answer to that. And if you’d like to make a final pitch for NetFoundry and more for our audience, please do that now.
[Mike Gorman] We are always on the hunt for top talent. We’re not on a big hiring spree, but we’re a fairly small company, and we’ll always investigate top talent. And please come by netfoundry.io. We are a .io. Or OpenZiti. I didn’t mention it before, I think, which is our software, our SDKs and the entire networking system. It’s called Ziti – both a delicious Italian pasta and secure programmable networking. We are in the OpenZiti RPO [Phonetic 00:30:36] on GitHub. Come by, take a look, leave us a star so we know you were there. And fill in the code. Come play. We really want to build a community both in the developer space and in the networking solutions space.
[David Spark] When you said it was a programming code and also a delicious pasta, it reminds me of the old SNL sketch. “It’s a dessert topping.” “No, it’s a floor wax.”
[David Spark] So, it’s kind of the same thing.
[Steve Zalewski] Mike, I do want to take a moment to say thank you. This was a really interesting topic and really appreciate the opportunity and flexibility in thinking as we kind of looked at the different views. So, really appreciate having you on the show and thought it was great to have you here to talk about what I think is a very important topic.
[Mike Gorman] Thank you very much. It’s fun to have that discussion with people who get it.
[David Spark] Thank you very much, Mike. Thank you very much, Steve. And thank you to your company, Mike, NetFoundry, for sponsoring this episode and multiple episodes of Defense in Depth. We also greatly appreciate our audience, even when Steve starts trashing them. Steve, you’re going to have to do a lot of apologizing.
[Steve Zalewski] I will, but I think the audience appreciates when I call it like I see it sometimes.
[David Spark] Yeah, they do. They do. We appreciate your contributions and for listening to Defense in Depth.
[Voiceover] That wraps up another episode. If you haven’t subscribed to the podcast, please do. We have lots more shows on our website, CISOseries.com. Please join us on Fridays for our live shows – Super Cyber Friday, our virtual meet up, and Cyber Security Headlines – Week in Review. We’re always looking for fascinating discussions for Defense in Depth. If you’ve seen one or started one yourself, send us the link. We’d love to see it. And when any of our hosts posts a discussion on LinkedIn, participate. Your comment could be heard in a future episode. If you’re interested in sponsoring the podcast, contact David Spark directly at David@CISOseries.com. Thanks for listening to the CISO Series Podcast.