Defense in Depth: Security Hygiene for Software Development

How do we improve the quality of our software? In the rush to be competitive, security has often taken a back seat to be first to market. What’s the formula for fast and secure applications?

Subscribe to CISO Series Podcasts - Defense in Depth

Check out this post for the basis for our conversation on this week’s episode which features me, David Spark (@dspark), producer of CISO Series, co-host, Geoff Belknap (@geoffbelknap), CISO, LinkedIn, and sponsored guest Wayne Jackson, CEO, Sonatype.

Thanks to our episode sponsor, Sonatype

With security concerns around software supply chains ushered to center stage in recent months, organizations around the world are turning to Sonatype as trusted advisors. The company’s Nexus platform offers the only full-spectrum control of the cloud-native software development lifecycle including third-party open source code, first-party source code, infrastructure as code, and containerized code.

Got feedback? Join the conversation on LinkedIn.

Full transcript

Defense in Depth

Steve Prentice

Hello, I am Steve Prentice, one of the reporters for the daily Cyber Security Headlines podcast. If you enjoy our short six minute summaries of the day’s most important headlines, but are looking for a little more insight, then join me and a special guest every Thursday at 4 PM Pacific, 7 PM Eastern for Cyber Security Headlines week in review. A video webcast where we dig a little deeper and explore just what these stories might mean to you. You can even contribute your own comments. Just go to CISOSeries.com, click on the register for video chats button, and look for Cyber Security Headlines, week in review.

David Spark

How do we improve the quality of our software? In the rush to be competitive, security has often taken a back seat to be first to market. What is the formula for fast and secure applications?

Voiceover

You are listening to Defense In Depth.

David Spark

Welcome to Defense In Depth. My name is David Spark, I am the producer of the CISO series. And joining me on an alternating basis is Geoff Belknap, CISO over at LinkedIn. Geoff thank you so much for joining.

Geoff Belknap

Hi friends and thanks for having me.

David Spark

Alright. Let me mention that our sponsor for today’s episode is Sonatype and they have brought our guest today. So I am excited about that. And this is a great, great discussion about cybersecurity hygiene, specifically for software. I actually reached out to a ton of experts on this topic. And wow, this is heavily involved. The old discussion on this, and when I say old, I am saying maybe only two or three years ago, was shift left. But there is way more to this discussion than shift left, right Geoff?

Geoff Belknap

Yes I think we frequently talk about software eating the world. And I think it is more true now than it has ever been. And as part of that discussion, especially as we think about cybersecurity, we frequently spend time talking about how do we shift left? How do we spend more time with our software developers, ensuring that we have high quality, very secure, very private software products that we are shipping to the world? And since so much of people’s lives rely on that, it is a constant conversation. So I think we have somebody today that can really help us flesh that conversation out.

David Spark

Yes and I am very excited that he is joining us. It is our sponsored guest, Wayne Jackson, the CEO for Sonatype. Wayne, thank you so much for joining us.

Wayne Jackson

It is so great to be with you.

What are they doing wrong?

00:02:38:00

David Spark

Pat Benoit of CBRE said, “I believe the biggest failure is being so focused on moving fast that you lose sight of ensuring good practice, quality, and security. We have developed a culture of “agility” without always retaining the appropriate balance with quality and security.” And Todd Cronin of Ryu Team said, “Software development is a grinding environment. Forces always seem to be pulling in opposite directions, between management, client, and developer ideologies. The only real winner at the end are the hackers who benefit from code rushed into production.” So, this is a big question about balance and do you find that many when developing software are struggling to try to maintain this balance and that it is lop sided often?

Geoff Belknap

Yes, I think the reality is developing software, like building a business, is an ecosystem fraught with choices and equities that we all have to balance and I think especially now that we recognize that very secure software is a high quality software and that it is important to us and our customers that it be that way. That we are bringing the discussion of software development to the same level that we discuss growing the business, and what risks we take there. I think both of these quotes are really extreme. I think you can go fast, you can develop fast patterns for secure software and you can also be stuck in a position where, you know, really because you are optimizing for speed and the business, you might be insecure. But the reality is like many people build secure software quickly and it is really just a matter of what discipline you are following, and how accountable you are holding your teams.

David Spark

Do you agree with that last statement there, Wayne, that it really is what is the system you are building to build the secure software? Because it does not necessarily need to slow down, yes?

Wayne Jackson

I am torn on this one. I think there are clearly ways and methodologies and learning systems that enable developers to develop quickly in a hygiene way. But the uniformity of those practices is so diverse. And Geoff you are in a very sophisticated organization with a very sophisticated development culture, which is entirely different than many of the industries, where software is suddenly their last route to differentiation and so a big challenge for us as a group is how we can make those practices that you are referring to Geoff, more easily consumed and more easily propagated. One thing that I will also add, especially with regards to Pat’s quote, is that we hear a lot about quality and speed, where quality is mostly defined as security. But I would offer that maintainability is also a really interesting part of this dynamic. Developing things quickly that might be secure in the short term, using for example open source projects that are really poorly managed is not going to make for long term innovation outcomes either.

David Spark

How do you know that on the front end though? Like if I used this open source code here, how well will I be able to handle it down the road? I guess you can see the history of how well it has been managed, but do you know if it is going to still be decent five years down the road? Is there any tell-tale signs or have you got to kind of roll the dice?

Wayne Jackson

There are ways that you can measure hygiene. How well they manage committers is one of the most important ones in my view. How well they deal with their own dependencies. Are they managing those well? How predictably are they burning down their technical data? And what is the stability of the project over time? One of the things– and Geoff you may know better than I– but last time I saw there was something 40 million open source projects on GitHub. There is not uniform quality across those 40 million and so I think especially once you start talking to architects, it’s a huge responsibility to actually do the homework. To figure out what the hygiene of the projects are that we are relying on.

David Spark

Geoff, I just want to know what is the way you do your homework to know how hygienic an open source GitHub, code dump is?

Geoff Belknap

I think like all things you have to assume that is it not and then keep assessing. I think to Wayne’s point, and just to build on that, companies like mine, like LinkedIn, like Microsoft and others, we can put together the teams and the internal processes to do that. But I think most organizations need to rely on other organizations, or other software, or other products to help them do that. But the point is you have to do that work. You have to treat it just like you are bringing in a third party vendor to your environment. But in this case that third party vendor, or that third party project has direct access to your production environment. So you have to take it seriously. You cannot just go, oh it is open source, it is secure. It is a process.

What else is required?

00:07:31:22

David Spark

What level of scanning should we do, and how often? There was lots of discussion of static and dynamic testing and how often you should do it. But then many argued that this type of scanning misses a lot. Steve Giguere over at Palo Alto Networks said, “As we move to native technologies and architectures, your microservices have smaller code bases, less than 10,000 lines of code. Using diverse technology stacks which embrace a larger open source base. This may render any enterprise Static Application Security Testing solution ineffective.” And Eoin Keary over at Edgescan said, “Automation is great for the detection of technical vulnerabilities but, for example, if we look at the OWASP API Top 10, many of such vulnerabilities would not be detected using automation. Automation is poor at detection of logical or business logic vulnerabilities.” I will start with you Wayne. When is automation for scanning like this good? And when does it not cut it?

Wayne Jackson

That is a bit of a loaded question. It depends a lot on where the automation is intended to happen. And who you expect to consume or be affected by that automation. The foundational key there is how accurate is the information on which we are making judgments. Obviously automating pass fail based on static analysis results which are fuzzy kind of by design, sounds like a recipe for disaster. Having super actionable information that is very precise and very consumable, is a great place for automation. And in my view the closer that automation can get to the actual invention or the developer, the better we are going to be. The really important part here is what is the intent of your automation? You cannot rely on automation alone to do everything for you. That is the same mindset that gets people to think there is a state that is called secure for software, or for an environment, or for a computer and there is no state where software is just secure. The automation will help you. It will elevate the decisions that your teams need to make. It can help enhance choices that you need to make about when to ship or what you are shipping. But it does not do all the work for you. You have to put a smart human in the driver’s seat to use and synthesize that information, make decisions for your business. Geoff makes a great point and there is no magic bullet in software hygiene. And ultimately as challenging as this is in terms of scale, humans and truly expert humans just have to be in the loop.

What is the situation?

00:10:06:01

David Spark

Jatinder Singh over at Informatica said, “Up to 80% of the code flowing into any “main pipeline” these days is not written by your developers. It is third party Library code written by someone you do not know and have no control over.” Stephen Porter of VMware said, “Development of guard rails to help Dev use and be creative with code or pre-built capabilities is essential. But then the guard rails must help track and trace where code is used. The creation of a catalog of approved elements and third party applications reduces the variability.” And we talked actually on another Podcast where we recorded with Brian, your CTO Wayne, and there was a lot of talk of how that is not even feasible but you may have to lean on others for sort of expertise here, or approval and we can get into that in a second. But let me read this last quote from Francis Dinha of OpenVPN. And he says kind of the opposite here. He says, “The problem is you do not have enough open source software” “Open source software is subject to review by security experts. This means security issues are isolated and resolved prior to the launch of the production version of the software. Not only that, but when an open source community is continually testing it, that ultimately improves the quality of the software as well as the security element of the product.” So let me start with you Geoff on this. There are sort arguments for and against using all this code and I think it also comes back to something you said before, how well is this being reviewed by peers?

Geoff Belknap

I think that really is what it comes down to. First of all just because it is open source does not– I want to underscore that– does not make it secure. It does not matter how many people are looking at it. Very few of those people are looking at it to make sure it is safe for you. There have been a ton of massive vulnerabilities that are related to open source software. I think that just like objectively that is a false statement to say it is secure. But the upside is it can be very secure. It can be very valuable to your business and to the piece of software that you are building. Or the piece of infrastructure that you are trying to ship. But you have to stop thinking that it is secure just because it is coming to you from a third party. I also point out there are lots of software supply chain attacks, intentional and unintentional that take place, that require you to either have a team or have tooling or have a software product that you have purchased, that can help you review that code. Just like you would review code that you wrote. It does not take any longer or it is not anymore technically difficult. But you have to do that work and I think that is the most important part. I think Stephen’s quote here is really important. You have to develop those guardrails. You have to enable your team to be creative and use building blocks that exist. But you cannot just make the assumption that something is going to be good because that way lies madness.

David Spark

So Wayne, the assumption line. You cannot assume anything but what do you do to make yourself more confident?

Wayne Jackson

Geoff I could not agree with you more. The notion that many eyeballs makes for better security is for most open source a fantasy. The problem is that there just are not enough eyeballs with the skills in application security to scrutinize millions of applications and it gets back to the quality of the project and the nature of the project leadership. And Geoff also made a great point, when we are outsourcing effectively, 80 percent of our software development, to use Tinder’s statistic, think about what that means in terms of your aggregate developer base. You have got, take a big bank, JP Morgan, I will pick on them, they have got 30, 40 thousand software developers. If 80 percent of their software is actually coming from external providers, how big is their effective developer base? And how well do they know them? The committers to the open source projects? What are their motivations? What are their skills? What are their backgrounds? Is this a hobby or is this trade craft? So there is just a huge amount of variability. One of the pieces of research that we have been doing lately is to watch committer integrity and how that aligns with project hygiene. Because again, as Geoff mentioned, some of these committers on some of these projects do not have the best of intentions.

David Spark

What did you discover from your research of committers? And the quality of that software? The quality of the commit? What did you learn from that?

Wayne Jackson

That the vast majority of committers are thoughtful and well intentioned. But a very malignant minority are not and we started out doing the research, just sort of on a hunch really using credit card-ish techniques to watch for commit behavior. And we started identifying through algorithms things that looked “odd”. So if a commit happens in two geographies in relatively similar time spans, that is odd. If a committer in a more advanced state starts providing algorithms that are out of character for that committer, if they start introducing new dependencies to our project. If they install certain algorithms like listeners and in logging frameworks, just as an example. That is suspicious and so we have started getting better and better at identifying things that should spark research. And in the course, especially in– again I hate to pick on specific eco systems, but Job Descript is kind of notoriously the wild, wild west. And in that eco-system we routinely now see intentional malware pushed into projects that are otherwise assumed to be good.

Whose issue is this?

00:15:47:10

David Spark

Jeff Williams of Contrast Security says the issue is systemic, because software is a market for lemons. Referencing Nobel prize winning economist George Akerloff who noted that buyers will not pay full price for a used car because it might be a lemon. Wiliams said the same is true for software. “Buyers will not pay more for secure software, because there is no way to know if it’s full of vulnerabilities. Consequently, the market is flooded with insecure software. The way out is “security in sunshine” – making everything relevant to security observable.” And Ahsan Mir of Rapitcore said, “We want a secure application, not just secure software. We can solve for one vertical but end up with an insecure application due to failure in another area. Getting an end-to-end picture from design to deployment of an application is the holy grail.” Wayne I am going to start with you here. I love Jeff Williams comment about there is no drive to make it secure because people will not pay for it, if it is more secure. Do you believe that to be true?

Wayne Jackson

I think we have been trained to tolerate insecure sadly. And you could pick any of the blue chip vendors that puts software on boxes, but fail to tell us what the ingredients are for that box. One of the big initiatives right now relates to requiring from these vendors the software build materials. Which to me seems completely logical. You would not go and buy a can of spaghetti sauce without knowing the ingredients. Why would I consume software which may govern the fate of my business, without knowing what those ingredients might be? To just point with that sunshine, at least you can make a judgment about the integrity and the practices even of these vendors.

David Spark

Let me lose your jar of spaghetti sauce theory. We often do not look at the ingredients because all of this, other organization called the FDA, and if there is poison in this they would have caught it and I would not hurt myself, damage myself as well. So we kind of walk around in this sort of sense of security because there is some other group looking out for us. But there is not an equivalent of that in software, is there Wayne?

Wayne Jackson

There is not. And I have heard various folks propose that there might be, sort of UL labs for software. Josh Corman who is a long term friend of the company and a well-known guy in the software hygiene arena has proposed things like that. And he is a big advocate for these software builds initiatives. But until there is, and I am not sure I really practically want UL labs for software because of what that might do for innovation. But I think transparency is an important part of the answer here.

David Spark

Taking this to you Geoff, starting with Jeff Williams comment about nobody really demands secure software yet. When it becomes insecure they are like what happened?

Geoff Belknap

I think first of all it is really dangerous when we start to use analogues to the physical world for software. Because it does not work. I think the same thing when we talked about cyber war and weapons, the analogues are easy and they are accessible but they do not work. And I think this is a case where I agree with Jeff’s conclusion but I do not agree with the example here. Because people definitely pay for more for higher quality vehicles. Ask Ferrari or…

David Spark

Well he is arguing the used car market specifically.

Geoff Belknap

I think you can look at used car values and see the luxury vehicles like Mercedes or BMW are always going to hold value over other vehicles. And that is because to some extent they are paying for the quality of the vehicle. Now in this case I think where it starts to break down is in software people are making the assumption that they are buying secure software. No-one is paying a price and going this is probably insecure so I am going to take 10 percent off the top of the price. In fact that would be great because if that was the buyers’ market out there, there would be incentive for more secure software. But the reality is he is exactly right. People will not pay more for more secure software because they are assuming it is secure from the get go. They might pay more for higher quality software but there is not a direct correlation between quality and security always, I wish there was. So in this case I think it is true, people will not pay more. That is a problem. There are not enough capitalist economic incentives out there in the market for people to make more secure software. And I think a great example, if you look just beyond software to systems in general is most organizations that get breached, their stock takes a short term hit but they will be right back to where they were, selling groceries or selling wood, whatever they are doing…

David Spark

And by the way sometimes it does not go down, sometimes it goes up. Like Facebook has proved.

Geoff Belknap

There is just a really weird set of perverse incentives for security in the market place. But I think the reality is people want those secure applications. They want their data to be safe. And they are not going to pay you less because your data is less safe. So I think the reality is as we move along in the world, there will be incentives. There will be regulatory requirements, as there are developing around privacy, for you to build and ship secure software. I think software building materials is a great way to get there. I would rather see something that is more like nutrition facts so that consumers can make decisions about how much risk they are taking, sort of decide it is okay for me to have this much fat or sodium in my diet, or maybe it is not because of other risks that I am taking in other places. But until we get there, and it is going to take a long time because to get there, especially in the US, which is a dominant market, you have to have more informed legislators and regulators. And right now dear old legislators and regulators listening to this, do not take this personally but right now we do not have that level of information in the space. And I think it will just take a while to get there.

Close

00:21:39:12

David Spark

An excellent point to close on and thank you very, very much Geoff, Wayne. We have now come to the point of the conversation where I ask what is your quote and why? And I am going to start with you Geoff. Which quote was your favorite and why?

Geoff Belknap

My favorite is Stephen Porter from VMware. I think development of guardrails is fantastic. I frequently talk about guardrails, not gates. Did not invent that, I think that has been there for a while. But the idea that you can enable people to move faster if you build these guardrails, if you use either tools that you build or tools that you buy to help enable quicker decisions about risk in the environment, that enables your organization to be agile and adapt to what the business needs.

David Spark

Alright Wayne it is over to you, your favorite quote and why?

Wayne Jackson

Well I cry foul because Geoff stole my favorite quote.

David Spark

Do you have a second favorite?

Geoff Belknap

Or you could just tell everybody why it is your favorite.

Wayne Jackson

It is not as potentially controversial as Geoff’s quote, and Geoff is a fantastic guy in that way. But it is just a great bit of practical advice. Developers need to go fast. But give them some guardrails, let them do their best to stay in the center of the highway and go as fast as they possibly can. I just think that is really solid practical advice.

David Spark

Excellent. Thank you very much Wayne. Thank you Geoff. Now I am going to get you have the very last word on this, Wayne. And also if you have got anything to tell us our audience about Sonatype or any offer you would like to give them, please let us know or how to follow up with you to learn more. And what you guys are offering in this space, please let us know. Hold on to it for a second. And I also ask my guests are you hiring? So have an answer for that one too. Geoff. Any last comments you would like to make on this discussion?

Geoff Belknap

I think I will use this to do my usual plug. If you are interested in making secure software and having a big impact on the world, we are always hiring over at LinkedIn. You can find us on LinkedIn.

David Spark

Good place to look. Thank you so much for Sonatype. Sonatype by the way for listeners benefits, have been a spectacular sponsor of the entire CISO series and we really, really appreciate it. Wayne, any closing comments and what would you like our audience to know about Sonatype, how to learn more and get in contact with you.

Wayne Jackson

So first, David and Geoff thank you for the great conversation. It has been a pleasure. From Sonatype’s perspective, we think of the world of software and software innovation in supply chain terms. So helping to manage and identifying the best possible inputs that the manufacturing process, is modern software development. And to enable developers to make better decisions in the context of those guardrails we just talked about. To Geoff’s point we also are hiring so would love for anyone who would like to learn more about software supply chain management and what we do at Sonatype, would love to hear from you at Sonatype.com. And love for any reach outs directly. My email is wayne@sonatype.com. Happy to entertain any questions that anyone has.

David Spark

Thank you very much Wayne. Thank you Geoff. And to our audience, as always as I have always said I greatly appreciate your contributions, that you see the show is built on your contributions. So thank you very much for that. Thank you for contributing and listening to Defense In Depth.

Voiceover

We’ve reached the end of Defense in Depth. Make sure to subscribe so you don’t miss yet another hot topic in cyber security. This show thrives on your contributions. Please write a review, leave a comment on LinkedIn or on our site: CISOSeries.com where you’ll also see plenty of ways to participate, including recording a question or a comment for the show. If you’re interested in sponsoring the podcast, contact David Spark directly at David@Cisoseries.com. Thank you for listening to Defense in Depth.

David Spark
David Spark is the founder of CISO Series where he produces and co-hosts many of the shows. Spark is a veteran tech journalist having appeared in dozens of media outlets for almost three decades.