What would it take to build your entire security program on open source software, tools, and intelligence?
Check out this post for the discussion that is the basis of our conversation on this week’s episode co-hosted by me, David Spark (@dspark), the producer of CISO Series, and Geoff Belknap (@geoffbelknap), CISO, LinkedIn. We welcome guest DJ Schleen (@djschleen), distinguished security architect, Yahoo Paranoids.
Got feedback? Join the conversation on LinkedIn.
HUGE thanks to our sponsor, SPMB Executive Search
Full transcript
[David Spark] What would it take to build your entire security program on open source software, tools, and intelligence? Can it be done?
[Voiceover] You’re listening to Defense in Depth.
[David Spark] Welcome to Defense in Depth. My name is David Spark, producer of the CISO Series. Joining me for this very episode, you know him very well because you’ve heard his voice many times, and you’re going to hear it again right now – it’s Geoff Belknap, the CISO of LinkedIn.
[Geoff Belknap] Hey, David. Thanks for having me. And looking forward to this.
[David Spark] This is going to be a great discussion. But first, let me mention our sponsor. It’s SPMB Executive Search – the executive search firm for innovators. More about what they do…it’s pretty darn cool…later in the show. Geoff, let’s get to our topic at hand. This has actually come up on our other shows before, but we’re going to actually have a deep dive on just this topic. I think it’s very interesting here. This question was posed – “Is it possible/feasible/practical to run a security program entirely based upon free and open source software, open source tools, and open source intelligence?” This was asked by Rafeeq Rehman of Verizon Business. And as I said, we talked about it on the other podcast as a thought exercise. But could a security team really pull this off? And I’ll ask you, Geoff, do you know of anyone who’s actually done this?
[Geoff Belknap] I think could a security pull this off, absolutely. And has a security team done this? I feel like every security team that is starting from scratch goes down this journey and starts this way. And it’s a perfectly reasonable approach to take, and I think it’s just a question of what are the tradeoffs – what are you missing out on if you do open source. Is it are you missing out on time? Are you missing out on features? What are your tradeoffs? And I think this is going to be a great topic to get into with our guest.
[David Spark] Like most of our discussions, there is a lot of it depends in this discussion as well.
[Geoff Belknap] We’re basically lawyers on this show, yes.
[David Spark] So, joining us… Thrilled to have him on. We had him on our live show, Super Cyber Friday. I wanted to get him on this podcast. It is the distinguished security architect for Yahoo Paranoids, DJ Schleen. DJ, thank you so much for joining us.
[DJ Schleen] Hey, David. Nice to see you, and thanks for having me on the show.
Would this work?
2:22.718
[David Spark] Now, this is the big question – would this actually work. And Bruno Guerreiro Diniz of Datasec said, “Possible? Yes. Feasible? Yes. Practical? Not much.” Ricardo Bastos of Sierra Wireless said, “I believe it is impossible to do well with only open source. Some solutions do not have good open source alternatives. Would you use ClamAV as your AV or OpenVAS as your vulnerability scanner?” And Ofer Shaked of Deepblue said, “Yes, but you’ll be spending a lot more on set up and maintenance and have a lower security level.” So, this sort of tags to what you were saying in the opening here, Geoff, that there is possibilities. There is feasibilities. But there’s questions about how practical doing a full open source program would be.
[Geoff Belknap] Yeah, I think the reality is like is it practical? Yes. But in my experience, what you are trading off on is time. So, we talk about this a lot when we’re looking at buying security product is like what is time to value – how fast between cutting a check and getting that piece of software infrastructure installed to when I’m receiving value from that infrastructure. And then over the time… And I hate that I’m saying this out loud. But what’s the total cost of ownership of that. Like how much time and energy does it take to maintain that. I think the reality is there’s a lot of really great open source projects that can cover just about everything you would need to run a security program. But if you go all in on open source and you don’t have an incredibly talented pool of engineers at your disposal, it’s going to take a significant amount of your time to maintain those things and keep them running. Whereas generally if you spend money, generally, you’re getting a lot of that automation of how you maintain it, how you upgrade it, where you get updates through rules, and intelligence, and things like that. you’re getting that for free, built into the cost of the product. Whereas otherwise you’ve got to go and hunt for that yourself. So, there is a tradeoff everywhere.
[David Spark] You’re even jumping ahead into the discussion we’re going to have even later on this talk as well.
[Geoff Belknap] Oh, all right.
[David Spark] Because this very much comes up. All right, DJ, have much have you personally gone down the road of open source, and at what point do you just sort of cut loose and go, it’s like what Geoff said, “This is just going to take too much time.”
[DJ Schleen] I think it’s a bit of a mixed approach. I’ve been at some small startups where you don’t have the funds to really invest in a lot of these very expensive tools. You look at some of the software composition analysis tools, and you’re looking at the millions of dollars for licensing. And even looking at 90,000 servers that you might have on prem, which is a reality in some large organizations, how are you going to scale a lot of your security program to that? Are you going to use end point protection? It might not be financially feasible. But I think it is possible. I’ve done it in smaller organizations where you can do code analysis. You can do deployment. You can do a lot of things around the software that you deploy. I think the hard parts are when you start getting into end points, laptops, identity and access management, and those kinds of things. But to start off, I think that you can. I think it can work, but I thin it’s going to be a hybrid as you get bigger.
[Geoff Belknap] That’s a great point. When you’re dealing with large fleet scale things like end points, especially if you’re a medium to large size enterprise, that’s where that maintaining it at scale, where there’s all those edge cases, that’s where that really bites you.
What needs to be considered?
5:57.624
[David Spark] Dennis Merenguelli of Verizon said, “If the company is small enough, sure. But not for the enterprise.” Just tagging off what the two of you just said. And Jonathan R., CISO over at Lightspin, said, “If you had a relatively small Cloud native footprint, sure. Outside of that, not really. Open source EDR or next generation AV is very poor. You’ll be missing advanced tools if you need them like DLP and CASB.” So, I’ll start with you, DJ. Can you get decent advanced tools like DLP and CASB like Jonathan is saying? Through open source that is.
[DJ Schleen] You might be able to find it, but I think the cost of ownership is going to be high because you’re going to have to potentially customize it, bend it into the environment that you’re trying to deploy this into. That’s I think one of the biggest things about the risk about going down this road is that there’s going to be certain things that are really mature and certain things that aren’t. There’s going to be applications that you have to host internally, and you have to provide infrastructure for. Or do you go to the SaaS model and purchase some service from another organization? I think there’s a lot of considerations that feed into this.
[David Spark] Let me ask this… I got to assume that a lot of people like to go down the open source road to say, “Hey, can we ‘get this for free?’” But does there come a point where just the discovery process alone is so long and so painful that you’re like, “We’re not even going down that road. There are better paid solutions out there that are going to do this quicker and easier for us.” Either of you jump in on that. Let me put it a different way of does open source become, “Let’s look first there before we even try a commercial solution.”
[Geoff Belknap] I think a lot of times, yes. I think just as DJ said and has been my experience, if you’re starting from scratch, whether you’re starting a brand new security program at a brand new company or you’re starting to build a capability that you didn’t have before… Even if you’re at a company the size of Yahoo or LinkedIn. A lot of times you’re going to start with open source because the cost to you to talk to somebody and get a demo or etc. is very low. You can just get on GitHub or something like that and download something and try it. You can also experiment with, “Oh, does this thing actually add value to my environment?” And so I think a lot of engineers like that, and it becomes very accessible to them before they have to go and actually talk to sales people. So, I think it’s very helpful to do that. I think really it comes down to deciding whether you’re going to build or buy something. I would say if you’re using open source software, you’re really… I’m putting that in the build category. You’re going to have to do a fair amount of work yourself. That product can work fantastic for you. I think the issue I would take with some of our quotes here is the time investment that you have to put into it to make it scale, to make it resilient, and to make those features work is much higher than typically than if you buy the product. Sometimes that is perfect for what you need, and sometimes you just have to spend the money to make the problem go away.
[DJ Schleen] Yeah, I completely agree. When you look at developers who are potentially choosing security tools… Like you’re building software, you’re going to do software composition analysis. Or you’re going to do container security analysis, and you’re going to do this in your CICD process. Developers are going to look at those quick tools that they can manage and that they can get in right there because the security organization is the one holding the budget. Right, so when you start looking at smaller security or even absentee security teams for those small startups, the developers are the ones who are in charge of the security. And they’re going to turn to the open source community first. There may be no alternative in the commercial space or the commercial space for the startup might be a little bit prohibitive from a cost perspective. As you start growing though, you’re going to need to tie in that information more on an enterprise scale. Like the things that come out of these processes. But for the initial bang for your buck, you might get this into your pipelines. You might get this into your development processes, and it might be fine.
Sponsor – SPMB
10:04.580
[David Spark] DJ, before we go on any further, I want to ask you a question about Yahoo Paranoids, where you work. First of all, as I understand, Yahoo Paranoids is just like the name of the security department at Yahoo. Correct?
[DJ Schleen] Yes, it is. Yeah, we’re known as the paranoids. Been around since I believe 1998 in various forms.
[David Spark] Okay. So, it’s not like a different product or anything. It’s just you’ve got a brand for the security department. Which by the way, I love that idea. More companies… Geoff, you should think about this. Having a brand name for the whole security team. By the way, is there some paranoid swag, or T-shirts, or jackets, or something like that? Like you’re a gang of some sort?
[DJ Schleen] We got stickers all over our laptops, jackets. We throw some pretty big events at DEV Con.
[David Spark] Okay.
[DJ Schleen] So, if you’re going to be going to DEV Con, check us out over there.
[David Spark] All right. And the big question that I always ask on the show is are you hiring.
[DJ Schleen] Absolutely. We’re open for business all across Yahoo for hiring new folks in various different organizations – engineering, non-engineering, sales. And we have a lot of open positions for the paranoids, and we’re growing.
[David Spark] All right. So, I’m assuming they can see it on the site, but also we’ll have a link to DJ’s LinkedIn as well. And you can reach out, I’m assuming, to you via LinkedIn as well, yes?
[DJ Schleen] Anytime. Yeah, reach out as much as you want. I’d love to talk to you.
[David Spark] Before I go on any further, I do want to mention our sponsor. They are pretty fantastic, and you’re going to want to hear this – especially if you’re looking for your next leadership role in cyber security or you’re looking for the great talent in cyber security. SPMB is the number one executive search firm serving the technology market and one of the largest independent retained search firms in the country. For 45 years they have specialized in recruiting C-level executives and board members to large multinationals across all categories – media, consumer, financial services, healthcare, renewables – on their path to digital transformation. SPMB also partners with disruptive growth oriented startups, building out the leadership teams at the most innovative companies in the tech space.
They bring their knowledge of the large global firm and combine it with what you need – the personalized service and attention of a boutique to connect top executive talent to the best and fastest growing innovators across the country. Closing hundreds of C-level searches annually, SPMB has recruited key leaders into companies that have generated over one trillion in market value for our clients. A key area that SPMB brings extensive knowledge and expertise to is its dedicated security practice, leading both functional services, so CISO and VPs, defining security strategy, and building out executive teams at top security software companies. To learn more just go visit them at their website. That’s the SPMB website. And what do you think the address is? Spmb.com.
Can it be solved?
13:06.065
[David Spark] Dan Holden of BigCommerce said, “In my experience, the more open source you use, the more people you need.” All right, this is getting to the topic at hand.
[Geoff Belknap] That sounds familiar.
[Laughter]
[David Spark] And Toby Lewis of Darktrace said, “I imagine what you save in software licensing you spend in human expertise and blood, sweat, and tears to keep the thing working.” Robert Blumofe of Akamai Technologies said, “In my cases, commercial software systems, tools, and intelligence will have value above what can be found in open source.” And James Nofsinger of Lightstep said, “Yes, you can to a point. Sooner or later, something won’t scale – either people or process.” Now, I’ll go to you, Geoff, and I’ll go off of James’ comment. Every one is making the same comment that both of you have made multiple times this show of you’re going to need people to deal with this. But you argued about a lot of people claiming you can’t scale, but they’re specifically saying about people and processes is going to have the problem of scaling. What do you say to that?
[Geoff Belknap] I think that’s exactly right. At some point you either have to invest enough people that you are building either the automation or the advanced features that you’re not getting because you’re using open source that you have to ask yourself, “Is this worth it?” So, if I look at the fully loaded cost of a security engineer, which just I’m going to nominally say it’s $250,000… How many of those am I willing to invest…? And by the way, it’s probably higher than that. But how many of those am I willing to invest before I just go, “This is not worth it. I should just be buying the software.” And then moving those extremely valuable resources onto something that’s more core and strategic to my business.
I think the thing to keep in mind is like let’s say I can find ten engineers at that price, and I’m talking about two and a half million dollars for the year. Those people should be working on the things for my business, not running an EDR or running like a CASB or something like that. That is not a good use of my people, the best resource that I have. So, I think this is where we talk about what’s the tradeoff. Am I going to just spend the money to make the problem go away, or am I really going to invest ten of the brightest, most in demand people with the best skillset that is hard to find on building the same thing? I think at the end of the day, that cost just doesn’t make sense.
[DJ Schleen] I think with any of these tools or solutions that you use, you got to matrix them out when you’re doing your evaluation. Because if you take into consideration air gapped environments or places where you have to install a vendor software on prem, you’re in the management business of that infrastructure and that platform, so you’re going to have to take that into consideration. Again, if you have these software as a service, that’s one thing. We can offload it. Things get handled for us. But if we have bugs or issues internally, we’re still in that outage period. We have to look at how we’re going to maintain it, availability, identity and access management, all these different things around some commercial software that we might be putting into our environment as well.
[David Spark] Let me throw this out – one theme that we hear again and again is complexity if the enemy of security. Well, as you were saying, if you have to put more people to deal with the complexity of some open source software and to finetune it, aren’t you fighting against what you’re trying to achieve with security, Geoff?
[Geoff Belknap] Yeah, I think that’s exactly right. Look, there is a reason why commercial software exists. It’s not because open source is better, but nobody knows about it. It’s because there are a bunch of fantastic open source projects out there that can absolutely satisfy some needs. But the reality is a lot of innovation still happens in the commercial space. There are absolutely commercial pieces of software that do things that open source software doesn’t. And whether that be find vulnerabilities, or find tacts [Phonetic 00:17:02] that your open source software isn’t, or whether it just be making it easy to manage at scale, there is value in spending money on software to solve a problem for you.
What aspects haven’t been considered?
17:10.283
[David Spark] Juan Pablo Castro of Trend Micro said, “If you find a way to ensure that cyber criminals don’t infiltrate malicious open source components, whitelist malicious components and create malicious open source intelligence, it might work.” Juan Pablo pointing out another issue that is specific to open source. And Ofer Shaked, again, of Deepblue, said, “Some people who are a DYI, do it yourself, enthusiasts do all sorts of creative developments, and even they use a lot of paid tooling. I would never go that far in a business setting.” So, I think Juan Pablo makes the good point of, well, sometimes these open source projects get poisoned by malicious software, and that becomes an issue. And then Ofer said, “Well, even people who are into this, they do a mix of open source and paid.” DJ, I get the sense that Ofer’s comment is kind of the theme here of we all like open source, but all the way, that’s a tough call.
[DJ Schleen] It’s going to be tough to get 100% coverage across all the elements of a security program. Like GRC? Yeah, you can use open source, things like a Ramba [Phonetic 00:18:32]. But who’s supporting that? It’s this whole community. We got to be careful when you talk about the software supply chain. There’s a lot of components that come in. You can have these supply chain attacks on the open source code. But as well, the vendors who are using it or developing the products are also using the same open source code and components at times. So, we have to treat both with the same amount of cautious analysis and cautious review as we look at a security analysis of the software that we’re bringing in.
[David Spark] So, DJ, in all the work that you’ve done, has this been your solution? Like you’ve had some open source, and you’ve had a commercial solution. And it is the mixture that creates your security program? Yes?
[DJ Schleen] Yep. Actually the past ten years, it’s been that way. It’s sort of a race. Sometimes we’ll have open source software that’s addressing a new problem in the industry that hasn’t been taken care of yet or hasn’t been looked at. And then there will be other software that is the commercial, heavy duty, heavy lifting software that’s doing some scanning, or some monitoring, or observation, or data analysis, or those kind of things. So, it’s definitely a combination of the two. I think as the business matures and grows, you’re going to go from that point where you’re using a lot of open source to a hybrid. I don’t think you’re ever going to see one extreme or the other.
[David Spark] Geoff, I do want you to touch on Juan Pablo’s comment. Is that a serious concern of open source software getting poisoned?
[Geoff Belknap] If I’m honest and certainly no disrespect to Juan, but I think that’s a nonsense concern. I think the target of cyber criminals when it comes to open source projects are usually broad spectrum. They’re going to hit a Chrome extension, or they’re going to hit something that everybody has installed because they want the broadest total addressable market to monetize. I think in the case of sort of the context of the discussion that we’re having, it really is going to depend on the choice that you make in terms of the mix between paid product and open source product… It’s going to depend on both your threat model and the kind of organization that you’re in. If you are a small nonprofit, you are one person, maybe two, and you’re almost certainly going to choose open source software because your threat model is probably very relaxed. And you just don’t have the people or the resources, and you’re never going to have them. But you’re trying to do the best you can for a charitable organization.
If you are a giant tech company or a defense contractor, you are going to be much more thoughtful about the choices that you make and the threats that you’re actually defending against. I think everything comes into play then of yeah, you might be concerned about foreign intelligence services invading the software that you might be choosing. But you’re also going to be concerned about the fact that you have a lot more money and resources, but you have limited people to put into different focused areas. You’re going to choose commercial software that gives you the most lift with the people that you have. The reality is exactly at Ofer is saying here – that you’re going to use some amount of paid tooling, and you might either develop or implement some open source products that you already know about.
Closing
21:34.577
[David Spark] Excellent. Well, that brings us to the end of our conversation today. This has been great. And I think we’ve also confirmed that the commercial cyber security industry does not have to worry – that they will still be very successful selling their products, that open source is not taking away all their business.
[Geoff Belknap] Yeah, don’t worry, vendor. We still need and love you.
[David Spark] Yes. And by the way, the comment that you made, DJ, about sometimes open source jumps on something before a commercial product, I’ve definitely seen that happen. And I have seen the commercial products because they’re like the first to market, they have to sell themselves as, “We’re better than the open source thing that you’re currently using.” And I’ve seen that happen often.
[DJ Schleen] A lot of times, as well, you start looking at companies like Semgrep who are doing static analysis. They started off with an open source product, and then they ended up bringing a commercial product on top of it that had a whole bunch of enterprise features. So, you see that a lot, a lot incubated as open source and them come into the commercial space.
[Geoff Belknap] So much good stuff comes from that path.
[David Spark] I have seen that path. Again, we’ve had sponsors take that very path as well. Now we’ve come to the point of the show where I ask both of you which quote was your favorite, and why. We’re going to wrap it up with this. So I will start with you, DJ. Which quote was your favorite, and why?
[DJ Schleen] I think my favorite quote was Bruno from Datasec where he said, “Possible? Yes. Feasible? Yes. Practical? Not much.” Just because you can do something doesn’t mean you should do something. And if you do do something, it might not turn out the way you want it to. So, I really like that quote and how it resonates to the topic.
[David Spark] I think it thematically kind of wraps up our entire show. I like it as well. Geoff, your favorite quote?
[Geoff Belknap] Let’s see. There’s a bunch of good ones here, but I’m going to go with Toby from Darktrace who said, “I imagine what you save in software licensing you spend in human expertise to keep the thing working.” I think that has been my experience time and time again. You’re just trading off. Like whatever I’m not spending in cash, I’m going to spend in people. And sometimes that is the right choice but not always.
[David Spark] Very good point. Well, let’s wrap up this whole show. Thank you very much to my cohost, Geoff Belknap, CISO of LinkedIn, who is usually always looking for great cyber security talent. And if for some reason you didn’t want to work with him, there’s also DJ Schleen, who is also looking for great talent over at Yahoo Paranoids where you can find him also on LinkedIn. The link to his page is available on the post for this very episode as well. And a huge thanks to our sponsor, SPMB Executive Search. They are the executive search firm for innovators. Remember that. They have a very much cyber security focused effort. They’ve been doing this for decades. Check them out at spmb.com. Again, to our audience as well, thank you so much. We greatly appreciate your contributions and listening to Defense in Depth.
[Voiceover] We’ve reached the end of Defense in Depth. Make sure to subscribe so you don’t miss yet another hot topic in cyber security. This show thrives on your contributions. Please write a review, leave a comment on LinkedIn or on our site, cisoseries.com, where you’ll also see plenty of ways to participate including recording a question or a comment for the show. If you’re interested in sponsoring the podcast, contact David Spark directly at david@cisoseries.com. Thank you for listening to Defense in Depth.