Defense in Depth: Measuring the Success of Cloud Security

How are you measuring your progress and success with cloud security? How much visibility into this are you providing to your engineering teams?

Check out this post and this post for the basis for our conversation on this week’s episode which features me, David Spark (@dspark), producer of CISO Series, co-host Geoff Belknap (@geoffbelknap), CISO, LinkedIn and our sponsored guest Matthew Chiodi (@mattchiodi), CSO, public cloud, Palo Alto Networks.

Got feedback? Join the conversation on LinkedIn.

Thanks to our podcast sponsor, Palo Alto Networks

If you’re doing cloud security right, no one knows if you’ve done anything. When you do it wrong, well, you end up on Cybersecurity Headlines. Prisma Cloud from Palo Alto Networks helps ensure your security stays in the quietly appreciated group. It’s a single security platform that delivers comprehensive protection from code to cloud. Learn more at paloaltonetworks.com/prisma/cloud.

Full transcript

David Spark

How are you measuring your progress and success with Cloud security? How much visibility into this are you actually providing to your engineering teams?

Voiceover

You’re listening to Defense in Depth.

David Spark

Welcome to Defense in Depth. My name is David Spark, I am the Producer of the CISO Series and joining me on this episode is Geoff Belknap, CISO of LinkedIn. Say hello to the audience, Geoff.

Geoff Belknap

Hello friends, thanks for coming.

David Spark

That is Geoff. You’ll hear his voice far more often than just that. Our sponsor for today’s episode is Palo Alto Networks. They have been a very good sponsor in the past and, guess what? They’re back again. Thrilled to have Palo Alto Networks back. Today’s topic is measuring the success of your cloud security program. Like anything in cybersecurity, we want to measure how good, or poorly we’re doing. But, I think we’re more talking about the fact that Cloud security needs constant management working and tweaking. How do we go about actually doing that and I now you asked this question to the community and, how satisfied were you with the answers and how confused do you think the audience still is? Geoff, go ahead.

Geoff Belknap

Yes, I think, I was really excited that we had a lot of engagement of the topic. I think, overall, my assessment is, people are still figuring this out; so, I think it’s a great thing for us to talk about today. Especially with our guest.

David Spark

Yes. I felt there was still a lot of confusion, as per you said and, hopefully, we’re going to come to some more understanding and direction, I think is the best way to put this. Because, as I had heard from another CISO in the past, he said, I would like to know, when I set it up today, in six months from now, how messed up is my configuration from, you know, that place? Hopefully, by the way, I’ve done something in the past six months as well. I would like to know, how far from optimal am I at this point? That’s where we want to get this conversation to go. The person to help us with that very conversation is someone from Palo Alto Networks, actually. It is our sponsored guest, Matthew Chiodi, who is the CSO of the Public Cloud. Matthew, I’m expecting you to know something about this. Yes?

Matthew Chiodi

Absolutely, yes I do and I’m glad to be here.

How do we handle this?

00:02:27:22

David Spark

Ashish Rajan of Cloud Security Podcast points to elements that should be in place before you start measuring. He said, “right skills in the security team, visibility with context, sharing what the security team knows or uses with the engineering team”. If you feel you have some control of any of these, you have the right datapoints to start planning for what success in a Cloud/hybrid environment can be. Abdoulkader Dirieh of BD said, you should first be asking, “what requirements do you tend to choose to decide what needs to be measured?” I will start with you, Geoff. Do a lot of sort of questions, or things need to be in place before you can do any kind of Cloud security management?

Geoff Belknap

Depending on what kind of organization you’re in and the age of the organization, these are the right set of questions to start with. Do you have a functional and performant security team? If you do, then you can really start measuring the impact of that team. Whether it be on Cloud security, or on Premise, or your application security, you have to start with having the right pieces and making sure that they work together. Once you have that together, I think Abdulkader is on the right track; which is, what is Cloud security to you? What is successful Cloud security to you? Because, yes, there are some fundamentals and I think some of the other commenters on some of the conversation talked about the basics; you know, whether it be CIS top 20, or whether you’ve built your own framework for that. But, beyond that, what is important to your business to ensure that it can succeed and grow and thrive? There is no single set of metrics that are right, it really comes down to what the right choice is, based on your threat model and your business goals.

Geoff Belknap

I will throw this to you, Matthew. Are these the right questions to be asking and are we missing some ones to be asking before we even begin the measurement process?

Matthew Chiodi

Looking at some of that feedback that came online, I think there was some really good wisdom in there. But, for me, measuring Cloud security success has to start with mapping it back to the business’ Cloud goals. Organizations are moving to the Cloud for various different reasons and I know a lot of times, as security practitioners, we tend to think that security is only about security all the time. I would work back from whatever the business’ Cloud goals are. There’s a reason they’re going.

David Spark

Can you just give me one classic example. We’re moving to be more elastic, or to save money or, you know, whatever it is. Give me, like, ok, if this is a goal, this is the security parameter we want to be looking at.

Matthew Chiodi

Absolutely. One of the main reasons that I often hear from organizations is, they want to be able to release faster. Whatever it might be. Whatever their product is, they want to be able to release faster. That is something that you can very clearly then map to metrics. I did a blog on this probably almost a year ago, on the New Stack and I think it was called something like, How Dev Ops can save security. I called out on there, just a number of metrics that you can use that will clearly tie to that type of organizational requirement when moving to the Cloud. I think there’s some nuances we can get into there; but, that’s an example of starting at that place. Instead of starting with the control, which is a part of it, I want to start from those organizational requirements and then the visibility, sharing, etc, those then become inputs to help measure progress toward those goals. As security teams, those are top level metrics. They’re going to have their own metrics that are specific to the security team, that are likely to be mapped to, you know, a NIST 800-53 or something like that; but, I think you’ve got to start with the organizational goals and work back. A lot times, we don’t do that in security.

David Spark

Is there a case, Geoff, where you have set something up and gone, ah, I think this is what we should do and, then, it clearly is not the thing. Are you able to shift gears easily, as a result?

Geoff Belknap

Look, all metrics start out life this way, if you have highlighted that you want to be able to iterate quickly, or ship quickly. Because you want to be able to move fast as a business, you start with one thesis of what the security metrics are that impact your ability to ship quickly and you zero in on understanding what metrics are useful and what ones aren’t. This, again, goes to the beginning of this conversation where, you’re going to measure some of them wrong things. This is why we tell people, don’t just start measuring everything, don’t just tick every button and send 1000 graphs to your executive leadership team because, that’s a great way to lose credibility. A great way to start is, just pick four or five metrics and see if you can actually impact those metrics with other changes that you make. If those actually have some impact on the thing that you’re trying to improve, then you have the right metrics. All metrics start with a guess of what you’re trying to measure and whether that impacts your goal and it’s a process, like everything else.

What are the best practices?

00:07:48:10

David Spark

Raymond Pompon over at FI said, “compliance tells you something about the organization hygiene efforts and commitment to security. You will never have definitive proof that something is “secure”” Courtney Bramlett of VerSprite Cybersecurity said, “measuring Cloud is against frameworks and best practice. Then apply these frameworks via a lens created by risk-based threat models.” We heard a lot of argument about pro and con, about using frameworks as sort of a measurement model. What say you, Matthew, on that?

Matthew Chiodi

I think that they are a good baseline. I’m a huge fan of what CIS created, the Version 8, they came out with there CIS 18. One of the teams that I lead at Palo Alto is our Unit 42 Cloud Threat Research team and we put out a Cloud threat report about every six months. We analyzed about 18 months worth of Cloud breaches, we did this probably a year to year and a half ago. What we found was that, the vast majority of those 65% were the result of customer misconfigurations. 65%, so the bulk of them. For me, what we continue to see, when we do these reports, is that organizations are struggling with the basics. I always tell people that, I don’t know what you’re doing, maybe you’re trying to meet PCI, whatever it might be; obviously you’ve got to meet those minimum baselines. But, I just say, go back to something like a CIS 18 and start to measure around, how well are you doing those? Those should be considered the baseline. Once you are doing those well, or you’re making progress towards those, then you can start to move onto perhaps some of the more advanced things. As security practitioners, we love shiny new objects, new tools, new projects; but, what we consistently see, in our threat research again, is that organizations are struggling with even just some of the basics. Again, if you are subject to a regulatory framework, you’ve got to meet that, you don’t have a choice not to. But, at the same time, I think that, starting with the basics like a CIS 18 is really important.

Geoff Belknap

I think people are also really surprised about how difficult it can be to just measure things consistently. Because, if you haven’t been doing that already, especially if you’re new to your Cloud journey, I think the Mac gives advice that I would also give people. Start with some framework, any framework. It almost doesn’t matter. Obviously if you have regulatory requirements, start there. But it doesn’t matter because, what you need to learn first is, are you any good at measuring that thing consistently? I’ll give you a great example. If you’re going to measure, for example, whether you’re getting telemetry from every hose, well that sounds really simple. But now you have to unpack that and go, well, how are we measuring what every host equals and how that changes from day-to-day because of the Cloud? You don’t start with ten hosts and end with ten hosts. You might start with ten hosts and, the middle of the week, go to eight and then end with 15. Well, that brings complication to metrics and I think, a lot of people overlook that that takes a fair amount of engineering work and craftsmanship and execution to be able to do well and all of that underpins your ability to understand the security posture that you have in your Cloud. Just start somewhere and then you can get to really understanding how to interpret that data.

Matthew Chiodi

Someone wise once said that, you get what you measure.

Geoff Belknap

Yes. Although, sometimes you have no idea what you’re measuring; so you’re surprised by

Matthew Chiodi

That is also true.

There must be a better solution.

00:11:28:18

David Spark

Andrew Scully of Ampion said, “I personally couldn’t care less about compliance to frameworks, unless it’s a regulatory requirement, if we are meeting these outcomes. For me, frameworks are fantastic tool sets but not a measure of success.” So, very much tagging on what was just discussed; but, a good starting point, as you said, Matthew. I also quote what Stu Hirst, CSO of Trustpilot said, “I’ve set up a company-specific guardrails framework covering all the areas of AWS we use. We can then measure whether we are preventing those things from happening.” I found Stu’s comment very interesting there. Have you see that before, Matt?

Matthew Chiodi

I love what he said and I’ll drill on the one part of his quote. He said, “all the areas of AWS we use.” I like that because, it shows me that he is fully aware of what they are actually using in AWS. This is a CISO who is very aware of what services are actually being used.

David Spark

Which turns out not to be easy, right?

Matthew Chiodi

Not to be easy and not very common. That is really important. We’re talking about AWS here, but it could be any Cloud service provider. They offer hundreds, if not thousands of services; but, what we’ve seen is that, many organizations, they’re typically only using a small fraction of those services. In Stu’s case, he doesn’t have to worry about each and every service that AWS offers, he knows he’s only using, you know, these ones. He can focus his time on measuring how well he’s securing those services and, potentially, maybe he can map those then back to the organizational goal. We started off kind of like, ok, here’s the high level ones, here’s organizational and now Stu was kind of talking more of the tactical level of measurement. You’re not likely to report those types of low level metrics to a board; but there’s a way to bubble those up to those higher level organizational metrics. I love that. I think what he said is spot on.

David Spark

I will throw this again to you, Geoff, in that, have you created your own custom framework to what you have set up specifically for Cloud environments? Are we meeting, essentially, the goals of what we’re trying to pull off here, but specific to this environment?

Geoff Belknap

I think, you know, Andrew and I are of like mind here where, I don’t really care about frameworks explicitly. I think Stu is on the right track, there’s a set of policies and standards that we articulate, as a central security organization, that other teams and people that are shipping product need to follow. I want to know if we’re staying true to that. I want to know when there’s drift both in configuration, but also drift from policy. Then I want to be able to measure, did we approve that drift from policy, or did that just happen all on its own? To Matt’s earlier point, a lot of companies, including mine, want to be able to go really fast and what comes with that is this trust relationship. Great, we’re going to put guardrails policies in place. But a lot of people don’t realize that AJAand AWS and other Cloud providers have policy engines built into that, where you can articulate what your business policy is, like AJA in AJA policy. Then you can measure when there’s drift from that. Whether that be to the micromanagement level of the version of an image you’re using, or whether it be like a NACL, or Network ACL,or something that’s in place. I want something that I guess you could define as a framework. I want to understand when there has been drift from policies, when there’s been drift from standard configurations and then I want to be able to take action based on that. People can call that a framework and I think most people would call their framework when they download it from the Internet, or borrow from something somebody else has built. But that’s all we’re really talking about, when we use the fancy term framework here, we just want to make sure that what we’re running with right now is in the same kind of state as it was running an hour, or a year ago. We want to understand how much drift exists and what we need to do about it.

David Spark

Let me ask both of you this quick question. I’m guessing it’s obvious, but there might be a more nuanced answer to it. What does drift look like? Because, when you set something up, you don’t know if you’re in the optimal state and, like you say, are we better or worse than we were before? Matt, what does drift look like?

Matthew Chiodi

Yes. I mean, to your earlier point, there is a way to know what good looks like. You’ll hear me always talk about CIS. I just like what they do so much.

David Spark

We quote them all the time, by the way.

Geoff Belknap

OK, that’s awesome. They’ve got their secured OS’s, they’ve got secure benchmarks for all the major Cloud service providers; things like Kubernetes. We do know what good looks like to start. Let’s say, for example, you start and you have a Kubernetes cluster and you’ve followed CIS’s frameworks and you’ve secured it to that. That is a good known state. Then, let’s say, three months goes by, someone goes in and they’ve now made a change to that configuration and now you’re drifting from that. Maybe they’ve configured a cluster in such a way that they’ve made remote administration to that cluster more promiscuous. That is a very concrete example of drift. It’s great that you start out secure, you start out from a CIS benchmark, or something like that, or from a DISA STIG if you’re on the public sector side. But, as soon as people start to interact with that, that is a huge part of it.

What else is required?

00:17:19:13

David Spark

Sort of an interesting prescriptive comment made by Christoph Puppe of System Vertrieb Alexander. He said, “define secure configuration and code quality baseline, automate the audit, measure the number of code issues, compliant resources and policies. Give each team access to the issues created by the audits by pushing them into their repos. Measure team performance by number of opened and closed issues.” Sort of a basic linear explanation. Does this read well to you, Geoff?

Geoff Belknap

Yes, I think, generally, this is what we’re going for. I might snipe at a couple of things here. But, I think, the spirit of this is exactly right. In a perfect world you’re defining, declaratively, what the configuration standard should be in part of your build pipe; however you build and ship. The only things that should be changed, or drifted, or a delta from this standard configuration are things that you’ve already approved; or things that are application specific, or specific to whatever you’re building that’s not part of the standard image. When it’s not, you need a process and I think Christoph here a very simple version of a complex idea; which is, when the configuration is different, you need to push that information to the person that’s most able to understand whether that’s a problem or not and most able to do something about it. Sending all these alerts to the security team to say, hey, this bit is set to zero instead of one on these 1000 machines and last week is different, the security team doesn’t necessary know that that is good, or bad. That might have been a memory optimization on Java running, or something like that. But the team that made the change, they should know whether that’s good or bad and then, they should be able to explain to you, oh, that’s a change we made in a running production system that’s different from the configuration standard because of x. Or, they’ll tell you, we have no idea why that change happened and now the security team can get involved. That is an articulation of a high-functioning, mature security environment. Notice I said nothing about zero days, or fancy vendor technology. Because I think, the thing we really need to underscore is, this is a conversation that is both critical and extremely boring and thee most extremely boring conversations are the sexiest part of security sometimes. Because, this stuff will protect your environment far better than almost any vendor technology you can buy. Certainly you can buy technology to make this stuff easier and I encourage you to do that in all cases; but, this is the kind of stuff that people forget about all the time.

David Spark

The critical and extremely boring is sadly when it is breached and what makes headlines.

Geoff Belknap

The boring stuff is always the hardest stuff.

David Spark

Yes. Matt?

Matthew Chiodi

Yes, I mean, I think that I love what Christoph said. I mean, organizations that operate at scale in the Cloud and do it securely, they’ve got to automate as much of their security processes as possible. I always tell security teams, you should be the ones that are championing the infrastructurer’s code; doing everything as code. Because, if it’s in code, I can inspect it. When it’s manual, it’s way harder to identify. If it’s in code, I can also measure it. Peter Drucker said that, efficiency is doing things right and effectiveness is doing the right things. I always tell people, when they’re looking at measuring Cloud security, or whatever it might be, that they make sure they have both effectiveness metrics, as well as efficiency metrics. Like I said, I did that blog about a year ago and I called out a handful of those from a DevSecOps perspective that I think can be really helpful. You can find those on the New Stack.

David Spark

Can you echo any of them right now?

Matthew Chiodi

Yes, I have a couple of them in front of me. One of them would be, for example, if you’re looking at your CI/CD pipeline, one metric might be something we would call continuous integration, vulnerability management discovery. That would fall under the category of, how effective are we? It answers the question, how effective are we at discovering vulnerabilities, pre-production across the CI pipeline? That’s an effectiveness one. Now, you can look at that same piece from an efficiency perspective. You can say, the metric would be CI vulnerability management backlog remediation. That answers the question, how efficient are we at addressing vulnerabilities, pre-production in the CI pipeline. These are the kinds of metrics that you can tie, very well, to an organization goal that says, hey, we want to be able to move faster and release faster. Now, am I going to report this to board? No. They don’t want this level of detail. Is it something that I’m going to talk about and have conversations and make a scoreboard around for the security and the dev ops team? Absolutely.

Closing

00:22:12:01

David Spark

Well, that brings us to the close of the show. Thank you for those great examples, Matt. At this point, I will start with you, Geoff. What was your favorite quote and why? By the way, my favorite quote, obviously, was yours that you just mentioned ago, about critical and extremely boring.

Geoff Belknap

Well, I’m always happy to provide.

David Spark

Not a way I would describe you, Geoff.

Geoff Belknap

Oh, well, I think there are many people that would differ on that.

David Spark

Critical and extremely boring.

Geoff Belknap

Critical and extremely boring. Maybe cynical and extremely boring. You know, a lot of great choices here. I’m going to have to say it’s a tie between, for me, the key elements you need in place before you can get started. Because, I frequently have to bring people back to that. This is not an easy thing to do. It’s an easy concept to understand, but sometimes executing it is much more complicated than you think. Then I have to go to Christoph. Although I wouldn’t articulate it exactly the way Christoph does. But I think, what he lays out here is, define your secure configuration, define your policies, build some automation around it and then start measuring. This is a great basic road-map for how you get from zero to one, if you haven’t done this already. I think that’s a great way to be.

David Spark

Good example. Matt, your favorite quote and why?

Matthew Chiodi

It would be Andrew Scully. He said, I always measure security against defined organizational outcomes. I’m a huge strategy person. I feel like so many security teams are just so stuck in the weeds with tacticals, that they never have the time to actually stand up above and look out and say, are we actually even in the right forest? I think this is where you’ve got to start. It’s got to start top down.

David Spark

Great advice. We have come to the very end of the show and, Matt, I’ll let you have the very last word. Any offers, or anything you’d like to say about Palo Alto Networks, specifically in this area we’ve been talking about; with regards to measuring the success of Cloud security. Please suggest. I always ask to my co-Host and guest, are you hiring? Make sure you have an answer for that. But, first, a big thank you to our sponsor, Palo Alto Networks. Thank you so much for sponsoring this episode and multiple episodes of the CISO series. Geoff, any last thoughts and I’ll say it for you, because I always know, you’re always hiring, right?

Geoff Belknap

We’re always hiring. If you’ve got some Cloud security skill-sets, or application security, or any security skill-set whatsoever, we are interested in talking to you and you can find us at LinkedIn.com.

David Spark

I know how to lock a door. Does that count as a security skill-set?

Geoff Belknap

I am happy to talk to you about your basic skill-sets and see if you can build yourself up into a fully performing engineer on our team. Come and join us, David.

David Spark

All right. Thank you for the offer. Matt, any specific offer from Palo Alto Networks, or you’re hiring? Or, anything you’d like to call out?

Matthew Chiodi

Yes, absolutely. We are also always hiring. We just surpassed 10,000 employees in the last month or two; so, we’ve had a massive amount of growth. We are always hiring. I would just say, if people want more information on our Cloud threat research, you can just go to Cloudthreat.report. You can find our biggest Cloud threat report. If you want to find more information on our Cloud security offerings, just go to Prismacloud.io and everything you want to know will be there. It was fun being here Dave, it’s a great conversation.

David Spark

I am glad you liked that. Let me ask you, Matt, how would someone get in contact with you?

Matthew Chiodi

Ah, well there’s a couple of ways you can do that. Number one would be on Twitter; it’s just @MattChiodi.

David Spark

Spelt just the way it sounds.

Matthew Chiodi

Yes, just the way it sounds; especially if you know Italian. @MattChiodi. I also have my own podcast, which I hope I can plug here. Cloudsecurity Today. If you go to Cloudsecuritytoday.com, you can hear my fledgling podcast, which just released its fifth episode.

David Spark

Oh, congrats. Good stuff. Does take a lot of work, I know, so I’m sure you’re discovering that.

Matthew Chiodi

Yes I do. I do it on my own, so it’s a lot.

David Spark

Thank you very much Matt; thank you very much Geoff; thank you very much Palo Alto Networks and thank you audience, as always. We greatly appreciate your contributions and listening to Defense in Depth.

Voiceover

We’ve reached the end of Defense in Depth. Make sure to subscribe so you don’t miss yet another hot topic in cybersecurity. This show thrives on your contributions. Please write a review, leave a comment on LinkedIn or on our site: CISOSeries.com where you’ll also see plenty of ways to participate, including recording a question or a comment for the show. If you’re interested in sponsoring the podcast, contact David Spark directly at David@Cisoseries.com. Thank you for listening to Defense in Depth.

David Spark
David Spark is the founder of CISO Series where he produces and co-hosts many of the shows. Spark is a veteran tech journalist having appeared in dozens of media outlets for almost three decades.