Defense in Depth: How Do You Measure Cybersecurity Success?

In most jobs there’s often a clear indicator if you’re doing a good job. In security, specifically security leadership, it’s not so easy to tell. “Nothing happening” is not an effective measurement. So how should security performance be graded?

Check out this post for the basis for our conversation on this week’s episode which features me, David Spark (@dspark), producer of CISO Series, co-host Geoff Belknap (@geoffbelknap), CISO, LinkedIn, and our guest is Deneen DeFiore (@deneendefiore), CISO, United Airlines.

Got feedback? Join the conversation on LinkedIn.

Huge thanks to our sponsor Tessian

95% of breaches are caused by human error.
But you can prevent them. Learn how Tessian can stop “OH SH*T!” moments before they happen, why Tessian has been recognized by analysts like Gartner and Forrester, and which world-renowned companies trust the platform to protect their data.

Full transcript

David Spark

In most jobs, there’s often a clear indicator if you’re actually doing a good job. But in security, specifically security leadership, it’s not so easy to tell. “Nothing happened is not an effective measurement.” So how should security performance be graded?

Voiceover

You’re listening to Defense in Depth.

David Spark

Welcome to Defense in Depth. My name is David Spark, I’m the Producer of the CISO series and joining me for this very episode is Geoff Belknap who is the CISO over at LinkedIn. Geoff, the sound of your voice?

Geoff Belknap

Welcome to another fantastic episode of Defense in Depth, David.

David Spark

You do an amazing impression of yourself.

Geoff Belknap

It’s something I work at.

David Spark

You’ve nailed it just to a tee.

Geoff Belknap

Well it is an art, thank you.

David Spark

Let me mention our sponsor for today’s episode, Tessian. We are thrilled to have Tessian on, a new sponsor with us. They do human layer security. You’re going to hear a little bit more about that later in the show. But first I want to mention our topic. It comes from Helen Patton, who’s the advisory CISO over at Cisco, and she asks the community this question: how do you know when a security leader is doing a good job? What’s interesting about this question is I actually talked about it on the other Podcast, the CISO Security Vendor Relationship Podcast, and the leaders themselves often don’t know if they are doing a good job or how well they are performing. So let me ask, Geoff, do you sometimes question to how well you’re performing?

Geoff Belknap

Only on days that end in Y! It’s especially acute for CISOs because, like we talked about before on this Podcast and many others in the network, it’s a relatively new career path, it’s a relatively new role, and unlike some others, we’re still kind of talking about what makes us good at what we do, what validates that we’re adding value to our organization. And I think today’s a great day and our guest is great guest to have this conversation.

David Spark

Yes, very thrilled to have her on. We had her on the CISO Security Vendor Relationship Podcast. She was so amazing, I said, “You have to come on Defense and Depth,” and here she is. She is the CISO for United Airlines, Deneen DeFiore. Deneen, thank you so much for joining us.

Deneen DeFiore

What’s going on?

Thanks, it’s great to be back. I’m excited to be here with you again.

David Spark

Julian Cohen, who is CISO over at Ocrolus, said, “Is the security team designed, built and managed well for the type of organization it serves?” And David Nolan, of the Aaron’s Company, said, “Applying problem solving and creativity to security versus just tools,” so he was arguing that, that is the sign of a security leader doing well, not just plugging in tools but actually true problem solving. So design and problem solving I think is what they sum it up to be. Geoff, your take.

Geoff Belknap

Yeah, I think in most cases when Julian has a thought on a topic like this I’m going to lean towards him.

David Spark

By the way, he had lots of good comments on this, I should mention.

Geoff Belknap

Always does. And hey Julian, I miss you. But the real heart of the matter here is if we unpack what Julian’s saying. Is the security team designed, built and managed well for the type of organization it serves? So a security team’s objective is not to buy a bunch of stuff and plug it in, or spend a bunch of money on services or contractors; a security team’s job in most organizations is to make sure that the organization can go fast and far managing its risk as best to its ability. So that really is the measure of whether a security team, and by extension a security leader is doing well, is the business succeeding because of the security work or in spite of the security work. And I think when you dive into what that really means, then you have your answer.

David Spark

Deneen, can you understand what that really means? Is there any way to figure that out?

Deneen DeFiore

I definitely understand it and I’m hopefully trying to push and live towards those objectives in what I’m doing at United Airlines right now. I think like any organization, a security team has to be purpose built. So there’s different contexts, there’s different threats, there’s different risk and there’s different business outcomes that an organization is trying to drive. And depending on what those are, the security organization has to be aligned to those very nicely. At United Airlines, I’m a commercial airline, we do a little bit of government business, but I’m not an aerospace defense contractor. So my organization is going to look very different and the skill sets that I have although at the baseline will be the same, where I put expertize or maybe more resources and priority will look different than a financial services company, for an example. I always say that organizational alignment, especially in security, because it is so dynamic when you’re trying to keep up with what the business is trying to do and what the threat landscape is changing, the regulatory landscape is changing, it’s more than an art than a science. All those models and every frameworks are great, but when it comes down to it, it’s like do you have the people that have the right skill sets, that are focused on the right things, driving the outcomes you need for the business?

David Spark

So could we get just one concrete example from both of you as to what is an example of building for my team specifically versus just a generic thing? You kind of set up, don’t just buy tools and measure. So Deneen, can you just give me one concrete example?

Deneen DeFiore

In my world I had to look at cybersecurity at the enterprise, but I also had to look at the safety aspects of it. So I have a whole team that does cybersecurity engineering for aviation’s specific products, so our aircrafts, or the aircraft ground systems, which is a very different way to approach cybersecurity because there’s regulatory constraints and safety considerations versus protecting data or operational processes. So the way you would come at that is different and you need different skill sets and different people to be able to do that.

David Spark

And that’s not something you would have to worry about because that’s not in your business model at all, Geoff?

Geoff Belknap

Yeah. LinkedIn has shockingly few commercial airliners! I think in that kind of example, one of the main things that LinkedIn deals with is people’s personal information and user generated content and we spend an inordinate amount of time focused on making sure that that is safe and protected and that we are handling it with the care and with the responsibility that members have given us. I think if I took that singular focus and moved those teams over to United, if Deneen and I switched jobs, I would probably fail in that role. If I didn’t change anything and I just moved my program from here to there, I think United Airlines would be very disappointed.

David Spark

The LinkedIn program wouldn’t work at United Airlines, the United Airlines program wouldn’t work at LinkedIn is essentially what you’re saying?

Geoff Belknap

Does it play nicely with others?

Exactly.

David Spark

Jonathan Waldrop over at Insight Global said, “I think the best indicator is when your security is accessible to all users. It’s so simple, they don’t have to try to find a workaround, it works, and it’s an integral part of the business process.” And Christophe Foulon of Captain One said, “Security is part of the culture. The business is actively considering business and security risks when making decisions.” And Brandon Scherer of Charles Schwab says, “The business sees security as a trusted advisor who they willingly and deliberately bring to the table to help solve problems.” You would always want to join an organization like that, but if it isn’t there you wouldn’t want to build organization to get there, yes, Deneen?

Deneen DeFiore

Absolutely. And I think that’s probably an evolution because a lot of times when you’re coming into an organization building a program, even if you’re new in your role and there’s an established program, there’s a level of trust that you have to accomplish with the business leaders that gives you that [UNSURE OF WORD] seat at the table. So I think that’s one thing. The other thing too is, more mature organizations are going to invite their CISO to the table. Okay, we need to change the commercial model, what does that look like, what are the digital risks that we need to be thinking about? Or we’re going to go into this new market, this high risk market, what are the considerations that we have to think about from a data, privacy or a regulatory standpoint or even threat standpoint. And that takes time for everybody to get on the same page. I think that’s everybody’s nirvana. But there’s bits and pieces that I think you experience, but I wouldn’t be discouraged if you’re a CISO and that’s not happening because that does take a lot of time to get there.

Geoff Belknap

Yes, I think if you want to have a spectrum of whether you’re adding a lot of value to an organization, you can really look at exactly as Deneen is calling out. If you are just giving information to your peers and to the senior executives or the organization but you’re not really adding value, you’ve got some room for improvement. Adding value means that you are influencing the decisions that those decision makers might make about how they operate the company, not about how much head count, what money to give your organization. When you are influencing how the company is operated, you are doing a fantastic job at security because at the end of the day every company is a technology company these days and all decisions really should be made in the context of what’s the technological risk? Or what’s the risk the organization to making this decision? And if you’re involved in that decision, and if you’re involved whether it be influencing or making the decision, things are going well, you’re performing well.

David Spark

One of things I want to point out and I think this section’s good in terms of indicating whether you’re doing a good job, is all of this is kind of based on building trust. One doesn’t all of a sudden get to the point of, oh well we definitely want to bring to security into this – you have to build trust that they want to say that and that doesn’t happen overnight and so it becomes incremental actions to get there like any trust relationship, Deneen?

Deneen DeFiore

I totally agree. It is that foundation of trust. I tell people all the time you can be the best at what you do, you can be the best technical security architect or cybersecurity application engineer, but if people don’t have trust in what you do to enable what they need to accomplish, you’re not going to accomplish what you need to.

David Spark

Do you think people trust you, Geoff? Did you have to incremental things to get trust because you had credibility, but you still had to build some level of trust, yes?

Geoff Belknap

I think there’s definitely a discussion to be had about whether I’m credible! So few relationships start with a default state of I’m just going to give you trust. I think it’s like we always talk about, you have to focus on the relationships. Security can only do so much by itself and really what security can do by itself is very little. If you build those relationships, if you really invest in the relationships with your peers and your customers and your partners, that is a place where you can build a trust upon. And I’m not saying take them out for drinks and pizza, I mean the relationship building happens when you’re working together on projects or programs and you’re implementing things and people see you as a contributor to the successful outcome that they’re trying to drive, not as the person, as Deneen pointed out, really correctly so, the smartest, specialized IT person in the room but as a real partner to drive that outcome. That’s where trust comes from.

Sponsor – Tessian

Steve Prentice

Phishing and social engineering continue to be a scourge for cybersecurity because they prey on a company’s most vulnerable element – the human being. Josh Yavor, who is CISO at Tessian, recognizes that there needs to be improvements in security at the human level and that’s where you’ll find him.

Josh Yavor

It’s not really possible to train everyone to be individually resilient from all types of social engineering attacks. As humans, that’s not how we work. I’m a security leader and I will tell you I have fallen for phishing simulations, it’s a human element, we all can fall victim for these types of attacks. So our focus is on doing two things – how do we apply the right types of technologies so that we’re able to provide as much safety as possibly be default when people click links, open attachments, engagement with other humans? That’s mostly a job that technology needs to do the heavy lifting on. And the second is to really take advantage of our AI and what I mean by that is sometimes Artificial Intelligence, but also actual intelligence, so actually getting humans involved in the feedback loops in decision making when these events come up. Rather than having a traditional data loss prevention role, we can prompt and provide coaching in the moment to actually enable you to avoid making what may have been a mistake, but also not be blocked by something that was actually legitimate and intentional.

Steve Prentice

For more information, visit Tessian.com.

How do we go about measuring the risk?

David Spark

Matt Stamper, CISO over at Evotek, said, ” Organization’s adoption of a risk register that is used ubiquitously by departments beyond IT and security. Having a universal approach to risk management is integral to a successful security program.” And Dwayne Edwards over at Tenable said, “A clearly defined, well articulated with meaningful metrics, one where every stakeholder is well aware of their role and responsibility and how it affects business risk.” So this is interesting, I’ll throw this to you first, Geoff. This whole idea that everyone’s sort of defining risk the same and is on the same page as risk, is that wishful thinking?

Geoff Belknap

I think it’s wishful thinking, but I think it’s directionally the right way to go.

David Spark

But hard to get there, I would assume.

Geoff Belknap

It’s very hard, but it really depends on where you’re starting from. If your organization is new to having a CISO or new to having risk management, then yes, it’s going to take a little while to build up to that and you have to be realistic about what you can achieve in a certain amount of time. But this is what it’s all about. There are some technical things that an information security program can provide but most of what InfoSec is doing is helping add value to the organization as it thinks about risk and it thinks about how it manages that risk. Security risk is just one part of that – there are all kinds of other risks that the organization should be thinking about and a lot of times engaging really positively here can do exactly what Dwayne and Matt are talking about. It can start to help formalize how they measure that risk and how they manage that risk.

David Spark

Deneen, how do you create a one sort of risk, not that we’re all using the same risk but that everyone’s sort of measuring the risk in the same way so that you can compare apples to apples if you will.

Deneen DeFiore

It’s hard. You can’t do it in a vacuum – I think we’ve said that before. So the cybersecurity organization can’t drive that on its own. It has to be inter-connected with whatever other risk management programs are in the business. So of course if the organizations have ERM programs – Enterprise Risk Management Programs – that’s a great start, but even you can go find those inter-connections in business processes. For instance, if you’re working with procurement or sourcing or you’re supply chain organization, they have a third party risk management framework so that managers, vendors, suppliers, they’re a financial risk, they’re quality, they’re an operational risk, so why not add cyber to that and integrate that into the business process and that whole 360 risk profile, than trying to do your own cybersecurity look. So even though I may have the five or ten metrics, that I say okay this is a successful cybersecurity program and posture because we are doing these things and we are lowering the risk in these areas, what I find is more meaningful and more relate-able is when I have that into the business process and I can say, slight risk in total, looks like this and it includes cyber as a component. Operational risk, same way. So that’s how I’m trying to think about it now and moving forward, changing it up a little bit.

David Spark

The other real key here is to speak about risk in security in the same way that the rest of the business speaks about risk. So I imagine, and Deneen for you, if you’re a commercial airline you’re managing risk.

Deneen DeFiore

Yes.

David Spark

Not just a technological risk. Are you adapting your metrics to the way of the rest of the managers thinks about risk?

Deneen DeFiore

Yes, absolutely. They don’t want to see how many vulnerabilities you have, the volume and age, and severity and how long it took you to close them, they want to understand how does that translate to disruption to systems that are critical to ride my operation. So that’s how you have to think about it, so that’s how we look at it, that business process is not vulnerable because we’re doing x, y, z in it from the cybersecurity side and we manage it that.

What aspects haven’t been considered?

David Spark

Ann Kramer over at Living Security said, “One way to know you’re doing things right is when people not only don’t click but they forward it to you,” so one sort of narrow focus way of describing that people are being security aware, security is on their mind. And Dave Cason of News Corps said, “When things get safer and less expensive at the same time.” I’m going to throw to you first, Geoff, does that ever happen because my feeling as one is building at a security program, and this is just my theory, I’ve never built one before, but the ideas that security programs should be incremental in the sense that over time you’re building it, you’re making it better and so it’s kind of all working together and you’re not fighting it on a day to basis, but the thing is getting incrementally better and hopefully, as a result, somewhat less expensive. Or is it just everything else is getting more difficult so you’re always chasing it? What’s happening?

Geoff Belknap

If you’re early stage up at a start up or a tech company or where ever you’re doing security, if the organization’s just small and just starting, the costs are going to be significant for you to start. If you have an established organization, if you’re managing the costs well, they should be mostly incremental and they’ll have to put an asterisk as regulations change or you find new contractual obligations you have to meet, sometimes they can be significant. Honestly, even if we talk about it as incremental costs, like some of these things are expensive. I do not run an inexpensive department and we do not generate revenue so there’s always that discussion. But the reality is if you’re doing this well, and you’re building security into the things that you do, it should become part of the overhead costs of operating the business, not some significant material additional cost that it’s a bolt out of the business. That’s not only you’re doing security well, but that’s you managing the business well, and your well informed decisions you’re making about the business in the risk context.

David Spark

Deneen, have you even ever heard of this: have you ever one year said, “I don’t need as much budget this year”? Have you ever heard anybody say that?

Deneen DeFiore

What I have done is probably turn down a chunk of money because I couldn’t spend that much. If I’m going to spend the money, then I’m going to have to really do something and show the outcomes. I’m not going to just have a bunch of money to spend on things, so I want to get value out of that spend. I wouldn’t say that I said, “Hey, I don’t need this money,” because I just didn’t have the people and the resources to be able to get everything done in that amount of time.

David Spark

That’s a good point, yes, you would like money, but if I can’t show the value of what I’m going to spend on this, what’s the point?

Deneen DeFiore

Right.

Geoff Belknap

No one’s going to forgive you for that later – if you take the money and then don’t use it, you’re not going to be looked on favorably. I can say for me, I had an instance recently where we budgeted a significant amount of money for a project and then were able to optimize how we were doing along the way, we needed about half as much money to operate that program year over year. And that’s pretty rare. But I think the most common thing is along the lines of what Deneen is saying. When you’re very lucky and you have a supportive, engaged board and executive team, they will say, “How would you like all these resources to do these things?” And you have to contextualize that in the sense of what can you accomplish. Right now, it would be fantastic if my executive team is like, “Geoff, here’s 200 more people.” That would be fantastic, does this come with a recruiting people to bring all these people on and train them and engage them because if not, we would be very hard pressed to make that usable within a fiscal year.

David Spark

I want to go to Ann Kramer’s comment, which is essentially summarizing a good sign that you’re doing well as a leader, is that everyone is becoming more security aware. And one of the things that Mike Johnson, my co-host at the CISO Security Vendor Relationship Podcast said, an indicator for him is when people come to him asking questions about their own personal cybersecurity. Are there things that you’re seeing about employees done, security people, that indicate that we’re getting to them, this sort of security awareness is actually working.

Deneen DeFiore

Just coming off the last days of wrapping the cybersecurity awareness month of October, our theme this year was ‘Be a reporter’ because what we wanted to do was make sure people had understood and had the resources that if they saw something they could report it and it wasn’t a consequence. It was like analogous to the safety culture in aviation. Pilots report near misses all the time and there’s no retribution and it actually helps when you look at that data, get the whole eco system to another level of safety. So that’s what we were trying to do at United Airlines and I think it’s resonating with people because it’s familiar to the way we operate and run the airline, but it also is something that they can see is making a difference and we’re rewarding them for them too. So we’re like giving them a little badge, a note to their manager and their leader and saying “Great job, keep doing this, we appreciate it.”

David Spark

Any similar, Geoff?

Geoff Belknap

Absolutely. If your culture as an organization is such that people are coming to you are people are thinking about security, you’re winning. But it takes more than just you, just like at United I imagine the culture of safety doesn’t come from the maintenance team. It’s not on the maintenance crew to say, “Let’s all fly safe everybody,” it’s the entire company doing that. So if you can help put your company on that path, your organization on that path, to be thinking about security in what you do, not in terms of being paranoid about clicking things, but really working at how they work, you’re doing a phenomenal job and you should feel great about how it’s going.

David Spark

Excellent. And that brings us to the very end of this show. This was a great discussion and I think we all now know how to measure security. It’s still complicated but we understand the nuance of it. So don’t measure Deneen’s performance and Geoff’s performance with the same yardstick, if you will, because they are very different environments. But I want to ask you, what was your favorite quote from this episode. Geoff, I’ll start with you.

Geoff Belknap

I’m going to go with Christophe Foulon from Capital One, “Security is part of the culture. The business is actively considering business and security risks when making decisions,” and I think no matter what your organization does or how you operate your security program, you can feel really good about how it’s going if you find this to be true.

David Spark

Deneen, your favorite quote and why.

Deneen DeFiore

I think its Brandon Scherer from Charles Schwab: “The business sees security as a trusted advisor who they willingly and deliberately bring to the table solve problems.” That’s ultimately the angle – everybody wants to drive the outcomes and add value to the organization that you’re in a really meaningful way.

David Spark

Everyone wants to be wanted, I think is what it really gets down to. Have the two of you worked in organizations because we’ve heard from plenty of security leaders where you were an after thought, have you been in organizations where that was the case? Geoff? Deneen?

Geoff Belknap

I’ll say there are definitely teams that I’ve worked with that would like us to be an after thought and I think that’s just where you realize you have to engage and spend more time in the relationship to help them understand what security really does.

David Spark

And that’s part of you building up your trust.

Deneen DeFiore

I only have worked at two companies, so it’s a little different. GE’s like 75 different companies in one company, but getting growth and revenue was way more important than stopping to think about cybersecurity risk and the added cost and time, especially in that start up time mentality. It’s a journey, we got through it and it ended up in a good place.

David Spark

That brings us to the very close of the show. I want to thank our sponsor, Human Layer Security, because we know many of the issues in security have to do with humans, hence why we have Cybersecurity Awareness Month. More at Tessian.com. I always ask for any closing thoughts you have on this and also if you’re hiring. Geoff, on every show says he’s hiring, so I’ll say it for him – is it LinkedIn.com/jobs?

Geoff Belknap

It is. We’ve got you well trained!

David Spark

I’ve learned something in LinkedIn Job Awareness Month. But if one would want a job somewhere else, they can also find that in LinkedIn as well. But why would you want to work anywhere else if you worked in cybersecurity than linked in, possibly also of United. I’m assuming you’re hiring, Deneen.

Deneen DeFiore

Definitely, we’re hiring. You will find our jobs at united.com/careers and there’s a lot out there.

David Spark

Geoff, any last thoughts?

Geoff Belknap

It is a hard job to be a CISO and it’s exceptionally lonely and I know we all find ourselves wondering are we even doing a great job? I think we’ve had a great conversation about what that means, but just keep in mind if you’re driving your organization forward, you’re adding value, you’re doing the right thing. Everything else will fall in place.

David Spark

Deneen, any final thoughts on the topic?

Deneen DeFiore

I agree. A CISO is an unforgiving job. Most of the days you’re just like, what? The problems are insurmountable. You turn around and you’re like, yes, I took two steps forward and now you look over here and it’s like 15 steps back. But in the end, some of the things we talked about, when you do get that recognition from the business or someone on the front line or in a different department comes to you and says, “Hey, that really resonated with me,” or “I saw this, what do you think about it as a risk?” Those are all rewarding moments and I think everybody has those and they start to uplift us and get us re-energized to keep doing what we’re doing.

Geoff Belknap

But if it doesn’t LinkedIn/jobs!

David Spark

Thank you very much, Deneen. Thank you very much, Geoff. Thank you to our audience for all your awesome contributions and for listening to Defense in Depth.

Voiceover

We’ve reached the end of Defense in Depth. Make sure to subscribe so you don’t miss yet another hot topic in cybersecurity. This show thrives on your contributions. Please write a review, leave a comment on LinkedIn or on our site: CISOSeries.com where you’ll also see plenty of ways to participate, including recording a question or a comment for the show. If you’re interested in sponsoring the podcast, contact David Spark directly at David@Cisoseries.com. Thank you for listening to Defense in Depth.

David Spark
David Spark is the founder of CISO Series where he produces and co-hosts many of the shows. Spark is a veteran tech journalist having appeared in dozens of media outlets for almost three decades.