How Can You Tell If Your Security Program Is Improving?

How Can You Tell If Your Security Program Is Improving?

What’s your best indicator that your security program is actually improving? And besides you and your team, is anyone impressed?

Check out this post for the discussion that are the basis of our conversation on this week’s episode co-hosted by me, David Spark (@dspark), the producer of CISO Series, and Steve Zalewski. Our guest is Simon Goldsmith (@cybergoldsmith), director of information security, OVO Energy.

Got feedback? Join the conversation on LinkedIn.

Huge thanks to our podcast sponsor, Votiro

Can you trust that your content and data is free of malware and ransomware? With Votiro you can. Votiro removes evasive and unknown malware from content in milliseconds, without impacting file fidelity or usability. It even works on password-protected and zipped files. Plus, it’s an API, so it integrates with everything – including Microsoft 365. Learn more at

Full transcript

[David Spark] What’s your best indicator that your security program is actually improving? And besides you and your team, is anyone impressed?

[Voiceover] You’re listening to Defense in Depth.

[David Spark] Welcome to Defense in Depth. My name is David Spark, I’m the producer of the CISO Series. And joining for this very episode is Steve Zalewski. You may know Steve because his voice sounds a lot like this.

[Steve Zalewski] Hello, audience.

[David Spark] That’s exactly how it sounds. It hasn’t changed since probably prior to puberty, would I say that’s probably a good time when it changed?

[Steve Zalewski] Yeah, that I would say is a true statement.

[David Spark] Okay, good. Our sponsor for today’s show is Votiro. They’ve been, by the way, a phenomenal sponsor, and they have a very interesting solution for malware and about extracting malware that gets added to files but doesn’t actually destroy the file or completely erase the file, so you can actually still operate. Anyway, more about that later in the show. But first our discussion for today. On Twitter, I asked the question, andI posed it in the tease of today’s episode,”What’s your best indicator your security program is actually improving?” And since security professionals suffer from the Rodney Dangerfield “I get no respect” syndrome, I asked, “Does anyone care that you’re actually improving your security posture?” So, what’s your take on sort of everyone’s sort of initial response to this, Steve?

[Steve Zalewski] So, I think this is a very serious question for Defense in Depth. Part of me, it says, “Look. If you don’t think this is the number one problem that you need to address for your security program, you got to rethink.” On the other hand, I love the Rodney Dangerfield because really, the point being, A, “If we cry wolf, who cares? And so let’s have some fun with it,” right? Which was, “Okay. Well then, who really should care, or what does caring look like?” That’s kind of the two sides of my thinking as we were going to go through this episode.

[David Spark] Well, I am very excited to bring our guest on for today’s episode because this person, I know solely because he’s been very involved with the CISO Series, and I’ve quoted him umpteen times on our various shows. He is always extremely eloquent in the written word, and I’m going to put the pressure on now. But I but the pressure on before we set up the recording, I said, “Will the eloquence of his written words translate to his spoken words?” I think so. So, I’m very excited to have him on. He’s the director of information security over at OVO Energy, it is Simon Goldsmith. Simon, thank you so much for joining us.

[Simon Goldsmith] Thanks for having me, David. It’s a pleasure to be here.

What would a successful engagement look like?


[David Spark] David Peach of Illuminate Risk said, “Your stakeholders start telling you what they did or didn’t do as a direct result of our advice, and they’re proud of it.” And Bill Diekmann of Cupertino Electric said, “When you are invited to the business strategy sessions because they want to ensure that security is being considered in the planning stages. When they offer more budget to your program without you asking.” And lastly, and note this was a Twitter discussion, so I’ll be mentioning Twitter handles here, Shak The Hack@shaktavist said, “You see an increase in the number of issues being reported by the staff.” So, we’ve talked about this before many times, Steve, that a good sign the security’s working is others are visibly showing they care about it, and what you’ve taught them they’re actually applying.

[Steve Zalewski] Yes. And from a stakeholder perspective, these are several examples of what success looks like. That what you’re seeing is an engagement in your business, that they’re excited about being part of the solution. The other half to this though is how does your own team feel? Because if your own team doesn’t have that same level of engagement and pride and feeling like they’re part of the solution with the business, then you have an equally difficult problem. And so what a successful engagement looked like for me is, yes, make sure that the business is showing that there’s value, and they’re proud, but as a leader of your organization, make sure that your team feels that it’s equally important. And if either of those is out of whack, then get on it.

[David Spark] Good point. So, Simon, have you experienced any of the things that were mentioned here by David, Bill, or Shak the Hack?

[Simon Goldsmith] Yeah, I have actually. And at first I thought, well, who offers you more budget without you asking for it? But then I actually did think back to a couple of boardroom discussions where we were asked if we had more money would we be able to go quicker. So, that has happened. And what I really liked about these three quotes, and particularly the first by David, is they covered the emotional impact that a security program is having on those outside of the security team. And given that we as information security discipline can focus on rational intelligence before emotional intelligence, that was encouraging to see that. But I do think it does, and to Steve’s point, I think it falls a little short of appreciating the scale, the reach, and the business objectives of a typical cybersecurity program.

[David Spark] I would also say that these quotes here also speak to the second part of my question of, “Is anyone impressed? Does anyone care?” because when they come to you, like Steve, “I did the thing that you told me to do and look how awesome it is,” that’s kind of a major lift for you. Like, “Oh, my God. I’m not screaming into a void,” right?

[Steve Zalewski] Well, there is that sense of self-satisfaction, meaning as a security practitioner, it’s exciting that you can teach somebody about a new topic. And sometimes we get a little hung up on that, that our job is to educate you on why security’s important. As opposed to realizing that a successful engagement could look very different. And as an example, I’ll offer when we implemented single sign-on at Levi’sthrough a single sign-on vendor. That turned out to be, over the course of four months, one of the most successful engagement programs that we ever had. And you know what made it so successful? The fact that the business sponsors went back to the CIO and said, “Of all the things that we’ve gotten out of IT in the last two years, this is the one that’s had the most value to us. It’s made our job easier.” It had nothing to do with security. And yet, the CIO is now being able to say, “Look. The security team actually was able to do something that made the whole IT organization look good.” How’s that for a successful business engagement? So, I offer that as another example of thinking out of the box to know what success looks like.

[Simon Goldsmith] 100% agree, Steve. I think it’s that. It doesn’t need to be security for it to be a change program, and I think that’s what we’re really talking about here. We’re not talking just about continuous improvement of existing processes. We’re trying to deliver organizational, technical, and behavioral change which results in business benefits. That’s a program to me, and that’s what we’re really talking about.

[Steve Zalewski] And that’s why successful engagement, right? Why I kind of took it to start with. Which is, well, there’s the strategic thinking of what does a successful engagement look like. Rodney Dangerfield I think is great because there’s a certain amount of hilarity, right? Jocularity that you have to have, camaraderie, to talk about all the things that go bad for people. Where you can make light of it so that you can orient around make sure your people are happy, make sure your business is happy,make sure your company is happy, all of those are successful engagements so don’t lose sight of the bigger picture.

What should we be measuring?


[David Spark] Benjamin Henriksen of ATP said, “Improved assessment results, improved results during red/purple team testing.” So, I should mention that all these responses were very literal in terms of metrics here as well. And Walter Williams of Monotype said, “Number of known vulnerabilities decrease, and speed of incident closure increases.” And George Al-Koura who’s a CISO over atRuby offered lots of advice, but repeatedly said between each piece of advice, “We are not breached.” So, I’m going to start with you, Simon. These are just kind of metrics to measure the improvement of a program or what they say, and we’ve had lots of debates on metrics. What do you say to these and if you were just looking at numbers, how do you measure a successful program?

[Simon Goldsmith] I think those are good answers, but they speak to control and technical effectiveness rather than the effectiveness in the outcomes of a program. So, my answer to this question is really it’s a combination of what we can measure easily and what’s actually relevant and important to measure. So, this is dangerous, but I like to use an analogy in this kind of situation, and the one I tend to use is the one of the boiling frog. And the fable goes that if you put a frog in boiling water, it will hop out straight away. But if you put a frog in water that is the same temperature as the body temperature of the frog and then heat the water, the frog will slowly boil to death without noticing.

So, basically in this situation, a security incident is boiling hot water. If an organization is aware of an attack, it’ll act quickly and it’ll act with urgency. But most of the time, a security program is dealing with a steady accumulation of vulnerabilities, loose access permissions, misconfigurations, missing controls, and all of that stuff which we’re familiar with, all of which are increasing the temperature of the water. And the problem in security is right now we’re not very good at measuring the water temperature, and we’re not very good at agreeing with the business on what temperature is kind of the range where with a little bit of attack of heat, we’re going to get frog death.

So, my conceptual answer is we should be measuring the agreements we reach with system owners to operate controls that can help them to measure and stay within an acceptable range of water temperature, get feedback on the quality of the help thatthe security team and the security program is giving them in implementing those controls, and then measure our ability to detect, respond to, learn from, and anticipate threats which could take us out of that range. So, it’s a bit of a conceptual answer, but that speaks to more outcomes and more kind of business benefits.

[David Spark] I like it. Steve, I love the boiling frog analogy – finding that temperature that we are well-suited for, if you will.

[Steve Zalewski] Great answer, and I’m going to pick a different analogy because I think also to the point here is that different analogies are the different ways of the problem. The first thing I’m going to say is what you should be measuring, and I’ll remind everybody – figures don’t lie but liars figure. Okay? And you have to bear that in mind as a good thing. Because the second part I’m going to say is my analogy is, hey, look, as a security guy, every line of business, every customer that I have, is a member of my family, they’re my kids. Now, at any point in time if you have children, has a child come up to you to say, “Am I your favorite child?” and of course, “You are my favorite child,” and then they said, “But you say that to everybody else.” And the answer is, “Yes, you’re all my favorite in different ways.” And that speaks to how I measure that success.

Measurement is not metrics. Measurement is a way of encouraging good behavior. A metric is what is my bonus based on, okay, so I want to maximize bonus as opposed to I want to encourage you to go to bed at nine o’clock every night, and so let me give you candy for every five minutes, plus or minus, right? So, therefore, I’m going to measure your promptness. That seems a little arcane maybe to people, but it’s really important that I say measure where everybody is your favorite child at that point in time and know it’s appropriate.

But when you think about the metrics and the true business value, that’s where you have to separate. So, that’s why I like to say figures don’t lie but liars figure. And that what we’re really doing is we’re using that measurement and those relationships to encourage the good behavior. Because how do you measure success? How do you feel good, as Rodney Dangerfield says? And the answer is you have to understand you’re going to get a different answer for each of the audiences, and you want to know how to have those conversations so that in the end, all of your kids in the end are happy with how you raised them.

Sponsor – Votiro


[Steve Prentice] So much of business relies on the sharing of documents but as we all know, much of the cybercrime world relies on this as a carrier for its malware. Votiro is a company that allows information exchange by delivering only the good safe stuff. Aviv Grafi, founder of Votiro, explains how this works.

[Aviv Grafi] So, if we thinkwhat is the good stuff, what is the relevant stuff for productivity within a content, let’s say a Word document, we know that the end user’s interested in the text, in the paragraphs, in the embedded images. We’re taking all the productivity-relevant content and deliver that over a safe template of that Word document. What we do is we keep the active content; we know how to do that for macros and for other interesting stuff. In that way, we’re allowing yourfinancial organizations like insurance companies and banks to use their macro-based manipulations without the need toworry.

[Steve Prentice] Votiro started out by delivering this service for email but now offers a fully hosted software solution as an API.

[Aviv Grafi] So, on top of our state-of-the-art technology, what we did is we’re now offering that as a fully hosted software service solution. So, in fact, using Votiro’s API, every application can enhance its security and productivity but just within minutes by plugging Votiro’s technology that is hosted into the traffic and business flow.

[Steve Prentice] For more information about Votiro and how it can work for you, check out

Does it play nicely with others?


[David Spark] Jonathan Dupre of Cloud Security Labs said, “Managers make accurate predictions about security risks, compliance projects get leveraged to win more business, IT cuts cost by orders of magnitude, and people stop making dumpster fire jokes.” And Meg Brejwo of Atlantic Federal Credit Union said, “Senior leadership asking me what I need from them to support the program, and employees proactively asking me how they can be more secure. I don’t just feel like an annoying nag anymore.” That’s sort ofa call to our first segment here. And lastly, AccidentalCISO said, “I think the best indicator is that you are having less and less trouble getting buy-in across the organization.” So, I think that very last comment, I’m going to throw this to you, Steve, is there’s a partnership not a butting of heads, and if you’ve got a butting heads, that’s a miserable existence. And have you had a butting heads environment?

[Steve Zalewski] Unfortunately, yes. And for most of us, I’m going to say it’s inevitable. You can plan for success, but you got to accommodate the failure, and here’s where it comes into it. If you are either told or believe your job is to make changes to the business processes to make sure that you’re secure, the business will potentially rebel because they’re ultimately making money or doing what’s important to the company and the control friction that you’re implementing, they will not accept. In which case, what do they do? Well, if it’s too onerous, they don’t do it, and they’ll find any number of reasons, and now you’re in trouble because you’ve lost that allegiance with the business to know where the tradeoff is. And so what you got to do is be sensitive to where that control friction has now kind of crossed the line and you’ve got to back off. And you got to now go back to the table and say, “It’s not working and let’s talk about why it’s not working because it isn’t that you’re not doing what I’m telling you. And maybe we have unreasonable expectations in how we have to play with each other and let’s have that conversation.”

And so that’s why buy-in across the organization is a continuous exercise. And on any given day, they may like you or they may not like you. The key is do they feel confident and comfortable with talking to you. So, at the end of the day if a mistake is made, and they feel like, “You know something, I made a mistake, but I can go talk to Steve. Because even though I made the mistake, I trust him that he’ll help me do the right thing to get out of it.” That ultimately is where you want to be.

[Simon Goldsmith] Yeah. And I think the most common situation that I’ve found, certainly more common than stakeholders just saying a flat-out no or not having the buy-in is it’s they understand the why and generally the security team have thought through quite carefully about the why, but we struggle with the why now. And so buy-in tends to be problematic on the why now question, particularly over the last couple of years with the likes of COVID where most companies have been living a business continuity event. Why now sort out security, or why now make this change for security which might be some event that happens in the future can be a bit of a challenge.

But I like Jonathan’s answer here in that I think quite a fair bit about stakeholder groups. And to Steve’s point earlier, I like to think about what a good security program looks like to customers, investors, regulators, engineering teams, and even our own security team. And the challenge that we’ve often got is that regulators and even investor requirements tend to be quite well articulated in audits or in assessments. But the success metrics for customers and engineering teams are a lot more subjective. But I think Jonathan makes a good point about B2B companies, where they can start to differentiate on security and internal security teams being positioned as a competitive advantage, and that’s a good hook to take advantage of.

[Steve Zalewski] There’s one more point here too which was we got to realize that the leadership team are spoiled children in many cases if I’m going to use my analogy. And they can be spoiled in one of two ways. They can acknowledge that they’re spoiled, and people can appreciate that and not mind it. Okay? “Hey, look. This executive team is in such-and-such a situation that they can’t do multifactor authentication.” “This executive team is breaking laws that we’re holding everybody else in the company is accountable for, but they have special dispensation, and this is why.” That we’re acknowledging it’s important for the business as a whole. As opposed to the ones that just want to do it because they want to. And therefore, the rest of the organization becomes demotivated and what they’re actually doing is harm to the organization because they want to be treated like special children but they’re not stepping up to that responsibility.

And so does it play nicely with others? I want to remind everybody is unfair as it is, there really are two classes. Make sure you understand the two classes and make sure that you get both to play because that’s going to go a long way for you so that you’re not always having to explain how come the CEO doesn’t have to do what I have to do. Right? And if the CEO’s the one willing to say, “Look. I understand. Let’s talk about the relative security to the business and that the security teams are working with us.” People may grumble but they appreciate the fact that there’s transparency there.

[Simon Goldsmith] If I could just jump on that very quickly around the senior leadership because I think Meg’s quote was great here. I think it really successfully summed up what good looks like from a people perspective, so people have understood the change with both their minds and their hearts, and security’s now something they both understand and they’re passionate about. She really gets the kind of the emotional impact that a security program needs to have.

Why is this happening?


[David Spark] @Jedediah6 said, “When people come to me with questions, when they actively seek out security to ask for advice, that means that, A, we’re approachable; B, we’ve showed the importance of security to the organization; C, they understand that it’s not something to be scared of and often makes life better.” And @IMTheNachoMan said, “Facts and good security don’t matter. If they are happy, the program has achieved its goals. Security teams/functions should operate behind the curtains, they should not be seen or heard, but their presence should be everywhere.” So, I kind of see where IMTheNachoMan is going with that, but facts and good security do matter, but the question is how much do others need to see. I mean, it’s an interesting take here. People need to be security-minded, but they don’t need to see how the sausage is made. Yes, Steve?

[Steve Zalewski] So, there’s two things that I kind of picked up here which is if you look at Jedediah on B where he says, “We’ve shown the importance of security to the org,” I would say the responsibility is not to the org. It’s almost to the individual. That what you’re really saying is, “We’re approachable. So, anybody in the company knows who are and will approach us to anything to do with security.” My credit card got breached,I’ve got a personal compromise, what should I do around my firewalls, what should I do for route… It doesn’t make any difference. It’s that each of them is coming to you understanding how the value to the company is being internalized by each person. Because at the end of the day, the person is the weakest link. And if the people are engaged, you’re doing everything you can to make it as hard as possible for the bad guys.

And then the second part about being behind the curtain. Okay? Here’s what’s interesting about that. Yes. Most people do not want to overtly have to change how they do business in order to do business. And security oftentimes is a change agent, not necessarily in a good way. But if you have good stories like we’re talking about where somebody says, “Hey, I talked to the security guys because I got popped on my personal account,” or “There was a phishing email, and they reached out to me, and they told me what was going on, and they sent me gummy bears as a thank you for me just sending the note.” Okay? That’s behind the curtain where they tell those stories to everybody else and that’s the positive impact. Which was people simply say security is really if I talk with them, a positive experience, and stories that come through the anecdotes are what build that. Here’s what a good security team is. It’s behind the curtains, you see it once in a while, it’s there to support you, but it’s not their job to tell you what to do always. It’s your responsibility as well to go to them if you have questions. And these are all the kinds of questions that anecdotes come out as ways that you can approach anybody on the security team. And that is actually how we approached it at Levi’s.

[David Spark] Simon?

[Simon Goldsmith] I mean, the only word that I can think of when I read the second quote was “wow.” I mean, I like the sentiment that security’s so omnipresent in an organization and all its people are so supremely competent that the security team can just kind of sit in their bat cave and only respond to kind of the worst kind of incidents. It’s kind of like a Zen mastery of security wherea company is practicing security with kind of unconscious competence. I’ve never seen it. I like the sentiment, but I just don’t think it exists.

And there’s a great quote from David Marquet in Turn the Ship Around when he talks about if you want people to think, give them intent, not instruction. And he talks about competence and clarity being the two key components to give control and decision making to teams so that they can make more secure, or in his case, safer decisions. And he talks about there being quick wins but developing that kind of competence and clarity, taking time. The great thing about kind of thinking about it in terms of competence in the teams that Steve is talking to and the clarity from the security team of what good looks like is those are also measurable outcomes for a security program.

[Steve Zalewski] Spot on.

[David Spark] You nailed it.



[David Spark] And that brings us to the end of the show and where I ask both of you what was your favorite quote and why. And I will start with you, Simon, as our guest. Please tell me – which quote was your favorite and why?

[Simon Goldsmith] I think a security program should ultimately be a change leadership program and we talked about appealing to both the kind of the emotional and the rational sides of our colleagues. And I think Meg expressed what that felt like, both for the security team and the wider business brilliantly, so I’m going to go with her quote.

[David Spark] Excellent. And Steve, your favorite quote?

[Steve Zalewski] I’m going to go with George Al-Koura, the CISO of Ruby, because measurement and metric, right? Measurement makes you feel good, metric makes you profit, but it’s never bad to remind everybody we haven’t been breached as the ultimate reason why we’re doing this, to keep everybody aligned to the mission.

[David Spark] Excellent point. All right. Well, this brings us to the very end of the show, and I want a huge thanks to our sponsor Votiro. Thank you very much, Votiro, for sponsoring us, and being such a phenomenal sponsor with the CISO Series. And by the way, I’m going to give myself a pat on the back, I do this every now and then, where I predicted Simon’s eloquence in the written word would transfer to the verbal word. So, again, Simon, yes, you were the one who did it, but I got to give myself credit for predicting that as well.

[Steve Zalewski] Was that a measurement of success or a metric of success for you?

[David Spark] I’m just going to call it a blanket level of success.

[Simon Goldsmith] Kind of humble, David, kind of humble.

[David Spark] Yes. I am all of that. I’ll let you have the very last word, Simon, so hold tight, and the question I always ask all our guests is are you hiring, so make sure you have an answer for that too. But first, Steve?

[Steve Zalewski] So, in summary, like we said when I opened was this is a very serious topic for Defense in Depth, because measurement and knowing success and doing that well is a multifaceted exercise. But I think hopefully like we’ve seen, have some fun with it, right? The conversations around what success looks like, this is an ongoing exercise, and your ability to be able to laugh and cry at the same time on such a difficult topic is really important to bring to the conversation with your company and your stakeholders.

[David Spark] Very good point. And Simon, are you hiring and any last thoughts?

[Simon Goldsmith] Yes. Yes, we are hiring, and I should have some roles available particularly around junior security engineers and mid-level security engineers. So, those are available. It’s a great organization, delivering zero carbon living to people is just about the most motivating mission I’ve ever been a part of. So, it’s a great organization and great people, so please do look us up.

[David Spark] Simon, by the way, got his job prior to this at Adidas because of this very show. Simon, explain.

[Simon Goldsmith] Yeah, so I was listening to one of the CISO Series shows, and Brandon Newman [Phonetic 00:28:42] was on, I think talking about SAP security and liked what I heard and reached out and got in contact with him. And he gave me a gig out in Asia-Pacific as a kind of a regional BSO. So, I have the show to thank for a very enjoyable and informative part of my career and if Brandon’s hiring, go work for him, he’s brilliant.

[David Spark] But he’s not on the show right now, so you want to work for Simon. Now, your other advice, Simon?

[Simon Goldsmith] If I was going to make an ask, I would like to ask that everybody listening does connect with at least two people who don’t look like or think like you. We’ve got a huge diversity problem in security. It’s a big echo chamber, and it’s holding us back from becoming a true profession. So, being competent and paid is only a little small part of professions like engineering and medicine and accountancy. So, get involved, and it benefits it all when we act with integrity in a free public discussion on the diversity of the profession. It’s for all of us, so please do reach out and connect with others.

[David Spark] Seek out others who do not agree with you.

[Simon Goldsmith] Correct.

[David Spark] And challenge yourself. I love that, it’s a great piece of final advice. I’m going to actually think about that actively this week. Thank you very much, Simon Goldsmith, who is the director of information security for OVO Energy. And thank you to my co-host, Steve Zalewski. And thank you as well to our phenomenal audience who I always adore and always appreciate. It is not me just glad-handing. I do really appreciate you, especially your contributions, and for listening to Defense in Depth.

[Voiceover] That wraps up another episode. If you haven’t subscribed to the podcast, please do. We have lots more shows on our website, Please join us on Fridays for our live shows – Super Cyber Friday, our Virtual Meetup, and Cybersecurity Headlines Week in Review. We’re always looking for fascinating discussions for Defense in Depth. If you’ve seen one or started one yourself, send us the link. We’d love to see it. And when any of our hosts posts a discussion on LinkedIn, participate. Your comment could be heard in a future episode. If you’re interested in sponsoring the podcast, contact David Spark directly at Thanks for listening to Defense in Depth.