Security awareness is a key part of any security program. So why are we so skeptical of security awareness programs?
Check out this post for the discussion that is the basis of our conversation on this week’s episode co-hosted by me, David Spark (@dspark), the producer of CISO Series, and Dan Walsh, CISO, Datavant. Joining us is Sharon Milz, CISO, Time.
Got feedback? Join the conversation on LinkedIn.
Huge thanks to our sponsor, Intezer
Full Transcript
Intro
0:00.000
[David Spark] Security awareness is a key part of any security program. So, why are we so skeptical of security awareness programs?
[Voiceover] You’re listening to Defense in Depth.
[David Spark] Welcome to Defense in Depth. My name is David Spark, I’m the producer of the CISO Series. And joining me as my guest co-host for this very episode, it is the CISO for Datavant, none other than Dan Walsh. Dan, say hello to our audience.
[Dan Walsh] Hello, it’s great to be here, and I’m excited to talk about this topic with you.
[David Spark] This is a very high topic we’ve been talking about on all of our programs, and we’re going to get to that in a second. But first, I do want to mention our sponsor, and that’s Intezer. Extend your security team with AI. We’ll talk about exactly how Intezer is doing that a little bit later in the show.
I’m going to begin with this quote. “I don’t think security awareness training programs are that effective.” This was the argument from Jacob Friedman of Three Tree Tech. And there are so many security awareness programs out there, and there are so many debates as to how to do training and how to measure successful training.
But given those variables are so subjective, and opinions on the effectiveness in security awareness programs can vary wildly, Dan, is a security awareness program that can demonstrate a reduction in click-through rates actually delivering an improved security program?
[Dan Walsh] Well, I would say yes and no. So, yes, in terms of that’s the unlocked front door to the house that’s about to be robbed. But there are other controls that once inside are usually what result in a failure, right? And so I think the bottom line is it’s important. It does reduce it. But the reason why these breaches are happening isn’t because a small fraction of people aren’t aware.
It’s because of other underlying problems in the program.
[David Spark] We’re going to get into that. We’re going to talk about security awareness programs, how they try to pitch themselves, the effectiveness, and how essentially CISOs respond to what they believe is effective in their environment. And somebody who’s actually going through this issue right now is our guest.
She is the CISO over at TIME, none other than Sharon Milz. Sharon, thank you so much for joining us.
[Sharon Milz] Thank you for having me. Really glad to be here.
Where does this effort fall flat?
2:19.339
[David Spark] Sam Oberholtzer of ComplySAM said, “Whenever you buy tech, it won’t just magically solve all your problems. I would not just state this is specific to security awareness training because any training is better than none. Ultimately, you need the right people to facilitate the training and enforce or else it is useless.” So, David Volkov has a very negative view of security awareness training in general, and also saying that it’s a race to the bottom.
And Sam says, “Well, you need training for everything.” Dan, what do you think?
[Dan Walsh] Well, I think we’re confusing two different topics inside of security awareness. I think the first topic that we are talking about is compliance, check the box. If you’re a regulated company or even if you’re not, different frameworks call that there’s an annual security training, right?
Or when people on board, you have to take training. So, that’s compliance security awareness. I think what the first quote was referring to was culture. Like how do you change the culture? It’s not going to be watch this terrible PowerPoint slide deck or this terrible video. And if that’s the case, yeah, then you should absolutely buy the cheapest solution out there because it is going to be a race to the bottom if you’re just trying to check a box.
So, I think what gets lost in this argument, as I mentioned, is it’s are we talking about changing culture or are we talking about compliance and checking a box?
[David Spark] That is a very good point. That is a good dividing line right there. Sharon, this is an effort that you’re working through right now. I’m assuming you have both problems you have to worry about. You do have probably a compliance or regulation issue, and you also want to change culture, yes?
[Sharon Milz] Yes, so not as much in compliance. We’re not a public traded company or under any regulations. But one of the challenges Dan set up perfectly was it’s not about checking the box for me. It’s about how do I change the behavior and the culture of the organization, right? And how will I ensure that my users think of them as part of the program, right?
Or part of the security program on how to keep the organization safe. I agree in some degree with David on like most of the security awareness training tools out there have become repetitive and stall. So, one of the things that we have incorporated, or I’ve started to incorporate, is more of a personal approach where my team members are outside of the program.
We’re setting up one-on-ones with departments, right, to really target what are the challenges and concerns that that department should be facing, right? It’s not the same from my perspective to look at editorial or look at finance and thinking that they have the same problems, right? So, that goes back to, I think, to Sam’s quote around it’s not just buying a tool and put it out there.
It’s when you buy that tool, how do you implement it and how you leverage across the organization.
[David Spark] And like we’ve said about a lot of solutions, it’s people, process, and technology. A vendor would like you to believe, or this is how a lot of security professionals feel, they would like you to believe that if you install this piece of software, it’ll magically solve your problems. Yes, Dan, you’ve had that pitch before?
[Dan Walsh] A hundred percent. I mean, you think about it from a human risk management point of view, and those solutions are very often tied to our called [Phonetic 00:05:25] security awareness platforms. They’re useful. If you can get the data, it’s very useful. But again, that data is not going to change unless you address the root cause, which is culture.
As an example, in a previous company, the CEO was traveling to a country that was on our banned list of places you could access company resources from. And he wanted an exemption for himself. He wanted an exemption for all the people that were traveling with him versus I’ve been in other companies where the CEO is like, “Oh, that’s the rule?
Then I don’t want any special exception for that. I’m just like everyone else because that is how important culturally security is to the CEO.” And so I do think it always goes back to a cultural issue. And in fact, if you look at like any of the major breach reports, like Verizon, IBM, and Google, they all say the same thing.
It all comes down to culture.
Would this work?
6:11.050
[David Spark] Val Dobrushkin of Akamai Technology said, “Some training is still beneficial, but in a very limited and targeted way, where it can help people on their regular lives as well. For example, phishing prevention and password management practices.” In fact, my cohost Mike Johnson has brought this up many times.
When the employees ask you about their personal cybersecurity, then you know they’re actually paying attention, and they’re interested. Kevin Walker of Black Swan Cyber Security Solutions said, “This is where defense in depth helps. Email filtering, DNS filtering, and browser extensions all help protect end users.
Security awareness training still has a place in my opinion, but it’s part of a bigger picture and not a silver bullet.” Now, it is though one of many controls, right, Sharon? Like you can’t just divorce sort of the security culture away and just hope all your controls are going to work. The humans are a control.
Do you see it equivalent to your other sort of controls you have in place or do you see it in a different way?
[Sharon Milz] Yeah, I feel like security awareness training is part of my overall security program. It is not mutually exclusive, right? It comes to layer in defense, right? You got to have your tools and platforms to prevent the phishing attack to come in place to your users or your inbox, but you still need to have your users be aware of what they should and should not be doing, right?
If you think about – I think I’ve heard this quote somewhere else – but think about an athlete, right? They’re constantly training and doing repetitive drills so when they go in the game, they know how to react, right? It’s the same thing that we’re doing with users, right? We’re training them on what to do and not to do.
So, when that happens, they know how to react. And now you also have platforms, and when their training doesn’t come in place, you got platforms that can prevent it and resolve it. So, it’s an overall part of the security program. You can’t do one without the other.
[David Spark] Let me ask you this, and this goes back to what we said at the very beginning about like phishing tests. We have some CISOs out there that just flat out will not do them because they think that they create a bad sort of culture. But I just had a guest on the show that said, “Oh, yeah. We will do the nastiest, the meanest,” now I’m paraphrasing, but those phishing tests that get people really upset, saying like, “Oh, you will get a raise if you click this button,” or something, or “There’s a big virus breaking out in the company.” And their attitude is this is what the criminals will do.
But the fear the other CISOs have is, but it creates a really bad culture kind of like the security team is sort of beating up on the employees. Where do you stand on this, Dan?
[Dan Walsh] I think you have to read the tea leaves in terms of the broader culture at the company. How are people punished, or what’s the consequence when there’s a failure? Like another failure, outside of clicking on an email or something like that? Is it very punitive and heavy-handed to begin with?
If it’s not, if it’s more of like, “Hey! You know what? We’re going to phish you as hard as we can. Around the holidays, you’re going to get those FedEx emails, and you just purchased something on Amazon, and so you’re probably thinking this is the real deal, and we’re going to phish you as hard as we can because we want you to be better.” And really focus it on we’re trying to make you a better, more savvy employee, and we’re trying to keep the company safe.
I think it just depends on how all that is managed at the company from a cultural point of view.
[David Spark] Also, have either of you had this situation where somebody falls for a phishing test, and they feel awful about it, and they feel they have screwed up? And also one of the punitive damages of failing a phishing test is additional training, which no one really enjoys. So, how do you manage that to keep sort of, I guess, the lighter mood or keep the positive attitude, the security culture?
Either one of you jump in.
[Sharon Milz] I mean, I think I’ve seen both where people celebrate that they were able to capture it, right? And then actually, sometimes it completely mess up our test because they end up sharing with all their colleagues, “Hey, there’s a phish going on now,” and they don’t know if it’s actually an email coming from us or it’s actually a real phish attack, right?
But it also proved my point where it’s actually working because now they’re sharing, “There’s a phish campaign going in our organization.” I don’t necessarily do the punitive part of it. Like if you fail, you have to do it again. You fail the process of the surprise attack, right? Of like, they’re going to be expecting another phishing campaign coming to them.
I think it’s more around, yes, you fail. They understand they fail, and they pay more attention going forward.
I have a perfect example. We weren’t really doing phishing tests before I started, and I did the first that we were failing through the roof. I mean, it was crazy how high our rate was, right? And when we do the test now, I think it’s less than 15% of people. And I don’t know if you guys recall the recent flaw that happened with one of the major email security companies, right, and we saw a huge increase on attacks, right?
From valid domains, which they were extremely hard for even our security team to confirm that they’re actually accurate or not. And I was extremely proud of how many people in our organization just sent the emails to the phishing alert and say, “Hey, can you confirm? I’m not sure this is actually real.
I don’t remember like talking to this vendor before or anything like that,” right? So, it shows that even though most people think they’re boring and where people think that they might not do a change in the behavior of the users, they end up doing it, right? Users are starting to become more careful about it.
It all depends on how much training you actually emphasize.
Sponsor – Intezer
11:41.168
[David Spark] Before I go any further, let me tell you about our sponsor and that would be Intezer. And you’re going to want to listen to this because it has to do with your SOC and alert fatigue. So, alert triage and investigations are time consuming for security teams. I don’t need to be telling you this, but I’m just setting you up because it doesn’t have to be that way.
Smart security teams are using AI to automatically investigate alerts from the security tools 24/7 with an average triage time of just two minutes. So, how does this actually work? Well, Intezer is an innovative platform that integrates with your security tools to monitor alerts, collect evidence, and investigate every artifact.
When Intezer uncovers evidence of a serious threat, it escalates the findings to the SOC analyst. Intezer also reduces noise, correlates alerts, and automatically resolves over 90% of false positives.
This means even low severity and informational alerts get investigated. You aren’t wasting time on false alarms, and you have actionable recommendations for serious threats. Intezer extends your team by emulating an experienced SOC analyst with an AI framework backed by years of industry research. It’s designed to be a cost-effective, easy-to-set-up platform that provides detailed transparent results that SOC analysts can trust.
No playbooks, no chatbots, no engineering to set up. With Intezer supporting your SOC analysts, your team can eliminate alert fatigue, uncover hidden threats, and stay focused on what matters most. AI won’t replace your SOC team, but it can be a game changer. For more, go to Intezer’s site, it’s Intezer.com.
No one said it was going to be easy.
13:30.883
[David Spark] Tim Golden of Compliance Scorecard said, “Security awareness training companies need to hire staff that have a deep understanding of how learners learn. Maybe focus on hiring one or two behavioral analysts that can understand the psychology of how people actually learn and pivot the education message to meet the learner where they are at.” And Bill Schneller of Geffen Mesher said, “I’m wary of placing too much value in them as a preventative control since I think anyone, even security professionals, can be phished if the context of the phish is good enough.” We have heard this many times.
“I’m fine with user training provided companies are covering other basic controls as well and not using user training as a basis of their security program.” Meaning no defense in depth, it sounds like. Bill continues to say, “To paraphrase someone smarter than me, ‘If your security program consists of training users not to click on bad links, you’ve already lost.” So, yes, we’ve heard that one too.
So, let’s get back to the original theory though here, Dan, of the effectiveness of these programs. I think this may also fall into this thing that we hear all the time in security – don’t let perfection be the enemy of good or very good. Are we doing the same thing here?
[Dan Walsh] We are. And the other thing that I mentioned too, David, is like if you think about an EDR solution, you set it and forget it, right? You got the agent running on your laptop, it catches the bad things, life goes on, you respond to them. Security awareness solutions, however technical the software might be, is not set it and forget it.
Okay, every year, “Oh, your one-year anniversary, congratulations, now go on and take your security training.” You really have to contextualize it and choose your timing to make sure that you get the most out of it.
So, as an example, when the MGM breach occurred, where it was a social engineering call to the IT help desk and basically got into their network and it was a really bad situation for them. I sent an email out to the company that said, “Hey, here’s what happened. It was a 10-minute phone call. It resulted in $110 million loss.
Here’s the things you should be careful about.” We’ve had specific company attacks at different companies that I’ve been at – send out a Slack message, send out an email, ping people on specific functions, “Hey, around the holidays, again, like I mentioned before, the phishing is going to go up.” We see phishing go up, like the phishing super bowl, at least for all the companies I’ve been at, is like Thanksgiving to New Year’s.
That’s when all the phishing comes in, right? And so just I think trying to take those tools that are valuable and then contextualizing it in a real-world scenario for the people. So that way they can like link it to something, I think is really key. And so to answer your question, yes, it’s very hard.
[Laughter] It’s definitely more art than science.
[David Spark] Do you think, Sharon, though, you’re winning with your other controls better than you are with security culture? Because it just seems like that’s always going to be the case, I mean, because humans don’t work like computers.
[Sharon Milz] I think it’s a combination of both. I don’t think it can just depend on one.
[David Spark] No, I’m not saying just depend, but the thing is the dealing with a computer is always going to be probably an easier situation than dealing with a human, right?
[Sharon Milz] Agree. And I think that what Dan explained is it’s not easy to change the behavior of users, right, and to make them think how you want them to think first, right? And things are changing quickly, right? If you think about now, we have seen an increase in voice phish attacks, right? That perfect example of MGM, and I’m seeing [Inaudible 00:17:00] editorial case, right?
So, we just have to continue to adapt, and that’s not something that I think any platform is going to be able to do. No matter how much tools or platforms you’re going to implement in your security program, I think you need to have that human piece of it in the loop. And it is the hardest part of our security program, we can’t control it, right?
We can just simply tune it and say, “All right, they know what to do going forward.”
[Dan Walsh] The one thing I would add is this. If you think about that Titanic sinking, right? Why’d it sink so fast? And it wasn’t because it hit an iceberg. It was because the ship was designed poorly, and all the chambers, the way they were designed allowed it to flood quickly. So, in the same analogy, hitting that iceberg is just clicking on the email.
Once the attacker gets in through the email, if your network or your environment is wide open, just like the Titanic, you’re going to sink quick. It is just an entry point, and you have to have very strong controls throughout the rest of your program to be successful.
[David Spark] And this is why we talk a lot about segmentation and blast radius essentially.
[Dan Walsh] That’s right.
[Sharon Milz] Right.
[David Spark] Just don’t let the damage go too far.
What are the elements that make a great solution?
18:02.816
[David Spark] Carl J. of Phishbusters Audit and Consulting said, “Phishing awareness training needs these three things. One, simplified definition and approach to training that covers all phishing, regardless of the channel.” We talked about that, he refers to text, email, voice, social media. “Two, a specialized focus in order to keep up with current trends and sophistication.
And three, objective metrics that bring value by providing behavioral change.” All right, Dan, I want to lean on that very last one there. The one that is most used is click rates for phishing emails. But what we have heard again and again, all depends on sort of the difficulty level of that phish. So, my question to you is how do you show, demonstrate change to yourself, to the team, to sort of applaud or say, “Hey, we need to work harder,” and to the board?
Like, what do you do to show change?
[Dan Walsh] It depends on the maturity of the organization that I’m currently in. I’ve just started at Datavant, so I’m still evaluating this. But in previous organizations, if I understand the identities of the people clicking, someone clicking in a very specialized role in finance that has all the company bank accounts and top financial secrets, that’s a much worse click than maybe someone who’s in the mailroom, as an example.
And that’s a terrible analogy, but you get my point, right? The more access you have, the more damage that you can presumably do the company.
[David Spark] And I would assume you give that person a lot more training.
[Dan Walsh] That’s right.
[David Spark] And you want them to be more ingrained with the security culture than somebody else.
[Dan Walsh] That’s right. So, showing that, I think, is very, very eye-opening to executives. I showed one report to a VP. His response was like, “You can shut off the internet to my employees’ computers.” Because he was so scared about it because of the function that he was overseeing. Okay? Now, that’s obvious you wouldn’t do that.
I think those metrics are good. I think the other thing too, is a board is going to want to see that you’re doing the basics. And to a board member’s mind, the basics is just phishing. Your average board member, your average business executive knows what phishing is, right? They don’t know what defense in depth is.
They don’t know what application security is. They have no idea what that stuff is. They do know what phishing is. And so to a degree, that helps them understand that, yes, we are doing the basics, and we would consider this the basics.
[David Spark] Do they understand that phishing comes through multiple channels, not just email?
[Dan Walsh] I would say three to five years ago, no. I would say today, yes. They’re getting the same smishing, and I’m getting a lot of questions now about what about deep fakes and things like that. So, I think with like the rise of AI and these alternative social engineering tactics, they’re definitely becoming aware of that.
[David Spark] We are hearing a lot about deep fakes specifically through WhatsApp right now.
[Dan Walsh] Right.
[David Spark] Voice first, by the way. All right, Sharon, what do you do to measure? Because I know this, again, you had mentioned this is something you’re sort of struggling with right now, and you don’t know whether these security programs are working or not. So, what do you do to sort of make you feel like you’re spending the right money and doing the right training?
[Sharon Milz] So, two things. Dan mentioned one, it’s kind of looking at the reports. But one of the things I’ve noticed is there’ve been an increase in response from our teams, including all the way to our executive level, right, where no concerns before. Now it’s getting a text in the middle of afternoon saying, “Hey, I got this email.
Can you check it? I’m not sure it’s safe.” Right? That wasn’t happening before, right? So, that means the culture and the behavior of the organization is changing to be more cautious than just clicking on a thing or looking at a text and responding.
[David Spark] Can I confirm that that is the number one response you really want? “Please tell me. This looks phishy.” Please tell me if that’s the case.
[Sharon Milz] Exactly. That’s what we want our users to be, “Hey, I’m not sure. Check it for me.” I’ll take a look at it. Or ask around. Ask your co-worker, “Hey, I got an email about paying this vendor. Is this right?” Or “I got an email to share this report with somebody. This seems too good to you, right?” That’s what I’m seeing lately from a culture perspective within the organization, and I think that proves the value of the things that we’re doing to change that behavior.
[David Spark] It may not be a one-to-one, meaning like that you’re seeing clicking rates go down because it’s hard to tell depending on how difficult or easy the phishes are. But if you’re seeing that kind of behavior and unfortunately, that seems more anecdotal than truly measurable, right, at that point?
[Sharon Milz] Correct. But I think it also, because we’re seeing it across the board where people are reporting the actual email as phish, right? So, we actually have set up a phishing report in our email platform, right, where people are reporting that we are seeing those things increase. Right? Where people like, “Hey, I got this.
I don’t know what to do. What should I do?” We are seeing even the executive level team is getting back to my team, to my management team, and say, “Hey, I got this email. I got this text. I don’t want to say anything until you guys take a look at it. Please take a look.”
[David Spark] By the way, of the percentage they say, “This seems phishy,” what percentage is it phishy?
[Sharon Milz] It’s actually pretty high. I would say if I have to put a number, I’ll say 70% is phish, either be it email or text. We actually have encountered a few voice as well. So, yes, a lot of it is phish.
[David Spark] Dan, I’ll ask you the same question. Of the people say, “Hey, this seems a little phishy to me,” what percentage are they truly like phishy?
[Dan Walsh] I’d say half.
[David Spark] At least half?
[Dan Walsh] Yeah. I mean, there’s also different types of phishes too. Sometimes people will send a random email to just see if the email’s a valid one. And so it’s more like you’re not necessarily phishing for… I mean, you’re phishing to see if the email is valid, not necessarily phishing to send malware.
[David Spark] Right.
[Dan Walsh] Maybe doing some reconnaissance or something like that. But yeah, I’d say generally it’s about half.
[David Spark] Well, then that’s definitely worth that they’re reporting if those percentages are so darn high. Well, now we are at the point of the show, Sharon and Dan, where I ask each of you which quote was your favorite and why?
[Sharon Milz] I think Sam Oberholtzer’s quote about, what is it, that no fancy platform or tool is going to fix everything. If you think about when you buy gym equipment, you don’t get fit just because you have it in your house. Right? You have to work out every day to get consistent. It’s the same thing that happens with security awareness training or any other platform that you set up on your program, right?
You have to customize. So, you have to spend the time. You have to think about how it fits your organization’s culture and ensure that you’re applying that accordingly.
[David Spark] All right, Dan, your favorite quote and why?
[Dan Walsh] Kevin Walker, where he talks about defense in depth, right? You’re going to penalize someone for clicking on an email, and you have no network segmentation, you have no automated technical security controls in place. That seems pretty ridiculous. So, I think he mentioned email filtering, DNS filtering, browser extension, all those tools that are kind of surrounding the user as they’re doing their job.
I think he hit the nail on the head with that quote.
Closing
24:53.217
[David Spark] Well, that comes to our very end of our show, and I want to thank our sponsor. That’s Intezer. Intezer, extend your security team with AI. Remember, go to their website, Intezer.com, for more on exactly that. Sharon, thank you so much for coming. I appreciate. We unfortunately missed each other.
We were at the same conference over in Florida. I had no idea you were there. I wish I had known so we could meet in person. But I hope you had a good time at the conference. This was a cool conference where they gave out badges that were 5-1/2″ floppies, which is pretty darn cool. Dan, you have a new gig.
Are you hiring at your new gig?
[Dan Walsh] I am. If you search Datavant careers and look at the IT and Security section of the website, we are hiring a variety of security and IT professionals and would love to chat with folks if they’re looking to make a move.
[David Spark] And are you hiring people remotely?
[Dan Walsh] I am. Yep. So, Datavant is a remote-first company, and so that’s definitely a possibility.
[David Spark] Well, thank you. Thank you very much to Sharon Milz, who’s the CISO over at TIME, and also Dan Walsh, who’s the CISO over at Datavant. And thank you, our audience. We greatly appreciate your contributions and for listening to Defense in Depth.
[Voiceover] We’ve reached the end of Defense in Depth. Make sure to subscribe so you don’t miss yet another hot topic in cybersecurity. This show thrives on your contributions. Please write a review, leave a comment on LinkedIn or on our site CISOseries.com where you’ll also see plenty of ways to participate, including recording a question or a comment for the show.
If you’re interested in sponsoring the podcast, contact David Spark directly at [email protected]. Thank you for listening to Defense in Depth.