Are Phishing Tests Helping or Hurting Our Security Program?

Are we missing the point with phishing tests? We know attackers will just craft better messages to get clicks. So how can we make our own testing more meaningful?

Check out this post for the discussion that is the basis of our conversation on this week’s episode co-hosted by me, David Spark (@dspark), the producer of CISO Series, and Geoff Belknap (@geoffbelknap). Joining us is Dennis Pickett, vp, CISO, Westat.

Got feedback? Join the conversation on LinkedIn.

Huge thanks to our sponsor, Concentric AI

Full Transcript

Intro

0:00.000

[David Spark] Are we missing the point with phishing tests? We know attackers will just craft better messages to get clicks, so how can we make our own testing more meaningful?

[Voiceover] You’re listening to Defense in Depth.

[David Spark] Welcome to Defense in Depth. My name is David Spark. I am the producer of the CISO Series. And joining me for this very episode, you’ve loved him since he was… You know, I’ve mentioned this before. Since he was a child actor. And now, he’s become a CISO. We’re so excited to see him grow up.

And you know what I think most I enjoy, Geoff, is you didn’t go the path of the other child actors. You kept it together. I’m very proud of you.

[Geoff Belknap] Yeah. When I got my first Academy Award, it kept me humble. I didn’t let it change me.

[David Spark] I appreciate that.

[Geoff Belknap] So, hey, everybody, thanks for tuning in. I don’t know what podcast this is now.

[David Spark] [Laughs] It is actually a cyber security podcast. Hang tight. You’ve been on it before, Geoff.

[Geoff Belknap] All right.

[David Spark] Our sponsor for today’s episode is Concentric AI. Find and protect your data with Concentric AI. We’re going to talk about exactly how they do that a little bit later in the show. Geoff, phishing tests in and of themselves don’t tell us much. You either make them too obvious to get a good response rate or devious enough to say, “Gotcha,” to your employees, as Mike Johnson, CISO of Rivian and CISO Series Podcast cohost said on LinkedIn.

Now, Mike has many times mentioned that he is not a fan of phishing tests. So, the tests themselves he believes aren’t worth much. So, I want to know your opinion as we talk about this, but how do we start answering the what happens next after a phishing message rather than just assigning a pass/fail grade and make the person take more awareness training?

[Geoff Belknap] I think this is a great way to think about. It shouldn’t be how do we decide whether employees are any good or not based on the phishing training but really what do we do with this information. And I think that is really the crux of where we’re at with this. I think we’ve got a great guest to have a conversation about today.

[David Spark] And I think also the question is do we value that information. Is there value in that information from a phishing test? Which there is debate on that, and that is the point of this episode. So, I’m excited about our guest. He is the VP CISO of Westat. None other than Dennis Pickett. Dennis, thank you so much for joining us today.

[Dennis Pickett] Thank you very much for having me. I’m excited to talk about the topic. It is always on people’s minds. It’s something that’s happening more and more these days. And the more we discuss it and learn about it and make it more affective, the better we’re all going to be.

What aspects haven’t been considered?

2:46.661

[David Spark] Rohyt Belani said, “Abandon viewing phishing simulations as tests. They are immersive education opportunities. The goal shouldn’t be to focus on the organization’s click rate or, worse still, any one person’s but rather the organization’s resilience as measured by the ratio of the number of employees that report the email as suspect verses those that take the bait.

Now, driving that reporting culture can be the difference between an attack turning into an infection versus a full blown breach.” And we hear that a lot, as well as that it’s more important about the reporting rather than you clicked on it. Although they don’t want you clicking on it. Jonathan Waldrop, CISO over at the Weather Company, said, “The best way to protect against phishing attacks is to give your employee base clear guidance on how to escalate when they do accidentally click.

And to be clear, nobody should be fired for admitting that they made a mistake. If your security relies on one person not clicking, you are doomed to fail.” And that…I think that very last line is key because if that’s your security program, you haven’t built that good of a security program, have you, Geoff?

[Geoff Belknap] No. No, not at all. I think Jonathan hits the nail on the head. Although I think he gets there in a roundabout way. And here’s where I think it’s really important to zero in. The best way to protect against phishing attacks is to use either phishing resistant credentials to transition everybody to passkeys, or password lists, or something that is very difficult to phish, or really build a bunch of automation either into the browser, into your desktop, whatever they can detect, and block phishing attacks.

You have to just attack this methodically and mechanically. It is not hoping that somebody is going to report a link. Now, if your organization doesn’t have the budget for or just is at a different point in their maturity journey along the journey of security where you don’t have this kind of automation or you can’t roll out phishing resistant credentials, yes, phishing tests are fantastic.

They’re very useful. But I think both of these commentors make a great point that they’re not useful in the sense of you can use them as a punitive tool against your users. They’re useful to just understand where do you have more opportunities to help people understand where to report things, is your tooling working, what’s going on.

And I think that’s the key. It’s not about your users. It’s about what you can do for them.

[David Spark] Very good point. All right. Dennis, your take on this. Where do you see the value? And do you think both Rohyt and Jonathan set it up well here?

[Dennis Pickett] They definitely set it up well, and they mentioned a few things that I think are very important. One of the things talked about was pass/fail and punitive measures. And I totally agree. You can’t just have phishing testing as a pass/fail event and look at your employes like, “You are bad.

You clicked on something once. You clicked on something again. That’s a bad thing.” You need to use it to derive data and information from your employees, from your organization, and see where you need to be doing better because not all phishing messages are equal. You can run a simple phishing message that says, “Please click this link,” versus one that says, “You have a package waiting.

Please click this link.” And you’ll get vastly different click rates. Understanding why things are happening and which kinds of things your employees are more susceptible to and then reacting to that I feel is an excellent tool to have in the toolbox. We can’t always expect to be able to block things at the perimeter.

It’s the whole defense in depth concept that the podcast is named after. You have to assume at some point something is going to get through, and what can you do about it then?

Why are we blaming users?

6:26.839

[David Spark] Gadi Evron of Knostic said, “Phish tests are measurable, and I’d like to see them used for risk scoring of users with associated permissions rather than ‘sent to website for a learning experience.’ 4% will always click. Track those who read your intentional emails, attend your webinars, and care about security.

They are natural champions for the cause and act as human sensors who will report issues you want to know about. The human detection layer, if you like.” David Jones of RxBenefits said, “Don’t condemn the ones that click but celebrate the ones that report them.” All right. So, this is something that we referenced, that this is the thing that we want is more reporting.

So, is phishing tests the best way to go about reporting, Dennis, or is there maybe a different way? How do you sort of I guess build that reporting muscle, if you will?

[Dennis Pickett] Sure. I’m always in favor of building the better mouse trap, but phishing is what we have now, and it’s what a lot of the tools center around, so we try to use them as affectively as we can. At Westat, we’ve implemented a phish alert button. I know many companies have done such a thing.

But it’s one way you can track that reporting rate, and you can see how affective your education has been and get an idea of how many people are trying to help there.

[David Spark] Let me pause you there for a second. You said a phish alert button. So, like in people’s inbox, there is a very clear button that says, “I think this is maybe a phish.” So, it’s like ever present, and someone could click on it at any given time.

[Dennis Pickett] Yes, exactly right. It’s even got a little fishhook on there, so it’s very recognizable by the icon. We’ve done a lot of employee/user training around this. Even asking them to click it on a sample test message, so they kind of get the hang of it. And it’s there to do a couple of things.

One, it’s to sort of crowdsource reporting. If enough people click on something, it’s going to automatically be quarantined until it’s investigated. And then we add some automation behind that that can investigate those emails because these are the ones that have gotten through. These are the ones that have ended up in the user’s inbox, and now we’re relying on them in our layer of defense to report things that we can take action on.

So, if enough people report a particular message, we get flagged, and somebody on the security team does an investigation. And we have automated tools where we can, but those automated tools aren’t perfect, and sometimes they can’t make a clear determination. We have a process for doing investigations after that.

Sometimes the users don’t care. They report something. It’s not something they’re concerned about. Other times it looks like something they might need to open or an attachment they might need for business purposes, and they’re just not sure.

[David Spark] And one of the things that we’ve heard that is valuable is that looping back. Like, “Hey, because you reported that email we investigated, and we discovered this.” So, they realized there was some value to it. Do you do something like this?

[Dennis Pickett] Absolutely. They want that feedback. In fact, when we first started this we weren’t providing that, and we have users following up with us, saying, “Whatever happened to that thing I reported.” And that’s when we realized, “You know what? It’s not only just a good practice to let them know, but that reward, that positive feedback and the thank you for having done that and the results of that investigation causes them to want to do that more.”

[David Spark] That’s a good point. Now, let me… I’m going to repeat a story that I’ve mentioned before, and I’m going to throw this to you, Geoff, in just a second. Is actually a good friend of mine, who… She works in HR for a company. They actually had to let someone go because they repeatedly kept failing phishing tests.

Now, this… I know. Everyone I’ve mentioned this to, they’re squeamish. Andy Ellis was horrified by this. Everyone… But now here’s the thing that was weird – this guy was a mechanic. He was not a knowledge worker. He did not work on the computer. He used his computer very, very little. But everyone was required to take this security awareness.

As many times as they tried and as many times as they trained him, this guy just kept repeatedly failing, and they had to let him go. First of all, have you ever heard of a story like this before? And B, just the way you’re shaking your head, it sounds like you’re completely and utterly horrified, Geoff, yes?

[Geoff Belknap] No, that’s the stupidest thing I could imagine. You don’t… Look, if you’re running a business, regardless of what the business is, you are not hiring people with the requirement that they know how to identify phishing emails. And I hate to just come out too hard against this, but I’ll use this as a performative example of where security fails the business.

Our job as security professionals is to support, and accelerate, and enhance the success of the business by derisking it, by helping build automation, by helping it win and be successful. You’re not helping the business if you construct a mindset that everyone needs this basic security skill and if they don’t it, they should be fired.

Right?

If I’m running a F1 team, I’m probably hiring engineers that know how to build specific aerodynamic tools or whatever. If I’m running an AI program, I’m looking for people that have very specific skillsets. And whether or not they can identify a phishing email is not relative to the value that they’re bringing to the organization.

You shouldn’t be punishing them. What you should be doing is investing any of that time and energy in doing better at detecting, and preventing, or responding when somebody does click on it. Because the one thing you can predict with absolute certainty is that people will click on something they’re not supposed to.

There is nothing you can do to stop that.

[David Spark] So, let me go back to the earlier question that I was asking Dennis, is what do you think is the best mechanism or the best way to help people flex that muscle of being aware to phishing in general if it’s not doing phishing tests? Yeah. Well, I think you can do phishing tests, but I think the way you should think about them is not as somebody…as part of somebody’s performance as an employee, but you think about them as two things primarily or the way I think about them at least, is, one, as a datapoint to your detection and your tooling.

Like is it easy for people to report these? Does your tooling automatically find them? Blah, blah, blah. The other part of it is how are you building incentives for a more security positive culture? And one of the things you can do as security with phishing tests and actual phishing campaigns is incentivize people that report them and celebrate them as champions.

One of the things we did at another company I worked for is we had buttons, and stickers, and t-shirts. And it was like if you… If there was a real phishing campaign and you were the first person or one of the first couple of people to report it, we would give you a t-shirt, and people would wear them with pride and be like, “I saved…” And it said the company name.

And people… I’ll tell you one thing people really are is competitive. They were like, “I want to find that next time.” And that is driving positive cultural change in your organization without it having to be punitive.

Sponsor – Concentric AI

13:27.958

[David Spark] Before I go on any further, I do want to tell you about our absolutely spectacular brand new sponsor, and that is Concentric AI. So, as the leader in AI driven data security posture management or DSPM, Concentric AI understands that gen AI tools like Microsoft Copilot, super, duper popular…well, those are generating new data protection challenges that are just unheard of not that long ago, so that’s why they’ve developed a cutting edge solution designed to protect your most sensitive data no matter where it resides – on premises, in the cloud, or within SaaS applications.

So, today’s data protection solutions need to go beyond just identifying risks. Concentric AI was designed to do more. It proactively and automatically discovers, classifies, and remediates at risk data across your entire organization.

Plus it makes sure that any content generated or accessed by Copilot remains protected from unauthorized access or accidental exposure. Now, whether it’s structured or unstructured data, Concentric AI adapts to your unique environment, providing the intelligence and oversight you need to stay compliant and secure.

So, by integrating seamlessly with Microsoft Copilot, Concentric AI empowers your organization to harness the full potential of AI technologies without compromising on security. You can trust Concentric AI to keep your data safe so you can focus on innovation, growth, and success. So, go ahead, visit Concentric AI today.

Concentric.ai. Go there to discover how you can protect your PII, your PHI, and your intellectual property with ease.

What’s the optimal approach?

15:28.210

[David Spark] Thomas August of AltaMed Health Services said, “I think of phishing testing as a routine practice of fundamentals. Much like pro basketball players will shoot hundreds of free throws a day, I think that drilling workforce members on basic fundamentals is essential. You’re not trying to trick them, just practice the fundamentals so that it becomes second nature.” Interesting take.

Jason Hoenich of Arctic Wolf said, “We are finally accepting that humans aren’t the last layer of defense, sure they may be a link in the chain – but often the ‘weakest’ link is under the most load.” By the way, I’m going to just take an aside. This is something that we’ve heard said. That people aren’t the weakest link, but they’re the number one attack vector.

All right, going back to Jason’s quote here. “In my career I’ve done live-hacking demonstrations that showed users what happened ‘after the click’ and was far more engaging and effective than my phishing sim program was. Let’s ACCEPT the fact that users will click, just like users will plug in a USB, just like users will lose devices, and build a support system around those behaviors.” I find… By the way, I find both of these interesting.

Thomas said, “Look, it’s like working out. That’s why we do phishing tests.” And Jason says, “Look, they’re a good layer of defense. They’re not the best layer. But they are a good layer, and people are fallible. It’s part of their makeup. Let’s build defenses to it.” What say you on this, Dennis?

[Dennis Pickett] Yeah. I mean I like both those points, and I think they’re complimentary as opposed to contradictive. I think that practicing the fundamentals is important and having your user base be more aware of these kinds of things does not hurt anything. You could argue is the cost not just in terms of the tools you use but the user’s time to take the training, and to do the testing, and things like that…does that provide the value for what you’re looking for.

But let’s assume for the moment that it does. Then I think why not have these things in place? Why not have your users practice these fundamentals? I think an earlier quote talked about a culture shift that you’re driving within the organization that’s just more awareness. You know, TSA has their famous motto, if you see something, say something.

And it took years before that came something that when you walk into an airport, if you see an unattended bag, you are going to remember that saying and go, “I better tell someone about this.” Why not have people trained with that kind of awareness and that kind of reaction?

[David Spark] In your program, Geoff, you don’t see people as the weakest link, but you realize people are human, and we all make mistakes. Geoff, I believe you’ve made a mistake in your life before, yes?

[Geoff Belknap] Never. I have never made a mistake. I think that’s what my wife keeps saying. It could be the opposite of that. Yeah, look, people are human. And not only do they regularly make mistakes, we’re talking about the intersection here of most people whose jobs is something else, having to learn a skill which is affectively becoming a part-time security person, they are not going to get it right most of the time.

And frankly, it is really easy to trick people. The phishing emails out there are really, really good. And people are just humans. They will fall for tricks when you lie to them. So, while I think it’s absolutely valid to make these tests a part of life and to sort of see and get the data about what percentage of your user base or your user population is falling for these and using that information to inform decisions you make about purchasing, or priorities, or more training, we should just be really clear, this is not a performance evaluation tool.

You should not be judging, grading, or deciding people’s fate based on how they perform on one of these.

What’s the best tool for the job?

19:21.629

[David Spark] Sunday McDickson Samuel, a great name, of SMSAM SYSTEMS, said “I’d rather take the pressure off the employees by implementing PROACTIVE security controls that PREVENTS the phishing email from getting in the first instance. Isolate all web contents to the network (or work from home). While phishing awareness training is great, it does put unnecessary pressure on employees.

Implementing DMARC to reject phase prevent most phishing mails from getting in. Additionally, Implementing API based ESG prevents over 95% of anytime of phishing getting in.” So, by the way, I think most of us, God willing, are doing the things that Sunday is saying right here. Yes, Geoff? I would hope.

[Geoff Belknap] I think… Again, I sort of talk about this at the top of show. I think this is where the journey of security takes you at the higher end of the maturity curve to fight phishing. Right? You’re going to implement these things in your email. You’re going to implement strict DMARC rules. You’re going to build, buy, or borrow automation.

You’re going to implement very phishing resistant credentials and ways to manage identities. It’s like, yes, this is where you’re all going. And frankly, this is where your energy and your investment should go. If you’re spending money on phishing testing, it should be to validate that doing this is a positive force for your organization.

[David Spark] Agreed. But I think it’s…previously with Jason’s comment, it’s saying they’re a weak link, but it’s a good link. And Dennis, you were just saying, “Look, this is all part of Defense in Depth,” right?

[Dennis Pickett] Yeah, absolutely. I think in the quote you were just reading they said the goal should be to prevent it from coming in at all, and I think we can all agree that is something we would love to have happen, but that’s just not the reality these days with the tools at our disposal because things are still getting through.

And I think some of this is context. If it gets through to a regular user, that’s one thing. If it gets through to somebody in your finance department who’s sending money and payments out because they think the messages they’re getting are from legitimate people within your organization, that’s a much bigger risk and a much bigger problem.

[David Spark] All right, I want to wrap up this episode, just stating our sort of opinions on the value of phish testing and to what level is it useful, to what level is it not useful, and if it’s not phish simulations, what other mechanism would you sort of rely on more? I will start with you, Geoff.

[Geoff Belknap] There is definitely some value to phishing simulation. If for nothing else, if you’re already implemented every tool, technology, and mature process you can…

[David Spark] Like Sunday pointed out.

[Geoff Belknap] Yeah, exactly. To just understand what the problem still looks like, and your user’s behaviors, and what, if anything, you can do to continuously improve. And if you haven’t made all those investments, I think it’s still really valuable to understand what kind of behaviors you’ve got in your user population.

But I think it’s just really useful to you as a security program, not necessarily useful to HR.

[David Spark] By the way, I’m glad you brought that up at the end, because that’s a really good point to make is this gives you… The whole point of being a security professional is to understand your environment. Your humans are part of your environment, and how they are behaving is part of your understanding of your environment.

So, these phish testings are not really…should be designed for who’s failing and not failing, and who’s reporting and not reporting but more of how does my community behave on these kinds of different tests. All right, I throw this to you. Where do you stand on the value of phish simulations and other possible solutions, Dennis?

[Dennis Pickett] Sure. Now, if we weren’t doing these phish testings or we thought they weren’t affectively, we should be relying on other layers of protection that we should be doing as well. You know, we talked about between multi factor, and pass phrases, and things. But that’s one kind of phishing where somebody is trying to capture credentials from a bad link.

I mentioned a moment ago about if your finance department gets something that says, “Please make a payment,” and everything looks perfectly legitimate, you still need some other measure like password, a code, a phrase. Even if you’re getting a call or even these days a video call that appears to be from people within your organization, you can’t always trust those things anymore.

So, you know, the whole trust but verify. You’ve got to… I think you need to be relying on other measures you should have in place already. These should be additional protection so that if credentials are lost nobody else can actually use them because they’re not getting the whole part of the credential.

Or if somebody requesting a payment to be made, you’re not sending that out right away because there’s another measure to verify that this is actually an authorized request.

[David Spark] Two keys that have to be turned at the same time essentially from different… It can’t be one person stretching their arms, if you will.

[Dennis Pickett] Yes. Yes.

Closing

24:29.897

[David Spark] Excellent. This was awesome. All right. Now we’ve come to the point of the recording where I ask you, our guest, Dennis, which of these quotes was your favorite and why.

[Dennis Pickett] I want to call out Rohyt Belani’s quote about driving culture change within the organization because I think that phishing tests are more than just, as we said, a pass/fail, did this user do this or not. It’s about getting your organization more focused on security and being more aware that not everything is going to be something you should be trusting.

You should have extra measures in place. You should be verifying that what’s happening and what you’re reading and seeing is real.

[David Spark] All right, Geoff. Your favorite quote and why.

[Geoff Belknap] I’m going to go with David Jones from RxBenefits who said exactly what’s always in my brain, which is don’t condemn the ones that click. Celebrate the ones that report the emails. And I think…

[David Spark] Give them a T-shirt.

[Geoff Belknap] Give them a T-shirt, a button, a well written handwritten note, whatever it might be. But let people know that reporting it is the thing you’re really valuing. You’re not really looking for people that missed it, but you’re celebrating the people that caught it.

[David Spark] And that dovetails very nicely to what Dennis said about culture change, and that is all about building up that sort of positive culture around security. Well, that brings us to the very end of this very episode. I want to thank, first, our sponsor of this episode, which is Concentric AI.

Find and protect your data with Concentric AI. If you’ve got Microsoft Copilot in your environment, you owe it to yourself to take a look at Concentric AI. Just go to their website. Concentric.ai. That’s concentric.ai. Dennis, I will let you have the very last word. Thank you so much, Geoff, as always.

And, Dennis, first of all, I want to ask, at Westat, are you hiring over there?

[Dennis Pickett] We are often hiring. I don’t know about necessarily this moment. But if anybody is interested, you can always go to the career page at westat.com to see what’s posted and if there might be something that is a fit for you.

[David Spark] And if it’s something in security, is it appropriate if someone reaches out to you directly?

[Dennis Pickett] Yes, please do. Please do.

[David Spark] Okay. So, the link to his LinkedIn page will be on the blog post for this episode so you can follow up with him as well. Thank you so much, Dennis. Thank you so much, Geoff. And thank you to our audience. We greatly appreciate your contributions and listening to Defense in Depth.

[Voiceover] We’ve reached the end of Defense in Depth. Make sure to subscribe so you don’t miss yet another hot topic in cyber security. This show thrives on your contributions. Please write a review, leave a comment on LinkedIn or on our site, cisoseries.com, where you’ll also see plenty of ways to participate, including recording a question or a comment for the show.

If you’re interested in sponsoring the podcast, contact David Spark directly at [email protected]. Thank you for listening to Defense in Depth.