Defense in Depth: Training for a Cyber Disaster

What are you doing to prepare for the next cyber disaster? You must train for it, because when it happens, and it will happen, everyone should know what they need to do.

Check out this post for the discussions that are the basis of our conversation on this week’s episode co-hosted by me, David Spark (@dspark), the producer of CISO Series, and Geoff Belknap (@geoffbelknap), CISO, LinkedIn. Our guest is Roland Cloutier (@CSORoland), CISO, TikTok.

Got feedback? Join the conversation on LinkedIn.

Huge thanks to our sponsor Keyavi

Data that protects itself?  Now it does! We made data so smart it can think for itself. Secure itself. Stay continually aware of its surroundings. Control where, when and who is allowed access. And automatically report back to its owner. This changes the entire cybersecurity paradigm. Learn how.

Full transcript

[David Spark] What are you doing to prepare for the next cyber disaster? You must train for it. Because when it happens, and it will happen, everyone should know what they need to do.

[Voiceover] You’re listening to Defense in Depth.

[David Spark] Welcome to Defense in Depth. My name is David Spark. I am the producer of the CISO Series. And joining me for this very episode is Geoff Belknap. You also know him as the CISO over at LinkedIn. Geoff, people know you by this sound that your voice makes.

[Geoff Belknap] Hello, and welcome to Defense in Depth.

[David Spark] Very good. Our sponsor for today’s episode is Keyavi. They are the self-intelligent data company. Very, very tool technology they have. They’ve been, by the way, a phenomenal, phenomenal sponsor of the CISO Series ever since its very, very beginning. So, more about Keyavi data later in the show. But first a quote, Geoff. “A crisis is chaotic. And your response has to be muscle memory. If you aren’t doing crisis management exercises, you clearly don’t believe one can happen,” said Jamil Farshchi, CISO of Equifax. First, Geoff, is everyone accepting of the need to conduct crisis management exercises? And if so… And I know this is what we’re going to talk about for the entire show. But as a sort of headline, what do you actually do, how often, and how do you learn?

[Geoff Belknap] Well, only a Sith speaks in absolutes. But I think most people are accepting of the fact that a little bit of preparation can go a long way. And I think it’s really variable in orgs that have had an incident are definitely much more willing to invest more time and preparation because they’ve probably learned they weren’t so prepared. Orgs that have been blessed with never being affected by kind of major incident, it’s going to be a little bit harder. But at the end of the day, as I think our guest will agree, preparation is pretty important.

[David Spark] I’m interested to know, because I think there is agreement on preparation, but the question is when you do it, how often you do it, and is there a diminishing rate of return on all this preparation because you can’t overdo… Because there’s other stuff you need to do to run a business and all this. Anyways, joining us is Roland Cloutier, CISO of TikTok. Roland, than you so much for joining us.

[Roland Cloutier] Gentlemen, thanks for having me. Glad to be here.

How do I start?

2:21.696

[David Spark] Troels Oerting of BULLWALL said I think what we all think, and that is, “It’s unfortunately not if you get breached but when.” And we hear this line a lot. That’s why cyber crisis management is so important and needs to be trained frequently. Now, it is not an absolute. Yes, Geoff, as you said. But a lot of us feel that we get hit by these incidents inevitably. Let me also point out that Matt Stamper, CISO of EVOTEK said, “There is one certainty in our profession, and that’s that we’ll be hit with an incident when it’s least opportune, and we’d better be prepared.” So, there’s never a time you get hit by some type of incident and you go, “Well, this was the time we wanted this.” Is it?

[Geoff Belknap] I think there is rarely a moment where you’re sitting around going, “Whew, thank God that happened then or that happened now.” And there’s rarely a time even if you’ve spent a fair amount of time preparing where you’re like, “Oh, this feels fine. This is good.” But there are a lot of ways you can closer to that where it doesn’t feel like panic, where it doesn’t feel like no one knows what to do. And that’s really what you’re aiming for. You want to make sure that when a major incident happens, not just phishing or anything like that… Although those are fine things to start with, too. But when a major incident happens where it’s going to be all hands on deck and you have worried board members, and worried CEOs, things like that, those are the things where you want those people that might otherwise be unprepared for what’s about to happen to at least have some semblance of what they should do, what they should expect, who’s going to do what. And you want your executive team, and your board, and your employees especially to feel like, “We’ve got this. We’ve done this kind of thing before. We know what to do.” And it’s not going to terrify the people that are working on it. If you’ve done enough to get there, you are ahead of the game.

[David Spark] That’s a good point. You just want to at least pull down are bare minimum…pull down from panic. Yes, Roland?

[Roland Cloutier] [Laughs] Yeah, I think we want to go a couple layers deeper than that. Geoff, I think you hit a real good point. Incidents are like onions. There’s the outer core of people are barely touching it, and there’s the inner core. Wherever you are standing inside that onion, you want people to be confident. And confident that they can handle it, not necessarily that they know exactly what’s going on but that they understand how to manage through it. Because all crisis management is is management at the end of the day. There’s teams that focus on remediation, and investigation, and every other component thereof. But I think what we’re trying to do when we talk about issues and incidents within businesses and agencies is to get people focused on a process that they can follow down a path to success. What I also think is interesting that this topic brings up is what is an incident versus a crisis, and how do you deal with those independently. It’s not just cyber. It’s not just [Inaudible 00:05:24] service. It’s not just a breach or a data exfiltration. It’s a whole lot of things. So, how are we preparing as leaders to help our teams prepare and be successful in those areas. And those are some other points I think we should touch on today, too.

[David Spark] Let me go to you, Geoff. And I’d like quick comments from both of you on the whole topic of incident versus disaster. Because incidents could stay conceivably within the cyber security organization of the company where a disaster all of a sudden involves other people. Is that a good way to define it or no?

[Geoff Belknap] It’s a great question. And I don’t know if I have a strict definition here, but I generally feel like, Roland, you’re making a great point. That a crisis is really more about the scope and your response than it is about the specific details. I feel like a crisis is really where you’re not entirely sure how to respond. You’re not entirely sure what’s going on. People are not quite sure what to do. I think in terms of comms lingo, like a crisis is going to be anything where it takes a whole of company response to manage that incident. And it’s really important to point out security incidents maybe 5, 10 years ago really were just a technical thing that you responded to. Now most security incidents are a thing that is having direct impact on whether that business is an ongoing concern as a result of how you respond to it. I don’t know. What do you think here, Roland?

[Roland Cloutier] Yeah, I’m with you. I think when we talk about crisis versus incident, versus any other terminology you want to use, it’s about scale. It’s about scale and impact to the business. And if you’re in this for business operations protection, our job is sustainability and a resilience of the business that we’re protecting. And so I disagree a little bit with Dave thinking that incidents remain inside security because I think if you’re in a digital ecosystem of a business that it goes all over. It can be compliance. It can be the fraud organization, financial crimes. It can lead to so many different areas that a lot of people have to get involved. So, what we talk about is focusing on the five pillars of incidence response – people, standards, process, technology, and partners. Every single time. If you train your organization on those five basic pillars, involve your partners in that training, you get to the best outcome each time.

Where does the solution fall short?

7:42.159

[David Spark] Justin Clarke-Salt said, “I can’t help but wonder why the uptake on this practice has lagged behind other forms of crisis preparedness such as business continuity planning.” And Jamil Farshchi, CISO of Equifax who penned this post that we’re talking about said, “These exercises are viewed as check the box at a lot of organizations, and therefore of limited value. You have to have thoughtful scenarios, leadership support, engagement, and it’s a not if but when type of culture to make these successful and value added.” So, I’ll start with you, Roland, on this. Do you find that crisis preparedness is kind of the last thing that is prepared for after like disaster recovery, business continuity planning?

[Roland Cloutier] I love this question, and I think Justin is so right in asking the question because what people often fail to realize is that crisis management does not sit in a silo unto its own. When you think about that practice, that service delivery that you have to deliver, it’s really around the concept of business resiliency. And you think, “What’s involved in that?” There’s business impact analysis. There’s business continuity planning. There’s disaster recovery. And then there’s crisis management. And if you’re not looking through all four areas of business resiliency then you’re not going to be good at any of them. So, I think what we have to do within the concept of preparing is to ensure that we understand, educate, prepare in each one of those areas.

[Geoff Belknap] Yeah, I couldn’t agree more. I think more to the point here is that… And this will be a little bit cynical. There’s a lot of marketing dollars spent to get security teams to focus on some of the technical issues, and the training, and we’ve had lots of discussions about these across all of the CISO Series podcasts. But I think there’s very little marketing push so to speak even either directed as security teams or directed at executives in non-security roles that sort of focus on the fundamentals. And I think this is something I find Bob Lloyd [Phonetic 00:09:55] and I talking about a lot, which is we have all these crazy stories, these [Inaudible 00:09:58] stories about like, “Don’t use coffee shop Wi-Fi. Don’t ever click on a QR code link.” That’s not… Fundamentally if you’re running security correctly, you’re not worried about those problems the same way that the panic is… Fundamentally if you’re running security well, you’re intrinsically having crisis preparedness meetings. You might not be calling them crisis preparedness. You might not be calling it a tabletop or formally inviting somebody in, but you are building into the workflow and the life cycle of security in your organization preparedness for when something goes wrong. And it just turns out to be really important. But it turns out to be something that nobody really talks about. I think cynically, again, because it’s not monetized.

[Roland Cloutier] I don’t think it’s cynical, Geoff. I think you’re spot on. I think there’s this whole theory about crisis management – that it operates in its own area and its own specialty. But it’s not. It’s how we operate every day. It’s how we build run books. It’s how we do muscle memory, as you said, at the beginning. This is to Jamil’s quote…this is muscle memory. And I think the same way that we got really good with technical standards, certifications, accreditations, and everything else within the cyber security and cyber defensive operations area, back to the rainbow series. I might be dating myself. But we follow a path. We can do that with crisis management. You have major multinational organizations, ISO standards around crisis management. Build your program off that. Dive in the same in that we did it to ISO 27-001. Use free programs like sims from FEMA in the United States that enables you to train all of your business around how to do command incident management. These are simple things. But really back to Geoff’s point, if you build it into your operations spectrum if you will, it’s part of how you work incidents and crises, and it’s the vernacular and taxonomy of which your business operates on. It just becomes how you address crises every day.

Sponsor – Keyavi

11:55.672

[Steve Prentice] One of the top level concerns about being prepared for a cyber disaster is of course data. It is generally accepted that in the case of a breach, data loss is inevitable. But Elliot Lewis, CEO of Keyavi, has issues with this.

[Elliot Lewis] Why is it always inevitable? Because data couldn’t protect itself. Now data can. Data can protect itself, its intelligence. Self-protecting, self-aware, in perpetuity. Which means that when you’re saying prepare for a data breach, you’re using all the old tech, all the old paradigms, and all the assumption that data cannot protect itself. And when you use things like Keyavi, which puts protection and the control right into data itself no matter where it goes, what is a breach now. Well, a breach is somebody breaking into your systems and working on your devices, and your servers, and your operating systems. But a breach when you use Keyavi is no longer about data loss. Because your data is not losable. It someone takes it, it will just geofence itself. It’ll protect itself. It will delete itself, and it will report all the things happening to it in the real time, which is the other side of the equation, too. So, when you’re preparing for a breach, now it’s all about the systems and the architecture and everything that’s around data. But data itself, that’s not going to be a breach target with Keyavi involved.

[Steve Prentice] For more information, visit Keyavi.com.

What would a successful engagement look like?

13:33.563

[David Spark] David Kennedy-Pitt of Capita said, “One of the biggest challenges in these situations isn’t always the issue itself but actually how the crisis team will affectively work together to get the organization through it.” And Graeme Payne of Kudelski Security said, “This is one of the key lessons learned from major data breaches. Unfortunately many companies do not carry out breach simulations with their executive team and boards.” So, we hear this all the time. The whole point of the tabletop exercises is for people to understand, “Oh, this is my job when this happens, and this is what I do. And this is who I communicate to this person.” Is there any other way to figure this out unless you do a tabletop, Geoff?

[Geoff Belknap] Well, there really isn’t. And I think just in my ongoing effort to lower the bar, again, you don’t even need to run a tabletop. But you do at bare minimum need to have a discussion with your executive team and your board about what happens in an incident – who is going to be engaged, who is going to do what, who is going to talk to them, and how often are they going to communicate. You don’t have to start with a very elaborate tabletop. It can help. And if your team is ready to do that, do it. I think the other side is some of this is just you don’t learn, and especially non security people. You don’t really learn what happens in an incident or what an incident looks like until you’ve sort of experienced one – either a simulated one or a real one. And I think, again, if you’ve been blessed by never having experienced a major incident in your organization, a tabletop is a great way to get a real feel for what are the kinds of things that can actually cause incidents. Spoiler alert, it’s not all zero days and foreign spy hackers. And two, what does it actually look like to respond to one. Also spoiler alert, your team probably can’t go through 13 petabytes of data in 15 minutes to answer that pressing question of who did it, and why. So, I think it’s really good both for the security team to learn the fact that you can’t respond as fast as you’d like, but it’s especially good for the rest of the org to learn what are they going to be doing while the security team is responding to this. Who is going to be handling press calls or customer calls? How are you going to respond to those? And I just can’t say enough about preparing your organization, especially the non-security parts of your organization, for what happens will really just ensure the success that your organization will see when it exits the incident.

[David Spark] Okay, Roland, quick question for you. You’ve done tabletop exercises. What have been findings that you’re like, “Had we not done this there’s no way we would have figured this out,” kind of a thing.

[Roland Cloutier] Yeah, I think it comes in two parts. One is that the executive management level to be quite frank outside of the security leadership area. So, this is when you’re involving your executive leadership team, your CHROs, your CFOs, your COOs. How’s the business going to operate? Where can you operate if you lose the sight? How does insurance come into play? Do we have partners in that part o the globe? All of those things are super important when you’re doing tabletop exercises to make sure that you get the business side of what you’re dealing with. Because all of a sudden…

[David Spark] Let me ask you a question regarding that, and it’s a guest actually I’m going to have in studio later today. We did this on the other show. They asked a question of their team saying, “How long can you be without power before it becomes critical where we’re now losing the business?” And they said, “Oh, we could be without power for 72 hours.” Then they had an actual incident that took them out for 72 hours, and they realized no, the real answer to that was two hours. So, the question is how do you not run into a problem with that where they answer, “Oh, we can handle more,” when really no, they can’t.

[Roland Cloutier] Yeah, I think having the right people in the room and asking the hard questions and the last question always is, “And how do you know?” I think that is something…

[David Spark] That’s a good point.

[Roland Cloutier] …that is a great question to ask when someone fires off an answer, and you say, “And how do you know?” And the follow ups coming out of those is to go back, validate, and verify. You’re never going to solve everything in a tabletop exercise. Matter of fact, you’re going to push forward and skip a bunch of things because even if you’re doing a four-hour or a six-hour tabletop exercise, you’re going to end up pushing through a lot of stuff. And what you leave is a list of things that you have to go find out, achieve, bake back into your plan. That’s the power of these. It’s not just getting people trained on it and building that muscle memory. It’s being able to pick something that’s important to you, important to your business, a probability. You’re not going to go and do the top 20 tabletop exercises. You’re lucky if you do the top four in a year and do them well. And pull out those things that you need to go work on and work back into your plan.

No one said it was going to be easy.

18:28.714

[David Spark] Jamie Henderson of Interos said, “Simulation exercises are the road to a skillful breach response. They are the only way to find inevitable DR process gaps and other entrances to business continuity.” And Kevin Angone of Equifax said, “This activity not only allows people to walk these paths. It also opens them up to thinking about how else they might prepare for any vent. There is literally no downside to taking actions to be prepared.” Geoff, let me start with you on this. First of all, have you ever done a tabletop exercise and someone said, “Well, this was a waste of time.”

[Laughter]

[David Spark] And then two, my second question is I like this last comment that Kevin made is to be thinking differently. Like let’s give ourselves the breathing room now in the tabletop, like how could we think differently about that. And do you use that kind of methodology?

[Geoff Belknap] Yeah, I think first of all, no one has ever said, “This was a waste of time.” Or if they did, it was probably a poorly executed tabletop. I do think, as a side note, really good tabletop exercises take preparation. And they’re expensive in the sense that you’ve got to have the right people in the room. It doesn’t work if it’s just security and one person from legal or something like that. You want to have the whole executive team in the room, and plus you want to have sufficient representation from the technical teams that are going to respond, as well as the other than technical teams like comms, and legal, and customer support, and sales. And getting all those people in one place is expensive and takes a lot of preparation, but it always, always pays off because you always hit this thing, like Kevin is mentioning here, where people in a tabletop have the first opportunity that they might have had in other situation to think, “Oh, this is what this other person does all day,” or, “This is what they’re going to do during a security incident.” And it really shifts how they think about an incident and how it impacts the business.

[Roland Cloutier] I got to tell you from my time in the military, this is one of my favorite things I love doing. Often I will take my leadership team, and I will take and do a training exercise. This past year I took the entirety of them… They didn’t know where they were going. We put them on a bus. We took them to Cambridge to the IBM Cyber Range. We dropped them on the… We walked them up to the building. Again, they had no idea where they were going. They walk into, and then minutes later they were dealing with incidents. Halfway through, we swap their jobs. And we said, “Okay, now you’re doing this.” We put them in other peoples’ shoes. So, the power of well trained, well planned, and well executed crisis management training and exercises gives you the ability to have a span of control around operators that know how to do their job, back up the people next to them, and help support the entirety of the operation that they’re working for. I think I love Geoff’s perspective in that a lot of people have never done this before. But when you’re in that room, and you’re like, “How do I get a million dollars to get a cold site up and operating,” that’s not going to happen in six hours. That has to be done prior. The CFO now has to learn that there has to be a mechanism by which we can do emergency orders for specific equipment. All of these things are learned during these exercises, and I have never been in a position where anyone walked away saying, “Boy, that was a waste of time.”

[David Spark] I was literally just talking to some CISOs about saying that they had equipment on hand for just these incidents. Do you do this, Geoff? Do you have just essentially spare equipment for emergency preparedness?

[Geoff Belknap] We have contingencies for devices like end user devices, things like that. Like laptops, desktops, etc. The short answer is yes, we have in the…in the data centers that we own, we have spare equipment. I think the reality though is if you’re in a much more regulated space or the thing that you’re providing is medical care or something like that, this becomes part of that process. And the funny thing is… There was another quote here earlier about that. This used to be a really important thing, especially in regulated spaces where you used to have full scale disaster recovery exercises. And you’d go take that giant binder that you had of how to recover your banking system and your mainframe, and you’d go try it out. And then what you’d find is none of it worked, or none of it worked the way you expected it. And then you found these exercises to be incredible valuable in understanding how to recover your business. It feels like a lot of that has sort of fallen by the wayside as we’ve moved along, and now we’re coming back to it.

Closing

23:11.437

[David Spark] All right. Well, that is going to bring us to the end of our conversation today. I want the two of you though to pick your favorite quotes from the discussion. All right, Roland, what was your favorite quote, and why?

[Roland Cloutier] I think the first quote of the day said it all. Jamil says a crisis is chaotic, and your response has to be muscle memory. If you aren’t doing crisis management exercise, you clearly don’t believe one can happen. It is so true. If you’re not exercising, training, and expecting it, you’re probably in the wrong job.

[David Spark] And, Geoff, your favorite?

[Geoff Belknap] I’m going to go with Jamie Henderson from Interos that said, “Simulation exercises are the road to a skillful breach response. They are the only way to find inevitable DR process gaps and other hinderances to business continuity.” And I couldn’t agree more.

[David Spark] Excellent. Excellent point. Well, thank you very much, Roland and Geoff, for coming her. I’m going to… Roland, you’re going to get the very last word. And the question I ask all my guests is are you hiring. And my guess is latest I’ve seen, TikTok is growing, so my guess is you are hiring. But I’ll get to that in a second. Geoff, I’m going to say for you, you’re always hiring. And if for some bent, demented reason you wouldn’t want to work with someone like Geoff Belknap, LinkedIn is another great place to look for jobs in cyber security and in other walks of life as well. I want to thank our sponsor, Keyavi. This is the data that knows where it is, what it’s doing, and whose hands it should and should not be in. If you were interested in learning more about self-aware data, go to keyavidata.com. All right, Geoff, any last thoughts and anybody specific you’re hiring for right now if you’re looking for specific talent?

[Geoff Belknap] We are absolutely looking for people that, A, would love to work with us on disaster recovery and business continuity, so this topic is very near and dear to my heart. But also if you have a passion for all things Cloud or identity, we are especially looking for you. And if we’re not looking for you, certainly Roland or anyone else hiring on LinkedIn is.

[David Spark] All right. Roland, any last words on this topic? And I’m assuming you’re hiring. Am I right?

[Roland Cloutier] You are a 100% right. If you’re mission focused and want to protect the last sunniest corner of the internet, we have about 200 jobs open in 19 different disciplines including our business resiliency program.

[David Spark] 200 positions specifically in cyber open?

[Roland Cloutier] In our security program.

[David Spark] Wow.

[Roland Cloutier] So, like I said, 19 different disciplines from financial crimes through cyber defensive operations, to data defense, and everything in between. So, it’s an exciting time.

[David Spark] Hold up. May I ask this question? Because we have a lot of very green people. How many sort of entry level positions do you think you got open?

[Roland Cloutier] Many. Many. We go from entry up to leadership and everything in between.

[David Spark] Awesome. Good to hear. I’m sorry. I cut you off. Continue.

[Roland Cloutier] Hey, listen. We’re advertising on LinkedIn, of course, and anywhere else. But listen, take this job seriously. If you’re in crisis management, business resiliency, train hard, fight hard I think is what we used to say in the military all the time. And it applies here. And just apply your capabilities with action, urgency, and excellence, and keep carrying forward.

[David Spark] And if people want to find those specific jobs at TikTok and possibly reach out to you, what’s the best way to do that?

[Roland Cloutier] TikTok.com/careers or find us on LinkedIn of course.

[David Spark] Awesome. Thank you very much. Well, thank you, Roland Cloutier, who is the CISO over at TikTok. Also Geoff Belknap. He’s the CISO of LinkedIn. Two rather sizeable social media networks. Glad to have both of you. And as always, thank you to our audience for your amazing contributions and for listening to Defense in Depth.

[Voiceover] That wraps up another episode. If you haven’t subscribed to the podcast, please do. We have lots more shows on our website, CISOseries.com. Please join us on Fridays for our live shows – Super Cyber Friday, our virtual meet up, and Cyber Security Headlines – Week in Review. We’re always looking for fascinating discussions for Defense in Depth. If you’ve seen one or started one yourself, send us the link. We’d love to see it. And when any of our hosts posts a discussion on LinkedIn, participate. Your comment could be heard in a future episode. If you’re interested in sponsoring the podcast, contact David Spark directly at David@CISOseries.com. Thanks for listening to Defense in Depth.

David Spark is the founder of CISO Series where he produces and co-hosts many of the shows. Spark is a veteran tech journalist having appeared in dozens of media outlets for almost three decades.