How Security Leaders Deal with Intense Stress

When you have an incident and you’re engulfed by the stress that lasts more than a day, how do you manage and deal with it? And not only how do you manage your stress, but how do you manage everyone else’s?

Check out this post for the discussion that is the basis of our conversation on this week’s episode co-hosted by me, David Spark (@dspark), the producer of CISO Series, and Geoff Belknap (@geoffbelknap), CISO, LinkedIn. Joining us is our special guest, Tim Brown, CISO, Solarwinds.

Got feedback? Join the conversation on LinkedIn.

Huge thanks to our sponsor, Push Security

Full Transcript

[David Spark] When you have an incident and you’re engulfed by the stress that lasts more than a day, how do you manage and deal with it? And not only how do you manage your stress, but how do you manage everyone else’s?

[Voiceover] You’re listening to Defense in Depth.

[David Spark] Welcome to Defense in Depth. My name is David Spark, I am the producer of the CISO Series. And joining me for this very episode, you’ve heard him before, you’re going to hear him again right now. It’s Geoff Belknap, he’s the CISO of LinkedIn. Geoff, say hello to the nice audience.

[Geoff Belknap] Hey, David, and hey, everybody else. I’m super excited about the episode we have today.

[David Spark] I’m very excited about it as well. I do want to mention to everybody, if they’re not aware, we’re available at CISOseries.com. We mention all our sponsors’ names by spelling out their dot-com address. I never spell out my own, and there I go, I spell it out. Speaking of that, our sponsor for today’s episode is Push Security – find and secure shadow SaaS apps.

They’ve actually done some pretty impressive research on this very topic, and we’re going to talk about that and what Push Security does a little bit later in the show.

The topic for today’s episode came as a result of a conversation I had with today’s guest, that is Tim Brown, CISO of Solarwinds. I met Tim earlier this year and he told me that during the first month of the Solarwinds incident, he lived across the street in a campus apartment, he was working 6:00 AM to midnight every day and lost 30 pounds in less than a month.

And I thought who better to talk about dealing with unbelievable stress than our guest? You can’t really talk about it until you live it, right, Geoff? Part of the stress though is not only what you’re dealing with, but also the stress of all the people around you. You need to manage all their stress as well.

So, Geoff, I don’t believe you’ve had a major incident as big as Solarwinds, and we’re going to bring in Tim in a second, but during a major incident which stress is more difficult to manage – your own, or those around you?

[Geoff Belknap] I think it would be really hard to pick one or the other. I think it’s hard to manage your own stress. You then have the stress of ensuring that your team feels safe and has the sort of like mental acuity and safety needed to work on a very challenging problem. You got to talk to all the stakeholders, whether they be internal or external.

You’re dealing with people saying crazy things, potentially in the press. I don’t think there is one easy part about this.

[David Spark] Not one easy part. Ah! I’m going to actually ask our guest if there is an easy part of this at all. First I want to introduce our guest Tim Brown who is the CISO of Solarwinds. Tim, thank you so much for joining us for this conversation.

[Tim Brown] Absolutely. It’s great to be here.

[David Spark] As I understand it, just to reiterate, this was about two-and-a-half years ago this happened. Am I getting my timing right, about two-and-a-half years ago?

[Tim Brown] December 12th, a date I will never forget.

[Geoff Belknap] Yeah, sounds like that rolled right off the tongue for some reason.

[Laughter]

[Geoff Belknap] Some trauma related to that perhaps.

[David Spark] This event was so public, still is referred to as a you don’t want any “Solarwinds” incident. You’re still around, you’re still there. You went through massive stress. You’re still dealing with issues today, as I understand. Walk us through, we just want an encapsulation, what were the levels of stress you were dealing with?

And I also want to note something that Tim dealt with that others don’t – public abuse. You got a lot of it.

[Tim Brown] Yeah. Oh, absolutely.

[David Spark] Walk us through it. Like, how you managed that and then I want to throw out, you’re a married man, how your wife dealt with you being a mess.

[Tim Brown] Yeah. So December 12th, really a surprise, right? That’s the biggest thing you have to understand. This is not what anybody’s expecting and it comes out of the blue, right? You get the call in the morning that says, “You shipped tainted code.” Right? Our worst nightmare from a CISO’s perspective.

You find out that it’s going to get announced the next day, right? It’s already been leaked to the press and…

[David Spark] Hold it, wait. Did you sleep even one minute that night?

[Tim Brown] Not at all. And we attempted… This was the middle of COVID too, so we’re right in the heart of COVID, everything’s going on, we’re working remotely on Saturday. Sunday we all come into the office. Yeah, just COVID be damned, right? We’re getting together, getting into war rooms, doing what we need to do.

They brought people in to test us for COVID, but yeah, just really crazy trying to get through the first days, the first weeks, the first months, and now into the years.

[David Spark] Did you try to shut out all news and all social media as much as possible?

[Tim Brown] I actually did, right? I heard some of kind of the glimpses of what was being said, but I definitely did not spend a ton of time looking.

[David Spark] And please tell me you had people around you knowing better not to say, “Do you know what they’re saying about you online?”

[Tim Brown] Yeah. “Did you see the 60 Minutes’ article? Did you see what CNN said? Did you…?” No. Because at that point in time, it’s all bad, right? You have to remember – during one of these incidents, the people, the direct company, doesn’t really speak to the press very much, right? We speak through lawyers; we speak through our marketing people.

So what news they get is usually kind of incorrect or it’s just explosive or it’s people who haven’t been with the company for six years.

[David Spark] Well, how long had you been with the company prior to the incident?

[Tim Brown] So, I’d say about five now. I’ve been here for the six years total, so I guess it was four years at that point. And actually I was planning to leave. We were splitting the company in two.

[David Spark] Prior to the incident you were planning to leave?

[Tim Brown] Yeah.

[David Spark] Man, your timing was bad! [Laughter]

[Tim Brown] Yeah. We were splitting the company into two and I was planning to go over with the other company. And already planned and then this came up, so there was no way I could leave in the middle of it, of course. So, yeah, those plans got kind of scrapped.

[David Spark] And then two-and-a-half years later you’re still there?

[Tim Brown] Yeah, yeah, absolutely. And that just says that you have to get things through, you have to really make sure that the company’s in a good spot, you need to work through it, you need to make sure everything works.

[David Spark] Who was – and to our audience, we’re going to make this a slightly longer episode than we normally do – who was your best ally, your best support, the thing that with everything that was negative that was bringing you down and exhausting you and beating you, this was – and it could have been your wife, it could have been a supervisor or CEO, whomever, or it could have just been a colleague, whatever – who were those people and what did they do to make things good for you?

[Tim Brown] So, a combination, right? The number of goodwill messages that I got from my community, from CISOs around the world, was incredible.

[David Spark] Oh, that’s good to hear.

[Tim Brown] I got, “Tim, you got it. It’s okay. Who better to work through this than you? It will get better.”

[David Spark] First of all, is there anything anybody could do for you or no?

[Tim Brown] Not much except say, “Hey, if you need anything, I’m here. If you need to yell at somebody, I’m here.” Just that, “You’ve got it. But if you need to go scream, I’m there.” Then family, of course.

[David Spark] By the way, did you take advantage of that a few times?

[Tim Brown] I did. I actually did, yeah.

[David Spark] [Laughter]

[Tim Brown] Because at some point you just have to yell, right? It’s like, oh, crazy. But those little glimpses of support – our board. Our board wrote me notes of support, “Tim, you’re doing a great job. Please keep it up.”

[David Spark] Just in our big moments of our life that are very difficult, like through death and things like that, these small things that people do, they don’t realize the huge impact they make and the fact that literally that those small comments those people made to you, you still remember today.

[Tim Brown] Absolutely. I moved my office the other week and I found a card from my CEO, and we had a CEO change at the same time, so the new CEO. And he was just like, “Thank you. I know it’s been hard. I know we’ve been at it for a month and thank you for your support. Thank you for going forward.” Just little notes of thanks and encouragement go such a far…

More than anybody realizes. So if anybody you know is ever in one of these situations, reach out, don’t think it’s silly. Reach out, show them support. They will appreciate it.

How do we make this everyone’s concern?

8:27.871

[David Spark] Steve Zalewski, who is the other co-host of this show, Defense in Depth, said, “I had a SOC manager whose philosophy was ‘during a major incident, everyone in IT worked for him.’ It was a way of surging the size of the team during those first hours/days of critical investigation. It effectively shared the stress across a broader team rather than focusing it on a few key individuals.

It allowed the team to function at full capacity longer.” Yaron Levi, CISO of Dolby Laboratories, said, “Never do it alone! Bring the entire team together. If possible, send a third of the team home at the beginning of the incident so you can rotate people in and out as the incident continues to roll.

At least one of these people should be someone who can rotate with you.”

And lastly, Justin Bumpus who’s the CISO of Medalogix said, “I have used the tag team approach rather than the all-hands-on-deck to prevent responders from burning out during the incident response. The important thing is having good handoffs and coverage in the impacted areas through the event.” Geoff, I’m going to throw this to you.

This is all good philosophy, I’m all aboard. Is it conceivable? Can you do this? Can you make the whole IT team yours? Can you literally divide the team in three and have 24-hour rotation going on?

[Geoff Belknap] I think you can do all these things, but I think the most important thing to keep in mind is it’s not so much the rest of the company works for you, but an acknowledgement of as much as our job is hard and sometimes we conflict with other parts of the org, security is everybody’s job.

Nobody at the company is like, “Oh. That sounds like a you problem, man. I got to do my own thing.” What I find in incidents is most people are more than willing to help out and appreciate that this is a very challenging issue to work on and that you need help. People need help to work on big issues.

I think to Yaron and Justin’s point, you have to set some boundaries. Security people – and I mean this I think as a sort of personality trait – if you’re a security person, sometimes what I see as a common trait here is you desire to be in the service of others, you want to help people. And if you take somebody that inherently believes that it is good and valuable for them to help people and you tell them, “Don’t burn yourself out,” they’re going to be like, “I absolutely won’t, but I’m just going to work these next 50 hours straight to work on this issue.” You have to sort of enforce some structure and give them an opportunity to not burn themselves out.

They’re still going to work super hard, they’re still going to work long hours, but you have to make room for them to be useful to others, not a peril of their own health.

[David Spark] All right. I’m going to throw this to you, Tim. Did you have other members all of a sudden become members of your team, and were you able to divide up your team to have the sort of 24-hour coverage?

[Tim Brown] Yeah. So, one of the things that we did right, we had DLA Piper who’s the largest legal firm in the country, they have a great cyber team. And they came in and acted as a quarterback because we had streams that we had to go for across the company. We had a stream for engineering – figuring out how it happened, what they did, how did they get in, where did they affect things.

IT, right? Same stream. A marketing stream, escalation stream, support stream. So all these streams they helped coordinate. Because normally for normal incidents, my team would take incident commander, right? We would be able to do that. But in something this large, we needed to be able to spread the load across many.

We had people on the phones answering support calls from anybody that had skills.

[David Spark] And was there one source of truth? Because I got to imagine with that much, all of a sudden there’s going to be a ton of different stories being told. How did you handle that?

[Tim Brown] Documentation and publicking, making public knowledge, synced up with SISA to get our information on our website equal to what they had. And then our messaging, right? And FAQs for anybody that was on phone. Some people just wanted to call and say, “What version do we have? Were we affected?” So we were able to get a talk track and enough information to be able to put non-support people on phones, to be able to get the load down.

All of those things helped to spread the load across the company. Now, this is an incident of unprecedented scale, right? It’s not often you’re on the cover of every news story. So, with those, we just brought everybody together and then kind of coordinated every night around nine o’clock at night, had all of the groups come together and discuss where we were at.

Yeah, that just ends up in late nights and getting everything done. Did I say that I limited my stress because of that? No.

[David Spark] [Laughter]

[Tim Brown] But we were able to accomplish what we needed to do.

What’s going on?

13:24.307

[David Spark] Shweta Kshirsagar of Myntra said, “Give regular and to the point updates to leadership before they ask, communicate periodically of such updates to leadership.” Edwin Covert of Bowhead Specialty said, “Managing expectations is critical – being realistic with timelines and due dates.” Ooh, I’m going to want to get to this, yes.

And Tiago Rosado of Gravitee said, “First the incident manager sets the pace and rate of communication to the C-level to avoid the usual ‘are we there yet’ kind of time-wasting stress-creating attitude.” And Justin Bumpus, CISO of Medalogix, again said, “Ask questions like ‘What do you know’ or ‘What can you tell me’ rather than ‘Tell me what happened,’ setting the expectation that incidents are complex, and it’s okay to not have all the answers.” I’m actually going to start with you, Tim, on this one.

It seems like you had a kind of a controlled environment because of DLA Piper and then you had these 9:00 PM end-of-day sort of rehashes. So, it was kind of already built into your structure, yes?

[Tim Brown] Yes. We had a structure in place, we had people to call. They knew better how to do it than us though. Absolutely, no question. Their concepts were how often we get together. Their concept was put the leaders of each group, the CTO in front of stuff, the CIO in front of one working group.

And then structure that, “Hey, we’re getting together every night. Here’s what we’re going to do. Here’s how we’re going to discuss it. Here’s what our open items are.” So, the coordination, having them in place as an independent external kind of third party, helped. They were the ones that kept everybody kind of calm in the mix and said, “Okay, we got these actions to do.

Let’s get them done.” And then setting the expectations of time, setting expectations of discovery, setting expectation of disclosing what we know and what we don’t know because that was a big important part.

[David Spark] And were you able to, like what Edwin Covert said, direct, “This is when you’re going to get it, this is the best we can do,” if they start pushing, “No! We need this information now!” Did you ever have any of those incidents or no?

[Tim Brown] Yeah, absolutely. We’re trying to figure out what to do, we’re trying to get information, get answers, answers for customers, answers for countries. Yes, countries call you.

[David Spark] Oh, right. But was there a buffer? Like did DLA act as the buffer? There’s certain calls you don’t want going directly to you. Did they act like the buffer?

[Tim Brown] They did. And others, right? So, we had business escalation. We had business escalation, we took those, we found out the answers we could. We published as much as we could, so our published answers were our general answer and then when somebody asked things outside of published, we would try to work on it and figure it out and get back to them.

But you should always publish what you do know and what you don’t know. It’s okay not to know things at first. But you want to be conservative in your answers, you want to be able to not tell somebody 10 people are affected and then 1,000 people were affected, right? You want to be able to go high numbers.

But publish what you know and then go from there.

[David Spark] All right. Geoff, I’m going to throw this to you. I got to assume the whole thing about managing expectations is literally having good relations before an incident ever happens, right?

[Geoff Belknap] Yeah, I think so. You have to have that relationship beforehand because when there’s an incident – and let’s be clear for a second because I think Tim already said this. All incidents are not this company existential incident. There’s a broad spectrum of maybe you just had a phishing attack or maybe somebody’s sending SMS messages as the CEO to people.

All the way to maybe you had a massive black swan event. Sometimes the CEO or the CFO or the general counsel will call you 30 seconds after finding and go, “Please tell me everything that happened.” That is a moment where that person has a valid question. To do their job, they probably do need to know all that information.

And you would love to give it to them, but you have to use the relationship to set expectations that, “Hey. Here’s what’s going on. Here’s where we’re at. I’m going to update you every hour,” or 6 hours, every 24 hours, whatever it is, to help them understand how they can get information from you but also, to Tim’s point, to help them be a buffer so that they understand you’re working very hard.

And shockingly, when you’re on the cover of every news organization about this incident, people don’t usually treat the story with like, “Hey, but they’re working on it. Why don’t you check in about three weeks? They’ll have everything you need to know then.” They want to know right now and they would like updates immediately, and it doesn’t work that way.

You cannot find out all the answers you need to know and all the dependencies you need to chase down in an hour or 24 hours. Unlike TV, it takes a lot of time for a lot of talented people to investigate these things. That relationship helps cascade out and helps, honestly, defer the stress. Because now you’re having people help as a buffer.

You’re helping them understand and communicate out the seriousness of this and how much effort it takes. And that’s what you want. You want room for the people that need to do the hard work to do it without feeling all the stress that they could possibly feel in that moment.

Sponsor – Push Security

18:51.544

[David Spark] Before I go on any further, I do want to mention our sponsor Push Security – find and secure shadow SaaS apps. Now, we all love SaaS apps, whether you’re in security or not. It’s great that we can try out a new SaaS app before we buy, and it’s obviously an approach that’s working very well for the software vendors.

But when you look at this with your security hat on, every time one of your employees signs up to a new app, they’re creating a new identity, a new front door that could potentially give attackers access to sensitive corporate data. All these shadow identities on SaaS apps are giving attackers a huge new attack surface that’s largely unmonitored.

So Push Security’s own tool gives security teams visibility of all the SaaS apps and accounts their employees are using, including any shadow IT and identities. It then finds security issues that are putting your data at risk. Things like, well, compromised and weak passwords, MFA not being used where it could be, as well as reused credentials and shared accounts.

Now here’s the cool bit. Rather than giving your team a long list of issues to just go out and fix, Push automatically engages with the SaaS users in your business and gets them to fix the issues themselves by providing them with super simple guidance anyone can follow.

So, what does this mean for your business? Well, it means your employees can carry on using the tools they need to do their jobs and you don’t have to compromise on security or get swamped with a load of issues that need fixing. So you can have your cake and eat it. Sounds pretty awesome. So go to their website, head to pushsecurity.com to see if they could help you find and secure your shadow identities.

Plus – and this is a bonus – check out their research looking at the SaaS native attack techniques, 30 of which are mapped to the kill chain. That’s pushsecurity.com.

No one said it was gonna be easy.

20:56.050

[David Spark] Jerich Beason, CISO over at Capital One, said, “In a past role, during one incident we ordered breakfast, lunch, and dinner for the core responding team and their families until work…” and the families, I thought it was good, “…until work got back to a sense of normalcy, and we gave the spouses an option of a cleaning crew or a spa day.” Did your company offer all that too?

[Tim Brown] No cleaning crews or spa days, no.

[David Spark] Did they offer meals?

[Tim Brown] Absolutely. We over-ate for the first month, no question.

[David Spark] And yet you lost the weight. That is the amazing thing of this all. That is, by the way, an amazing workout plan. Incredible.

[Tim Brown] Not one I would recommend to anyone, but it does work.

[Geoff Belknap] Being CISO may be more effective than Ozempic but probably not the route for most people.

[David Spark] No. All right. Let me finish this quote. Jerich Beason, he goes on to say, “A big part of the stress is how life at home is affected and just wanted to do a small part in acknowledging and limiting that impact.” That is pretty powerful, I thought that’s impressive. Simon Goldsmith of OVO said, “Teams under extreme pressure become frustrated, frantic, and fractured.

The antidote/alternative is the three C’s – have the team commit to calmness, control, and connectedness. We have to provide the tools to enable this to happen but having a vision of what kind of mindset you want your team to adopt when it really matters.” So, Geoff, this is really interesting stuff that both Jerich and Simon talk about in that we’re getting into the how others are reacting.

We talked mostly about ourselves, but how does other people’s stress and even how it trickles into the home life as well, what have you seen?

[Geoff Belknap] This is great, and I think what I’ve seen is you have to sort of address this in if you think about Maslow’s hierarchy. The first thing is to address all your responders and your own personal needs, right? And I think sometimes this is as easy as being clear to people like, “Hey, you have a hard job.

We have an incident now. You are not fired because of this incident; your job is safe. We need you now to respond to this thing.” Even though you might feel a deep sense of guilt or stress because you’re on the security team and ostensibly it’s your job to help prevent these things. We need people to understand this is a natural part of security.

You can’t be perfect all the time.

I think then from there it extends out. You as the CISO or as the security leader are the first line of support for your team that’s responding to this. You need to make sure that they have what they need, that every person in sales or marketing is not calling them individually asking for things, and then you extend out.

What makes us healthy, happy human beings? It’s our support networks. It’s the people, our loved ones, our families, our friends, our neighbors. And those people, you need to put together strategies to help them understand what’s going on and understand what’s going on with you. And you’re going to find out your neighbors and your friends are going to offer help and support.

They’re not going to be mad; they’re probably not going to be pestering you for information. But I think Jerich’s offer is fantastic. Whatever you can do to stabilize that support system is like a multiplier for the impact that your team that’s working on this can have. Genius.

[David Spark] Tim, I know that you mentioned that your CEO gave you a really nice vacation for you and your wife, which I want you to mention in a second, but tell me about that and also tell me about how the stress of your team. And some people handle things better than others, how did you manage that while you are visibly stressed?

Because, I mean, you’re losing a lot of weight. That cannot be hidden in this process. So, did you have certain members that just really had a tough time with stress and how did you manage it being clear that you were not handling it as best… I mean, I guess you were handling it as best as you can, even though you’re losing all this weight.

Maybe that was the way you were dealing with it.

[Tim Brown] Yeah. I guess I internalized a lot of things. That’s probably what ended up having me lose weight. But on the outside, it was calm. On the outside, it was just get the work done. On the outside, it was fix the problems as they arise. The outside was take calls with customers, have some yell at you, have some support you, and just be kind of prepared for that.

And then team members, try to get them to not go through the model that I was going through.

[David Spark] Did you see certain team members maybe spinning a little too much out of control?

[Tim Brown] Absolutely.

[David Spark] How did you right that ship with them?

[Tim Brown] Yeah, just have them relax, understand we’ll get through it, and just talk through things. Get them to take some time off, get them to not get as kind of internalized of… A lot of people are hurt, right? You have to understand that people worked on this product, they loved this product.

Just like our customers loved the product, our employees loved the product. And to have somebody kind of break into your house and mess with your stuff is a huge… Just think of the same thing from when you’re broken into from an apartment or a house, how internalized that gets. So, sometimes it’s time, sometimes it’s talking, sometimes it’s just letting people know, “Hey, no, take the day off, take the weekend off.

It’s okay. Right? We will get through it, we will work through it, we will get better on the other side of this.”

[David Spark] And there was some kind of light at the end of the tunnel, yes?

[Tim Brown] Yeah, absolutely. About what kind of the company did, right?

[David Spark] Tell me what they did for you and your wife.

[Tim Brown] Yeah. So, for me and my wife, they basically said, “This was…” We probably got to June, right, and said, “Tim, you need to take some time off and go wherever you want and we’ll pick up the tab but shut your phone off and just take your wife and go away for a week.” That just did a lot.

Instead of giving you money or a bonus or those types of things, forcing me to just shut down, right, was a great effort.

[David Spark] Yeah, I don’t want any graphic detail but tell me what did that week that you were off do to you? How did your body change during that week?

[Tim Brown] Yeah, just refreshed, right? I read books. I just sat there and didn’t do much, right? We just hung out. It wasn’t like a busy week and it just gave you back the energy that you had, put things into focus for you, just helped to really… We did focus on how do we relieve kind of stress, right?

Did a spa thing, did a massage, all the things that you kind of hear to just, “Okay, relax.” Right? And not worry about work and just put that behind you for a little bit.

[David Spark] Were you able to truly not think about work during that week?

[Tim Brown] I tried. I tried.

[David Spark] And did you turn off your phone? Did you answer any messages from work during that week?

[Tim Brown] Yeah, I did.

[Laughter]

[Tim Brown] But I tried.

[David Spark] It’s so hard not to, I know.

[Geoff Belknap] You get credit for trying, yeah.

[David Spark] You did your best. Your wife was supportive though, it sounds like, all through this.

[Tim Brown] Oh, extremely, yeah, extremely. And that’s so important, and to have family, have friends, have support, and just tell you, “You got it,” and then… Yeah. I would get home and I’d been talking for 20 hours, so she said… Just to sit there and be quiet and her to understand that I can’t talk to anybody anymore.

[Laughter] I’ve been talking for 20 hours. I just need quiet, right? So, she got that and very, very supportive.

[David Spark] One more question about in the very beginning, it’s very intense. December 12th is a very intense period of time. When was the first moment you and your team got a moment of relief in the sense of you either went for a drink, you went bowling, you had dinner, you did something. How long did it take before you did something like that?

[Tim Brown] So, Christmas Day we ended up taking off, so that was the biggest kind of break we had.

[David Spark] So you didn’t have anything for two weeks?

[Tim Brown] No, no. We were completely at it. The timing was wonderful in that during Christmas week, the world slowed down, right? And the world slowed down helped us catch up, right? So we took Christmas Day off but not the week before or week after or anything like that. But that allowed the world to slow down, for us to catch up, for us to get our messages, for us to understand more, to have more solidified information for people.

That timing just couldn’t have been better because once the world came back, we were in better shape. Our queues had slowed down, our information was out there, we were able to get a tremendous amount done during that slower period for the world. So it would have been much longer if we didn’t end up getting there.

What do most people think it is, and what’s the reality?

29:46.011

[David Spark] So, Shawn Bowen, CISO of World Fuel Services, who has served in the military and also I believe he’s a Reservist as well, said, “Many people and organizations simply haven’t had enough training or aren’t well-suited for incident management. If you’ve only planned once, your ability to observe, orient, decide, and act will be slow and the stress mounts because of the requirement to process so quickly.

The more scenarios you’ve observed, the more comfortable you’ll be in any situation. In professions such as the military, firefighting, and law enforcement, trainees go through rigorous training that includes handling stressful environments. The cybersecurity field doesn’t necessarily have the same level of crisis response requirement and training.” That is very true, Geoff, and how much do you have and how much would be ideal I guess is the situation.

Because I got to imagine that no one’s at an ideal level.

[Geoff Belknap] No one’s at the ideal level. Well, honestly, that’s not true. I think the Department of Defense, military.

[David Spark] Yes. Because that’s what they do.

[Geoff Belknap] That’s what they do, they understand this inherently. There’s a lot to be learned. But I think if we just go back for a second, this is a weird career path to be in, whether it be individual contributor or security leader. One of the things that Tim mentioned a little while ago, it’s like someone broke into your house and that really disturbs you.

But this is also the only career path where people go like, “Why did you let someone break into your house? What’s your excuse for this? How dare you?” while you’re picking up the pieces of this traumatic thing that happened to you, that you didn’t want to happen. And this happens regularly. All the time.

Even little, teeny things, people are sort of victim-blaming you for having this issue happen.

And I understand where that comes from a little bit but it definitely is something that we as an industry have not standardized on yet. There are a zillion certifications that I can go get and send people to right now about how to do the technical incident response. I can’t think of a single certification or a process I can put people through to manage the emotional and mental part of incidence response.

And that’s something where we really need to invest more heavily in, especially because – this is sort of the deep, dark secret of InfoSec – alcohol and drug abuse are definitely a significant part of how some people cope with these things, and that’s not healthy for us as a long-term practice in the industry.

[David Spark] Good point. So, let me ask – prior to this incident, Tim, how much training did you guys have?

[Tim Brown] Not training on this level of an incident, right?

[David Spark] No one could, no one could conceivably.

[Tim Brown] Yeah. But on general incidents, we handled lots of general incidents, lots of little things, and you can handle those pretty much by a book, by a playbook. Yeah, and they’re stressful, sure, but they’re manageable and you get how to do them. One of the things that people that do deal with these big incidents, they’re the folks like DLA, CrowdStrike came in as a partner, others, KPMG’s team came in as partners, they have more experience at the big, at the really ugly.

They helped by taking away things. So, they helped by…

[David Spark] Give me an example. What are the things they took away? I’m interested in that.

[Tim Brown] Yeah. So, they took away quarterback, right? I couldn’t be incident commander for this thing, I was spread all over everywhere. If I tried to be incident commander, it would have been terrible.

[David Spark] That’s a good point. Let me pause you on that right there. When you are so in the mud, you don’t have the viewpoint of someone outside, especially of you’re not seeing how the world’s reacting right now. We do, we have to control it. I realize how important and critical that is, and that must be an enormous relief off your back that you don’t have to be quarterback.

[Tim Brown] Absolutely, absolutely. Because being quarterback is hard. Program managing stuff is hard. Getting people together and organizing what the right cadence for a thing is. Okay, how do we deal with all of these people that are going to talk to us? All I had to do was show up, right? I had to show up to these things.

[David Spark] And by the way, let’s pause on that a second. A lot of people [Inaudible 00:34:05] like when they say they want to help, a lot of times, especially during something like this, people don’t know what they should do, and if there’s someone that just says, “Do this,” people are, like, they’re so happy that someone told them the very thing to do so they don’t have to think about the thing to do, right?

[Tim Brown] Absolutely. “Get on the phone, here’s the playbook, answer some of these thousands of calls we’ve got,” right? “Oh, great. I can help.” Absolutely, right? Just that level of maturity, that level of practice that they had. “Okay, press is going to be doing this to you. Here’s what you do with it.” Right?

“Oh, this is the messages you want to send, FBI’s going to want this, SISA’s going to want this, other people are going to want this.” So show up to the meetings and give them what they want but you don’t have to organize it, you don’t need to structure it, you don’t need to do all of those things around the outside.

It’s a huge thing for something that’s competent that’s done it before to help take it away from you. And again, this is for the big ones, right? This is for the ones that are just so large that you can’t do on your own. But that helps control the stress, that helps say, “All right. Somebody else is helping on this side, then others are helping here, others are helping here,” and we’ve spread the load across many.

Closing

35:20.774

[David Spark] That is awesome. Well, thank you so much for your time, Tim. I was very much looking forward to this recording and this definitely paid off, I’m thrilled. Now it comes to the portion of the show where I ask you which quote was your favorite. There’s so much good material in today’s episode, even before either you or Geoff spoke, I just love what the community said.

Which quote was your favorite, Tim, and why?

[Tim Brown] So, in the area of no one said it was going to be easy, Jerich Beason, the CISO for Capital One, really focused on how we respond as a team, how you get support from your team, how you extend that team support to others. Now one of the things you remember that you may be going through a hugely stressful time but your customers are as well.

And that’s one of the things that, as I often said, my Christmas was ruined but many of our customers’ Christmas was ruined. They’re investigating, they’re trying to figure stuff out. So don’t forget them and lean on your family, lean on your friends, lean on others to help you through this and it will get better.

It is a matter of time but it will get better, so it will end. And that’s one of the most important messages that people sent me, “Just hang in there.”

[David Spark] So when it’s going six months, they’re still believing you?

[Tim Brown] Yep, yep, absolutely. I’m still going to believe it’s going to get better. The other day I was at a conference, or just last week, and somebody came up and said, “I saw your presentation at a conference somewhere and I have that as mandatory listening to my employees. That’s how much you’ve taught people about kind of what goes on through an incident.” That says, “Good.

I’m helping the world here.” So that’s the other part that’s really great, if you can help the world.

[David Spark] The shining light is that this horrible incident for you has turned out to be a benefit for others and I hope actually this episode as well, has turned as well. I’ve loved it. Geoff, your favorite quote and why?

[Geoff Belknap] I don’t know how I follow Tim’s favorite quote here, but I’m going to use a very easy one to just close this out easily, which is Edwin Covert’s from Bowhead Specialty, “Managing expectations is critical – being realistic with timelines and due dates.” This is the core of managing large incident response.

You just have to tell people what they should expect and when. Because the immediate expectation is like what they’ve learned from a phishing campaign or something like that like, “Oh, we were phished, we took care of it, there was this many people that were involved,” blah-blah-blah. For a major incident, you have to reset expectations with everybody.

This is not going to be, “There’ll be an email in an hour explaining everything.” This is going to be, “We’ve got to get more rooms [Phonetic 00:37:57] in, we’ve got to bring the customer communications people in, we’ve got to bring in comms and legal,” and blah-blah-blah-blah-blah. You just have to set those expectations as a basic starting point to kicking off this major response effort.

[David Spark] Excellent. Well, couldn’t be happier with this episode. Thank you so much. Tim Brown, still the CISO over at SolarWinds, kudos to you. Geoff, thank you so much as well. And I want to thank our sponsor as well – Push Security, available at pushsecurity.com. Find and secure shadow SaaS apps and in addition go to their site to check out their research they’ve done on SaaS native attack techniques.

Tim, any last words you’d like to say? By the way, are you hiring people who are good incident responders over at SolarWinds?

[Tim Brown] Yep. Always looking for good folks, no question. But nope, great episode, great to talk about kind of [Inaudible 00:38:50] stress and the big message is that no matter what you’re going through, do it right, be honest, be humble, own it, and you will get through it and you try to look for learning opportunities for others during them.

[David Spark] Very good point. I want to thank our audience as well, we greatly… Especially on this one. God, the audience’s comments on this were just fantastic. Thank you so much for your contributions and for listening to Defense in Depth.

[Voiceover] We’ve reached the end of Defense in Depth. Make sure to subscribe so you don’t miss yet another hot topic in cybersecurity. This show thrives on your contributions. Please write a review, leave a comment on LinkedIn or on our site CISOseries.com where you’ll also see plenty of ways to participate, including recording a question or a comment for the show.

If you’re interested in sponsoring the podcast, contact David Spark directly at David@CISOseries.com. Thank you for listening to Defense in Depth.