We explore the world of dishonesty in cybersecurity. Practitioners know that marketers will stretch the truth, but how far are we willing to let that go? Isn’t this industry built on trust? Can cybersecurity continue to thrive if we can’t trust each other?
This week’s episode is hosted by me, David Spark (@dspark), producer of CISO Series and Mike Johnson. Our sponsored guest is Anna Belak (@aabelak), director of thought leadership, Sysdig.
Got feedback? Join the conversation on LinkedIn.
HUGE thanks to our sponsor, Sysdig
Full transcript
[Voiceover] Best advice for a CISO. Go!
[Anna Belak] Pick vendors that are great partners in addition to selling you a good product, especially when you’re dealing with emerging technologies and new operating models.
[Voiceover] It’s time to begin the CISO Series Podcast.
[David Spark] Welcome to the CISO Series Podcast. My name is David Spark, I’m the producer of the CISO Series. My co-host for this episode, since the very first episode, is Mike Johnson. Mike, are you there?
[Mike Johnson] I’m here, David.
[David Spark] He’s there.
[Mike Johnson] I’m here, it’s true.
[David Spark] I can see him. His lips are moving.
[Mike Johnson] It’s true. Lips and voice are working in concert.
[David Spark] Yes. We appreciate that. There’s no lag. Your audio and video are in sync.
[Mike Johnson] I cannot tolerate lag at all. That is not okay.
[David Spark] Even when it’s off just a couple of frames, it is the most disconcerting thing.
[Mike Johnson] It is so distracting. Just like the slight delay, you just focus on it, and it draws your attention, then you just…
[David Spark] Can’t think about anything else.
[Mike Johnson] All you can think about is what are those packets doing.
[David Spark] Yeah. I don’t think I’m thinking about what are those packets doing, just unnerved by it.
[Mike Johnson] Yes.
[David Spark] You know what it is? It’s the equivalent of the uncanny valley.
[Mike Johnson] Oh, very much so. It’s one of those. If it’s very far apart, you can deal with it. But the fact that it’s really close, it’s almost there, it’s in that uncanny valley. Yes. It’s a really good way of thinking about it.
[David Spark] You know, we’re available at CISOSeries.com. We’re doing a lot to this website these days actually because we found some errors and problems with it, but we’re working on it. No need to get into the details of it because it’s not interesting even the slightest. Our sponsor for today’s episode is Sysdig, a brand new sponsor for the CISO Series, and guess what? They’re in the area of cloud security, an area that many of our listeners are quite involved in. Did you know that, Mike?
[Mike Johnson] I’ve heard of it. I’ve heard of this cloud security thing.
[David Spark] Well, it’s important. Guess what? They brought our guest today who’s going to help us a lot in that discussion.
[Mike Johnson] Awesome.
[David Spark] But before we jump into that, today, the day that we’re recording is June 1st, 2022, which is important to the two of us and is important for me doubly. One is my son’s birthday is today.
[Mike Johnson] Happy Birthday.
[David Spark] That’s good. But secondly, you and I four years ago released our first episode.
[Mike Johnson] Four years.
[David Spark] Mm-hmm.
[Mike Johnson] Wow.
[David Spark] I never ever want to listen to that first episode again.
[Laughter]
[David Spark] I can’t imagine how much that thing stinks.
[Mike Johnson] It was a different time, David. We were different people.
[David Spark] We were different people, that’s true. I want to do a huge shout out to Dwayne Melançon who was on our very first episode. Anyone who agrees to do the first episode of anybody’s podcast is essentially taking a bullet.
[Mike Johnson] Yes.
[David Spark] Because it’s going to stink, and I am thanking Dwayne a thousand times over for doing that.
[Mike Johnson] Thank you, Dwayne.
[David Spark] So thank you. Dwayne has kind of exited. He runs a technology group now, but he’s no longer in cyber-specific anymore.
[Mike Johnson] Well, he was willing to take a shot on us the first time, and here we are four years later.
[David Spark] Yes. So, we greatly appreciate that, and yes, we have improved. If any of you want to be so brave as to listen to the first episode, go ahead. But don’t let that reflect on the quality of the show we do today. By the way, those first episodes I edited, another reason they stink.
[Mike Johnson] So, listener – you have been warned.
[David Spark] All the warnings have been out.
[Mike Johnson] Disclaimed.
[David Spark] All right. Let’s bring in our guest today. Enough of this nonsense. I am so thrilled to have our guest on today. From the East Coast, from where I was born, my area, the neck of the wood not far from where I went to high school. It is the director of thought leadership for Sysdig, our sponsor – Anna Belak. Anna, thank you so much for joining us today.
[Anna Belak] Thank you so very much for having me.
Walk a mile in this CISO’s shoes
4:03.320
[David Spark] What are the questions a CISO should be able to answer? On Medium, Vicente Aceituno Canal provided a good list that’s in order that starts with, “Who are the clients of the information security team?” and it ends with, “What do you plan to do to improve the level of security or decrease risk?” Now, on the cybersecurity subreddit, the redditors appeared to be very cocky, saying, “These seem very basic,” and yes, they are, but often very hard to answer. We’ve addressed all these issues on the show at one point, but correct me if I’m wrong Mike, a good security leader should be able to answer these questions, know where to find the answers, or be able to tell how they’re going to produce those answers. I’ll ask you – how well do you think you can answer these questions and has your ability to answer them improved over time? If so, what do you attribute that to?
[Mike Johnson] So first, I want to say that I don’t necessarily agree with all the questions, but the real takeaway for me here was, reminder, is that often the easiest questions to ask are the hardest ones to answer.
[David Spark] Well, the classic, “Are we secure?”
[Mike Johnson] Exactly. Oh, my gosh.
[David Spark] Yeah.
[Mike Johnson] At least that wasn’t on the list. But what the author really seems to be focusing on and is something that I’ve had to actively learn is talking to the business about security. Is talking, frankly, to people who don’t do security day in and day out. That’s a skill that I think most security professionals when they’re coming up in the industry, that’s not one they’re developing. They’re developing their security knowledge, and that absolutely makes sense. But as you’re trying to get the business more and more involved in security, more and more bought in security, you have to meet them where they are. You can’t expect that they’re going to suddenly learn and become experts in security.
[David Spark] So, are these more a question for the non-CISOs rather than the CISOs? Like, you should be asking me these questions, and let me slowly reveal them over time?
[Mike Johnson] I hadn’t thought about it that way. I think that’s another way of looking at this list. That if it’s framed as, “These are questions that CISOs should be able to answer,” whoever’s asking them should be able to expect an answer on these. So, absolutely. Anyone should be willing to ask these questions. And quite often, it should be the folks who aren’t as steeped in security.
[David Spark] Anna, I’m throwing this to you. Let me generally ask you what you thought of the list of questions and how has your ability to answer them, find the answers, know how to get the answers, changed over time?
[Anna Belak] So, I’ll agree with Mike. I think that I don’t like the entire list.
[David Spark] Go ahead. Spotlight one of the questions you didn’t care for.
[Anna Belak] I feel like some of them are a little bit too granular, actually. Like how many vulnerabilities or things like that, right? And so this is kind of where I want to go and say, as a CISO, you should be taking care to enable your team to provide you with the useful information to communicate to the business and to make your decisions, right?
So, technical folks often, like Mike said, focus on building their technical skills, and they may not necessarily think about which metrics does the business care about, or which metrics does the CISO care about. And so if you are a CISO, and you’re thinking about that aggregation broadly, and you can go to your team and say, “Hey. If you’re a vulnerability management team, I don’t care how many phones [Phonetic 00:07:42] there are, but I care about our progress month over month, or our progress in terms of quick wins in the past month that we’ve improved the program,” or something like that. Because those folks won’t know, and the more you enable them and the more you train them to think this way, the easier your job actually becomes.
Are we having communication issues?
7:59.679
[David Spark] How much dishonesty do you find in cybersecurity? On the CyberSecProfessionals subreddit, a lot of people talked about dishonesty, and the majority of responses dealt with services promised but not offered, or a sales rep lying about what their products can do. One person noted that this post invites negative stories but from their experience, things aren’t so bad. And others mentioned that often stress is what’s driving the dishonesty. So, I’ll start with you Anna. What’s been your experience with the level of dishonesty in the industry?
[Anna Belak] I’m going to have a bias here because I used to be a Gartner analyst, so I’ve heard it all pretty much. I’m super-allergic to vendor dishonesty in particular because after you’ve heard enough briefings, it’s really easy to tell what’s true and what’s kind of pitch, right?
[David Spark] Can you give us an example of a, “I see this one coming,” and how you avoid that? Just a blatant lie. What do you hear?
[Anna Belak] So, there’s a lot in machine learning, actually, because it’s very popular right now, but a lot of machine learning claims are borderline false, or at least they’re very interestingly veiled in things that are somewhat true but not really true. A big one is false positive reduction or noise reduction which can be super-contextual, can be true in a specific scenario, it may not be true in your environment depending on what the vendor’s actually doing and what dataset they’re using and so on. So, it can be really hard to tell when the product they’re selling is very technically complex, right?
So, I have two pieces of advice which I’ve kind of developed over time, and I think make sense. One is you can look at open source vendors or open source base vendors because they tend to be more transparent, or at least you can see what the product’s built on. If you look at Falco, can see all the rules that are in there because it’s just in the GitHub repository. And the other thing you can do specifically for detection but for other tools as well is essentially use a testing framework like Atomic Red Team where you can actually simulate an attack or simulate an adversarial scenario against a tool in real-time and see if it actually works as advertised.
[David Spark] Mm-hmm. Mike, I throw this to you, and let’s think about when we started four years ago because anecdotally I see differences, and I see some things have never changed. What have you seen that changed? And let’s just use the period of time of our show – four years. What have you seen a change in the industry with regards to dishonesty over those four years?
[Mike Johnson] Over those four years, I actually don’t know that I’ve seen much of a change, but what I’ll say is I don’t feel like I’ve been outright lied to at any point.
[David Spark] There’s a lot of little mini ones that come in.
[Mike Johnson] There’s exaggerations.
[David Spark] And do you brush those off? You’re like, “Yeah, yeah. You would obviously say that, but I’m not going to buy it. But that doesn’t mean I hate your product or anything like that.”
[Mike Johnson] Again, that’s kind of how Anna was talking about it is it’s part of the game. Like, you recognize that those are going to come, and you take everything with a grain of salt. And recognize especially – again, as Anna said – that the more complex a thing is, the more difficult it is to understand what’s going on. And it could be very easy for you to have an assumption or an impression about what the thing does, and that was never even mentioned, that was never brought up. You can almost create these misunderstandings by jumping to conclusions yourself. And the more complex it is, the more greater a chance that is.
[David Spark] Anna, when you were an analyst, did you find any vendors really good? Because this issue comes up a lot on our shows. It’s that CISOs want to hear, “Great. Tell me what it can do but tell me what it can’t do,” and that when they admit what it can’t do, that buys a lot of trust. Did you find vendors were willing to do that?
[Anna Belak] I would say there definitely are a few. It’s more rare than not, I would say. But actually, those do tend to be the more trustworthy vendors. And the reality is they’ve often innovated in some particular domain, and they’re doing a fantastic technical job of solving some problem, and so they know their scope very well, and they’re very secure in that scope. Which usually sends a pretty good signal like, “We know what we’re doing.” So, there are a few of those, and they’ve actually been the ones that I’ve respected the most probably as an analyst, so there’s something to that.
[David Spark] I’ll just tell you honestly my own sales, because I have to sell sponsorship for this show, I’m always straight up like, “This will do this, this won’t do that.” I mean, I make it clear that everything we sell has a pro and a con to it, and no one thing does everything. Which I think when I talk to vendors, they’re appreciative of that as well.
It’s time to play “What’s Worse?”
12:43.423
[David Spark] All right, Anna, do you know how this game is played?
[Anna Belak] I believe I do, yes.
[David Spark] Okay. Two horrible scenarios. You’re not going to like either one, but you got to pick one. I make Mike go first. I like it when our guests disagree with Mike.
[Mike Johnson] It’s true.
[David Spark] No pressure. And I’ll just say that because there’s obviously pressure. All right. So, this comes in from Jason Dance of Greenwich Associates who has given us I’m going to say dozens, I don’t know if 24, but he’s definitely given us over the years at least a dozen, if not more. We’ve quoted many of his “What’s Worse?” scenarios. This has got kind of a tricky little twist to it I think, Mike.
[Mike Johnson] Okay.
[David Spark] What’s worse – a business leader that does not know to ask about security, or a business leader that knows to ask but doesn’t?
[Mike Johnson] Okay. So, in a way, what you’ve got is an ignorance versus negligence way of thinking about it. The first one just isn’t even thinking about it, the second one is actively trying to avoid it, is the way that I’m hearing this. And to me, that makes it pretty simple. The one who’s actively ignoring it, that’s going to be the worst in every situation. The first one – I can educate them.
[David Spark] No, no. This is the thing – you’re not going to educate them.
[Mike Johnson] No, no. But I can track them down, and I can talk security at them.
[David Spark] They’re still going to never know what to ask.
[Mike Johnson] And that’s fine, I will bring it to them.
[David Spark] You could bring it to the other person too.
[Mike Johnson] But they’re going to actively ignore it. The first one is just like, “Eh, I wasn’t even aware that this was a thing.” The second one is like, “I hate security, I never want to talk about this.”
[David Spark] Have you ever run into that person, by the way?
[Mike Johnson] No. No.
[David Spark] Thank God.
[Mike Johnson] Because I don’t think, as with almost all of our scenarios, neither of these is the real world. Most business leaders recognize that there is risk to their business, and cybersecurity is one of those, and so they’re not going to actively avoid it.
[David Spark] Hold it. But let me ask this. Cyber’s very much the front pages of the mainstream news now, there was a time it was not. When it was not the cover, were there leaders that were actively avoiding it?
[Mike Johnson] None that I ran into, but I’ve heard stories that those folks did exist, yes.
[David Spark] All right, Anna, I throw this to you. Do you agree or disagree with Mike?
[Anna Belak] I mean, the easy path is to agree, but I’m going to disagree just for the fun of it.
[David Spark] Way to go, Anna.
[Mike Johnson] Great, great.
[Anna Belak] So, I’m going to try to be generous to the second individual because maybe they’re actually avoiding it for some good reason, like maybe they don’t have the budget, and they’ve prioritized some other things because it’s more critical to their business.
[David Spark] They have other risks to worry about it.
[Anna Belak] Yeah. Maybe there are non-cyber risks that they’re concerned about, or maybe they’ve just had a really bad experience with leadership in the past, and they kind of need someone to help them advocate or liaise or something like that. So, the fact that they’re aware and perhaps even knowledgeable about security might be usable to my advantage to kind of persuade them to reconsider their risk appetite.
[David Spark] Now, here’s the other thing to throw out, and Anna makes a good point. They’re aware. They’re not asking the questions, so they’re aware and concerned but not asking the questions. So, that actually has another twist to it, think about it that way. A leader that is aware of the risk of cybersecurity but is not asking questions is at least aware of the risk and is putting that in their calculations versus the one that’s not even asking the question, that’s not even in their calculations.
[Anna Belak] They’ve accepted the risk.
[David Spark] Mike, does that change your tune? Yeah?
[Mike Johnson] I think you could look at it that way. I mean, and that might be a possibility. Anna made… The key thing that she said there is there are other risks that are perhaps more important than cybersecurity, and I can totally buy that as a reasonable explanation. But again, you have to kind of pick one of these and run with it, and I’m sticking with my answer.
[David Spark] You picked your horse.
[Mike Johnson] I picked my horse.
[David Spark] Old Glue Boy.
[Mike Johnson] Even if I don’t win, I picked my horse.
Please. Enough. No more.
16:55.691
[David Spark] So, Mike, our topic today for “Please. Enough. No more.” is cloud migration, but with a twist. How does one lead a cloud migration? Mike, what have you heard enough about enough with leading a cloud migration and what would you like to hear a lot more?
[Mike Johnson] So, it’s an area that I haven’t thought too much about with regards to cloud migration.
[David Spark] Because you’ve always walked into cloud, haven’t you?
[Mike Johnson] I’ve basically, for as long as I can remember what cloud is, I’ve kind of lived in that space.
[David Spark] You were at Salesforce. They didn’t have anything but that.
[Mike Johnson] That’s all we did. That’s all we did. So, it’s not something that I have too much experience with, so I haven’t thought a whole lot about it. Which brings me to the flip side is I would actually love to learn more. I would really like to understand.
[David Spark] You want to go into the world of on prem and then literally do a shift. There are other people dealing with this saying, “Mike, no! You’re making a huge mistake!”
[Mike Johnson] I want the scars. These are scars that I don’t have that I feel like I…
[David Spark] Anna’s cracking up.
[Mike Johnson] That I need… I haven’t paid my dues when it comes right down to it.
[David Spark] You think? That’s what other cybersecurity leaders are saying, “Mike, he’s such a fake. He’s never actually had to connect a box.”
[Mike Johnson] Ah, you haven’t walked in my shoes.
[David Spark] Okay. So, what have you heard enough of, what do you want to hear a lot more?
[Mike Johnson] I would really like to understand what are the ways that you prioritize the path that you take. It’s not something that you suddenly… You literally don’t back up a truck to your data center, load it up, and then drive down to Seattle and to Amazon’s headquarters and dump it on the floor. I would like to understand how you prioritize, how you’re picking it out.
[David Spark] By the way, if they could do that, think about how much more money they could make. They should really consider that.
[Mike Johnson] I think it would cost them a whole lot of money in return. They might have considered it. That might not be the first time that that’s been thought of.
[David Spark] Back up the truck with all your servers.
[Mike Johnson] Yes, yeah.
[David Spark] I love that. I just love the visual of that.
[Mike Johnson] Yeah, it’d be pretty cool, just push them off the side, see what happens.
[David Spark] All right. So, you want to know more about what it is because it’s not that.
[Mike Johnson] The process. Just I don’t understand. I’ve been a part of smaller pieces where we’ve had some systems that we’ve been moving from one place to another. There’s obviously a lot more to it. There’s obviously a lot of planning that goes into it and prioritization. How do you start?
[David Spark] How do you start? I think step one is the one that a lot of people… Okay, we’re going to get to that. All right. I’m throwing this to you, Anna. First, though, answer the first part of the question. You’ve heard enough about this cloud migration, and what would you like to learn a lot more? And please address Mike’s concern.
[Anna Belak] My answer’s going to be a little bit snarky, and I’m going to say I don’t want to hear any more about “lift and shift.”
[Laughter]
[David Spark] No, that’s not snarky.
[Anna Belak] I’m sorry!
[David Spark] I think that’s a knee-jerk reaction.
[Mike Johnson] No trucks and forklifts, that’s just…
[David Spark] No, no. But he created a nice visual cartoon, I like it. He didn’t say the words “lift and shift” because everyone knows. You know what it is? It’s saying the thing that 100% of us already know. There isn’t a single person’s going, “Well, why don’t we just lift and shift it?” Like, I’ve never heard that. Ever.
[Anna Belak] Yeah. But to be fair, even if you’re lift and shifting a specific application, it’s still just, for a cloud native kind of fangirl, it’s very painful to hear this because I’m like, “Oh. You’re going to a new operating model, a new environment with all these newfangled toys that you can use to make this better, and you’re just going to pretend it’s someone else’s server.” Like, what even is the point? But I do recognize that it’s very, very hard to actually rebuild an application to be fit for cloud or cloud native, so that’s why I’m saying it’s a little snarky. I’m very sorry for all you folks who are trying to accomplish this and just have no choice because it’s sort of a necessary evil. But it does make me hurt to hear about it.
[David Spark] So, what would you like to hear a lot more? And you can speak to Mike’s concern.
[Anna Belak] First of all, I have to laugh at the new form of gatekeeping of, “You’re not a real CISO unless you’ve gone through a cloud migration.” I think that’s pretty sweet. So, I’d like to hear a lot more actually from CISOs, to be honest, about their strategy around securing cloud native and securing containers and securing some of these new patterns and technologies that often actually manifest as shadow IT. A lot of folks will say they don’t have containers in their environment or will just not know, and I would bet you if you are a large enough company and somebody in your company is writing software, there are containers in your environment in some cloud, and you just haven’t found them yet. So, I would like to hear a lot more about folks’ proactive strategy around securing these things.
[David Spark] So, have you then, in the work that you do over at Sysdig, is a lot of it just gathering tips and tricks of what others have done in the process?
[Anna Belak] Some of it is certainly part of that. I mean, we are building and selling a product, obviously, that helps address a lot of the relevant pain points. So, our story’s interesting because we actually started from the container perspective. We were not originally broadly a cloud security company. We solved this very cool problem in 2013, I want to say, of, “Hey, a new technology that seems very promising for software development,” and we saw the need to secure that technology, and so that’s what we started with. Eventually what happened was we were super-early to market with a super-innovative solution, and folks who are going through cloud migrations are struggling with a lot simpler problem. So, we ultimately expanded our portfolio to address the needs that you have to meet before you can deploy containers, like checking configurations and kind of like securing sort of the cloud control plane and the cloud bits that aren’t even necessarily related to containers themselves.
So, what we do now is much more broad. We kind of have the source to run story where we secure all your artifacts from source to production, so that means we’ll scan your configs, we’ll scan for vulnerabilities, but then we also do threat detection in real-time. And the reality is many of our customers will interact with one or several of the product components because they’re maybe at a different stage of maturity, or they may just be addressing a different use case.
[David Spark] So, going also to Mike’s comment about process about step one, you developed your original product to address an issue that was a little bit later down the line, and then you realized, “Well, wait. If we’re going to bring people in, we’ve got to address the earlier problems,” and you mentioned some of them. Let’s just go to Mike’s question – what is often step one for a lot of people, for security leaders, when considering a cloud migration?
[Anna Belak] I’m going to feel a little dirty saying this but in many cases, step one is choosing whom you’re going to outsource the migration too. [Laughter] Like, you got to pick a really good managed service or consulting firm to actually help you through it because, again, if you’re a large enough org, there’s a very good chance you’re simply not equipped to do this yourself.
[David Spark] Okay. Well, that’s a good point. Now, here’s the thing – if I’m not equipped to do this myself, going back to the question we had in the very first segment, what is the questions I’m asking?
[Anna Belak] The first question is I guess how much it’s going to cost.
[Laughter]
[Anna Belak] How much it’s going to save you. Which brings me back to my “lift and shift” pet peeve. You could take your entire data center, as per Mike’s visual, and just dump it into the cloud. That is an option. It doesn’t quite work like the truck situation. It takes a lot longer, actually, and it will cost literally millions and millions of dollars to do that. And then the assumption would be that you’ll pay this cost, and then your operating costs in the cloud will be lower, and that’s actually usually not true. If you don’t refactor your state to be cloud native and to leverage the cloud resources in an efficient way, you’re going to end up paying more money for running it in the cloud than you were running it on prem. Almost always. So, you have to ask yourself the question of are you actually making a good business decision.
[David Spark] So, really and also, it sounds like the question you should be asking is how much does it cost, but B, what are you going to do to make our applications cloud native, and they will hopefully tell you that story at that point.
[Anna Belak] Exactly. So, you then want to scope your migration to include things that would be good to migrate because either they’re in the process of being refactored, or there’s some other reason why them being in the cloud is more useful to you or more cost-efficient or what have you, and then you may want to leave a lot of things on premise, which happens a lot. Like hybrid scenarios are perhaps the most common still.
More bad security advice.
25:48.531
[David Spark] What lies are told to us about machine learning? It just dawned on me that we have two segments about lies.
[Mike Johnson] Yep. Yep.
[David Spark] This was asked by someone on Quora, Václav Krpec of MSD, and he echoed these lies that he has heard, again, we’re talking about machine learning, “It’s a magical way to solve all your problems without the need to program anything.” “It just works, it’s brilliant.” “We must do machine learning in order to be competitive.” The reality, said Vaclav, is that programming is necessary to train the model. Most often machine learning doesn’t work, but you don’t hear those stories because they’re often not interesting. And the last quote is from people who think that technology without a business case and an implementation will solve their problems. So, Mike, I’ll start with you. What are some other lies about machine learning that everyone needs to be aware of? And I’ll throw out – do you agree that these are common ones you hear?
[Mike Johnson] [Laughter] So, all of the lies that we’re talking about today, we’re evidently unveiling them all and debunking all of the lies.
[David Spark] Well, yeah, and also the “lift and shift” one.
[Mike Johnson] Yes.
[David Spark] Which, by the way, everyone knows that one.
[Mike Johnson] Yes. Well, maybe not, and maybe we have educated some more people.
[David Spark] If you’re listening to the show, and we’re the first ones to ever tell it to you, please, I’m not shaming anybody, just I want to know who that person is. I’m not going to post your name and promote it. I’m just interested to know if there’s a person like that exists. By the way, in our industry. My mom, if she’s listening, I know she doesn’t know.
[Mike Johnson] But with regards to machine learning, I do not claim to be an expert, and at the same time, I’ve heard a lot of these. That it’s a magic box, it solves all off your problems. For me, what I don’t get, and I don’t understand why we’ve forgotten the saying of garbage in/garbage out. Why does that not apply to machine learning as well as everything else? If you’re feeding a magic box bad data, you’re going to get bad data back out of it. And that’s something that I think somewhere people have forgotten, that it’s magic, and it just does not work that way.
[David Spark] By the way, you can feed mostly good data. We had done another segment on a previous episode that found that just 0.7% of bad data can ruin an entire set.
[Mike Johnson] I buy that, and I think another thing that I didn’t see in this list that I was surprised to not see is related to that, of the bias of the programmers. The people who are building the machine learning models and training them in the first place, what data they feed it and what data they teach it has huge impact on the outcomes.
[David Spark] But the thing is there’s bias everywhere. Even if you don’t want to believe it, you’re creating bias at some point. At some point, bias is coming in.
[Mike Johnson] Yes. But if you’re aware of it, you can actually do something about it. Some of the biases that are there, you actually can deal with. People have just chosen not to because it’s easier.
[David Spark] Anna, as you mentioned earlier, that this is kind of a stump speech you have on machine learning, that you hate the lies. All right. Do you agree with the ones that were mentioned here, and what more would you add that you’ve heard that drive you crazy?
[Anna Belak] I have a very love/hate relationship with machine learning in general. I was a scientist in a previous life, and I did a lot of statistical mechanics.
[David Spark] Gartner analyst, scientist, geez, all these things.
[Anna Belak] A very aggressive career, yes. Yes, so I actually do agree with almost everything Václav says, and the garbage in/garbage out thing is so real, but I think it’s actually kind of more nuanced than that in many ways. Because even if you do have good data, like 100% good data, what people forget about is they focus on this magic box thing, and they forget about the fact that most of these models are tailored to deal with a very specific use case. They usually have a very narrow scope of applicability, and so if you are interested in looking for cats on the models trained on dogs, you’re not getting anywhere even if it’s a really great cat detection model. So, you really have to be careful to ask the right questions of the vendor or whoever’s peddling it to you to understand what exactly they’ve trained the model to do and is it actually meeting your need.
[David Spark] By the way, there’s a great episode of Defense in Depth which we had Davi Ottenheimer on who’s written a lot about this. I believe he’s in the process of finishing up a book on the biases of machine learning for that matter. I suggest everyone go check out that episode of Defense in Depth, just search “machine learning Defense in Depth” on our site, you’ll find it. The thing that I found interesting, you brought it up, and I mention with a small discrepancy is how, I guess, gentle and sensitive machine learning is. It’s got to be treated with kid gloves. Yes, Anna?
[Anna Belak] Yes. I mean, the real thing is you need to understand not only what you’re getting in the magic box, but how it would interact with your environment, with potential exploitation of the model itself, for example, of what care was taken in the model development from a supply chain perspective. So, there are all these angles that we almost just shut our ears and refuse to think about because we want to believe in the magic.
[David Spark] Well, yeah, because it’s going to spit out a result, and we’re going to be dazzled by it no matter what it says, right?
[Anna Belak] And that’s the other lie, right? The result is actually kind of sus, right? It spits out this probabilistic outcome, so you’ll get like, “This is 90% cat.” And the machine learning developer may say, “Okay, well, 90% is cat enough, so 90% plus equals cat,” right? So, you as the end user can’t actually tune that. You can’t decide that, for your scenario, it should be 95% cat at least. And then once you get the outcome, there’s often no context. So, if you’re trying to triage a situation or respond, you may just be told like, “95% cat,” and you’ll go, “Okay. Why? Why do you think that? Is it because the ears were pointy? Like, I have nothing to go on.”
[David Spark] When do you love then? We’re going to end positive. We’ve been so down on so much technology on this episode, I want to end on a positive note. You said you have a love/hate relationship, I want to hear the love. Why do you love it?
[Anna Belak] I do. So, I do actually love it because in the use cases which are well-selected, it can be awesome. Usually it’s awesome in scenarios where there’s a hybrid approach. So, you’re going to have, let’s say a malware detection use case, this is one of my favorite examples. So, originally we detected malware by signatures. We hash this thing and then if we see that hash we’re like, “Okay. This is an evil thing.” However, there’s mutating malware, and all kinds of other clever methods to dodge the hashing signature detection. So, and mal is because of this wobbly probabilistic nature, we’ll see small deviations because it kind of can be trained to loosely understand what bad looks like in a malware situation, and then it can detect things that would dodge the hash.
Now, of course, the situation is not that simple because as it turns out, there are many types of malware that the ML will not detect but signatures would. Because unfortunately, when you introduce probability, you will miss some things that should be obvious hits, right? So, the best models are hybrid. They have a signature-based method that’s combined with a machine learning method, and then you get the best coverage, and it is so obviously superior to just either method alone that every single endpoint protection engine today uses a hybrid method.
[David Spark] Go hybrid! With your cloud, with your machine learning, that’s the way to go.
[Anna Belak] With your car.
[Mike Johnson] With your car.
[David Spark] With your automobiles too. We all said that, we all said that. Yes. All right.
Closing
33:31.864
[David Spark] Thank you very much, Anna. I’ll let you have the last word, so hold tight here and one of the questions I ask all my guests is are you hiring, so be prepared to answer that question. I want to thank your company, Anna. Sysdig – they’re available at sysdig.com. Guess what? If you’ve got a cloud environment, and if you’re listening, guess what? You’ve got one, I don’t even need to know. There’s nobody listening to the show that is 100% on prem. I feel confident in saying that. What do you think, Mike? Do you think anyone’s listening to the show that’s 100% on prem?
[Mike Johnson] I’m sure there’s someone out there going, “Oh, I’ve got everything…”
[David Spark] Email me if you’ve never heard “lift and shift,” and you’re 100% on prem. That I want to know, I want to know who you are.
[Mike Johnson] Yes, yes. You don’t have to name yourself, but that you’re out there, we’d love to hear.
[David Spark] Just say, “I’m that person. I’m not budging. I’m staying on prem. I’m loving it.”
[Mike Johnson] “I need to hug my server.”
[David Spark] “I put clothes on it, dress it up like a doll.”
[Mike Johnson] Now, that would be a good reason. That’s a good reason to not go to the cloud.
[David Spark] Send me the photo of that too, please. Geez, so many things we want from our audience. We have a lot of demands, don’t we?
[Mike Johnson] Yes. One or two.
[David Spark] Thank you very much, sysdig.com, for all your cloud security needs. And guess what? No one’s saying, “I don’t have any cloud security needs,” so why don’t you just check them out? Or Anna will tell you more in just a second. But Mike – any last words?
[Mike Johnson] Anna, thank you so much for joining us today. It was lovely to sit down, have a conversation with you, and hear your very strong opinions. I love when someone comes on the show and has an opinion.
[David Spark] That is the best part. That’s what makes you an awesome guest. I agree with you, Mike, on that.
[Mike Johnson] That was so great, and I really loved both your ML opinions, your cloud opinions, and your experience and knowledge really shone through in having those conversations. And I love your hatred of “lift and shift” and not taking advantage of the native capabilities of the cloud, that should be a pet peeve for everyone. So, thank you for bringing that specifically to our show. Thank you for coming on, sharing your experience, your knowledge. It was wonderful to have the conversation. Thank you for joining us.
[David Spark] From the world of science and Gartner as well, I will add. All right, Anna. Please, make any pitch or plea for Sysdig to our audience. If you’re hiring, let us know. Anything else you’d like to say in closing?
[Anna Belak] We are absolutely hiring. We are an awesome place to work, we do cool and exciting things, and you get to work with me.
[David Spark] The best part of the job.
[Mike Johnson] Yes.
[Anna Belak] Absolutely yes. I will plug a couple of assets we have. So, we have something called the Cloud Native Security and Usage Report that we publish every year with real, real back-end data of what we see our customers doing out there. It’s kind of spooky, actually. You should check it out. And another asset I’ll publish that should interest CISOs and actually everyone else is we have a paper that is called Anatomy of a Cloud Attack, I believe, or something in that vein, and it actually talks about how certain attack types change as an attacker might want to execute them in cloud. So, for example, like ransomware on prem versus ransomware in cloud, what would that look like. So, pretty cool stuff. Do take a look.
[David Spark] We will make these links available on the blog post for this episode.
[Anna Belak] Thank you, gentlemen, for having me. You have made me giggle, which is awesome, and happy fourth anniversary.
[David Spark] Thank you so much. Mike, are you clear for another four years? Check your calendar.
[Mike Johnson] I’ll have to tell you, next year I’m starting to get a little worried about, it’s starting to fill up. I’ll see what I can do.
[David Spark] Start clearing it out.
[Mike Johnson] I’ll clear it out.
[David Spark] Clear it out. Clear out next year. Thank you very much, Anna Belak from Sysdig. Thank you very much, Sysdig, for sponsoring. Thank you, Mike. Thank you to our audience. Send us more subjects to discuss “What’s Worse?” scenarios. We want to hear it all. And also I need that photo of your server in doll’s clothing, if you’ve never heard “lift and shift,” and you’re 100% on prem and you’re never going to the cloud. Please let me know. Bye, everyone.
[Voiceover] That wraps up another episode. If you haven’t subscribed to the podcast, please do. We have lots more shows on our website, CISOSeries.com. Please join us on Fridays for our live shows – Super Cyber Friday, our Virtual Meetup, and Cybersecurity Headlines Week in Review. This show thrives on your input. Go to the Participate menu on our site for plenty of ways to get involved, including recording a question or a comment for the show. If you’re interested in sponsoring the podcast, contact David Spark directly at David@CISOSeries.com. Thank you for listening to the CISO Series Podcast.