A threat intelligence program sounds like a worthy effort in any security program. But, can you pull it off? There are so many phases to execute properly. Blow it with any one of them and your threat intelligence effort is moot.
Check out this post for the discussion that is the basis of our conversation on this week’s episode co-hosted by me, David Spark (@dspark), the producer of CISO Series, and Steve Zalewski. Joining us today is our special guest Jon Oltsik, distinguished analyst and fellow, Enterprise Strategy Group.
Got feedback? Join the conversation on LinkedIn.
Huge thanks to our sponsor, DataBee, from Comcast Technology Solutions
[David Spark] We’re excited to announce that Capture the CISO is coming back for a new season. If you missed our first season, then you missed out on a truly unique competitive experience. Each episode sees CISOs asking vendors deep questions about their demos, judging each competitor on innovation, need, and implementation.
Winners move on, hoping to win a guest spot on the CISO Series Podcast. It’s an inside look at something that usually happens behind closed doors. If you want to compete on the show, we’re still looking for sponsors, so email us at firstname.lastname@example.org.
A threat intelligence program sounds like a sound effort in any security program, but can you pull it off? There are so many phases to execute properly. Blow it with any one of them and your threat intelligence effort is moot.
[Voiceover] You’re listening to Defense in Depth.
[David Spark] Welcome to Defense in Depth. My name is David Spark, I am the producer of the CISO Series, and guess who’s joining me today? Why, it’s Steve Zalewski! Steve, say hello to the nice audience.
[Steve Zalewski] Hi, audience.
[David Spark] There he is. You’re going to hear a lot more of him. But first, I do want to mention that we are on cisoseries.com, and this is not the only program we have on the show. We have many, many other shows so you should be listening to all of them and then deleting all your other podcasts. That’s what I advise.
You think that’s good sound advice, Steve?
[Steve Zalewski] Damn straight.
[David Spark] I’m with you. Our sponsor for today’s episode is Comcast and their really cool data intelligence solution, DataBee. It gives you great insights into what’s going on in your environment. More about that a little bit later in the show. Steve, on LinkedIn you ask, “Which phase of a threat intelligence program gives you the most trouble and why?” Now this was based on an article by our guest today, who I’m about to introduce, and he did the research and found that trouble was more or less evenly split across all the phases, which are planning and direction, collection, processing, analysis, dissemination, and feedback.
So, first I want to just ask what has been your personal experience and does it change organization to organization?
[Steve Zalewski] Yes. So, from a personal experience when I answer, I found that number one, planning and direction, setting expectations so to speak, and then number four, the analysis, having the people that could translate the firehose of data into something actionable, not just interesting, were from my perspective my personal bugaboos on being able to do this.
What I will also share that when I posted this with our guest and was thinking about this from a technical perspective more or less, about how do you execute this, I came to realize that it wasn’t that I asked the wrong question. It’s that I didn’t have the right lens on the context for the question, and I think that’s where we’re going to talk in today’s episode about one through six are necessary but the, what I want to say, the bias on which ones are the hardest really depends upon the context for the company that you’re in.
[David Spark] Well, I definitely have the right guest for this episode because he has done a ton of research and can actually give some insight, far more than just your post on LinkedIn provided. I’m very excited to have him and I’ve actually known him for quite some time. It is the distinguished analyst and fellow over at the Enterprise Strategy Group, Jon Oltsik.
Jon, thanks for joining us today.
[Jon Oltsik] Great to be here, David. Thank you.
Why is this so darn hard?
[David Spark] Lisa Young of Netflix said, “Planning and direction is the most troublesome as there is often a reluctance to fully understand, describe, document, and recognize the potential impact of the real threats of an organization. Recognition of the real threats is how to get the most value out of a CTI program.” And Lisa Ackerman of GSK said, “Feedback,” so that’s the first and last here.
“Feedback,” said Lisa, “It is so hard to get feedback on the reports/briefings we present. How can we improve the service we are providing if we don’t get feedback? Another area, although not listed, would be metrics,” although I kind of throw that into feedback as well. And Lisa goes on to say, “How do you measure the success of the program to prove the value of the work being done?” So, from both Lisas, we got Lisa Young and Ackerman, are arguments about the first and the last here.
Again, it seems like it’s equally divided as our guest will explain, but Steve, you first.
[Steve Zalewski] How do you prove the value of the work being done? That’s what I was alluding to to the context. Because if you’re not getting feedback, you can argue, “Why aren’t I getting feedback?” But what you really should be saying is, “If I’m not getting feedback, I’m doing something wrong because they don’t know how to provide feedback.” Because this is not an area where it’s yes or no.
Nobody is ever satisfied, but how do you measure success?
[David Spark] That’s a good point right there is that the [Inaudible 00:05:43] of options or not knowing how to make a decision. Jon, I alluded multiple times that from your research, you saw equal problems across all. Yes? Confirm and what’s your relation of why you saw equal problems across all?
[Jon Oltsik] I think that the rationale is that it’s a life cycle process and everything builds on the phase before it. So in fact if you think of a phase zero before planning and direction, it really comes down to who owns the threat intelligence program and what do they expect out of it. So before you hire a threat intelligence analyst, before they can put their effort into the planning and direction and learning the priority research requirements or intelligence requirements, you have to set that up correctly.
There are obstacles in every phase and unless you do the right thing in the first phase, the second phase will be problematic, and I think that’s what we’re after here. There are also some very important skillsets and esoteric type of skillsets that also come into play.
[David Spark] Then let me dig down into that. What is an esoteric skillset that came into play that maybe people didn’t have and then therefore because they didn’t have them, that phase was like, “Oh. Well, we can’t do it because we just don’t have the capability.”
[Jon Oltsik] What I heard in my research was that you really do need people who are threat intelligence people, not incident responders, not security analysts. You need people who understand the intelligence tradecraft. So, if you think of national intelligence, when the national intelligence community comes out and says, “We have a high degree of confidence on this particular intelligence,” that means something, that’s a highly defined term.
And you have to understand someone who knows that tradecraft. And then you also need someone who can go out and find the priority intelligence requirements, so it involves talking to the CISO, talking to the security operations teams, but also talking to line of business managers, talking to M&A, talking to Finance.
What are the existential threats to the business? Because a good intelligence program is strategic, operational, and tactical, not just tactical.
[Steve Zalewski] I’m going to jump in on that, and I’m going to offer my own perspective which was, again, when I said context, a threat intelligence program is viewed as a luxury, not as a necessary function for many organizations. They’re just not big enough to be able to get to the point that they can afford the luxury of this.
And so they’re looking for a way to minimize the risk and minimize the cost. That’s the first challenge we have, right, is, “I just can’t get there. It’s interesting but it’s just not important because I’m trying to lay down an MFA control and not go hunting.”
The second thing is the analysts themselves, in many cases, come across like enterprise architects, ivory tower. They sit up there, they’re smart, they have tools, they’re looking at stuff, and then they come down from high with observations of something that’s interesting that then requires a whole bunch of work to determine if it’s important.
And so therefore, as a CISO, I’m working with very intelligent people but I’m having to demonstrate the business value of what they’re finding, not the sheer interest in finding interesting things. And so we have two things, which is luxury and this enterprise architecture high on the mountain telling me interesting things but not helping me sell more jeans.
What are they doing right? What are they doing wrong?
[David Spark] Scott Ponte of Amazon said, “In my experience, teams are at one extreme or the other and thus fail to produce actionable intel for the teams who use it. No strategic intel and the tactical intel is too broad, lacks aim, and is irrelevant. No tactical intel and the strategic intel is vague and meant for leadership and not the majority of people who depend on it.”
And Bil Harmer of Craft Ventures said, “Collecting the right data in a timely manner. So much of the data out there is pre-scrubbed before being shared, never shared, or just old enough that it’s a view into the past, not a representation of what’s happening to me today. It’s why I found cloud vendors that see across a wide swath of customers to be valuable as they can help me prevent issues without sharing the details of the who was previously attacked.”
So, Jon, there’s this aggravation that the data they’re getting is not valuable or the way that they’re handling it is just it’s not good for anybody. This goes back to sort of, maybe like you said, these are all phases, planning and direction. Maybe a lot of this would have been solved if they had better planning and direction.
[Jon Oltsik] Absolutely. The first thing that you need to do is, again, define the intelligence that you want to gather, and it has to be relevant to your business, your industry, your geography. One of the mistakes I see is that people begin with threat intelligence collection. There’s this notion in the industry that more is better because the 99th threat feed that I get may tell me something that the other 98 didn’t.
Okay, that is a mistake, and that quickly overwhelms threat intelligence teams. They don’t know what to do, they’re sorting through different threat intelligence feeds that basically have a lot of the same information but slightly different information. It may not be timely. When that happens, they’re so overwhelmed with the collection process and the processing of that data that it’s really difficult for them to get to the analysis phase.
So that’s certainly one of the problems I see.
[David Spark] Steve, what has been your experience and do you find that some of these intelligence programs aren’t serving anybody?
[Steve Zalewski] I think Bil Harmer kind of hit on it, which is – and Jon said it – I don’t need more data, I need the right data, and I need the raw data. But what didn’t get said is, “But I also need to be told which specific problems I am going after.” Because the exercise here is, “Hey, I’m allowing you to see the entire United States and you’re going to tell me where you might see a bad guy operating in Canada, right, that could influence Michigan.” What I’m more interested in is to say, “Well, no.
What I need to do is protect my grid, and so tell me when you find nefarious behavior that can impact my electric grid.”
That gets back to number one, planning and direction, is you got to give them the windows that they’re allowed to look through to see all the data but tell me for these particular types of problems. And the larger issue here is stop telling me about vulnerabilities, stop telling me about what happened 275 days ago, and give me actionable intelligence on materially exploitable events in near real time.
And now, we’re having a conversation that you’re adding real value for all of that experience and expertise that you bring to the table.
[Jon Oltsik] Yeah. To what Steve said, it’s all about shifting left, or getting left of “boom” as the military says, is what do I know about my adversaries, what are their TTPs, what are they talking about on the deep and dark web. The more I can anticipate that, the more I can plan for it.
Sponsor – Comcast
[David Spark] Before I go on any further, I do want to mention our sponsor Comcast. Now, I’m going to tell you something you may not know about Comcast and it has to do with data. Data is the currency of the 21st century, right? We all know that. It can be used to understand the health of the business, to continually adapt as needed to meet customer needs, to remain competitive, and to innovate.
It can also be used to better understand where threats are lurking or where security compromises have been made. But collecting and normalizing security data isn’t a small feat, and neither is contextualizing it with relevant business insights.
This is where DataBee enters. It’s made by security professionals for security professionals. DataBee, so it’s like data and then the buzz, buzz bee. DataBee from Comcast Technology Solutions is a cloud-native security risk and compliance data fabric platform. It integrates and enriches data from disparate sources across your security technology stack to deliver more connected insights that drive better business decisions.
That’s what we’re all about here, right?
So, with DataBee, GRC teams can validate security controls and address noncompliance, data teams can accelerate AI initiatives and unlock business insights, and security teams can quickly discover and stop threats. It’s the one solution for everyone! Awesome! So learn more about how DataBee can deliver the security data insights you need to stay ahead of the ever-evolving threat and compliance landscape quickly and cost-effectively.
Now, I’m going to tell you where to go, but listen, it gets a little complicated here. So, you’re going to go to comca.st/databee but it’s spelled a little weird, it’s comca.st, so it’s like comcast but you put the period just before the “st” and then /databee.
Can anything be done?
[David Spark] Yishay Yovel of Cato Networks said, “Which organization has the resources and skills to run a threat intelligence program at all?” I alluded to this earlier, so Yishay is bringing it up, and he goes on to say, “What is the point of giving end user organizations – with the exception of the absolutely largest – raw data and expecting them to continuously consume, deploy, optimize, integrate, and use it?
How many organizations like this even exist?”
I think, by the way, Jon will have an answer for this, but let me also read a quote from Mathew Biby, CISO over at Satcom Direct, “It can be incredibly challenging, especially in small/medium organizations where budgets and resources are very limited and you simply do not have the necessary time to work through each of the phases in a linear fashion.” So let me start with you, Jon.
In your research, what types of organizations actually had a threat intelligence program, and were you surprised by any of them? Like, “Oh, my God. I can’t believe you’ve got one.”
[Jon Oltsik] This goes back to something Steve said about a threat intelligence program being a luxury. I agree with Steve with a caveat and that is five years ago, a lot of companies I talked to said, “I can’t afford a threat intelligence program, all I want to do is block threats.” But with ransomware, with the growing attack surface, more are investing in threat intelligence.
In fact, in our research and it was enterprise-focused, 98% of organizations were increasing their threat intelligence budget.
Now, what we do see is a lot of organizations using managed services in this area. So, “I don’t have the skills, I can’t afford to get these people,” or “I’m in a remote area,” or “I can’t compete. I’m in New York City and I can’t compete with the financial services firms and things like that.” So I look for outsourcers to help me.
Still have to understand what a program is, still have to understand the division of labor with a managed service provider, but there are many good threat intelligence providers who can at least work with you and give you some inkling of a threat intelligence program.
[David Spark] Steve? Now, let me ask you – have you been in organizations that have not had threat intelligence programs and others that have?
[Steve Zalewski] The three S&P 500s all had threat intelligence programs, but they were all at different levels of maturity and I joined at different times. So in some cases they had some, in some cases we built some. I will also say, like Jon, even in the last five years the expectations of what a threat intelligence program is and what it’s designed to accomplish has undergone, in my mind, some pretty dramatic change in expectations compared to 10 years ago or 15 years ago when we were building them out.
Okay? Where I’m going with that is what I said earlier around exploitability versus vulnerability is I don’t want to build out a program like I built out my SIEM, and then have level one, level two, level three analysts, right, that are feeding the engine to then do eyes on glass.
The expectations with an MSSP is that I want to only get level three reports. I want actionable and therefore I’m leveraging the MSSP to be able to have the expertise that I can have a contract with them around what’s important for me to be alerted on, and not have to build the expertise and the six stages to get there.
But I can understand and they should hopefully help me understand whether I need to see business impact versus raw threat intelligence going into my functions for how I run my automated response. But it’s to get me to automated material exploitability as quickly as I can in a way that I can measure the value to get the money to do it.
[David Spark] Jon, you were nodding your head there. I mean, is this what you saw from your own research?
[Jon Oltsik] Yeah, absolutely. Going back to a point Steve made before – where these projects fail, what I heard as a term or terminology often when I did my research was, “It’s an academic exercise.” Meaning there’s no guidance. And what are threat intelligence people going to do if they don’t have any guidance?
They’re going to do research, whether it’s relevant to the company or not. So the relevancy, you should really start with what outputs do you want to get and then work backward to what inputs does that require, what analysis does that require, who needs to see that, what should they do with that. So again, it’s a life cycle, but if you start at the end game and work backward, that may be a helpful hint.
[David Spark] So let me throw this argument. You said from your research that people said, “I have difficulties equally across I think all six phases,” but you also mentioned at the beginning that each phase relies on the first, and everything we’re talking about goes back to the first phase, which also you say work from the back and go forward.
That honestly, if you don’t figure that out, I mean, everything else becomes mud it seems. Yes, Jon?
[Jon Oltsik] You’re decreasing the value of the program. So for instance, if the program is managed by the SOC, the SOC will want IOCs, the SOC will want to understand what’s happening in a MITRE ATT&CK framework context, but that may not help the business at large. So, I’ve seen programs that sort of muddle through the planning and direction and get some value, but the key is to just continuously try to improve that program.
One of the things, one of the quotes was really apropos of that and that is you need to push on your consumers for their feedback, “Am I hitting the mark? What am I missing? What would you like to see that I haven’t given you?” It may be as simple as the writing style, it may be too much technical information, it may be not enough technical information, but a good threat intelligence analyst program manager is going to seek that out because they want to succeed, they want to make, as Steve also said, they want to make their intelligence actionable, not just have it be something that’s nice to have or that people ignore.
Why is this happening?
[David Spark] Ryan Franklin of Amazon said, “Very few threat intelligence programs are doing this to drive real risk reduction for the business. Most threat intelligence teams look at security operations as their primary customer. Perhaps we need to shift and start doing more with the rest of IT.” Malcolm Harkins of Epiphany Systems said, “Threat intelligence programs are built to be tactical and reactive versus strategic and predictive so we can proactively adjust our postures to mitigate risk versus respond to it.” I mean, this goes to what Jon said earlier, Steve, that sometimes they just become academic exercises and the reality is, “No, the point is to reduce risk!” And all of this discussion is like, “Hey, where’s that discussion coming in?” Like aren’t we supposed to reduce risk here?
Steve, you’re nodding your head.
[Steve Zalewski] This gets back to the comment about five years ago. Right? It wasn’t about risk. It was about finding interesting things.
[David Spark] By the way, security people just like to do that. They like to find interesting things.
[Steve Zalewski] It gets back to the nature of technologists and curiosity that we always say we want that is very good in certain capacities. But the fact that we can be interesting, right, and find interesting things versus what we need to do, I say today that’s just it. Which was we now know how to find interesting, and back then that was enough to fund it.
Okay? Because we were still novices in being able to do this and demonstrate the value. Over five years, we’ve not been very good at demonstrating the value, and the leadership teams and the funding organizations are simply saying, “We gave you the money. You’re not stopping the ransomware attacks. We’re going to have to start pulling money back.” And so now you’re having to make hard decisions, right, on whether this is truly valuable or not given multiple years of not succeeding.
And then the last thing I’ll say is, and Jon kind of said this too, is there are six steps, okay? But they’re not linear. It’s not you get the first one right, then the second one, then the third one, and then it’s continuous improvement. This is more like an identity and access management program where the integrity of the program as a whole relies on the integrity of every one of those pieces foundationally, and then you want to get better at the data knowing the integrity is there.
But if you drop the ball in any one of those six, the value of the entire program goes to zero because it’s garbage in, garbage out. And I think that’s what we’re learning as well is that any one of those components when it dies kills the entire program, and it’s pretty uniform about all the places that you can basically have it fail.
[David Spark] Jon? And I think what I’m most interested in as we’re kind of wrapping up this segment was what shocked you the most in terms of people’s effort to get value out of their threat intelligence programs?
[Jon Oltsik] Well, I should say I’m encouraged because organizations are recognizing the value of threat intelligence. They’re investing in threat intelligence. So the notion of a few years ago that, “Just tell me what to block,” we’re beyond that for the most part. But once you’re beyond that, the question becomes, “What do I do?” and organizations are muddling through that.
I guess what surprised me is how many organizations still have the old mindset of let’s just collect everything we can, let’s block every IOC we can, let’s get machine-readable threat intelligence and integrate that into our SIEM and our firewall and point security and things like that. And so there’s this notion that, “Well, if we get that right, we’ve got a mature program,” and that’s absolutely not true.
The thought is, again, how does this help my business? You really need to have that mindset from, again, from before you hire your threat intelligence team through the dissemination period and then really stress the feedback.
And so I guess I’d say that if you want to do pure threat intelligence, that’s great. Go work for Mandiant, go work for CrowdStrike, go work for CISA. They do a lot of threat intelligence that’s general purpose. If you want to be a threat intelligence analyst at a large organization, you’re going to have to find out what the stakeholders need, you’re going to have to define that in terms of intelligence requirements, and then you’re going to have to put together the life cycle and the program to make sure that people get that, again, so they can react, so that they have the intelligence to make decisions.
And those decisions will not only help improve security but they’ll help direct investments, they’ll help with validation of security controls. So it can be a very, very fruitful program, but it’s not easy. It really does take that commitment to the full life cycle and the kinds of things that Steve alluded to from his experience.
[David Spark] Steve, you want to close on this? Although that’s a great close right there. Is there something you want to add, Steve?
[Steve Zalewski] Here’s what I say – I am incredibly excited about actually this area of innovation because I think what we’re hearing is that we realize that there’s a lot more value in this program to tap, but we have to go back and rethink the value of the program itself. And the MSSPs and the outsourcing that’s going on is actually a huge step in the right direction to take this from a luxury to mandatory in a cost-effective way.
[David Spark] Good point.
[David Spark] All right, we have come to the end of our show but it’s the part where I ask both of you which quote was your favorite and why. I’m going to start with you, Jon. Which quote was your favorite and why?
[Jon Oltsik] David, I love Ryan Franklin’s quote from Amazon which was, “Very few threat intelligence programs are doing things to drive real risk reduction for the business. Most threat intelligence teams look at security operations as their primary customer. Perhaps we need to shift and start doing more with the rest of IT.” I totally agree with Ryan, although I would expand it beyond just more with IT.
That’s a good place to start, but can you produce results or intelligence and decision-making data to the business, to the executives, for things like mergers and acquisitions, for things like operational risk, third-party risk. That’s where a mature program has to go, so Ryan’s right, they’re tactical today, I think that’s where they have to strive to go.
[David Spark] Ah, excellent point. Steve, your favorite quote and why.
[Steve Zalewski] I liked Ryan’s, I liked Bil Hamer’s as well, but I’m going to go with Malcolm Harkins from Epiphany and the key here is, “Threat intelligence programs are built to be tactical and reactive versus strategic and predictive so we can proactively adjust our postures to mitigate risk versus respond to it.” I think what it really speaks to is the excitement I have is that we’re realizing that there’s a role to play in both and we’ve been biased to the former which is the tactical and reactive to find things.
Whereas now we’re starting to understand what’s important to the business, what types of threats are we trying to identify, and now we’re looking for evidence that those threats are actioning against us and not just from potential vulnerabilities but that they’re actually in the process of it. And so therefore we can work on containment and not restrict ourselves to simply prevention and recovery.
[David Spark] Very good. Excellent. So much, Steve. Excellent, Jon. This was so packed with information, and your research on this was so enormously valuable, so thank you very much. I want to thank our sponsor Comcast. Remember DataBee, an amazing solution from Comcast. Check them out comca.st/databee.
And thank you, Steve. Jon, do you have any last comments on this topic or would you like to make a plug for Enterprise Strategy Group?
[Jon Oltsik] Well, on the topic, I think Steve nailed it. That this is one area of security where we can squeeze a lot more juice out of the lemon, and so moving in that strategic direction, really maturing your program is worth the investment. And my next project that I’m really excited about is about generative AI and security.
There’s a lot of buzz out there.
[David Spark] A lot is the understatement. [Laughter]
[Jon Oltsik] But I can tell you – I’ve talked to nine CISOs in the last two weeks and there isn’t a lot of planning on the demand side, so I think that will change, but I want to understand what are the use cases, what are the challenges, what are the requirements, and really important – what is the governance that organizations are putting in place for generative AI and I’ll be doing research along those lines.
[David Spark] Awesome. Thank you very much, Jon. Thank you, Steve. And thank you, audience. We greatly appreciate your contributions and listening to Defense in Depth.
[Voiceover] We’ve reached the end of Defense in Depth. Make sure to subscribe so you don’t miss yet another hot topic in cybersecurity. This show thrives on your contributions. Please write a review, leave a comment on LinkedIn or on our site CISOseries.com where you’ll also see plenty of ways to participate, including recording a question or a comment for the show.
If you’re interested in sponsoring the podcast, contact David Spark directly at David@CISOseries.com. Thank you for listening to Defense in Depth.