Why do we hear so many stories about incidents related to poor or misconfigured cloud services?

Check out this post and this post for the basis for our conversation on this week’s episode which features me, David Spark (@dspark), producer of CISO Series, co-host Geoff Belknap (@geoffbelknap), CISO, LinkedIn and our sponsored guest, Brendan O’Connor, CEO, AppOmni.

Got feedback? Join the conversation on LinkedIn.

Thanks to our sponsor, AppOmni

AppOmni is building the future of SaaS security. We empower our users to enforce security standards across their SaaS applications, and enable them to remediate in confidence knowing they’re fixing the most important SaaS security issues first. Contact us at www.appomni.com to find out who – and what – has access to your SaaS data.

Full transcript

David Spark

Why do we hear so many stories about incidents related to poor or mis-configured cloud services?

Voiceover

You’re listening to Defense in Depth.

David Spark

Welcome to Defense in Depth. My name is David Spark and I am the producer of the CISO Series, and my co-host for today is Geoff Belknap who is the CISO for LinkedIn. Geoff, thanks for joining us today.

Geoff Belknap

Absolutely. Hey everybody.

David Spark

Our sponsor for this episode is AppOmni, who also brought along our guest today, whom I will introduce shortly. Geoff, you conducted a poll for LinkedIn asking why so many cloud configuration-related breaches occur. From 640 votes, the conclusion was that it’s easy to use cloud as a scapegoat and that cloud management is hard. But many still believed that configuration drift and third party access are also significant issues. What did you learn from this poll?

Geoff Belknap

The main thing I learned was that cloud is blamed for everything. We had so much engagement on this post, and from the conversations it’s clear that cloud configuration is really hard and that’s an important recognition. We have the ideal person here to talk about that today.

David Spark

Our sponsored guest, Brendan O’Connor, CEO of AppOmni has been with us before. Thank you for joining us.

Brendan O’Connor

Great to see you guys again.

How did we get here?

00:01:50:06

David Spark

Chris Hughes, from Rise8 said, “Cloud environments can be very complex and dynamic, allowing for a myriad of configurations, and mis-configurations. Organizations are quickly adopting cloud while having little or no plan for upskilling their existing workforce or bringing in those with the needed skill sets.” This comment from Chris received a lot of likes. We also heard from Bruce Gibson of Ermetic who commented, “It boils down to lack of time, resources and expertise. The cloud providers’ native tools are not easy to use and the cloud is proliferating faster than folks can hire and onboard.” To me, all these issues boil down to the fact that we do have cloud mis-configurations. Do you agree, Geoff?

Geoff Belknap

Yes, there are a myriad issues and cloud configuration is simultaneously the best, and the most dangerous thing about cloud.

David Spark

That’s a good point and I would repeat that what makes cloud so wonderful, is also it’s greatest fallibility. Can you give me some specifics of that, Geoff?

Geoff Belknap

We’ve discussed before on the show that you can eliminate a lot of the human error that is typically found in on-premises environment where there is low automation or perhaps a lot of one-off builds, by building all the automation into the cloud and letting the cloud infrastructure management systems do the work for you. However, if you make a small typo in a high scale environment, that small typo will be dutifully copied across the the entire environment with catastrophic consequences. It’s critical to pay attention to the automation processes in these cases.

David Spark

Let’s discuss that small typo issue further, Brendan. How simple is it to make a typo and cause a cascade of problems?

Brendan O’Connor

I’m not sure it’s about a typo, but I do know that with the cloud you can shoot yourself in the foot with a cannon, making a mistake that has a huge blast radius and, therefore, impact. Customers demand that cloud is configurable and the cloud providers are listening to their customers allowing them to configure to run almost any conceivable business process in Software as a Service public cloud infrastructure. This makes for tremendous value, but we can draw a contrast between a tram that follows a simple route – for example from one airport terminal to another – with a car that can take you anywhere, forwards or backwards, and fast when required. Some car drivers don’t want to say in the same lane, and in the same way cloud users want to be able to customize their applications and the infrastructure to support their business, and the needs of their business are changing faster than ever. Customers are demanding customization and the ability to modify and integrate. This comes with complexity. So, going backwards in a car is not inherently good or bad. If you’re doing 70 mph on the highway and throw the car into reverse, it’s a terrible decision, but you can’t blame the car manufacturer for putting a reverse gear in the vehicle and causing the resultant chaos.

David Spark

Our contributors, Chris and Bruce, listed a whole myriad of problems: time resources, expertise, upskilling an existing workforce. When potential customers come to you at AppOmni, why do they require the sort of solution you deliver?

Brendan O’Connor

I have the unique experience of leading security teams at two of the leading SaaS providers, Salesforce and ServiceNow, so I have seen this from the inside out and the outside in. Visibility is a big challenge, but it’s also bridging the gap between intention and implementation. It isn’t that people don’t know what they want to do, it’s more that there is a lack of specific knowledge of how to do it. When you look at SaaS platforms, the challenge is they are all different and there is no commonality on how certain tasks are executed. I was at Salesforce for ten years and knew that platform like the back of my hand. When I joined ServiceNow, with ten years of cloud security experience, I believed I would add value on day one, but ServiceNow works completely differently. None of my experience translated, so I had to re-learn how to do things in the language of ServiceNow. If I go to a foreign country being fluent in English, it won’t mean I am 50% or 75% fluent in that local language. If I want to be fluent in that language, I need to take the time to learn it and be able to speak it. That doesn’t mean I don’t know how to speak, it just means I don’t know how to express myself in that language. The problem with SaaS is that there are so many major platforms that organizations rely on, and they are all different. Microsoft 365, Salesforce, ServiceNow, Workday, GitHub; they are not alike.

Can this problem become even more complicated?

00:07:04:07

David Spark

Christina Morillo of Trimarc Security said, “The dynamic nature of cloud means things change often. You cannot trust the configuration you set three months ago is still correct, or even in the same blade for that matter. Cloud providers have also been known to make changes to tenant-wide settings, sometimes with no prior warnings, meaning that organizations moving to the cloud need processes and tools like security assessments to validate configuration settings often.” This illustrates the classic mistake of setting and forgetting, right, Geoff?

Geoff Belknap

Yes, and to the point made by Brendan earlier, you have many options, and the beautiful thing about the cloud is that you can update the environment to the current state of your own needs. If you are not doing so, you must question why not, and expect some additional risk as a result.

Brendan O’Connor

I agree, and what I see is that a lot of times organizations haven’t taken the time to properly define what their baseline is, not properly evaluating the state of the cloud compared to the desired state. If the desired state has never been documented, that should be step one. Ask yourself what good looks like, what out of the box security looks like, and do you understand what guardrails you need to put in place to make a continual comparison.

David Spark

Brendan, could you give me an example of what configuration drift looks like?

Brendan O’Connor

I think a lot of organizations drastically under-estimate the amount of change that happens in cloud platforms, particularly Software as a Service. The cloud providers are constantly pushing new features and functionality. It’s what you’re paying for as a subscribed customer, the ongoing support and automatic upgrades. People live inside these applications, and if you’re a security professional and you have never been in sales, you probably don’t know that much about Salesforce. If you’ve never been in marketing, you probably don’t know that much about HubSpot. If you’re not in HR, you may not know about Workday. It’s easy to under-estimate how powerful and complex some of these platforms are, but people are living in these applications and making changes every single day, and it’s possible to support almost any conceivable business process, running through cloud applications and cloud infrastructure. They have become hubs within the organization with many integrated workloads, APIs, data connections, and it can be very difficult to understand the downstream impact of one small change. You can think you’re making a small change to solve your problem by adding permissions or changing permissions on a database table for example, but you may not realize that’s connected to an external API and now all of that data is exposed to the public internet.

Geoff Belknap

A common trope that people know of and which really applies here is risk surface, as in how much surface area of risk is there in a given environment. I believe there is also mistake surface, especially to those who are new to operating in a cloud environment. Of course there is risk that someone may do something maliciously, but there’s also simply the opportunity for you to make a mistake in what is a complicated environment. It can accelerate the growth of your business, but you really have to understand what you’re doing with all these options available to you, otherwise you can shoot yourself in the foot.

Brendan O’Connor

That’s a great point because when you look at how code is developed today, no-one would ever say that their code security strategy is to ask their developers if they followed best practices and did they write any bugs today, and then base their need for pen testing and scanning source code on that. No-one would ever say that’s there code security strategy, but when you consider cloud applications, these are highly complex state machines. They are a lot closer to code than to a simple UI, but the way that we support our developers and our IT teams that are configuring these applications, is to ask them to just do their best and not make any mistakes, and that’s a very unfair place to put the line of business. When security is not giving detailed guidance, or there is no sort of automation that provides guardrails to help IT and these administrators stay in the right direction, it’s unfair to put them in that position and expect them to be perfect, because we don’t expect that anywhere else in our organization.

How do we make this everyone’s concern?

00:11:23:23

David Spark

John Bailey of JP Morgan Chase says, “It’s very easy, in a busy, understaffed organization, to simply say “that’s not my job,” and just walk away, leaving a mess that no-one will address until after it causes an incident.” Amrita Mukherjee of Vigilant Technologies adds, “Customers believe that the CSP – the cloud service provider – is responsible for a lot of things like OS patching, and even disaster recovery. They simply do not know about the shared responsibility.” Patrick Garrity of Blumira goes on, “Security is an afterthought. People consistently delay security controls in an effort to make migration easier and then up never putting them in place.” All these comments suggest to me that this is just something that is chronically being avoided and that’s why it results in a monstrous problem.

Brendan O’Connor

I love that quote from John Bailey at JP Morgan Chase. He says, “That’s not my job,” or that the organization says that’s not my job, and I think that is a big part of the problem. In the organization, whose ongoing job is it to work with IT for their cloud application in the lines of business? Who knows what kind of data is in it? Is it social security numbers, is it PII, is it immigration documents and passport information? Is it your forward-looking financials and you’re a public company, and do you have all of your sales forecasts out there on the public internet because someone made a mistake? Someone in security needs to assume responsibility to work with these teams because these applications are truly mission-critical, and the data that’s in them is among the most sensitive data in your entire organization. How can it not be anyone’s job to make sure that we’re protecting that data?

David Spark

I don’t read that many job listings, but I don’t know if cloud configuration has been named on a job listing anywhere, yet we keep hearing that so many breach happen because of this. Geoff, have you ever put cloud configuration on a job listing, or have you seen it?

Geoff Belknap

I don’t think it is being listed explicitly, but I do believe this is an underlying problem, especially as we continue to evolve in the security space. 15 years ago, it was pretty easy to draw boundaries between whose job was it to manage the network, and whose job it was to manage storage, but now it can be very confusing. In a very high scale, complex environment, is it the data team’s job to manage access controls or identity and access management? Is it the privacy team’s job to decide who can access what data? Also, overall, whose job is it to make sure that the configurations are continually enforcing privacy, safety and security, because all of those controls are usually used by multiple people. In most cases, especially if you’re a cloud native environment, you are looking for people that understand how cloud works holistically as one entity. You’re looking for smart adults that will take ownership and ensure that whatever they are deploying probably is safe for the entire organization, and not just for their niche application.

This problem doesn’t end here.

00:14:42:14

David Spark

Thomas M of the Universal Music Group said, “There are 30 ways to solve a problem in cloud, but only one might be the most legitimate way to solve said problem. It’s also not the easiest to track consistently in the cloud, depending on what technology stacks you use.” From Trimarc Security, Christina Morillo adds, “It’s work that requires outsources and continuous supervision. Like a child.” I like that comment, although we’ve all been children before but we all haven’t been Salesforce, Workday or HubSpot before so, as has already been mentioned, we have to learn all new languages we don’t understand. Brendan, what can you add to our running theme which is the complexity of cloud?

Brendan O’Connor

I think that in technology we naturally gravitate towards mono and duo cultures: iPhone and Android; Windows and Mac on the end point; Windows and Linux; Oracle and SQL Server. You look at infrastructures as a service, there’s AWS, there’s Google Cloud, there’s Azure and in those three vendors, that’s most of the market. When you look at SaaS however, it’s not three vendors. It’s not 300. You have some of the largest software companies in the world and they all have SaaS offerings. You have Microsoft and Salesforce, ServiceNow and Workday, SAP and Oracle, Adobe, Atlassian, GitHub, Box, Zoom. The list goes on, and I would guess that most people listening are using many, if not all, of those applications I just mentioned. We don’t have one or two systems to specialize in, so we have a highly heterogeneous environment where different lines of business and different GOs are all in on different SaaS applications, that all behave differently. Therefore, you don’t have the leverage of setting, for example, a Windows end point standard which you push out and enforce across millions of end points through group policy. You simply can’t do that in SaaS today.

Geoff Belknap

I think it’s important to underscore this point because while my mind always goes to IaaS, infrastructure as a service, one of the deepest problems that people don’t always think of, you have to invest resources to ensure that you know how to solve this across all your SaaS providers. Right now, you do have Workday and Salesforce and probably other things, but you’re not thinking about those. And if you ask the security team or the IT team, if they are sure that SSO is set up correctly on Salesforce, the team will probably say “Yes.” Then you explore with them the 30 other SaaS providers to see if they are all set up consistently, in the same way, with the same specific expiration. If a change is required, do we know how to make that change across all the 30 different providers that are doing SSO? The answer then is probably, “No.” And it turns out we are not doing 30 providers, we are doing 40 providers. That’s a common discovery when organizations go to look at these things. We also shouldn’t forget that those providers change sometimes. Options change, settings change, the location, the API call, all of that is very difficult to keep on top of and we hear all the time, it is the leading cause of breach sometimes that you have mis-configured something that may have been configured correctly initially.

David Spark

In the recent Fastly breach, it wasn’t an outside problem, it was their own internal working. So these are problems you can create yourself without any outside influence, right?

Brendan O’Connor

I think that mis-configuration is not a binary definition. It’s not like there is a button that says, “secure versus insecure” and someone in the organization pressed the insecure button and set your SaaS application to insecure. I’ll go back to the car analogy. If you’re backing out of your driveway, reverse is the right gear to use and reverse is not inherently wrong. If you’re doing 70 mph on the highway, throwing the car into reverse is definitely the wrong thing to do. It’s all about context, and the difference is your intention versus your implementation. This is what you did; is that what you meant to do? Almost every time we see these breaches, it is good people with good intentions that made a mistake because they didn’t fully understand the blast radius, or the downstream impact of the change that they made, and the truth is, it’s hard. You can’t blame the cloud provider for letting you reverse. It depends on how and when you choose to use reverse, and sometimes you want to put the car in reverse. It’s not the gear’s fault; it’s the driver’s fault.

David Spark

You highlight a good point that sometimes it’s simply that the interface being used does not communicate that the user shouldn’t have done something.

Geoff Belknap

You could be doing all the right things and then suddenly, on this new SaaS application, it isn’t called reverse, it’s called backwards, or it’s called inverse drive, or something like that, and all your engineers speak one language and in this application it has a different name. Good luck if you didn’t know that that was what you were looking for.

Brendan O’Connor

You need a translator.

David Spark

Since that’s what AppOmni offers in this very solution, I’m going to let you have a comment on just that, Brendan. But first, as I always ask, Brendan, what was your favorite quote and why?

Brendan O’Connor

You know I’m gonna go back to what John Bailey at JP Morgan Chase said, “It’s very easy in a busy, understaffed organization to simply say, “that’s not my job,” and walk away.” I think that the reason that people aren’t paying attention to SaaS is they think, or hope, that it’s someone else’s job or that someone else is doing this for them. The truth is, everyone thinks it’s someone else’s job, so it’s no-one’s job.

David Spark

Good point. Geoff, your favorite quote and why?

Geoff Belknap

My number one favorite quote was when Brendan said, “You’re not walking around the office asking people if they wrote bugs today,” which I will now begin to do. In terms of our contributors, I think Christina from Trimarc Security really hit it on the head. Yes, it is great if you’re gonna invest in cloud and SaaS and IaaS, but doing these things you do have to invest resources and money into making sure that you’re managing them effectively. It’s work that requires resources and continuous supervision. I don’t know that it’s like a child, but you do have to pay attention to it.

00:21:14:23

David Spark

In closing, I thank you, Brendan, and the whole AppOmni community for sponsoring this, and many, episodes with CISO series. We greatly appreciate that. Brendan, have the last word and make final pitch for the translation capabilities of AppOmni in SaaS environments. But first, Geoff, any last words?

Geoff Belknap

Don’t forget to care and feed for your cloud, and LinkedIn is always hiring.

David Spark

Brendan, last words and please, give a pitch for AppOmni and how you play in this environment.

Brendan O’Connor

AppOmni is the leader in SaaS security management. We are 100 per cent purpose-built for Software as a Service. We don’t do on-prem, we don’t do infrastructure as a service. We are focused on SaaS and the SaaS problem. So we offer a custom-built tool and we are a team of security practitioners who all worked at in-house security at Salesforce and ServiceNow, and Workday and Slack and Apple. The team is founded not just by lifelong security practitioners, but people that worked inside some of the leading SaaS and cloud providers. We’re building the tool that we wish we had when we had to manage SaaS at scale.

David Spark

And now you have it.

Brendan O’Connor

We do. This problem isn’t going away on its own. It’s not getting better on its own. Security debt accumulates with interest very quickly, and it only ends in one place: you either address it or there’s a breach and it forces you to address it. Now is the time. Run to the pain, because the problem is only going to be more painful and worse later. Now is the time to get your arms around it, to really understand what your cloud configuration is, and put automation in place so that you can stop configuration drift, because it’s only going to get worse over time if you don’t address it now.

David Spark

Brendan, as I always ask everybody, are you hiring?

Brendan O’Connor

We are absolutely hiring, in all positions. We’re growing like crazy.

David Spark

I want to thank our audience, as always, for their awesome contributions. Please keep them coming and let us know if you see a good online discussion with a lot of feedback. We can turn that into an entire episode of Defense In Depth. As always, we appreciate your participation and listening to Defense In Depth.

Voiceover

We’ve reached the end of Defense in Depth. Make sure to subscribe so you don’t miss yet another hot topic in cybersecurity. This show thrives on your contributions. Please write a review, leave a comment on LinkedIn or on our site: CISOSeries.com where you’ll also see plenty of ways to participate, including recording a question or a comment for the show. If you’re interested in sponsoring the podcast, contact David Spark directly at David@Cisoseries.com. Thank you for listening to Defense in Depth.