Two years ago, at Black Hat, I produced a silly video asking attendees if security and developers should be in couples counseling. Everyone agreed, yes they should be, but the response from the predominantly security audience was “they should just listen to me.”
If you’ve ever been in couples counseling, you know that technique doesn’t work, no matter how right you think you are.
Developers and IT have already started a great party called DevOps and security has some serious FOMO. The InfoSec team is trying to get involved by placing themselves at the front (shifting left) or inserting themselves (DevSecOps) into the process.
It’s a rather self-interested goal.
“A security team needs to be viewed as a partner that helps move the organization forward vs keeping things stationary,” said Norman Hunt (@normanhunt3), deputy CISO, GEICO.
“Security cannot ask people to reinvent their workflows,” advised Mitch Parker (@mitchparkerciso), CISO, Indiana University Health.
Instead, “work within their existing processes,” said John Prokap, former CISO, HarperCollins.
What would that be? I asked security professionals, “What is it that security is already doing or could do that would be embraced by DevOps?”
Here’s their advice.
Thanks to our sponsor, Capsule8
Editor’s note: This article is part of CISO Series’ “Topic Takeover” program. While the article sponsor, Capsule8, and our editors agreed on the topic of aligning with DevOps, all production and editorial is fully controlled by CISO Series’ editorial staff.
Got feedback? Join the conversation on LinkedIn.
1: From the bottom up and the top down
“Tackle from both directions,” said John Karabin, national director cybersecurity, NTT, who recommends both beginner-level security education for developers and supportive direction from management. “Leaders at every level have to insist on it culturally and procedurally.”
2: Stop trying to control DevOps (you’re an imposition)
“Often the security team feels that they still need to control, oversee, and even intervene directly in the testing cycle,” said John Meakin, CISO, Equiniti.
“Listen and learn instead of demand and annoy,” advised Steve Zalewski, deputy CISO, Levi Strauss. “Security is about alignment, influence, and patience to get what you want. Learn the business of DevOps and become the trusted ally over time.”
“Your target is to support DevOps engineers in deploying secure applications and infrastructure without inserting an approval bottleneck that slows down releases,” said Keith McCartney (@kmflgator), CISO, Zenefits, who suggested that with any checks you’re doing (e.g., linters and CI checks) that you’re providing clear and actionable feedback for DevOps engineers.
3: Be of service to the developer-first mentality
“Security has gotten lazy, operating within the confines of its frameworks, and those cannot permeate the DevOps processes,” said William Gregorian (@WillGregorian), CISO, Addepar. “We have to reinvent and start a robust recommendation engine that sets the tone on how to align our operations with the DevOps manifesto, and not the other way around, making it less about right or wrong, more about getting it done.”
4: Relinquish control
“We have to be comfortable in being uncomfortable,” said Al Ghous, CISO, Envision Digital, who suggests relinquishing security responsibilities over to engineers, which would inherently make security part of DevOps.
“We always want to do things our way,” added Ghous. “Make security part of their job function… We have to be comfortable with DevOps and engineering being ‘responsible’ while we remain accountable.”
5: Security is a shared responsibility
“Asking about where security fits is the wrong question. Security is more of a mindset and shared responsibility that should transcend both development and operations,” said Mike Wiacek, CEO of a stealth security startup.
As a concrete example, said Zenefits’ McCartney, “Intentionally pair your security goals (risk control, vulnerability mitigation) with the goals of DevOps (speed, reliability, flexibility).”
6: We’re all trying to win the same game
We’re not on different teams.
“The first mistake is to think being left out is always a problem for security,” said Davi Ottenheimer (@daviottenheimer), vp, trust and digital ethics, Inrupt. “It’s more of a bell curve, where being left out can be symptomatic of success (understood) as much as failure (ignored).”
Think of a couple with no kids who are being pestered by the family (security mavens) to have children, said Levi’s Zalewski. Eventually, the couple bends to the pressure and adopts a teenager (security engineer).
“Security has two options on how to play their role,” said Zalewski. “They can be the obnoxious, attention seeking, disruptive security child or the intelligent, manipulative security child.”
Zalewski recommends the latter: “Divide and conquer the parents to maximize your shared wins. Everyone knows that if dad says, ‘no,’ you go to mom and visa-versa. So as the security child listen to both and don’t be afraid to pick your battles and ally with whichever parent gets you what you want. A little parental rivalry for your attention can be a good thing, especially if you always share your wins with one or both parents.”
7: Determine who owns what
“Have a frank conversation about system ownership,” said Zenefits’ McCartney. “Are there systems (identity, logging, configuration monitoring) that make sense for security to manage and operate?”
If you can successfully get security to own part of the tech stack, the security/DevOps relationship becomes operational and can grow faster, noted McCartney.
Nir Rothenberg, CISO, Rapyd, concurred, “If a security team could define solid, technically sound requirements, they would many times stand out above unclear, confused requirements by developers and product managers.”
8: Focus on quality, not security
“Security can use quality as a way to bring attention to security within engineering/development teams as well,” said Envision Digital’s Ghous. “A security defect should be seen as a quality defect.”
If addressed as quality, then security implementations should be implemented within DevOps’ existing model.
“All security teams do is provide the thought leadership and guidance/oversight to make sure quality, and therefore security, is not compromised,” said Ghous.
9: Poor security is just another flaw that DevOps needs to resolve
“No amount of security geeks intervention will ever achieve as good production security as will the security-minded DevOps guy,” said Equiniti’s Meakin. “DevOps guys can no longer escape the consequences of poor secure design and coding. The whole agile ‘fail fast, learn quickly’ encourages rapid resolution of all flaws.”
10: You get to advance once you reach this milestone
To make sure that both security and DevOps are operating on the same timeline, present achievable milestones. Simply put, the development teams need to complete certain security-related work, such as deploying various analysis tools at different points in the development process.
“This puts our teams (security and development) working together to achieve the milestones,” said Tom Garrison (@tommgarrison), vp and gm of client security strategy at Intel. “It also avoids the last-minute diving catches for various security-related work/testing that is often found very late in the development process.”
11: Let them know about the exposure they didn’t know they had
According to the 2019 State of the Software Supply Chain Report, 85 percent of component releases are open source. Most organizations don’t realize this nor do they realize what risk these components impose. A Software Composition Analysis (SCA) generates an inventory report of all open source components that have direct and transitive dependencies.
“If you aren’t performing SCA you don’t know how many operations support system (OSS) components you have; your asset inventory is incomplete,” said Dan Walsh, CISO, VillageMD. “Imagine the visibility security can provide when they are able to report those inventory metrics.”
12: Identity management solutions
Security and developers both love solutions that greatly improve user validation, such as multi-factor authentication (MFA). Add on top of that a software-defined perimeter around the users.
Combining the two, said Nick Espinosa (@NickAEsp), CIO, Security Fanatics, “It allows their identities to be verified using technologies like biometrics, and allows for greater control of application access. It’s vastly more secure than the standard LDAP based login the world has used for decades.”
13: Measure, share, improve, repeat
“Measuring progress of improving resiliency and sharing that across teams will be the proverbial tie that binds and make for a collaborative, productive, and secure culture,” said Mike D. Kail (@mdkail), CTO, Everest.org.
“Look for ways to introduce security improvements in small and regular increments, gathering metrics and feedback to ensure progress,” said Lenny Zeltser (@lennyzeltser), CISO, Axonius.
Nir Valtman (@valtmanir), vp, head of product & data security, Finastra suggested companies “Publish a dashboard to engineering and DevOps with a ‘product/service health’ from a security standpoint. For example, at Finastra we have a power business intelligence-based dashboard that reflects the compliance with all security tools per product, and it refreshes on a daily basis automatically.”
14: Let DevOps peer inside your real-time threat tools
“Both security and DevOps love to have high visibility into heir environment,” said Rapyd’s Rothenberg.
“The same metrics that security uses to see attackers are the same metrics that can also see bad form flows, logic errors, and other non-security issues,” said Jimmy Sanders (@jfireluv), head of information security, Netflix DVD. “DevOps is now and would be thrilled to be able to utilize this level of visibility to quickly analyze pain points within their environment to accurately address and correct issues, security related or otherwise.”
“Deploying an AI-backed live threat monitoring via technologies like SIEM that can detect and analyze both north and south, as well as east and west traffic, is something that all DevOps teams should easily embrace,” added Security Fanatics’ Espinosa.
15: Walk a mile in their shoes
“You need to be able to articulate the entire however many step process that an engineer goes through from ideation to production,” said VillageMD’s Walsh, who recommends having security ship code or deploy infrastructure in the cloud.
Zenefits’ McCartney suggests job rotation between security, DevOps, and software development.
“You may be surprised,” said McCartney. “But many developers are curious to learn more about security and would jump at the chance to dedicate time to learn.”
And vice versa, let the security team delve into the code.
“If your DevOps team operates in an Infrastructure-as-Code model, use their code as your go-to source to understand how the infrastructure has been built and operates,” added McCartney who noted this is the step needed to plug your security team into the continuous integration (CI) processes to provide approval of changes directly in git.
16: Build guardrails that still allow for innovation
“Safe by default,” explained JJ Agha, CISO, Compass. “Embrace security controls from the start and embed it to check for secrets and keys.”
“Work with DevOps teams to develop standards (aka guardrails) that support an organization’s security policies and risk objectives so that it enables DevOps to go as fast as possible,” said GEICO’s Hunt.
Additional examples, explained Hunt, could be standard deployments in a cloud setting that include standard security and configuration settings, and minimum data encryption standards for data in transit and at rest.
17: Break down larger tasks into smaller discrete tasks
“Operate within the behavior and rhythms of the dev team,” said Adrian Ludwig, CISO, Atlassian. “Break down large tasks into smaller pieces, and weave those into the tools and processes that the dev teams are using.”
Big projects, such as pen tests and vulnerability scans, can still happen, said Ludwig, but results need to show up to the dev team as either discrete bugs to be fixed, or individual tickets that are dropped directly into the normal developer workflow.
18: Train security champions in the dev team
A key component in making security part of the DevOps culture is having security champions within the DevOps team itself,” said Naomi Buckwalter (@ineedmorecyber), director of information security and privacy, Energage. “Instead of me pushing security ‘down’ onto the DevOps team, my security champions are pushing for security ‘within’ the team itself.”
“Benefits of those security champions are conducting application/service threat modeling, self-service application code vulnerability scanning, appropriate 3rd-party service/library lifecycle management, and dedicated sprints to ensure any finding from those items are addressed appropriately,” said GEICO’s Hunt.
“I meet with [a security champion] regularly and we find a plethora of opportunities to bolster both the DevOps structure and our security posture and visibility,” said Jeff Hudesman, head of information security, DailyPay.
“By having a champion in each team it allows the dev teams to feel they are more in control of their destiny and we get an insight into the directions they are heading,” said Benjamin Carr, CISO, Qualys. “The security champions program helps to break down barriers and it acts a force multiplier.”
19: Like DevOps, security champions’ process improvements will be incremental
While the idea of a security champion sounds awesome, don’t expect their introduction to result in a revolutionary change of your DevOps environment.
“Within nearly all engineering organizations, most of the engineers are primarily focused on tasks that are not security related. That means there’s a huge amount of inertia in mindshare, process, and code that quite correctly ignores security,” said Atlassian’s Ludwig. “But [your security champions] (likely <5% of engineers) can tweak things slightly to make things a little more secure (and this can be repeated every single day to make things a lot more secure over time).”
20: More risk-based decision making
“DevOps is so focused on speed, that it sometimes makes short-sighted decisions,” said Axonius’ Zeltser. “Proceeding in a way seems easiest or fastest in the moment can create a backlog that weighs down the organization in the longer term.”
For that reason Zeltser advises teaching DevOps about the importance of making risk-based decisions. They should already be fixing things based on business impact. It requires the same type of prioritization, but now with a focus on business risk, which is a form of business impact.
21: Validation process should be commensurate with associated risk
Security can help by building and monitoring validation processes that allow them to “identify risks, exposures, or vulnerabilities within applications at a frequency commensurate with the risk associated with the application,” said Nina Wyatt, CISO, Sunflower Bank. “When issues are identified, it offers a ‘real-world scenario’ to add more meat to the training and guidance.”
22: Distribute security responsibilities to avoid bottleneck
“DevOps requires developers to take accountability for software performance, stability, and operations. We can similarly empower developers to include security knowledge and decisions in their work. Distributing security responsibilities helps us avoid being a bottleneck,” advised Axonius’ Zeltser.
23: Cross functional training refined with metrics
“Security has to be invested in DevOps or DevOps has to be invested in security,” said Roger Hale (@haleroger), CSO, BigID. “This means looking at more cross-functional training and experience to support the must have goal of true responsibilities and metrics for data protection across engineering, DevOps, and security.”
While DevOps likes to measure their performance, and modify procedures as a result, the measurements need to be updated, said Hale, “DevOps and engineering need to be trained on, and measured on, security metrics, not just uptime and service level agreements (SLAs).”
24: Give developers a security framework that they can use in their code
“Developers hate to fix code and even more hate fixing others’ code,” noted Yaron Levi (@0xL3v1), CISO, Blue Cross and Blue Shield of Kansas City. “So a good way to align with the structure that DevOps already loves is to build a security framework that the developers can use in their code such that they do not need to develop these security capabilities by themselves every time they develop something.”
Just like a developer will normally utilize a component repository (e.g., open source) when developing, they could do the same by calling up a security component that already has the protections to their code built in. That way they’ll be able to develop securely a lot faster.
25: Choose threat modeling over documentation
“Use threat modeling to help developers identify the types of requirements they need to build in throughout the development process, instead having to track down all the things they have to do to meet security requirements in the multiple long and not easy to digest ‘security policy documents,’” said Michelle Valdez (@scauzim), CISO, OneMain Financial.
“Ask them how they think a nefarious human could monetize parts of the system they build and operate, and their creativity will likely surprise you,” said Kelly Shortridge (@swagitda_), vp of product management and product strategy, Capsule8.
“Threat modeling is a great way for developers and security to work together, understand each other’s requirements, and identify only those security requirements necessary vs making developers have to try to figure out what they need to do,” said Valdez.
26: Bridge the gap between compliance and IT
“A developer’s first day is never going to be reading up on all the compliance, security, and privacy requirements specific to the organization,” said Sunflower Bank’s Wyatt. “Security’s number one goal should always be to bridge the gap of knowledge that exists between compliance and IT.”
That involves articulating the conditions and criteria that determine what security requirements apply and when they do they’ve got the controls to apply them along with a validation check.
27: Documentation is useless in DevOps, keep pushing to automate
“Security tools and procedures have historically been very manual in nature,” said Everest’s Kail.
“DevOps works because of symbiotic automation, so if you want to be more than a third wheel on a date you need to adopt the same mindset,” said VillageMD’s Walsh.
“Find ways to automate the necessary controls throughout the development lifecycle. Automated application security testing, vulnerability scanning, data protection can be policy as code. Another example is secrets and key management can be written into runbooks (i.e., Lambda in AWS),” explained OneMain Financial’s Valdez.
“More automation means more efficiency and less friction,” added Axonius’ Zeltser.
28: Have a clear definition of what “done” looks like
Policies that are open to interpretation invite inconsistency.
“If you believe 10 developers tried to follow your policy, do you believe that the outcomes would be (CARE) Consistent, Adequate, Reasonable, and Effective,” asked Ross Young, CISO, Caterpillar Financial Services Corporation.
Security policies need a clear definition of what ‘done’ looks like.
It shouldn’t be something vague like all systems should be logged and monitored. Instead, Young suggests a functional test to show if a policy is currently compliant. An example would be all systems shall use FileBeats to send data back to the enterprise data store (Elasticsearch). Each month Kibana dashboards show the percentage of applications that are compliant to this criteria.
29: Auto scan the code before it’s compiled
Static application security testing (SAST) or “static code analysis takes the code that was written and looks at it for potential security vulnerabilities,” said Indiana University Health’s Parker. “It can integrate directly into the workflow and provide automated feedback as part of existing code sync processes.”
“It’s not just scanning code though, there is also automated vulnerability testing that can be performed against APIs and validating DDoS protections under high load [of malicious traffic],” said Michael Wilkes (@eclectiqus), CISO, SecurityScorecard.
30: Security should adopt DevOps, not the other way around
“This whole DevSecOps concept comes from security teams that are not collaborating or embedding themselves with the DevOps philosophy and are looking for a way to be included,” said Emilio Escobar (@eaescob), vp of information security, Hulu. “It’s up to the security teams and their leadership to align their security story to the rest of the organization and realize that their responsibility is to grow the company just like engineering and IT.”
Harking back to some aforementioned tips, Escobar recommends starting with tools that already exist within DevOps’ CI/CD environment. Then, implement security controls via infrastructure as code. And finally, scan source code either using linters, commercial products, and scan for errant code in open source libraries.
Conclusion: Hey security, it’s a DevOps world, accept it
“We generally try to meet people where they are so why is it any different for DevOps,” asked VillageMD’s Walsh.
“If the security team can focus on making it easy to consume security services within the DevOps workflow, then a DevOps-oriented organization will consume those services and product security will improve,” said Atlassian’s Ludwig. “If, on the other hand, the security team asks the DevOps team to change the way they work in order to be more secure, then they will fail.”
And nobody wants to fail. Security wants to be accepted, and developers, IT, and the business have all accepted DevOps just the way it is. It’s time for security to do the same.
The opinions expressed within this article by Nina Wyatt are hers alone and are not associated or representative of her professional network, associations, or her employer, Sunflower Bank.