Vulnerability management (VM) has become synonymous with frustration. Breaches often result from exploited vulnerabilities that are known, yet not patched. The compromise was avoidable. Why couldn’t we stop it?

The problem stems from security “trying to manage a mountain of work they usually have little to no control over by pushing other overtaxed teams, such as IT and engineering, to remediate during non-ideal times,” said Yaron Levi (@0xL3v1), CISO, Blue Cross and Blue Shield of Kansas City.

While not sexy, the return on investment (ROI) for VM is fantastic, noted Mark Eggleston (@meggleston), vp, chief information security and privacy officer, Health Partners Plans, “VM encompasses the first three Center for Internet Security (CIS) controls – those which are not only basic but can return the biggest bang for your buck.”

Sure VM doesn’t have the dazzle of hunting advanced persistent threats (APTs), but if you get a handle on it you’ll stun your C-suite. Read on for some expert tips.

Thanks to our sponsor, Vulcan Cyber

Editor’s note: This article is part of CISO Series’ “Topic Takeover” program. While the article sponsor, Vulcan Cyber, and our editors agreed on the topic of vulnerability management, all production and editorial is fully controlled by our editorial staff with no sponsor participation.

Vulcan’s vulnerability response automation platform allows enterprises to automate their TVM programs. Vulcan integrates to existing IT DevOps and security tools to fuse enterprise data with propriety intelligence which allows to accurately and subjectively priorities and remediate vulnerabilities – either using a patch workaround or compensating control.

1: Don’t go it alone

“People still tend to work too much in silos. Vulnerability management is a vital area of self-protection that must involve more than just IT and security experts,” said Steve Prentice (@stevenprentice), tech journalist,

Vulnerability management is everybody’s job. It’s a mistake to think it should be managed solely by security.

“InfoSec and operations need to treat tracking remediation as a joint project,” added Richard Greenberg (@RAGreenberg), InfoSec officer, LA County Department of Public Health.

“Ownership is key,” exclaimed Brian Fricke (@brianfricke), CISO, BBVA Compass. “IT personnel need to know what asset groups they own, when they are supposed to be patching them, and they must be empowered to run ad hoc scans on those asset groups.”

Got feedback? Join the discussion on LinkedIn.

2: Give IT teams direct access to vulnerability scans

“Whenever possible, enable IT teams to directly access vulnerability management platforms to access scan data versus receiving reports,” said Branden Newman, CISO, adidas. “I find that this collaborative approach eases tension between IT and security teams because they experience the entire process in a way that they understand better than reports or tickets.”

3: Become best friends with the help desk

“Have a true alliance with your infrastructure and applications teams, particularly the help desk. These folks are the ‘boot on the ground’ when it comes to deploying patches or mitigations and the first to field trouble tickets or complaints raised by users,” said Josh Stabiner, CISO, General Atlantic. “Be respectful of their bandwidth and projects. Be honest and measured regarding risk; not every advisory is a ‘DEFCON 1’ situation, even if the vendor calls it an ‘emergency’.  Be prepared to discuss options for alternative mitigations in the event patches can’t be deployed on the normal cycle… If you take a collaborative tack and understand the process from their perspective you’ll find that they’ll reciprocate in kind and the frustration normally associated with vulnerability management evaporates and is replaced by teamwork devoted to confronting a common challenge.”

4: Let remediation happen on the technology owners’ platform of choice

Don’t force those who don’t work in security to learn security’s solution when they have a solution they already know and use.

“If your vulnerability management tool has great tracking capabilities to assist with remediation, it does not matter. The export function is more important. This is because tracking and fixing needs to take place where the owners of the target systems and applications live,” said Allan Alford (@AllanAlfordinTX), CISO of Mitel, and co-host of Defense in Depth podcast.

5: Connect vulnerability management scanners with ticketing

“The biggest challenge that I have is tracking and remediating vulnerabilities,” said Dr. Jackson Muhirwe (@muhirwe), deputy CISO, University of California, Davis. “We have great tools for scanning and discovering vulnerabilities however it is always the actions that are taken after that become a challenge.

“Integrate your vulnerability management tool with your ticketing and configuration management database (CMDB). If you are able to do this, then it would be possible for you to easily track and follow up with all the leads that have been assigned remediation tasks.”

6: Accept the fact that you can’t fix everything

“One of the biggest sources of frustration in vulnerability management comes from not having a strategy and a blanket ‘fix everything’ mindset,” said Hemanth Srinivasan, senior manager, application security, Autodesk. “Not all bugs will and can be fixed.”

Emilio Escobar (@eaescob), head of information security, Hulu, follows a simple pragmatism philosophy.

“Accept the reality that not every vulnerability is equal,” said Escobar. “Focus more on things that truly do matter.”

7: Prioritize vulnerabilities by criticality, risk, and simplicity

“You can’t boil the ocean and you shouldn’t,” argued Tomas Maldonado (@tomas_mald), CISO, International Flavors and Fragrances. “Take a risk-based approach to vulnerability management.” 

“Prioritize assets to be patched based upon the risk each specific asset presents to the business,” said Scott Foote, CISO and founder, Phenomenati.

When you begin, “focus only on the ones that can be remediated in a single cycle with available resources,” added adidas’ Newman.

It’s a simple formula. If you want an easy win with high ROI, target vulnerabilities that are business critical and easy to fix, first.

8: Asset prioritization will be a challenge

Figure out what the company’s most valuable assets are, protect them, and make sure all vulnerabilities are patched.

Wouldn’t it be wonderful if it were that easy? The C-suite just happens to have a handy printed list of all their most valuable assets ranked in order of importance.

Too bad that never happens.  

“The CISO’s dirty little secret is that nobody tells you what the business crown jewels are on day one of the job,” said Rich Mason, former VP and CSO, Honeywell. “This is because the C-suite doesn’t know, can’t agree, or doesn’t trust you enough yet to fully disclose.”

Since cyber-attacks don’t follow the board’s timeline of crown jewel discovery, this puts the CISO and the organization in a very uncomfortable position. To mitigate, do whatever you can to point out the conundrum and get working on creating that list.

9: Less risk refining, more patching

While it’s important to determine your most critical assets, don’t get bogged down on it.

“Stick to the MSP – map, scan, and patch,” said Health Partners Plans’ Eggleston. “Resist the temptation to purchase additional platforms which sit on top of VM to refine risk scores. While these add-on tools can be helpful, most shops don’t require the complexity.”

10: Agree on what is and isn’t a priority vulnerability

“There is nothing that introduces as much friction in a vulnerability management program than two teams not aligned on what’s critical, medium, and low,” said Suzie Smibert, CISO, Finning International. “The velocity at which one team will react and remediate a vulnerability is highly dependent on how critical and important they believe it to be; having a consistent language brings clarity around expectations.”

Mark Stanislav, director of security engineering, Duo Security, notes one of the problems is vulnerability scanning presents most concerns as a binary decision. It’s either vulnerable or not.

“In practice, an application that has a known vulnerability doesn’t mean that the in-use technology is actually at risk. Without technical engineers to evaluate what ‘critical’ patches are actually critical, organizations may spend time mitigating for attacks that aren’t even possible,” said Stanislav who noted the complete opposite is also possible. A small single vulnerability may open the floodgates to larger vulnerabilities is an organization’s software.

You simply need security engineers and operational engineers to work together to evaluate what is critical and not rely solely on vulnerability scans.

11: Run threat intelligence against your vulnerability scans

Unfortunately, the aforementioned advice simply isn’t manageable when you have to determine the validity of so many vulnerabilities to your environment.

To prioritize which ones to look at first, Gary Hayslip (@ghayslip), CISO, Webroot, suggested, “Use threat intelligence to highlight those vulnerabilities actively being used in the wild against organizations. With that added insight I provide it to my patch management teams so they focus on those issues first.”

12: Create a clear governance process

“The most important first step is to establish a formal governance process for the identification, prioritization and patching of vulnerabilities,” said Larry Ponemon (@ponemon) Chairman, Ponemon Institute. “This governance process should establish clear lines of accountability and ownership of vulnerability management.”

Of course that’s easier said than done, but when people start pointing fingers, “Blame the process instead of people,” advised Honeywell’s Mason. “It takes the emotion out of the issue.”

Simply look at how your existing process prioritizes vulnerabilities, what’s being identified, automated, and acted upon, said Mason.

“The governance process should centralize the reporting of key performance indicators (KPIs) to the CISO, such as the time to identify the vulnerability, time to patch, and cost associated with the vulnerability management failure,” added Ponemon. 

13: Add business context

To help with governance, add business context to each vulnerability. Who and what application does it affect? Who’s responsible for patching? How critical is it and who has access to it?

“This business context can significantly improve your vulnerability program by helping responsible teams prioritize based on business criticality and severity and ensuring each vulnerability has a clear owner for remediation,” said Claude Mandy, owner, Five Consulting, LLC.

14: Separate patch and vulnerability management

Patch management and vulnerability management don’t have to be synonymous. Vulnerabilities have to be discovered or reported, while you can always expect patches at regular intervals.

“Patch management should just be an operational task that gets done at a certain cadence usually every month,” said Hulu’s Escobar. “This addresses most vulnerability management service level agreements (SLAs) as required by regulatory compliance.”

15: Make it a habit

“We need to approach [vulnerability management] the same way we approach mouth hygiene. As parents we teach each kid to brush their teeth when they wake up and before they go to bed. We don’t remind them weekly or monthly and give them 72 hours to complete the task,” said Blue Cross and Blue Shield of Kansas City’s Levi. “We need to convert DevOps into DevSecOps and have every individual own his or her little security responsibility until it becomes a habit… We need to stop treating security as some kind of black magic that is only known to the very few skilled wizards. We need to educate everyone on the basic things they can and must do.”

16: Gamify vulnerability remediation

Since vulnerability management is process and people intensive, if you were to gamify the process, you could make it fun and eventually a habit.

“If you have an organization where the responsibility for remediating vulnerabilities is quite segmented, such as by region, you could implement a ranking or scoring system that puts the teams in competition,” suggested adidas’ Newman. “The success of this is very dependent on the company culture and you can even spice it up a bit with awards for the team that ranked the highest over the year.”

17: Improve coordination with data integration

“Feed vulnerability data into a central location where reports can be consolidated, prioritized, and turned into specific IT action items,” said Jeremiah Grossman (@jeremiahg), CEO, Bit Discovery. “Over time this data can also be used for metric tracking to better understand where the organization can best improve, such as patch coverage or patch speed.” 

While there should be ‘IT action items’ that doesn’t necessarily mean putting the weight of the vulnerability management world on IT’s shoulders. As mentioned before, vulnerability management should be a responsibility of all department

18: Don’t start with vulnerability management reports

While Grossman’s advice is valuable, it’s designed for vulnerability management refinement.

“Vulnerability management analysis work comes in a later phase. Those VM reports lack the context and the functional use of the asset perspective that only the business can provide,” said Mark Butler (@mbinc), CISO and SVP, MegaplanIT.

The beginning of a vulnerability management program, said Butler, should begin with asset inventory, configuration management, hardening systems, privileged access controls, and auditing.

“After you get those foundational disciplines in place, then you can add the active exploit prioritized vulnerability data on top to direct any vetted remediation work against threats,” said Butler. “Teams typically lazily do this in reverse and it’s like staring at the backside of broken process and wondering why there’s only one thing that comes out of it.”

19: Know your assets

Why isn’t this tip #1?

It’s bizarre that after decades of network development and sprawl, one of the hottest issues in 2019 is what the heck do we have?

It just so happens that #1 and #2 of the CIS top three are hardware and software asset inventor.

“While vulnerability management is critical to understanding the current organizational risks, this visibility is only as good as the accuracy of the known attack surface — Internet-accessible assets,” said Bit Discovery’s Grossman.

“From a technical perspective inventory management trumps all else. If you don’t know what assets you have and where they are then it’s impossible to manage vulnerabilities effectively,” said General Atlantic’s Stabiner. “Create an inventory of all five asset classes – devices, applications, networks, users, and data – and implement ways to keep these inventories up to date in an automated fashion. Be sure your inventory includes the asset owner, location, and current version information where applicable.”

“The unknown and unmanaged systems are a battle, and without a solid and up-to-date inventory with relevant data you will always be chasing the unknowns in your environment or be being exploited via them,” said Jack Daniel (@jack_daniel), Tenable Network Security.

20: At least start asset discovery

Trying to understand what you have is a monstrous task. But don’t let full discovery of your inventory hold up your vulnerability management program.

“Don’t let perfect be the enemy of the good. Just start,” advised Michael Strong, CISO, GCI General Communication. “Discover the key assets and add more over time. Don’t let your program stall out because you’re trying to discover everything before starting the next step.”

21: Reduce your attack surface

“Make a concerted effort to reduce your end-point attack surface so that the need for vulnerability management is on a constant reduction trajectory. The less vulnerability management you have to do, the more effective you can be with the vulnerability management you need to do,” said Elliot Lewis (@ElliotDLewis), CEO, Keyavi Data (formerly Encryptics).

For example, one obvious place to look are the effectiveness of your security mitigations, said Lewis. Given that an organization’s cybersecurity state is always fluid, what was once effective may no longer be and that could result in an expanding attack surface.

22: Educate to reduce complaints

“The complaint I receive the most is about the personal disruption or time it takes to perform patching or upgrades,” said Jim Marshall, director, enterprise architecture and InfoSec officer, Versiti.

This is understandable as individuals are just trying to do their jobs and any downtime is preventing them from doing their jobs.

“The business should have an understanding as to why it’s important to maintain hygiene,” said International Flavors and Fragrances’ Maldonado.

Marshall’s education took the form of a monthly newsletter citing examples of risks and events and how his colleagues could protect themselves.

“Since I started [the newsletter,] about six months ago, the complaints have been less, and the support has risen as people begin to see themselves as part of the cybersecurity solution rather than just a victim of its’ requirements,” said Marshall.

23: Automate the process you want to happen

“Don’t automate a bad process. Don’t automate the way processes are currently working. Automate the process you want to occur,” said Joey Johnson, CISO, Premise Health.

It often doesn’t make sense to automate the process you currently have because human-managed systems have a way of constantly morphing, changing various needs for decisions, actions, and approvals, noted Johnson.

24: Automate in stages

Claim victory at each step in any vulnerability automation effort.

“The ‘win’ isn’t always in getting to the final goal out of the gate,” noted Premise Health’s Johnson. “Focus on things you know will continue, and require complex communication and alignment.”

A great example is Java patching. It’s frequent and affects many applications. As a result, it requires lots of humans to collaborate and reach consensus. Instead of trying to patch an issue right away, Johnson suggests starting with automating QA testing routines to even determine if the patch can be applied to target environments. Once that works, and you get agreement, then you can move to the next stage of automating the workflow of delegating patching tasks as Java releases emerge.

Since this process will involve a lot of people, accept the fact that you’re going to feel a little pain.

“Automation can make things more complex before it makes them easier,” said Johnson. “For the workforce tasked with implementing automation routines it takes a while before the promised land of ROI becomes real.”

25: An agreeable downtime window is a vulnerability mitigation dream

“All of this advice is all fine and good, but it leaves out the most important part: when can you actually fix the broken stuff?” asked Taylor Lehmann (@BostonCyberGuy), CISO, Wellforce.

“Many organizations force patches with little consideration for the impact to the business,” said Phenomenati’s Foote.

“Getting the regular, business-friendly downtime window to actually fix problems is almost more important than your scanner or your prioritization math.”

Work with everyone to figure out when you can create an acceptable maintenance window.

“Getting agreement from the business to actually take systems down is very hard. It requires negotiation and trust building. Maintenance windows provide IT and information security a tool that allows them to operate without impacting the business and customers,” Lehmann. “Without a maintenance window, vulnerability management will only occur during times of crisis with a high likelihood of unintended negative outcomes that follow… The crisis-driven approach increases outage durations, feeds a distrust for IT, and lowers confidence in the security team.”

Conclusion: Your first line of defense

It’s clear that nothing is static in networking or operations. Ever-growing and -changing vulnerabilities are going to be a constant. Success requires involvement from all departments and if vulnerability management is made into a habit it will ingrain good security hygiene across your entire staff.

Got feedback? Join the discussion on LinkedIn.

Creative Commons photo attributions to Olivier Engel, Peter Richardson, Fiat Chrysler Automobiles, and Josh Morgan.