Why CISOs need to rethink security metrics and move beyond scan counts

Cybersecurity leaders are facing a measurement problem. While threats continue to escalate, many organisations are still relying on outdated indicators to assess how well they’re protected. Meanwhile, according to our research, half of organisations are carrying critical security debt (vulnerabilities that have been left unresolved for longer than a year), which creates persistent exposure that attackers are quick to exploit.  

At the same time, new technologies are accelerating how quickly those risks can be uncovered and connected. Projects like the recently announced Glasswing initiative point to a future where vulnerabilities can be mapped into complex attack paths in a fraction of the time it once took, surfacing weaknesses that may have gone unnoticed for years. But crucially, these advances still centre on discovery, not fixing risk or reducing exposure over time. Alongside this, emerging tools like Anthropic’s Mythos and guidance from organisations like the Cloud Security Alliance, reinforce the need for risk-based prioritising, focusing on what truly matters rather than simply what’s easiest to measure. 

Despite this shift, success is still often defined by how much security activity is taking place, rather than whether risk is really being reduced. More scans, more alerts, and more findings may look reassuring on paper, but they don’t necessarily translate into stronger defences. In fast-moving environments shaped by complex software supply chains and CI/CD pipelines, this disconnect is widening.  

The illusion of progress in security metrics  

Measuring against volume-based KPIs, like the number of scans run, vulnerabilities found and alerts generated, only tracks the effort taken to increase security — not the actual outcome. These traditional KPIs tell you how needed security measures are, but not whether they are stopping anything meaningful.  

For example, a scan finding 10,000 low impact issues might look productive on a dashboard, but at the same time a single exploitable dependency might have been untouched for months, presenting a critical, unresolved security risk. Board members and the C-suite see rising KPI numbers and automatically assume strengthened protection when, in fact, it could be quite the opposite. This blurred measurement line skews the reality of how security teams are tackling security risk. 

These industry wide tropes are inadvertently rewarding security teams for generating noise but not reducing actual risk. And with the average fix time for security flaws rising from 171 days to 252 days over the past five years, the delay to remediation quietly backlogs security risks. Those vulnerabilities hidden in the depths of the supply chain and pipeline are a ticking time bomb. 

With security teams already stretched and struggling to find the capacity for detecting and fixing vulnerabilities, these outdated metrics encourage a culture where security teams and CISOs look “on top of it”, right up until an old, known flaw gets exploited – at which point, it could be too late. 

Why point-in-time scanning no longer works 

With the rapid pace of technological advancement and the apparent rise in successful cyberattacks, point-in-time scanning is now inadequate. It overlooks critical time factors—such as the mean time to remediate or the duration an attacker can operate undetected—which are precisely what attackers exploit. 

Modern attacks happen in the gap between scans, with security snapshots unable to catch moving targets. For CI/CD pipelines, they are obsolete. Code changes multiple times a day and dependencies update automatically.  

And nowadays, an attacker doesn’t even need to evade a scan. They just wait for the next build, commit, or dependency pull and, by the time the scan report is read, the environment it assessed no longer exists. Scanners traditionally inspect source or binaries, but not the inner workings of the build process, meaning a malicious build step can inject code after a scan has passed.  

This happened with the infamous SolarWinds Orion hack, which compromised thousands of organisations (including US government agencies) back in 2020, injecting malicious code into software updates that were then distributed to the unsuspecting customers. 

If the build is already poisoned, then the scan is irrelevant. 

Rethinking security metrics for modern risk 

What needs to change is not the volume of measurement but its purpose. CISOs must move away from reporting on activity towards demonstrating tangible risk reduction. That means prioritising metrics that show how quickly exploitable vulnerabilities are eliminated and how long they remained undetected.  

In an era of continuous delivery and AI-enabled attacks, security performance needs to be tied directly to resilience—in other words, how well systems withstand attack and how quickly they recover.  

Better metrics ultimately lead to better decisions. By focusing on outcomes over output, CISOs can provide clearer insight into the business and align security with operational risk to ensure effort is being directed where it makes the greatest difference.