When a vulnerability is discovered in production, a non-technical executive is bound to ask, “Why didn’t we discover this before we put it in front of customers?”
The question is understandable and, frankly, justifiable. After several decades of software development and software testing, why are we still discovering security flaws when it’s too late?
The short-sighted answer might be: “We don’t run the right tests until we’ve written all the code.” The far-sighted answer might be: “Software development processes have evolved over time, but are still largely sequential despite moving from waterfall to agile methodologies.” Even at agile organizations, QA and security operations groups feel like the caboose on the release train.
In actuality, a vast majority of the vulnerabilities are in code we never wrote ourselves. Devs use open-source components and open-source operating systems that can pack hundreds of vulnerabilities in them. But since the devs didn’t write the code, remediation options are limited to:
- Investigate if the vulnerability is exploitable or reachable, reducing lateral movement in case of a breach.
- Patch the code if a patch exists. Often, it doesn’t in which case the issue remains unresolved. And if it does it often entails an entirely new QA cycle adding time and expense.
- Replace with another component. But these new components often bring their own vulnerabilities.
- Try to fix the vulnerability by developing ones own bespoke patch and the ensuing host of backward compatibility nightmares it brings.
- Redesign the software to omit the problematic component. This is usually neither feasible nor economical, and it’s hard to determine if a component or dependency is needed at run time.
No matter the option, devs are stuck with limited options and unlimited vulnerabilities to fix.
The industry’s answer is to scan early and often. This shifts scanning processes left, though it’s more of a “spread” of relevant, time-consuming scans across the entire SDLC. It sounds like the right thing to do—but is it?
The approach is called “shift left” or “shift left security.” It started gaining industry traction in 2018, and for good reason. Product teams rightly decided to stop waiting until the last steps in the SDLC to detect problems. For many development teams, though, it feels less like sharing product security concerns and more like adding work to their increasingly-impossible workloads.
The principle of shift left is a difficult road paved with good intentions. There’s a much better way to prevent vulnerabilities from getting to production that’s a win-win-win for product teams, business leadership, and customers.
What Shift Left Does Right
Devopedia cites a 2016 paper by Stefan Friese saying, “Shift left is less about problem detection and more about problem prevention.” How does the practice accomplish problem prevention?
In principle, “shift left” is about much more than testing and software scanning. All aspects of the software development cycle can be tested. For example, software designs and requirements can be tested for clarity, measurability, completeness, logical consistency, correctness, and security. All of this is achievable when using software requirement specifications and developing maturity around requirements gathering and documentation.
Furthermore, infrastructure and vulnerability scanning don’t need to wait until a staging deployment. In fact, teams can get a good understanding of the security risks and software attack surface well before they get to production.
A holistic shift left approach involves appropriate types of testing and scanning for each stage of the development lifecycle. This is what mature and responsible software organizations do to establish a foundation of good software hygiene.
Organizations with high software maturity are cognizant of the flaws in every stage of the life cycle. Shift left detects and prevents problems using strong automation, governance, and autonomy. As a result, software can be developed faster, at higher quality, and with fewer flaws reaching production. Mature organizations are fixing problems before they surface, increasing developer awareness of SecOps teams’ concerns (and vice versa), and reducing the significant costs of fixing code later in the SDLC.
The theory behind shifting left sounds awesome! At RapidFort, we are all for mature and responsible software development—but there’s more than one flaw to address, even at mature, responsible organizations.
What Shift Left Doesn’t Do Right
There are many good articles describing the problems with shift left in software testing and DevSecOps concerns in pre-production.
One of the biggest problems is organizational leadership buy-in. Many product companies are paid and incentivized to deliver value to the market with acceptable risk. But what are “acceptable risks” and how are those risks determined? The problems here are multifold: identifying and documenting the risks, adequately communicating the risks, and understanding, accepting, and managing the risks.
The process of shifting left can be dramatic—and for large organizations, quite costly. Changing SDLC processes, adding lightweight governance, incorporating input and involvement from non-software groups (e.g. operations and security teams), and motivating teams to change are huge impediments to delivering value to markets at high velocity.
Furthermore, poor implementation often leads to overworked teams. Despite good intentions, an unclear vision of a functional shift to the left can have catastrophic effects on already overwhelmed delivery and support teams.
It’s not “shift left” specifically that leads to problems: It’s the process of getting there. It’s not unlike the transition from waterfall to agile, or from monolithic software to microservices, or from hosted services to serverless.
In fact, the biggest problem with shifting left cannot possibly be solved by shifting left.
What Shift Left Will Never Accomplish
Shift left is a powerful concept that works really well for discovering software problems in custom code—but 80-90% of the code in a container isn’t custom. None of that third-party code can be fixed or modified by the organization deploying it.
For example, let’s say you’ve built a container with your custom code in it. It’s running a Redis database and an open source operating system like Ubuntu. The SBOM and SCA scan might say you have 500 vulnerabilities in Ubuntu and another 250 in Redis. Your developers have no tools or abilities to modify the operating system or the database.
You decide to make a custom patch for the most critical vulnerabilities and then the next versions of Redis and Ubuntu come out a few weeks later. Now you need to see if your patch is backwards compatible or if it needs to be updated. From now on, you need to reconcile third-party patches with your custom patch. This creates a massive spider web of problems for your teams to manage.
The only thing you can do for open source software is: ensure you’ve applied all the patches and use the fewest number of components. You can design around the flaws, but the big gap in understanding is that you can’t get rid of OSS vulnerabilities by patching them yourself.
Shift Left Security Needs Software Optimization
There is only one shift left strategy for minimizing the software attack surface: software optimization. The best solution available today is to eliminate unused components from your workload and use shift left to ensure your custom code is safe.
One of our Enterprise clients discovered 108,384 vulnerabilities across 128 containers before using RapidFort. After scanning and optimizing their infrastructure, RapidFort reduced the vulnerability count to 6,503—a 94% reduction in vulnerabilities!
We strongly believe in shift left, but it’s not a silver bullet for software security. Every day we scan and measure container workloads and find that 50-90% of software components can be eliminated with no risk to workload functionality.
At the beginning of this article, we outlined five vulnerability remediation options available to dev teams today. If you are looking for a software optimization solution that can eliminate every unused component in your infrastructure, our scanner is available for free.