You Can’t Fix Issues if You Can’t Find Them

Decades ago, software was developed either primarily in-house or purchased from a vendor. The in-house software likely had a few libraries reused across the organization. Purchased software had contracts that specified quality and security requirements. Leaders had a clear view of the software being used in the organization.

‍

Modern software development is different. High-quality open source software has been a boon to developers, but has also increased the complexity of the supply chain. Instead of a few well-defined and contractually-bound relationships with vendors, organizations are sourcing their software from many individuals and groups on an as-is basis. Open source projects range from organized efforts led by companies and foundations to individual solo maintainers working in their spare time. When a vulnerability is discovered, it’s often fixed on a best-effort basis. If this fix doesn’t come fast enough, affected organizations have no recourse except to pitch in and create a patch — or just cross their fingers and live with the risk.

‍

Popular programming language ecosystems like Node.js and Go encourage small packages with narrow functionality, which means applications need to pull in many dependencies. A 2019 research study found that the average package in npm had 86.55 dependencies. The isarray package — a 5-line function that checks whether an object is an array — is a dependency of over 1,900 other packages in the npm Registry. The dependency graph of a modern application can get complicated quickly, with several layers of dependencies, circular dependencies, and other headaches.

‍

The rise of generative AI tools has only added to the complexity. While developers have copied and pasted code snippets from the web as long as the web has existed, AI coding assistants speed up the process. Generative AI can help developers write code more quickly, but without proper testing to verify the code, applications can gain vulnerabilities and unnecessary dependencies.

‍

While code may flow downstream faster than ever, the growing complexity makes it harder to track vulnerabilities and dependencies.

‍

Increasing attacks require a holistic approach

It’s not getting any safer out there. Attacks on the software supply chain are becoming more prevalent as bad actors see how effective they can be. Just in 2024, we’ve already seen several major supply chain attacks. 

‍

• In February, unidentified attackers compromised the GitHub accounts of several developers and the Top.gg organization, a popular chat bot discovery site. The attackers were able to introduce malicious code into the developer toolkit and modify Python applications to pull from a malicious Python package repository.

‍

• In March, a developer discovered a backdoor inserted into the widely-used xz compression utility. Thankfully, only a few users were affected, since the backdoor was discovered before most Linux distributions shipped the affected releases. Had the backdoor spread, it would have allowed attackers to gain privileged remote access to affected computers.

‍

• In June, the director of infrastructure at the Python Software Foundation accidentally added GitHub credentials to a public repository. As with the xz backdoor, this was quickly discovered by a third party, averting what could have been a disastrous supply chain vulnerability for the world’s most popular programming language.

‍

Even today, the 2021 Log4Shell remote code execution vulnerability remains a prime example of the risks. In late 2023 — a full two years after Log4Shell was disclosed and fixed — researchers found that 38% of applications were still using a vulnerable version of Log4j.

‍

Supply chain risks don’t just come from malicious attacks. In July, a faulty update in CrowdStrike’s Falcon platform caused global IT outages. Delta Airlines claims a loss of $500 million or more due to the outage. It’s clear that technology leaders must take a constant and holistic approach to securing their software supply chain. Vulnerabilities can come from any part of the software development life cycle — from compromised developer workstations to malicious software dependencies to attacks on code repositories, build systems, and production servers. But guarding against attacks is only part of the story. You can’t prevent every vulnerability, so how do you address the ones you know of?

‍

What does it mean to know your software? 

You can’t fix issues if you don’t know they exist. This is an obvious statement, but it’s hard to know what software is in the supply chain of a modern application. To secure the software supply chain, you have to know its components, how they interact, and their security postures.

‍

Knowing the software dependencies that you pull in is only the first step. You have to know about your dependencies’ dependencies, and their dependencies, and so on. What vulnerabilities exist in those packages? Do the developers follow secure practices? Are the build and delivery systems secure?

The problem isn’t a lack of information — the information is out there. The challenge is collecting and correlating all of the facts into a coherent and searchable set of relationships. When you can interrogate your supply chain, you can know it. With a holistic understanding of the software supply chain, developers and security engineers can perform fast and accurate queries to quickly identify risks and devise effective remediation plans.