A year ago, as Russia amassed troops at its border with Ukraine and the Covid-19 Omicron variant began to surge around the world, the Apache Software Foundation disclosed a vulnerability that set off a frenzy across the global tech industry. The bug, known as Log4Shell, was in the ubiquitous open-source logging library Log4j and exposed a wide range of applications and services—from popular consumer and enterprise platforms to critical infrastructure and internet-of-things devices. Now, after weeks of intensive remediation last December and a year of cumulative progress on patching, Log4Shell no longer poses the universal threat it once did. But researchers warn that the vulnerability is still present in far too many systems worldwide, and that attackers will be successfully exploiting it for years.
Many critical vulnerabilities get discovered every year that are of high urgency to address, but Log4Shell was unusual because it was so easy to exploit wherever it was present, with few caveats or subtleties for attackers to navigate. Developers use logging utilities to record operations in a given application. All attackers need to do to exploit Log4Shell is get the system to log a special string of code. From there, they can take control of their target to install malware or mount other digital attacks. Loggers gonna log, so introducing the malicious snippet can be as easy as including it in an account username or sending it in an email.
“Logging is fundamental to essentially any computer software or hardware operation. Whether it’s a phlebotomy machine or an application server, logging is going to be present,” says David Nalley, president of the nonprofit Apache Software Foundation. “We knew Log4j was widely deployed, we saw the download numbers, but it's hard to fully grasp, since in open source you're not selling a product and tracking contracts. I don’t think you fully appreciate it until you have a full accounting of where software is, everything it's doing, and who's using it. And I think the fact that it was so incredibly ubiquitous was a factor in everyone reacting so immediately. It's a little humbling, frankly.”
The situation resonates with larger discussions about the software supply chain and the fact that many organizations do not have an adequate accounting of all the software they use in their systems, making it more difficult to identify and patch vulnerable code. Part of the challenge, though, is that even if an organization has a list of all the software it's bought or deployed, those programs can still contain other software components—particularly open-source libraries and utilities like Log4j—that the end customer isn't specifically aware of and didn't intentionally choose. This creates the ripple effect of a vulnerability like Log4Shell as well as the long tail of patching, in which organizations either aren't aware that they have exposure or don't recognize the urgency of investing in upgrades.
Meanwhile, attackers are still actively exploiting Log4Shell everywhere they can, from criminal hackers looking for a way into targets' systems to Chinese and Iranian state-backed attackers deploying the exploit in their espionage campaigns.
“Log4Shell is one that’s going to show up in data breaches for the next decade as part of the root cause—all it takes is one instance of Log4Shell to be vulnerable,” says Dan Lorenc, CEO of the software supply-chain security company Chainguard. “Thankfully, most consumers didn’t feel an impact last year, because the severity of it was so high that folks scrambled over that terrible weekend and throughout the holidays in a race with attackers. But there's an economic impact to that, a massive effort cost to do that remediation. And we’re not going to be able to scramble everybody for something that is even slightly less severe.”
Apache had to scramble at the beginning of December 2021 to be ready to release patches for Log4Shell when it publicly disclosed the situation on December 9 of last year. As a result, researchers quickly found edge cases and workarounds to the patches, and Apache was forced to release multiple iterations, which added to the confusion.
“This thing was everywhere, truly everywhere,” says Jonathan Leitschuh, an open source security researcher. “Attackers were jumping on it, the security community was jumping on it, payloads were flying everywhere.”
Researchers say, though, that Apache's overall response was solid. Nalley adds that Apache has made changes and improvements in reaction to the Log4Shell saga and hired dedicated staff to expand the security support it can offer to open-source projects to catch bugs before they ship in code and respond to incidents when necessary.
“In a short period of time, two weeks, we had fixes out, which is great,” Nalley says. “In some ways, this is not a new situation to us, and I would love to say we dealt with it perfectly. But the reality is, even at the Apache Software Foundation, this highlighted what a responsibility we have to everyone who consumes our software.”
Going forward, the more concerning aspect of the situation is that, even a year later, roughly a quarter or more of the Log4j downloads from the Apache repository Maven Central and other repository servers are still full of vulnerable versions of Log4j. In other words, software developers are still actively maintaining systems running vulnerable versions of the utility or even building new software that is vulnerable.
"The reality is that the majority of the time when people are choosing a vulnerable open-source software component, there's already a fix available,” says Brian Fox, cofounder and chief technology officer of the software supply-chain firm Sonatype, which operates Maven Central and is also a third-party Apache repository provider. “I've been around for a long time, and I'm jaded, but that really is shocking. And the only explanation is that people really do not understand what’s inside their software.”
Fox says that after the initial scramble to address Log4Shell, version downloads in Maven Central and other repositories hit a shelf where roughly 60 percent of the downloads were of patched versions and 40 percent were still of vulnerable versions. Over the last three months or so, Fox and Apache's Nalley say they've seen the numbers fall for the first time to roughly a 75/25 percent split. As Fox puts it, though, “After a year, a quarter of the downloads is still pretty terrible."
“Some people feel Log4j was a big wake-up to the industry, a collective freak-out and awakening,” he says. “And it has helped us really expand upon the message about software supply-chain security, because no longer were people in denial. The thing we were all talking about was real now' we were all living it. But the peer pressure alone of Log4j should have forced everyone to upgrade, so if we can’t get this one to 100 percent, what about all the other ones?”
For security researchers, the question of how to address the long tail of a vulnerability is always present. And the issue applies not just to open-source software, but proprietary systems as well. Just think about how many years it took to move the last 10 percent of Windows users off of XP.
“With these worst-case scenarios—black swan events in open source—you just know they're going to keep happening, because the community has gotten a lot better at reacting, but the pace of open-source development is even faster,” ChainGuard's Lorenc says. “So we have to find the balance of prevention and mitigation, and keep coming up with efforts to reduce the frequency as much as possible. It's like The Simpsons meme when Bart says, ‘This is the worst day of my life.’ And Homer says no, ‘The worst day of your life so far.’”
