The Untold History of America’s Zero-Day Market

The lucrative business of dealing in code vulnerabilities is central to espionage and war planning, which is why brokers never spoke about it—until now.
A burglar pries open the back door of a computer with an American flag.
Illustration: Elena Lacey

This story is adapted from This Is How They Tell Me the World Ends, by Nicole Perlroth.

Getting to the bottom of the zero-day market was a fool’s errand, they told me. When it came to zero-days, secret vulnerabilities in code, governments weren’t regulators; they were clients. These holes made up the raw material for their espionage tools and cyberweapons. They had little incentive to disclose a highly secretive program, which dealt in highly secretive goods, to a reporter like me.

“You’re going to run into a lot of walls, Nicole,” Leon Panetta, the secretary of defense at the time, warned me. Michael Hayden, the former NSA director, laughed when I told him what I was up to. “Good luck,” he said, with an audible pat on the back.

Word about my quest that year, 2013, traveled fast. The zero-day dealers, the men who dealt in vulnerabilities and the code to exploit them, prepared for me with bug spray. I was disinvited from hacking conferences. At one point, someone on the dark web offered good money to anyone who could hack my email or phone. But I’d glimpsed enough to know I had to keep going.

The world’s infrastructure was racing online. So was its data. The most reliable way to access those systems and data was a zero-day exploit. Zero-days had become a critical component of American espionage and war planning. The Snowden leaks made clear that the US was the biggest player in this space, but I knew that it was hardly the only one. Oppressive regimes were catching on, and a market was cropping up to meet their demand. There were vulnerabilities everywhere, many of them of our own making, and powerful forces—including our own government—were ensuring it stayed this way. Many did not want this story to be told.

It took years to find a zero-day broker from the market’s earliest days who would talk. Many never responded. Some just hung up. One told me that not only would he not speak with me about the market, but he’d already warned everyone he knew not to. If I continued on this thread, he told me, I would only be putting myself in “danger.”

Most just feared for their bottom line. In their line of work, keeping your mouth shut was essential. Every deal required discretion, and most were wrapped in nondisclosure agreements and, increasingly, classified. The most profitable brokers kept their zero-day business, the sheer fact there was a business, a secret. The more discreet the broker, the more governments coveted his business. A broker’s quickest road to bankruptcy was to talk to the media. It still is.

This was not a matter of paranoia. Brokers have a case study in the perils of talking to a reporter about the zero-day market: a well-known South African exploit broker, based in Bangkok, called “the Grugq.” The Grugq just couldn’t help himself. Unlike most zero-day brokers, who avoid any platform that leaves a digital trace, the Grugq is on Twitter, where he has more than 100,000 followers. In 2012 he made the fatal mistake of openly discussing his business with a reporter. He would later tell me he was speaking off the record, but he was also happy to pose for a photo next to a large bag of cash. When Andy Greenberg’s story appeared in Forbes magazine, the Grugq became persona non grata. Governments stopped buying from him.

No broker was looking to follow in his footsteps, to forsake their fortune and reputation for fame or transparency. And so I reported out this market the only way I knew how, scratching at what was public and working my way in from there—until I found a zero-day dealer who would relay the industry’s untold history.

Every market starts with a wager. I learned that the zero-day market—or the public face of it at least—started with 10 bucks. That was what John P. Watters paid to acquire the Chantilly, Virginia-based cybersecurity company iDefense in the summer of 2002. Watters, a Texan with virtually no knowledge of cybersecurity, figured it was a fair price for a company that was hemorrhaging a million dollars a month, with no obvious plans to make it back. Employees hadn’t been paid in weeks. The Nasdaq would reach its lowest point of the dotcom crash the following month. Five trillion dollars in paper wealth up and vanished. In another two years, half of all dotcom companies would disappear. Even employees didn’t think iDefense stood a chance.

The company sold “threat intelligence” to its customers at the big banks and some government agencies. Usually that meant flaws in software that could be used to break into their networks and eventually steal their data. But what iDefense offered was hardly unique; the raw information was freely available on hacker forums like BugTraq, where hackers dumped and traded the vulnerabilities they discovered at all hours of the day and night. And now there was a good chance the company would lose access to that too: The same day Watters walked into iDefense headquarters, Symantec scooped up SecurityFocus, the company that ran BugTraq, for a cool $75 million. iDefense would never be able to compete with Symantec’s deep pockets. And if Symantec shut off access to BugTraq, iDefense was screwed.

Two young hackers in iDefense’s research lab offered Watters the Hail Mary. David Endler had spent his early career at the NSA. Sunil James was just a few years out of college. Both were tinkerers in their own right, and they knew there was an untapped pool of hackers the world over, discovering vulnerabilities at all hours of the day and night. For years, these hackers had no good options. When they discovered vulnerabilities in Oracle software or Sun Microsystems, there was no 1-800 number to call. When they did find an engineer at the company to report a coding error to, hackers were often ignored. If they received any return call, frequently it came from a lawyer warning them to stop poking at their products. It didn’t help that code, not diplomacy, was their strong suit. And even when companies did fix the issues, those fixes often contained glaring errors of their own. What these hackers offered was free quality assurance, and more often than not they were penalized for it.

Endler and James relayed this dynamic to Watters one day. Vulnerabilities were being introduced into code every day. Blackhat hackers were exploiting these vulnerabilities for profit, espionage, and digital mayhem. The whitehats who wanted to do the right thing were losing incentive to report them to vendors. Companies much preferred threatening hackers with lawsuits to fixing the bugs in their products. Inevitably, the losers in this quasi-abusive arrangement were iDefense’s customers: The banks and agencies that relied on vulnerable software. Their systems were being left wide open to attack.

“What if we started a program?” Endler pitched Watters. “We could pay hackers to turn over their bugs.” iDefense could still turn the bugs over to tech companies to patch, but until a patch was available, iDefense could offer customers workarounds to protect themselves. iDefense would be offering its customers something concrete and unique. It wouldn’t be just another noisy feed. And it would allow Watters to justify raising fees and turn a profit.

Any other CEO might have balked. But it was the Wild West elements of the cybersecurity business that brought Watters to iDefense in the first place. And so, in 2003, iDefense became one of the first companies to publicly throw open its doors to hackers and pay them for their bugs.

At first, James and Endler didn’t know what they were doing. There was no market, no competitive program, as far as they knew. They drew up a quaint little price list, $75 for this bug, $500 for that. Of the thousand vulnerabilities submitted in the first 18 months, half were crap. They contemplated turning hackers away, but they knew they needed to build trust so hackers would come back to them with bigger and better hauls.

It worked. Hackers in Turkey, New Zealand, and Argentina, even 13-year-olds in Kansas, started turning over bugs that revealed how attackers could burrow into iDefense’s customers’ systems via antivirus software or intercept passwords and siphon off data between a user and their web browser.

As the program gained notice in 2003, Watters started getting calls. Most were from big tech companies furious that he was inviting, paying, hackers to probe their products. But in late 2003 and 2004, he started hearing from a new breed of caller: They claimed to work for government contractors Watters had never heard of, and they asked if he would consider withholding some of hackers’ bug submissions from vendors and clients in exchange for higher profits. That bug iDefense paid a maximum of $10,000 bucks for? These callers offered up $150,000, so long as iDefense didn’t tip off anyone—not its customers, not the software vendors. These bugs would make their way into espionage tools and cyberweapons and be used against America’s enemies, and nobody would ever have to know they existed. The fact these callers were willing to pay so much for bugs blew Watters’ mind.

When Watters declined, the contractors pivoted to patriotism. The old "Do it for your country" pitch. These bugs would enable the government to spy on terror cells, suspected Russian spies, double-dealing Pakistani intelligence officials. Watters was a patriot, but he was also a businessman. “It would have killed us,” he told me. “If you’re co-conspiring with the government to leave gaping holes in core technology used by your customers, you’re inherently working against your customers.”

His callers eventually got the message. But Watters could see the wind shifting. Hackers started demanding six figures for bugs that only a year earlier they had gotten a few thousand dollars for. They alluded to other options. Mystery outfits, like Digital Armaments, started popping up online, offering high five-figure bounties for zero-days in Oracle, Microsoft, and VMWare products. Beyond a bare-bones website registered in Tokyo, it wasn’t clear who the customers of these bugs would be. They solicited “exclusive rights” and said only that they planned to notify tech companies “eventually.”

Soon, iDefense was getting priced out of the very market it helped spawn. Watters read the writing on the wall. In July 2005 he sold the company he bought for ten bucks to Verisign for $40 million. It was time to let the market run where it would.

“It would have been a huge business,” one of the zero-day market’s first brokers told me, in between bites of his enchilada. Finally, in the fall of 2015—after two years of trying—a zero-day dealer agreed, against his better judgment, to sit down with me face-to-face.

That October I flew to Dulles to meet with a man I’ll have to call Jimmy Sabien. Sabien had been out of the market for years, but given that he still worked with the same government agencies he did back then, he would only speak to me on the condition I not use his real name. It was Sabien who first offered Watters $150,000 for a bug that iDefense paid less than a thousand bucks for. “You couldn’t dream up better profits,” he told me.

Ten years later, he still shook his head at the slight.

Before he helped pioneer the zero-day exploit business, Sabien was in the military, protecting military computer networks around the world, and he looked and played the part. Tall, broad-shouldered, hair cut high and tight, with a soldier’s gallows humor.

We arranged to meet at a Mexican restaurant in Ballston—just a few miles from several of his former customers—where he relayed the history of a market that very few people even know exists.

In the late 1990s, Sabien was recruited to one of three boutique government contractors that first started dealing in zero-days on behalf of US intelligence agencies. Back then the deals were not yet classified, which meant he wasn’t breaking any laws by talking to me. Even so, he’d insisted I obscure his real name.

Protecting networks for the military, Sabien told me, had left him intimately acquainted with technology’s flaws. In the military, secure communication means the difference between life and death, but the big technology companies didn’t seem to grasp that. “People were clearly designing these systems for functionality, not for security. They weren’t thinking how they could be manipulated.”

Manipulating computer systems was pretty much all Sabien thought about after he left the military and joined a boutique Beltway contractor, where he managed a 25-person team that developed intrusion tools for the US government. Sabien learned that the hacking tools his team built were useless without a way to deploy them. Reliable access to a target’s computer system was critical.

“You could be the best jewelry thief in the world, but unless you know how to bypass the Bulgari store alarm system, it doesn’t get you anywhere,” he told me. “Access is king.”

Sabien’s team trafficked in digital access, searching for bugs and writing the code for customers to exploit them. The bulk of revenues—more than 80 percent—came from the Pentagon and intelligence agencies, with the remainder from law enforcement. The goal was to deliver these agencies secret tried-and-tested ways into every system used by the adversary, be it nation-states, terrorists, drug cartels, or low-level criminals.

Some of their work was opportunistic. If they found a bug in a widely used product like Microsoft Windows, they’d develop an exploit and sell it to as many agencies as possible. But much of their work was targeted: Agencies would come to Sabien’s team looking for a way to monitor the Russian embassy in Kyiv or the Pakistani Consulate in Jalalabad. Sabien’s team would have to do reconnaissance, decipher which computers the targets were using and what kind of operating environment they were running. Then find a way inside.

There was always a way. So long as humans are responsible for writing code and designing, building, and configuring machines, Sabien’s team knew there would be mistakes. Finding those flaws was only half the battle. The other half was writing and honing the exploit code that would give government agencies a reliable, clean entry point.

Sabien’s clients didn’t just want a way in. They wanted a way to crawl through networks undetected, a way to implant invisible backdoors that kept them in even after their breach was discovered, and ways to pull the adversary’s data back to their command-and-control servers without tripping up alarms.

“They wanted the entire kill chain—a way in, a way to beacon out to their command-and-control server, an exfiltration capability, an obfuscation capability,” he said, using military-speak. “It makes sense when you think of the Special Forces and SEAL Team Six. They have snipers, sweepers, exfil specialists, and people who break down the doors.”

The holy trifecta was a string of zero-day exploits and implants that offered reliability, invisibility, and persistence. Rarely did you get all three. But when you did: “Ka-ching!” Sabien said.

When I asked him to discuss specific exploits, he recalled some with the same affection others might feel in recalling their first love. His favorite was a stubborn zero-day in a video memory card. The memory card ran on the computer’s firmware—the software closest to the bare metal of the machine—making the exploit nearly impossible to find and harder still to eradicate. Even if someone wiped their machine clean, the exploit stuck. The only way to confidently rid the machine of spies was to throw the computer in the dumpster. “That exploit was the best,” Sabien recalled with a twinkle in his eye.

The first thing spies do after breaking into a machine, Sabien told me, is listen in for other spies. If they found evidence that the infected machine was beaconing out to another command-and-control center, they would scrape whatever others were catching. It wasn’t abnormal, Sabien said, to find multiple spies listening in on the same machine—especially in the case of high-profile diplomats, arms dealers, or terror networks. There was one zero-day in HP printers that for years, Sabien told me, was exploited by “government agencies all over the world.” The exploit allowed spies to capture any files that passed through the printer and gave them a beachhead where IT administrators would least suspect. The day Hewlett-Packard patched the printer bug, Sabien said, “I just remember thinking to myself, ‘A lot of people are having a very bad day.’”

The short list of government agencies looking to acquire their own zero-day arsenals did not stay short for long. The NSA boasted the largest and brightest army of cyberwarriors of the intelligence community, and in those early days, the agency didn’t require much outside help. But in the mid-1990s, as the masses took to the web and email, sharing a fine-grain record of their daily lives, relationships, inner thoughts, and deepest secrets, a growing number of intelligence agencies feared they were not prepared to exploit the rapid adoption of the internet. In late 1995 a special CIA working group determined that the agency was woefully underprepared. The same was true for other agencies, who were even further behind. A growing number started looking to acquire their way into these capabilities.

The nearly simultaneous bombings of the American embassies in Nairobi, Kenya, and Dar es Salaam, Tanzania, only fueled the government’s demand for more intelligence, more data, and the digital intrusion tools to capture it. Amassing zero-days became a competitive enterprise. Congress, meanwhile, which has slashed military spending throughout the ’90s, continued to approve vague “cybersecurity” budgets without much grasp of how those dollars funneled into offense or defense. Policymakers’ thinking on cyber conflict was, as former commander of US Strategic Command James Ellis put it, “like the Rio Grande, a mile wide and an inch deep.” But inside each agency, officials were learning that the best zero-day exploits netted the best intelligence, which in turn translated to bigger cyber budgets down the road.

And there was Sabien, right in the middle of all of it.

His team couldn’t churn out zero-day exploits fast enough. Disparate agencies wanted ways into the same systems, which played well from a bottom-line perspective, but not so much from the American taxpayers’. His company sold the same zero-day exploits two, three, four times over. The overlap and waste was only exacerbated after the 1998 bombings and, then, 9/11. As defense and intelligence spending ballooned by more than 50 percent over the next five years, there was a virtual stampede from the Pentagon and intelligence community to Beltway contractors that specialized in digital espionage.

But bugs and exploits took time to find and develop, and Sabien concluded that it would be a far more efficient use of their time to outsource the bug-hunting to hackers. They would still write the code to exploit them, but why not tap hackers to provide them with the raw material?

“We knew we couldn’t find them all, but we also knew there was a low barrier to entry,” Sabien recalled. “Anyone with $2,000 to buy a Dell is in the game.”

And so in the late ’90s, Sabien’s team started reaching out to hackers, and America’s underground zero-day market was born—the same one that would eventually consume iDefense, and the rest of us.

Sabien’s stories from those early days landed like a spy novel, complete with cloak-and-dagger meetings, bags of cash, and murky middlemen—only in this case none of it was literary or imagined. It all checked out.

In the ’90s, government agencies would pay contractors like Sabien’s roughly $1 million for a set of 10 zero-day exploits. Sabien’s team would budget half of that to buy bugs and then develop them into exploits themselves. A decent bug in a widely used system like Windows might fetch $50,000; a bug in an obscure system used by a key adversary might fetch twice as much. A bug that allowed government spies to burrow deep into an adversary’s system, undetected, and stay awhile? Easily $150,000.

Sabien’s team avoided idealists and whiners. And because there were no rules to this market, the bulk of their suppliers were hackers in Eastern Europe.

“With the breakup of the Soviet Union, you had a lot of people with skills, without jobs,” Sabien explained. In Europe, hackers, some as young as 15 and 16, were trading their discoveries to zero-day dealers who would turn around and sell them directly to government agencies and their brokers. Some of the most talented hackers, Sabien told me, were in Israel, veterans of Israel’s Unit 8200. One of the best was a 16-year-old Israeli kid.

It was a secretive business and mind-blowingly convoluted. Sabien’s team couldn’t exactly call up hackers, ask them to send their exploit by email, and mail them back a check. Bugs and exploits had to be carefully tested across multiple systems. Sometimes hackers could do this over video. But most deals were done face-to-face, often in hotel rooms at hacker conventions.

Sabien’s team increasingly relied on these murky middlemen. For years, he said, his employer dispatched an Israeli middleman with duffel bags stuffed full of half a million dollars in cash to buy zero-day bugs from hackers in Poland and across Eastern Europe.

Every step in this insanely complex deal-making structure relied on trust and omertà. Governments had to trust contractors to deliver a zero-day that worked. Contractors had to trust middlemen and hackers not to blow the exploit in the course of their own escapades, or resell it to our worst enemies. Hackers had to trust contractors would pay them, not just take their demonstrations and develop their own variation of their bugs. This was before bitcoin. Some payments were doled out via Western Union, but most were done in cash.

You couldn’t dream up a less efficient market if you tried.

Which is why, in 2003, Sabien took note that iDefense was openly paying hackers for their bugs and called Watters.

To a businessman like Watters, who was trying to push the market out into the open, what the contractors were doing was idiotic, dangerous even.

“Nobody wanted to talk openly about what they were doing,” Watters recalled. “There was this whole air of mystery to it. But the darker the market, the less efficient it is. The more open the market, the more it matures, the more buyers are in charge. Instead they chose to work out of Pandora’s box, and the prices just kept going up.”

By late 2004, there was new demand from other governments and front companies, all of whom kept driving up the price of exploits and making it difficult for iDefense to compete.

As the market spread, what troubled Watters wasn’t the effect the market would have on iDefense; it was the increasing potential for an all-out cyberwar. “It’s like having cyber nukes in an unregulated market that can be bought and sold anywhere in the world without discretion,” he told me.

The certainty of the Cold War era—with its chilling equilibrium—was giving way to a vast uncharted digital wilderness. You weren’t quite sure where the enemy would pop up or when.

American intelligence agencies began relying more and more on cyberespionage to collect as much data about as many adversaries, and allies, as possible. But it wasn’t just spying. They also sought code that could sabotage infrastructure, take out the grid. The number of Beltway contractors eager to traffic in these tools began to double every year, Sabien said.

The big contractors—Lockheed Martin, Raytheon, Northrop Grumman, Boeing—couldn’t hire cyber specialists fast enough. They poached from inside the intel agencies and acquired smaller shops like Sabien’s. The agencies started procuring zero-day exploits from catalogs, offered by Vupen, a zero day broker in Montpelier, France, who would later rebrand as Zerodium. It set up shop closer to its best customers in the Beltway and started openly publishing its price lists online, offering as much as $1 million (and later $2.5 million) for a tried-and-tested way to remotely hack the iPhone. “We pay BIG bounties, not bug bounties,” went the slogan. Former NSA operators started their own businesses, like Immunity Inc., and trained foreign governments in their tradecraft. Some contractors, like CyberPoint, took their business overseas, stationing themselves in Abu Dhabi, where the Emiratis rewarded former NSA hackers handsomely for hacking its enemies, real and perceived. Soon, zero-day dealers like Crowdfense, that sold exclusively to the Saudis and Emiratis, started outbidding Zerodium by a million dollars or more. Eventually, those tools would be turned on Americans.

By the time Sabien agreed to meet with me in 2015, the market was hard to avoid. “In the ’90s there was just a small community of people working on exploits and selling them. These days it’s so commoditized. It’s blown up. Now”—he swirled his finger in a wide circle to symbolize the Beltway— “we’re surrounded. There are more than a hundred contractors in this business.”

The market’s spread among US agencies didn’t bother Sabien. It was its spread abroad that rattled him.

“Everyone has their enemies,” he told me. “Even countries you would never suspect are stockpiling exploits for a rainy day. Most do it to protect themselves. But one day soon,” he added as we got up to leave, “they know they might have to reach out and touch someone.”

“Keep going,” he told me. “You’re onto something. This will not end well.”

And with that, he was gone.


Excerpted from This How They Tell Me the World Ends, by Nicole Perlroth. Copyright © Nicole Perlroth 2020. Published by Bloomsbury USA. Reprinted with permission.


More Great WIRED Stories