Concluding Act 3: A ceremony that changes everything. Image by author.

Something big happened on October 27. Something unprecedented. And like much high-level change that impacts the Internet’s basic infrastructure, this change came down to the actions of a handful of carefully chosen people. It involved a ceremony straight out of a sci-fi movie–seemingly rife with opportunity for espionage, intrigue, or a massive telegenic heist. For STS-focused social scientists, this story is compelling for the layers of trust involved, and the way technical security and human relationships intersect. That something so critical to global infrastructure can be reduced to concepts like duty and accountability is neither surprising nor novel, per se—but it is remarkable.

But let me backtrack, as this is really about two news stories.

Friday, October 21, 2016 saw a massive disruption in internet traffic, particularly for the Northeastern United States. The outage, a distributed denial of service attack (DDoS) started at 7am EST, appears intended as a show of force, and was directed at New Hampshire-based Internet infrastructure company Dyn. In a sense it also acted as “proof of concept” for Dyn data analyst Chris Baker’s recent blog article, “Recent IoT-based Attacks: What Is the Impact On Managed DNS Operators?” The article had posited a likely increase in DDoS attacks that exploit an ever-expanding “Internet of Things.” Indeed, if Dyn had wanted to draw traffic to its blog, this was a brilliant strategy.

DDoS? What happened?

A brief primer on DNS (Domain Name System)—the naming system for anything connected to a network. The way DNS works is that when you type in a web address, like blog.castag.org, it looks up a series of numbers, that is, an IP address needed to connect with the servers that will deliver the appropriate content. So if everything’s working right, you get Netflix’s actual website when you visit netflix.com. If someone was able to intercept that request, however, you could instead get a facsimile that harvests your login information when you try to use it. DNS has been compared to a phone book, or a system of street signs (which I can tell you, as someone whose primary field sites are in Japan, are something you miss when they’re not there).

A DDoS attack essentially creates too much “busywork” for servers to handle—it floods them with fake requests for information, followed by automatic re-requests; the servers also suffer from regular users repeatedly refreshing stalled pages to try to force them to load. Dyn sees these attacks often, so to cause it any real difficulties requires a big attack—Dyn said “tens of millions of IP addresses at the same time” in last Friday’s case. Sites lose money by being offline—people can’t pay their bills, sign up for new services, click on ads, or view paid content. So what’s annoying for you and me is extra stressful for anyone trying to do business on the Internet.

We are the botnets

The attacks started around 7am EST, and while the first wave was handled after approximately two hours, a second attack hit just before noon. Just after 4pm, a third wave came. Attacks like these take advantage of poorly secured networked devices—components of the “Internet of Things” including webcams, DVRs, routers, etc., and infect them with a kind of malware that allows hackers to control them remotely. These networks of enslaved devices (or as Bloomberg refers to them, “zombies”) are called botnets.

Not this kind of zombie (dramatic rendering by author).

A reporter at Gizmodo claimed: “Our internet is frightfully fragile in the face of increasingly sophisticated hacks,” and alluded to widely circulating conspiracy theories (mostly political) surrounding the attack’s timing. And as Matt Larson, vice president of research at The Internet Corporation for Assigned Names and Numbers (ICANN—more about them later) explains, DNS isn’t exactly a highly secure system:

The domain name system was designed when the internet was a friendlier place, and there wasn’t much thought of security put into it.

The source code for a malware application called Mirai, used to create these botnets, was recently made public, prompting concerns that Mirai-based attacks would soon increase. And indeed, much of the traffic generated by the October 21 attack was Mirai-driven.

Security expert Bruce Schneier explains the attack further:

Over the past year or two, someone has been probing the defenses of the companies that run critical pieces of the Internet. These probes take the form of precisely calibrated attacks designed to determine exactly how well these companies can defend themselves, and what would be required to take them down. We don’t know who is doing this, but it feels like a large nation state. China or Russia would be my first guesses.

In other words, DDoS-using hacker groups like Anonymous never attempted an attack of this sort—one might argue that it’s not their style, as they typically target particular entities rather than entire chunks of Internet infrastructure.

The DNS story gets weirder

I want to take this in a weird direction right now. We’ve established the central role that DNS plays in the functionality of the Internet. It is no exaggeration to say that if you control DNS, you control the ‘Net itself. At the top of the pyramid, DNS is secured by only a handful of people—seven crypto-officers, and seven backup officers.

And as I typed this sentence on October 27, 2016, they were meeting for a reoccurring ritual called the Root Signing Ceremony.

This ritual, which has happened every three months since 2010, requires some of the officers to gather in one of two dedicated facilities in order to update and verify the “keys to the internet’s metaphorical lock.”

These individuals are part of the non-profit ICANN, which assigns numerical addresses to websites and computers (IP addresses, that is). Essentially, to control ICANN’s databases is to control the Internet. ICANN has only been in charge beginning this month, when the US government turned over its role as “inventor-steward” to the US-based group.

In the interest of protecting DNS, ICANN uses an elaborate system and, just as in any good sci-fi novel, it distributes power between individual crypto-officers. Of the seven key holders (and the seven backup holders), at least three must attend a ceremony. Their keys unlock safe deposit boxes, from which key cards are extracted. These three key cards must be used at the same time in order to access a master key-generating device.

What participants see when watching the live streaming ceremony. Image by author.

To explain it simply, the master key is actually a code (a “root signing key”), a kind of password that allows access to the master ICANN database. The master key generates additional keys, used by internet security organizations to protect various parts of the Internet. Each of these organizations requires the signature of the key above it, starting with ICANN at the top.

And on October 27, in Culpepper, Virgina, the master key itself was changed for the first time. The ceremony streamed live from 1-7pm EST. ICANN also publishes the scripts, but the experience of being able to chat live with other viewers and ask questions is part of the draw.

If you read past scripts, you get a sense of what to expect. The ritual is extremely precise, down to the scheduling of one minute pauses. Participants know this script in advance, so that they might monitor for deviation. During the last ceremony, in August, a total of twenty-two parties serving different roles and referred to by code letters, entered a series of locked and increasingly secure rooms using key codes and hand scanners. With the assistance of a common key wielded by an ICANN representative, the crypto-officers extracted their key cards.

As you may have noted, the security surrounding the ceremony is so intense that the entire ritual takes six hours to execute.

Changing the master key. Image by author.

And at some point during the first quarter of 2017, two ICANN employees will take a copy of the encrypted key file to its other facility on the west coast. The changeover to a new master key will, all told, take about two years.

ICANN has said it considers potential DNS security threats to include everyone from individuals to entire nation-states.

Cybersecurity going forward

As a manufactured enterprise, the Internet’s basically functionality always comes back to the efforts of individual experts. So do threats to its safety. With security hacks and data breaches seemingly always in the news, and a major party candidate for American president accusing another country of hacking to manipulate the election during a national debate, the need for such elaborate security as that maintained by ICANN is essential. And with the Internet of Things expanding faster than our capacity to protect these devices, defense against malware will remain a hot issue. As far as defending our networked gadgets goes, we are in the toddler stage.

Ceremony image by author.

Further Reading

The excellent podcast Radiolab, discussing cybersecurity: http://www.radiolab.org/story/darkode/ 

And a podcast explanation of botnets: https://nakedsecurity.sophos.com/2014/01/09/sophos-techknow-understanding-botnets-podcast/

Explaining DNS further: http://gizmodo.com/what-is-dns-and-why-does-it-make-the-internet-break-1788065317

Visualizing a DDOS attack: http://gizmodo.com/5995429/how-a-ddos-attack-looks-as-it-happens

The Guardian’s story on the crypto-officers:  https://www.youtube.com/watch?v=fUJM9tTvCG4

Information Week’s DarkReading, on internet security: http://www.darkreading.com/