The Time Someone Actually Broke the Internet [CL95]

Posted on Thursday, Feb 29, 2024 | Series: Chaos Lever
Explore the story of how 11 lines of code disrupted the internet, the battle between open-source ethics and corporate power, and the fragile nature of our digital world.


00:00:00 Ned: Did you ever say when you were a child, like, I’ll never tell my kids, “Because I said so.” It’s amazing how quickly that goes out the window.

00:00:10 Chris: [laugh].

00:00:10 Ned: Like, “I could get into the details with you my son, but you know what? Just do it because I fucking say so.”

00:00:17 Chris: Right. “You’re twelve. Nothing you say matters.”

00:00:22 Ned: Oh, come now. It doesn’t matter now either. Hello, alleged human and welcome to the Chaos Lever podcast. My name is Ned, and I’m definitely not a robot. I’m a real human person with a staunch belief in extraterrestrial beings who have secretly infiltrated the government and media, and now rule us with mind rays beamed through television, music, and podcasts. Mwah-hahaha. Would a robot believe such illogical claptrap? Nay, I say. Nay. With me as Chris, who’s also here.

00:01:05 Chris: I am trying to remember the last time I ever heard someone say ‘claptrap,’ and I am coming up with nothing.

00:01:13 Ned: I’d like to blame it on, like, word-of-the-day calendar or something, but I don’t actually have one of those, so that really just popped out of my brain.

00:01:22 Chris: I mean, anything that rhymes is automatically better, so that’s part of it, I’m sure.

00:01:27 Ned: [laugh] Been saying it for years. Yeah. Claptrap is a good one. I do like a lot of the older words, like tomfoolery, shenanigans, I could probably go on, but now I’m drawing a blank because I’m putting myself on the spot.

00:01:42 Chris: No pressure like self-pressure.

00:01:44 Ned: [laugh] It’s how I think I’ve accomplished most of what I’ve done in my life is to set unrealistic goals, procrastinate to the last minute, and then stress myself out for 24 to 48 hours.

00:01:58 Chris: Panic to just a heroic degree.

00:02:01 Ned: Yes [laugh]. And somehow—

00:02:03 Chris: Stop—maybe stop for lunch.

00:02:07 Ned: [laugh] Yes. I mean, I’m not an animal, you know? Act like a civilized person. But—

00:02:13 Chris: Not a robot animal?

00:02:15 Ned: No, no, not an animal robot either, which is a slightly different thing, and reminds me of Chuck E. Cheese. Did you know that people reprogram the animatronics from saved Chuck E. Cheeses to perform other songs?

00:02:32 Chris: Yes, I too get bizarre recommendations from YouTube.

00:02:36 Ned: Excellent [laugh]. Well, I’m glad we’re all on the same page. Oh, why don’t we reflect on a… simpler time? Maybe you have a story about the internet.

00:02:48 Chris: I do, and we are going to do a small amount of time travel to talk about something that happened a bunch of years ago, but still has implications for how we do stuff now. And two things I want to get out right away before we get into it. The first is, I’ve been wanting to do this story forever, so [laugh] because I think it is just the best, you know? To our conversation earlier, it’s the bee’s knees.

00:03:13 Ned: Yes, it is. Well done.

00:03:15 Chris: But second, the big thing that we’re going to talk about here is really heavily rotating around programming, and as we have talked about on this program a number of times, I am not a programmer.

00:03:27 Ned: Mm-hm. Neither am I.

00:03:28 Chris: So, I’m going to be doing two things with regards to the details—like the real fine, nitty-gritty, right—two things. The first is summarizing because a lot of the detail is not necessarily important to the main thrust of the story.

00:03:43 Ned: Fair.

00:03:44 Chris: And the second is guessing because I’m not a programmer; I don’t know what the fuck I’m talking about. Right?

00:03:53 Ned: [laugh]. Oh—

00:03:54 Chris: So.

00:03:54 Ned: —yeah, no, I mean, you and ChatGPT have that in common.

00:03:58 Chris: You are welcome to keep me honest at any point, or just sit back and let me crash and burn.

00:04:04 Ned: Or both.

00:04:05 Chris: Or both [laugh]. So, having said that, have you ever noticed how many times someone has allegedly—air quotes here—“Broken the internet?” Well, there was one time that a dude actually did. And God help me, that is just an awful lead. Egh. It sounds like some BuzzFeed shit right there. And you know what? Fine. In the spirit of said Buzzfeeds, let’s do a list.

00:04:33 Ned: Oooh.

00:04:34 Chris: Times that the internet has been brokended, and number six will shock you.

00:04:40 Ned: Well, the exposed copper wires, really.

00:04:42 Chris: Almost everything that breaks the internet really is just something that is so popular, people can’t access a web page for a time. So, we’re talking about things, like, J Lo’s green dress, Lola Bunny in the Space Jam movie, Ronaldo and Messi pretending they know how to play chess, Ryan Gosling and Harrison Ford talk Blade Runner, and of course, Ralph.

00:05:07 Ned: Hmm.

00:05:07 Chris: But enough has already been said about him.

00:05:11 Ned: I do not want to tell you how many times I’ve had to watch that movie. But it’s more than ten, Chris.

00:05:18 Chris: At least it’s not the sequel.

00:05:20 Ned: No, that is the sequel.

00:05:22 Chris: I’m confused.

00:05:23 Ned: Wreck it Ralph is the original.

00:05:24 Chris: Aw crap, you’re right.

00:05:27 Ned: Haha-HA [laugh].

00:05:30 Chris: [sigh] moving on.

00:05:30 Ned: Let’s bring it full circle, man: children. [laugh] I’ve seen both of those movies [laugh] before, ten times.

00:05:36 Chris: The point here is, none of these things literally broke the internet. It just sounds like a cool thing to say. Just like ‘Trial of the Century.’ Did you know that the 20th century had, like, ten of those, roughly one a decade? You can actually look that one up; that’s really true.

00:05:55 Ned: Should have just called it ‘Trial of the Decade.’ But it doesn’t have the same ring.

00:05:59 Chris: ‘Trial of’ people are paying attention.

00:06:01 Ned: Mmm.

00:06:02 Chris: So anyway, to break the internet as a whole actually is possible, depending on how you define the word break, and depending on how you define the word internet. But what if I told you that one man did it in 2016, by making a measly 11 lines of code, a total of 216 characters—I counted; you’re welcome—he made them unavailable, and like, half the internet websites stopped working. That happened.

00:06:33 Ned: Mmm.

00:06:34 Chris: So, let’s talk about what happened, why it happened, and its implications on the wider world of software development, and by extension, the wider world in general, both in 2016, and in… now.

00:06:49 Ned: [laugh] Whenever now is.

00:06:51 Chris: So, I tried to pivot away from BuzzFeed. Did that sound dramatic enough? I don’t think I hit exactly the target. I feel like I went less impending doom and more beta-blockers before a TED talk.

00:07:04 Ned: No, I think you were right on the cusp there. I was feeling it. I’m intrigued.

00:07:09 Chris: Three shots of tequila and then, like, 90 seconds in a spinny chair.

00:07:15 Ned: [laugh] That will certainly put on the show. Very quickly.

00:07:21 Chris: Let’s just move on.

00:07:22 Ned: All right, then.

00:07:24 Chris: So, there’s a couple of things we have to talk about first so that we understand all of the different technological players in this story. So, people might have heard of a little thing called JavaScript. Surprisingly, has a lot less to do with actual Java than you want to believe, or you’d thought that it was.

00:07:45 Ned: Yeah.

00:07:45 Chris: But it’s still kind of a big deal. A lot of the internet is based on it in a number of different ways. It’s not just annoying crap on your website, unfortunately. It is become something that lives in that weird netherworld between a shell script and a quote-unquote, “Real” programming language that you hardcore compile, like C. But it’s got two things going for it that have kept it popular forever. One, it is good enough for internet work. And two, as it has evolved, it has gathered to itself a really neat way of pulling bits of random code together from all over the place to make the total running program. Those bits are called packages, and they’re used in the JavaScript runtime called Node.js. Now, remember, I’m summarizing.

00:08:41 Ned: Yes.

00:08:41 Chris: A lot of these details, not necessarily important. What is important is, a lot of websites use Node.js, a lot of packages can be used automatically by Node.js, referenced by name. And those packages can then be updated on the fly by developers that have nothing to do with the web program in question, which is pretty cool because it means that anytime a Node.js is put together, the people that are making the web application can expect the latest code from the packages that they’re using. And the way that this works is through something called NPM, Node Package Manager. All this stuff was in 2016, and still is as far as I understand—free. And it saves a ton of time in programming because this stuff is out there, pre-developed, tested, proven, it works.

00:09:35 Ned: Right. There’s a lot of nuance in there. Like you said, you’re summarizing. For instance, you don’t have to install the latest package from NPM. You can pin certain versions of different packages, if you want to, through a special file that defines the packages you’re using and the versions you want to use. There’s also a downside—not downside, but there’s also the issue that some packages will leverage other packages, so there’s a cascading effect there. And so, if you want to pin versions for all of them, you have to know exactly which packages you’re using. And you can also cache the packages locally. So, you don’t have to pull them every time you run a Node.js program. You can pull from a local package manager. But I mean, largely, broad strokes, everything you said is right.

00:10:28 Chris: Right. And I just—I agree with that, and I think it’s really good that you made that differential between what people do and what people could do.

00:10:36 Ned: [laugh] It’s very important. You could have a secure server, but you have to make some choices.

00:10:44 Chris: Right. So, that’s the background. And the whole point of this is it saves people time and programming fundamentals. Like, how are you going to display text, like… a zip code repository? You could write this all this stuff by yourself if you wanted to, or you could connect to an NPM module—or an NPM package, I’m sorry—and just have it there. Saves you time.

00:11:08 Ned: Yeah.

00:11:09 Chris: No, two ways about that. So, flashback to the mid-2010s. Did we ever name that decade?

00:11:18 Ned: I don’t think so. It’s just the 2010s. And we had the aughts before that, but that’s because it was really awkward to say the double zeros.

00:11:24 Chris: Yeah, and people tried to do the naughties, and that was—we didn’t like that.

00:11:29 Ned: I mean, most of us did not like that [laugh].

00:11:32 Chris: Anyway, 2010s, an open-source Turkish developer named Azer Koçulu was cruising—and I hope I pronounced that even remotely close to correctly—

00:11:45 Ned: Sounds good.

00:11:46 Chris: Azer was responsible for writing a lot of these freely available packages hosted on NPM. They were popular, to the extent that they had been used, referenced, downloaded, compiled, et cetera thousands upon thousands of programmers the world over. And he wasn’t doing it for money, per se. He was doing it because he could because what he was doing filled a gap, and he enjoyed it, and he really embraced the idea of open-source programming as the way to do everything. Then, out of nowhere he got a letter from a lawyer, which I don’t know if anybody’s ever been in that situation, never a good sign.

00:12:31 Ned: Nope. Not something you want.

00:12:33 Chris: I don’t even have any lawyers that are friends. Don’t want to take chances.

00:12:37 Ned: [laugh].

00:12:37 Chris: So, this particular lawyer was from an up-and-coming messaging app that was based in Canada called Kik. It’s spelled K-I-K, it’s pronounced, apparently, ‘kick’ because what they want in this story is for us to hate it from the jump.

00:12:53 Ned: Done. Hate it.

00:12:56 Chris: Now, here’s the problem. Azer had an NPM package called Kik [keek]. Now, I’m going to pronounce it properly because I’ve decided he’s the good guy, but they’re both spelled K-I-K. In all truth, I have no clue how any of them are pronounced. It’s probably pronounced ‘menagerie.’ Fucking internet.

00:13:16 Ned: [laugh].

00:13:16 Chris: Anyway, the letter was effectively a cease and desist because, as the lawyer man helpfully explained—and I want to quote this one particular piece of this back-and-forth; there’s a ton of emails in here, but this really gets to the heart of it, and I think it’s magical. It’s magical enough—let’s do a little play-acting.

00:13:38 Ned: Okay.

00:13:39 Chris: So, there’s two of us, and there’s two of them. Ned, would you rather be the dickhead lawyer or Azer, the hero programmer?

00:13:46 Ned: You know, I’m going to be the lawyer, right?

00:13:48 Chris: The stage is yours.

00:13:50 Ned: All right. He’s Canadian?

00:13:51 Chris: The lawyer is Canadian.

00:13:53 Ned: Okay. “We don’t mean to be a dick about it, eh, but it’s a registered trademark in most countries around the world, and if you actually release an open source project called Kik, our trademark lawyers are going to be banging on your door and taking down your accounts and stuff like that, eh—and we’d have no choice but to do all that because you have to enforce trademarks or you lose them. Eh.”

00:14:15 Chris: “Hahah, you’re actually being a dick. So, fuck you. Don’t email me back.” And, scene.

00:14:26 Ned: [laugh]. That’s magical.

00:14:27 Chris: I told you. I told you. Azer stuck to his guns and did what I think was the reasonable thing to do here. He appealed directly to NPM for help because they are an organization right? To their immense credit, NPM stood strong at Azer’s back against this obnoxious and unnecessary corporate overreach—[laugh] I’m [laugh] just kidding. They folded like a house of cards. And it didn’t even take long. The friggin’ chief executive of NPM emailed Azer, saying quote, “In this case, we believe that most users would come across a Kik package and reasonably expect it to be related to… in this context, transferring ownership of these package names achieves that goal,” unquote. What a hero.

00:15:14 Ned: Hero.

00:15:16 Chris: It’s a word that gets thrown around a lot these days. Not to me, and I don’t want to get into the intellectual property or trademarks or anything like that. To me, this is a nonsense argument. This is Apple Music versus Apple Computers. Yeah, two things can have the same name.

00:15:32 Ned: Yeah. How about that?

00:15:35 Chris: NPM disagreed. Azer flipped out—understandably—and said he wanted all of his packages taken out of NPM. Quote, “Quickly.” And after NPM predictably, did nothing. Two days later, Azer did the only thing he thought he could do, and completely unexisted all of his packages from NPM. All of the versions, all of the packages, absolutely everything. Delete all. rm -rf ./*.

00:16:07 Ned: That’s a bit of a scorched earth policy, but you know, he obviously had some principles, and he felt like sticking to them.

00:16:14 Chris: Yeah, he was mad, is the thing.

00:16:16 Ned: Yeah. Mmm.

00:16:18 Chris: And like I said, he didn’t just delete the Kik thingamajig; he deleted them all. And what I said at the top, if you’ll remember, he had a lot. And the list of deletions included an incredibly innocuous one called left dash pad, which I’m just going to call left-pad from now on. What did left-pad do, you ask? Well, it was being used by thousands of projects to pad out strings of numbers with extra spaces, or extra zeros. Formatting stuff. Now, there’s a few examples out there, but the simplest one is to say, let’s say you’re selling tickets to an event of some kind. You, as a human being, are numbering those tickets from 1 to 1000. You want to show them, so that the numbers on the computer and columns and screens make sense, the first ticket is 0001; the 10th ticket, 0010, et cetera. That just makes sorting easier, it makes life easier for computers, especially back then, right? Uniformity, right? Super helpful in computering. Also, possibly even more important than that, left-pad helped a lot with ASCII art where—

00:17:29 Ned: [laugh].

00:17:29 Chris: Spacing is incredibly important, too.

00:17:32 Ned: True, yeah. Okay. We got to get our priorities right. I don’t care about tickets, but ASCII art.

00:17:38 Chris: [laugh] I had a dragon on my BBS, and it was excellent. So, lots of packages, it turns out, needed to do formatting things like this. And they relied on left-pad to do it. And to do it, well, left-pad had to exist—

00:17:57 Ned: Mm-hm.

00:17:58 Chris: Which as of early morning, March 22nd, 2016, it no longer did. And that is when all hell broke loose.

00:18:10 Ned: [laugh]. Yeah.

00:18:11 Chris: So, you’re a little bit further down the programmer path than I am. You can probably help fill in some blanks here. You know what happens when a program is trying to run, and it tries to run a reference to another program, and that program, isn’t there? It’s usually totally fine, right?

00:18:27 Ned: Umm… yeah, no.

00:18:30 Chris: It’s the other one.

00:18:31 Ned: Sometimes it’ll error out in a nice way and tell you that it couldn’t find the package, but the ultimate end is that it will stop running.

00:18:42 Chris: Right. I believe the technical term for this is kablooey.

00:18:47 Ned: Yes, that’s what my training tells me.

00:18:49 Chris: Right. Coyote running into the wall kind of explosion.

00:18:53 Ned: Mm-hm.

00:18:54 Chris: Yeah. That’s what started happening a lot. And it happened to a lot of little boutique websites you’ve probably never heard of, like, Facebook, or Quartz, or any number of these other aggregator websites. [laugh] In a beautiful piece of irony, our good friends at the messaging platform Kik found themselves offline, too. Talk about poetic justice.

00:19:23 Ned: [laugh] I love it.

00:19:24 Chris: Why did this happen? It happened for exactly the reason you intimated at the top. All of these websites were built using a framework called React, and React uses tons of NPM packages. Somewhere in the dependency string. Guess what was in there? Left-pad.

00:19:43 Ned: Left-pad.

00:19:44 Chris: Yep. So, this happened, everything exploded. Azer publishes his side of the story, and what he did and why, on a Medium post called, quote, “I’ve just Liberated my Modules” unquote. Now, in years since, he has taken down the original, but never forgets. Link in the show notes, if you’re curious. In his story, he basically just says NPM is compromised by corporate interests, and he rejects their stance, and refuses to have his work hosted by them. Azer wrote, quote, “This situation made me realize that NPM is someone’s private land where corporate is more powerful than the people, and I do open source because, Power To The People,” unquote.

00:20:33 Ned: All right.

00:20:34 Chris: Standing on principle. Amen, sir.

00:20:36 Ned: How about that?

00:20:38 Chris: So yeah, Power To The People, indeed. So, much so [laugh] that NPM decided that they would unilaterally reinstate left-pad, manifestly against Azer, the freaking author’s wishes because it was causing so much chaos to our little friends at Facebook, Quartz, et cetera. Basically, anyone using React—among other people—were mystified why their sites weren’t updating or working anymore, so NPM decided to make all those sites work again. But there is no doubt, for two-odd hours in March of 2016, a single open-source programmer standing on his principles truly did break the internet.

00:21:24 Ned: [laugh] Well done, sir. I tip my hat to you. An interesting thing is that React was actually created—the framework was created by Facebook. So, that’s part of the reason it broke Facebook is because it was literally the thing that they were using. Unfortunately, because he was developing in open-source, the code, even if he removed it from NPM, the packages, all the code was still available for anyone to fork and then republish on NPM. I’m not sure exactly how packages are referenced on NPM, and NPM, would have had to recreate that reference exactly as it was, otherwise, every one who was using that package would have to update their dependency file to include whatever the new package path was, and that would be a lot of work—

00:22:16 Chris: Right.

00:22:16 Ned: —for some very powerful people.

00:22:19 Chris: And eventually, all of that stuff came to pass. But to explain where the crisis came from, first of all, let’s remember that the websites were talking about, even in 2016, there was an expectation that everything was always going to be up. Downtime was no longer an issue.

00:22:37 Ned: Right.

00:22:37 Chris: Except for the good old days of Twitter.

00:22:40 Ned: [laugh] I haven’t seen the fail-whale in a while.

00:22:43 Chris: To my understanding, this is what caused the crisis. Let’s say you run a build on March 21st. Left-pad still exists, easy-peasy, everything works. We’re in a world of continuous development. You make a minor change on March 22nd, try to rebuild, left-pad doesn’t exist. Weird things start happening, stuff starts breaking, nobody knows why. Now, you could fail back to the March 21st build if your DevOps is responsible and did a lot of the caching and things you were talking about with secure and safe programmatic principles, but there’s that word ‘if’ again.

00:23:25 Ned: Mm-hm.

00:23:27 Chris: But even if you did, this effectively means that your web app development is done until you figure out why the March 22nd build didn’t work. Hence, the alarm.

00:23:39 Ned: Right.

00:23:40 Chris: Won’t someone think of the DevOps engineers?

00:23:43 Ned: Mmm.

00:23:45 Chris: But on the bright side, they’ll have a lot to talk about on the afternoon sprint.

00:23:49 Ned: [laugh] The stand-up? Yeah. “Everything’s broken.” [laugh].

00:23:54 Chris: So does that makes sense? Does that sound about right, like, the gist of why there would be such a crisis?

00:23:59 Ned: So, the critical thing that you’re hitting on there is the fact that most mature software development shops, build and deploy multiple times a day. And so, what that means is, multiple times a day, they’re pulling in the dependencies and building out the application and deploying it to their environments. And a lot of them should be deploying to a development or staging environment first, which should have caught this issue before it rolled out directly to production, but I’m sure there are some pipelines that say, “Well, if it’s this level of change, we don’t need to do that because it’s such a minor change.” And you just have the fact that workloads themselves—the applications that run—tend to be very short-lived. We’re living in the era of containers and Kubernetes, and so the actual lifetime of a particular running instance of that application is going to be pretty short. Could be on a few seconds to a few minutes for a container to exist, and then poof, it’s gone. And if you are grabbing ‘latest,’ [laugh] instead of grabbing a specific build for that container, that could also lead to some issues. So, it’s surprising that this wasn’t caught in the staging environments, but chances are that this direct-deploy to production was what was actually happening in a lot of cases, or they had regular rebuilds happening outside of their standard code promotion process. And since, quote-unquote, “Nothing had changed,” during the new build, there was no reason to do testing on the new build. Because it was the same code. It worked yesterday. Why wouldn’t it work today?

00:25:44 Chris: Yeah. And it’s also important to remember, the things that you’re talking about there, I mean, this was the… I would say this was the high watermark of that kind of aggressive programming. This was Facebook’s famous, “Move fast and break stuff,” time of living. You know, 2016 was back when they still cared about their product and wanted to actually make it better.

00:26:04 Ned: [laugh] Did they ever? That’s not nice. There are plenty of people at Facebook that cared about the product.

00:26:11 Chris: Yes, but if we’re talking about Facebook, the company…

00:26:15 Ned: [laugh].

00:26:16 Chris: I think this is amazingly illustrative of the way that we do modern software development. In favor of speed, we end up creating a world of Jenga. And you pull one block—as we saw clearly in 2016—the whole thing comes crashing down. Now, what I don’t have for you today is a solution. The first thought people probably have is, everything in NPM should be forever immutable. You put something out there for the public, it’s something that public relies upon, maybe it should stay there. The other side of that coin is that takes away the rights of the author. But then again, NPM showed clearly what they think about author’s rights in the way that they handled the crisis as it unfolded. Here’s the thing, though. At its core, ideas like NPM, packages instead of writing everything from scratch are not bad. Neither are they new.

00:27:15 Ned: No.

00:27:16 Chris: Neither are the problems that come with reusing someone else’s code. After all, anybody that’s taken even an intro to programming class knows that the ethos of responsible software development—and frankly, practical software development—is, write stuff you can reuse. One-offs are a bad idea. Create modulars; use them as much as you possibly can. Everybody’s favorite programming language—and by that I mean, like, six people at this point—C has the very concept of reusable code built into it at its core. Now, you can write every single inch of a C program on your own, but nobody ever does. They use what are called header files. And if you think anyone, even the great, magnificent Linus writes anything of consequence without header files you are dreaming. All you have to do is use a magic little code block called#include, and then you get to use these pre-written things to work at really sophisticated levels with strings if you need them, booleans if you need them, math if you need it, harder math if you need it, quantum math if you need it—I’m only slightly exaggerating; there’s a lot of math—error handling, and most importantly, standard input/output. They are a staple of every single C program you can imagine, and those header files are all exactly the same. I mean, sort of. There are differences over time. There have been versions of C that have come out and updates to those header files. The header files don’t have the same speed of development of NPM packages, by orders of magnitude. I mean, we’re talking about days over a period of five, six years. NPM packages can be updated every day. But counterpoint, they’re also not ephemeral. They come with the compiler, so they’re on your computer right now. Whether you use them or not, they’re there, and they’re always going to be there. But if there’s a security bug in a header file, guess what? There’s a security bug in every program compiled that uses it. One famous example is the string.h header file. This provides much more convenient ways to handle text strings. Unfortunately, they are as of this day, known insecure, capable of being abused to cause buffer overflows—in a previous life everyone would drink—

00:29:47 Ned: Mm-hm.

00:29:48 Chris: —if they’re not programmed around properly. So, I mean, which is better? You have an NPM that is ephemeral, rejects the rights of its authors’ autonomy, but gets a tremendous amount of attention and much faster development speed, or a header file, which doesn’t change for years at a time. Because surely it’s not better for people to be writing things like string.h or left-pad by hand from scratch for every single program, right?

00:30:22 Ned: I guess. I write everything in assembly, so it’s not really a problem for me [laugh].

00:30:27 Chris: I was totally going to throw some assembly jokes in here, but I was like this is already… impenetrable enough.

00:30:32 Ned: [laugh] I suppose. Yeah, it is, it’s quite the quandary, right? People want things to move at a certain velocity, and it’s become pretty standard in all of the modern programming languages that there is some sort of package repository, or a way to reference packages that are constantly maintained outside of your local system. And the way that it actually goes about grabbing those and compiling them is different, but the result is the same. I have to pin things to specific versions, and I have to really hope that the person who’s maintaining that library doesn’t decide, I’m going to blow away the repository, where it lives. That—you know, with Go modules, that is a possibility. Someone could decide tomorrow, “I don’t want anybody using my cool Go library anymore, and so I’m just going to delete or make private the repository that it lives on.” And that’s—that would be bad. NPM, by the same token, it’s also a huge security risk. If someone’s able to compromise NPM’s hosting in some way, they can distribute malware to thousands and thousands of applications, if they can, like, break into NPM. And there have been vulnerabilities on NPM that have allowed that exact sort of thing. So, it’s kind of like, you got all your eggs in one basket with NPM because it’s a hosting site. GitHub, kind of the same thing. If someone can hack one of those GitHub repositories, they can drop in some code in that library that everybody uses, and that can backdoor hundreds of applications. So—

00:32:15 Chris: Right.

00:32:16 Ned: Yeah, there’s the nice side that it’s very convenient, and I don’t have to wait for the next version of the programming language to roll out to get updates to these things, but at the same time, it also introduces tons of security vulnerabilities. But I guess, like, AI is going to fix all of it, right?

00:32:34 Chris: Oh, totally.

00:32:36 Ned: Awesome.

00:32:36 Chris: Actually, AI wrote this entire thing. I’ve been AI this entire time.

00:32:40 Ned: [laugh] Well, let’s hope not as we’ll—well, as you may already know about if you listen to our Tech News of the Week, this week. But hey, thanks for listening or something. I guess you found it worthwhile enough if you made it all the way to the end, so congratulations to you, friend. You accomplished something today. Now, you can go sit on the couch, fire up JavaScript and download some NPM packages. You’ve earned it. You can find more about this show by visiting our LinkedIn page, just search ‘Chaos Lever,’ or go to our website, where you’ll find show notes, blog posts, and general tomfoolery. We’ll be back next week to see what fresh hell is upon us. Ta-ta for now.

00:33:26 Ned: Hear my voice crack? That was nice.

00:33:27 Chris: Yeah, it sounded—sounded great. Very, very masculine.

00:33:30 Ned: [laugh] I don’t know what you’re saying [laugh].

Show Notes

The Time Someone Actually Broke The Internet

Episode: 95 Published: 2/29/2024

Explore the story of how 11 lines of code disrupted the internet, the battle between open-source ethics and corporate power, and the fragile nature of our digital world.

Technology, power, and unexpected consequences

In this episode, Ned and Chris take us through the dramatic story of a single code change that temporarily crippled the internet, emphasizing the delicate web of dependencies in software development. They explore the ethical dilemmas and power dynamics at play when open-source contributions clash with corporate interests. It’s a revealing look at the unexpected consequences that arise from the interconnected nature of modern technology.

Links Referenced:

Intro and outro music by James Bellavance copyright 2022


Chris Hayner

Chris Hayner (He/Him)

Our story starts with a young Chris growing up in the agrarian community of Central New Jersey. Son of an eccentric sheep herder, Chris’ early life was that of toil and misery. When he wasn’t pressing cheese for his father’s failing upscale Fromage emporium, he languished on a meager diet of Dinty Moore and boiled socks. His teenage years introduced new wrinkles in an already beleaguered existence with the arrival of an Atari 2600. While at first it seemed a blessed distraction from milking ornery sheep, Chris fell victim to an obsession with achieving the perfect Pitfall game. Hours spent in the grips of Indiana Jones-esque adventure warped poor Chris’ mind and brought him to the maw of madness. It was at that moment he met our hero, Ned Bellavance, who shepherded him along a path of freedom out of his feverish, vine-filled hellscape. To this day Chris is haunted by visions of alligator jaws snapping shut, but with the help of Ned, he freed himself from the confines of Atari obsession to become a somewhat productive member of society. You can find Chris at coin operated laundromats, lecturing ironing boards for being itinerant. And as the cohost on the Chaos Lever podcast.

Ned Bellavance

Ned Bellavance (He/Him)

Ned is an industry veteran with piercing blue eyes, an indomitable spirit, and the thick hair of someone half his age. He is the founder and sole employee of the ludicrously successful Ned in the Cloud LLC, which has rocked the tech world with its meteoric rise in power and prestige. You can find Ned and his company at the most lavish and exclusive tech events, or at least in theory you could, since you wouldn’t actually be allowed into such hallowed circles. When Ned isn’t sailing on his 500 ft. yacht with Sir Richard Branson or volunteering at a local youth steeplechase charity, you can find him doing charity work of another kind, cohosting the Chaos Lever podcast with Chris Hayner. Really, he’s doing Chris a huge favor by even showing up. You should feel grateful Chris. Oaths of fealty, acts of contrition, and tokens of appreciation may be sent via carrier pigeon to his palatial estate on the Isle of Man.