Transcript

[00:00:00] Ned: You’re an untrustworthy soul. As, we all know that’s fair. Jeez. Talking about souls, we are really just going hard on their religious talk this fine.

[00:00:11] Chris: Maybe we were just talking about shoe construction.

[00:00:14] Ned: That could be a two. All right, fair enough. There’s something that tends to crop up often, you know, shoe construction. I know. I’m often talking about uppers and souls and tongues.

[00:00:26] Chris: I live for it. I live for it.

[00:00:28] Ned: And aglets.

[00:00:31] Chris: At one time, you talked about waxed shoelaces. I needed to take a half a day off of work.

[00:00:35] Ned: It was too exciting. That’s too much content for a mere mortal to handle.

[00:00:41] Chris: Over stimulated, which is what most of my teachers called me. I thought it was a little inappropriate in grad school, but I guess you take what you can get.

[00:00:52] Ned: You do. I have to say, because we work in the tech industry and it is filled with jargon, I have to assume that other industries are equally filled with nonsense jargon and we just don’t know it. And part of it oh, yes. I don’t want to know all of it. But I am curious to know about sort of the subculture of jargon that exists in any other given industry. Like the shoe industry, for instance. I bet they just have words for all kinds of weird shit that you never thought about.

[00:01:27] Chris: Yeah, I saw not how it’s made, but something along those lines of a company that did shoe construction for just generic trainers. Nothing special. Not like the insane Nike $500 running shoes or anything like that. Just a regular pair of tennis shoes. Yeah, there were, like, 40 parts.

[00:01:47] Ned: Yeah. And I’m sure like the construction of the sole and the components that it’s made up of all have their own acronyms. And then the process by which you create it has its own set of weird things. It’s just delightful to know that all exists and I don’t have to care about it.

[00:02:05] Chris: Right.

[00:02:07] Ned: That second part is important.

[00:02:09] Chris: Somebody somewhere spent their entire life figuring out the difference between one molecule in the construction of glue that holds my shoe together.

[00:02:19] Ned: Exactly.

[00:02:20] Chris: And good for you, sir and or ma’am.

[00:02:23] Ned: Yes. I think it’s a useful perspective when we get too caught up in our own bullshit to take the long view and be like, oh, no one beyond this very small segment of people actually cares about this.

[00:02:37] Chris: Wait, are you implying that we have bullshit?

[00:02:42] Ned: I’m implying that the tech world in general has an overinflated sense of ego.

[00:02:51] Chris: Yeah, that’s true.

[00:02:54] Ned: And we can get a little true.

[00:02:55] Chris: But then, you know, point Counterpoint, corey’s latest newsletter, used the phrase piss balls, which I found entertaining.

[00:03:02] Ned: I have not read it yet, but I love it. On the other hand, and as we’ll cover in this episode, sometimes tech is kind of important.

[00:03:14] Chris: Wow. A segue this early in the morning? Are you on drugs?

[00:03:21] Ned: Impossible. Hello. Alleged human and welcome to the Chaos Lever podcast. My name is Ned and I’m definitely not a robot. I am a real human person who communicates regularly with other human persons. I can even do humor. Now, how about these? Why did the robot go on a diet? It wanted to reduce its bite size.

[00:03:45] Chris: Wow.

[00:03:46] Ned: Why did the robot cross the road? To get to the other side of humanity.

[00:03:54] Chris: What?

[00:03:58] Ned: We are now full of mirth, aren’t we? Doing humor truly brings all of us silicone based, life likelike simulations together, doesn’t it? It does. With me is Chris, who’s also here. Let’s chat. Chris, how are you?

[00:04:22] Chris: We already chatted. We have to chat again.

[00:04:25] Ned: That’s ridiculous that I’m asking you to think of things to say that you haven’t written down yet.

[00:04:31] Chris: All right, here’s a stupid question. Have you ever heard of Nick Cave?

[00:04:35] Ned: Yes. And the back.

[00:04:36] Chris: Oh, no, you haven’t stopped it.

[00:04:38] Ned: I’m ruining it for you. Sorry.

[00:04:42] Chris: So, no, right. Like, literally five minutes before we started, I saw an article where somebody used Chat GPT to fake a Nick Cave style song and put it on YouTube and Nick Cave found it and his response was expected. He says that the song, quote, sucks and is, quote, a grotesque mockery of what it is to be human.

[00:05:13] Ned: He’s just all sunshine and rainbows, isn’t he? You know, I thought what you were going to say is he found it and then he made it a song, because that would have been almost better. But his response is fine and I get where he’s coming from. I think if you wanted to target an artist that would be perfect for this sort of thing, counting Crows, I feel like you could easily write one of their songs using Chat GPT with a prompt. Something like, Angela is stuck in a tree. Go.

[00:05:47] Chris: But the tree is a metaphor for my deep emotions that I don’t really want to relate to directly. So here are a lot of allegories. Someone’s might drown.

[00:05:59] Ned: Indeed. And also, I love Bruce Springsteen, which.

[00:06:04] Chris: Sure, how that’s related, but I’m fine with it.

[00:06:06] Ned: Well, I just mean he does. As someone who came of age listening to Counting Crows, but not necessarily the earlier works of Bruce Springsteen, I did not understand the cultural impact that he had on others songwriting. And now that I’m a little bit older, I’ve gone back and listened to some things that were made before 1985. I’ve discovered that, oh, the past can have an impact on the future.

[00:06:37] Chris: Yeah.

[00:06:38] Ned: It’s almost like the past comes back to haunt us, whether that’s music or old ass technology.

[00:06:44] Chris: Another segue.

[00:06:46] Ned: I am so good at this.

[00:06:49] Chris: So, yeah, let’s do that. Let’s do that thing.

[00:06:53] Ned: All right.

[00:06:54] Chris: And this one is going to be fun because I get to use more material that we weren’t able to get to because we were on break. All right, so I want to talk about two stories that are similar in nature in the sense that they are in the same industry. There were two what do we call them? Let’s call them interesting oopsies that happened. The Southwest meltdown, people probably remember, notably causing something like 65% of their flights to be canceled over the holidays. And then just this past week, the failure of the FAA’s Notice to Air Missions or NOTAM system. That one caused around a three hour what they called a pause on all domestic departures. What nontechnical commonality do these two disasters share, you are probably asking yourself. The answer will surprise you. I haven’t been reading way too much marketing material. Why would you ask?

[00:07:59] Ned: It’s all floppy disks, isn’t it? It all came back to this eight inch floppy disks.

[00:08:06] Chris: I honestly, I was going to talk about that because that’s still there, but this is already long enough. So both the Southwest system and the NOTAM system are super complex, as we will see. But the problems were not caused because the systems were complex. The problems were caused because the systems were allowed to be deprioritized and generally forgotten about by the organizations that were responsible for their upkeep.

[00:08:36] Ned: Yeah, you know that if it ain’t broke, don’t fix it. That can be sometimes true, but if it’s slowly breaking, you might want to fix it occasionally.

[00:08:49] Chris: If it ain’t broke yet, it will go nuclear eventually, I think is how that should really be pronounced.

[00:08:55] Ned: Indeed.

[00:08:56] Chris: Yeah, there’s a couple of accent marks over the letters. It’s not easy to see unless you’re looking at it. It doesn’t work as well verbally, but anyway, unfortunate.

[00:09:05] Ned: All right, let’s start me out with Southwest.

[00:09:09] Chris: Okay, so let’s start with one of the problems with Southwest. And this is something that people who don’t really care about the airline industry, except for getting from point A to point B, probably don’t know. Southwest is unique in that they do not fly the same kind of patterns that other airlines do. Other airlines have a very classic hub and spoke model. They organize X amount of flights from one central place, let’s say Denver. Then from that hub, they have spoke flights that go out to an endpoint, and then the plane turns around and goes back to the hub. The endpoints can be to other hubs, but they’re only ever one link long. Now, there are exceptions to that, but basically that’s the rule, right? Incidentally, this is why sometimes you as a traveler have a situation where there’s no direct flight and your connection is somewhere completely bizarre. American Airlines, for example, might not have a direct flight from San Francisco to Albany. Albany is simply not an endpoint from San Francisco, but it is a spoke endpoint from Atlanta or from Charlotte. So now you buy one plane ticket and you get the opportunity to hockey stick yourself across the country.

[00:10:29] Ned: You know, I actually have always wondered why some flights that seem obvious to me that it should be a direct connection. Don’t have that available unless you search on a different airline. And I knew it had something to do with hubs and where the airlines were locating those hubs, but I didn’t understand. It was this very specific diagram that they had built or data model that they had built up for themselves. But wow, now I kind of get it. So Southwest does it totally different.

[00:11:02] Chris: Correct. Southwest southwest operates like a local airline, which is how they started. They were called Southwest for a reason. They only flew in the southwest of the United States. And what they do is operate on a point to point model. Every flight is a direct flight from one city to the next. And here’s the kicker. The planes do not just fly out and back. They fly in loops that are based on traffic. So a plane could be scheduled to start in, say, San Francisco, have a direct flight to Denver, have a direct flight to CLT, and then finally have a direct flight back to San Francisco.

[00:11:45] Ned: Okay.

[00:11:46] Chris: And yes, CLT is the airport code for Charlotte. It’s very funny. So, based on needs, this is a dynamic system. The loops could theoretically change every day. Okay, so there’s a couple of reasons that that makes things vastly more complicated than a hub and spoke model. So let’s line these problems up. Number one, they use a vastly more intricate and dynamic loop based system. Southwest prides itself on moving in and out of gates as fast as possible. In fact, in one or two airports around the country, southwest allows passengers to deep plane from the front of the plane and the back of the plane. They are the only major airline in the United States that do that, and they can only do it in two airports. Right.

[00:12:37] Ned: Now, when you think about it, it makes sense. You got two doors.

[00:12:41] Chris: Exactly. Now, additional complexities or variables, the planes, as long as they’re not in a maintenance window, can fly 24 hours a day, seven days a week, nonstop. Sure, humans cannot slackers, and you need humans to fly planes.

[00:13:04] Ned: For now.

[00:13:06] Chris: Southwest runs Lean. They have exactly enough planes to handle all of their loops, excluding planes that are in maintenance. Southwest also has humans who work for them, who take vacations, say, hypothetically, around the holidays. So this is a lot of variables to keep track of. And luckily, and I’m using gigantic air quotes here, southwest has a system to handle this. The system is called Skysolver, and it is a commercial, off the shelf package that was sold by GE and, quote, heavily customized by Southwest. Skysolver was first offered on the market by GKE in 2006. And while I cannot find a definitive date as to when it went into service, my guess is Southwest version is from somewhere between 2006 and 2010.

[00:14:03] Ned: So it’s at least twelve or 13 years old.

[00:14:07] Chris: Yep. I forget if I talk about this later. But that package is also end of life, so just keep that in the back of your head.

[00:14:16] Ned: Sweet.

[00:14:16] Chris: So the system that they built grew organically and works okay? It works pretty well the majority of the time it’s fine. The software itself is, in fact, slow and very old. And if the software breaks down, gate attendants are reduced to reworking their routes for both themselves and other dependent flights by a glorified phone tree. Okay, so it works especially when the weather is good, because that is a variable I neglected to put in my list of problems. The weather. As I was doing this research, I saw someone online called Southwest, quote, the best fair weather airline in the world, which is, as you will see, a backhanded compliment with all of these variables stacked up and lined up and dependent on one another. The problem is, if there’s a problem with the system anywhere, it has the capacity to affect traffic everywhere. And if there are problems in more than one place or one of these loci of variables, forget about it. So let’s fast forward, or I guess in this case, rewind to the holiday system or the holiday season. System failure. This year, the weather was bad in more than one place.

[00:15:49] Chris: Shocking, the system was overloaded and then stopped working.

[00:15:56] Ned: That sounds bad.

[00:15:59] Chris: Yeah, not good. When your only recourse is a phone tree, there’s only so much one can overcome, right? The end result for Southwest was approximately 17,000 flights being canceled between December 21 and the end of the year, which, in parlance, again, is bad.

[00:16:23] Ned: For many people.

[00:16:27] Chris: This had way more problems along with it in terms of people getting stranded, but also people losing luggage. It was an absolute fiasco.

[00:16:36] Ned: Right?

[00:16:38] Chris: Now, here’s the kicker. There are reports from pilots, gate attendants and other employees that Skysolver has been inadequate regarding the size of the Southwest network going back as far as 2015. This is online. This is not hard to find. Issues with Skysolver have occurred on a small scale regularly. This is why we know about the phone tree. Employees talk about it and complain about it all the time online. And it’s not even that. This December’s incident was an outlier. There was a major sky solver based spate of flight cancellations around this time last year for exactly the same reasons.

[00:17:22] Ned: So two things jump to my mind. One is the nature of complex systems and how adding a variable doesn’t increase the complexity in a linear way. It tends to do it in an exponential, if not factorial, way. So I add one more variable. I haven’t increased the complexity of my system by two or ten. I may have increased the complexity of my system by 100 or 1000.

[00:17:48] Chris: Right.

[00:17:49] Ned: So as a growing airline, in theory, as Southwest adds more flights and more destinations, they are ever increasing the complexity of the calculations that skysolver has to do exactly right. The other thing that I immediately think of is the impact of weather and the fact that weather systems due to climate change have become less predictable and more extreme, more severe? More severe, so less predictability, more complex systems. And I’m guessing the next part is that they haven’t kept up with that system. Kind of spells disaster.

[00:18:34] Chris: That would be correct, yes. There have been no serious efforts to replace Skysolver, which is problematic because, as I said before, skysolver as the generic software sold by GE end of life.

[00:18:48] Ned: Right.

[00:18:50] Chris: All they have are patches and vague statements that they would, quote, do better. Now, employees and people that work with and for Southwest are getting a little sick of this, with the head of the pilots union stating, quote, the company has had its head buried in the sand when it comes to its operational processes. And it which is probably a common refrain to a lot of people who listen to this podcast.

[00:19:17] Ned: It seems familiar now, not a technical.

[00:19:20] Chris: Thing, but adding insult to injury for a lot of people, southwest received approximately $3 billion in public funding from the COVID relief effort. Southwest then took that $3 billion and used it to affect stock buybacks, to float the price on Wall Street instead of plowing it into the business and fixing things like the Skysolver situation. So taking all that and trying to boil it down, my perspective on the issues, how they came about looks like this. Number one, southwest runs a complex and unique flight plan model. Step two, this required Southwest to run unique flight plan software. Three, that software was a behemoth and also a unicorn. A behemacorn.

[00:20:11] Ned: I’ll accept it.

[00:20:12] Chris: Thank you. Four, the business saw the software working, quote, fine and consistently neglected to update it. Now, fine in this instance means that yeah, sure, there were repeated problems, issues, and people complaining loudly in public, but they weren’t so bad that the business wanted to spend money on this ridiculous. Five, importantly, Southwest kept growing back to your exponential point. Number six, see step four. Seven, the business started running razor thin on personnel to save money. Eight, the holidays happened and scheduling became hard. Even asking people to work overtime has its limits. Nine, the weather happened, the software couldn’t keep up. And ten, catastrophic failure.

[00:21:13] Ned: Yeah, it’s my favorite flavor of failure, really. I mean, when I go to the grocery store, I always go for the catastrophic you know what? The chocolate swirl is just not as good.

[00:21:26] Chris: It’s just missing some sort of a joy of eva.

[00:21:29] Ned: Yes. Or she’s shouting fruit.

[00:21:33] Chris: That’s not how you pronounce that.

[00:21:34] Ned: That’s how I pronounce it.

[00:21:36] Chris: Get out. So not only was this failure predictable to many in the industry, many Southwest employees have been warning for years that it was going to happen, and the business did nothing. It’s okay, though. If you check Twitter, you’ll see that.

[00:21:54] Ned: They’Re real sorry thoughts and prayers.

[00:21:58] Chris: Estimated losses from revenue from refunds having to be given and a recently announced goodwill credit program for people that were affected is going to end up costing Southwest nearly a billion dollars. Now, airlines are giant and they make a lot of money, but in terms of profit, they run pretty thin. They basically wiped eleven months of profit off the board because they thought it would be better to do cost savings and not replace Skysolver aggressively. Ouch.

[00:22:32] Ned: There’s also the long term impact of this where people who may have chosen to book on Southwest will now think twice, especially around the holidays.

[00:22:42] Chris: Right. It’s a reputational damage that’s going to take years to figure out how much that cost and how to undo it. And the really sad thing about all of this is that Southwest recently got a new CEO about a year or so ago, Bob Jordan, who was a 34 year veteran of Southwest as an employee, an engineer, and a leader. He had a computer science degree and worked for Real as a programmer and tech team lead at companies like HP and at Southwest before becoming an executive. If anyone was in a position of authority who would be able to understand something so basic as tech debt is bad, you would think it would have been him. You would think.

[00:23:35] Ned: You would think, alas.

[00:23:38] Chris: Alas. So let’s put a pin in that and move on to part two and talk about the FAA’s failure from just this past week. Now, we’re not going to be able to get into as much detail because they’re being a little tight lipped. So a lot of what I’m going to talk about is based on supposition and people saying things online, so your mileage may vary. See what I did there?

[00:24:02] Ned: I see what I did there. Yeah. Yeah, it’s funny. It’s fine. Let’s move on.

[00:24:06] Chris: So a lot of people might not necessarily exactly know how intimately connected the FAA is to airlines and the airline industry. Okay, wait. Well, somebody might not know that.

[00:24:22] Ned: I was going to say it’s kind of like in the name.

[00:24:28] Chris: Basically, the FAA owns the sky. If you want to fly a plane or anything larger than like a 490 grams drone, you have to deal with.

[00:24:39] Ned: The FAA and they’re coming for your drone.

[00:24:42] Chris: Yeah, they approve pilots, they provide clearance, they approve flight paths, and generally work to make sure that planes are not taking off landing or flying anywhere near hazards or especially other planes. To do this, they rely on a lot of systems. The one that we had the issue with this week is called NOTAM Again. That stands for Notice to Air Missions. And what it does is provide real time updates to pilots both on the runway and in the sky about abnormal situations or hazards. So these can be big and obvious. Things like a disabled plane causes the closure of runway six or not necessarily obvious hazards like, say, the change in the path of a flock of migrating birds. That’s the sort of thing a pilot is going to need to know in.

[00:25:40] Ned: Advance, especially if you’re flying one of those smaller planes, right? Yes. Hitting a flock of geese is not just going to damage an engine. It might knock you out of the sky.

[00:25:51] Chris: Exactly. And what you want for an airplane is to stay in the sky until you land it out of the sky on purpose?

[00:25:57] Ned: Yes.

[00:25:58] Chris: When that’s an accident, it becomes problematic. So that’s the purpose of NOTAM. And according to many pilots, it’s actually, as a program, quasiuseful generally a huge pain in the ass because the other thing that they will do is err on the side of caution and send a NOTAM alert for things that can possibly never impact a flying plane. So the joke that was made online is no tam alert came in because somebody raised a crane 50ft off the ground 5 miles from the airport. That could conceivably be a no tam transmission. Okay. And if you want to see how insane and you were talking about jargon before, take a look at what a NOTAM transmission transmission looks like. Wikipedia has an example of one from Europe, but they’re the same effectively all over the world. Pilots have to read them, though they are not legally allowed to take off until they are up to date with all relevant no tam alerts. And the way that it’s defined as relevant is a certain amount of mileage from where you are and from your flight path out. So 5 miles out in either direction, if there’s a NOTAM in there, you need to read it.

[00:27:16] Chris: For busy airports, this can mean as many as 200 no tam alerts that have to be read before takeoff. So if you’re ever wondering why the plane is just sitting still doing nothing, it could very well be the pilot is grinding through NOTAMs fun.

[00:27:35] Ned: Awesome.

[00:27:36] Chris: But remember what I said. Legally, they’re not allowed to take off until they’re up to date.

[00:27:42] Ned: Right.

[00:27:43] Chris: So why does this system exist? I think it’s obvious. It’s for security and safety and it’s been around for a while. It actually started off as a phonebased way for airports to notify other airports about issues. It was eventually migrated online. How it was brought online is still a matter of debate, but it definitely appears that it was a homegrown solution. And if the rumors are true, the running NOTAM system that failed was written in Ada and is running on hardware that’s probably older than the both of us, possibly combined.

[00:28:19] Ned: I don’t even know what Ada is, so that’s probably not a good start.

[00:28:24] Chris: I’ll tell you when you’re even older. It is, actually. So there were complaints about the usual jokes that the government never updates their software, and Transportation Secretary Pete Bootyg fiercely pushed back on the idea that NOTAM was out of date, saying that the system was, quote, constantly updated, which is its own kind of frightening. I don’t know about you, but I’m not sure I want the government doing DevOps. I mean, seriously. During this incident, the FAA had a running conference call that was hopefully trying to describe the process of recovering from the outage. The call was intended for airline professionals, airports, airline executives, et cetera. But they posted the number publicly and they didn’t have any mechanism in place to mute people that didn’t need to contribute. The world being what it is and Twitter being a thing. Yeah, I think you know what happened. Someone on the call actually had to stand up and yell, can we please stop the cat meowing?

[00:29:33] Ned: This is why we can’t have nice things.

[00:29:35] Chris: Yeah. So this is the level of technological expertise we’re working with. Maybe let’s just stick to waterfall deployments for the time being, perhaps.

[00:29:44] Ned: Yeah. Okay.

[00:29:45] Chris: And just because somebody is asking the question in their head. According to people in the industry, there was in fact, a NOTAM released about NOTAM being down.

[00:29:56] Ned: Cycle complete.

[00:29:58] Chris: Obviously, it didn’t go out on time. But you do have to love the commitment to the process.

[00:30:03] Ned: Well, it is the government. If there’s one thing they can do, it’s follow the process.

[00:30:09] Chris: So what happened here? This is where the details get a little scarce. What we were told is there was a, quote, damaged database file that was introduced into the system during a regular reboot. The system was not adequately checked. The fact that there was a damaged database file was missed, and this caused the system to go into read only mode, meaning no new NOTAMs could be added.

[00:30:36] Ned: That’s a problem.

[00:30:38] Chris: Remember how everybody has to be up to date before you can take off?

[00:30:43] Ned: Yes.

[00:30:43] Chris: No new note. AWS being added to the system means that the FAA had no choice but to ground every single plane in the United States for something like two and a half hours. The speculation on what a, quote, damaged database file means is running rampant. Was it corrupted input of a previous NOTAM entry? Was it a bobby table situation? Was it a bad block on the file system that was inadvertently backed up before it was read? Was it the world famous Oraclesque error in the sense that a software bug introduced by the database software itself inadvertently caused damage to its own data? We unfortunately simply don’t know. All we know is that the damage file was there long enough to be in the standard backup. The NOTAM system, as I said, was rebooted and, quote, not properly validated. When it came back up, there were problems. The solution to the problem, I kid you not, boil down to turn it off and turn it back on again.

[00:31:53] Ned: And because of the age of the hardware, wait 2 hours.

[00:31:57] Chris: Yes. So one likely reason this occurred is because there are overlapping systems in NOTAM. There is the very old system, which I alluded to before that’s running Ada USNS. And there is the new system FNS. FNS was implemented in 2013. The two systems are apparently still running in parallel as the migration to FNS is still not complete. Oh, and there’s a new new system called Swim that’s slated to go active in 2025. Now, I’m not going to bother breaking down those acronyms because let’s be honest, you don’t care.

[00:32:41] Ned: No.

[00:32:42] Chris: The failure appears to have occurred in the older USNS system, which allegedly only handles 20% of the traffic. But it begs the question why? What is unquestionable is that there are a lot of problems in their process with validation, updates, backups, and did I mention validations in their system? Clearly, either a lot of steps were skipped or were just not there in the first place. And because there is an ongoing nine years worth of ongoing migration happening, I am guessing both systems have a lot of cluges in place that just aren’t documented.

[00:33:25] Ned: Yeah. As someone who has been part of a migration from Lotus Notes to Microsoft 365, I can tell you that migrations take a super long time because the 80 20 rule applies heavily. The first 80% migrates, no problem, and then you have the remainder 20%. That is the weird crap that somebody got up to in Lotus Notes. And I’m sure there’s a similar situation here, but obviously nine years is unacceptable.

[00:33:54] Chris: Right. So the problems in both cases there are technical problems, sure. But the problems are organizational. In my opinion, Southwest and the government are both cash based businesses because let’s just call the FAA what it is, and based on this breakdown, I would call them both toxically risk averse. In Southwest case, they chose not to fund the updating or replacement of Skyscalver in any significant way. In the FAA’s case, though, it’s probably more likely that they barely have enough money to stay afloat. And any efforts towards migrations are what’s a worse word than optional unlikely?

[00:34:43] Ned: I don’t know.

[00:34:44] Chris: In both cases, the systems were able to mutate over a period of years into a solution that was, to put it kindly, rickety. And those are the worst kind. You get to a point where you’re too scared to touch it because if it breaks, you don’t know if you’re going to be able to fix it.

[00:35:06] Ned: Yes, we’ve all accidentally unplugged the desktop that was running in the back of a closet and suddenly that has taken down the entire system that your business is dependent on. Just plug it back in.

[00:35:21] Chris: You unplug the coffee machine and the power goes out in the whole building. I don’t remember integrating that into the system. But the two incidents show two different sides of the same problem, and that is the risk of doing nothing. Yes, a decade ago, you solved a crazy complex technical problem and you should be. Lauded for that. Congratulations. Here’s a lolly. But that’s not the end of the situation. Things change. Your business gets larger, the amount of customers gets bigger. If you don’t update as things change and look to the future, always, all you’re doing is setting yourself up for another, larger problem down the line.

[00:36:08] Ned: Absolutely. And when you’re talking about typical technology companies like the Facebooks of the world, if they don’t keep up to date, they’ll lose their customers and go out of business. But will anybody be really that inconvenienced to buy? It like we’ll all eventually migrate to other platforms and be perfectly content with that. When it comes to the FAA failing, we don’t have a choice. We can’t move to a different organization or a different platform. It’s the FAA. Our only option is to fund them better and demand they do a better job.

[00:36:49] Chris: Yes. And for everybody that’s chuckling to themselves and saying, oh, well, I’ll just move to Canada, canada had their own NOTAM problems.

[00:37:01] Ned: So it seems like it’s more of a systemic issue than it is anything specific to the US.

[00:37:08] Chris: Yeah, the issue there. They claim that it’s not related, but again, nobody’s giving us any details, so we really don’t know what caused the problem. So it could just be a coincidence, or it could be that they shared a code base.

[00:37:22] Ned: Either way, it always pays to invest in the maintenance of your systems exactly.

[00:37:28] Chris: And make it an organizational priority.

[00:37:31] Ned: Yes. Lightning round.

[00:37:35] Chris: Lightning round.

[00:37:39] Ned: Magic Leap makes the entirely expected jump to the enterprise. If at first you don’t succeed, ride the Hypetrain to industry town. Magic Leap was a much lauded AR startup who kept their cards close to their chest and overpromised on virtually every dimension of their software and hardware. Anyone who got to play with the tech had to sign onerous NDAs, and they all seemed to sing well. Described as strained praise about what Magic Leap was doing, the headset itself was priced at over $2,000, a price tag that in 2019 seemed ridiculous and has become no more reasonable over the past three years. With a change of CEO in 2020, Magic Leap pivoted from the consumer space to the Enterprise, a group that constantly complains about money while wasting metric truckloads of it on tech to solve what are actually organizational problems. At CES 2023, Magic Leaps booth was focused on industrial and medical challenges that could be solved with its AR headset in the same vein as what Microsoft has done with their HoloLens technology and Google with their revised version of Glasses. While virtual reality may be slowly finding a toehold in the consumer space, thanks to the efforts of Facebook and HTC, I think that augmented reality is better suited for business applications, and the shift in focus by Magic Leap seems to confirm that theory.

[00:39:12] Ned: Of course, Apple is rumored to have a headset ready for release in 2023, so I have to see if that’s another iPhone moment or closer to ping.

[00:39:24] Chris: Pine, we’ll talk about CES.

[00:39:27] Ned: Well, I did bring it up TLDR.

[00:39:31] Chris: Most everything there was terrible 60 security wise, of course. A representative of the EFF stated, quote I think there’s a chronic problem with consumer electronics. They are not giving people the full picture they need to evaluate whether they want to use these tools. Now, realistically, she could have just stopped at there’s a chronic problem with consumer electronics. Example just one of the items that received the coveted worst in show listing was a urine scanner from a company called Whittings. The device is health oriented and analyzes hormones in your P requiring an account and a cloud connection, of course, and in their terms of service proudly claim that they will go ahead and store your data indefinitely. How many things did you count that were wrong with everything I just said?

[00:40:29] Ned: All of it correct.

[00:40:33] Chris: The problem, as always, when it comes to this kind of cheap, disposable swill and I’m not talking about the urine here is that companies are only focused on cost and speed to market. Safety and privacy are way below the horizon, and that is only for things that actually exist. As usual, CES was also filled with plenty of vapourware. The other problem that we have had is that people continue to try to make things connected for literally no reason. One headline from the event read, and I swear to God this is true, quote the oven won’t talk to the fridge, which what are we doing here? I cannot believe I have to say this out loud, but your fridge does not need to be online, and neither does your toothbrush. Stop it.

[00:41:29] Ned: But my scale does absolutely stop it. The inevitable march to Rust continues with Chrome metal based jokes about blah blah blah blah, whatever. 2023 will truly be the year of Rust, as Dana Janssen’s from the Chrome security team has announced that third party Rust libraries will be supported by Project Chromium. With the long term goal of bringing Rust directly into Chrome proper. Rust offers several advantages over C Plus Plus, the language currently used by Chrome and Chromium. Google’s software engineering team has done the hard work of adding a production rust tool chain to their build process. But one cannot simply smoosh Rust and C Plus Plus together and expect good things to happen. Instead, they are waiting in cautiously restricting rust components to third party libraries that can only interact with C code via APIs. This prevents possible memory safe flaws in the C Plus Plus code from leaking over to the Rust side of the house. C Plus Plus code, on the other hand, will be able to call Rust directly through a number of tools like CXX and bind gen. With the Linux kernel adopting Rustbased code, Mark Rosinovich publicly issuing C for Rust, and now the prominent and popular Chromium project adopting Rust.

[00:42:57] Ned: It certainly seems as though the language is ascendant. Yet another project pioneered by the good folks at Mozilla, who never seem to get the love they deserve. And no, I’m still not paying for the firefox. VPN.

[00:43:10] Chris: No, it’s still moved. Stop it. Intel Releases Next Generation CPU Line sapphire Rapids some other stuff too. It has been literally years in the making, but the fourth generation Zeon CPU is finally here. Sapphire Rapids was released. Too much applause. We applauded, right? Did we?

[00:43:38] Ned: No, didn’t.

[00:43:40] Chris: Okay. The chip does have some interesting things going for it. It’s got a max of 60 cores, which is higher than last generation max of 40. Quick reminder that the AMD max is 96. Intel also has implemented a pay as you go model for four different on chip accelerators. Cannot wait to see how fast that gets hacked. And then the interesting stuff PCIe five DDR, five compatibility, and what we’ve all been waiting for a CXL 1.1 interface.

[00:44:15] Ned: Yay see, now we clap.

[00:44:20] Chris: In this FHIR Rapids line, there are a dizzying 52 models to choose from, with prices going from a low of $415 to a peak of 17,000.

[00:44:33] Ned: I’ll take two.

[00:44:36] Chris: The 17,000 gets you the 8490 H with all of the fixings. 60 full cores, 120 full threads, all four accelerator types, fully enabled. The chip has a max of 112.5 megabytes of L, three cache, and a 350 watt thermal design power rating. That’s a lot.

[00:44:59] Ned: A little bit, yeah.

[00:45:01] Chris: Intel also released a new GPU series called Ponte Vecchio. I’d love to tell you more about it, but that’s literally all the article says. Oh, intel also released a new I nine codenamed, Raptor Lake. This is a tweak on the already out there I 913 9000 chip, but it adds a PCORE turbo capability, allowing two of the cores, I think two, to hit 6 GHz. All it takes to do that is all of the power on Earth. Seriously, this thing’s base draw is 150 watts, and its max draw is 320. Wow. What kind of power supply do they think that people have these days? All of that, though, to be fair, for a reasonable $699 base price. Interesting though, that Ryzen AMD’s CPU line is showing the kind of performance you can get from low power designs, whereas intel is content to brag about just blasting more electrons down the wires of their old stuff and calling it a win.

[00:46:12] Ned: Come on, guys, we needed a win. We paid Pat Gelsinger $180,000,000 to come work for us. Itunes replacement for Windows. Hope it flogs the alpaca’s posterior.

[00:46:25] Chris: Nailed it. Yeah, nailed it.

[00:46:27] Ned: In this, the year of our Lord 2023, I was shocked. Shocked to discover some people still use music programs on their computers. It’s like finding out someone still has a pager. But like, why? But amazingly, yes, people are still using itunes on Windows. Even though the app always ran like a drunken cow in a swamp and received about as much attention as your fourth child. A while back, Apple wound down itunes for Mac OS, choosing to break the app’s functionality into four separate apps for Music, TV Storefront, and Device management.

[00:47:04] Chris: OOH, don’t forget podcasts. That needed to be its own application, too, apparently.

[00:47:08] Ned: Naturally, Windows is receiving similar treatment with dedicated apps for Apple Music, Apple TV, and Apple Devices, but notably none for podcasts. The last one, Apple Devices, is for updating and synchronizing devices. What baffles me is that the ipod touch was discontinued last year, and there hasn’t been an eye device that required wired syncing since the nano and shuffle were discontinued in 2017. I suppose there are still nanos and shuffles out there, but it’s safe to say most people have moved on to streaming and wireless devices, or chose to use a different application to manage their nano like I did. And I guess that’s what this announcement is really all about. Apple doesn’t want to even pretend to support itunes anymore. Instead, they’d like dedicated apps for streaming stuff, the stuff that makes them money. And if you have a device that needs to be reset, there will be a separate, simple application for that. For those who use itunes as a media library management tool. Poor, poor souls. I’ve heard good things about music, Be. And of course, there’s always Win, amp, llama and all. I expect you to have feelings about this, Chris.

[00:48:32] Chris: We can’t have feelings in the lightning round.

[00:48:36] Ned: There’s no place for your feelings. Hey, thanks for listening or something. I guess you found it worthwhile enough if you made it all the way to the end. So congratulations to you, friend. You accomplished something today. Now you can sit on the couch, book a flight, wait for that flight to be canceled, and once again, consult your economics textbook on the Sunk Cost fallacy. You’ve earned it. Or at least you’ve paid for it. You can find me Horse on Twitter at ned 1313 and at heiner 80, respectively, or follow the show at Chaos underscore Lebra, if that’s the kind of thing that you’re into. Show notes and the newsletter are available@chaoslever.com. You can sign up for our weekly newsletter that encapsulates our episode as well. AWS, the lightning rounds and anything else we felt like talking about, but you probably shouldn’t. Podcasts continue to be better in every conceivable way. We’ll be back next week to see what fresh hell is upon us. Tata for now. I had this whole thing that I wanted to talk to you about that had to do with Wedding Crashers and how Owen Wilson’s character is an irredeemable, horrible person.

[00:49:50] Chris: Monster.

[00:49:51] Ned: Yes. But I felt like we’d already gone on enough at the beginning, so we have to sidebar that for next time. Time.

[00:50:01] Chris: I just want to know what Nick Cave has to say about it.

[00:50:05] Ned: Just make up a song. Nick Cave writes a song about Wedding Crashers and send it to him.

[00:50:13] Chris: Nick cave, shows up in front of my house with a sharpened shovel. That’s what happens.