Transcript

[00:00:00] Chris: I really wish you had that queued up because that would have been very funny.

[00:00:03] Ned: I could try one of these buttons. I don’t know what’s going to happen. Yeah, that was close.

[00:00:09] Chris: Was that a high hat?

[00:00:10] Ned: It was. There we go. There’s our live studio audience, everybody.

[00:00:16] Chris: I didn’t hear them. I’m sure they’re very excited, though.

[00:00:19] Ned: They sure were. I don’t really use the buttons that are on this thing.

[00:00:22] Chris: You don’t say. You were supposed to. That was one of your New Year’s resolutions.

[00:00:28] Ned: That can’t possibly be right.

[00:00:32] Chris: What do you suppose, like, percentage wise, worldwide, all time? What are the percentages of New Year’s resolutions that get resoluted?

[00:00:44] Ned: Resolutide or that I’m going with sub 5%. Yeah.

[00:00:52] Chris: I mean, I was absolutely thinking single digits. No two ways about it.

[00:00:55] Ned: Yeah. For me personally, I would say definitely around that range.

[00:01:02] Chris: Right.

[00:01:02] Ned: When I make any kind of resolution to begin with, which typically I avoid them at this point. I think there was a time in my 20s where I did make resolutions with the full intention of actually following.

[00:01:14] Chris: Through and gosh darn it, it was going to happen.

[00:01:18] Ned: And then maybe I grew somewhat wiser, definitely older, and realized it’s pretty unlikely. So why don’t I just do what I’m going to do, right?

[00:01:29] Chris: In living color.

[00:01:31] Ned: It’s the only way to do it. Sketch TV like mad. TV in Living Color really gave birth to some of the biggest stars that we have today. Well, maybe not today, but that’s going.

[00:01:48] Chris: To say you’re forgetting the part where you’re old. AWS, hell. Right?

[00:01:51] Ned: Yeah. Okay. I’m not forgetting.

[00:01:53] Chris: Jim Carrey retired.

[00:01:54] Ned: I’m choosing to forget Chris. Well, Jim Carrey was one of the obvious ones. But you also have Jlo. She was one of the original Fly Girls on a little bit.

[00:02:06] Chris: That’s true.

[00:02:07] Ned: You know who else was a Fly Girl? Paul Abdul.

[00:02:11] Chris: I think that I knew that she.

[00:02:14] Ned: Was also one of the La, the leaders. She was the head of the Cheerleaders, I think.

[00:02:19] Chris: Right. And did the coordinating and all that stuff.

[00:02:23] Ned: Yeah, she certainly a dancer. First singer, second for sure. Maybe dancer, entertainer, then singer. Not that she’s a bad singer, but I wouldn’t put her in, like, the top tier.

[00:02:38] Chris: I mean, she’s fine.

[00:02:39] Ned: She’s fine. I saw her live.

[00:02:40] Chris: She’s fine.

[00:02:42] Ned: I got to see her as part of a, you know, early 90s nostalgia tour. It was new kids on the block. Boys to Men and Paula Abdul.

[00:02:55] Chris: Wow.

[00:02:58] Ned: Yes. Wow. I was really there for the opening acts, which was Paula and Boys to Men, but, you know, Andrea was there for New Kids because that was, like, squarely in her preteen years.

[00:03:14] Chris: Right.

[00:03:16] Ned: Yeah. It sort of skipped over me for the best, really. Yeah. But they were fairly entertaining, so I’ll give them that. They still enjoy what they’re doing. Or at least give close. Yes, that’s the one. Close enough that I believed it. Who knows?

[00:03:37] Chris: I really don’t have any new kids on the block. Hot takes. Sorry. I think one of them was named Donnie. And that’s all I have to say about that.

[00:03:46] Ned: There’s definitely a Jordan in there of some kind. Michael, at least one. Michael, probably two.

[00:03:55] Chris: And then BIFF and Eric.

[00:03:58] Ned: Yeah, sure, we’ll round out with that. I mean, I’m pretty sure Donny was the bad boy or as close as they had to the prototypical bad boy of the time.

[00:04:07] Chris: Right? The bad boy of Martha’s Vineyard.

[00:04:11] Ned: Oh, yeah. Stomping around, upsetting everybody over in Nantucket. Keep that riff raff out, shall we?

[00:04:24] Chris: Surely.

[00:04:25] Ned: Hello, alleged human, and welcome to the Chaos Lever podcast. My name is Ned, and I’m definitely not a robot. Beneath this dermis of living cells, a thought that I do not find it all repugnant pumps, the viscous red fluid that I did not steal from anyone else, giving me life, love and laughter. Ha ha. Blood. It’s so funny. Enjoy us. Right, Chris?

[00:04:50] Chris: And nutritional wait, what are we talking about?

[00:04:55] Ned: More of a supplement, really. I mean, I don’t know what we’re talking about. The joy of life, of course.

[00:05:02] Chris: Yay. Yeah, I just finished reading a rereading, actually, a book about the Essex. Do you know what the Essex is? The tragic tale of the Essex?

[00:05:13] Ned: I can’t say that I do, but I feel like you’re going to tell me.

[00:05:16] Chris: Long story short, it’s the book that inspired Moby Dick or I’m sorry, it’s the story that inspired Moby Dick, which.

[00:05:23] Ned: As we all know, is a very happy story, full of joy. Yeah.

[00:05:26] Chris: Lots of fun, lots of excitement. There are sparklers. At one point. I recommend reading it to your kids as a bedtime story.

[00:05:35] Ned: Could you imagine? Well, it certainly put them to sleep, if nothing else.

[00:05:40] Chris: Well, yeah, until you got to the cannibalism.

[00:05:43] Ned: Yeah, valid point, valid point. I did try to read my middle one, the Linode, the Witch and the Wardrobe. And we got to was essentially the crucifixion of Aslin, and we had to stop too much. Little too much.

[00:06:05] Chris: That’s fair.

[00:06:06] Ned: Yeah. And I don’t blame her, because you know what valid that is not to go off on a tangent, but we’re already here.

[00:06:18] Chris: Not that we’ve ever done that before.

[00:06:20] Ned: No, nothing against Catholicism, be Catholic, be happy, whatever, but some of the imagery is kind of brutal and it’s just there in the church all the time. And I’m like, wow, if you’re like a five year old looking up and you’re like, why does that poor man have nails through his wrists? That doesn’t seem comfortable.

[00:06:43] Chris: No, I don’t think so. And he’s real cold.

[00:06:46] Ned: He does not look like a happy person.

[00:06:48] Chris: January outside.

[00:06:50] Ned: Yeah. So I guess I don’t know, maybe I prefer if I have the option to happy Jesus and the thumbs up Jesus, if possible.

[00:07:02] Chris: What was his name? Shit.

[00:07:05] Ned: Buddy Jesus.

[00:07:06] Chris: Buddy Jesus. That’s right. Maybe that you can barely ever find anywhere anymore.

[00:07:14] Ned: I actually got in trouble my first legitimate office job because I’d been working retail up until then. I think we used AOL Instant Messenger because this is like way before the office communicator and all those. And so for my profile icon, I chose Buddy Jesus.

[00:07:35] Chris: Yeah, that’s not going to go over great.

[00:07:39] Ned: My boss came over a little bit later and was like, so Ned, you’re going to have to change your icon to something else. I’m like, but she’s like, yeah, and we all learn something to go over well, sure. Ding. I was like, oh, I am not amongst my kin anymore where this sort of joke is just enjoyed. There’s going to be some people in this organization that will take umbrage, and I need to be I need to be conscious of them.

[00:08:08] Chris: Yeah, you shouldn’t, umber anybody you should rage.

[00:08:12] Ned: People, though, all the time.

[00:08:14] Chris: Only machines. Think about it.

[00:08:18] Ned: Think about it. Let’s talk about something else. Let’s talk about some tech garbage.

[00:08:23] Chris: Let’s do it.

[00:08:24] Ned: So I wrote 1800 words. Get ready. It’s actually, I think, a pretty interesting topic, though. And feel free to interrupt me as you usually do. So the topic is cloud boring. Damn it. I knew it. I shouldn’t have given you the in cloud repatriation or the art of two things being true at once. That can happen.

[00:08:50] Chris: Wee.

[00:08:50] Ned: Yeah. Not everything is just black or white. So just as a baseline, whenever a trend becomes large enough, there’s always going to be detractors. Naysayers, those who claim that the trend isn’t happening or isn’t as widespread as everyone thinks, or it isn’t ready for prime time, or it’s already over, or is just inherently terrible. Kubernetes, janko jeans, manic pixie, dream girls, walking nachos. These have all had their series of detractors, and most of them were wrong. There’s nothing wrong with Jacobs.

[00:09:32] Chris: There’s definitely nothing wrong with walking nachos.

[00:09:35] Ned: Yeah, I don’t know what the hell those people are.

[00:09:37] Chris: We’re going to have to have some tough conversations, if that’s what you think.

[00:09:39] Ned: I do not. I’m fully in favor.

[00:09:41] Chris: All right, good. So carry on.

[00:09:43] Ned: They all go through this cycle, the trend, the Hype cycle, maybe you call it. And right now there’s a bit of an ongoing debate over whether this whole public cloud thing was all one massive mistake and maybe we should all go live under a rock and drink PVR or move our Compute back on Prem. It was one of the two. And honestly, I was barely listening.

[00:10:06] Chris: What I heard was drink tbr on prem. Was that not what you said?

[00:10:15] Ned: It’s the only way to drink it. Really? Who knows? So the opening salvo of the current debate was really kicked off by Martin Cassado and Sarah Wang over at a 16 Z with their contentious post titled the Cost of Cloud a Trillion Dollar Paradox. Now, the core premise of the post, despite it lacking a paradox entirely, is that as a company grows from a startup to a big boy SAS, their technology needs and their cost model also change. And maybe running everything in the cloud doesn’t make financial sense for them. Their example is the evergreen poster child for so called cloud repatriation, good old Dropbox. Everybody likes to bring up Dropbox because Dropbox claims to have saved $75 million by moving their workloads off of the public cloud. There’s a lot of provisos asterisks after that 75 million, but sure, we’ll give it to him.

[00:11:26] Chris: Sounds good in a newspaper headline.

[00:11:28] Ned: Sure does. Now, that post was early 2021, I believe, and since then there’s been a series of posts from David Heinenmeyer Hansen, known as Dhh, for obvious reasons, who is, among other things, the CTO of Basecamp hey and 37 Signals. I don’t know if that’s how you say hey, but that’s how I’m going to say it. Hey.

[00:11:55] Chris: I thought you were just saying hi to Basecamp. I didn’t realize it was even a company.

[00:11:58] Ned: It sure is neat. Is it? Anyway, sure. So the series of posts he has document the efforts of Basecamp hey and 37 Signals to move off of AWS and Gcp and onto their own infrastructure, housed in a set of colocation facilities. And according to his back of the envelope calculations, hey and Base Camp will save a total of $7 million over five years by making this move. So that’s not $7 million every year. That’s over five years. So it’s not still, it’s money, right? It’s not the $75 million that Dropbox is claiming, but $7 million. It’s hard to shake a stick at. I’ve never been able to shake a stick at any millions of dollars, so.

[00:12:56] Chris: I saw a million dollars once. It doesn’t take up as much space as you’d think.

[00:13:00] Ned: I’m reminded of that. God, I don’t even remember what show it was where guy offers him like, a hundred thousand dollars and he opens a suitcase and it’s one stack of bills. Yes.

[00:13:12] Chris: Yeah, I don’t know what that is either, but that was very funny visual.

[00:13:15] Ned: Gags for an audio medium. Anyway, so a series of questions comes to me. Is there a massive repatriation wave coming? Who is pushing that narrative? And what can you, the listener, the technology person in theory, take away from all of this? So let’s try to separate the signal from the noise a little bit. I want to start with the fact that basically everyone involved in both sides of the conversation has an agenda. The public cloud providers continue to tout their superiority of solutions versus traditional on prem, although their focus has changed in interesting ways. The hosting providers and colocation facilities like Equinix and Digital Realty are hoping for a wave of revenue to roll in from the shifting tide. And the hardware vendors are all excited to push for repatriation because, let’s face it, they get much better margin selling to the enterprise than they do to the cloud hyperscalers who are for their part, increasingly building their own hardware solutions.

[00:14:24] Chris: Anyhow AWS laughs hysterically in graviton too.

[00:14:30] Ned: It’s not just Graviton too, it’s the what’s the name of their hypervisor? The network chip that runs their hypervisor. Nitro. Yeah. I mean they don’t fab it themselves, but they sub that out to an ODM that’s not going to any of the hardware vendors. I don’t know this for a fact, but I got to imagine a significant portion of their EC Two fleet is in the same boat.

[00:14:57] Chris: Would make sense.

[00:14:59] Ned: Yeah. So if the vendors can sell back into the enterprise, they’d be more than happy to do so.

[00:15:07] Chris: Right.

[00:15:08] Ned: So my point here is anything that you read about cloud repatriation from anyone, including me, I probably have an agenda somewhere. It’s usually written by someone with a vested interest in one outcome versus the other and it should be consumed with an appropriate dose of skepticism. So instead of simply trusting what all of the paid for bloggers have to say, I thought we could look at this from a financial perspective and look at what the various parties are doing. If it’s a major shift away from the public cloud, I would expect to see roughly three things declining revenue and capex spending in the public cloud providers. They’re not expecting additional growth, so they’re not going to buy a bunch of hardware and their revenues are going to go down because less people are moving to and possibly moving away. Colo providers like Equinix should see increased revenue because people need somewhere to move to and they’re probably going to have to spend a bunch on capex for that build out.

[00:16:11] Chris: Right.

[00:16:12] Ned: And for the hardware vendors, you should see a shifting of revenue mixes favoring their infrastructure groups selling into the enterprise. That’s what I would expect to see. So what do we see?

[00:16:27] Chris: It would make sense because if they’re doing the more because this was one of the things that was so appealing about the cloud in the first place was the idea that, hey, it costs AWS a lot less to buy a million servers than it does for John’s Flower Shop to buy five.

[00:16:43] Ned: And they could pass those savings on to you. Not that they will, but they could.

[00:16:49] Chris: That’s what’s really important.

[00:16:51] Ned: Yeah, and we’re not going to get into it, but if you do track the cloud costs, how much it costs to run a particular instance of EC Two or how much storage costs, et cetera, the costs have not really dropped in the last few years. They’ve kind of just stayed steady, state.

[00:17:11] Chris: It’S flatlined, more or less. But one of the things they haven’t done is gone up.

[00:17:15] Ned: Right, they haven’t gone up.

[00:17:17] Chris: Unless you think about new services coming in to supplant old services, which is kind of a little weasely way to get around the whole thing and say we’ve never dropped or I’m sorry, we’ve never raised prices.

[00:17:27] Ned: True. But I think that the point is they should be reaching economies of scale where compute continues to get cheaper, and it’s not doing that. It’s staying the same price.

[00:17:38] Chris: Right, that’s fair.

[00:17:40] Ned: And that’s because they’ve found what the market will bear, essentially. So since we’re dealing with mostly public companies here, we can actually dig into the numbers because they have to publish that information. Let’s start with the public cloud providers and their growth rate and revenue, since, well, we’ll go a little farther back than the 2021 blog post from May 16. Z. So looking at AWS, and this can be a little tricky because Amazon doesn’t necessarily break everything out for AWS, but it does to a certain degree. So if you look at the growth rate that’s calculated as the percentage of change over the previous year’s revenue, their growth rate has been trending downwards since 2015, with occasional bumps in 2018 and late 2021, also known as the Pandemic. So the reason the growth number has been trending downward is because it’s very hard to maintain consistent growth based on ever larger numbers. If we’re looking at actual dollars, AWS generated $80 billion in revenue in 2022 versus 45 in 2020.

[00:18:58] Chris: That’s good, right?

[00:19:00] Ned: Seems like they’re doing all right.

[00:19:02] Chris: Yeah.

[00:19:04] Ned: So even if they have a similar increase over the next two years, just because of the way that numbers work, the percentage growth will actually get smaller. So what happens? So what about azure? How’s that going? If I look at the annual reports, their growth has been steady or on ever increasing revenue numbers. So they’ve been averaging somewhere around 20% plus or minus year over year growth. Now, you might find different numbers if you go and search, because the way that they break out stuff in their financial reporting is they have a group called Intelligent Cloud, which includes Azure, but might include some other things. And then some of the Azure stuff is in a different segment. So sometimes they give you an Azure number that’s different than what’s? Intelligent Cloud. I went with Intelligent Cloud because that’s what’s actually on their balance sheets.

[00:19:56] Chris: Right.

[00:19:57] Ned: The point is they’ve been averaging 20% year over year growth, and in 2022, they pulled in 75 billion in revenue. So they’re almost the same size.

[00:20:08] Chris: That’s also good.

[00:20:09] Ned: Yeah. And then we’ll have Gcp bringing up the rear. The trend is similar, slowing growth over increasing revenue since at least 2017. Now, they’re still well behind the other two, with only 26 billion in revenue for 2022. But they also have the highest growth rate of 37%. Smaller numbers, bigger growth percentage. So if cloud repatriation is rampant, I am not seeing it in the numbers. For the public, cloud growth is healthy but slowing, as the law of large numbers tends to do. We are still seeing year over year increases of more than $10 billion for AWS and Azure. So they’re doing all right. Okay.

[00:21:00] Chris: And that probably doesn’t go into anything about some of the smaller players that are still growing and expanding and cloudifying for other less major Big Three style audiences.

[00:21:12] Ned: Yeah, I mean, you have Digital Lotion, which I believe is still independent. I don’t know if they’ve been bought up. You have Lenode, that was scooped up by Akamai last year, I believe. And you’ve got other companies like Vulture. And then there’s the more international ones. What’s the big? Chinese, 110 cent? Yeah, ten cent. There’s a lot of cloud providers out there, and they seem to be doing okay. So what about the colocation providers? These are essentially real estate companies that own data centers. Let’s start with Equinix. They’re the biggest. Their revenue is comparatively tiny, with a mere 7.26 billion for 2022. Their growth has been holding steady at about 10% year over year growth since 2019. So nothing drastic, nothing bombastic there.

[00:22:09] Chris: What about stable progress?

[00:22:12] Ned: Yeah. What about Digital Realty? Well, their revenue in 2022 was 4.69 billion. So they’re about two thirds of size. Yeah. And their growth has actually been shrinking over the last two years. They had a major spike in 2020 with 24% growth as a result of that whole pandemic thing. And now they’ve settled back to 6% growth in 2022. So if I look at those numbers and I look at the trends, I’m not seeing dramatic growth on the hosting side, which I would expect to see. But hey, maybe folks aren’t repatriating to colocation facilities. Maybe they’re going back to their old data centers, their old haunts. Which brings us to the hardware vendors. So let’s pick two giants of the industry, hewlett Packard, Enterprise, and Dell. Now, in terms of revenue, HPE has seen annual growth of about 3% for the last two years and a 7% increase over the last reported quarter. Now, if we dig into the actual segment earnings, because they don’t sell just servers and storage, they actually have a bunch of different segments, including a finance one that’s growing. Compute and storage only represent 28% of their revenue. But if you look at the growth numbers for both segments, they were either minimal growth, like 1%, or at a loss for the last two years.

[00:23:44] Ned: The growth area for them was actually an intelligent edge, which we’ll get to. What about Dell? Now, Dell has done pretty well for itself with revenue of $101,000,000,000 for 2022, which is just a gobsmacking, ridiculous amount of money. But bear in mind that Dell has a lot of different segments. Now, that’s a 16.7% increase over 2021. Now that’s the company as a whole. And we can’t forget about some of the weird financial machinations happening over there as the company was brought public again and also divested VMware. So I don’t know what to make of some of those numbers. But the segment that actually deals with infrastructure, it’s called the Infrastructure Solutions Group. They logged 34 billion in revenue for 2022. Which is growth of 4% over 21, and they shrunk 4% in 2021 and 7% in 2020. So while the overall picture of Dell is quite rosy, their ISG segment is not the one driving growth. So that’s all the financial data I I dredged up and looked at. But based off of that, I am not seeing a massive migration of workloads from the public cloud to colocation facilities or existing data centers. And honestly, it’s not just me.

[00:25:14] Ned: I found another series of excellent blog posts by Charles Fitzgerald over on platformomics, and he came to the very same conclusion. So with all that in mind, why are people making all this noise about cloud repatriation?

[00:25:33] Chris: Do they just like to say repatriation?

[00:25:35] Ned: I think it is a fun word to say.

[00:25:38] Chris: It definitely is a lot of syllables.

[00:25:42] Ned: You know what they say, if there’s smoke, there’s fire. Right now, if there’s smoke, there’s some idiot with a fog machine hoping you will come see his 80s hair metal tribute band. And frankly, George, it’s all about the it’s moving to the Grunge era.

[00:25:57] Chris: We have moved on.

[00:25:58] Ned: Okay, but there are people making a lot of noise, including the aforementioned A 16 Z and Dhh. Do they both have agendas? Yes, absolutely. A 16 Z is a venture capital firm, and they primarily invest in, get this, SaaS companies, and they want those companies to go public at massive multiples. In order to do that, they have to offer up companies that are doing what the market wants, and suddenly people appear to care about profits again, which is wild.

[00:26:36] Chris: Yeah, we should probably not have ever gotten away from that as like a thought. It’s like an idea.

[00:26:40] Ned: Growth at all costs. So in their myopic view, any SaaS company of significant size should probably move off of the public cloud to lower costs. And they might be correct with a giant Asterisk. Now, Dhh, for his part, likes attention and money, but mostly attention. So just to put things into perspective, he’s talking about moving the bulk of compute, not data. He still has like five petabytes of data locked up in AWS, but it’s compute onto four racks of servers across two colo facilities, two racks per facility. That’s it.

[00:27:28] Chris: Not much.

[00:27:29] Ned: No, we’re not talking about Google scale architectures here. We’re talking about four racks of gear for his people to manage across two data centers. And the colo is providing the remote hands, power and cooling. His departure from AWS and Gcp will barely be a fart and a strong wind for them. So, bearing all that in mind, do these posts make any good points? Yeah, I’d reckon they do. Moving to the public cloud was once touted as a cost saving exercise with all kinds of total cost of ownership calculators to push you in one direction or the other. Surprise, it was always cheaper to move to AWS. What if you use their Tco calculator?

[00:28:21] Chris: Oh, of course.

[00:28:23] Ned: But now the public cloud providers have mostly dropped the cost savings narrative in favor of innovation and breadth of services and locations and AI. And AI, well, that’s one of the services, right? I personally can cheaply run four servers in a rack in my basement for basically no cost. But what if I want to use an advanced AI service or store petabytes of data or perform complex data analysis? I might need the cloud for that. And that’s probably what the cloud is actually good for. So I think what we’re seeing is some amount of acknowledgement that the public cloud isn’t the end all and be all. Some people are swinging a little too far in the other direction, decrying the use of public cloud in all of its forms. Those people are silly. I’d like to think that the emerging consensus is somewhat more nuanced that the public cloud is good for some things, the private cloud is good for others. And then there’s the nebulous edge, which also can solve some problems. As always, the best place for a workload depends entirely on the requirements.

[00:29:39] Chris: It’s a question that everybody hates the answer to, because it’s always that it depends.

[00:29:46] Ned: Yes, that’s what technology almost always comes down to. But I think it’s important that the rest of that sense is it depends on these things. Go measure those things and then you’ll have your answer. So if you’re a technology professional, the thing that you might be wondering is, what does this mean for me? And what should I be worried about? If you’re a person who got their start in the cloud or has built their career on cloud, don’t worry, there’s no massive wave of repatriation that’s going to crush you on the shoulders of on premises. The public cloud providers continue their growth. They don’t show any signs of stopping $80 billion, 75 billion billion dollars and growing for the top two. It’s going to be all right.

[00:30:41] Chris: Yes.

[00:30:43] Ned: At the same time, the rise of open source standards for packaging and managing applications has made things a little more portable than they were in the past. Shuffling between public cloud to on prem to edge is less of a burden, and so you should expect cloud operations to infiltrate all aspects of your environments. So maybe learned about cloud operations more than any of the specific cloud providers.

[00:31:13] Chris: And networking?

[00:31:14] Ned: Yes.

[00:31:15] Chris: Don’t skimp on networking.

[00:31:16] Ned: Dear God. Networking. You can see the public cloud providers getting in on this trend as well. AWS has local zones and outposts. Microsoft has Azure stack and Azure arc. Google has Anthos. They’re all trying to bring their services to your on prem and colo environments and also out to the edge. And I would like to recommend paying special attention to what’s happening at the edge, which before you say anything, Chris, I realize that is a nebulous term that means almost nothing.

[00:31:49] Chris: That’s the pointy part.

[00:31:54] Ned: So for our purposes, when I say Edge. What I mean here is that there’s something important happening in two spheres, and one is the rise of Akamai and cloudflare as more than just Cdns. Both of them have taken steps to get into application hosting and security. And I would expect a whole new class of applications and services to live at their edges. I don’t remember all the different things, but what was it? Two weeks ago, we talked about the Mastodon service that cloudflare had put together using all their weird components, right? And it’s not running on a VM. It’s just built from all these serverless abstractions. That’s the kind of thing you can do now. So that’s one edge. The other Edge that I would reference is that of nontraditional computing environments. So think cell phone towers, factory floors, oil refineries, or just more generally, industrialized IoT. That is why HPE has seen any real growth in their revenue. Their Intelligent Edge segment had growth of 15% in 2021 and 11% in 2022. And that’s what it focuses on. So you can expect to see more Edge hardware being deployed in your organization, and you’re going to be on the hook to manage some portion of it.

[00:33:17] Ned: So get familiar with it. So, in conclusion, public cloud will be with us for the foreseeable future. But that doesn’t preclude the growth of hosting on Prem data centers and the Edge looking to the past, it’s not like we’ve been scaling back on our use of technology in the last 50 years. Quite the opposite. Thanks to Moore’s Law, this is in no way a zerosum game, and I expect that all sectors will continue to see growth. It turns out two things can be true at once. Cloud and on Prem are both the solution and this whole grand enterprise was probably a mistake.

[00:34:01] Chris: Yeah. And one thing that drives a lot of repatriation efforts is people being surprised by what the cloud is, how different it is to operate in that environment. And most importantly, I think, for especially big companies. What’s the cost model? Not the same as buying a bunch of servers from HPE or Dell every five years. You talked about those total cost of ownership calculators, and those are important, but they’re also incomplete, deeply flawed. As someone that has seen an AWS bill before for a company of even moderate size, I am sure you remember the literal thousands of line items.

[00:34:44] Ned: Yeah.

[00:34:46] Chris: The whole idea that you can put in, I want this server and I want this server and I want this firewall, and that’s the end of your calculations when it comes to the cost of the cloud almost always ends in tragedy. And somebody saying the cloud is a joke. It costs too much money. Let’s put these servers back on my secretary’s desk where they belong, right.

[00:35:09] Ned: There’s. This idea that the costs of onprem infrastructure are going to be predictable and the cost of cloud is not correct. And I’d like to say that’s wrong, but it’s actually quite true. It’s hard to predict the cost of cloud unless that you actually put some controls in place.

[00:35:30] Chris: Well, yeah. Setting up an environment where people can create unlimited resources in the cloud isn’t a recipe for disaster.

[00:35:38] Ned: Right.

[00:35:39] Chris: Setting things up where you don’t have designs or tags or models that show what are the things that actually matter to this workload. Because if you have twelve systems in your data center that are running and doing nothing and they just run, it’s a waste of electricity, but that’s about it.

[00:35:57] Ned: Exactly.

[00:35:58] Chris: If you have that in the cloud, that could be thousands of dollars a month. That’s just going down the drain for services that you are literally not using at all. FinOps is one thing, cloud operations in general is another one. Being disciplined about how you build out in the cloud is super important. I often wonder when you hear about these repatriation efforts, I often wonder how much of a Wild Wild West show was it at Dropbox. Exactly.

[00:36:30] Ned: They also were running at a certain scale where you could make the argument that running steady state in your own data centers makes more financial sense because.

[00:36:41] Chris: It uses that’s definitely true, but all I’m trying to say is you want me to waste $75 million in the cloud? I can waste $75 million in the cloud.

[00:36:50] Ned: Oh yeah.

[00:36:51] Chris: With nail polish?

[00:36:52] Ned: Easily. Yeah.

[00:36:55] Chris: So I’m really curious to see how that goes in terms of the pushback from the pro cloud side. It’s not that the cloud is always like you said, it’s not a zero sum game. And the answer is always going to be it depends. But there are absolutely use cases where the cloud will make more sense both operationally and financially. But if you don’t design it right, you’re never going to get there and you’re just going to automatically assume the cloud is a problem.

[00:37:20] Ned: Right. And what was interesting is that a 16 z post really went with the SaaS operating model, which is a very specific type of company that has to solve for specific types of problems early on. You need to be able to iterate quickly. You need to be able to set up and fail fast. So the cloud is the perfect environment for you. As you move on and grow and mature and you reach some sort of steady state, then cost management becomes your challenge and bringing stuff on prem helps you handle that portion of it. So it’s like, yes, your cost model works great for this very specific type of company. But not every company is a SaaS company, even though that’s all you deal with.

[00:38:03] Chris: Right. A little bit of a myopic perspective.

[00:38:08] Ned: Precisely. So I think one of the interesting side effects of cloud repatriation in that it happens at all is some companies made the journey to the cloud while they were in the cloud, were able to make some fundamental changes. To their application architecture and their operation of it. And they’re going to reap those benefits when they move it back down and maybe actually be more efficient in their data center because they’ve learned those lessons from the Cloud.

[00:38:42] Chris: Possible. That is possible too.

[00:38:44] Ned: But you also have the scenario where they moved everything up, got scared by the cost, and then started an immediate two year program of moving everything back and learned nothing. I think it’s going to happen, too. All right. Lightning round.

[00:39:01] Chris: Lightning round. Tried to switch it up there. Was that exciting?

[00:39:06] Ned: I felt excited.

[00:39:08] Chris: Good. AWS changes, billing and cost management permissions taking effect both yesterday and also in July. So this is an IAM article. Buckle up.

[00:39:23] Ned: Fair enough.

[00:39:23] Chris: In July, AWS will be retiring IAM permissions that use the service prefix AWS dash portal for access control in the AWS billing and cost management console. They will also be sunsetting two additional actions that are listed under purchase orders. Specifically, view purchase orders and modify purchase orders. Overall, these newer controls that are taking the place, they’re going to benefit administrators looking to minimize unnecessary permissions in the console make everything a lot more granular net positive. However, it’s going to be very important for people to keep an eye on the timeline, as there is a hard stop to the old permissions, ie. Once we get to a specific date, they’re going to stop working. Now, what’s going to happen next depends greatly on when the accounts to be affected were created. So here is the high level breakdown of what is happening. Note that this applies in both service control policies SaaS and in IAM policies. Now, if your account was created before March 6, 2023, which is to say, yesterday, you will be able to continue using the existing AWS portal actions and the other two sunsetting ones. If your account was created after March 6, 2023, you will be required to build in the new fine grained Actions.

[00:40:53] Chris: In either case, on July 6, 2023, all previously working AWS portal actions will cease functioning, and access that relied upon them will stop working. Note that everything that I just said is only a concern if you’re using custom policies. If you’re using AWS managed policies, they will be updated automatically. Confused? Of course you are. It’s. AWS IAM. Confused is what they’re going for. Which, of course, means that this is one of those rare times where you really should read the Linked article.

[00:41:37] Ned: Need kubernetes native Storage Akamai is on that? Listen, I’m not the one who named the company. I just make the jokes. Kubernetes storage startup ondat is being acquired by CDN and Edge player Akamai, originally called Storage OS. Ondat has been around since the early days of Kubernetes. Starting in 2015, their solution is deployed directly on a Kubernetes cluster and interfaces with the Linux IO stack to talk to whatever storage you want to present akamai has been on a bit of an acquisition terraform lately, scooping up Philadelphia based cloud provider Linode. Back in February of 2022, they cited the On Dat acquisition as a way to round out the storage functionality in the Linode Kubernetes environment service. Chris and I have both attended Tech Field Day events where Ondat was presenting, and I think it’s safe to say we both felt that they were looking to be acquired at some point. I’m glad to see they got their wish, even though I wouldn’t have thought Akamai would be their eventual home. So congratulations to the Ondat team. I hope they continue to improve the product as it is integrated into Akamai’s offerings.

[00:42:51] Chris: Elon Musk chose not to pay his AWS bill. Hilarity. Ensues. Elon Musk, that bastion of fiscal integrity that he is, apparently has chosen yet another controversial and poorly planned cost saving strategy. What if I just don’t pay AWS?

[00:43:13] Ned: Wow.

[00:43:14] Chris: Back in 2020, Twitter signed a five year contract with AWS. Much ballyhooed. At the time, the goal was to move a significant amount of services over to AWS from Twitter’s own hosted data center environments. Now, a good amount of Twitter does, in fact run on AWS, but a lot of those migrations never happened for totally unexplainable reasons. So Musk, in his infinite wisdom, decided he wasn’t using the services, then he wasn’t going to pay the bill. Not how contracts work, bubby. Aws’s first retaliation was to stop paying their own bill to Twitter for the ads that they run there for both AWS and Amazon Studios. Now it will be fun to see how this all plays out. AWS has indicated that they have no interest whatsoever in renegotiating their contract with Twitter. And thanks to Musk, Twitter doesn’t have anything close to the engineering resources to migrate off of AWS anyway. According to reports, Twitter is still $70 million behind on payments. And this is all fun and games until AWS just, like, turns off Twitter. You know what, they can do that.

[00:44:37] Ned: Their image service went down today. I don’t know if that had anything to do with AWS, but it sure did for a while.

[00:44:43] Chris: Nice.

[00:44:45] Ned: Software providers may be held liable for bad cybersecurity can’t we just give you more identity protection? Vouchers? No. The National Cybersecurity document has just been released by the Biden administration, and within is a proposal to hold companies liable for vulnerabilities in their software or services. To say that this is controversial would be an understatement. Of course, it’s really only controversial for vendors who make shitty, vulnerability riddled software that major industries rely on. Imagine for a moment a software vendor was similar to a company that produces physical goods, let’s say a catheter manufacturer. And the shoddy workmanship of the catheter led to viral infections in one out of every ten patients. Should the manufacturer be held liable for the piss poor quality assurance standards and the real damage caused by their negligence? Yeah, they probably should, and so should software vendors who manufacture shitty software. It’s almost like there should be some kind of software development standards that they’re held to, especially in industries like health care, transportation, finance. And it shouldn’t be up to the customers of those vendors to perform the testing or pay for the consequences.

[00:46:11] Chris: Attack Vector for Major Lastpass Breach identified as an unpatched Plex server.

[00:46:23] Ned: That just tickles me.

[00:46:24] Chris: I don’t know why LastPass’s reputation for security and data stewardship has kind of been through the ringer. Over the past 18 months, we had a security breach that was massive enough to earn it a large amount of showtime a few months back. Few short days ago, they were hit again finally, over a series of press releases, Lastpass let us all know what happened. Their press release stated that the attack vector was, quote, an earlier, unpatched version of Plex media server on a Last PaaS DevOps engineers home computer unquote. This vulnerability allowed malware with a keylogger to be installed, giving the attackers all the usernames and passwords that they needed. The press release then Lowkey tried to blame Plex for the breach. Plex responded by saying that the CVE in question was from 2020, was resolved with a patch the first day that it was announced, and was updated in Plex, quote, roughly 75 versions ago. Zing while there is plenty of blame to point at Last PaaS, this is also a reminder that Internet facing services need to be taken seriously at all times, even when it’s just a service you use to watch TV.

[00:47:51] Chris: Regardless of the Lastpass service security failures, and there were many, this is just bad personal security hygiene. If this employee had kept his server updated, the breach would not have happened. If the Plex was on its own server that was on its own isolated Vlan, then this breach couldn’t have happened. Defense in depth, people and separate your servers from your desktops.

[00:48:21] Ned: That’s the one.

[00:48:22] Chris: Yeah.

[00:48:25] Ned: Mobile web congress. More like metaverse web congress. There’s a frankly over ambitious article on TechCrunch detailing their experience at MWC 2023, and specifically all the hype over the metaverse, which appears to be MWC’s raisin deache. I don’t know how to say that.

[00:48:54] Chris: It’s close enough.

[00:48:55] Ned: Okay, honestly, more thought went into the article than the whole of the metaverse hype train brain trust, but it’s also a bear of a read, so allow me to distill it down for you. .1 no one knows what the metaverse is, but they’re really excited. .2 the ISPs in the EU. See the increased bandwidth needs of the metaverse as a new excuse to add more fees and start charging app providers .3 The MWC is not really interested in debating ethical and societal implications of the metaverse and .4 people really need to stop referencing Snow Crash if they haven’t even read it. I know Neil Stevenson can be a little bit long winded, but this ain’t the Baroque cycle regardless, it would appear that the Metaverse, in whatever form it takes, will be arriving in about five years. And I personally look forward to saying the same thing five years from now. Hey, thanks for listening or something. I guess you found it worthwhile enough if you made it all the way to the end. So congratulations to you, friend. You accomplished something today. Now you can sit on the couch, sink into the Metaverse, and remember your place in the universe is as insignificant spec on an most unremarkable rock.

[00:50:11] Ned: You burned it. You can find me or Chris on Twitter at ned 1313 and heiner 80 respectively. Or follow the show at chaos underscore Lebr, if that’s the kind of thing you’re into. Show notes and the newsletter are available at chaoslevercom. If you like reading things, we’ll be back next week to see what fresh hell is upon us. Tata for now.

[00:50:34] Chris: So I know you already feel old, but you want to feel old?

[00:50:38] Ned: Hit me with your best shot.

[00:50:39] Chris: Do you want to know what came out approximately this month, 30 years ago?

[00:50:47] Ned: Go ahead.

[00:50:49] Chris: Simcity. 2000.

[00:50:50] Ned: I hate you.