Transcript

[00:00:00.250] Chris: It just like in life, we’re borderline functional.

[00:00:05.650] Ned: And which side of that border we’re on changes day by day.

[00:00:09.090] Chris: Hello. Wait, that’s your line?

[00:00:11.060] Ned: No. Hey, what was that all about? I should make you do it sometime.

[00:00:16.360] Chris: We don’t need to talk about the details of our functionality.

[00:00:21.420] Ned: No, that’s probably true. And it would just be sad for everyone fault. So let’s just dig right in. Hello, alleged human, and welcome to the Chaos Lover podcast. My name is Ned, and I’m definitely not a robot. I am a real human person with a Microsoft MVP award that is definitely not about to be imperiled because of this podcast. What I do, I do out of love. Human love for a giant corporation and not robotic harmony with my overlords. With me is Chris, who is also here. Hey, buddy.

[00:00:54.850] Chris: Is that better?

[00:00:57.970] Ned: It’s hard to say. We have deeply unhealthy relationships with giant corporations these days.

[00:01:04.960] Chris: It actually feels like the name of an unreleased Jonathan Colton song.

[00:01:10.710] Ned: Which part? Human love for a giant corporation or robotic harmony with my overlords? Or just the whole thing?

[00:01:17.030] Chris: Let’s call that a single.

[00:01:18.680] Ned: Yeah, I can’t wait.

[00:01:20.640] Chris: Got the A and the B side just nailed down and we’re admitted one.

[00:01:25.370] Ned: I love it. I like that. We’re talking about a physical medium in the year of our Lord 2023.

[00:01:33.230] Chris: So a single was a physical device. It was called a record, and they were released with one song per side.

[00:01:43.490] Ned: That sounds dubious. I don’t believe you.

[00:01:46.770] Chris: Back in the before times.

[00:01:50.530] Ned: Oh, back cassettes have been making a comeback, which is mind boggling to me. Like, I don’t like everything about the audio on cassettes was demonstrably worse than CDs.

[00:02:05.850] Chris: But you could make mixed tapes.

[00:02:09.290] Ned: You could make mixed CDs.

[00:02:10.740] Chris: It wasn’t even that hard eventually.

[00:02:14.890] Ned: It’s weird because the people who are collecting tapes now are too young to have nostalgia about collecting tapes. So I’m just like, Whatever, man.

[00:02:24.160] Chris: They’re probably doing it ironically.

[00:02:28.350] Ned: Aren’t we all just doing life ironically? Let’s talk about some tech garbage. Microsoft honesty is the worst policy. Wait a minute, that doesn’t sound right. Their best worst policy, I suppose. Hasn’t been a good couple of weeks for old Microsoft. Let me count up the hits. I’ve got a few here. They had their Q Four 2023 earnings call, and even though they did very well, their guidance was generally down for the next year or the next quarter. So their share slipped by 4%. That activision deal that just keeps dragging out still hasn’t closed. Azure Active directory was renamed to Microsoft entraid. Yeah, apparently there’s a critical flaw with the power platform that just got patched. And also an Azure ad private key was swiped with national security Ramifications. Wait, whoa. One of those slightly more important than the other. And it would be the Azure ad being renamed.

[00:03:41.270] Chris: Entra? More like.

[00:03:45.430] Ned: Yeah. Generally for a security product, you don’t want to invite people to enter. And so it’s weird that they called it Entra. I had some pretty clear feelings about the renaming, and I shared them on LinkedIn, and apparently I wasn’t alone because that posted fucking crazy numbers. It’s the most popular post I have ever written by a wide margin.

[00:04:10.010] Chris: I mean, it’s good that you’re finally understanding social media, and in order to get attention, all you have to do is complain about things.

[00:04:19.570] Ned: Yes, completely. Unbounded rage seems to be what really attracts the followers on LinkedIn.

[00:04:28.150] Chris: Don’t forget about kids these days. Avocado, toast, et cetera, et cetera.

[00:04:32.630] Ned: So generational put downs. Yeah. Okay. So complaining about Zennials or something, is.

[00:04:40.110] Chris: That what we’re calling them?

[00:04:41.500] Ned: I honestly don’t know. Anyway, so while I do have my own feelings about the renaming, the thing that actually mattered was this loss of the Azure Ad key that was like, way worse. Yes. So let’s dig into what the key does, why it matters, and what Microsoft’s response was. Spoiler it wasn’t great. So we’re going to start with what happened and do a little bit of a timeline thing here, something that is super clear to follow in an audio medium. But hey, you tuned in, you made your media bed, now you got to lie in it. On May 15, the Storm 58 China based threat actor started using a forge using forged authentication tokens.

[00:05:31.930] Chris: It’s not like, a great name.

[00:05:34.410] Ned: Microsoft came up with that name. I don’t know why, and I didn’t dig any deeper. June 16, microsoft discovered the malicious campaign and started taking steps to mediate it and also notify impacted customers.

[00:05:52.030] Chris: That would be a month later, a.

[00:05:53.360] Ned: Few months later at home. And the only reason they knew is because a customer brought the issue to them. So it wasn’t like their crack security team found this issue on their own. Someone who was using their Outlook web. Well, Outlook.com Outlook Web Access noticed some weird traffic coming from IPS that were unexpected and escalated it up to Microsoft. So that’s how it was actually discovered. So that was June 16. June 26 through the 29th, they revised a bunch of OA token settings, OA being Outlook Web Access, and all the current MSA keys were replaced. July 11. You’ll notice this is almost a full month after they discovered the problem. They finally published two blogs detailing the malicious campaign of Storm Five. Five 8. July 19, they expanded cloud logging for per view audit to include the premium level for all customers. Two days after that, security outfit Wiz published a blog claiming that the MSA key extended well beyond the initial OA andoutlook.com scope that Microsoft was claiming. And on July 27, which, if we’re doing our math right, was about two weeks ago, senator Ron Wyden from Oregon wrote a formal wow words letter of complaint to CISA, the Attorney General, and the FTC accusing Microsoft of gross negligence.

[00:07:37.290] Ned: So that’s the high level overview. And where we are today. This is an evolving, you know, it’s going to be out of date by the time it’s published. That’s life.

[00:07:49.210] Chris: But on the brighter side, it means we can talk about it like four more times.

[00:07:52.900] Ned: At least we’re going to get some mileage out of this one. Yay for content. Before we get into the specifics of what happened, it’s probably good to get a broader context, as we are want to do on Chaos lever. So let’s talk about tokens and the St no arcade game. These are different kinds of tokens.

[00:08:16.470] Chris: Play me in Street Fighter two for money?

[00:08:19.070] Ned: Definitely not. Unless I can be Blanca.

[00:08:23.730] Chris: I accept your ridiculous terms.

[00:08:25.780] Ned: All right, I’m going to shock the shit out of you anyway. So the lingua franca of the modern Internet is tokens. And what are tokens? They are cryptographically signed packets of data that are used to determine access and permissions on a system to understand how tokens are used, who issues them, and who trusts them. We have to talk a little bit about OIDC and Sam’l. So sorry in advance for that.

[00:08:54.170] Chris: TLDR OIDC is the new one. Sam’s the old one. Go take a bathroom break.

[00:08:58.380] Ned: Carry on. Okay, so let’s start with a very simple example. Say you’re logging into the website for your local branch of antique spoon Collectors Anonymous.

[00:09:08.680] Chris: I love ASCIA.

[00:09:10.370] Ned: Who doesn’t? In the olden days, and sometimes right now, you would log into a site by providing a username and password that’s specific to that site. And when you submit your credentials through a web form, what happens? How does the website know you’ve logged in successfully? It uses a token, usually in the form of a cookie, which is not as delicious as it sounds.

[00:09:34.710] Chris: No, but what’s important is it means that you don’t have to reuse your password on every single page and every single time you log in. And usually they have some type of expiration. So you log in once you’re good, 30 days, 60 days, and then you have to log in again.

[00:09:51.870] Ned: Exactly. That web form is submitted to the site with a post request. And assuming you typed your password of spoon freak with the right number of exclamation marks, it’s twelve. The website would return a response with a token and that token would be then resubmitted by your browser with all future requests as proof that you’ve logged in successfully. And now the web server is happy.

[00:10:17.750] Chris: Right.

[00:10:18.150] Ned: And like you said, the token is typically written out to your local file system as a cookie, and it is good for a certain amount of time. Which is why when you go into your browser settings and you delete your browser cookies and history, you need to log back into all of your websites because all of those tokens are gone.

[00:10:37.790] Chris: Right. This is an example of a good cookie. This is helpful with convenience, making websites more efficient. There are many other kinds of cookies that we can get into in a different episode.

[00:10:54.390] Ned: Yeah, maybe we should do like a Cookie Monster episode.

[00:10:58.450] Chris: Write that down.

[00:11:00.130] Ned: I don’t have any paper.

[00:11:01.890] Chris: Well, remember that then.

[00:11:03.540] Ned: Sure. So that token, like you said, it typically includes a validity period. It also includes information like who issued the token, who it was issued to, and whether or not the token can be renewed. And to prove that the token was issued by the website in question, it will be cryptographically signed with a private key, which can be then verified with the site’s public key. So as a quick reminder, we’re not going to get super deep into this, but in the world of cryptography, a public private key pair can be used together to perform complementary operations. So you can encrypt data with a public key and then decrypt it with a private key, which is what you use when you’re doing like PGP with email. You can also sign data with a private key and then verify that signature with a public key. The public key is well, it’s public, so anyone should be able to get access to the public key. That’s kind of the whole point.

[00:12:06.940] Chris: I mean, it’s right there in the name.

[00:12:08.590] Ned: I know sometimes we got to spell these things out. The private key.

[00:12:14.090] Chris: Oh, you weren’t being literal. Sorry. Carry on.

[00:12:16.360] Ned: Not the moment. The private key also in the name should be kept private. It’s pretty important, so it should be closely guarded. That’s what we call foreshadowing in the biz. Having every single website run its own identity service is, let’s say, enormously wasteful. So standards like OIDC, OpenID, Connect were created so that you could have an identity provider that’s leveraged by one or more relying parties. And now I’ve introduced some new terminology. I suppose I need to define it. So basically, an identity provider performs the authentication process and holds the database of identities that will use a service. So like Google has an identity provider. If you want to sign in with Google, facebook can act as an identity provider, though don’t do that. Other examples are Azure Active Directory or Traditional Active Directory. All of those are identity providers. The relying party trusts the identity provider to do the authentication. And when a client wants to authenticate to the relying party or access the relying party, the relying party redirects it to an IDP to perform that authentication. And assuming the authentication goes well, the relying party will grant access to resources based on the authentication status and identity of the client.

[00:13:48.410] Ned: So we got our identity provider, we’ve got our relying party, and then we’ve got the client. And there’s this whole dance happening here. And there’s really good graphics that demonstrate this whole thing that you can’t see right now.

[00:14:01.070] Chris: They look amazing. I did not know you had this much skill in animation.

[00:14:05.630] Ned: I definitely didn’t use AI anyway. So when you log into Asca with your Google account. The Asca server redirects your Web client to get an authentication token from Google first. Once you get that token, you present it to the Aska server, which verifies its authenticity with Google and then issues you a new token that grants you access to see all those sexy, sexy silver spoons from the 1880s. Hubba hubba. Oh, yeah.

[00:14:38.250] Chris: First of all, the real collector’s items are Pewter, but.

[00:14:43.690] Ned: Oh, they’re much more bendable, which makes them more valuable. It’s weird, and I think you can get lead poisoning from them, but it’s fine, it’s fine. As you can imagine, the private keys used by Google to sign the token are, like, really important. Really important. Like, it would be super bad if someone stole one of those private keys and started signing tokens to gain access to your sexy spoon pictures or, I don’t know, like, national security documents on Outlook. So Microsoft loses the keys to the.

[00:15:21.060] Chris: Outlook castle in case anyone’s reading ahead, I think you know where we’re going with oh.

[00:15:29.220] Ned: So, getting back to our little timeline, as you may have already surmised, and Chris has certainly figured out, sometime before May 15, 2023, the Chinese based hacking group that Microsoft dubbed Storm 58 somehow stole one of the seven or eight Microsoft account, aka MSA keys, used by Microsoft Azure Active Directory to sign tokens. Now, how they were actually able to steal the key is still completely unclear, and Microsoft has been less than forthcoming with those details, by which I mean they’ve said nothing super cool. Between May 15 and June 16, the hacking group went on to forge tokens and gain access to at least 25 customers of Outlook microsoft 365, including high value targets like governmental accounts, including US. Commerce secretary Gina Raimondo.

[00:16:33.000] Chris: Sorry, Gina, she’s got all the good stuff.

[00:16:35.380] Ned: I bet she does. All that commerce. It’s happening with her.

[00:16:38.850] Chris: You should see her spoon collection is all I’m saying.

[00:16:41.480] Ned: Now, I’ll point out that Microsoft looked back through their logs and were able to pinpoint that May 16 as the first time they saw that activity. But that doesn’t mean that these adversaries weren’t successful in previous attacks. It’s just that’s the first one they found.

[00:16:59.610] Chris: Right.

[00:17:01.750] Ned: In a blog post from Microsoft published around July 11, they claimed that only Outlook Web Access and Outlook.com were affected by the leak key and advised customers to review their logs for specific source IP addresses, token, hashes, and suspicious activity. That proved to be more difficult than you would think. More on that in a moment. Sheer Tamari, a researcher over at the infosec company Wiz, published an investigative blog on July 21 claiming that the leaked key could be used for much more than just Outlook Web Access and Outlook.com. So he did this thing where he looked through Microsoft’s documentation. Crazy, I know. Specifically, their documentation around the OpenID token verification process. Version 20 for Azure Active Directory. It’s a blistering read. I highly recommend it or not. It’s actually very boring and dry. But in there he determined that there are eight public keys that are used to verify signed tokens for personal accounts and seven public keys that are used for multi tenant applications. The stolen key in question appears to be in both of these lists, meaning that forged requests could potentially access any Azure Active Directory multi tenant applications or applications that use the login with Microsoft functionality.

[00:18:35.670] Chris: So it’s just OA is what you’re saying.

[00:18:39.670] Ned: Yeah, about that. The list of applications that falls under that umbrella is a bit larger than OA andoutlook.com including your favorite hits like SharePoint Teams OneDrive, Skype, Xbox, and any other application that uses the login with Microsoft functionality and version 20 of their OIDC protocol. That’s a lot of applications.

[00:19:10.520] Chris: I would go so far as to say that’s probably all of them.

[00:19:16.050] Ned: There are a few application types that would not be impacted by this. If you have a single tenant application that would not be impacted by this, and if your application doesn’t use the login with Microsoft functionality at all, I believe that’s also not impacted. But anything else that is multi tenant or multi login type would be impacted by this. Now, Microsoft recommends in the documentation that when developers are writing their app, they verify the organization ID in the issuer claim along with the signing key when they’re validating a token. However, that’s not a hard requirement and a busy or inexperienced programmer could have easily missed this piece of advice because it’s an extension specific to just Microsoft. So if you’re a developer who’s used OIDC in the past and you didn’t read the Microsoft documentation real closely, you might have missed it. The other important thing is that it’s not uncommon for applications to cache the public keys locally so they don’t have to reach out to Microsoft every time they need to validate a token. And that’s because the keys don’t change very often the key that they revoked, the one that was stolen, that was issued in 2016.

[00:20:40.350] Ned: So reaching out to Microsoft every time you need to validate a token is kind of a waste. So even though Microsoft did revoke the token, revoke the key, I’m sorry, in question, there’s a really good chance that it lives on in some systems that just hold a local cache that is very infrequently refreshed.

[00:21:02.210] Chris: And that’s one of those things that if you don’t know it’s there, you probably don’t know that you need to do anything about it.

[00:21:09.350] Ned: Exactly. So hopefully if you’re listening to this and you know that you’re using some Azure ad applications, might be time to comb through that code and see exactly where you’re caching those keys.

[00:21:21.210] Chris: I would go so far as to say that if you don’t know, you.

[00:21:23.950] Ned: Should just assume there’s a good chance you’re using. Yeah. Oh. So a Microsoft spokesperson contacted by the Register said that, quote, many of the claims made in the Wiz blog are speculative and not evidence based. We recommend that customers review our blogs, specifically our Microsoft Threat Intelligence blog, to learn more about this incident, end quote. You’ll note they didn’t actually refute any of the claims made in the blog, like any of them. They didn’t even use the word any or all. They said that many of the claims are speculative and without evidence and many is not the same as all of them. And the reason they’re speculative is because the keys in question have since been revoked. So it’s pretty hard to test the theories that Wiz came up with. Now, as I mentioned, Microsoft is advising folks to check the logs of their cloud applications for possible signs of breach. The thing is, collecting logs that include all the relevant information and retaining them requires a Microsoft Purview Audit Premium license, which is not cheap. You need an e three or e five license, I believe, to add that functionality. And you can guess that not everybody has one of those, right?

[00:22:52.950] Ned: So after coming under a little bit of scrutiny and pressure, microsoft expanded cloud logging access to essentially everyone with an Azure ad tenant, which they probably should.

[00:23:05.710] Chris: Have done in the first place, but.

[00:23:07.660] Ned: Here we are, probably should have included as just having an Azure ad tenant to begin with. But we’ll get as you can imagine, the US. Government is not super pleased about Microsoft’s, whoopsie doodle. So much so that Senator Ron Wyden of Oregon wrote an open letter to the director of CISA, jen Easterly, the Attorney General, Merrick Garland, and the FTC Chair Lena Khan lambasting Microsoft for their negligence in securing these incredibly important signing keys. Wyden rightly points out that Microsoft makes billions of dollars selling security products to protect their own services. And I don’t think it’s a stretch to say that Microsoft should be offering at least some of their security services for free to all customers and after this mess, a serious discount for all their premium products as well.

[00:24:10.110] Chris: Yeah, I think that’s fair.

[00:24:12.370] Ned: It’s kind of like how Windows Eleven comes with endpoint protection, whatever they’re calling it today, windows Defender. That’s just part of the product because Microsoft knows that their product is insecure and so it needs this bolt on. That should be an approach for everything in Azure as well.

[00:24:30.490] Chris: Well, especially something as basic as this type of logging and logging analysis. Because realistically you need those logs for analysis because there is so much going on and it’s so disparate across your entire environment that you can’t just look at it and go, oh, there’s something wrong with Server Six.

[00:24:49.400] Ned: Right?

[00:24:50.410] Chris: Like you couldn’t do that when you were actually in a data center, but you definitely can’t do that when you’re in the cloud.

[00:24:55.870] Ned: Yes, and Microsoft will point at their shared responsibility model and like but that’s not in our portion of the shared responsibility. That’s your responsibility. Which I mean is true, but you’re also being a dick about it. The other thing that Wyden points out is that MSA Private Keys should have been held in a hardware security module that’s known as an HSM. Those are designed to prevent exfiltration of the Private Key. In fact, if you tamper with an HSM, it destroys the Private Key. And Microsoft recommends using an HSM for Private Keys and they even offer one as a service as part of Azure Key Vault. So why weren’t these incredibly important keys secured on an HSM? Inquiring minds want to know.

[00:25:55.170] Chris: I’m going to guess that they were put in place before HSMs were common. And like you said, 2016.

[00:26:06.630] Ned: That doesn’t jive with me. Only because I was working on a healthcare application in 2014 that was required to use HSMs in AWS.

[00:26:19.690] Chris: I’m not saying that they didn’t exist, I’m just saying increased popularity over time.

[00:26:24.410] Ned: Right, but it’s not like they were uncommon.

[00:26:26.430] Chris: Right, they were not uncommon.

[00:26:29.880] Ned: Sure. The other larger question is how did Microsoft get all of these certifications of compliance with standards like FedRAMP and the ISO 20,000 ones, while lacking this basic level of protection for these Private Keys? There’s a lot of questions here and basically no good answers coming from Microsoft.

[00:26:56.070] Chris: TLDR.

[00:26:58.070] Ned: Microsoft failed. They failed hard and worse, they’re being like real ninnies about it. I hate to use harsh language, but ninnies is just when you fuck up, Chris, it’s best to fess up and not compound the lie. I mean, have we learned nothing from 40 years of sitcoms and after school specials? I mean, Webster taught me that I need to own up to my mistakes and not make it worse. Just follow Webster.

[00:27:31.560] Chris: Man, I got that lesson from Silver Spoons. See what I did there? That was a spoon callback.

[00:27:37.770] Ned: Oh, well done. I’m ashamed that I didn’t think of it. That’s the way we get by it’s. Just because I love Emmanuel Lewis.

[00:27:45.980] Chris: That’s another spoon reference that you missed.

[00:27:49.830] Ned: I’m just dropping the ball left and right. So the very angry letter from Senator Wyden is likely to spark a full blown investigation by some sector of the government. He explicitly calls out to each of the people on the letter to use their investigatory powers against Microsoft in whatever way they can. We’ll see what happens and what’s revealed. But regardless, I think this is a serious blow to Microsoft’s reputation in the security realm. And it didn’t exactly have a sterling one to begin with. Not even pewter.

[00:28:26.770] Chris: Right? Oh, you got one?

[00:28:29.080] Ned: I got one. I got one in. Yeah.

[00:28:30.610] Chris: I feel good about mean there’s so there’s two ways to think about this and when it comes to infrastructure, it’s always the yin and the yang. With this particular point running in the cloud, it seems like you’re using Microsoft to protect Microsoft, right?

[00:28:48.140] Ned: Yeah.

[00:28:49.370] Chris: If you were using a third party product like, okta, you wouldn’t be using the login with Microsoft button and none of this would be an issue.

[00:28:58.750] Ned: True.

[00:28:59.280] Chris: So do you go with the single pane of glass, everything under one roof kind of Microsoft, Microsoft, Microsoft argument, which has value and merit and can be less expensive, can be easier to manage. But if one piece of that starts to waver, shall we say the entire portfolio is at risk, whereas if you go third party, you’re spreading the risk out. So if there is an issue, like now there’s a Microsoft issue if you’re using Okta, it’s not a concern.

[00:29:34.790] Ned: The thing I come back to is when you have these massive companies that have such a large swath of customers, when they mess up, the blast radius is just so much bigger. Like when AWS S Three goes down or they mess up their DNS.

[00:29:52.970] Chris: First of all, AWS S three never goes down, sir.

[00:29:58.190] Ned: Anyway, the blast from that is huge. But also you can say, well, nobody ever got fired from using S Three. But I think you’re right to point out that it’s probably better to go with a company that specializes in a particular area. I mean, Microsoft can make the claim that using all these things together because they’re all part of the same company means that they’ll work better together. But A, I found that to not be the case across many companies that have made that. B, I think, yeah, you’re putting all your eggs in one basket and I’d rather have my risk spread out across multiple companies, especially ones that are focused on one particular area.

[00:30:41.870] Chris: Right. And I mean, like I said, the argument can very easily be made in both directions. The problem is, when you have an incident like this, it really weakens one side of that argument.

[00:30:52.760] Ned: It really does. So I guess buy octa stock is what we’re saying. I don’t oh well, hey, thanks for listening or something. I guess you found it worthwhile enough if you made it all the way to the end. So congratulations to you, friend. You accomplished something today. Now you can spend the rest of your day combing through Azure Ad logs oh, I’m sorry, entraid logs looking for possible breaches. You’ve earned it. That does it for today’s main episode, but stay tuned for Thursday. We will have our Tech News of the Week episode airing, then you can find me or Chris on Twitter. I refuse to call it X at Ned 1313 and at Hayner respectively. Or follow the show at Chaos underscore Lever if that’s the kind of thing you’re into. Maybe we should just get rid of that section. I don’t know. What do you think? Should we just say LinkedIn instead?

[00:31:44.750] Chris: Oh, I wasn’t listening. What are you talking about?

[00:31:46.660] Ned: That’s fair. Show notes are available@chaoslever.com if you like reading things, which you shouldn’t. Podcasts are better in every conceivable way. We’ll be back next week to see what fresh hell is upon us. Tata for now.

[00:32:00.370] Chris: I think the funniest thing dude, every single show, podcast, YouTube noob, that I look at, everybody says the same exact thing. Yeah, we’re just going to keep calling it Twitter.

[00:32:11.930] Ned: Why would somebody was like, well, we no longer tweet, we excrete. And I’m like but no.