Leo the Angry Penguin: A Malware Cautionary Tale [55]

Posted on Tuesday, Apr 25, 2023
Chris tells a tale of malware misery and how defense in depth could help, Ned is amazed by what people leave on their used devices, and we’re all going to Sizzler!


00:00.00 Chris: Any better is that any better or no okay, you didn’t say that.

00:00.48 Ned: You’re weird or whatever. It sounds fine to me. Yeah, it’s going to normalize some of this when I put it through the production process. So As long as you’re audible enough and your signals Clean. It’ll be Fine. We’ll boost things up. Yeah, and you know hopefully this time I can.

00:21.65 Chris: Well good.

00:28.52 Ned: Key out the green without also keying out your eyes though. The.

00:32.76 Chris: Be pretty funny if my eyes turn green this time just you know the technology clearly exists and it only exists to annoy you.

00:42.17 Ned: That is how I feel about most technologies if we’re being honest, this is not a particularly good week for me in technology. There were a lot of fights I mostly lost them.

00:53.88 Chris: I Was that was gonna be my next question but I didn’t really feel like the question needed to be asked.

01:00.14 Ned: Well I did win 1 kind of battle as something locked up with my router on Sunday morning and suddenly the internet was gone as far as the home was concerned and so.

01:07.95 Chris: Yeah, yeah.

01:13.82 Chris: Is that bad.

01:17.36 Ned: It’s bad when the kids are like watching their various shows on streaming and the locally cached information runs out.

01:26.71 Chris: Yeah, so then the streaming turns into screaming.

01:30.19 Ned: Yeah, so a quick reset of the router fix the problem but then 1 Tv was being super weird. It would connect to the wireless but it would only get an ip address some of the time and then the communication wasn’t real solid. And what I eventually figured out was whatever happened with the router caused one of my access points to sort of flake out and so my Tv was trying to talk to the much further away access point and because the signal wasn’t particularly strong. It was exhibiting really strange behavior so I had to fix the access point which again a simple reset was all needed. How about that I turned it off and I turned it on again. God damn it.

02:15.72 Chris: And.

02:24.32 Chris: Friend of mine had to get ah a refrigerator fixed and not a smart refrigerator mind you just the old fashion kind the one that you you know it’s a sealed box that keeps food cold. The main problem being that at this point in time.

02:32.54 Ned: Um, yeah, we are though. Yes.

02:40.28 Chris: Said Box was not keeping said food cold.

02:41.84 Ned: That is a serious problem.

02:45.26 Chris: So a trouble ticket was opened or whatever they call them in the normal world and I swear to God This is true. Do you know what? they told him.

02:55.35 Ned: To unplug the refrigerator and plug it back in and did it work.

02:59.45 Chris: Yeah, turn it off and turn it back on again. Now No, but you know it was funnier if you didn’t have to ask the idiotic followup question and we could have just left it. We could have just let it hang. People could sit there in silent wonderment and curiosity. But no.

03:06.14 Ned: Ah, but did it work. Ok.

03:15.27 Ned: What more it’s It’s a valid question. No our every single person listening to the podcast would be screaming Well did it work and I just I have to be the voice of the audience in that case.

03:28.51 Chris: So.

03:33.16 Ned: And and speak what they cannot to you in real time. That’s true. He he’s a whisperer.

03:34.12 Chris: Listen First of all you and I both know that Dave is not a screamer.

03:43.36 Chris: Um, it would have been more like a a polite letter emailed I mean Mail Mailed Snail mailed probably calligraphy. Maybe a small hand done sketch an emotive one. You know.

03:49.39 Ned: Snail Mailed he does have that ink pot. Yes, he’s good about that.

04:01.20 Chris: Describing the feelings and.

04:06.82 Ned: Well I look forward to receiving that letter. But I guess I won’t now because I asked the question so that’s on me hello alleged human and welcome to the Chaos Lebra Podcast My name is ned and I’m definitely not a robot.

04:10.21 Chris: And I hope you’re happy. Okay.

04:24.30 Ned: I Too gaze up at the spring night sky bewildered by the beauty of our lunar satellite and feel myself filled with an eerie calm and dare I say hope for tomorrow. It’s not because my cpu is pigged making me prone to manic hallucinations.

04:36.39 Chris: This.

04:43.36 Ned: Have heard such a thing with me as Chris and a stuffed penguin named Leo hanging in front of my field of vision radiating peace and serenity hello Leo hello Chris: ello system fault.

04:56.31 Chris: I think we should just let Leo do the show.

04:59.84 Ned: He is exuding a strange confidence along with all the peace and serenity. So I do appreciate that.

05:06.83 Chris: Um, and also he’s waving a gun around which makes me feel like we should listen real close.

05:12.44 Ned: Ah, ah, you’re not wrong. He is. He’s prone to go ah to fly off the handle are off the ice floe as they say.

05:21.49 Chris: Because Penguins can’t can’t fly off of handles or or otherwise yeah.

05:26.90 Ned: No no, he swos off and swims off the ice moe. Okay, oh good I’m glad we we landed on an episode title so early on in the recording. That’s that’s skin that’s going to pay dividends later. Um.

05:43.41 Chris: I did not I wrote a short story with sort of a happy ending.

05:43.43 Ned: So you wrote a freaking novel. We okay maybe a novella can we can we agree on that.

05:53.84 Chris: I’ll accept although I will tell you that as I was writing this I had to cut it in half this is the easy half if I were to have continued I would have gotten into the more difficult and technical half.

06:05.90 Ned: I see and since you don’t really understand any of that. It’s probably best that you don’t put it in for posterity.

06:13.72 Chris: What’s a computer him? No, but but what’s a computer.

06:18.40 Ned: Ah, but oh oh, you’re ask. It’s a person who adds numbers and subtracts them says you.

06:28.37 Chris: This is not the year nineteen thirty

06:35.17 Ned: All right? Let’s talk about the thing that you wrote a short novella about okay.

06:37.63 Chris: Ok I want to tell a true. Well true enough story of 1 company’s adventure in almost going bankrupt because of malware. So. There are 2 competing initiatives that business leaders have to grapple with when it comes to it security and stop me if you’ve heard this one before the first one is getting hacked I’m ignoring you completely for the next twenty minutes the first

07:03.37 Ned: Up. Excellent.

07:12.81 Chris: 1 is getting hacked would be bad and we don’t want that to happen and the second is this security stuff sounds expensive and we don’t want to pay for it and this can be especially true for things that.

07:22.30 Ned: M.

07:29.27 Chris: On their face seem redundant too often to business leaders the whole concept of defense in depth can be deemed wasteful if that leadership thinks that the outside security is good enough.

07:44.40 Ned: Yeah, thus we have the the hard candy shell with the soft gui interior that is all of your user data.

07:52.87 Chris: Exactly and for the past infinity years. That’s exactly what we did that was where you had your physical walls and you had your virtual walls. The physical walls are made of concrete. The virtual walls were made of firewalls.

08:13.80 Ned: Were those also made of concrete ah out.

08:15.80 Chris: Um, only the ones from Arista zing. Um anyway, security strategy and architecture has since moved on.

08:30.98 Ned: Um.

08:31.21 Chris: Now we’ve talked this concept to death on this show and in our defense we are not alone everything I just said can easily and snottily be prefixed with well everybody knows that fill in the blank something something micro segmentation.

08:46.62 Ned: Right.

08:51.40 Chris: Something something zero trust that’ll be $50000

08:54.66 Ned: Yeah, but that can be a little condescending.

08:59.53 Chris: Yeah, that’s 1 word for it and it occurs to me that one of the reasons that this might not be a helpful way of going about it aside from the condescension aspect is people might not understand why that.

09:10.99 Ned: Me.

09:17.81 Chris: Ah, multifaceted approach to security is essential. So I says to myself I says why not tell a real story from history of a company that got hit by some malware and the consequences of that attack and how defense in depth. Strategies would have saved their bacon. Okay.

09:38.37 Ned: Yeah, we certainly hear these stories in a more general sense. But it’s very seldom from someone who was personally involved in the process and saw it firsthand and I think that hearing that story is going to bring really bring it home in a way that the more general. Marketing that you get from all the security vendors does not.

09:57.91 Chris: Right? And one of the things that I hope will come across when we talk about this is that there is no such thing as 1 single magic pill. You cannot buy 1 product to have all of your problem solved. Voting me president will not make all your dreams come true. For example, so disclaimers one the story I’m about to describe is a real thing that more or less happened I was definitely involved and that is as far as I’m going to take that.

10:15.60 Ned: Or any of them.

10:31.76 Chris: Have generalized the crap out of this story to eliminate all specifically identifying details. No I will not tell you who it was ned for God’s sake stop asking and finally unfortunately this story is not a unique one.

10:41.61 Ned: Oh.

10:49.74 Chris: I guarantee that a close variation of this has happened tens of thousands of times all over the world and that’s just this week. So just because you think you know who I’m talking about doesn’t mean you know who I’m talking about but you know what I’m talking about you know? so.

10:58.50 Ned: Yep.

11:08.84 Chris: What I’m talking about what am I talking about.

11:09.50 Ned: I’ll just let’s just begin of only Leo knows.

11:15.43 Chris: Um, okay so June twenty third two thousand and nineteen a 38 year old Caucasian Male advertising executive named Francis Nunnerie from balt waitde a minute hoop. But so let’s.

11:24.20 Ned: A whoa that’s weird. Yes I do that all the time. Okay.

11:32.20 Chris: Well, we’ll cut that in post right. Sweet so let’s tell a story about a person in a company and the person we’re going to focus on. We’ll call him Steve it’s actually Steve Steve

11:46.78 Ned: Steve Dave oh okay like Mario Mario okay

11:54.20 Chris: Exactly nice, well done Steve worked for a company in their IT department Steve got a phishing email that purported to be from a supplier asking a question about a recent order said email included a link.

12:13.50 Ned: Um I think I know all Steve.

12:13.34 Chris: Like guess what happened next our man Steve went ahead and clicked the link his web browser did some weird things sat and spun for a moment his computer got a little hinky and then an error came up. Telling him that there was a problem and to try again later Steve wrote a mental note for himself to try again later and just moved on with his life like you do needless to say that was unwise and that link was in fact.

12:34.51 Ned: Um.

12:41.62 Ned: Um, like you do.

12:50.37 Chris: Connected to malware now Steve’s computer was infected with malware.

12:50.42 Ned: Oh.

12:57.85 Ned: Which is the beginning but not devastating yet.

13:01.90 Chris: Correct. But before we go further and then I want to do this a couple of times as we go through just some of the thoughts in terms of things that count as security in depth that would have helped ameliorate and or eliminate this from happening number 1 easy 1

13:16.11 Ned: Oh.

13:21.36 Chris: User education if Steve had been given even basic regular security training. He would have been less inclined to click on that link or he would have investigated the email further or he would have had it just something in the back of his head rang a bell and he just deleted it.

13:38.24 Ned: Right.

13:41.50 Chris: Now it is true that overdosing on user training is not helpful. In fact, that is counterproductive that has been shown but it has also been shown that giving 0 training gives 0 protection.

13:44.70 Ned: M.

13:57.50 Ned: I.

14:00.23 Chris: Number 2 a link scanning service of some kind the link that Steve clicked on in retrospect was an obvious bad address. It was not even one of those clever things where you think you’re going to write aid and you’re going to write. A 1 d um, this was an overseas. Yeah Url which did not make any sense based on the vendor that the email was purporting to be from local businesses generally don’t order from company name redacted.

14:20.92 Ned: Yep yep.

14:35.17 Ned: But probably from Russia.

14:37.99 Chris: Point is Steve missed it. So the other point is every modern major email provider offers some kind of basic link scanning service had 1 been employed the email.

14:40.75 Ned: Yeah.

14:55.88 Chris: And that link would have been flagged as malicious and put into some type of quarantine.

14:59.78 Ned: In fact, he might never have seen that email.

15:02.45 Chris: Correct and I don’t want to go down the the path of the different products that exist but any of them would have helped the situation as opposed to what happened so whatever he didn’t know not to click on it. He ended up clicking on the bad link number 3

15:09.44 Ned: Right? right.

15:21.26 Chris: Dns security. Let’s say you didn’t have link scanning in email. Okay, fine. There could have been dns-based security at the router in this case, this is an easy one like I said the link went overseas.

15:27.83 Ned: Ah.

15:36.23 Ned: Right.

15:38.65 Chris: There’s no way it makes sense for Steve’s computer to be going to country name redacted from a work pc none this is something that can be easily prevented and this is something that can be prevented for free. Using Router rules if that had been the case and Steve had clicked on the link. It would have thrown a 4 4 page not found or some other kind of routing-based error and again no malware makes it to the computer.

15:57.27 Ned: Yep.

16:10.49 Ned: Right.

16:14.72 Chris: And finally for this initial area of um, what do we call it unfortunateness user accounts that had too much permission to do things.

16:20.54 Ned: Right.

16:32.67 Chris: Here’s one that users hate but is quickly becoming a security necessity users such as Steve should not be allowed to install anything period no executables that are not on the approved whitelist. No scripts no macros.

16:50.45 Ned: And.

16:51.34 Chris: Nothing now in this case, there was no device security so that was the problem in and of itself. But Steve was an it person. So even if there was. There is totally the possibility that Steve could have granted himself exceptions to this rule because he knew better does that sound like anyone you’ve ever met in your life.

17:14.66 Ned: It does it actually sounds like 2 distinct categories of people that I found at companies now. The first one is kind of obvious because you’ve already mentioned it. It’s I t admins and they think they know better and that they can give themselves special permissions on their machines I am just as guilty as. Any other I t person of doing this. Do you know who the other group of individuals are not yet, but maybe in the future corporate executives specifically anyone in the c-suite. Yes.

17:32.33 Chris: I Agree it is you your children.

17:46.83 Chris: VI p’s as it were.

17:51.29 Ned: They often demand having any kind of privileges they want on their computer and they want access to everything in the organization which makes them ah doubly dangerous.

18:01.33 Chris: Right? for similar reasons now and to be fair in both cases I’ve seen this trend on the downward slope but it definitely still happens and it really really shouldn’t because one of the things about security of any kind defense in depth or otherwise consistency matters.

18:12.58 Ned: Um.

18:21.18 Chris: Whatever the rules are is what the rules are.

18:22.87 Ned: Right now try telling that to the person who writes your paycheck.

18:29.20 Chris: So that’s step 1 Let’s move on through the timeline to the next phase or the next episode. Nothing fair enough anyway.

18:43.87 Ned: Nope Nope I’m not going there.

18:48.62 Chris: Steve in fact, forgot about the email and the link he clicked on remember that mental note yeah he didn’t more like a nope am I right? So the malware that he.

18:54.19 Ned: Um, yeah, maybe legitimated a note note. Ah, ah no.

19:06.91 Chris: Unintentionally installed on his computer started probing the local ah computer and his account for information now we have to do a little speculation here. It is likely that this probing. Started to happen after hours so that the activity wouldn’t be noticed by Steve working on his computer seeing weird things flash on the screen terrible performance etc. We don’t know this for sure because spoiler alert.

19:29.25 Ned: Ah.

19:32.40 Chris: 1 of the things the malware did was delete every single log file. It could find anywhere at every step of the process.

19:39.26 Ned: Um, yeah, that sounds like malware.

19:40.96 Chris: 1 thing we know for sure is that it found Steve’s passwords which were stored in his browser. These were easily accessed by the malware because passwords in the browser are protected by Ned and.

19:56.11 Ned: Um, oh that’s not good.

19:58.19 Chris: Nothing the malware was logged in as Steve’s legitimate account so that was good enough for Chrome.

20:10.48 Ned: Um, so yeah, maybe using a password manager is a good idea that’s not Chrome how about I didn’t have to.

20:18.26 Chris: Don’t um, don’t read ahead. How about that so some additional security and security and depth concepts that would have helped here number 1 and probably 1 of the more annoying ones.

20:23.70 Ned: Yeah.

20:33.87 Ned: Ah.

20:34.18 Chris: Whatever endpoint protection they had in place did not include behavioral mapping modern endpoint protection tools do more than just check executables for virus fingerprints. The concepts of Edr and xdr exist specifically to watch systems and user accounts for. Not virus fingerprints but virus-like behavior now these can get annoying expensive and complicated especially in the setup phase but they exist precisely for these kinds of situations you have system.

20:54.80 Ned: Right.

21:04.42 Ned: Um, and um.

21:10.19 Chris: Observation software that goes hey so um, like Steve doesn’t usually run deep end map scans of our internal networks from his work pc at 2 in the morning. Do we think that might be a problem.

21:21.87 Ned: But yeah.

21:24.37 Chris: That is not the kind of question that Norton Antivirus is ever going to ask so.

21:29.56 Ned: Now you need to upgrade to something ah a little more robust.

21:31.61 Chris: Yes, number 2 your logs should be exported to a read only location now this is one that again can be done for pay or for free should you have the expertise or. You know people that can do it But in either case, it is very rarely done Logs for everything should be sent off system for recording and analysis the system they are sent to should be read only meaning that they get to write the log one time and that’s it. It is now Forever immutable This way had this been done. The best thing an attacker could do is stop the logs from being sent but the system would have logs of that happening right up until the command was entered.

22:10.22 Ned: Right.

22:25.68 Ned: Right.

22:28.85 Chris: So ideally 2 a similarly those logs that are stored are monitored and analyzed and alerts are sent if say for example, logs all the sudden stop being sent at all that could be construed as a problem.

22:38.84 Ned: And.

22:45.20 Chris: Now you could get way into the weeds with this with like scanning for the logs to look for specific commands or behaviors. But like I said I don’t want to get that far down the product line. The point is logs that are deleted are not logs that you can use in retrospect to try to reconfigure or. Ah, reconstruct everything that happened and like you said Password managers people the point right here is why having them in the browser isn’t good enough storing passwords in the browser has to be considered.

23:03.90 Ned: Right.

23:15.94 Ned: And.

23:20.72 Chris: Ah, convenience and it is better than nothing but using an external password manager one that is ideally protected by Mfa is security.

23:32.24 Ned: Yeah, so just pick something that isn’t lastpass.

23:36.89 Chris: I Think we did a number of things about that. So I’m not going to So I’m not going to dwell.

23:41.45 Ned: Yeah, it’s probably not worth harping on that one any more than we already have.

23:46.86 Chris: So a lot of what we talked about to this point is either user or user workstation focused which if you look at the numbers is really it makes sense the numbers show that as of 2022

24:00.48 Ned: Um.

24:03.64 Chris: Something like 90% of breaches are caused by some kind of human error most of which is in the categories of lost credentials and phishing sound familiar to anything I’ve been talking about thus far now I don’t want to understate.

24:17.35 Ned: Yeah, ah, only slightly? yeah.

24:21.82 Chris: Technology and the importance of keeping your systems patched Csv vulnerabilities are obviously bad because that’s the how but the thing that opens the door all too often is human error in many cases including this one.

24:32.23 Ned: Nothing.

24:37.66 Chris: The exploit of the vulnerabilities is only possible because an employee had an oopsie.

24:42.19 Ned: Write.

24:44.68 Chris: Now let’s move on in the timeline to where the real fun begins. So what do we have? We have a attacker who has complete control over Steve’s computer

24:50.91 Ned: Oh this wasn’t the real fun yet.

25:03.30 Chris: Steve an it person has helpfully given out usernames and passwords to any number of systems and the attacker knows where those systems are thanks to that password list. They can demolish the onsite systems.

25:05.28 Ned: Um, then.

25:08.48 Ned: Right.

25:21.10 Chris: But they’ve also got another target Steve’s company’s Aws environment as the malware crawled the local network. The attacker manually went after the cloud so this is something that we have seen a lot of the attacks on.

25:27.75 Ned: Ooh. Okay.

25:40.69 Chris: Prem are so historically standardized and prevalent and script- kiddified that you don’t need a human to type commands. You can just run any number of scripts and let it happen whereas things online in the Cloud are a little bit more custom and a little bit harder to.

25:45.60 Ned: Right.

26:00.27 Chris: Scripts not least because Aws changes how their knobs and dials work every 15 seconds

26:06.25 Ned: It’s true though that is quickly becoming less an issue I mean the the script Kitty side of it. Not Aws changing things they they change things. Ah but let me let me make a guess here because I’ve worked with other I T people especially in I’m guessing this wasn’t the largest shop.

26:14.47 Chris: Um, right.

26:24.15 Ned: We won’t say how big or small but in my experience, there’s always too many people who are domain admins in a given organization and all of those people use their domain admin credentials. Locally to do things.

26:44.83 Chris: Unfortunately, you couldn’t be more right.

26:45.69 Ned: So not only does the Attacker have the full list of systems. They have domain admin credentials that they can scrape from his local credential manager on his desktop.

27:01.65 Chris: Um, not could there. You have it. That’s the short short version of what happened. The only thing that you missed was the local file servers Also easy targets.

27:02.49 Ned: And there goes your entire 80 environment. Hooray.

27:15.20 Ned: Well yeah, they’re part of a D that this as soon as you have domain admin credentials as long as anything’s joined to the domain you have full control over it.

27:18.70 Chris: Exactly yeah.

27:23.80 Chris: Correct now that was part one and that’s bad part 2 is what happened in the cloud and that was kind of interesting and i’m.

27:30.17 Ned: Not great.

27:38.36 Chris: At the end of this I’m going to I’m going to explain what happened here and generally what happened is security completely by accident. So the environment in the Cloud was pretty simplistic in its setup. They were not very far down the Cloud Journey um the account.

27:44.56 Ned: Fair enough.

27:57.99 Chris: That Steve had in his I’m going to use password manager in gigantic air quotes had access but he was only able to do things like list and read he couldn’t destroy anything. He couldn’t create anything. Nothing was able to be deleted but the data was able to be.

28:01.64 Ned: Right.

28:16.20 Ned: Moving.

28:16.90 Chris: Download it which is still fine for any number of companies. The Attacker tried to go after all the standard things creating new shadow accounts going after and deleting versioning all of which failed. Ah, the Attacker also tried to delete Logs in Cloud Trailil which also failed. Yeah so they were not able to delete anything but they did get the data and they did send a ransom that said basically if you want this data to stay private pay us X Y z.

28:35.38 Ned: Yeah, it’s pretty hard to do That’s kind of the point.

28:53.00 Chris: All the usual things that came from that so security in depth concepts that would have helped in this part and this is one that I probably should have said at every single stage but have you heard me say the phrase Mfa yet.

29:05.67 Ned: I Feel like it came up once but you could say it again.

29:09.18 Chris: Yeah I was talking about that degree I got before I got serious about technology sculptures making a comeback man number two East West traffic was permitted with.

29:16.67 Ned: Ah I’m not sure if you’re joking or not.

29:27.82 Chris: Any without any re-authentication requirements now this is sadly far too common. This is also a bedrock concept of 0 trust Steve’s workstation was the 1 logging into the on-prem servers. So it was trusted implicitly as you can tell.

29:43.50 Ned: M.

29:47.74 Chris: This was a mistake and again we could go into micro segmentation and all the details of how to do this type of thing. But the point is there should have been some kind of a technological question asked every single time a login happened and that simply didn’t occur.

30:00.43 Ned: Right.

30:06.80 Chris: So backups On-prem were not immutable when I say that the On-prem environment was a disaster I mean it everything that they could find the attackers could find was exfiltrated and then destroyed and I mean thrashed.

30:23.91 Ned: Oh.

30:24.86 Chris: On site This company legitimately could have gone out of business because of this data loss because as you know data is the lifeblood of any modern organization Now the company didn’t.

30:35.51 Ned: Yes, very much. So.

30:39.82 Chris: Go out of business for 1 hilarious reason that I don’t want to talk about on Mike but it’s just a fair reminder. Why backups must be considered part of the security process too right now if you go through.

30:52.16 Ned: Write and you.

30:56.92 Chris: Defense in depth. Obviously we’re starting at the user level going down the the timeline getting all the way to the center of this company which is the data if we’re talking about having to protect backups things have gotten real bad. But again, this is why so many different layers are necessary and important to have.

31:13.21 Ned: Right.

31:15.26 Chris: And it’s also why taking care of backups as though none of those other things existed is important too. Yeah.

31:20.21 Ned: Right? because eventually especially in a ransomware type attack you’re going to get to the backups at some point and it’s probably good if they aren’t also erased or encrypted by the ransomware.

31:34.10 Chris: Right? So finally storing logs in aws was great and the fact that Steve’s accounts didn’t have any access to delete them was awesome now I’ll tell you in just a second why that was the case but it’s important to note. Another thing that we can and should do in defense in depth is monitor and alert based on those logs remember cloud trail is expensive sure but it can also see absolutely everything that happens within Aws.

32:01.69 Ned: Um.

32:11.66 Chris: Probably not an equivalent product that can do that in that environment. Why not if you’re using it take advantage of that to its fullest, 1 simple way would be to set up alerts for when data type events happen things like downloading an entire s 3 bucket for example or.

32:21.75 Ned: Um.

32:31.64 Chris: Attempts to to disable s 3 versioning very simple to set up. You could very quickly turn them into 100 decibel claxon level situations and the last thing is.

32:41.47 Ned: Absolutely yeah.

32:46.37 Chris: The separation of duties and accounts using roles and not assigning permanent permissions which this customer did by accident because you know what happened remember I said that this customer was very early in their Cloud journey.

32:57.35 Ned: I Can’t say I do.

33:04.76 Chris: They made a mistake that a lot of companies going very early in their cloud journey make which is everything was set up using the rood account Steve did not have access to the rood account. So when I say security by accident. This is what I meant.

33:24.90 Ned: Right? If he had had access to that root account the whole Aws environment would been wiped too which again you would hope they have Mfa on that rooto account I think you I think you have to now I don’t think you can possibly have a rood account at this point.

33:28.32 Chris: Exactly.

33:41.74 Ned: And not had Mfa enabled on it.

33:41.86 Chris: I write I believe that is correct and I also believe that you can set up 2 2 factor authentication which is also awesome which is you know? um.

33:49.20 Ned: Right.

33:55.66 Chris: Not hunt for red October crimson tide where the 2 guys have to go in and turn the key at the same time. It’s only ever happened in 1 movie ever ned. so so I mean like I said I’ll stop there.

33:58.56 Ned: I think that was Superman 3 but I’ll out.

34:06.66 Ned: Yeah, also in stargate like 4 times.

34:15.10 Chris: Because we’re running up to our time limit but you’d kind of I think you get the idea.

34:19.86 Ned: Yes, adding additional security at any of these stages could have prevented the attack or prevented it from expanding out from from where it was the fact that they weren’t willing to spend the money to add these defense in depth. Features is what allowed the attack to be as big as it was and I’m sure the cleanup effort was not inexpensive.

34:45.79 Chris: Right? So The next thing that happens is you have to have a conversation with yourself as a business to say what’s the value of security versus the danger or the risk of this type of a breach happening. And what kind of mathematics. Can we do to make that make sense and that gets us into risk management which could legitimately be its own episode but I wanted to do just the first most simplistic way of doing this? um.

35:08.61 Ned: Ah.

35:12.87 Ned: Might have to be.

35:22.30 Chris: From the data that I’ve been using is a fantastic report that has been around since I think 2007 which is the verizon data breach investigations report using the 2022 version which came out in I think July we know that 18% of data breaches.

35:36.79 Ned: Ah.

35:40.30 Chris: Involve a phishing attack similar to the one that I described and the average cost of a data breach is around three and a half million dollars in global aggregate. So very simple math. This means that the potential cost of any single data breach by phishing could be described.

35:43.60 Ned: Um.

35:58.69 Chris: To be worth six hundred and thirty thousand dollars three point 5000000 times 18% chance of it happening the risk valuation is six hundred and fifty thousand six hundred and thirty thousand dollars so you could make the case that spending anything less than that.

36:09.80 Ned: Ok.

36:16.62 Chris: Would be considered profitable from a risk management perspective. So if your security policy costs. Ah your security practice I should say costs six hundred and twenty Nine Thousand Nine hundred and ninety nine dollars a year. You’re okay now like I said this is a simplistic way of doing it.

36:29.40 Ned: Right. Sure.

36:35.91 Chris: But this is the short short version of how all risk is evaluated. We could go into the detailed stuff like sle and ale and any number of other risk management models. But frankly, even Leo’s calculator fell asleep.

36:49.65 Ned: As it is prone to dough.

36:52.41 Chris: Now the other side of this is the cost of the breach should it happen three point five million globally in aggregate pretty substantial which could have ended the life of any number of small to medium businesses.

37:07.73 Ned: Me.

37:10.17 Chris: Steve’s company did have cyber insurance. But even with insurance which I have to remind people is getting harder and harder and more expensive to get if your data is gone. It doesn’t matter that the insurance company cut you a check. You might be done anyway.

37:26.27 Ned: Right? You have enough to pay out some severances.

37:29.80 Chris: Exactly you can’t put a price tag on reputation for example, which is why I’m not allowed back in the cal door anymore. So long story short too late.

37:37.92 Ned: Understood. Yeah.

37:44.66 Chris: This is kind of all this stuff added together is why defense in depth is the essential way to go about it. Many layers of security means that if 1 happens to be breached or fail for whatever reason there are layers of security behind it. Steve’s company had the old model. They had a firewall. And they had a small staff of it. Pros who were quote unquote good humans though make mistakes and there is no such thing as a security measure that is 100 % foolproof 100% of the time. The bad guys will always.

38:15.57 Ned: Ah.

38:24.19 Chris: Evolve and users are always going to want to download and install efflux from that sketchy website that their cousin said was totally cool.

38:30.83 Ned: A cousin Joe has never steered me wrong. He was right about listen. Okay, he was he was wrong about the the shrimp business he was trying to get me involved in.

38:34.33 Chris: Is that before or after he got out of prison.

38:45.59 Ned: But fortunately I had already lost all my money in the previous doxin breeding thing that he was doing so I didn’t have anything to invest in the shrimp. It’s all about happy accidents. Apparently all right shut lightning round.

38:52.17 Chris: Call that a happy accident.

39:02.13 Chris: I Think we must. We have to look at it.

39:05.26 Ned: We shall who would end and we’re going to talk about insurance. So everybody’s immediately asleep if they weren’t already nap time with the boys cloud outage insurance is a real thing. You should probably have. Remember how I predicted that cloud outages would become more frequent and no one would really care I totally did and I was correct about the first half, but it turns out some people might care more than I thought at least about cloud based services that sit near anything financial. Or health care related. There’s a new company called parametrics which is spelled not like it sounds because company names. Parametrics insurance is looking to provide financial protection for companies that are impacted by a cloud outage that lasts up to 24 hours after which your typical cyber insurance tends to kick in since most cloud outages are resolved in far less than 24 hours existing insurance often doesn’t cover the financial cost incurred by a policyholder. Parametrics has set up their own monitoring system to track outages across all the major public clouds and their services and it automatically flags outages that impact their customers. The policy holders are reimbursed within fifteen days after an outage occurs and it seems like the whole thing happens.

40:34.98 Ned: Kind of automatically the policies aren’t cheap though. Starting at $100000 a year and quickly going up into the tens of millions so this might not be a good fit for your aunt’s flower shop parametrics is only the first of what I’m sure will be several companies getting into this market segment. So listen if you’re too lazy to architect architect your application to survive a cloud outage I guess you now have an easy way out slacker.

41:01.89 Chris: Easy way out. That’s the best way out Microsoft to change functionality of print screen button to just open the snipping tool following the famous tech adage of if it ain’t broke. Let’s go ahead and fix it. Microsoft’s latest preview release of windows eleven is changing the twenty five odd year history of the functionality of hitting print screen in the olden days hitting that weird button that nobody knows why it’s there in the first place would take a screenshot of the entire screen and save it to the clipboard. Next release this could be changed to simply bring up the snipping tool menu now to be fair I haven’t used this new mapping yet and it’s definitely true that Microsoft has historically been terrible at screenshots which. Is why for at least 20 years the way to get good screenshots on windows has always been install program x instead popular enough that I guarantee everyone listening is thinking of at least 1 program. They’ve recommended at some point now the snipping tool has gotten decent.

42:10.43 Ned: Um.

42:16.10 Chris: Over the past few years and I’m sure that nobody knows that it’s there or maybe they don’t know its full potential. So this key mapping change could be a positive but based on the fact that it’s different has made the internet decide that it is in fact, a negative.

42:32.13 Ned: You know I had to check my keyboard to see if I even have a print screen button I do it shares the cis rick function which I truly have no idea what that does right? who calls Joe who calls Leo.

42:46.20 Chris: Um I think it calls Rick.

42:51.39 Ned: How it works bard a I sings a song of misery and woe does that make chat gp juliet or Romeo or maybe Tybalt who knows ours Technica has put together a collection of reports regarding advertising company Google. And their mad dash to get barred out of the door seems as though the surprise success of chat gp really caught the old goog off guard and that led to a code red situation a phrase which may or may not have actually been uttered by Sundar Pachai this ah damn the torpedoes full speed ahead stance has landed barred on some rocky ethical shores with na and ethical advisor to help them from the very public firing of timnit gebru and Margaret Mitchell to the defection of Sammy Benggio the cadre of ethics Ai researchers at advertising company Google has dwindled down to basically nothing meanwhile anyone working on a bard related project is being actively told to ignore the ethics questions and keep doing quote real work has that real work. Resulted in a product that is at all competitive with chat Gpt which is also not really known for its accurate information or responsible disclosure. Why no no, it hasn’t and that’s because advertising company Google is not actually good at making things.

44:20.73 Ned: They’re good at selling ads.

44:26.20 Chris: Um, cold fusion Wait No ah Sodium Ion batteries set to make it to production soon. Ah Lithium Great song, Really good element to use in batteries too except for the whole massive scarcity problem.

44:27.80 Ned: Yes.

44:38.70 Ned: Are.

44:43.37 Chris: The industry has been working feverishly to replace lithium as a power storage ingredient for decades and it seems that enough progress has been made that can we can even swap out the word soon with an actual date to wit battery Manufacturer C Atl which I really hope is pronounced cattle.

44:55.93 Ned: Um, tone.

45:03.22 Chris: Announced a two hundred watt per Kilogram battery that will be ready for large scale production this year not only that they announced this week at the shanghai auto show that the battery will be installed and sold in the Sherry Icar line by the end of 2023

45:22.24 Ned: Um.

45:22.52 Chris: Now Icar is a new quasi-independent brand of the chinese based sherry manufacturing company kind of like scion was for too toota tooda what’s a car there are at least 6 car types in the Icar brand that can and will benefit from this new battery. Which while not initially more efficient than lithium-based batteries is far easier to produce on both workers. And on the environment. It’s also far cheaper estimates put lithium costs at $80000 per ton to mine and produced compared to its sodium equivalent at drumroll please three hundred bucks

46:15.95 Ned: Wow Ok, slightly different used routers provide a path to company secrets path route. You get it right? Listen it’s It’s early on.

46:31.29 Chris: Leo kill.

46:35.22 Ned: And I’m only on my second cup of coffee but even in my caffeine deprivved state I know enough to wipe the data on all my electronics that I might put up for sale. Sadly, the same cannot be said of most companies. Security firm eset decided to see what folks were leaving behind on their network gear they purchased 18 used routers from ebay and cracked them open fully half of the routers had not been wiped or encrypted in any way with vpn credentials and unhashed root passwords. Just. Hanging out in the open for anyone with a console cable to see 2 of the devices were at least encrypted but not wiped and 5 were actually properly wiped also on the unwiped devices were a router to router authentication keys network connection credentials for. Other companies and customer data. This is a freaking gold mine for any cybercrial or would-be hacker you’re literally being handed the keys to the castle and in some cases the keys to partner castles as well. In 1 case, the researcher had remote access credentials. To a major accounting firm not because the device was originally from that firm but because it was from a partner who was connecting via a site to site Vpn. This is just the laziest kind of security lapse as wiping a device is not exactly a difficult process.

48:07.62 Ned: I suspect in many cases the effort to resell was not spearheaded by the networking or the security team but instead was an effort by someone else in the company to save a few dollars and it will until they get sued into oblivion for corporate negligence.

48:23.67 Chris: Facebook settles privacy violation class action lawsuit for seven hundred and twenty five million dollars that doesn’t seem like enough.

48:32.20 Ned: Not nearly.

48:35.89 Chris: A class action suit that covers users of Facebook through the preposterous timeframe of may 2007 through December Two Thousand and twenty two has finally settled. That’s basically everyone ever the lawsuit originated with Facebook’s gross negligence regarding the Cambridge analytica.

48:42.31 Ned: So all of y’all.

48:53.63 Chris: Quote unquote breach back in 2018 where if you’ll recall in total violation of their terms of service Facebook basically handed over all of their data on at least 87000000 users just because.

49:04.61 Ned: Bro.

49:09.33 Chris: Um, the settlement understandably covers. Basically everyone who’s ever had a Facebook account and you’re eligible to get your share too simple to do make a claim before August Twenty Fifth 2023 at Facebook user privacycysettlement.com confirm your usage during the aforementioned decades and sit back and wait eighteen months for your $4 and fourteen cents to show up in your mailbox because you are worth it.

49:35.37 Ned: Daddy’s going to sizzler hey thanks for listening or something I guess you found it worthwhile enough if you made it all the way to the end so congratulations to you friend you accomplished something today now you can go sit on the couch watch some hockey and contemplate the fact that. Song lithium came out nearly thirty one years ago for the rest of the day you’ve earned it. You can find me or Chris on Twitter at ned 1313 and at hay or 80 respectively or follow the show at chaos underscore lever if that’s the kind of thing that you’re into. Show notes are available at chaoslever.com as is the sign up for our newsletter if you like reading things you can get that there. We’ll be. We’ll be back next week to see what fresh hell is upon us Tata for now.

50:22.82 Chris: So did you ever even get any money back from the equifax thing.

50:27.13 Ned: No was that an option because I haven’t been to Sizzler in forever and I really want to go.

50:33.62 Chris: Well with the money that I got back from it. You could buy a napkin.

50:38.60 Ned: Or go to Golden Corral. It’s not as expensive but it’s still okay.


Chris Hayner

Chris Hayner (He/Him)

Our story starts with a young Chris growing up in the agrarian community of Central New Jersey. Son of an eccentric sheep herder, Chris’ early life was that of toil and misery. When he wasn’t pressing cheese for his father’s failing upscale Fromage emporium, he languished on a meager diet of Dinty Moore and boiled socks. His teenage years introduced new wrinkles in an already beleaguered existence with the arrival of an Atari 2600. While at first it seemed a blessed distraction from milking ornery sheep, Chris fell victim to an obsession with achieving the perfect Pitfall game. Hours spent in the grips of Indiana Jones-esque adventure warped poor Chris’ mind and brought him to the maw of madness. It was at that moment he met our hero, Ned Bellavance, who shepherded him along a path of freedom out of his feverish, vine-filled hellscape. To this day Chris is haunted by visions of alligator jaws snapping shut, but with the help of Ned, he freed himself from the confines of Atari obsession to become a somewhat productive member of society. You can find Chris at coin operated laundromats, lecturing ironing boards for being itinerant. And as the cohost on the Chaos Lever podcast.

Ned Bellavance

Ned Bellavance (He/Him)

Ned is an industry veteran with piercing blue eyes, an indomitable spirit, and the thick hair of someone half his age. He is the founder and sole employee of the ludicrously successful Ned in the Cloud LLC, which has rocked the tech world with its meteoric rise in power and prestige. You can find Ned and his company at the most lavish and exclusive tech events, or at least in theory you could, since you wouldn’t actually be allowed into such hallowed circles. When Ned isn’t sailing on his 500 ft. yacht with Sir Richard Branson or volunteering at a local youth steeplechase charity, you can find him doing charity work of another kind, cohosting the Chaos Lever podcast with Chris Hayner. Really, he’s doing Chris a huge favor by even showing up. You should feel grateful Chris. Oaths of fealty, acts of contrition, and tokens of appreciation may be sent via carrier pigeon to his palatial estate on the Isle of Man.