Transcript

[00:00:01] Chris: Great. Now I’m all bothered. I had a question for you and I don’t remember what it was. So I guess, really what I’m asking is why won’t you answer the question?

[00:00:11] Ned: I’m just trying to keep things chill. I’m trying to keep things relaxed. Chris hakuna matata, if you will.

[00:00:18] Chris: Monster.

[00:00:22] Ned: It really is true that if someone says, when I was a young warthog, it’s just, when I was a.

[00:00:28] Chris: Young ward hog, it’s impossible not to.

[00:00:34] Ned: For people of a certain age. You’re absolutely correct. And I apologize to everyone in the world, our Sunday night neighbors, who were.

[00:00:43] Chris: Not expecting that shit at all.

[00:00:48] Ned: There are worse things to be woken up by.

[00:00:51] Chris: True. And actually, it’s a work day. They’re still asleep. Come on, now.

[00:00:57] Ned: Well, not everybody works in nine to five.

[00:01:00] Chris: Put it together, Claude.

[00:01:05] Ned: I can’t tell if I’m Claude in this situation or if Claude is your neighbor.

[00:01:09] Chris: No, Claude’s the neighbor.

[00:01:10] Ned: Okay. All right.

[00:01:11] Chris: Claude and Genevieve, we’ve talked about this.

[00:01:15] Ned: I still think it’s weird that my daughter lives next to you. It’s a different genevieve seems very young to move out at six, but I have to admit, she’s got Hutzpah.

[00:01:25] Chris: She’s got the mansion up the hill.

[00:01:28] Ned: Well, she did invest early in Bitcoin around the time she was born and divested just in time for the market to crash.

[00:01:36] Chris: So she’s always been she’s always been savvy when it comes to Ponzi schemes.

[00:01:41] Ned: Especially when she’s running them. I can’t tell you how many times I’ve been fleeced. Right.

[00:01:47] Chris: That’s the best way.

[00:01:49] Ned: Isn’t being a child just one big Ponzi scheme? The parents are never going to get that money back.

[00:01:55] Chris: That’s true. But I think the difference there is they don’t expect to.

[00:02:00] Ned: Not the way you do it. Yeah, no, that is a good point. Parents, the reward is the child. Right? Right.

[00:02:14] Chris: Yeah. You look like you’re really buying everything that you’re saying right now.

[00:02:18] Ned: There was someone on Twitter that was talking about how their children are a blessing, blah, blah, blah, and like, yeah, children are a blessing, that’s cool. But then people were like, Sometimes they’re not. And he’s like, no, they always are. And I’m like, Whoa, buddy, that’s some toxic Positivity you’ve got rolling in here, and I don’t need that in my life. We can admit that sometimes good things are not good.

[00:02:44] Chris: Right.

[00:02:45] Ned: And that’s fine. That’s healthy. Let’s be healthy about this.

[00:02:50] Chris: Yeah, it’s like short term, long term. Sometimes you’re mad at somebody and then you make up and everything’s good, and sometimes you don’t. Right.

[00:02:59] Ned: And sometimes that’s also good.

[00:03:02] Chris: What are you trying to say?

[00:03:04] Ned: Well, I’m not thinking of anyone in particular, but I am, and why are.

[00:03:07] Chris: You pointing at me?

[00:03:10] Ned: It’s clawed behind you. He’s in the house. He’s not in the house. I mean, he’s in the house in a colloquial sense.

[00:03:23] Chris: You’re just going to keep going, aren’t you?

[00:03:25] Ned: Until you stop me.

[00:03:27] Chris: Hello?

[00:03:28] Ned: Alleged human. And welcome to the Chaos Lever podcast. My name is Ned, and I’m definitely not a robot. And since I am definitely a human person, I would like to celebrate this episode being number 42 for celebrating 42 for its famous for being one of the lowest Catalan numbers. That, of course, means it describes an n sided convex polygon that can be cut into triangles by connecting vertices with line segments. We all know that with the Nth element of the sequence described, and I’m sure I don’t have to remind you by the formula C of n equals two times n factorial over n factorial times n plus one factorial. Of course. Right. And obviously the density of numbers is null at infinity. Neat. What was that about? A book or something? Why is everybody crying? Me. AWS, Chris, you’re here, right? You all right, buddy? What?

[00:04:46] Chris: I understood triangle.

[00:04:50] Ned: It’s a good start. Are you familiar with the complex plane and the Fibonacci sequence?

[00:05:03] Chris: I understand airplane.

[00:05:05] Ned: Go fly, go, fly far away.

[00:05:09] Chris: Zoom, zoom. Wait, no, that’s Mazda.

[00:05:14] Ned: When I was a young warthog. We’re back to it when I was, I don’t know, nine or ten, probably. We got the Apple two E, maybe, or it might have been the two GS. It was one of the ones that could render some graphics, had a little bit of horsepower to it. And we had a program that would create Mandal broad sets, which is those.

[00:05:39] Chris: Cool fractal design, the never ending fractal thing of a jet.

[00:05:42] Ned: That’s the Mandal broad set. And the way that you construct a mandel broad set is through complex plane, which involves complex numbers, which includes the square root of negative one, which, as we all know, is I.

[00:05:59] Chris: You are pretty negative and square and egotistical.

[00:06:03] Ned: So I’ve got all of it covered. So what was amazing about this computer was that it was able to do these complex plane calculations and create the Mandal Brat set in a fractal on your screen. And you want to know how long it took? 26 hours. Wow, lightning fast.

[00:06:27] Chris: Time flies.

[00:06:31] Ned: In the not too far distant future from then, that became a screen saver that rendered in seconds and then kept rendering because it could anyway.

[00:06:43] Chris: So anyway, math is awesome. I’ve always said so, and I’ve always respected math’s. Right. To exist.

[00:06:53] Ned: All right, should we talk about some tech garbage?

[00:06:56] Chris: Surely.

[00:06:57] Ned: All right. No. Great segue this fine day. You guys didn’t have it in me. That’s sad.

[00:07:04] Chris: But you used, like, ten of them last week.

[00:07:06] Ned: That’s right, the well ranth dry. Quick reminder for listeners out there, we have a newsletter now. So if you prefer not to listen to Chris and I ramble on, you can instead read our newsletter and get a more succinct view of what we talked about. But you still listen to us because we’re back.

[00:07:24] Chris: If they aren’t listening, then they won’t know about the newsletter. So it’s a paradox. It’s a reverse paradox.

[00:07:31] Ned: It’s a catch 42. Today we are going to talk about It versus OT. Battle of the century.

[00:07:44] Chris: What if my company doesn’t do overtime?

[00:07:47] Ned: It’s not that kind of OT, my friend. I’m sorry to say, though, problems with your OT can cause overtime. That’s certainly true.

[00:07:55] Chris: That’s what we call a teaser.

[00:08:02] Ned: So we have been focusing a lot on security lately, and I see no reason to change that. But instead of talking about the cloud and compromised SaaS companies, instead, I thought we could learn a little bit about some of the most sensitive and critical systems we have out there. Systems that people rely on to operate consistently and safely. Otherwise they could cause catastrophic harm and in some cases, cost human life. I’m speaking of operational technology, or OT for short. That’s what it means, yes. Now, in fairness, that was a term that was completely new to me until I started working on a consulting project for a major manufacturer regarding a network refresh across their entire environment, including their offices and manufacturing plants. And suddenly the term OT networks was thrown around with words like skada and Plc and historian. And I realized I was in over my head, a constant state of affairs in the world of consultants. I know, Chris, we can do a whole episode on how being a consultant is like constantly treading water while pretending you know what’s going on.

[00:09:29] Chris: The secret to being a consultant is doing your hyperventilating in private.

[00:09:33] Ned: That’s right. That’s what you do on the train on the way to the consulting engagement, Chris. So like any good consultant, I immediately fessed up my ignorance and referred them to a qualified vendor who took over the contract. And no, I didn’t do that. That’s not what we do in consulting. I nodded and tried to say things that didn’t make me look like a fool. Difficult, but I may have accomplished it. And then I went home and spent the next three days reading everything I could about OT and the security requirements around it. And then I could sound smart. Ish. Ish after that project concluded, I ended up being an analyst on Edge Technologies for a while. And once again, operational technology reared its ugly. That’s not cool. It’s not ugly. It’s functional. It reared its functional head once again. After all, one big aspect of Edge Technologies is the industrial Internet of Things IIoT and those things plug into OT systems.

[00:10:45] Chris: It save functional without fun.

[00:10:49] Ned: Because no, you’re right, you can’t. So, as you recently reminded me, Chris, edge is not a thing and should be retired in the same vein as super cloud. For those who are interested, there’s a whole Register article about that. I don’t entirely agree with the article, but here’s one case where we can point to a real thing and say that’s Edge OT systems are quite literally at the beginning of where It systems end. And we can call that the Edge.

[00:11:26] Chris: Sure.

[00:11:27] Ned: The liminal space. Liminal? It.

[00:11:32] Chris: Sounds tough. It’s lit.

[00:11:34] Ned: Lit. I love it. Okay, we are starting a new company, Liminal It. And our tagline will be it’s lit. Get lit. Get lit. There it is. We did it. Everybody. This will be the last episode of Castle, right?

[00:11:51] Chris: Please invest in our spack.

[00:11:53] Ned: Yes. Oh, God. That just makes me feel awful every time I hear it. Yeah, it sounds like something you catch in a hospital. I got an awful case to SPAC. So based on my somewhat recent ignorance on the topic, I think it’s probably safe to assume that other people are also ignorant about operational technology. And based on some recent security news, more on that later. And the rise in ransomware attacks on critical infrastructure because people are awful, I thought now might be a good time to bone up, if you will. Let’s start with definitions. Woo. So OT systems are systems that directly interact and control physical equipment. This includes the actual controller boards that manipulate machines, the monitoring systems that log sensor data, and the decision systems that make changes to individual devices or the system as a whole. Previously, these were referred to as industrial control systems. ICS. But that definition has been expanded AWS more things become automated and sensorized and whatnot to include stuff like building automation systems, for instance, the HVAC in your building or the fire alarm and control and suppression systems. The transportation system like your subway. Or the controls for traffic lights and physical access control.

[00:13:32] Ned: Remote door locks, and my favorite, man traps.

[00:13:37] Chris: Good name. Yeah, everyone likes that name.

[00:13:40] Ned: First time I went into data center and they’re like, oh, you got to go through the man trap. I was like, Is Darryl Hall here? What’s happening? And you also have stuff like physical environment monitoring and measurement systems. So a weather station, for instance.

[00:13:57] Chris: But the key differentiator here is it’s either software or a system that actually interacts with the physical environment in some way.

[00:14:06] Ned: Right. This is not an app on your phone. This is not a web server serving you content. This is actually like toggling relays and making changes to the physical world that could have varying levels of impact.

[00:14:20] Chris: Right?

[00:14:21] Ned: Like perhaps if you switch the wrong rail and two trains collide.

[00:14:26] Chris: Is that bad? I thought that was just survival of the fittest.

[00:14:31] Ned: I guess it depends on whether you’re on the train or not. So before the Internet and yes, dear listener of a certain age, there was a time before the Internet we call that heaven.

[00:14:46] Chris: We called it ARPANET.

[00:14:48] Ned: We call it a better time. OT systems were not connected to a larger network. And even once local area networks and wide area networks became prevalent, OT systems were generally kept air gapped or bridged via some firewall or DMZ type setup. It’s really only in the last ten or 15 years that these systems have become increasingly online and it sure was not the OT engineers idea and we’ll.

[00:15:18] Chris: Get to whether or not it was a good idea at all, I’m sure.

[00:15:23] Ned: Spoiler? The answer is no. But it gets to a central core tenet of OT that is different than It information technology. If there’s one main difference between the two, it’s that of speed and stability. If it has a motto of move fast and break things, OT’s motto would be move slow and break nothing.

[00:15:50] Chris: Emphasis on that last one.

[00:15:52] Ned: Yes, OT will almost always trade new features for stability. And I’m going to be honest, I had to write that down twice because I kept saying it would trade stability for new features because that’s what we do in consumer tech and it but no, it’s called CES. No, I’m just set. But now, given the choice of a stable system versus some shiny new buttons and options, the OT engineer will always choose stability. And the reason why? Because when OT systems break, they can maim or kill people. And I hear that’s bad.

[00:16:38] Chris: That’s the whole consequences of interacting with the physical world thing.

[00:16:43] Ned: Yeah.

[00:16:44] Chris: If your website goes down, it’s one thing, right? If you’re say, oh, I don’t know, assembly line has a problem, that’s something else.

[00:16:56] Ned: Exactly. And before you reply, guys show up. And yes, it’s always guys, don’t at me. I just want to point out that, yes, it can cause the death of human beings through malpractice and bad judgment. You are entirely correct about that. But that doesn’t seem to give us more than a moment’s pause. So get down from your high horse and shut up. We’re talking about OT. So the difference in approach is one that results in behaviors like running a 30 year old operating system because they just want it to keep working and that one keeps working. I’m sure you have stories as well, Chris, but I remember specifically talking to someone in print media who told me their printing press facility was still running Windows Three one One in 2012 because it worked. And you don’t want someone’s hand getting crushed because Microsoft introduced a new device driver in Windows Eleven that flipped the wrong bit. The problem they were having was finding hardware that could still run the old OS as the PaaS on their existing system started to fail.

[00:18:10] Chris: Yeah, I mean this is staggeringly common even now in 2023, maybe we’ve moved on from Windows 3.1 to say, I don’t know, Windows XP SP Three, potentially the best operating system Microsoft ever made.

[00:18:27] Ned: Oh yeah, I agree. Fight us on that.

[00:18:31] Chris: But like you said, one of the big things about these types of softwares and the types of controllers is they were not historically online at all.

[00:18:39] Ned: Right.

[00:18:40] Chris: So the risk environment is significantly lower just because you don’t have a network card or a WiFi card. So what are you going to do? Who cares if there’s a zero day for XP that hasn’t been patched for 16 years, nobody can get to it.

[00:18:57] Ned: And nothing highlights the popularity of that specific release. Then how long Microsoft continued to support Windows XP after it was technically end of life.

[00:19:07] Chris: Right.

[00:19:08] Ned: They were only supposed to support it for ten years, and I think it stretched out to almost 20. And if you pay enough money, you can still get support for it, I’m sure. So, like I said, when those OT systems were mostly offline and they required physical access to operate, using a 30 year old piece of software that was riddled with security issues was fine. I mean, it wasn’t fine, but it was fine enough.

[00:19:37] Chris: Certain definitions of fine.

[00:19:39] Ned: Yeah, for their definition of fine. Right. But as these systems became increasingly connected, suddenly that 30 year old software with a password of admin and no username or vice versa becomes a little more problematic. Especially if that system controls, say, an industrial waste treatment plant, just as a for example, did you use that one.

[00:20:07] Chris: Randomly or did you pick that one specifically?

[00:20:11] Ned: Yes.

[00:20:13] Chris: Okay, remind me at the end, because I have a fun story about an industrial waste plant that highlights this very well.

[00:20:19] Ned: I seem to recall something in the back of my subconscious, but I couldn’t find the article through Quick Googling, so I just left it in. Fair OT systems also tend to run on specialized hardware using proprietary control and networking protocols. And like we said before, they’re in an isolated environment. So when you think about it, it’s really not all that different than the mainframe era of computing. You had these proprietary systems, they didn’t necessarily talk to each other over any standardized format. Hacking one of them was pretty hard because there weren’t that many of them and they weren’t connected. It wasn’t until the advent of hardware and software standards, along with the commodification, that changed it into the structure we know today, the rollout of X 86, the rollout of Linux and Windows basically being the operating systems for everyone. Now that has started to change in OT as well, but much more slowly. And before we get into how it’s changed, let’s cover what is actually being changed.

[00:21:28] Chris: More definitions. Yay.

[00:21:29] Ned: I know you’re excited, and I titled this section Pieces of you. That’s a jewel song and an album, and I know that, and I recommend that you do not, under any circumstances, look up the lyrics to that song. I know the 90s was a different time, but yikes. Okay, so I’ve talked about OT systems in the abstract and mentioned a few different kinds of systems, but let’s nail down some basic components. We have our programmable logic controllers, or PLCs, and they’re basically what they sound like. The PLCs take input from sensors, process that information based on a program, and then send instructions to actuators if required. The processing loop can involve a human or be fully automated. So an assembly line or an HVA system might be Plc based. Your thermostat at home is probably a fairly rudimentary Plc, although depending on how much you’ve spent on your thermostat, it could be wireless and increasingly more complicated. But basically, it checks its sensors. What’s the temperature in the room? If it’s too hot, it closes a contact that turns on the AC. And if it’s too cold, it closes a contact that turns on the furnace.

[00:22:54] Ned: Very simple. Then we have Distributed Control Systems, or DCs, and that would take inputs from multiple sources and make a coordinated decision on changes to the process. A DCs is deployed on a larger scale system, like an oil and oil or gas refinery. But a DCs is geographically local in nature. All the systems are in close proximity, as opposed to SCADA. SCADA stands for supervisory control and data acquisition. And that’s a network of OT systems that includes PLCs Remote Terminal Units, or RTUs. And those transmit data to a central controller for processing and monitoring. A human operator may be monitoring the systems that are part of the SCADA network from a control center. And those control centers are connected to field sites via whatever telecommunications form is available. So that could be copper, wireless, satellite or even Pots, plain old telephone service, whatever they could get out to that site. SCADA networks can be deployed in a variety of topologies, including point to point hub and spoke or a hierarchy of sites. We’ve got two more here. Industrial Internet of things IIoT. The S is for security.

[00:24:21] Chris: I get it.

[00:24:23] Ned: This is basically sensors, actuators and other devices that are now using the Internet and Internet protocols to connect to each other and a centralized controller for OT processes. When we talk about edge computing, we’re usually talking about IIoT and then finally, the historian. That’s a system that keeps logs in a time series database for analysis and compliance. Typically, it receives data from the PLCs or through a DCs or a SCADA network, meaning it has an interface in both the OT and It networks. Good.

[00:25:01] Chris: Got it.

[00:25:02] Ned: Okay, awesome.

[00:25:03] Chris: Do I get a certification now?

[00:25:05] Ned: You’re pretty close, honestly. So the goals of let’s talk about OT security now I feel like we’ve primed the pump with what’s in the OT network, what the high level principles are, or the tenants of running an OT environment. So the goals of OT security, they’re really not that different from traditional It, but there are unique constraints and some priorities that need to be taken into account. NIST has published a really excellent paper on deploying cybersecurity for operational technology. And so I just want to go over the high level goals that they’ve outlined for an OT network. And some of these are going to be super familiar, right? Restrict logical access to networking systems. Restrict physical access. Protect individual OT components. That one is actually not done as much because they traditionally were not networked. Protect data in transit and at rest. Again, something that was not very common in the past fail open and fail working. What this means is if something fails in the system, it should fail in such a way that it keeps the people using that system safe.

[00:26:27] Chris: Right? Yes. The famous example for the difference between fail open and fail closed is if you have a door into a room with no other exit.

[00:26:35] Ned: Right.

[00:26:36] Chris: And that door is automatically controlled by some type of a system. If that system fails and it fails closed, you’re going to die in the closet.

[00:26:46] Ned: Yes.

[00:26:47] Chris: But if it fails open, the door will be able to you will be able to go through the door because the lock will have opened as a result of the failure.

[00:26:55] Ned: Right. And for instance, if the power were to go out, it’s being held closed by power, and the absence of power unlocks the door.

[00:27:04] Chris: Right.

[00:27:04] Ned: Which may not be great from a security perspective, but is really good from a don’t kill people.

[00:27:12] Chris: Right? Yeah. There’s informational security, but there’s also human life, which in almost every case is more important.

[00:27:22] Ned: Depends on 51%. And lastly, rapid restoration of service. So downtime should be kept to a minimum and you should be able to bring things back up extremely quickly because lives might be depending on it. So there’s three big constraints I want to highlight around OT systems that are a little unique versus traditional information technology systems. So in It to protect systems, we would typically deploy agents or a process to monitor for things like virus and ransomware. We would want to run regular file scans. I mean, your computer probably ran a virus scan today. Might not have told you that, but it probably did. We also want to encrypt all data before sending that data along and decrypt data that was sent to us using something like Mutual TLS. We want authentication and authorization for each request and automatic rotation of certificates. Unfortunately, many of the OT systems that are out there simply do not have the capacity to run an agent or do encryption on the fly or run comprehensive file and process scans. These, especially, like PLCs, have extremely limited resources that can really only do the thing that they were designed to do.

[00:28:49] Ned: And priority always has to be given to that primary purpose.

[00:28:54] Chris: Right. It’s a difference between a generic CPU of some type that just has horsepower that you might not ever need. If you need examples, look at your laptop.

[00:29:05] Ned: That’s usually right. 10%.

[00:29:07] Chris: Right. Versus something that is literally designed for this primary purpose and absolutely nothing else. Plc will run at 90% utilization all the time.

[00:29:19] Ned: Right.

[00:29:19] Chris: We ain’t got time for BitTitan.

[00:29:24] Ned: Exactly. Or whatever weird thing you wanted to play out to it. And also just simple things like that. CPU was never designed to do encryption and decryption. And if you tried to do it. It’s extremely inefficient at It, right? So what you have is a whole cottage industry of startups collusions. Oh, startups, yeah, startups, aka collusion that basically will sit in front of this device and add that security layer to it because the device itself can’t do it. The next big constraint is the high cost of testing. So in the world of It, we’re used to the marginal costs of a second system for testing to be minimal. Hell, the cloud has made infrastructure disposable half of the time. Do you need a dev environment to test something? Sure. I can clone Prod and deliver your environment in about 30 minutes and tear it down when you’re done. Cost us like $2. That’s the benefit of using commodity hardware and standardized software. It’s very easy to do this sort of thing. I can pretty accurately test my changes and updates in a development environment that is 99% the same as Prod without spending a shit ton of money.

[00:30:43] Ned: And if that 1% slips through, maybe 1% of my customers won’t be able to buy my bespoke crafted holiday soaps made from 100% quail fat. That’s Pennsylvania quail fat.

[00:30:56] Chris: Those are the fattest quails.

[00:30:59] Ned: Have you seen it eat a Philadelphia pretzel? Now take all of that information and throw it at a dumpster, because none of it is true in OT systems. We’re talking about proprietary systems that are controlling physical devices with no virtual equivalent. The cost of a duplicate setup is the cost of the original setup. You want to test a change in dev? Ha. There is no dev, only zool. I mean Prod, but also zool. This is one reason why the digital twin idea is so attractive in IIoT and OT. Imagine if you could build a virtual version of your factory gas pipeline, water treatment plant, whatever and test your changes on that virtual version. That sounds pretty compelling. Sure hope you don’t get that model wrong. Which brings me to the second high cost of testing getting it wrong. So a failure of 1% is not great, but I’d say acceptable on most It systems. My unsold quail soap notwithstanding, things are going to sort themselves out and a little downtime. It’s good for the soul. Not so when you’re running a transit network or a rail system carrying freight. A 1% failure rate means, at best, the loss of thousands of dollars and at worst, the loss of life.

[00:32:28] Ned: Just rebooting it is probably not an option, and updates need to be planned out weeks or months ahead of time at a minimum. So that zero day exploit for Java.

[00:32:40] Chris: Also known as Java we’ll get to.

[00:32:43] Ned: That at the next six month change window. Assuming the vendor has developed and tested a patch in time, which assumes that the vendor still exists, which is another big challenge.

[00:32:55] Chris: I mean, that’s a problem with all kinds of IoT internet, good God. Industrial or otherwise. Yes, but yeah, I mean, the failure rate is a great exploratory question when you want to define risk. So if you have a website and it fails a user logging in one out of 100 times, that user is going to go, oh, what the crap, and then log in again and they’ll be fine. If you run an airline that crashes one every 100 times, I think you’re going to have more of a problem.

[00:33:30] Ned: Well, there’s a 99% chance that my flight won’t fall out of the sky in a fiery blaze of explosion. And I really have to get to Omaha.

[00:33:43] Chris: Somewhere in the middle of America.

[00:33:46] Ned: They get right to the heart of matters. And that matters what matters more? You better turn your ticket in. God. All right, that’s enough.

[00:33:57] Chris: We’re going to get sued.

[00:33:58] Ned: All right, number three components, lifetime and location. You’re a developer and you want to swap out the library in your net application. Easy, PaaS. Now, you need to power cycle a virtual machine. No problem. That’s a simple shell command. Need to swap out a drive on your physical server? That’s a little more work, but you probably got someone in the data center for that. Now, what about a Plc board living on a desolate track of rail 250 miles from the closest town? Uploading data across a satellite feed with screaming 64 Kbps upload speeds.

[00:34:39] Chris: Great. So we’ve gone from counting crows to a sad Bruce Springsteen song.

[00:34:43] Ned: Yeah. So put down the telecaster. To even push an update down might not be feasible from a remote location to begin with. And even if you can, if the device requires a physical reset, you’re going to have to roll a tech out there. And chances are you Azure going to deploy that hardware and leave it in place for the next 15 to 20 years with minimal maintenance and patching, in part because it’s too hard to get to the device. And the consequence of rolling out a botched update to a thousand remote devices situated in similar areas would be devastating impossible. Really expensive.

[00:35:24] Chris: Oh, yeah, that’s bad, too.

[00:35:25] Ned: That’s the one. Even if the Plc board isn’t in a remote location, chances are you aren’t going to swap it out on a regular basis. So any new software updates and security patches have to support hardware and compatibility going back 15 plus years. Your vendor is usually responsible for blessing the updates that go out, meaning they have to maintain a back catalog and inventory of the devices going back a couple of decades. Their incentive to roll out new hardware models is pretty low, and their incentive to roll out and test security updates is even lower. Yeah, those are all working against you when you’re trying to implement security in an OT environment. But eventually, the benefits of newfangled technology went out in any field of battle, and OT is no different. The renewed concerns for OT security are largely due to the introduction of It systems and processes. Into the OT world. So the cloud, and more specifically machine learning, have introduced the concept of a digital twin, a virtual facsimile, and corporations are constantly looking for ways to lower costs and improve efficiency. And things like the digital twin and other tools available from modern data science offer just such an avenue.

[00:36:53] Ned: And you know what wins over human life? Money.

[00:36:57] Chris: Oh yeah.

[00:37:00] Ned: So as a result, the OT systems that were once completely cut off from the rest of the world and required physical proximity to access, we’re talking about actual air gapping here, not the crap that goes into marketing materials that talk about zero trust, because that’s not air gapping. Those are now being integrated with varying degrees of success with the larger enterprise It environment historian systems are being mined for insights. PLCs are uploading data directly to the cloud, and patches are being pushed from the Internet down to OT components. This is a service that both AWS and Microsoft offer. There are real benefits to introducing It practices to OT, just as I would argue there are benefits to maybe introducing some OT practices to It, especially around stability. But if you’re an It professional and you’re starting to get into this world of OT related environments like you get placed on a project, here’s a couple pieces of advice I’d offer you. First of all, you don’t know shit about running OT, so shut up and listen to the person in charge of the OT processes.

[00:38:17] Chris: Probably got a certification, don’t we all.

[00:38:22] Ned: Assume competence, assume they know as much about their systems as you do yours, and hopefully that’s a lot. Ask questions, don’t make assumptions and defer to their expertise in anything OT related. You don’t know better. Number two, things are going to move really slow. That’s just the nature of OT. If you’re used to moving fast and knocking things off your punch list, prepare to hit the brakes and spend a lot of time in long, drawn out meetings where knowledgeable engineers worst case, your solution to death. And be ready to accept that they’re right and go back to the drawing board. You ever been around engineers?

[00:39:07] Chris: I’ve never even heard the word before.

[00:39:09] Ned: Like mechanical engineers, when they’re talking about all the different ways systems can fail, they’re really good at that. They kind of have to be.

[00:39:17] Chris: I wonder if it’s that they’re really competent or they just have very expensive mechanical pencils.

[00:39:22] Ned: A little bit of both. Last thing, security is not going to be the priority. I mean, security is never the priority, right? In It, it’s because the cost of a security incident is secondary to the business making money. And in OT, it’s because the cost of a security incident is secondary to the business making money. And also people not losing limbs. Safety will always trump security in OT. So you just kind of have to get used to it, right? If you want to build a solid case for cybersecurity in OT environments make it about safety. So I fully expect the opportunity for It professionals who understand cybersecurity to only expand in the next ten years, and doubly so for those who want to take a crack at working with OT systems. Almost every edge security startup I saw as an analyst was trying to augment existing OT systems with some security solution that didn’t get in the way of safety and stability. And if some estimates of the Nascent edge market are correct, there’s tremendous growth coming. And at the same time, we have so many aging OT systems that will need a complete overhaul and replacement in the next ten years.

[00:40:45] Ned: So if you’re looking for a new area to get into, grow your career. Maybe you were recently laid off, might not be a bad area to go.

[00:40:53] Chris: Into, especially if you still have all that XP expertise.

[00:40:58] Ned: And I do. Lightning round.

[00:41:05] Chris: Lightning round. Now they want AI lawyers. What’s the worst that could happen? I just want to start by saying, listen, I get it. The law is an impenetral morass of precedent, etiquette some random Latin, and overall, the whole situation is borderline incomprehensible to the average person. Even something as banal as traffic court is still court, and it’s always better to have representation so you don’t accidentally confess to a murderer or something. Joshua Browder, the CEO of consumer advocacy company Do Not Pay, has had it with this situation. His company has long had automated tools that help you understand how to fight traffic tickets. Now they want to take the next step AI in your ear, listening to the court proceedings and literally telling you what to say and do in real time. As you can imagine, this idea is not without its detractors. First of all, the 300 traffic court cases Do Not Pay looked at for their experiment. Only two of them would even allow a two way Bluetooth connection, aka a microphone, so that AI could hear. Secondly, the lawyers of the world are just not going to like it. Bad roar. And third, this is a roll your own AI solution that’s being tested here, and you know how we feel about rolling your own.

[00:42:46] Chris: On the right side, though, Browder did note the many limitations stock GPT applications had with the law and like facts. So hopefully his model is better. Do Not Pay explicitly stated that they are doing this for publicity to help educate consumers and hopefully inspire change. They also said that if the AI bot loses, the company would cover any fines. So that’s good.

[00:43:21] Ned: Yeah. Vulnerable historians that aren’t Chris stephanie Thornton’s.

[00:43:28] Chris: Secret history is better than procopia’s secret history. Except for all the espionage, murder and that one minor massacre. Emperor Theodora did nothing wrong. Wait, what was the question?

[00:43:41] Ned: I ran out of space in the main story, as Chris so kindly reminded me. So I shoved this in here, if you’ll remember from my longwinded exploration into OT systems, one type of system is the historian, and it’s nice. It’s not a bespeckled bookworm sitting on a chesterfield with a nice cup of earl gray, as nice as that may be. Nay, it is essentially a data lake of time series information from the OT systems that report to It. The time series data can be of great use to folks on the business side, so the historian often has to straddle both the It network and OT network, providing access to the data scientist while also collecting data from the OT systems. The US. Cybersecurity and Infrastructure Agency has issued a warning about five vulnerabilities in the GE Prophecy. That’s a terrible name, but it’s what it’s called prophecy historian joining previous vulnerabilities found in the Schneider electric vehicle historian and the Siemens cinematic process historian. Wow. They do not like naming things nicely. Because of the unique placement of historians, a client facing vulnerability could be exploited to gain access to industrial control systems, building automation systems, or any other OT system.

[00:44:59] Ned: This is compounded by the fact that most OT systems rely on network segmentation or air gapping for their security and are rarely patched. The historian, on the other hand, is not part of the process loop and should be part of the regular It patch cycle. You may want to check with your OT team to see if there are any historians on the network.

[00:45:22] Chris: And the AI generated art lawsuits have arrived. Well, it was bound to happen, and one night last week it did. A group of artists filed a class action lawsuit against two major generative art companies Mid Journey and Stability AI. The three plaintiffs, none of whom are Greg Britkowski, although I imagine he’s enjoying the show, have filed the case based on the, quote, Millions of artists whose work was used to train the AI’s models. This artwork, taken without compensation or attribution, allegedly allowed these companies to, quote, benefit commercially and profit richly unquote, leading to lost sales from the original artists. There are other suits in the works as well, including one from Getty Images Against Stability that alleges similar damages as the above, and a third high profile suit against Microsoft GitHub and OpenAI that is ostensibly about the fact that Microsoft has a lot of money. All of these cases are probably destined to go nowhere. The current state of copyright law is so behind the times that it doesn’t even adequately protect online image hosting, let alone AI. Whatever is going to happen to bring cases like these to a final resolution is going to take probably decades to sort out.

[00:46:50] Chris: I mean, maybe in that time, AI can find a way to simulate a remotely competent Supreme Court. Let’s be honest even the most powerful of computers have their limits.

[00:47:06] Ned: Twitter app I never used rendered useless. Wait. Am I the idiot? Don’t answer that. I’ve been using Twitter on a regular basis for close to ten years, and being the basic bitch that I am, I’ve never strayed from the basic app and Android app. I’ve flirted with Tweet Deck a few times, which is owned by Twitter, incidentally, but never have I strayed into the morass of third party apps that enhanced the experience. All of this to say, I had never heard of Twitterific before last week when their surprise demise popped up on my feed. Turns out Twitterific was built back in 2007 and helped coin the term Tweet as well as popularize the blue bird of Twitter. For their part, Twitter has long had a tumultuous relationship with third party applications and has broken those apps several times before by changing the Twitter API that they rely on for functionality. Although Twitterific and similar apps like Tweetbot and Phoenix have limped along and found workarounds over the years, that all came to a halt with nary a word from Twitter. Basically, the folks at Twitter made a change that blocked all third party clients without so much as a public announcement or private message to any of the client app developers.

[00:48:34] Ned: The apps just stopped working. Craig Hawkenbury. Great name. The creator of Twitterific, published a blog post called The Shit Show that makes for entertaining and frustrating reading. Finally, on the 17th, four days after blocking all third party apps, the Twitter Deb account acknowledged the change with the cryptic Tweet quote twitter is enforcing its long standing API rules that may result in some apps not working. End quote. That’s it. That was the whole explanation. Just a big old fuck you to all the third party apps out there. Now, most of the apps have now been removed from the Apple and Google App Store since they don’t work. And now everyone is just a basic bitch like me. So enjoy.

[00:49:27] Chris: Guitar center migrates most of its infrastructure into Oracle Cloud somewhere. Andy Jassy is sadly playing a kazoo, probably out of tune guitar center, the venerable place near the mall to go buy guitars and guitar based accessories when you’re particularly pissed at that goth kid who works at Sam, ash shared a story of their multi year migration to the cloud. The story is quite standard. Old custom app has run on prem for years, it barely works, and there’s far, far too much scrambling during times of high activity. The decision is made to move to modernize via. Say it with me, kids. Move to the cloud.

[00:50:13] Ned: Cloud.

[00:50:16] Chris: The interesting thing here is that they picked the Oracle cloud. Not AWS, not GCP, not Azure. Oracle sure, by all accounts, and by that I mean this press release puff piece masquerading as an article that Guitar Center in Oracle clearly authorized. The migration was a success and a money saver. Now, based on my reading of the article, this is probably because of Guitar Centers previously existing relationship with Oracle. They already had a significant investment in Oracle products. They are still using Exit data on prem, which leads me to believe that the 1.5 million they boast about saving comes down to the old. Well, if you use Oracle databases in the Oracle cloud, we’ll stop charging you out the Wazoo for licenses maneuver, and also talk nice about us in The Wall Street Journal. Oh, crap. I didn’t say that last part. Stop recording this. You’re cheating. I’m calling mom. Unquote. In all seriousness, though, the Oracle cloud, especially for IaaS, works perfectly fine. And as this example illustrates, it just seems to be working particularly finer if you’re, like, already in bed with Larry Ellison.

[00:51:45] Ned: EW, gross.

[00:51:47] Chris: Why did I have to word it like that?

[00:51:49] Ned: You can change it. More layoffs from Microsoft and Google record profits got you down. Have you tried firing some of your workforce? Last week, Microsoft announced the layoff of 10,000 people in its workforce, and now Google has followed suit with a reduction of 12,000 people. Look, we’re doing more, mom. Both companies have said something along the lines of, well, we overhired during the pandemic, and now the market and economic environment has shifted and we find that we are overemployed. While it might be true that they hired to meet projected demand during the pandemic and that the overall economic outlook isn’t as rosy as some might like a post from Gurgley Rose vROps we’ll workshop. It.

[00:52:41] Chris: Worth a try.

[00:52:43] Ned: Yeah, thanks. It rightly points out that Microsoft and Google aren’t exactly hurting for money. Microsoft has net income of $61.3 billion in 2022. That’s not revenue, that’s net income profit. Google hasn’t actually reported their year end results for 2022 yet. That’s next week, I believe. But for the first nine months, they are already sitting at $46.3 billion in net income. Not revenue income. I realize that it’s more complicated than that, but you know what? These two companies made over $100 billion in profit this year, and they can’t continue to employ 22,000 people during a financial downturn. I may not understand the complexities of the modern economy, however, I can still smell bullshit when it’s under my nose.

[00:53:42] Chris: Listen, they had to make a number of regrettable hard sacrifices while simultaneously watching a concert in a small room featuring Sting.

[00:53:52] Ned: Yes, well, we all make sacrifices. That sounds awful to me.

[00:53:56] Chris: The Microsoft guy had to sit off to the right instead of dead center. Hashtag sacrifices.

[00:54:05] Ned: Sting played fields of Goal twice. All right? It was a rough time for everybody. Hey, thanks for listening or something. I guess you found it worthwhile enough if you made it all the way to the end. So congratulations to you, friend. You accomplished something today. Now go sit on your fourth dimensional couch, sip some suds from a Klein bottle, and settle in for the latest Tesseract Theater. You’ve heard it. You can find me or Chris on Twitter at ned. 1313 and Heiner 80 respectively. Or you can follow the show at Chaos underscore Lever if that’s the kind of thing you’re into. Show notes are available@chaoslever.com, as well as our newsletter and previous issues of the newsletter. If you like reading things which you shouldn’t, podcasts continue to be better in every conceivable way. We’ll be back next week to see what fresh hell is upon us. Tata for now.

[00:54:59] Chris: So, wait, what’s? OT and I wasn’t super listening?