1 00:00:00,000 --> 00:00:04,560 Welcome to the Cloud Native in 15 Minutes podcast. The goal of this podcast is simple, 2 00:00:04,880 --> 00:00:09,020 to explain some complex but increasingly important and even strategic technologies 3 00:00:09,020 --> 00:00:14,400 and about the time it takes for a coffee break. So I'm Derek Harris from Pivotal and our topic 4 00:00:14,400 --> 00:00:20,940 today is SRE or Site Reliability Engineering. Joining me to explain SRE is Dave Rensen, 5 00:00:21,060 --> 00:00:27,400 a Senior Director of Engineering at Google and one of its, you know, for the past several years 6 00:00:27,400 --> 00:00:33,100 now, one of its public faces and primary spokespeople and engineers on the SRE team. So 7 00:00:33,100 --> 00:00:38,980 this, I think this is a really pretty interesting discussion if you're not familiar with SRE. 8 00:00:39,660 --> 00:00:45,740 Dave does a great job talking about it, not just as a technical problem, but as a business problem, 9 00:00:45,800 --> 00:00:51,120 as a problem that's designed to, you know, I think the way he puts it is make machines answer to humans, 10 00:00:51,220 --> 00:00:55,520 not humans answer to machines. It's really an interesting philosophical discussion and it 11 00:00:55,520 --> 00:01:00,240 touches on, you know, using SRE principles and everything. Yes, from reliability for your 12 00:01:00,240 --> 00:01:06,200 application, but also down to reliability for your marketing team. What's your error budget for mistakes 13 00:01:06,200 --> 00:01:12,780 and finance or whatever. So it's, I think it's a really interesting discussion. I think you'll enjoy it. 14 00:01:12,840 --> 00:01:17,040 So stick around and listen to Dave Rensen of Google talk about SRE. 15 00:01:19,220 --> 00:01:26,000 All right, Dave. Listen, thanks for being on to kick off. Can you talk listeners through what SRE 16 00:01:26,000 --> 00:01:30,680 stands for? I mean, obviously it stands for site reliability engineering, but beyond that, 17 00:01:30,740 --> 00:01:35,860 I mean, what it really, what it really means. Oh yeah, sure. So the, the SRE team here at Google, 18 00:01:35,940 --> 00:01:40,800 it's pretty easy. We're responsible for keeping all of our production systems running 24 seven. It's all 19 00:01:40,800 --> 00:01:46,540 the software systems, hardware systems, data centers, et cetera. We like to say that SRE is what you get 20 00:01:46,540 --> 00:01:51,720 when you treat operations like a software problem. All right. Can you flesh that out a bit? I mean, 21 00:01:51,760 --> 00:01:56,260 what is that? I've heard that described and I think it makes a lot of sense. If you, if you, if you know, 22 00:01:56,320 --> 00:01:59,740 if you know the kind of some context with that, but can you walk through a little bit? The way I like 23 00:01:59,740 --> 00:02:07,200 to think of it as this, you can live in one of two worlds. Um, in the first world, a machine called a 24 00:02:07,200 --> 00:02:12,240 pager wakes you up at three in the morning because some other machine is having a hard time. In that world, 25 00:02:12,240 --> 00:02:18,160 you work for the computers. The world you want to live in is one where some system you're responsible 26 00:02:18,160 --> 00:02:22,440 for is having a problem. It sort of mitigates itself. And then it writes a bunch of information 27 00:02:22,440 --> 00:02:26,800 out for you to debug the next morning after your morning coffee. That's a world where the machines 28 00:02:26,800 --> 00:02:32,740 work for you. SRE is about getting the machines to work for you. It's about performing all the 29 00:02:32,740 --> 00:02:37,060 operational things we have to do to keep our users happy, but getting the computers to perform as many 30 00:02:37,060 --> 00:02:41,300 of them is humanly possible. And that's the difference between sort of staring at a monitor 31 00:02:41,300 --> 00:02:47,740 and poking at a keyboard versus, you know, trying to write software or implement systems that fix 32 00:02:47,740 --> 00:02:53,260 themselves. All right. And it seems like, you know, from my perspective, the term SRE really took off. 33 00:02:53,300 --> 00:02:59,360 I think when Google published the book in about 2016, but I'm curious how far back, at least as it's 34 00:02:59,360 --> 00:03:02,600 currently practiced, right? How far back does that, does the, does the concept go? 35 00:03:03,560 --> 00:03:10,000 Uh, 2004, I think is really sort of the big bang moment. Uh, Ben Trainor Sloss is the, uh, VP for 24, 36 00:03:10,000 --> 00:03:16,280 seven operations here at Google. And, uh, we like, he's a bit of a Chuck Norris figure here at Google. 37 00:03:16,280 --> 00:03:22,220 You can, well, ironically enough, you can Google him and see why. Um, he coined the term in 2004. It's, 38 00:03:22,360 --> 00:03:28,040 it's hard to say exactly when it turned into what it is now, because it's been an evolution. 39 00:03:28,500 --> 00:03:32,040 You know, we like every other company on the planet in the early two thousands, we had systems 40 00:03:32,040 --> 00:03:38,520 administrators. Uh, and so slowly it was about evolving them from, uh, you know, log into a 41 00:03:38,520 --> 00:03:42,840 machine and type a bunch of commands and now do this a bunch of times because we have a lot of 42 00:03:42,840 --> 00:03:47,080 machines to manage, uh, to something that was more scalable with not as many humans. 43 00:03:47,360 --> 00:03:52,500 And how, how does SRE relate to DevOps? I mean, the story, the story I've heard of multiple times 44 00:03:52,500 --> 00:03:58,280 lately is like SRE is part of what you might call DevOps 2.0 or something, right? Which is like, 45 00:03:58,340 --> 00:04:04,920 it's, you know, this is just the silliest argument in the history. Well, maybe not in the history of 46 00:04:04,920 --> 00:04:12,520 tech, but certainly in modern tech. Okay. So here is the truth. SRE and DevOps developed mostly 47 00:04:12,520 --> 00:04:18,740 independently of one another, mostly at the same time in response to exactly the same set of 48 00:04:18,740 --> 00:04:26,380 problems. So unsurprisingly, they landed in really similar spaces. They share, you know, 99% of the 49 00:04:26,380 --> 00:04:31,340 same principles. So I don't, you know, I don't like to argue about like chronology and history and all 50 00:04:31,340 --> 00:04:38,500 that stuff. The way I like to think of it mentally is that SRE is a concrete opinionated, and it certainly 51 00:04:38,500 --> 00:04:45,740 is implementation of DevOps principles. Um, the thing I like about SRE is if you do SRE work 52 00:04:45,740 --> 00:04:51,480 at Google and then go work and then go to, I don't know, LinkedIn or Netflix or some other place with 53 00:04:51,480 --> 00:04:56,820 an SRE culture, you know, the, the activities will rhyme with one another. Like you will recognize 54 00:04:56,820 --> 00:05:02,880 the things. All right. Well, what's the, where's that 1% difference between SRE and, and DevOps? 55 00:05:03,880 --> 00:05:09,920 You know, there are like any, uh, any set of practices, uh, a set of sort of, um, theoretical 56 00:05:09,920 --> 00:05:14,380 or philosophical, or maybe even religious beliefs, uh, grow up around them. And, you know, you get 57 00:05:14,380 --> 00:05:19,700 little differences in how people want to implement things like, uh, you know, how, uh, what, what's the 58 00:05:19,700 --> 00:05:24,520 right default error budget policy is, I guess, a pretty good example. Oh, so I should maybe explain that, 59 00:05:24,520 --> 00:05:30,040 uh, for people. Sorry. Um, it just occurred to me, wait a minute. A lot of, a lot of folks may not 60 00:05:30,040 --> 00:05:33,720 know what that is. All right, let's try it this way. So, uh, I actually like to think of SRE as not 61 00:05:33,720 --> 00:05:39,260 a technical discipline. I like to think of it as, um, as a way to express humanity. And, and I know 62 00:05:39,260 --> 00:05:44,900 that sounds lofty, but here's the way I, I approach that. All right. The first thing to recognize 63 00:05:44,900 --> 00:05:52,380 is that a goal of no errors or a hundred percent, uh, reliability, because they're the same thing, 64 00:05:52,380 --> 00:05:59,060 uh, is not only not doable because there is no system in nature anywhere, uh, that has that 65 00:05:59,060 --> 00:06:04,980 property, but it is ultimately self-defeating and counterproductive. In fact, a goal of perfection 66 00:06:04,980 --> 00:06:10,220 will turn your employees into liars eventually like good, honest, hardworking people. You'll force 67 00:06:10,220 --> 00:06:15,160 them to lie. It's terrible. So you have to accept that whatever you do, there's going to be some level 68 00:06:15,160 --> 00:06:22,020 of error period because there is. And so the important question is how much error can you 69 00:06:22,020 --> 00:06:28,660 tolerate before it adversely affects your business or affects your users? And you want that number to 70 00:06:28,660 --> 00:06:32,740 be as high as possible, right? Because the closer you try to get to zero, the more expensive it is to 71 00:06:32,740 --> 00:06:38,580 maintain and the harder it is to achieve. So that amount of acceptable error in your system, 72 00:06:38,780 --> 00:06:43,300 right? And you want to pick a number that your users mostly don't notice that amount of acceptable 73 00:06:43,300 --> 00:06:49,720 error, we call the error budget. And then, so that the top of that bar, if you will, is, is perfection. 74 00:06:49,720 --> 00:06:56,320 And the bottom of that bar is what we call the service level objective. So I want to run my service 75 00:06:56,540 --> 00:07:06,120 just a teeny tiny bit better and have it consume just under the maximum, maximum amount of error. 76 00:07:06,260 --> 00:07:11,660 My users will basically not notice. All right. And so we call that the error budget. And so then the 77 00:07:11,660 --> 00:07:17,040 question happens, what do you do when from time to time you blow your error budget? And I should 78 00:07:17,040 --> 00:07:23,760 pause in and tell people there are actually two bad ways to blow an error budget. Obviously you can spend 79 00:07:23,760 --> 00:07:30,380 too much, meaning, you know, let's say, well, let's just use some math. Let's say that you have a system 80 00:07:30,380 --> 00:07:36,860 that you need where your SLO is 99.9% reliable, whatever reliable means in this context. It means 81 00:07:36,860 --> 00:07:43,700 you can have 43.2 bad minutes or that's your error budget every 30 days, right? So your system can be 82 00:07:43,700 --> 00:07:49,920 unreliable. However, we choose to measure that for 43 minutes every 30 days and still meet its three 83 00:07:49,920 --> 00:07:56,500 nines target. Okay. What happens though, when you blow that, when you're, you know, 50 minutes unreliable 84 00:07:56,500 --> 00:08:02,940 or 90 minutes or something in some 30 day period. Okay. That's an error budget policy, right? And there are 85 00:08:02,940 --> 00:08:08,520 two bad things that can happen. You can spend too much of the error budget. That's easy. You can also 86 00:08:08,520 --> 00:08:17,160 consistently spend too little. So if you determine that you can tolerate 43 minutes of error or bad minutes 87 00:08:17,160 --> 00:08:22,720 over 30 days and you're consistently only spending say 10 minutes, that's not a victory. That means 88 00:08:22,720 --> 00:08:27,500 you've over-engineered reliability. You, you have more reliability than your users need. And that's 89 00:08:27,500 --> 00:08:32,840 time and resource and expense. You could be applying to innovation or risk or some other thing. And so 90 00:08:32,840 --> 00:08:38,320 the question is, what do you do in that context? You know, how do you approach those things? And that's 91 00:08:38,320 --> 00:08:43,020 where you get to philosophical differences, you know, between companies and teams. You know, I, 92 00:08:43,180 --> 00:08:49,460 on, on my teams, I like to keep a pretty simple error budget policy. Um, if we blow our error budget, 93 00:08:49,740 --> 00:08:55,460 I want to feature freeze, right? No new features. We are going to spend the time it takes to earn back 94 00:08:55,460 --> 00:09:01,120 our good minutes doing reliability related things, but not every company has that, or every team has 95 00:09:01,120 --> 00:09:05,860 that as a default policy. So that's the kind of a thing that might be a little different going from 96 00:09:05,860 --> 00:09:11,460 company to company. All right. That's interesting. I think the, uh, yes, I've, I've heard that definition 97 00:09:11,460 --> 00:09:14,960 before too. And it's like, right. And the, you know, when you talk about humanity too, I think that 98 00:09:14,960 --> 00:09:18,900 there's this concept of like, well, okay. And, and your people are better and people are performing 99 00:09:18,900 --> 00:09:23,540 better, right. When you're not constantly striving for, for perfection and getting woken up and people 100 00:09:23,540 --> 00:09:26,520 when it's like, when people are healthier, your systems are going to be healthier. And I think that's, 101 00:09:26,600 --> 00:09:29,700 that's a, maybe a not natural way of thinking about it. 102 00:09:29,860 --> 00:09:33,880 Well, it's also, you know, I mean, for thousands of years, people have understood that, 103 00:09:33,880 --> 00:09:38,240 you know, Seneca said it, right? Irari humanimus said, and irari perseverari diabolicum, 104 00:09:38,280 --> 00:09:44,780 which is to err is human, but to persist in error is diabolical, you know, inhumane. What he meant 105 00:09:44,780 --> 00:09:50,140 by that is to make a well-intentioned mistake might actually be the best definition of what it is to be 106 00:09:50,140 --> 00:09:55,300 human. We make well-intentioned errors all the time, and that's okay. But to persist in a state of, 107 00:09:55,300 --> 00:09:59,740 to knowingly persist in a state of error, that is what is inhumane, uh, or is, 108 00:09:59,860 --> 00:10:05,560 um, when I, when I was interviewing at Google, uh, low these many years ago, actually it was Ben 109 00:10:05,560 --> 00:10:10,020 trainer who was interviewing me. He said a thing I'll never forget. He said, the great sin is not 110 00:10:10,020 --> 00:10:16,680 to fail. The sin isn't failing to notice. And I always thought that was a great way to, to live, 111 00:10:16,740 --> 00:10:20,820 right? Take the risk. That's fine. But just notice really quickly when things are going back 112 00:10:20,820 --> 00:10:26,000 badly and have a plan to mitigate and move on. That's a, that's a much more humane way to design 113 00:10:26,000 --> 00:10:30,820 and run things. All right. And so, so if I'm an enterprise and I'm looking to get into SRE, 114 00:10:30,820 --> 00:10:35,800 get, get into SRE practices and do it right. What, what do I need to be looking at beyond 115 00:10:35,800 --> 00:10:40,080 traditional metrics, right? I think they're, you know, historically it's a compute memory, 116 00:10:40,080 --> 00:10:48,700 right? Yeah. Those are, I mean, those are, uh, well, your users, our users, no one, no one's 117 00:10:48,700 --> 00:10:53,620 users care about CPU load or memory pressure or disk fullness, right? You know, they don't care 118 00:10:53,620 --> 00:10:59,220 about that stuff. They care about how long did the thing I want to take and did I get the correct 119 00:10:59,220 --> 00:11:05,180 answer, you know, or did the server throw an error or some other thing? Um, you want to measure the 120 00:11:05,180 --> 00:11:12,060 things your users carry about, excuse me, care about, um, we like to say, you know, the important 121 00:11:12,060 --> 00:11:15,680 things are the symptoms, not the causes. I mean, the causes are important because you need the data 122 00:11:15,680 --> 00:11:20,280 to be able to debug and fix the thing, but your users care about the symptoms. Uh, and so those 123 00:11:20,280 --> 00:11:24,420 are the things not only that you want to measure, but you want to learn on. Like if you have a pager 124 00:11:24,420 --> 00:11:30,420 set up, uh, to ring when your CPU spikes, you're doing it wrong. You, what you want to do is you want 125 00:11:30,420 --> 00:11:34,580 to be measuring how long is some, uh, you know, if it's a web application, how long is some request 126 00:11:34,580 --> 00:11:38,760 taking? And when it crosses some threshold that you think gives a bad user experience, 127 00:11:38,760 --> 00:11:42,920 that's, or it's starting to creep towards that threshold. That's when you want to page humans 128 00:11:42,920 --> 00:11:48,120 or take some corrective action. So that's the first thing is, uh, you know, I spent a lot of time 129 00:11:48,120 --> 00:11:55,840 working with large companies, uh, to, you know, onboard these principles and use them. And you know, 130 00:11:55,840 --> 00:12:01,760 the, an interesting thing is we find all the time that we say, okay, we want to learn on the symptoms, 131 00:12:01,760 --> 00:12:07,300 not on the causes. And that's, that's, we want to ring the pagers. Neat. So we go through and we 132 00:12:07,300 --> 00:12:10,580 help companies figure out what should their SLOs be for these various systems and their error budgets 133 00:12:10,580 --> 00:12:14,240 and so forth and so on. Okay. And then we go look at their monitoring systems and their alerting 134 00:12:14,240 --> 00:12:19,420 systems. And we ask, well, what are you actually measuring? And we find that, you know, a bunch of 135 00:12:19,420 --> 00:12:23,760 the SLOs, there actually aren't things being monitored that would tell us. So they're kind of blind in 136 00:12:23,760 --> 00:12:29,620 that way. But then we also find that they have alerts set up for sometimes hundreds of things 137 00:12:29,620 --> 00:12:36,760 that don't move the SLOs at all, or, or aren't the symptoms, but are the causes. And so very often 138 00:12:36,760 --> 00:12:42,160 when we talk to customers and help them through this process, they went up turning off like half of 139 00:12:42,160 --> 00:12:48,180 their pages in the first month because they actually don't do anything except, you know, cause interrupts 140 00:12:48,180 --> 00:12:55,280 for people. All right. Cool. And how do these broader, some broader tech initiatives play into 141 00:12:55,280 --> 00:13:00,140 this? Right. I think, I think of, you know, cloud computing containers, microservices, CICD, 142 00:13:00,260 --> 00:13:04,760 whatever. I mean, are these things required, do you think to achieve maximum success? Or is it really 143 00:13:04,760 --> 00:13:10,920 like, Hey, what's your application? And yeah, no, they're absolutely not. First of all, most of these 144 00:13:10,920 --> 00:13:17,860 things didn't exist when we were, when we started this. So that, um, no, SRE is a set of principles, 145 00:13:17,860 --> 00:13:22,380 not a technology. Uh, there's a set of principles and you apply those principles. And as long as you 146 00:13:22,380 --> 00:13:27,540 apply them consistently, uh, in, in a, not to be too redundant in a sort of a principled way, 147 00:13:27,540 --> 00:13:32,280 you get the outcome you want independent of the technology you're using. I mean, 148 00:13:32,920 --> 00:13:39,440 we have applied SRE principles, uh, to, you know, to legal teams and finance teams and, you know, 149 00:13:39,460 --> 00:13:44,240 for marketing risk and like principles of error budgets and error budget policies and whatever, 150 00:13:44,320 --> 00:13:47,980 those are applicable in almost all business domains. Actually they're applicable in all 151 00:13:47,980 --> 00:13:52,200 business domains as I think about it. Uh, and they're not particularly technology sensitive. 152 00:13:52,420 --> 00:13:56,800 Yeah. Some technology tools might make it easier to implement and enforce the principles. 153 00:13:57,140 --> 00:14:02,440 That's true, but it wouldn't be reasonable to say, Oh, this is only reasonable if you're on a cloud 154 00:14:02,440 --> 00:14:08,400 or if you're using CICD or microservices or blah, blah, blah. Like those, that's apples and sofas. 155 00:14:08,500 --> 00:14:14,100 They're not connected. All right. Makes sense. And how does SRE evolve? I mean, we touched on this 156 00:14:14,100 --> 00:14:18,860 earlier, I think, but the developer, the sysadmin and these other roles that it touches, like, is it, 157 00:14:18,980 --> 00:14:23,680 is it a new position as SRE engineer? And I've seen it actually, right? So I know it has like a new 158 00:14:23,680 --> 00:14:28,440 position to some degree, but is it? Yeah. I, I kind of think that's a mistake a little bit. I mean, 159 00:14:28,440 --> 00:14:31,960 you'll get disagreement from people. Um, the truth is, is at Google, 160 00:14:31,960 --> 00:14:38,860 it was the sysadmonds who became the first SREs, right? I like to say that, um, when, 161 00:14:38,960 --> 00:14:44,760 when I'm talking to a candidate, let's say about whether or not maybe they want to join SRE. 162 00:14:45,340 --> 00:14:50,480 One of the first questions I asked the person is when was the last time you got so frustrated 163 00:14:50,480 --> 00:14:54,000 with something that you said, I'm going to automate this no matter how long it takes. 164 00:14:54,340 --> 00:14:58,460 And then how many times did you have to do the thing before you reached your frustration level? 165 00:14:58,980 --> 00:15:03,700 And what I'm looking for is like, you know, two weeks or less, you know, since the last time that 166 00:15:03,700 --> 00:15:09,440 happened and, you know, maybe three or four times or less that I had to do it. Right. I'm looking for 167 00:15:09,440 --> 00:15:18,560 people who just cannot stand repetitive manual toily kind of things that are just that it just bothers 168 00:15:18,560 --> 00:15:22,940 them. It pushes some button emotionally for them. And they're just driven to automate that away. 169 00:15:23,500 --> 00:15:28,580 That's really the secret to all this. So you can take a sysadmin who's been running the same shell 170 00:15:28,580 --> 00:15:34,420 script or typing the same RPM command or whatever. And as long as they have a passion and are willing 171 00:15:34,420 --> 00:15:40,220 to learn how to automate those routine tasks away, they can become really excellent SREs. In fact, 172 00:15:40,720 --> 00:15:44,120 some of the best SREs I know started in traditional system administration roles. 173 00:15:45,560 --> 00:15:49,040 So I don't think of it as a whole cloth new position. And then of course, the other way to do 174 00:15:49,040 --> 00:15:55,660 it is you find software engineers, you know, people who already like to write code, who are really 175 00:15:55,660 --> 00:16:01,480 interested in how big systems run in production and how they fail and how to correct. And, you know, 176 00:16:01,540 --> 00:16:07,500 those are, those are people who, we all know these people who will spend a week optimizing to get 2% 177 00:16:07,500 --> 00:16:12,040 more out of a system or something. Right. That's the other kind of class of people who become really 178 00:16:12,040 --> 00:16:16,920 good SREs. Is there an inflection point, you think, at which companies are really going to start to see 179 00:16:16,920 --> 00:16:21,920 value with SRE? I mean, I would imagine a five person startup versus, I don't know, Verizon or 180 00:16:21,920 --> 00:16:29,540 something is a pretty big difference, right? Well, so I don't think of it that way. To me, SRE is a, 181 00:16:31,040 --> 00:16:36,320 the SRE principles are applicable from, you know, your first employee to your hundred thousandth. 182 00:16:37,500 --> 00:16:42,360 So measuring only the things that your users care about and not the other things is a good idea, 183 00:16:42,500 --> 00:16:49,760 whether you're a cash starved startup or a time starved enterprise. And so no, actually the 184 00:16:49,760 --> 00:16:54,560 principles are applicable. Now, how you choose to implement them, how you choose to organize your 185 00:16:54,560 --> 00:16:59,440 teams, those are absolutely going to evolve as you get bigger, but no, the principles aren't just 186 00:16:59,440 --> 00:17:03,000 horizontally scalable, you know, across the different business functions, but they really are 187 00:17:03,000 --> 00:17:07,520 very vertically scalable. If I were starting a new company from scratch, we would be doing 188 00:17:07,520 --> 00:17:14,520 error budgets and SLOs from day zero. Because candidly, reliability is, gets harder to bolt 189 00:17:14,520 --> 00:17:19,340 on afterwards. It's much better if you start. It's a, it's a lot like security or multi-tenancy 190 00:17:19,340 --> 00:17:21,120 or something. All right. You need to start at the beginning. 191 00:17:21,840 --> 00:17:25,240 That makes sense. Yeah. I guess what I was thinking was just the, the complexity of, I mean, 192 00:17:25,240 --> 00:17:29,980 the number of things. It doesn't, yeah, it doesn't have to be. Yeah. Right. But it doesn't have to be. 193 00:17:30,040 --> 00:17:35,080 That's the thing is you can keep your implementation really simple. The principles will scale that 194 00:17:35,080 --> 00:17:41,820 way. All right. And then ultimately, David, I'm curious, like, and, and, and, you know, and feel 195 00:17:41,820 --> 00:17:46,460 free to ask me if this doesn't make sense and I'll try to rephrase, but should organ, and I think I 196 00:17:46,460 --> 00:17:51,720 get a hint of this, should organizations look at SRE as like a technology practice or as it sounds 197 00:17:51,720 --> 00:17:55,760 almost like a business practice. Yeah. That's the way I think of it. And again, 198 00:17:55,760 --> 00:18:00,240 I think this is an area where you could get sort of theological disagreement among individual SREs. 199 00:18:00,280 --> 00:18:05,320 So this is just David K. Renson's opinion, but no, I think of SRE, I think strongly of SRE as a 200 00:18:05,320 --> 00:18:10,280 business practice that evolved in a technical culture. That's the way I like to think of it because it is 201 00:18:10,280 --> 00:18:16,640 so applicable to even what we would not think of as, as very technical things or more non-technical 202 00:18:16,640 --> 00:18:21,560 domains. All right. All right. And I mean, have you seen that, that I'm curious, 203 00:18:21,560 --> 00:18:25,920 like in your work at Google as, as you've worked with, with customers on this or even internally, 204 00:18:25,920 --> 00:18:30,020 like how you mentioned some of it at Google, but how, how have you seen it spread? I guess, 205 00:18:30,020 --> 00:18:33,240 or it seems like what kind of, maybe if this is bigger idea of, we always talk about culture 206 00:18:33,240 --> 00:18:35,780 change, right? If we're going to modernize, we're going to change the culture. 207 00:18:36,860 --> 00:18:40,440 Yeah. I actually, that's been one of the things I think has been really awesome in the last 208 00:18:40,440 --> 00:18:46,180 three or four years are the number of companies that have started doing SRE kind of their own 209 00:18:46,180 --> 00:18:51,360 way. Like, so we, we shipped the SRE book in 2016 and that was awesome. And then the most 210 00:18:51,400 --> 00:18:56,240 consistent piece of feedback we heard from the book is these are a great set of principles, 211 00:18:56,240 --> 00:19:01,800 but I'm not sure how to apply it in my company, you know, X. And so we shipped the follow-up book, 212 00:19:01,800 --> 00:19:08,320 the workbook, uh, in 2018, uh, to answer that, like, okay, here's some specific implemented, 213 00:19:08,760 --> 00:19:14,880 implementation details that you can use. Uh, and then we, we, and then what we found is there 214 00:19:14,880 --> 00:19:19,640 were a lot of companies, you know, old blind companies like the New York times or the home Depot 215 00:19:19,640 --> 00:19:24,560 and, you know, sort of cloud natives like Evernote. I mean, not just the big tech companies, 216 00:19:24,560 --> 00:19:29,280 you know, us or LinkedIn or whoever. Um, but a bunch of companies in a bunch of different sectors, 217 00:19:29,280 --> 00:19:33,860 uh, who started applying these things and going on their own, their own sort of journey. And that's 218 00:19:33,860 --> 00:19:39,080 been, that's been really awesome to watch. I'm a believer that you can't change culture from the 219 00:19:39,080 --> 00:19:43,480 top. You can't mandate things from the top. It has to be grassroots. And so you give people a, 220 00:19:43,640 --> 00:19:48,760 a set of easy to follow principles or at least easy to understand principles and autonomy to go 221 00:19:48,760 --> 00:19:52,880 implement them. And before you know it, you turn around and your culture is completely different. 222 00:19:53,100 --> 00:19:57,320 And, and we are seeing that like the home Depot is a great case. They talk a lot publicly about what 223 00:19:57,320 --> 00:20:01,020 happened to their culture, uh, in a positive way when they went on the rest of your journey and it 224 00:20:01,020 --> 00:20:06,060 didn't take long. It took them like a year, 18 months. All right. All right. Cool. With that, 225 00:20:06,160 --> 00:20:10,180 I think we're out of time, Dave. Thanks a lot. Derek did entirely. My pleasure. Always lovely to talk to 226 00:20:10,180 --> 00:20:15,720 you. Okay. And there you have it. I hope you learned something. I know I did, uh, for more 227 00:20:15,720 --> 00:20:21,820 information about SRE, specifically Google's approach to SRE, you can check out google.com slash SRE for, 228 00:20:21,820 --> 00:20:28,600 for lots of resources. And also you can just Google SRE and I'm sure you'll find a plethora of stuff 229 00:20:28,600 --> 00:20:33,460 that'll help you out. Uh, for more information about our Pivotal's broad suite of products, 230 00:20:33,500 --> 00:20:37,080 including a lot of, you know, we have a lot of information. We've published a lot of talks and 231 00:20:37,080 --> 00:20:41,160 papers and stuff in SRE because it's critical that we do things and how we'd like our clients to do 232 00:20:41,160 --> 00:20:46,400 things. Uh, check it, check us out at pivotal.io. And if you like this podcast and want to keep up 233 00:20:46,400 --> 00:20:52,480 with more of this, let's call it a thought leadershipy content, uh, about, about SRE and 234 00:20:52,480 --> 00:20:56,580 modern applications and digital transformation, whatever you want to call it, check out the, the website 235 00:20:56,580 --> 00:21:02,320 that, that, where this podcast actually lives, which is pivotal.io slash intersect. It's called 236 00:21:02,320 --> 00:21:06,320 intersect with the idea of being the intersection of business and technology. So at any rate, 237 00:21:07,020 --> 00:21:10,900 check us out there at pivotal.io slash intersect. Thanks for listening.