Welcome to the Cloud Native in 15 Minutes podcast. The goal of this podcast is simple, to explain some complex but increasingly important and even strategic technologies and about the time it takes for a coffee break. So I'm Derek Harris from Pivotal and our topic today is SRE or Site Reliability Engineering. Joining me to explain SRE is Dave Rensen, a Senior Director of Engineering at Google and one of its, you know, for the past several years now, one of its public faces and primary spokespeople and engineers on the SRE team. So this, I think this is a really pretty interesting discussion if you're not familiar with SRE. Dave does a great job talking about it, not just as a technical problem, but as a business problem, as a problem that's designed to, you know, I think the way he puts it is make machines answer to humans, not humans answer to machines. It's really an interesting philosophical discussion and it touches on, you know, using SRE principles and everything. Yes, from reliability for your application, but also down to reliability for your marketing team. What's your error budget for mistakes and finance or whatever. So it's, I think it's a really interesting discussion. I think you'll enjoy it. So stick around and listen to Dave Rensen of Google talk about SRE. All right, Dave. Listen, thanks for being on to kick off. Can you talk listeners through what SRE stands for? I mean, obviously it stands for site reliability engineering, but beyond that, I mean, what it really, what it really means. Oh yeah, sure. So the, the SRE team here at Google, it's pretty easy. We're responsible for keeping all of our production systems running 24 seven. It's all the software systems, hardware systems, data centers, et cetera. We like to say that SRE is what you get when you treat operations like a software problem. All right. Can you flesh that out a bit? I mean, what is that? I've heard that described and I think it makes a lot of sense. If you, if you, if you know, if you know the kind of some context with that, but can you walk through a little bit? The way I like to think of it as this, you can live in one of two worlds. Um, in the first world, a machine called a
2:07pager wakes you up at three in the morning because some other machine is having a hard time. In that world, you work for the computers. The world you want to live in is one where some system you're responsible for is having a problem. It sort of mitigates itself. And then it writes a bunch of information out for you to debug the next morning after your morning coffee. That's a world where the machines work for you. SRE is about getting the machines to work for you. It's about performing all the operational things we have to do to keep our users happy, but getting the computers to perform as many of them is humanly possible. And that's the difference between sort of staring at a monitor and poking at a keyboard versus, you know, trying to write software or implement systems that fix themselves. All right. And it seems like, you know, from my perspective, the term SRE really took off. I think when Google published the book in about 2016, but I'm curious how far back, at least as it's currently practiced, right? How far back does that, does the, does the concept go? Uh, 2004, I think is really sort of the big bang moment. Uh, Ben Trainor Sloss is the, uh, VP for 24, seven operations here at Google. And, uh, we like, he's a bit of a Chuck Norris figure here at Google. You can, well, ironically enough, you can Google him and see why. Um, he coined the term in 2004. It's, it's hard to say exactly when it turned into what it is now, because it's been an evolution. You know, we like every other company on the planet in the early two thousands, we had systems administrators. Uh, and so slowly it was about evolving them from, uh, you know, log into a machine and type a bunch of commands and now do this a bunch of times because we have a lot of machines to manage, uh, to something that was more scalable with not as many humans. And how, how does SRE relate to DevOps? I mean, the story, the story I've heard of multiple times lately is like SRE is part of what you might call DevOps 2.0 or something, right? Which is like, it's, you know, this is just the silliest argument in the history. Well, maybe not in the history of tech, but certainly in modern tech. Okay. So here is the truth. SRE and DevOps developed mostly
4:12independently of one another, mostly at the same time in response to exactly the same set of problems. So unsurprisingly, they landed in really similar spaces. They share, you know, 99% of the same principles. So I don't, you know, I don't like to argue about like chronology and history and all that stuff. The way I like to think of it mentally is that SRE is a concrete opinionated, and it certainly is implementation of DevOps principles. Um, the thing I like about SRE is if you do SRE work at Google and then go work and then go to, I don't know, LinkedIn or Netflix or some other place with an SRE culture, you know, the, the activities will rhyme with one another. Like you will recognize the things. All right. Well, what's the, where's that 1% difference between SRE and, and DevOps? You know, there are like any, uh, any set of practices, uh, a set of sort of, um, theoretical or philosophical, or maybe even religious beliefs, uh, grow up around them. And, you know, you get little differences in how people want to implement things like, uh, you know, how, uh, what, what's the right default error budget policy is, I guess, a pretty good example. Oh, so I should maybe explain that, uh, for people. Sorry. Um, it just occurred to me, wait a minute. A lot of, a lot of folks may not know what that is. All right, let's try it this way. So, uh, I actually like to think of SRE as not a technical discipline. I like to think of it as, um, as a way to express humanity. And, and I know that sounds lofty, but here's the way I, I approach that. All right. The first thing to recognize is that a goal of no errors or a hundred percent, uh, reliability, because they're the same thing, uh, is not only not doable because there is no system in nature anywhere, uh, that has that property, but it is ultimately self-defeating and counterproductive. In fact, a goal of perfection will turn your employees into liars eventually like good, honest, hardworking people. You'll force them to lie. It's terrible. So you have to accept that whatever you do, there's going to be some level
6:15of error period because there is. And so the important question is how much error can you tolerate before it adversely affects your business or affects your users? And you want that number to be as high as possible, right? Because the closer you try to get to zero, the more expensive it is to maintain and the harder it is to achieve. So that amount of acceptable error in your system, right? And you want to pick a number that your users mostly don't notice that amount of acceptable error, we call the error budget. And then, so that the top of that bar, if you will, is, is perfection. And the bottom of that bar is what we call the service level objective. So I want to run my service just a teeny tiny bit better and have it consume just under the maximum, maximum amount of error. My users will basically not notice. All right. And so we call that the error budget. And so then the question happens, what do you do when from time to time you blow your error budget? And I should pause in and tell people there are actually two bad ways to blow an error budget. Obviously you can spend too much, meaning, you know, let's say, well, let's just use some math. Let's say that you have a system that you need where your SLO is 99.9% reliable, whatever reliable means in this context. It means you can have 43.2 bad minutes or that's your error budget every 30 days, right? So your system can be unreliable. However, we choose to measure that for 43 minutes every 30 days and still meet its three nines target. Okay. What happens though, when you blow that, when you're, you know, 50 minutes unreliable or 90 minutes or something in some 30 day period. Okay. That's an error budget policy, right? And there are two bad things that can happen. You can spend too much of the error budget. That's easy. You can also consistently spend too little. So if you determine that you can tolerate 43 minutes of error or bad minutes
8:17over 30 days and you're consistently only spending say 10 minutes, that's not a victory. That means you've over-engineered reliability. You, you have more reliability than your users need. And that's time and resource and expense. You could be applying to innovation or risk or some other thing. And so the question is, what do you do in that context? You know, how do you approach those things? And that's where you get to philosophical differences, you know, between companies and teams. You know, I, on, on my teams, I like to keep a pretty simple error budget policy. Um, if we blow our error budget, I want to feature freeze, right? No new features. We are going to spend the time it takes to earn back our good minutes doing reliability related things, but not every company has that, or every team has that as a default policy. So that's the kind of a thing that might be a little different going from company to company. All right. That's interesting. I think the, uh, yes, I've, I've heard that definition before too. And it's like, right. And the, you know, when you talk about humanity too, I think that there's this concept of like, well, okay. And, and your people are better and people are performing better, right. When you're not constantly striving for, for perfection and getting woken up and people when it's like, when people are healthier, your systems are going to be healthier. And I think that's, that's a, maybe a not natural way of thinking about it. Well, it's also, you know, I mean, for thousands of years, people have understood that, you know, Seneca said it, right? Irari humanimus said, and irari perseverari diabolicum, which is to err is human, but to persist in error is diabolical, you know, inhumane. What he meant by that is to make a well-intentioned mistake might actually be the best definition of what it is to be human. We make well-intentioned errors all the time, and that's okay. But to persist in a state of, to knowingly persist in a state of error, that is what is inhumane, uh, or is, um, when I, when I was interviewing at Google, uh, low these many years ago, actually it was Ben trainer who was interviewing me. He said a thing I'll never forget. He said, the great sin is not to fail. The sin isn't failing to notice. And I always thought that was a great way to, to live, right? Take the risk. That's fine. But just notice really quickly when things are going back
10:20badly and have a plan to mitigate and move on. That's a, that's a much more humane way to design and run things. All right. And so, so if I'm an enterprise and I'm looking to get into SRE, get, get into SRE practices and do it right. What, what do I need to be looking at beyond traditional metrics, right? I think they're, you know, historically it's a compute memory, right? Yeah. Those are, I mean, those are, uh, well, your users, our users, no one, no one's users care about CPU load or memory pressure or disk fullness, right? You know, they don't care about that stuff. They care about how long did the thing I want to take and did I get the correct answer, you know, or did the server throw an error or some other thing? Um, you want to measure the things your users carry about, excuse me, care about, um, we like to say, you know, the important things are the symptoms, not the causes. I mean, the causes are important because you need the data to be able to debug and fix the thing, but your users care about the symptoms. Uh, and so those are the things not only that you want to measure, but you want to learn on. Like if you have a pager set up, uh, to ring when your CPU spikes, you're doing it wrong. You, what you want to do is you want to be measuring how long is some, uh, you know, if it's a web application, how long is some request taking? And when it crosses some threshold that you think gives a bad user experience, that's, or it's starting to creep towards that threshold. That's when you want to page humans or take some corrective action. So that's the first thing is, uh, you know, I spent a lot of time working with large companies, uh, to, you know, onboard these principles and use them. And you know, the, an interesting thing is we find all the time that we say, okay, we want to learn on the symptoms, not on the causes. And that's, that's, we want to ring the pagers. Neat. So we go through and we help companies figure out what should their SLOs be for these various systems and their error budgets and so forth and so on. Okay. And then we go look at their monitoring systems and their alerting systems. And we ask, well, what are you actually measuring? And we find that, you know, a bunch of the SLOs, there actually aren't things being monitored that would tell us. So they're kind of blind in
12:23that way. But then we also find that they have alerts set up for sometimes hundreds of things that don't move the SLOs at all, or, or aren't the symptoms, but are the causes. And so very often when we talk to customers and help them through this process, they went up turning off like half of their pages in the first month because they actually don't do anything except, you know, cause interrupts for people. All right. Cool. And how do these broader, some broader tech initiatives play into this? Right. I think, I think of, you know, cloud computing containers, microservices, CICD, whatever. I mean, are these things required, do you think to achieve maximum success? Or is it really like, Hey, what's your application? And yeah, no, they're absolutely not. First of all, most of these things didn't exist when we were, when we started this. So that, um, no, SRE is a set of principles, not a technology. Uh, there's a set of principles and you apply those principles. And as long as you apply them consistently, uh, in, in a, not to be too redundant in a sort of a principled way, you get the outcome you want independent of the technology you're using. I mean, we have applied SRE principles, uh, to, you know, to legal teams and finance teams and, you know, for marketing risk and like principles of error budgets and error budget policies and whatever, those are applicable in almost all business domains. Actually they're applicable in all business domains as I think about it. Uh, and they're not particularly technology sensitive. Yeah. Some technology tools might make it easier to implement and enforce the principles. That's true, but it wouldn't be reasonable to say, Oh, this is only reasonable if you're on a cloud or if you're using CICD or microservices or blah, blah, blah. Like those, that's apples and sofas. They're not connected. All right. Makes sense. And how does SRE evolve? I mean, we touched on this earlier, I think, but the developer, the sysadmin and these other roles that it touches, like, is it, is it a new position as SRE engineer? And I've seen it actually, right? So I know it has like a new
14:23position to some degree, but is it? Yeah. I, I kind of think that's a mistake a little bit. I mean, you'll get disagreement from people. Um, the truth is, is at Google, it was the sysadmonds who became the first SREs, right? I like to say that, um, when, when I'm talking to a candidate, let's say about whether or not maybe they want to join SRE. One of the first questions I asked the person is when was the last time you got so frustrated with something that you said, I'm going to automate this no matter how long it takes. And then how many times did you have to do the thing before you reached your frustration level? And what I'm looking for is like, you know, two weeks or less, you know, since the last time that happened and, you know, maybe three or four times or less that I had to do it. Right. I'm looking for people who just cannot stand repetitive manual toily kind of things that are just that it just bothers them. It pushes some button emotionally for them. And they're just driven to automate that away. That's really the secret to all this. So you can take a sysadmin who's been running the same shell script or typing the same RPM command or whatever. And as long as they have a passion and are willing to learn how to automate those routine tasks away, they can become really excellent SREs. In fact, some of the best SREs I know started in traditional system administration roles. So I don't think of it as a whole cloth new position. And then of course, the other way to do it is you find software engineers, you know, people who already like to write code, who are really interested in how big systems run in production and how they fail and how to correct. And, you know, those are, those are people who, we all know these people who will spend a week optimizing to get 2% more out of a system or something. Right. That's the other kind of class of people who become really good SREs. Is there an inflection point, you think, at which companies are really going to start to see value with SRE? I mean, I would imagine a five person startup versus, I don't know, Verizon or something is a pretty big difference, right? Well, so I don't think of it that way. To me, SRE is a,
16:31the SRE principles are applicable from, you know, your first employee to your hundred thousandth. So measuring only the things that your users care about and not the other things is a good idea, whether you're a cash starved startup or a time starved enterprise. And so no, actually the principles are applicable. Now, how you choose to implement them, how you choose to organize your teams, those are absolutely going to evolve as you get bigger, but no, the principles aren't just horizontally scalable, you know, across the different business functions, but they really are very vertically scalable. If I were starting a new company from scratch, we would be doing error budgets and SLOs from day zero. Because candidly, reliability is, gets harder to bolt on afterwards. It's much better if you start. It's a, it's a lot like security or multi-tenancy or something. All right. You need to start at the beginning. That makes sense. Yeah. I guess what I was thinking was just the, the complexity of, I mean, the number of things. It doesn't, yeah, it doesn't have to be. Yeah. Right. But it doesn't have to be. That's the thing is you can keep your implementation really simple. The principles will scale that way. All right. And then ultimately, David, I'm curious, like, and, and, and, you know, and feel free to ask me if this doesn't make sense and I'll try to rephrase, but should organ, and I think I get a hint of this, should organizations look at SRE as like a technology practice or as it sounds almost like a business practice. Yeah. That's the way I think of it. And again, I think this is an area where you could get sort of theological disagreement among individual SREs. So this is just David K. Renson's opinion, but no, I think of SRE, I think strongly of SRE as a business practice that evolved in a technical culture. That's the way I like to think of it because it is so applicable to even what we would not think of as, as very technical things or more non-technical domains. All right. All right. And I mean, have you seen that, that I'm curious, like in your work at Google as, as you've worked with, with customers on this or even internally, like how you mentioned some of it at Google, but how, how have you seen it spread? I guess, or it seems like what kind of, maybe if this is bigger idea of, we always talk about culture
18:33change, right? If we're going to modernize, we're going to change the culture. Yeah. I actually, that's been one of the things I think has been really awesome in the last three or four years are the number of companies that have started doing SRE kind of their own way. Like, so we, we shipped the SRE book in 2016 and that was awesome. And then the most consistent piece of feedback we heard from the book is these are a great set of principles, but I'm not sure how to apply it in my company, you know, X. And so we shipped the follow-up book, the workbook, uh, in 2018, uh, to answer that, like, okay, here's some specific implemented, implementation details that you can use. Uh, and then we, we, and then what we found is there were a lot of companies, you know, old blind companies like the New York times or the home Depot and, you know, sort of cloud natives like Evernote. I mean, not just the big tech companies, you know, us or LinkedIn or whoever. Um, but a bunch of companies in a bunch of different sectors, uh, who started applying these things and going on their own, their own sort of journey. And that's been, that's been really awesome to watch. I'm a believer that you can't change culture from the top. You can't mandate things from the top. It has to be grassroots. And so you give people a, a set of easy to follow principles or at least easy to understand principles and autonomy to go implement them. And before you know it, you turn around and your culture is completely different. And, and we are seeing that like the home Depot is a great case. They talk a lot publicly about what happened to their culture, uh, in a positive way when they went on the rest of your journey and it didn't take long. It took them like a year, 18 months. All right. All right. Cool. With that, I think we're out of time, Dave. Thanks a lot. Derek did entirely. My pleasure. Always lovely to talk to you. Okay. And there you have it. I hope you learned something. I know I did, uh, for more information about SRE, specifically Google's approach to SRE, you can check out google.com slash SRE for, for lots of resources. And also you can just Google SRE and I'm sure you'll find a plethora of stuff that'll help you out. Uh, for more information about our Pivotal's broad suite of products,
20:33including a lot of, you know, we have a lot of information. We've published a lot of talks and papers and stuff in SRE because it's critical that we do things and how we'd like our clients to do things. Uh, check it, check us out at pivotal.io. And if you like this podcast and want to keep up with more of this, let's call it a thought leadershipy content, uh, about, about SRE and modern applications and digital transformation, whatever you want to call it, check out the, the website that, that, where this podcast actually lives, which is pivotal.io slash intersect. It's called intersect with the idea of being the intersection of business and technology. So at any rate, check us out there at pivotal.io slash intersect. Thanks for listening.