The Google SRE Book With Andrew Shafer (Ep. 58)

Episode 46 · 57:08 · SRT · archive.org · all Tanzu Talk

Well, not we're. I'm recording from Paris. I assume you're still in the desert today? Tucson, Arizona. Well, anyhow, I'm in Paris. We've got a bunch of people here for DevOps, which will be fun. There's only a few episodes that you were on, and you were on most recently. And we were talking about Pivotal's involvement in the Google CRE program, which was good. It was nice to see people enjoyed that episode, and we had some good after conversation about it. But one of the topics that came up is that we kind of had to discuss in talking about CRE was about their SRE, their Site Reliability Engineer thing. And we mentioned that there's a book out about it, and so we thought we should get back together to talk about that in more detail. Now, there's a whole – what makes it possible to talk about this is there's a whole book about it now, which, as you pointed out last time, you can read for free. So I thought it would be good to just kind of discuss that book because it is – I can't say I've read through all of it, but I've read through a large amount of it. And it is interesting and sort of – to some extent, if you're into all this stuff, sort of fascinating to see how people think and put things together. Also, it's always delightful when people use like an uber-academic way of doing citations with the little square brackets and the first three letters of the name and the last two numbers of a year. You don't spend all that time in grad school for nothing. Exactly. And so first of all, I mean, let me just start off with a broad topic because you and I have talked about this a lot. But like what is this book? Like it's ostensibly – my first take on this book, which now is, I would say, is a fourth accurate, is sort of like this is a manual literally for if you get hired to be a Google SRE. And maybe not even like a hands-on-keys manual. There's a few extra things. But it is sort of like – and if you read the book, you realize that they don't have this attitude. But it would kind of be like, welcome to the job. And they would throw you this book and be like, read this, kid. I mean, there's a lot more to it going on than just like a written-down thing that's proprietary to Google. So I wonder how like you would describe what the deal with this book is. The framing last time, we kind of had the minimal viable SRE talk so we could talk a little bit about CRE.

And as a consequence, I had a number of great conversations and I even had some people from Google kind of reach out to me. And I wanted to do a more proper book review, if you will, or really focus on some of the stuff that's available. And I do think that there's some of the things you said are true. It's definitely Google-specific. I wouldn't quite go as far as a manual. But in some ways, it's also kind of got value as this anthropological work of a culture and a practice inside of – really, in some ways, if you're actually paying attention to what Google's done over the last two decades, it's kind of one of the wonders of the world that Google even exists. And to kind of see them open up and put out this level of detail about how they think about this problem and how they solved it, I think that's – I keep saying it's a gift to the world, but it's certainly there for free for anyone who wants to pick it up and read it. Yeah, I mean, like pretty typical good engineers, they're actually pretty modest about the scale of things. But there is one point to kind of like pin it down where they're like, this is perhaps probably one of the largest systems ever built by humans. And the wording is not exactly like that, but I mean, they have a point. Like it's the whole of Google and all the systems behind it and applications on top of it and a word they would not really ever use. But the governance, the norms around it are an incredibly large, complicated system. I also think that there's some general themes that emerge as you try to build these systems. And I've never been behind the veil and worked at Google, but the types of processes, the types of tools, the types of approaches that you see emerge in kind of all the big web companies have some similarities. And I think this is just Google kind of putting the Google stamp on there.

And this is something that got great response when I had this discussion with the actual SREs at Google Next. This is Google's DevOps, right? This is DevOps as she has spoken at Google. No, that's a good point, too. It is. And, you know, it's an interesting mental exercise to sort of compare and contrast DevOps because that's, I mean, to rat hole on one specific thing, I still haven't kind of reconciled it in my head. But there's an interesting discussion to be had about the difference between the SRE teams. Or not difference, but the relationship between the SRE teams and the product teams that's described. And it's not quite as strident as sort of like the notion of DevOps where all these roles are on one team. But then there is a certain amount of what almost feels like learned fluidity between the two teams, which is, I mean, I'm digging deep onto something without explaining the context for it. But the notion is you have an SRE who basically is responsible for, I don't know, I mean, tell me what you think of this. Like, it's defined pretty specifically most times. But the way I file it away is the site reliability engineer is responsible for the R, right? Keeping it up and reliable. But to a certain extent, and I don't know where this line is, they're also responsible for building and caretaking of the platform. And I don't know when that shifts over from our product team. But this is where the thing gets interesting is, we talked about this last time. So the product teams, like people who work on Maps or Gmail or whatever, they can rely on and sort of use the services of the SREs as long as they follow the rules. But going back to what I was rat-holing on, it's interesting to compare it to DevOps where these roles are, like, still more separate at Google than you would see in, like, a standard 30-minute DevOps talk. I mean, depending on who you ask, I might have some opinions about DevOps. But I never advocated or was a huge fan of the idea that there's not separate responsibilities, right?

Like, I think that everyone is responsible for everything 100%, but in a sense, like, everyone has to be responsible 100%. So I'm going to borrow a metaphor from some other part of my life. So when I used to be younger and not have three kids to chase around, then I used to do a lot of martial arts training. So the way that I was taught, and I did a lot of things that were, you know, considered kind of violent, and then I did some things that are maybe a little bit more new-agey. I did a lot of Aikido and that kind of stuff. And the thing that you kind of have to find a balance in this training and this practice is between the sincerity that you have to attack someone with to help them improve and the fact that if you really, really attack someone, then the probability that you'll hurt them is very, very high, right? And so if you think about this in terms of a tribe, a community of, and this goes back to a bunch of other stuff I've talked about before, communities of interest and communities of practice. So as a tribe, you're a community of interest. And everyone in the tribe might have a different thing. And when you're trying to help people in the tribe, you want them to be able to stand next to you on the field and be part of the tribe. And if you have been too soft on them, then they're not going to be able to help you because they've never experienced anything that approximates real combat. And if you're too harsh with them, then you've basically hurt everyone. So now you're alone because everyone else is injured, right? And so this sincerity, and this is the way some of my teachers thought it, is like when you're on this mat, when you're in this ring, then you are 100% responsible for your own safety.

But you're also 100% responsible for the other person, right? And it doesn't mean you're not trying to, in some sense, and in some real sense, often beat them up. But you're also, you're also, you're not, you're not trying to take it past the point where it's valuable to them in this other larger context. Not all gyms think this way, but that's a long way of saying that if you have everyone in this undifferentiated pile, then I believe you end up with, to borrow a metaphor from biology, a tumor, right? You need to have, you need to have functioning organ systems throughout the organism that are, in some sense, for these complicated organisms that we are, separated from each other. There's these feedback loops, they influence each other, they're dependent on each other, but they're not the same thing. Your heart is not your lungs, is not your liver, is not your bones. So that's a good way to raise this up out of the mire that I jumped in there all of a sudden. In that context, describe, like, what the role of the SRE is, especially in relation to sort of the product teams. So last time, I mentioned a DevOps Days talk from an SRE director in Pittsburgh that I think is still one of the best DevOps Days talks. And he kind of had this mentality, or I might butcher the quote a little bit, but it's something along this spirit, that you'd basically get 90% of the benefit of SRE if you just took a third of your engineering capability and told them your job is not to do anything except for make everything better. So the Google SRE is highly capable as a software engineer. And I think that's one of the things that is kind of a quantum leap for people to accept. In a lot of cases, people in organizations that have traditional IT,

the humans involved have attached their identity to their task. And this is not unique to Google. This is something that happened over and over. When we were having conversations about Puppet 10 years ago, then there's quote-unquote sysadmins that say, oh, I'm not going to look at Ruby. That's for developers. I'm not going to do this. That's for developers. So I think that one of the things that SRE does is discard any notion of, oh, I'm not going to be capable as a software developer when I'm doing my operations, when I'm thinking about operations. And it's inverting that and saying 100%, I'm going to look for every opportunity to solve every problem, to optimize every problem that can be optimized with software, with software. Early on, that's a point that's really like pounded home. Because I think it's exactly what you're saying. It is a, I mean, both of us have been around developers and ops people for a long time. That's a major difference of how people normally think of ops people. I mean, an ops mindset even has, to use a Bridget term, like, you know, janky bash scripts. There's like this self-awareness of like, oh, we're kind of not really programmers. Janky bash exists because that is the interface to talk to the system, right? Janky bash, janky shell scripts exist because we decided to build on Unix instead of Lisp machine. Yeah, you know, and as another side note, there's several fun walkthroughs of how we arrived at the platform that we have that start off with, you know, we had the command line at one point, and then we realized this series of problems, and now we do things this way throughout the book. But yeah, and so that is, as we mentioned last time, there's, I mean, I guess it is a division to not only use their words, but you're supposed to, as an SRE, spend about 50% of your time improving the system and then 50% on both traditional sysadmin stuff, if you will, or toil, as they would call it, and then also like what I would call paperwork. Like, you know, you've got to select your annual benefits compensation.

That is a, I think that's a pretty concise way of looking at the role of an SRE, being able to program things, but also do the, all the way up to traditional things like being on call, but keep the system up and running. Well, there's also a little bit of a bifurcation in the domain expertise, and the way that the SREs are hired, they go through the same interview process as the software engineers, and that's also quite explicit in the book. But then they're maybe given a higher ranking based on their level of expertise with things like the network stacks or the systems. That, in many cases, and, you know, people might be shocked to know this, a lot of developers don't actually know how computers work. Yes, yeah. They're usually, in their defense, they're usually much better at understanding how humans work with computers. I mean, that's, but yes, they're not so much into how computers work. Highly debatable, I would say. That's right. There is certainly the enlightened developer that has some notion of usability, but there's, you know, interesting archetypes that have, you know, many, many, many hours of their lives devoted to understanding complex algorithms and type systems and these layers of abstractions that, in some ways, rest upon but are also, in many ways, ignorant of things like LUNs and VIPs and, you know, the nasty stuff that actually happens that bits, you know, travel and get rid of the disks or whatever. Yeah, and that's a good level of detail that's representative of everything. It's like, here's our hiring criteria for when we do the hat sorting between a product developer or whatever they call it and an SRE type of developer, which is an interesting preview of the tasks that they end up doing. So, going back to one of the things that you just said, I think that the SRE, when you read the book, you come to realize that, for one, they make it explicit.

There's a target of 50% toil, and if you exceed the toil, then they actually have the mandate from the organization, you know, the highest levels of Google, to push toil back onto the dev teams, right? And the SRE that, it doesn't give specifics. There's a lot of things that have anecdotal stories. It doesn't give specifics about this, but it says it's even possible for services to kind of get kicked out of having SRE support if they're not able to reduce the toil. Yeah, you know, I haven't finished reading all of it, but did they ever tell an anecdote about that? Those would be good stories. No, no, they don't. I mean, I think that's one of those, you know, the names have been changed to protect the guilty. But it alludes that that's possible. But it also kind of says it's not common. It's just possible. Yeah. I think in parenthood, you call that a teaching moment. Yeah. Well, the other thing is that not all services that are born in Google are initially given SRE support. And this is one of the one of the big differences that you see in the way that Google does SRE from the way that things emerge at Amazon, that the role in Amazon for the dev team running a service is they run it. If you build it, you run it. That's common refrain that people say in the DevOps space for the last 10 years. In SRE land, in Google's mindset, they're basically saying, if you prove that you can live inside the house and not soil the carpet, then we'll feed your dog, right? So we'll keep it – we'll do the proper thing. And then if we're in something that we don't understand or there's too much toil or whatever, then you have to wake up with us. And, you know, I don't remember this part. But, like, so when I'm reading through this, I'm just – I can't help but compare it to, like, I don't know,

traditional IT service management and ITIL stuff and things like that. But one thing I don't really remember is at what point when you're doing a product does an SRE get involved to help you, like, plan out your infrastructure and things like that to sort of, like, co-design this is what it looks like. Now, in contrast versus, like, an extremely opinionated infrastructure approach, just to contrast, it would be, like, here's your platform. That's your answer, right? Like, you can only do the things in here. But, like, I don't remember any discussion where it's, like, we worked with the product team to come up with, like, a net new thing or a way of doing things, like, from the beginning. So this is my understanding from reading the book and from conversations with Google SRE and also from a bit of watching the CRE process. And this is also – we probably should have started the episode with this, but we'll bury the lead, I guess, that if anyone from Google, any SRE or not from Google, wants to do a podcast with me and talk about this stuff, I think that would be really fun to have that conversation. But this is my interpretation of the way that works. And there's kind of two levels. So there's the level that the SREs provide a platform. You know, and this is a – you know, come from a selfish, pivotal perspective. There's a lot of quotes that I pulled out and posted on Twitter about how the SRE in many ways are providing this platform that's opinionated already. This solves this problem by making the right thing the easy thing. So this is in some ways answering your question. This is a direct quote from the book. It says, SRE build framework modules to implement canonical solutions for the concerned production area. As a result, development teams can focus on the business logic because the framework already takes care of correct infrastructure use.

So in some sense, there's not an SRE going through in the early phase and saying, you should do this, you should do this, this is how you should use the infrastructure so much. As the libraries, the modules, the services that are available to the Google developer have already solved a lot of those problems. So the framework itself is making sure the infrastructure is used correctly. Then the second phase of that is as you transition from the ideation phase, and this is really where you prove that your services is going to be whatever, potty trained. Then the SREs, and this is where the CRE stuff really starts to be interesting. The SREs do a review of everything you built and how it works. And they walk through, and this is also, and we should probably get back to specifics of the book, where as a team, as a community of interest, the SREs on one side, the software engineers on the other, and also the business is kind of the third leg of that stool, decide what the service level objective should be for that service. And then from the service level objective, walk back into what may or may not need to be changed about what's been delivered to achieve that level. That makes sense, and that's helped me remember. So there's almost like a post-build consultation and fixing of new things that come along. And I'm sure, as with all human systems, you can also just go talk to someone whenever you want, if you can hustle that up and arrange it and go get some fancy catered lunch there at the Googleplex or whatever. Yeah, and it is like, in some of the anecdotes, you can also kind of reverse engineer that there are many vital Google systems that SREs have built and now run that were sort of learned through suffering, right? Like, there's a lot of, what do they call it, chubby or whatever?

Like, there's some discussion here and there of their, like, global locking service and then of their storage things. And the way they explain how they worked, like, it's sort of like, we had this problem and we had to come up with a way of solving it. And so it is, in not a bad way, but it's sort of reactive to problems that you have instead of pre-architecting big systems that you might think you need. This isn't 100% unique to Google, but it's certainly something that Google's uniquely been in a position to experience. That when you're on the edge of technology, quite literally, you kind of have to invent your solutions. There's no vendor that could show up and really solve these things for Google. And as a consequence, they've, by Darwinian force, been and continue to be a learning culture that adapts over, you know, to the pressure of their own service. Well, as you mentioned, I want to make sure we get to some specifics, but just one more abstract thing, but it's a good segue into specifics. So, like, how do you, like, think people should read this book? Like, what approach should they take towards, like, I don't know, consuming it? Well, it depends on you, right? Everything's about context. And if you've ever met me before, you've probably heard me say it depends. So, it depends. But there's obviously the read it straight through approach, and that's what I did. And I love consuming this kind of stuff. But from a practical perspective, I don't think that the book's equally practical all the way through. And to be fair to the book, like, it's self-aware enough to kind of lay some of this out in its own introduction. But from my perspective, the way that I think, and you can go read this book for free online right now, that everyone should basically read the principles. And they kind of start in the order that they're most important. So, the first few chapters, embracing risk, the service-level objective, eliminating toil, those three, that's just gold.

That's just gold right there. Everything about it, you should read and internalize. And then, you know, from there, monitoring distributed systems and the evolution of automation in Google, definitely interesting. Starts to get a little bit more, depending on the types of problems you need to solve in your organization, maybe a little bit far in the future for some people. And then there's this block of, like, 20 chapters on practices. I'm not saying you should skip that, but some of those chapters are very, very Google-specific. And you should read them in that kind of anthropological sense of, like, this is how this one tribe solved this problem this one way with their context. And, you know, all this stuff about being on call, effective troubleshooting, incident response, post-mortems, like, that stuff can translate across. But when they start talking about Google-specific tools and Google-specific services, some of that's going to be really hard for you to bring into your world unless you're going to work for Google soon. And then I would actually – I would probably recommend reading the principles and then skipping to the parts on management and then going back and cherry-picking the practices unless you're wired to just read a book straight through. That's what I would do. Yeah, I mean, just as an example of your point about the practices. So, like, I got – I think a couple weeks ago I was somewhat making fun of this part of it. But when I finally sat down to read it, like, all the parts about, like, monitoring and how – what do they call it? Borgmon were really interesting because, you know, I used to write that kind of stuff. So it was – as you say, it was fascinating to read, like, how one tribe solved that problem. And, yeah, that was interesting. Like, I was thinking, like, you can go back to the application and instrument it. Like, that's a fantastic luxury.

That would be great if you could go back and make your applications actually monitorable. But, no, like, the way they figure out how to, like – and, you know, to query systems and they arrange all that stuff. Like, it's – that was fun to read over. So let's go back to – let's start with those three main things, right? Like, service level objectives, toils, and embracing risk. I mean, take those in whatever order you want. But, like, as I think you rightly said, those are, like – especially – well, not especially. But I think the SLO conversation – I just did it there – is, like, is really well done in kind of discussing why you would care about O's instead of A's. The thing I love about the way that Google's framed this problem for themselves is that it makes it so dispassionate. It kind of – just by the nature of how quantitized it gets, you remove a lot of the finger pointing. And it starts with this notion of managing risk. And so this is – this is kind of the first chapter of the principles. It has this notion that you're going to try to match the profile of the service to the risk the business is willing to take, right? And the other thing that's sort of implicit or even explicit in a lot of places is that being 100% available is folly, right? It's impossible, it's expensive, and for every nine that you want to add, it's 10 times more than the last nine at least, right? And that's just from the hip estimate. But you don't get more nines for free, right? And so coming up with an agreed-upon level of risk based on what you need – and also it's diminished returns for each nine for most applications. Right? So the difference between Gmail being four nines and five nines is almost imperceptible to the users.

Yeah, and this is – at least for me, there's a point they make here that I think is one of the, like, top five – I don't know, epiphamatic? Is that a phrase? Like, sort of epiphany sort of things is, like, they make this great point that, like, you could have 100% reliability, but there's a series of things between you and the end user, like the cellular network they're on, the mobile phone, the browser the things rendering on, that basically is going to obsolete all of your work, right? Like, things are going to get – putting this all in my phrase, this is, like, things are going to get screwed up across the system of multiple parties and things that you have no control over. And this is not an accurate way of putting it. Maybe there's some phrase you know that's better. But it's almost like you should only have a few more nines, if not the same amount of the least common denominator in the system. It's like – I don't know. I mean, it's a hard way to boil it down, but they make a really good point there. It goes back to, like, you have this graph of, you know, conditional probabilities, and if any one of those are down, then the system's effectively down. Yeah. And kind of going to gold wrap through your constraints, unless you're removing or adding availability to the constraint in the system, you're actually not improving the overall system. Yeah. It's sort of like punch is defined by the one turd that's in it. Oh, that's a great visual. Yeah. But, yeah. No, and there's also – I'm asked a lot to speak on, like, ROI and business value. And, you know, we mentioned the Art of Business Value book last time. I think there's a pretty astute conversation of, like, figuring out and nailing down business value that an SRE would have to know, which I think is good that you went over a little bit there. I think that part is highly instructive for anyone doing computer stuff to read. So the other thing that I see missing from a lot of the conversations I participate in that this book gives a very nice vocabulary and framing around is how to think about the SLO in terms of the SLI, which is the service level indicators, right?

And that is things like the – you know, they give you some ideas. They're not terribly – they're not 100% prescriptive, but they say things like request latency, the error rate, throughput, availability. There's a little bit of talk about the context of, you know, a user-facing service might have a different service level indicator than your data – big data services or your storage or, you know. There's things that are unique to a storage system with respect to durability and where, like, more user-facing things might be a bit more stateless and that kind of stuff. So think about your – think about the service you're trying to provide and then think about service level indicators. And then there's kind of this tension between these exhaustive service level indicators and coming up with things that are simple enough to reason about. And their advice, you know, probably hard-fought advice is that as much as possible, you want to standardize on common definitions of service level indicators across your services because, at least for the different types, you don't want to be reasoning about each service from first principles based on, like, some new set of how the data is represented or aggregated. Right. You want to reinvent the wheel as little as possible. Well, and in some ways, it's also adding to your toil, right? Every time you have to swap in a new context and start reasoning about a different service, you're adding to the cognitive overhead that those humans involved have to deal with. But if you don't have – in your ops teams and in your software teams, if you don't have this language around latency and error rates and throughput, you know, availability, if you're not able to have those kind of conversations with a shared vocabulary, then you have really no hope of getting to, like, a service level objective.

So, like, it's kind of like a starting point to even think about that vocabulary. So then how would you define toil? We've mentioned a little bit here and there, but, like – and let me put the question another way to build on what we've said already is, how would you distinguish or talk about what toil is in a way that's kind of, like – it's not just, like, the crap work. Like, it's important stuff that happens, but you're trying to, like, minimize it. And I ask it that way because, as they say in the book, but I feel like a lot of people might feel like, but that's my job. Like, that's what I do is this thing that you're calling toil. And you're actually right. So I think that the value of toil is that things actually get done, right, and that there is a culture – and this is something that's been widely discussed in the DevOps circles – is the kind of culture of heroism where people are keeping services running with their blood, sweat, and tears. And that can kind of work, but then there's also all this talk about burnout and morale and depression, and those things are real too, right? And there's a toll that you pay on the organization over time if you don't eliminate toil that – and it's not enduring value, right? Like, the number of times that the janky shell gets run by that person is not necessarily adding more value. But at that moment, with the way that things were managed and the expectations that were set and the resources that were devoted, that might be the only way to get the service through the night, right? And so what Google did early in their – and this is Darwinian. I believe this is – it's worth reflecting on the fact that Google cannot be Google if they didn't solve this problem this way. The overhead of trying to manage systems at that scale with toil is just this unfathomable burden.

And probably impossible, you know, whatever task. So by taking and reframing all this work as software, then one, you let the machines do the thing that they're very good at, which is doing the same thing, exactly the same over and over. And then two, you get to free the humans up to be more creative, to solve those higher-level problems. And so over time, I think another way to think about this is – or a great framing, which it's not in the chapter on toil, but it comes out in the chapter that's like one or two after about the evolution of automation. And there's a couple bullet points that I think are really interesting and worth thinking about. And I kind of – I want to frame this in the toil discussion, which is that things start out – this is also – this dovetails into a lot of conversations about DevOps and tools and platforms, the rest of it, is that things start out with kind of no automation, right? So there's some databases, there's some disks, there's some web servers. And if things fail, then some human has to manually intervene, right? And that could be, you know, surgically failing over databases or restarting things or, you know, even, you know, in some cases, maybe swapping hardware, right? And then as you go forward, then you get to what they have bullet point step two is externally maintain system-specific automation. So we went from no automation to an SRE has a script in her home directory. So now we wrote, you know, the janky bash or whatever that starts to do some of the stuff that we had to do over and over. Then step three is an externally maintained generic automation. So now we've taken and added the janky bash from our home directory into version control and added a bit of documentation to the wiki or something.

And so now everyone can see this. And this is, you know, very similar progression to what you see in all the people that ever borrowed or started using any of the DevOps tools, right? And then the fourth level of Google or in the, in the framing of the, of the chapter is internally maintained specific system-specific automation. So now, so now instead of having like something about these, these systems are like generic kind of processes to restart this or whatever, that the debated base itself or the web server itself or that service itself has, has automation specific to itself. Right. And so you, this is the, this is where there's now a database specific failover script for that database that, you know, again, is, is maybe manually run. And then the, the highest level that we kind of aspire to, and in some sense, every puppet project ever aspire to is that you want to build systems that don't need any human intervention. Right. So at the point where you have a system or some subset of systems that don't require any human intervention, then all that toil, you know, anytime you run that janky shell, that's toil is, is eliminated. Right. Now, now there's some new toil because there's some new systems, but that, that's sort of the, the progression I think frames toil the best. Yeah. I mean, just two things that I would add to that. I think, I think the, one of the things you phrased there is that toil has no enduring value, right? Like it may have, like, there's another principle as with all sort of like emergency response stuff that they go over later on, which is, you know, when they talk about incident management, which is stop the bleeding. Right. Like, so fix, fix the fire that's on your head and then worry about solving how your fire got on head later on. Like how, how, how your, your head got on fire. And so there is that principle of like on a, on a daily basis, like you do want to fix the issue, but then you want to immediately going to like, is this as to use your phrase of enduring value?

If it's not, we should automate it and therefore it's toil. And then, and then another thing that's discussed that you kind of alluded to that I think is, is really interesting and valuable to take away from it is, you know, in the, thanks, thanks to, to, to, to John Willis and others in the DevOps world, we would call this like anti-burnout measures or whatever. But they almost spin it in a slightly more positive way, even though I'm sure it's just two faces of the same thing where it's also a nice career development to actually like, to actually like work on something and evolve something instead of just always troubleshooting things. And I think this is another one of those areas where I guess the anthropological study-ness of it comes through in Google where I forget the exact passage, but it's basically working on toil is not going to get you noticed and advanced your career, but coming up with new services and basically innovating new things will. And so it is a nice way of addressing one of the many human side things of how you make someone satisfied and retain them and make them thrive in their goal by reducing this stuff that is valuable, but it's not really going to advance them in any way. So I actually also want to editorialize just a slight bit because I think this is something some people miss sometimes, which is, and Google, Google in that, in that framing around the evolution says that sometimes manual operation is unavoidable. And obviously there's this, this budget for toil, but the, the thing that I think sometimes people miss is you, you probably will never get to the point where you can automate systems that don't need intervention until you kind of go through those phases. It is, it is, it is, it is, I've almost never seen someone get good results trying to automate things that they couldn't do manually. If you don't understand how to do it manually, the chance that you're going to have an automated solution do it without intervention is probably negative. And as, and as, as they catalog many times, the first several times you try to automate it, you're going to need to manually intervene to fix what you did. Right. Like there's, there's, there's, there's many delightful stories of like, uh, oh crap, that happened.

And, uh, I think, I think the best ones evolve around where they just delete data or, or potentially would have deleted data if they didn't intervene and do something about it. So here's one thing, and I almost wish this was the, the opening line of the, of the discussion. And it's something that I think is, has been missing from a lot of, um, what I'll call kind of like the agile canon, which is this, this notion of, of non-functional requirements. In, in, in, in many sense, I would say that what SRE does is take what we're, we're often framed as non-functional requirements and make them, um, first class. And, and there's a line from the book, you know, it's actually, uh, I think a chapter heading, I think it's, um, or a section heading. It's that reliability is the fundamental feature. Right. Everything, everything starts with reliability and, and all the features in the world. That are built on kind of a, um, you know, quicksand or unstable foundation are not going to be valuable in, in this kind of service oriented, highly available world that we've moved into. Yeah. As, as, as, as I like to subtweet when I get frustrated with computers, users would like software to work. I think that the, the next thing that I would focus on is, is sort of the, the management chapters. Yeah. Um, I really want to call out the, the communication and collaboration in SRE. And that in some ways answers some of the earlier questions you were having about how the SRE teams interface with the software development teams. And also how the SRE teams interact with each other. And, and it has, uh, the, the, those last chapters have some interesting things about how the, the SREs structure their, their meetings. You know, there's, there's certain means that are mandatory and, and that, that really, it just kind of puts a really concrete reified framing on, you know, these are the mandatory meetings. If you care about this service and, you know, you're going to work on this team, you're going to come to these things.

You're going to talk about these things. You're going to have an opinion about these things. You're going to make an effort on these things. And, and that I think is, is really easy to translate more, more easily translated into things than like, oh, here's how we run this distributed lock service or whatever. Yes. There's an amazing anecdote. Um, this is, this goes a little bit back to the kind of like the dependency. Um, and I just think it's in the book, but I'll, I'll just throw it out there. Like this super quick version is that the, the, the chubby lock service, um, they, they actually made it too, too, um, reliable, um, based on its service level objective. It was exceeding its service level objective. And then what happened is they, they, um, they had people sort of depending on that level of, of reliability. And then when they had a failure, then all these other subsystems that were depending on the lock level of, of, of availability, the lock level of, um, you know, service level, they were failing in like this kind of cascading tree of dependencies. And so now what they've done, um, not just with chubby, but with some of the other services is that if you are always above your level of availability, they'll, they'll, they'll kind of have these synthetic incidents to, to flush out where, where people are taking that availability for granted. You know, people talk about the, the chaos monkeys or the, you know, the different, the, you know, ways that people inject failure into their systems. And this is one of the things that, that, that Google does is, is on, on level of availability. If you're, if you're way over your kind of stated level of availability, then, then sometimes just to flesh out like where you might have these, these discoupling and dependency problem, then, then you have these synthetic outages just to, to force the issue. There's, there's, there's a lot in that story, but it's a good practice to not be too good because it's exhausting.

And then, and then being less snarky, like, uh, what do they call that reversion to mean? Like just over time, like you shouldn't be rated by your most excellent or even like your most excellent minus one, you know, it's, you're going to build unrealistic expectations. And then, and then I guess, ironically, you're going to violate, as we said earlier, the number one thing, which is being reliable, right? Like it's hard to be reliably 100%, 100% of the time, if not impossible. Well, the thing that, the thing that, that, that gets you in that case is not the fact that there was the failure. It's the fact that people were depending on you not to. Exactly. Exactly. Like they're taking for granted that you won't. So. And, and so, so let's, let's maybe pick, pick on, pick on, uh, two things and, and we can close out unless you have other stuff. But like one thing that I think is worth discussing, discussing, and we've mentioned it is, is sort of like the rigorous nature of postmortems that they do. They call them postmortems, right? Did they come up with some other fanciful term for it? No, they call them postmortems. So, so like, I think that's something interesting to discuss because that's, that's a good, like, that's a good example where there are, I guess, in practice or as she has spoke, as you say, differences between traditional IT management and, and at least as portrayed in the SRE book. And then the second one, which I think would be interesting just briefly to talk about is the, the onboarding process of an SRE person. And, and as we'll get to, and I don't think it'll take too long, but what I think is interesting about that discussion in the book of how you onboard a new one is how much thought they've put into it. Right. Which, which, as we both laughed, like, it's funny when you say it out loud, but if, if, if, if you can like take off your snarky view of it, you're sort of like, oh, right, I should probably put thought into training people and making sure that when we hire them, things work out. But, but, but let's go, let's go back to the postmortem thing. So when, when, when I, you know, and it's spread out through the book, but then there's at least one, if not a couple of dedicated chapters to it. What I think, what I, what I think is interesting about the postmortem process is that it's a very actionable embodiment of a large part, at least for SREs, what it would mean to be a learning culture.

Right. So a lot of, a lot of what we talk about in our, whatever it is, world we operate in our DevOps, cloud native agile world is trying to get this notion home that you, to, to people that you need to have like a blameless culture of doing things. And you're only going to improve if you learn from failures and, and you structure that. And this is one of the first instances, at least in practice. I mean, actually blameless postmortems has lots of details, but this is a good overview of what that looks like in practice and how you run it and standardize it so that it's beneficial and, and not a, a terrible situation. So this is not a hundred percent unique to Google, but certainly in the kind of cloud native DevOps world, the, the, the main difference between the way people are, are trying to do this. And this is really a part of the learning culture as well is get away from the, the, the blame, the, the blame hot potato game that, that happens in a lot of traditional organizations. Like who, who's going to be, who's going to take the blame for this? Who's going to take the fall for this? And this is, you know, this isn't even unique to it, right? You see this, this in, in lean, you see this in, you know, manufacturing where it's, it's not about, it's not about blame. It's about stopping the line and making sure that we don't do this, this way again, right? We, we, we make sure that whatever, whatever's gone wrong, uh, doesn't go wrong again. And then, you know, there's, there's mendacity and there's these other things that, you know, maybe require someone, someone taking a little bit of, uh, that blame. But for the most part, you want to start with the assumption that you have good people because you hired them and onboarded them, right? And then, and that they're doing the best they can. And if they, if they weren't doing the right thing, that's a failure in how you train them. That's a failure in how those, those systems fail, right? And so there's, that's the big thing, but then Google has, you know, as, as Google is wanting to do some kind of really specific templates and the types of information that they, that they gather.

There's, there's example postmortem in the book that you could kind of go through and then they, they do, um, another interesting thing. And so they have this culture of postmortems, like they're fascinated with postmortems. I think, you know, this is again, something that all the, the ops DevOps people are into is this sort of, you know, failure, failure porn. Um, talking about how things fail and how they're troubleshooting, how they recover is, is, is fascinating. And so they, they have these internal, uh, Google plus groups and, you know, postmortem of the month that's shared everywhere, postmortem reading clubs. And then they, they, they gather so much detail around the postmortems that as part of the, you know, this is the new SRE onboarding question, um, that they, they do this wheel of misfortune exercise where they kind of reenact. Um, some of the, some of the stuff in the postmortems for the, you know, for the newbies. I think, I think you hit on the two things that, that were, uh, interesting for me. I mean, one, the first part is interesting. Here's the template that we have. And the, uh, the second one is again, to use the phrase, the, the anthropological like fascination of like, there really is like this culture of postmortems of, of, of like really, really finding that as a primary tool for not only like figuring out, I mean, I'm kind of interpolating here, but figuring out pastmortems. Fast failure, but almost training yourself to figure out how to handle future failures and just future things. But then you can also kind of read that there's just this, this fascination with it, like really just this desire to see how things went wrong instead of the internals of it. And, you know, that, that pops up, you, you see some cultural examples of that or, or how the culture of Google enforces that here and there, where, as you mentioned, not only is it part of like, um, some exercises that you might do or, uh, the way you would onboard people. But there's a couple of examples of like every now and then at some all hands meeting all the way up to the CEO, we just do a postmortem or we just kind of like celebrate them and, and really go wallow in them as it were, which I think is not really something that you would encounter at a traditional place.

Now let's discuss how we failed and recovered from it. And then, and the other thing that, um, and this isn't unique to Google either, but they definitely make this explicit is that you're sharing that with everyone across the world, right? It's available to anyone and, and explicitly you want to make sure that the, the stakeholders are doing it where I think in, in a lot of, um, traditional IT ops culture, there, there's a lot of information hiding. Yeah. Yeah. It's sort of like, uh, as, as, uh, I'm, I'm not looking up all these quotes like you, you've been doing, but like there's, there's another way, good way of summarizing. This is that, um, negative data is very value valuable and, and, and it kind of, there's almost a toil take on it. And I don't know, I would assume scientists like capital S scientists think like this as well, but it's sort of like if you catalog and almost have a library of negative data and negative feedback, that means you don't have to go through the toil of rediscovering it over again. And so you're actually optimizing yourself in the future, uh, to, to catalog it and have everyone learn from it because you don't have to, as it were, reinvent the wheel. So, so then the last, the, the, the last point, and then, and then we'll close out. I mean, uh, I think, I think, I think the section on, on onboarding, it's really comforting because like you read through it and it starts off, like you kind of echoed some of these sentiments. It's sort of like, um, what you would want in a hiring body is to realize that like, uh, you had a responsibility to be good enough for us to hire you. And you have an ongoing responsibility as an employee to like improve and learn. But now our responsibility of the organization is to make sure that you're adequately trained and given everything you need to succeed at it. And you can see that they put a lot of care and thought into that. Uh, and, and, you know, the way they open it up, which I think anyone who's ever done development or probably even ops will be familiar with is like, here's what we don't do. We don't just like give them a login to like the bug tracking system and tell them to go fix a bunch of bugs. Like, like instead we, we have kind of a guided tour through things and actually to, to a limit, do, do a little bit of mentorship and helping them out and do shadowing and things like that too.

And, and, and, and this is humble on both sides, right? Sort of the teacher and the student, if you will, like we just, we bring them up and teach them how to do things in the system because no one really knows exactly what to teach them ongoing on an incident based basis. And they don't really know how to educate themselves. So we try to take on the role of like, here's kind of the core curriculum that we have, but just sort of like, instead of being thrown into the deep end of the pool, let's gradually have you go through this way and have someone with you who can teach you up. I want to make a call back to my, my rambling preamble about the, the tribe training itself to do martial arts, because in a very real sense, it's very similar to that. And they, they, they hired these smart people, they put them through the rigorous interviews and now they've got them. And if you just put them straight to, to on call on, on these complex, you know, highly scaled, highly available systems and something goes wrong and they're the one person there responsible for it, they might have a very bad time. And they have this really nice, like you said, they, they, they, they have this kind of thoughtful discussion around the, the, the way that they think about this problem and some of the practical things they do. But there's this infographic that that's there that has the, the, the, the time on one axis. And then it has these activities ranging from the, the thinking about kind of these mental abstract activities and then the doing. So, so on one end of the spectrum, you have this kind of role playing and the postmortem reading and, and, you know, reading documentation and kind of going through the, these sort of synthetic things. And then on the, on the other side, you have, you know, reverse engineering and, and breaking and fixing real things. They're, they're not necessarily in production, but they're really real, right? They're real systems that you're breaking as part of your kind of training and then, and then you're going to shadow someone on call. Like you're, you're kind of going to go be someone's squire basically, right? To go back to the, to the, to the metaphor, like you're going to, you're going to, you're going to be there with the, with the, the, the front line, but you're not the one, right?

You're not that you don't need to be the one that everything rests on. You can, you can watch, you can, you can also help too, but you're, you're shadowing. That's, that's very similar to what the doctors, you know, and some of these other high pressure professions do. And then, and then last but not least is as you've gone through SRE and you've kind of demonstrated understanding, then, then there's this kind of higher level, most applied projects and, and kind of building and extending the services that are available in the platform to make it available for, for the, you know, the rest of those teams to utilize the infrastructure in the right way. And then, you know, also this is one of my favorite things just because of all the other things I love. But they, they kind of underpin that with two clouds and on one end of the spectrum, they have continuous learning and on the other, and they have continuous teaching. So you, so you're not just there to learn, but as you, as you advance and as your understanding progresses, you're also, you're bringing up the next batch of, of the tribe. I think as one little, like, uh, anecdotal or like one little proof of all that, there's, there's an interesting little aside about, uh, you know, it's, it's, it's, it's a common thing that you would assign, I don't know, a squire to, to do much of the work of writing up a postmortem. And you have to be careful that like, this isn't just like crap work that you're giving them because you remember how much we respect postmortems. Right. And how, how much those are a vital, important thing of doing things. But it is, it is like, it's evidence of like the thoughtfulness that gets put into like the tasks that we have them do and, and that they're actually valuable instructive tasks to the organization. So that, that new people, when they're hired can, uh, become as good as you want them to be. I just want to make a quick call out and then, you know, we can kind of come to the end, but they, they have this, um, explicit call out to the different types of, of, of thinkers, right. And different types of, um, like not all, not all humans are fungible. Not all, not all SREs have exactly the same approach or understanding, right. And so building that, that organizational capability, utilizing some of the, those different, um, SRE qualities is also a big part of it.

Yeah, that's good. And, and it's sort of, uh, uh, it's sort of evidence that if you're, if you're going to build out a system of people who are in charge of keeping your system up and running, it's probably a good idea to take a systems-based approach to all of that, right. And, and to, to your point of like, here are the different nodes or the ways that people think that we know of, and therefore how you adapt that. And, and you go along with it in the system rather than just assuming that there's this one way of people thinking. And it's, uh, there's, there's a lot of that in the system engineering of the SRE governance and way of thinking that, that pokes into there. That's interesting. Well, that was good. This is, this is, uh, I don't know, our second book report episode, maybe, uh, as other topics come up. I mean, we won't have all book reports, but we'll have to find some, uh, other book reports. It's, it's Andrew and Coté's book report club. All right. Well, as always, this has been Pivotal Conversations. If you want to, the fastest way to get this stuff is if you subscribe to us in your iTunes or your Overcast, or maybe like any good Google SRE person, you have built a globally distributed podcast downloading and listening system just from the ground up on your own using all the services you have, however you're doing it. But if you, if you go into, uh, any of those places, you can subscribe to the RSS fee. And then of course, the second fastest way is if you go to soundcloud.com slash Pivotal Conversations, and you can see all the episodes, including the past ones listed there. And then we also post the show notes on pivotal.io slash podcast, uh, where you can find links to things we've, uh, we've mentioned. And it's always great if you go into, uh, iTunes or wherever and leave us a review or a star rating. It's, it's nice if you share things in social medias and whatnot. And with that, we'll see you next time. Bye-bye. Till next time.

Tanzu Talk · 🍕 Get more pizza.