1
00:00:00,000 --> 00:00:05,760
Well, not we're. I'm recording from Paris. I assume you're still in the desert today?

2
00:00:06,200 --> 00:00:07,720
Tucson, Arizona.

3
00:00:07,960 --> 00:00:13,240
Well, anyhow, I'm in Paris. We've got a bunch of people here for DevOps, which will be fun.

4
00:00:13,680 --> 00:00:17,920
There's only a few episodes that you were on, and you were on most recently.

5
00:00:18,060 --> 00:00:22,820
And we were talking about Pivotal's involvement in the Google CRE program, which was good.

6
00:00:22,980 --> 00:00:27,120
It was nice to see people enjoyed that episode, and we had some good after conversation about it.

7
00:00:27,120 --> 00:00:36,000
But one of the topics that came up is that we kind of had to discuss in talking about CRE was about their SRE, their Site Reliability Engineer thing.

8
00:00:36,160 --> 00:00:41,060
And we mentioned that there's a book out about it, and so we thought we should get back together to talk about that in more detail.

9
00:00:41,220 --> 00:00:48,200
Now, there's a whole – what makes it possible to talk about this is there's a whole book about it now, which, as you pointed out last time, you can read for free.

10
00:00:48,200 --> 00:00:56,600
So I thought it would be good to just kind of discuss that book because it is – I can't say I've read through all of it, but I've read through a large amount of it.

11
00:00:56,700 --> 00:01:04,140
And it is interesting and sort of – to some extent, if you're into all this stuff, sort of fascinating to see how people think and put things together.

12
00:01:04,300 --> 00:01:12,700
Also, it's always delightful when people use like an uber-academic way of doing citations with the little square brackets and the first three letters of the name and the last two numbers of a year.

13
00:01:13,080 --> 00:01:15,380
You don't spend all that time in grad school for nothing.

14
00:01:15,700 --> 00:01:16,100
Exactly.

15
00:01:16,840 --> 00:01:23,000
And so first of all, I mean, let me just start off with a broad topic because you and I have talked about this a lot.

16
00:01:23,100 --> 00:01:24,920
But like what is this book?

17
00:01:25,460 --> 00:01:36,580
Like it's ostensibly – my first take on this book, which now is, I would say, is a fourth accurate, is sort of like this is a manual literally for if you get hired to be a Google SRE.

18
00:01:37,140 --> 00:01:40,680
And maybe not even like a hands-on-keys manual.

19
00:01:40,800 --> 00:01:41,700
There's a few extra things.

20
00:01:41,700 --> 00:01:45,600
But it is sort of like – and if you read the book, you realize that they don't have this attitude.

21
00:01:45,800 --> 00:01:47,420
But it would kind of be like, welcome to the job.

22
00:01:47,480 --> 00:01:49,540
And they would throw you this book and be like, read this, kid.

23
00:01:49,680 --> 00:01:54,700
I mean, there's a lot more to it going on than just like a written-down thing that's proprietary to Google.

24
00:01:54,800 --> 00:01:57,760
So I wonder how like you would describe what the deal with this book is.

25
00:01:57,760 --> 00:02:04,980
The framing last time, we kind of had the minimal viable SRE talk so we could talk a little bit about CRE.

26
00:02:05,400 --> 00:02:12,960
And as a consequence, I had a number of great conversations and I even had some people from Google kind of reach out to me.

27
00:02:13,100 --> 00:02:20,480
And I wanted to do a more proper book review, if you will, or really focus on some of the stuff that's available.

28
00:02:21,040 --> 00:02:26,480
And I do think that there's some of the things you said are true.

29
00:02:26,720 --> 00:02:29,380
It's definitely Google-specific.

30
00:02:29,700 --> 00:02:31,320
I wouldn't quite go as far as a manual.

31
00:02:31,320 --> 00:02:47,800
But in some ways, it's also kind of got value as this anthropological work of a culture and a practice inside of – really, in some ways, if you're actually paying attention to what Google's done over the last two decades,

32
00:02:48,240 --> 00:02:51,840
it's kind of one of the wonders of the world that Google even exists.

33
00:02:52,560 --> 00:03:00,540
And to kind of see them open up and put out this level of detail about how they think about this problem and how they solved it,

34
00:03:00,600 --> 00:03:07,820
I think that's – I keep saying it's a gift to the world, but it's certainly there for free for anyone who wants to pick it up and read it.

35
00:03:07,920 --> 00:03:13,760
Yeah, I mean, like pretty typical good engineers, they're actually pretty modest about the scale of things.

36
00:03:13,840 --> 00:03:21,140
But there is one point to kind of like pin it down where they're like, this is perhaps probably one of the largest systems ever built by humans.

37
00:03:22,380 --> 00:03:25,180
And the wording is not exactly like that, but I mean, they have a point.

38
00:03:25,380 --> 00:03:32,660
Like it's the whole of Google and all the systems behind it and applications on top of it and a word they would not really ever use.

39
00:03:32,720 --> 00:03:37,380
But the governance, the norms around it are an incredibly large, complicated system.

40
00:03:37,380 --> 00:03:44,600
I also think that there's some general themes that emerge as you try to build these systems.

41
00:03:44,720 --> 00:03:59,560
And I've never been behind the veil and worked at Google, but the types of processes, the types of tools, the types of approaches that you see emerge in kind of all the big web companies have some similarities.

42
00:04:00,100 --> 00:04:06,760
And I think this is just Google kind of putting the Google stamp on there.

43
00:04:06,760 --> 00:04:13,080
And this is something that got great response when I had this discussion with the actual SREs at Google Next.

44
00:04:13,680 --> 00:04:15,660
This is Google's DevOps, right?

45
00:04:15,700 --> 00:04:17,940
This is DevOps as she has spoken at Google.

46
00:04:18,340 --> 00:04:20,360
No, that's a good point, too.

47
00:04:20,640 --> 00:04:20,900
It is.

48
00:04:21,360 --> 00:04:31,820
And, you know, it's an interesting mental exercise to sort of compare and contrast DevOps because that's, I mean, to rat hole on one specific thing, I still haven't kind of reconciled it in my head.

49
00:04:31,820 --> 00:04:38,120
But there's an interesting discussion to be had about the difference between the SRE teams.

50
00:04:38,500 --> 00:04:42,380
Or not difference, but the relationship between the SRE teams and the product teams that's described.

51
00:04:43,100 --> 00:04:51,400
And it's not quite as strident as sort of like the notion of DevOps where all these roles are on one team.

52
00:04:51,580 --> 00:05:01,700
But then there is a certain amount of what almost feels like learned fluidity between the two teams, which is, I mean, I'm digging deep onto something without explaining the context for it.

53
00:05:01,700 --> 00:05:09,580
But the notion is you have an SRE who basically is responsible for, I don't know, I mean, tell me what you think of this.

54
00:05:09,660 --> 00:05:12,240
Like, it's defined pretty specifically most times.

55
00:05:12,300 --> 00:05:19,940
But the way I file it away is the site reliability engineer is responsible for the R, right?

56
00:05:20,580 --> 00:05:21,960
Keeping it up and reliable.

57
00:05:22,640 --> 00:05:29,660
But to a certain extent, and I don't know where this line is, they're also responsible for building and caretaking of the platform.

58
00:05:30,060 --> 00:05:32,500
And I don't know when that shifts over from our product team.

59
00:05:32,780 --> 00:05:35,920
But this is where the thing gets interesting is, we talked about this last time.

60
00:05:36,040 --> 00:05:49,360
So the product teams, like people who work on Maps or Gmail or whatever, they can rely on and sort of use the services of the SREs as long as they follow the rules.

61
00:05:49,360 --> 00:05:58,720
But going back to what I was rat-holing on, it's interesting to compare it to DevOps where these roles are, like, still more separate at Google than you would see in, like, a standard 30-minute DevOps talk.

62
00:05:58,880 --> 00:06:02,600
I mean, depending on who you ask, I might have some opinions about DevOps.

63
00:06:03,180 --> 00:06:12,520
But I never advocated or was a huge fan of the idea that there's not separate responsibilities, right?

64
00:06:12,520 --> 00:06:21,940
Like, I think that everyone is responsible for everything 100%, but in a sense, like, everyone has to be responsible 100%.

65
00:06:21,940 --> 00:06:25,000
So I'm going to borrow a metaphor from some other part of my life.

66
00:06:25,360 --> 00:06:34,960
So when I used to be younger and not have three kids to chase around, then I used to do a lot of martial arts training.

67
00:06:36,760 --> 00:06:47,980
So the way that I was taught, and I did a lot of things that were, you know, considered kind of violent, and then I did some things that are maybe a little bit more new-agey.

68
00:06:48,140 --> 00:06:49,820
I did a lot of Aikido and that kind of stuff.

69
00:06:49,820 --> 00:07:13,280
And the thing that you kind of have to find a balance in this training and this practice is between the sincerity that you have to attack someone with to help them improve and the fact that if you really, really attack someone, then the probability that you'll hurt them is very, very high, right?

70
00:07:14,040 --> 00:07:23,060
And so if you think about this in terms of a tribe, a community of, and this goes back to a bunch of other stuff I've talked about before, communities of interest and communities of practice.

71
00:07:23,240 --> 00:07:25,960
So as a tribe, you're a community of interest.

72
00:07:26,700 --> 00:07:29,820
And everyone in the tribe might have a different thing.

73
00:07:30,160 --> 00:07:40,220
And when you're trying to help people in the tribe, you want them to be able to stand next to you on the field and be part of the tribe.

74
00:07:40,220 --> 00:07:49,060
And if you have been too soft on them, then they're not going to be able to help you because they've never experienced anything that approximates real combat.

75
00:07:49,760 --> 00:07:55,780
And if you're too harsh with them, then you've basically hurt everyone.

76
00:07:55,920 --> 00:07:58,540
So now you're alone because everyone else is injured, right?

77
00:07:58,540 --> 00:08:13,020
And so this sincerity, and this is the way some of my teachers thought it, is like when you're on this mat, when you're in this ring, then you are 100% responsible for your own safety.

78
00:08:13,860 --> 00:08:18,380
But you're also 100% responsible for the other person, right?

79
00:08:18,480 --> 00:08:25,120
And it doesn't mean you're not trying to, in some sense, and in some real sense, often beat them up.

80
00:08:25,260 --> 00:08:33,560
But you're also, you're also, you're not, you're not trying to take it past the point where it's valuable to them in this other larger context.

81
00:08:34,320 --> 00:08:48,380
Not all gyms think this way, but that's a long way of saying that if you have everyone in this undifferentiated pile, then I believe you end up with, to borrow a metaphor from biology, a tumor, right?

82
00:08:48,380 --> 00:08:59,320
You need to have, you need to have functioning organ systems throughout the organism that are, in some sense, for these complicated organisms that we are, separated from each other.

83
00:08:59,580 --> 00:09:05,620
There's these feedback loops, they influence each other, they're dependent on each other, but they're not the same thing.

84
00:09:05,680 --> 00:09:08,720
Your heart is not your lungs, is not your liver, is not your bones.

85
00:09:09,220 --> 00:09:13,860
So that's a good way to raise this up out of the mire that I jumped in there all of a sudden.

86
00:09:13,860 --> 00:09:21,020
In that context, describe, like, what the role of the SRE is, especially in relation to sort of the product teams.

87
00:09:21,020 --> 00:09:31,160
So last time, I mentioned a DevOps Days talk from an SRE director in Pittsburgh that I think is still one of the best DevOps Days talks.

88
00:09:32,300 --> 00:09:39,460
And he kind of had this mentality, or I might butcher the quote a little bit, but it's something along this spirit,

89
00:09:39,920 --> 00:09:49,880
that you'd basically get 90% of the benefit of SRE if you just took a third of your engineering capability

90
00:09:49,880 --> 00:09:56,520
and told them your job is not to do anything except for make everything better.

91
00:09:56,520 --> 00:10:03,200
So the Google SRE is highly capable as a software engineer.

92
00:10:03,420 --> 00:10:10,760
And I think that's one of the things that is kind of a quantum leap for people to accept.

93
00:10:11,180 --> 00:10:17,180
In a lot of cases, people in organizations that have traditional IT,

94
00:10:18,240 --> 00:10:22,060
the humans involved have attached their identity to their task.

95
00:10:22,060 --> 00:10:24,240
And this is not unique to Google.

96
00:10:25,020 --> 00:10:27,420
This is something that happened over and over.

97
00:10:27,760 --> 00:10:31,580
When we were having conversations about Puppet 10 years ago,

98
00:10:32,000 --> 00:10:35,040
then there's quote-unquote sysadmins that say,

99
00:10:35,520 --> 00:10:36,760
oh, I'm not going to look at Ruby.

100
00:10:36,860 --> 00:10:37,680
That's for developers.

101
00:10:37,940 --> 00:10:39,060
I'm not going to do this.

102
00:10:39,100 --> 00:10:39,800
That's for developers.

103
00:10:40,020 --> 00:10:46,260
So I think that one of the things that SRE does is discard any notion of,

104
00:10:46,660 --> 00:10:52,020
oh, I'm not going to be capable as a software developer when I'm doing my operations,

105
00:10:52,340 --> 00:10:53,600
when I'm thinking about operations.

106
00:10:54,280 --> 00:10:56,860
And it's inverting that and saying 100%,

107
00:10:56,860 --> 00:11:00,780
I'm going to look for every opportunity to solve every problem,

108
00:11:00,880 --> 00:11:05,080
to optimize every problem that can be optimized with software, with software.

109
00:11:05,240 --> 00:11:08,420
Early on, that's a point that's really like pounded home.

110
00:11:08,500 --> 00:11:09,860
Because I think it's exactly what you're saying.

111
00:11:10,000 --> 00:11:14,160
It is a, I mean, both of us have been around developers and ops people for a long time.

112
00:11:14,160 --> 00:11:18,820
That's a major difference of how people normally think of ops people.

113
00:11:19,060 --> 00:11:23,740
I mean, an ops mindset even has, to use a Bridget term, like, you know,

114
00:11:23,920 --> 00:11:24,800
janky bash scripts.

115
00:11:24,960 --> 00:11:28,680
There's like this self-awareness of like, oh, we're kind of not really programmers.

116
00:11:29,000 --> 00:11:34,680
Janky bash exists because that is the interface to talk to the system, right?

117
00:11:34,780 --> 00:11:41,420
Janky bash, janky shell scripts exist because we decided to build on Unix instead of Lisp machine.

118
00:11:41,420 --> 00:11:49,760
Yeah, you know, and as another side note, there's several fun walkthroughs of how we arrived at the platform that we have that start off with, you know,

119
00:11:50,180 --> 00:11:55,720
we had the command line at one point, and then we realized this series of problems, and now we do things this way throughout the book.

120
00:11:55,840 --> 00:12:04,060
But yeah, and so that is, as we mentioned last time, there's, I mean, I guess it is a division to not only use their words,

121
00:12:04,060 --> 00:12:12,560
but you're supposed to, as an SRE, spend about 50% of your time improving the system and then 50% on both traditional sysadmin stuff, if you will,

122
00:12:12,640 --> 00:12:15,860
or toil, as they would call it, and then also like what I would call paperwork.

123
00:12:16,360 --> 00:12:18,800
Like, you know, you've got to select your annual benefits compensation.

124
00:12:19,180 --> 00:12:23,940
That is a, I think that's a pretty concise way of looking at the role of an SRE, being able to program things,

125
00:12:24,060 --> 00:12:28,960
but also do the, all the way up to traditional things like being on call, but keep the system up and running.

126
00:12:28,960 --> 00:12:36,280
Well, there's also a little bit of a bifurcation in the domain expertise, and the way that the SREs are hired,

127
00:12:37,020 --> 00:12:45,500
they go through the same interview process as the software engineers, and that's also quite explicit in the book.

128
00:12:46,160 --> 00:12:56,100
But then they're maybe given a higher ranking based on their level of expertise with things like the network stacks or the systems.

129
00:12:56,100 --> 00:13:04,120
That, in many cases, and, you know, people might be shocked to know this, a lot of developers don't actually know how computers work.

130
00:13:04,260 --> 00:13:05,100
Yes, yeah.

131
00:13:05,480 --> 00:13:11,620
They're usually, in their defense, they're usually much better at understanding how humans work with computers.

132
00:13:12,180 --> 00:13:15,700
I mean, that's, but yes, they're not so much into how computers work.

133
00:13:16,000 --> 00:13:18,060
Highly debatable, I would say.

134
00:13:18,340 --> 00:13:19,080
That's right.

135
00:13:19,660 --> 00:13:25,920
There is certainly the enlightened developer that has some notion of usability,

136
00:13:26,220 --> 00:13:32,920
but there's, you know, interesting archetypes that have, you know, many, many, many hours of their lives

137
00:13:32,920 --> 00:13:39,660
devoted to understanding complex algorithms and type systems and these layers of abstractions

138
00:13:39,660 --> 00:13:47,980
that, in some ways, rest upon but are also, in many ways, ignorant of things like LUNs and VIPs

139
00:13:47,980 --> 00:13:54,360
and, you know, the nasty stuff that actually happens that bits, you know, travel and get rid of the disks or whatever.

140
00:13:54,600 --> 00:13:58,320
Yeah, and that's a good level of detail that's representative of everything.

141
00:13:58,440 --> 00:14:03,960
It's like, here's our hiring criteria for when we do the hat sorting between a product developer

142
00:14:03,960 --> 00:14:09,720
or whatever they call it and an SRE type of developer, which is an interesting preview of the tasks that they end up doing.

143
00:14:09,720 --> 00:14:15,960
So, going back to one of the things that you just said, I think that the SRE, when you read the book,

144
00:14:16,120 --> 00:14:19,400
you come to realize that, for one, they make it explicit.

145
00:14:19,680 --> 00:14:30,840
There's a target of 50% toil, and if you exceed the toil, then they actually have the mandate from the organization,

146
00:14:31,100 --> 00:14:35,860
you know, the highest levels of Google, to push toil back onto the dev teams, right?

147
00:14:35,860 --> 00:14:41,460
And the SRE that, it doesn't give specifics.

148
00:14:41,560 --> 00:14:43,160
There's a lot of things that have anecdotal stories.

149
00:14:43,220 --> 00:14:50,120
It doesn't give specifics about this, but it says it's even possible for services to kind of get kicked out of having SRE support

150
00:14:50,120 --> 00:14:53,820
if they're not able to reduce the toil.

151
00:14:54,440 --> 00:14:58,160
Yeah, you know, I haven't finished reading all of it, but did they ever tell an anecdote about that?

152
00:14:58,600 --> 00:14:59,840
Those would be good stories.

153
00:14:59,860 --> 00:15:01,160
No, no, they don't.

154
00:15:01,220 --> 00:15:04,900
I mean, I think that's one of those, you know, the names have been changed to protect the guilty.

155
00:15:05,300 --> 00:15:07,360
But it alludes that that's possible.

156
00:15:08,200 --> 00:15:10,940
But it also kind of says it's not common.

157
00:15:11,100 --> 00:15:12,480
It's just possible.

158
00:15:12,860 --> 00:15:13,100
Yeah.

159
00:15:13,220 --> 00:15:15,580
I think in parenthood, you call that a teaching moment.

160
00:15:15,760 --> 00:15:15,960
Yeah.

161
00:15:15,960 --> 00:15:23,320
Well, the other thing is that not all services that are born in Google are initially given SRE support.

162
00:15:23,640 --> 00:15:31,540
And this is one of the one of the big differences that you see in the way that Google does SRE from the way that things emerge at Amazon,

163
00:15:32,260 --> 00:15:37,520
that the role in Amazon for the dev team running a service is they run it.

164
00:15:37,880 --> 00:15:39,260
If you build it, you run it.

165
00:15:39,260 --> 00:15:45,340
That's common refrain that people say in the DevOps space for the last 10 years.

166
00:15:45,840 --> 00:15:51,260
In SRE land, in Google's mindset, they're basically saying,

167
00:15:51,500 --> 00:15:59,960
if you prove that you can live inside the house and not soil the carpet, then we'll feed your dog, right?

168
00:16:00,880 --> 00:16:05,140
So we'll keep it – we'll do the proper thing.

169
00:16:05,320 --> 00:16:13,160
And then if we're in something that we don't understand or there's too much toil or whatever, then you have to wake up with us.

170
00:16:13,340 --> 00:16:14,920
And, you know, I don't remember this part.

171
00:16:15,160 --> 00:16:19,360
But, like, so when I'm reading through this, I'm just – I can't help but compare it to, like, I don't know,

172
00:16:19,440 --> 00:16:22,640
traditional IT service management and ITIL stuff and things like that.

173
00:16:22,640 --> 00:16:31,260
But one thing I don't really remember is at what point when you're doing a product does an SRE get involved to help you, like,

174
00:16:31,380 --> 00:16:36,580
plan out your infrastructure and things like that to sort of, like, co-design this is what it looks like.

175
00:16:36,680 --> 00:16:43,580
Now, in contrast versus, like, an extremely opinionated infrastructure approach, just to contrast, it would be, like,

176
00:16:43,720 --> 00:16:44,740
here's your platform.

177
00:16:45,260 --> 00:16:46,720
That's your answer, right?

178
00:16:46,780 --> 00:16:48,920
Like, you can only do the things in here.

179
00:16:48,920 --> 00:16:53,580
But, like, I don't remember any discussion where it's, like, we worked with the product team to come up with, like,

180
00:16:54,360 --> 00:16:58,780
a net new thing or a way of doing things, like, from the beginning.

181
00:16:59,100 --> 00:17:06,640
So this is my understanding from reading the book and from conversations with Google SRE

182
00:17:06,640 --> 00:17:09,800
and also from a bit of watching the CRE process.

183
00:17:10,720 --> 00:17:16,580
And this is also – we probably should have started the episode with this, but we'll bury the lead, I guess,

184
00:17:16,580 --> 00:17:25,360
that if anyone from Google, any SRE or not from Google, wants to do a podcast with me and talk about this stuff,

185
00:17:25,920 --> 00:17:29,320
I think that would be really fun to have that conversation.

186
00:17:29,440 --> 00:17:32,180
But this is my interpretation of the way that works.

187
00:17:32,480 --> 00:17:34,880
And there's kind of two levels.

188
00:17:35,060 --> 00:17:38,940
So there's the level that the SREs provide a platform.

189
00:17:39,240 --> 00:17:42,840
You know, and this is a – you know, come from a selfish, pivotal perspective.

190
00:17:42,960 --> 00:17:54,240
There's a lot of quotes that I pulled out and posted on Twitter about how the SRE in many ways are providing this platform

191
00:17:54,240 --> 00:17:56,280
that's opinionated already.

192
00:17:56,440 --> 00:17:59,880
This solves this problem by making the right thing the easy thing.

193
00:18:00,180 --> 00:18:02,640
So this is in some ways answering your question.

194
00:18:02,760 --> 00:18:04,060
This is a direct quote from the book.

195
00:18:04,800 --> 00:18:11,080
It says, SRE build framework modules to implement canonical solutions for the concerned production area.

196
00:18:11,560 --> 00:18:19,460
As a result, development teams can focus on the business logic because the framework already takes care of correct infrastructure use.

197
00:18:20,460 --> 00:18:28,660
So in some sense, there's not an SRE going through in the early phase and saying, you should do this, you should do this,

198
00:18:28,700 --> 00:18:30,720
this is how you should use the infrastructure so much.

199
00:18:31,100 --> 00:18:42,180
As the libraries, the modules, the services that are available to the Google developer have already solved a lot of those problems.

200
00:18:42,580 --> 00:18:46,640
So the framework itself is making sure the infrastructure is used correctly.

201
00:18:47,140 --> 00:18:59,620
Then the second phase of that is as you transition from the ideation phase, and this is really where you prove that your services is going to be whatever, potty trained.

202
00:19:00,240 --> 00:19:05,400
Then the SREs, and this is where the CRE stuff really starts to be interesting.

203
00:19:06,160 --> 00:19:09,680
The SREs do a review of everything you built and how it works.

204
00:19:09,900 --> 00:19:24,060
And they walk through, and this is also, and we should probably get back to specifics of the book, where as a team, as a community of interest, the SREs on one side, the software engineers on the other,

205
00:19:24,560 --> 00:19:31,260
and also the business is kind of the third leg of that stool, decide what the service level objective should be for that service.

206
00:19:31,260 --> 00:19:40,900
And then from the service level objective, walk back into what may or may not need to be changed about what's been delivered to achieve that level.

207
00:19:41,200 --> 00:19:42,400
That makes sense, and that's helped me remember.

208
00:19:42,560 --> 00:19:49,560
So there's almost like a post-build consultation and fixing of new things that come along.

209
00:19:50,140 --> 00:20:02,180
And I'm sure, as with all human systems, you can also just go talk to someone whenever you want, if you can hustle that up and arrange it and go get some fancy catered lunch there at the Googleplex or whatever.

210
00:20:02,440 --> 00:20:17,720
Yeah, and it is like, in some of the anecdotes, you can also kind of reverse engineer that there are many vital Google systems that SREs have built and now run that were sort of learned through suffering, right?

211
00:20:17,720 --> 00:20:20,520
Like, there's a lot of, what do they call it, chubby or whatever?

212
00:20:20,660 --> 00:20:25,480
Like, there's some discussion here and there of their, like, global locking service and then of their storage things.

213
00:20:25,680 --> 00:20:31,680
And the way they explain how they worked, like, it's sort of like, we had this problem and we had to come up with a way of solving it.

214
00:20:31,800 --> 00:20:39,840
And so it is, in not a bad way, but it's sort of reactive to problems that you have instead of pre-architecting big systems that you might think you need.

215
00:20:39,840 --> 00:20:47,380
This isn't 100% unique to Google, but it's certainly something that Google's uniquely been in a position to experience.

216
00:20:47,720 --> 00:20:55,020
That when you're on the edge of technology, quite literally, you kind of have to invent your solutions.

217
00:20:55,520 --> 00:21:00,520
There's no vendor that could show up and really solve these things for Google.

218
00:21:00,520 --> 00:21:13,740
And as a consequence, they've, by Darwinian force, been and continue to be a learning culture that adapts over, you know, to the pressure of their own service.

219
00:21:13,980 --> 00:21:20,620
Well, as you mentioned, I want to make sure we get to some specifics, but just one more abstract thing, but it's a good segue into specifics.

220
00:21:20,980 --> 00:21:24,300
So, like, how do you, like, think people should read this book?

221
00:21:24,680 --> 00:21:28,460
Like, what approach should they take towards, like, I don't know, consuming it?

222
00:21:28,460 --> 00:21:31,100
Well, it depends on you, right?

223
00:21:31,120 --> 00:21:32,180
Everything's about context.

224
00:21:32,180 --> 00:21:36,100
And if you've ever met me before, you've probably heard me say it depends.

225
00:21:36,400 --> 00:21:38,340
So, it depends.

226
00:21:38,560 --> 00:21:44,920
But there's obviously the read it straight through approach, and that's what I did.

227
00:21:45,360 --> 00:21:49,040
And I love consuming this kind of stuff.

228
00:21:49,460 --> 00:21:54,820
But from a practical perspective, I don't think that the book's equally practical all the way through.

229
00:21:54,820 --> 00:22:01,140
And to be fair to the book, like, it's self-aware enough to kind of lay some of this out in its own introduction.

230
00:22:01,700 --> 00:22:11,580
But from my perspective, the way that I think, and you can go read this book for free online right now, that everyone should basically read the principles.

231
00:22:12,260 --> 00:22:15,040
And they kind of start in the order that they're most important.

232
00:22:15,040 --> 00:22:25,260
So, the first few chapters, embracing risk, the service-level objective, eliminating toil, those three, that's just gold.

233
00:22:25,520 --> 00:22:26,520
That's just gold right there.

234
00:22:26,640 --> 00:22:30,040
Everything about it, you should read and internalize.

235
00:22:30,760 --> 00:22:36,440
And then, you know, from there, monitoring distributed systems and the evolution of automation in Google, definitely interesting.

236
00:22:37,000 --> 00:22:46,960
Starts to get a little bit more, depending on the types of problems you need to solve in your organization, maybe a little bit far in the future for some people.

237
00:22:47,760 --> 00:22:53,600
And then there's this block of, like, 20 chapters on practices.

238
00:22:54,600 --> 00:23:00,260
I'm not saying you should skip that, but some of those chapters are very, very Google-specific.

239
00:23:01,700 --> 00:23:14,840
And you should read them in that kind of anthropological sense of, like, this is how this one tribe solved this problem this one way with their context.

240
00:23:16,120 --> 00:23:24,040
And, you know, all this stuff about being on call, effective troubleshooting, incident response, post-mortems, like, that stuff can translate across.

241
00:23:25,040 --> 00:23:35,160
But when they start talking about Google-specific tools and Google-specific services, some of that's going to be really hard for you to bring into your world unless you're going to work for Google soon.

242
00:23:35,520 --> 00:23:51,340
And then I would actually – I would probably recommend reading the principles and then skipping to the parts on management and then going back and cherry-picking the practices unless you're wired to just read a book straight through.

243
00:23:51,680 --> 00:23:52,360
That's what I would do.

244
00:23:52,360 --> 00:23:55,240
Yeah, I mean, just as an example of your point about the practices.

245
00:23:55,560 --> 00:24:00,320
So, like, I got – I think a couple weeks ago I was somewhat making fun of this part of it.

246
00:24:00,460 --> 00:24:06,180
But when I finally sat down to read it, like, all the parts about, like, monitoring and how – what do they call it?

247
00:24:06,200 --> 00:24:09,460
Borgmon were really interesting because, you know, I used to write that kind of stuff.

248
00:24:09,560 --> 00:24:14,140
So it was – as you say, it was fascinating to read, like, how one tribe solved that problem.

249
00:24:14,580 --> 00:24:17,180
And, yeah, that was interesting.

250
00:24:17,180 --> 00:24:21,520
Like, I was thinking, like, you can go back to the application and instrument it.

251
00:24:21,660 --> 00:24:23,160
Like, that's a fantastic luxury.

252
00:24:25,460 --> 00:24:29,960
That would be great if you could go back and make your applications actually monitorable.

253
00:24:30,620 --> 00:24:36,680
But, no, like, the way they figure out how to, like – and, you know, to query systems and they arrange all that stuff.

254
00:24:36,800 --> 00:24:39,060
Like, it's – that was fun to read over.

255
00:24:39,060 --> 00:24:42,820
So let's go back to – let's start with those three main things, right?

256
00:24:42,900 --> 00:24:46,520
Like, service level objectives, toils, and embracing risk.

257
00:24:46,620 --> 00:24:48,440
I mean, take those in whatever order you want.

258
00:24:48,520 --> 00:24:52,700
But, like, as I think you rightly said, those are, like – especially – well, not especially.

259
00:24:52,940 --> 00:25:04,540
But I think the SLO conversation – I just did it there – is, like, is really well done in kind of discussing why you would care about O's instead of A's.

260
00:25:04,540 --> 00:25:12,960
The thing I love about the way that Google's framed this problem for themselves is that it makes it so dispassionate.

261
00:25:13,420 --> 00:25:20,360
It kind of – just by the nature of how quantitized it gets, you remove a lot of the finger pointing.

262
00:25:20,920 --> 00:25:23,340
And it starts with this notion of managing risk.

263
00:25:23,400 --> 00:25:26,000
And so this is – this is kind of the first chapter of the principles.

264
00:25:27,040 --> 00:25:35,600
It has this notion that you're going to try to match the profile of the service to the risk the business is willing to take, right?

265
00:25:35,600 --> 00:25:47,440
And the other thing that's sort of implicit or even explicit in a lot of places is that being 100% available is folly, right?

266
00:25:47,480 --> 00:25:57,280
It's impossible, it's expensive, and for every nine that you want to add, it's 10 times more than the last nine at least, right?

267
00:25:57,280 --> 00:26:00,200
And that's just from the hip estimate.

268
00:26:00,420 --> 00:26:04,440
But you don't get more nines for free, right?

269
00:26:04,440 --> 00:26:17,440
And so coming up with an agreed-upon level of risk based on what you need – and also it's diminished returns for each nine for most applications.

270
00:26:18,480 --> 00:26:18,820
Right?

271
00:26:18,940 --> 00:26:28,620
So the difference between Gmail being four nines and five nines is almost imperceptible to the users.

272
00:26:28,820 --> 00:26:36,260
Yeah, and this is – at least for me, there's a point they make here that I think is one of the, like, top five – I don't know, epiphamatic?

273
00:26:36,400 --> 00:26:37,120
Is that a phrase?

274
00:26:37,340 --> 00:26:55,460
Like, sort of epiphany sort of things is, like, they make this great point that, like, you could have 100% reliability, but there's a series of things between you and the end user, like the cellular network they're on, the mobile phone, the browser the things rendering on, that basically is going to obsolete all of your work, right?

275
00:26:55,540 --> 00:27:03,800
Like, things are going to get – putting this all in my phrase, this is, like, things are going to get screwed up across the system of multiple parties and things that you have no control over.

276
00:27:03,960 --> 00:27:05,580
And this is not an accurate way of putting it.

277
00:27:05,620 --> 00:27:07,540
Maybe there's some phrase you know that's better.

278
00:27:07,600 --> 00:27:15,420
But it's almost like you should only have a few more nines, if not the same amount of the least common denominator in the system.

279
00:27:15,560 --> 00:27:16,420
It's like – I don't know.

280
00:27:16,500 --> 00:27:19,220
I mean, it's a hard way to boil it down, but they make a really good point there.

281
00:27:19,240 --> 00:27:27,660
It goes back to, like, you have this graph of, you know, conditional probabilities, and if any one of those are down, then the system's effectively down.

282
00:27:27,880 --> 00:27:28,120
Yeah.

283
00:27:28,120 --> 00:27:39,500
And kind of going to gold wrap through your constraints, unless you're removing or adding availability to the constraint in the system, you're actually not improving the overall system.

284
00:27:39,560 --> 00:27:39,700
Yeah.

285
00:27:39,800 --> 00:27:43,560
It's sort of like punch is defined by the one turd that's in it.

286
00:27:46,220 --> 00:27:47,600
Oh, that's a great visual.

287
00:27:48,660 --> 00:27:49,100
Yeah.

288
00:27:49,240 --> 00:27:49,520
But, yeah.

289
00:27:49,660 --> 00:27:56,580
No, and there's also – I'm asked a lot to speak on, like, ROI and business value.

290
00:27:56,580 --> 00:28:00,120
And, you know, we mentioned the Art of Business Value book last time.

291
00:28:00,220 --> 00:28:10,700
I think there's a pretty astute conversation of, like, figuring out and nailing down business value that an SRE would have to know, which I think is good that you went over a little bit there.

292
00:28:10,960 --> 00:28:14,760
I think that part is highly instructive for anyone doing computer stuff to read.

293
00:28:14,760 --> 00:28:31,920
So the other thing that I see missing from a lot of the conversations I participate in that this book gives a very nice vocabulary and framing around is how to think about the SLO in terms of the SLI, which is the service level indicators, right?

294
00:28:31,920 --> 00:28:36,920
And that is things like the – you know, they give you some ideas.

295
00:28:37,280 --> 00:28:45,840
They're not terribly – they're not 100% prescriptive, but they say things like request latency, the error rate, throughput, availability.

296
00:28:46,560 --> 00:28:59,680
There's a little bit of talk about the context of, you know, a user-facing service might have a different service level indicator than your data – big data services or your storage or, you know.

297
00:28:59,680 --> 00:29:10,720
There's things that are unique to a storage system with respect to durability and where, like, more user-facing things might be a bit more stateless and that kind of stuff.

298
00:29:10,880 --> 00:29:18,340
So think about your – think about the service you're trying to provide and then think about service level indicators.

299
00:29:18,800 --> 00:29:27,920
And then there's kind of this tension between these exhaustive service level indicators and coming up with things that are simple enough to reason about.

300
00:29:27,920 --> 00:29:44,720
And their advice, you know, probably hard-fought advice is that as much as possible, you want to standardize on common definitions of service level indicators across your services because, at least for the different types,

301
00:29:45,320 --> 00:29:56,620
you don't want to be reasoning about each service from first principles based on, like, some new set of how the data is represented or aggregated.

302
00:29:57,040 --> 00:29:59,800
Right. You want to reinvent the wheel as little as possible.

303
00:29:59,940 --> 00:30:02,360
Well, and in some ways, it's also adding to your toil, right?

304
00:30:02,360 --> 00:30:11,780
Every time you have to swap in a new context and start reasoning about a different service, you're adding to the cognitive overhead that those humans involved have to deal with.

305
00:30:13,020 --> 00:30:24,620
But if you don't have – in your ops teams and in your software teams, if you don't have this language around latency and error rates and throughput, you know, availability,

306
00:30:24,620 --> 00:30:35,560
if you're not able to have those kind of conversations with a shared vocabulary, then you have really no hope of getting to, like, a service level objective.

307
00:30:36,460 --> 00:30:40,960
So, like, it's kind of like a starting point to even think about that vocabulary.

308
00:30:41,320 --> 00:30:43,760
So then how would you define toil?

309
00:30:44,000 --> 00:30:50,660
We've mentioned a little bit here and there, but, like – and let me put the question another way to build on what we've said already is,

310
00:30:50,800 --> 00:30:58,480
how would you distinguish or talk about what toil is in a way that's kind of, like – it's not just, like, the crap work.

311
00:30:58,600 --> 00:31:02,300
Like, it's important stuff that happens, but you're trying to, like, minimize it.

312
00:31:02,400 --> 00:31:08,300
And I ask it that way because, as they say in the book, but I feel like a lot of people might feel like, but that's my job.

313
00:31:08,860 --> 00:31:12,360
Like, that's what I do is this thing that you're calling toil.

314
00:31:12,740 --> 00:31:14,620
And you're actually right.

315
00:31:14,880 --> 00:31:22,740
So I think that the value of toil is that things actually get done, right, and that there is a culture –

316
00:31:22,740 --> 00:31:27,440
and this is something that's been widely discussed in the DevOps circles –

317
00:31:27,440 --> 00:31:37,160
is the kind of culture of heroism where people are keeping services running with their blood, sweat, and tears.

318
00:31:37,680 --> 00:31:46,040
And that can kind of work, but then there's also all this talk about burnout and morale and depression, and those things are real too, right?

319
00:31:46,040 --> 00:31:58,360
And there's a toll that you pay on the organization over time if you don't eliminate toil that – and it's not enduring value, right?

320
00:31:58,400 --> 00:32:06,980
Like, the number of times that the janky shell gets run by that person is not necessarily adding more value.

321
00:32:07,540 --> 00:32:14,840
But at that moment, with the way that things were managed and the expectations that were set and the resources that were devoted,

322
00:32:15,180 --> 00:32:18,340
that might be the only way to get the service through the night, right?

323
00:32:18,340 --> 00:32:23,660
And so what Google did early in their – and this is Darwinian.

324
00:32:23,840 --> 00:32:29,040
I believe this is – it's worth reflecting on the fact that Google cannot be Google if they didn't solve this problem this way.

325
00:32:30,020 --> 00:32:39,440
The overhead of trying to manage systems at that scale with toil is just this unfathomable burden.

326
00:32:39,440 --> 00:32:44,420
And probably impossible, you know, whatever task.

327
00:32:44,720 --> 00:32:54,760
So by taking and reframing all this work as software, then one, you let the machines do the thing that they're very good at,

328
00:32:54,820 --> 00:32:58,320
which is doing the same thing, exactly the same over and over.

329
00:32:58,720 --> 00:33:05,020
And then two, you get to free the humans up to be more creative, to solve those higher-level problems.

330
00:33:05,380 --> 00:33:13,360
And so over time, I think another way to think about this is – or a great framing, which it's not in the chapter on toil,

331
00:33:13,460 --> 00:33:19,940
but it comes out in the chapter that's like one or two after about the evolution of automation.

332
00:33:20,320 --> 00:33:24,140
And there's a couple bullet points that I think are really interesting and worth thinking about.

333
00:33:24,280 --> 00:33:29,300
And I kind of – I want to frame this in the toil discussion, which is that things start out –

334
00:33:29,300 --> 00:33:34,500
this is also – this dovetails into a lot of conversations about DevOps and tools and platforms, the rest of it,

335
00:33:34,560 --> 00:33:37,700
is that things start out with kind of no automation, right?

336
00:33:37,740 --> 00:33:42,220
So there's some databases, there's some disks, there's some web servers.

337
00:33:42,660 --> 00:33:47,680
And if things fail, then some human has to manually intervene, right?

338
00:33:47,700 --> 00:33:57,160
And that could be, you know, surgically failing over databases or restarting things or, you know, even, you know,

339
00:33:57,180 --> 00:34:00,520
in some cases, maybe swapping hardware, right?

340
00:34:00,560 --> 00:34:10,680
And then as you go forward, then you get to what they have bullet point step two is externally maintain system-specific automation.

341
00:34:10,680 --> 00:34:18,120
So we went from no automation to an SRE has a script in her home directory.

342
00:34:18,580 --> 00:34:25,500
So now we wrote, you know, the janky bash or whatever that starts to do some of the stuff that we had to do over and over.

343
00:34:26,200 --> 00:34:29,840
Then step three is an externally maintained generic automation.

344
00:34:29,840 --> 00:34:42,300
So now we've taken and added the janky bash from our home directory into version control and added a bit of documentation to the wiki or something.

345
00:34:43,040 --> 00:34:44,140
And so now everyone can see this.

346
00:34:44,200 --> 00:34:54,340
And this is, you know, very similar progression to what you see in all the people that ever borrowed or started using any of the DevOps tools, right?

347
00:34:55,740 --> 00:35:05,080
And then the fourth level of Google or in the, in the framing of the, of the chapter is internally maintained specific system-specific automation.

348
00:35:05,620 --> 00:35:16,320
So now, so now instead of having like something about these, these systems are like generic kind of processes to restart this or whatever,

349
00:35:16,320 --> 00:35:25,540
that the debated base itself or the web server itself or that service itself has, has automation specific to itself.

350
00:35:25,740 --> 00:35:26,780
Right.

351
00:35:26,860 --> 00:35:36,140
And so you, this is the, this is where there's now a database specific failover script for that database that, you know, again, is, is maybe manually run.

352
00:35:36,140 --> 00:35:47,340
And then the, the highest level that we kind of aspire to, and in some sense, every puppet project ever aspire to is that you want to build systems that don't need any human intervention.

353
00:35:47,800 --> 00:35:48,360
Right.

354
00:35:48,440 --> 00:36:05,240
So at the point where you have a system or some subset of systems that don't require any human intervention, then all that toil, you know, anytime you run that janky shell, that's toil is, is eliminated.

355
00:36:05,580 --> 00:36:05,700
Right.

356
00:36:05,760 --> 00:36:14,140
Now, now there's some new toil because there's some new systems, but that, that's sort of the, the progression I think frames toil the best.

357
00:36:14,260 --> 00:36:14,460
Yeah.

358
00:36:14,580 --> 00:36:16,700
I mean, just two things that I would add to that.

359
00:36:16,700 --> 00:36:22,240
I think, I think the, one of the things you phrased there is that toil has no enduring value, right?

360
00:36:22,240 --> 00:36:31,500
Like it may have, like, there's another principle as with all sort of like emergency response stuff that they go over later on, which is, you know, when they talk about incident management, which is stop the bleeding.

361
00:36:31,500 --> 00:36:31,940
Right.

362
00:36:31,960 --> 00:36:38,000
Like, so fix, fix the fire that's on your head and then worry about solving how your fire got on head later on.

363
00:36:38,160 --> 00:36:40,220
Like how, how, how your, your head got on fire.

364
00:36:40,220 --> 00:36:50,460
And so there is that principle of like on a, on a daily basis, like you do want to fix the issue, but then you want to immediately going to like, is this as to use your phrase of enduring value?

365
00:36:50,560 --> 00:36:53,660
If it's not, we should automate it and therefore it's toil.

366
00:36:53,940 --> 00:37:08,220
And then, and then another thing that's discussed that you kind of alluded to that I think is, is really interesting and valuable to take away from it is, you know, in the, thanks, thanks to, to, to, to John Willis and others in the DevOps world, we would call this like anti-burnout measures or whatever.

367
00:37:08,220 --> 00:37:21,700
But they almost spin it in a slightly more positive way, even though I'm sure it's just two faces of the same thing where it's also a nice career development to actually like, to actually like work on something and evolve something instead of just always troubleshooting things.

368
00:37:21,700 --> 00:37:37,700
And I think this is another one of those areas where I guess the anthropological study-ness of it comes through in Google where I forget the exact passage, but it's basically working on toil is not going to get you noticed and advanced your career, but coming up with new services and basically innovating new things will.

369
00:37:37,700 --> 00:37:50,820
And so it is a nice way of addressing one of the many human side things of how you make someone satisfied and retain them and make them thrive in their goal by reducing this stuff that is valuable, but it's not really going to advance them in any way.

370
00:37:50,900 --> 00:38:04,420
So I actually also want to editorialize just a slight bit because I think this is something some people miss sometimes, which is, and Google, Google in that, in that framing around the evolution says that sometimes manual operation is unavoidable.

371
00:38:04,600 --> 00:38:20,420
And obviously there's this, this budget for toil, but the, the thing that I think sometimes people miss is you, you probably will never get to the point where you can automate systems that don't need intervention until you kind of go through those phases.

372
00:38:20,420 --> 00:38:29,820
It is, it is, it is, it is, I've almost never seen someone get good results trying to automate things that they couldn't do manually.

373
00:38:29,820 --> 00:38:38,020
If you don't understand how to do it manually, the chance that you're going to have an automated solution do it without intervention is probably negative.

374
00:38:38,360 --> 00:38:46,320
And as, and as, as they catalog many times, the first several times you try to automate it, you're going to need to manually intervene to fix what you did.

375
00:38:47,260 --> 00:38:47,700
Right.

376
00:38:47,880 --> 00:38:53,140
Like there's, there's, there's, there's many delightful stories of like, uh, oh crap, that happened.

377
00:38:53,340 --> 00:39:02,020
And, uh, I think, I think the best ones evolve around where they just delete data or, or potentially would have deleted data if they didn't intervene and do something about it.

378
00:39:02,020 --> 00:39:07,660
So here's one thing, and I almost wish this was the, the opening line of the, of the discussion.

379
00:39:08,300 --> 00:39:18,420
And it's something that I think is, has been missing from a lot of, um, what I'll call kind of like the agile canon, which is this, this notion of, of non-functional requirements.

380
00:39:18,420 --> 00:39:29,520
In, in, in, in many sense, I would say that what SRE does is take what we're, we're often framed as non-functional requirements and make them, um, first class.

381
00:39:30,280 --> 00:39:38,420
And, and there's a line from the book, you know, it's actually, uh, I think a chapter heading, I think it's, um, or a section heading.

382
00:39:38,420 --> 00:39:41,200
It's that reliability is the fundamental feature.

383
00:39:41,580 --> 00:39:41,720
Right.

384
00:39:42,380 --> 00:39:48,340
Everything, everything starts with reliability and, and all the features in the world.

385
00:39:48,420 --> 00:40:00,440
That are built on kind of a, um, you know, quicksand or unstable foundation are not going to be valuable in, in this kind of service oriented, highly available world that we've moved into.

386
00:40:00,660 --> 00:40:00,920
Yeah.

387
00:40:00,980 --> 00:40:06,180
As, as, as, as I like to subtweet when I get frustrated with computers, users would like software to work.

388
00:40:07,700 --> 00:40:14,540
I think that the, the next thing that I would focus on is, is sort of the, the management chapters.

389
00:40:14,540 --> 00:40:14,980
Yeah.

390
00:40:15,200 --> 00:40:19,140
Um, I really want to call out the, the communication and collaboration in SRE.

391
00:40:19,460 --> 00:40:26,700
And that in some ways answers some of the earlier questions you were having about how the SRE teams interface with the software development teams.

392
00:40:26,700 --> 00:40:30,480
And also how the SRE teams interact with each other.

393
00:40:31,140 --> 00:40:40,600
And, and it has, uh, the, the, those last chapters have some interesting things about how the, the SREs structure their, their meetings.

394
00:40:40,600 --> 00:40:52,920
You know, there's, there's certain means that are mandatory and, and that, that really, it just kind of puts a really concrete reified framing on, you know, these are the mandatory meetings.

395
00:40:52,920 --> 00:40:58,040
If you care about this service and, you know, you're going to work on this team, you're going to come to these things.

396
00:40:58,080 --> 00:40:59,060
You're going to talk about these things.

397
00:40:59,100 --> 00:41:00,320
You're going to have an opinion about these things.

398
00:41:00,360 --> 00:41:01,700
You're going to make an effort on these things.

399
00:41:02,280 --> 00:41:11,740
And, and that I think is, is really easy to translate more, more easily translated into things than like, oh, here's how we run this distributed lock service or whatever.

400
00:41:12,920 --> 00:41:13,320
Yes.

401
00:41:14,440 --> 00:41:15,900
There's an amazing anecdote.

402
00:41:16,400 --> 00:41:19,640
Um, this is, this goes a little bit back to the kind of like the dependency.

403
00:41:20,520 --> 00:41:23,780
Um, and I just think it's in the book, but I'll, I'll just throw it out there.

404
00:41:23,780 --> 00:41:34,980
Like this super quick version is that the, the, the chubby lock service, um, they, they actually made it too, too, um, reliable, um, based on its service level objective.

405
00:41:34,980 --> 00:41:37,460
It was exceeding its service level objective.

406
00:41:37,900 --> 00:41:44,420
And then what happened is they, they, um, they had people sort of depending on that level of, of reliability.

407
00:41:45,420 --> 00:42:01,040
And then when they had a failure, then all these other subsystems that were depending on the lock level of, of, of availability, the lock level of, um, you know, service level, they were failing in like this kind of cascading tree of dependencies.

408
00:42:01,640 --> 00:42:21,960
And so now what they've done, um, not just with chubby, but with some of the other services is that if you are always above your level of availability, they'll, they'll, they'll kind of have these synthetic incidents to, to flush out where, where people are taking that availability for granted.

409
00:42:22,760 --> 00:42:29,660
You know, people talk about the, the chaos monkeys or the, you know, the different, the, you know, ways that people inject failure into their systems.

410
00:42:29,840 --> 00:42:35,920
And this is one of the things that, that, that Google does is, is on, on level of availability.

411
00:42:35,920 --> 00:42:51,780
If you're, if you're way over your kind of stated level of availability, then, then sometimes just to flesh out like where you might have these, these discoupling and dependency problem, then, then you have these synthetic outages just to, to force the issue.

412
00:42:51,900 --> 00:42:57,660
There's, there's, there's a lot in that story, but it's a good practice to not be too good because it's exhausting.

413
00:42:58,780 --> 00:43:04,280
And then, and then being less snarky, like, uh, what do they call that reversion to mean?

414
00:43:04,280 --> 00:43:13,180
Like just over time, like you shouldn't be rated by your most excellent or even like your most excellent minus one, you know, it's, you're going to build unrealistic expectations.

415
00:43:13,180 --> 00:43:20,540
And then, and then I guess, ironically, you're going to violate, as we said earlier, the number one thing, which is being reliable, right?

416
00:43:20,560 --> 00:43:26,040
Like it's hard to be reliably 100%, 100% of the time, if not impossible.

417
00:43:26,440 --> 00:43:32,820
Well, the thing that, the thing that, that, that gets you in that case is not the fact that there was the failure.

418
00:43:32,820 --> 00:43:35,600
It's the fact that people were depending on you not to.

419
00:43:35,860 --> 00:43:36,220
Exactly.

420
00:43:36,580 --> 00:43:36,920
Exactly.

421
00:43:36,920 --> 00:43:38,900
Like they're taking for granted that you won't.

422
00:43:39,060 --> 00:43:39,180
So.

423
00:43:39,600 --> 00:43:46,860
And, and so, so let's, let's maybe pick, pick on, pick on, uh, two things and, and we can close out unless you have other stuff.

424
00:43:46,860 --> 00:43:54,360
But like one thing that I think is worth discussing, discussing, and we've mentioned it is, is sort of like the rigorous nature of postmortems that they do.

425
00:43:54,440 --> 00:43:55,900
They call them postmortems, right?

426
00:43:55,960 --> 00:43:57,900
Did they come up with some other fanciful term for it?

427
00:43:57,960 --> 00:43:58,720
No, they call them postmortems.

428
00:43:58,720 --> 00:44:14,860
So, so like, I think that's something interesting to discuss because that's, that's a good, like, that's a good example where there are, I guess, in practice or as she has spoke, as you say, differences between traditional IT management and, and at least as portrayed in the SRE book.

429
00:44:15,060 --> 00:44:22,460
And then the second one, which I think would be interesting just briefly to talk about is the, the onboarding process of an SRE person.

430
00:44:22,460 --> 00:44:32,300
And, and as we'll get to, and I don't think it'll take too long, but what I think is interesting about that discussion in the book of how you onboard a new one is how much thought they've put into it.

431
00:44:33,060 --> 00:44:33,420
Right.

432
00:44:34,480 --> 00:44:48,260
Which, which, as we both laughed, like, it's funny when you say it out loud, but if, if, if, if you can like take off your snarky view of it, you're sort of like, oh, right, I should probably put thought into training people and making sure that when we hire them, things work out.

433
00:44:48,260 --> 00:44:50,660
But, but, but let's go, let's go back to the postmortem thing.

434
00:44:50,800 --> 00:44:57,160
So when, when, when I, you know, and it's spread out through the book, but then there's at least one, if not a couple of dedicated chapters to it.

435
00:44:57,220 --> 00:45:07,540
What I think, what I, what I think is interesting about the postmortem process is that it's a very actionable embodiment of a large part, at least for SREs, what it would mean to be a learning culture.

436
00:45:07,540 --> 00:45:20,840
Right. So a lot of, a lot of what we talk about in our, whatever it is, world we operate in our DevOps, cloud native agile world is trying to get this notion home that you, to, to people that you need to have like a blameless culture of doing things.

437
00:45:20,980 --> 00:45:26,920
And you're only going to improve if you learn from failures and, and you structure that.

438
00:45:27,000 --> 00:45:30,640
And this is one of the first instances, at least in practice.

439
00:45:30,640 --> 00:45:41,780
I mean, actually blameless postmortems has lots of details, but this is a good overview of what that looks like in practice and how you run it and standardize it so that it's beneficial and, and not a, a terrible situation.

440
00:45:41,780 --> 00:45:53,020
So this is not a hundred percent unique to Google, but certainly in the kind of cloud native DevOps world, the, the, the main difference between the way people are, are trying to do this.

441
00:45:53,020 --> 00:46:05,780
And this is really a part of the learning culture as well is get away from the, the, the blame, the, the blame hot potato game that, that happens in a lot of traditional organizations.

442
00:46:05,780 --> 00:46:09,020
Like who, who's going to be, who's going to take the blame for this?

443
00:46:09,040 --> 00:46:10,380
Who's going to take the fall for this?

444
00:46:10,700 --> 00:46:13,520
And this is, you know, this isn't even unique to it, right?

445
00:46:13,540 --> 00:46:20,760
You see this, this in, in lean, you see this in, you know, manufacturing where it's, it's not about, it's not about blame.

446
00:46:20,760 --> 00:46:26,840
It's about stopping the line and making sure that we don't do this, this way again, right?

447
00:46:26,840 --> 00:46:32,220
We, we, we make sure that whatever, whatever's gone wrong, uh, doesn't go wrong again.

448
00:46:32,220 --> 00:46:40,360
And then, you know, there's, there's mendacity and there's these other things that, you know, maybe require someone, someone taking a little bit of, uh, that blame.

449
00:46:40,360 --> 00:46:46,580
But for the most part, you want to start with the assumption that you have good people because you hired them and onboarded them, right?

450
00:46:46,580 --> 00:46:48,760
And then, and that they're doing the best they can.

451
00:46:48,800 --> 00:46:53,520
And if they, if they weren't doing the right thing, that's a failure in how you train them.

452
00:46:53,520 --> 00:46:56,640
That's a failure in how those, those systems fail, right?

453
00:46:56,640 --> 00:47:07,760
And so there's, that's the big thing, but then Google has, you know, as, as Google is wanting to do some kind of really specific templates and the types of information that they, that they gather.

454
00:47:07,860 --> 00:47:15,000
There's, there's example postmortem in the book that you could kind of go through and then they, they do, um, another interesting thing.

455
00:47:15,000 --> 00:47:18,960
And so they have this culture of postmortems, like they're fascinated with postmortems.

456
00:47:18,960 --> 00:47:26,900
I think, you know, this is again, something that all the, the ops DevOps people are into is this sort of, you know, failure, failure porn.

457
00:47:27,460 --> 00:47:32,540
Um, talking about how things fail and how they're troubleshooting, how they recover is, is, is fascinating.

458
00:47:32,920 --> 00:47:42,320
And so they, they have these internal, uh, Google plus groups and, you know, postmortem of the month that's shared everywhere, postmortem reading clubs.

459
00:47:42,420 --> 00:47:56,320
And then they, they, they gather so much detail around the postmortems that as part of the, you know, this is the new SRE onboarding question, um, that they, they do this wheel of misfortune exercise where they kind of reenact.

460
00:47:56,880 --> 00:48:01,840
Um, some of the, some of the stuff in the postmortems for the, you know, for the newbies.

461
00:48:01,840 --> 00:48:05,000
I think, I think you hit on the two things that, that were, uh, interesting for me.

462
00:48:05,100 --> 00:48:06,720
I mean, one, the first part is interesting.

463
00:48:06,860 --> 00:48:08,100
Here's the template that we have.

464
00:48:08,300 --> 00:48:26,840
And the, uh, the second one is again, to use the phrase, the, the anthropological like fascination of like, there really is like this culture of postmortems of, of, of like really, really finding that as a primary tool for not only like figuring out, I mean, I'm kind of interpolating here, but figuring out pastmortems.

465
00:48:26,880 --> 00:48:31,800
Fast failure, but almost training yourself to figure out how to handle future failures and just future things.

466
00:48:31,800 --> 00:48:39,420
But then you can also kind of read that there's just this, this fascination with it, like really just this desire to see how things went wrong instead of the internals of it.

467
00:48:39,420 --> 00:48:54,440
And, you know, that, that pops up, you, you see some cultural examples of that or, or how the culture of Google enforces that here and there, where, as you mentioned, not only is it part of like, um, some exercises that you might do or, uh, the way you would onboard people.

468
00:48:54,440 --> 00:49:09,700
But there's a couple of examples of like every now and then at some all hands meeting all the way up to the CEO, we just do a postmortem or we just kind of like celebrate them and, and really go wallow in them as it were, which I think is not really something that you would encounter at a traditional place.

469
00:49:10,140 --> 00:49:13,420
Now let's discuss how we failed and recovered from it.

470
00:49:13,720 --> 00:49:21,980
And then, and the other thing that, um, and this isn't unique to Google either, but they definitely make this explicit is that you're sharing that with everyone across the world, right?

471
00:49:22,400 --> 00:49:35,200
It's available to anyone and, and explicitly you want to make sure that the, the stakeholders are doing it where I think in, in a lot of, um, traditional IT ops culture, there, there's a lot of information hiding.

472
00:49:35,900 --> 00:49:36,300
Yeah.

473
00:49:36,440 --> 00:49:36,640
Yeah.

474
00:49:36,640 --> 00:49:43,600
It's sort of like, uh, as, as, uh, I'm, I'm not looking up all these quotes like you, you've been doing, but like there's, there's another way, good way of summarizing.

475
00:49:43,600 --> 00:49:51,660
This is that, um, negative data is very value valuable and, and, and it kind of, there's almost a toil take on it.

476
00:49:51,660 --> 00:50:05,780
And I don't know, I would assume scientists like capital S scientists think like this as well, but it's sort of like if you catalog and almost have a library of negative data and negative feedback, that means you don't have to go through the toil of rediscovering it over again.

477
00:50:05,780 --> 00:50:14,280
And so you're actually optimizing yourself in the future, uh, to, to catalog it and have everyone learn from it because you don't have to, as it were, reinvent the wheel.

478
00:50:14,280 --> 00:50:17,740
So, so then the last, the, the, the last point, and then, and then we'll close out.

479
00:50:17,800 --> 00:50:26,480
I mean, uh, I think, I think, I think the section on, on onboarding, it's really comforting because like you read through it and it starts off, like you kind of echoed some of these sentiments.

480
00:50:26,480 --> 00:50:36,000
It's sort of like, um, what you would want in a hiring body is to realize that like, uh, you had a responsibility to be good enough for us to hire you.

481
00:50:36,000 --> 00:50:39,940
And you have an ongoing responsibility as an employee to like improve and learn.

482
00:50:40,100 --> 00:50:46,860
But now our responsibility of the organization is to make sure that you're adequately trained and given everything you need to succeed at it.

483
00:50:46,900 --> 00:50:49,360
And you can see that they put a lot of care and thought into that.

484
00:50:49,440 --> 00:50:57,260
Uh, and, and, you know, the way they open it up, which I think anyone who's ever done development or probably even ops will be familiar with is like, here's what we don't do.

485
00:50:57,260 --> 00:51:02,940
We don't just like give them a login to like the bug tracking system and tell them to go fix a bunch of bugs.

486
00:51:04,000 --> 00:51:13,800
Like, like instead we, we have kind of a guided tour through things and actually to, to a limit, do, do a little bit of mentorship and helping them out and do shadowing and things like that too.

487
00:51:13,800 --> 00:51:16,420
And, and, and, and this is humble on both sides, right?

488
00:51:16,480 --> 00:51:26,980
Sort of the teacher and the student, if you will, like we just, we bring them up and teach them how to do things in the system because no one really knows exactly what to teach them ongoing on an incident based basis.

489
00:51:26,980 --> 00:51:28,860
And they don't really know how to educate themselves.

490
00:51:28,860 --> 00:51:40,440
So we try to take on the role of like, here's kind of the core curriculum that we have, but just sort of like, instead of being thrown into the deep end of the pool, let's gradually have you go through this way and have someone with you who can teach you up.

491
00:51:40,440 --> 00:51:50,180
I want to make a call back to my, my rambling preamble about the, the tribe training itself to do martial arts, because in a very real sense, it's very similar to that.

492
00:51:50,300 --> 00:51:56,700
And they, they, they hired these smart people, they put them through the rigorous interviews and now they've got them.

493
00:51:56,700 --> 00:52:11,500
And if you just put them straight to, to on call on, on these complex, you know, highly scaled, highly available systems and something goes wrong and they're the one person there responsible for it, they might have a very bad time.

494
00:52:11,500 --> 00:52:23,760
And they have this really nice, like you said, they, they, they, they have this kind of thoughtful discussion around the, the, the way that they think about this problem and some of the practical things they do.

495
00:52:23,820 --> 00:52:29,080
But there's this infographic that that's there that has the, the, the, the time on one axis.

496
00:52:29,080 --> 00:52:38,300
And then it has these activities ranging from the, the thinking about kind of these mental abstract activities and then the doing.

497
00:52:39,100 --> 00:52:49,540
So, so on one end of the spectrum, you have this kind of role playing and the postmortem reading and, and, you know, reading documentation and kind of going through the, these sort of synthetic things.

498
00:52:49,540 --> 00:52:56,780
And then on the, on the other side, you have, you know, reverse engineering and, and breaking and fixing real things.

499
00:52:57,180 --> 00:53:01,100
They're, they're not necessarily in production, but they're really real, right?

500
00:53:01,100 --> 00:53:08,560
They're real systems that you're breaking as part of your kind of training and then, and then you're going to shadow someone on call.

501
00:53:08,980 --> 00:53:12,400
Like you're, you're kind of going to go be someone's squire basically, right?

502
00:53:12,400 --> 00:53:21,120
To go back to the, to the, to the metaphor, like you're going to, you're going to, you're going to be there with the, with the, the, the front line, but you're not the one, right?

503
00:53:21,120 --> 00:53:24,720
You're not that you don't need to be the one that everything rests on.

504
00:53:24,800 --> 00:53:28,780
You can, you can watch, you can, you can also help too, but you're, you're shadowing.

505
00:53:28,920 --> 00:53:34,880
That's, that's very similar to what the doctors, you know, and some of these other high pressure professions do.

506
00:53:35,180 --> 00:53:58,740
And then, and then last but not least is as you've gone through SRE and you've kind of demonstrated understanding, then, then there's this kind of higher level, most applied projects and, and kind of building and extending the services that are available in the platform to make it available for, for the, you know, the rest of those teams to utilize the infrastructure in the right way.

507
00:53:58,740 --> 00:54:04,780
And then, you know, also this is one of my favorite things just because of all the other things I love.

508
00:54:05,500 --> 00:54:13,600
But they, they kind of underpin that with two clouds and on one end of the spectrum, they have continuous learning and on the other, and they have continuous teaching.

509
00:54:14,040 --> 00:54:23,980
So you, so you're not just there to learn, but as you, as you advance and as your understanding progresses, you're also, you're bringing up the next batch of, of the tribe.

510
00:54:24,080 --> 00:54:38,240
I think as one little, like, uh, anecdotal or like one little proof of all that, there's, there's an interesting little aside about, uh, you know, it's, it's, it's, it's a common thing that you would assign, I don't know, a squire to, to do much of the work of writing up a postmortem.

511
00:54:38,240 --> 00:54:45,080
And you have to be careful that like, this isn't just like crap work that you're giving them because you remember how much we respect postmortems.

512
00:54:45,080 --> 00:54:45,360
Right.

513
00:54:45,660 --> 00:54:48,440
And how, how much those are a vital, important thing of doing things.

514
00:54:48,440 --> 00:54:57,480
But it is, it is like, it's evidence of like the thoughtfulness that gets put into like the tasks that we have them do and, and that they're actually valuable instructive tasks to the organization.

515
00:54:57,580 --> 00:55:01,520
So that, that new people, when they're hired can, uh, become as good as you want them to be.

516
00:55:01,520 --> 00:55:11,800
I just want to make a quick call out and then, you know, we can kind of come to the end, but they, they have this, um, explicit call out to the different types of, of, of thinkers, right.

517
00:55:11,820 --> 00:55:15,640
And different types of, um, like not all, not all humans are fungible.

518
00:55:15,640 --> 00:55:20,880
Not all, not all SREs have exactly the same approach or understanding, right.

519
00:55:20,920 --> 00:55:30,940
And so building that, that organizational capability, utilizing some of the, those different, um, SRE qualities is also a big part of it.

520
00:55:31,060 --> 00:55:31,740
Yeah, that's good.

521
00:55:31,840 --> 00:55:40,800
And, and it's sort of, uh, uh, it's sort of evidence that if you're, if you're going to build out a system of people who are in charge of keeping your system up and running,

522
00:55:40,800 --> 00:55:44,660
it's probably a good idea to take a systems-based approach to all of that, right.

523
00:55:44,880 --> 00:55:51,020
And, and to, to your point of like, here are the different nodes or the ways that people think that we know of, and therefore how you adapt that.

524
00:55:51,020 --> 00:55:55,720
And, and you go along with it in the system rather than just assuming that there's this one way of people thinking.

525
00:55:56,040 --> 00:56:02,340
And it's, uh, there's, there's a lot of that in the system engineering of the SRE governance and way of thinking that, that pokes into there.

526
00:56:02,400 --> 00:56:02,980
That's interesting.

527
00:56:03,060 --> 00:56:03,660
Well, that was good.

528
00:56:03,680 --> 00:56:08,780
This is, this is, uh, I don't know, our second book report episode, maybe, uh, as other topics come up.

529
00:56:08,800 --> 00:56:12,360
I mean, we won't have all book reports, but we'll have to find some, uh, other book reports.

530
00:56:12,360 --> 00:56:15,140
It's, it's Andrew and Coté's book report club.

531
00:56:17,240 --> 00:56:17,840
All right.

532
00:56:18,140 --> 00:56:20,560
Well, as always, this has been Pivotal Conversations.

533
00:56:20,800 --> 00:56:37,840
If you want to, the fastest way to get this stuff is if you subscribe to us in your iTunes or your Overcast, or maybe like any good Google SRE person, you have built a globally distributed podcast downloading and listening system just from the ground up on your own using all the services you have, however you're doing it.

534
00:56:37,900 --> 00:56:41,340
But if you, if you go into, uh, any of those places, you can subscribe to the RSS fee.

535
00:56:41,420 --> 00:56:49,320
And then of course, the second fastest way is if you go to soundcloud.com slash Pivotal Conversations, and you can see all the episodes, including the past ones listed there.

536
00:56:49,540 --> 00:56:56,680
And then we also post the show notes on pivotal.io slash podcast, uh, where you can find links to things we've, uh, we've mentioned.

537
00:56:56,680 --> 00:57:01,620
And it's always great if you go into, uh, iTunes or wherever and leave us a review or a star rating.

538
00:57:01,860 --> 00:57:04,720
It's, it's nice if you share things in social medias and whatnot.

539
00:57:04,980 --> 00:57:06,540
And with that, we'll see you next time.

540
00:57:06,720 --> 00:57:07,280
Bye-bye.

541
00:57:07,520 --> 00:57:08,360
Till next time.