1
00:00:00,000 --> 00:00:09,540
Our guest this time, Tammy Butow, you know, chaos engineering is one of those things that I've heard

2
00:00:09,540 --> 00:00:15,100
about for years, you know, the Simeon army, it sounds really cool. I can totally picture monkeys

3
00:00:15,100 --> 00:00:20,000
in a data center now. I never really would have thought about that before. Now I actually think

4
00:00:20,000 --> 00:00:25,960
that like having four-year-olds in a data center is actually way more chaotic, but that's a topic

5
00:00:25,960 --> 00:00:32,340
for another day. We wouldn't let that happen, would we? I mean, if you really want to see a random

6
00:00:32,340 --> 00:00:38,340
number generator in action, it's just like put like one of your files in front of a two-year-old

7
00:00:38,340 --> 00:00:46,620
and like watch them do like, there's no order to it. And so it's wonderful. It's like the best

8
00:00:46,620 --> 00:00:54,220
encryption you could ask for. So anyways, like, you know, I've heard about it. It sounded cool.

9
00:00:54,220 --> 00:00:59,820
I'm picturing these monkeys and kind of randomly shutting stuff off. And then hearing you talk last

10
00:00:59,820 --> 00:01:05,820
year, a light bulb kind of went off for me about like, oh, this is a lot more scientific and like

11
00:01:05,820 --> 00:01:11,880
the whole science process and running it as experiments, which ironically makes it feel a lot

12
00:01:11,880 --> 00:01:19,700
less chaotic. It's much more controlled than sort of what the name kind of lets on, but it was sort of

13
00:01:19,700 --> 00:01:25,300
an eye-opening moment for me. And then since then, just as it turns out, I've been hearing more and

14
00:01:25,300 --> 00:01:32,520
more mentions of it from the Cloud Foundry community and some of our pivotal customers and

15
00:01:32,520 --> 00:01:39,820
how they're mentioning like, oh yeah, we're doing chaos engineering in our PCF environment. So it just

16
00:01:39,820 --> 00:01:45,080
kind of prompted me to want to learn a little bit more. And in particular, what I'd love to understand

17
00:01:45,080 --> 00:01:53,500
is whose job is it to be doing chaos engineering? Because I can see this playing out in a couple of

18
00:01:53,500 --> 00:02:00,280
different ways. So I thought Tammy would be a great person to help us unpack that question.

19
00:02:01,680 --> 00:02:08,420
Awesome. Yeah, sounds great. So as you mentioned, chaos engineering to me, it really is a scientific

20
00:02:08,420 --> 00:02:13,580
approach. We like to call it controlled chaos. So you're injecting chaos into your system, but you're

21
00:02:13,580 --> 00:02:19,160
doing it in a really thoughtful, planned way. And it really is a disciplined approach to actually

22
00:02:19,160 --> 00:02:26,360
helping you identify failures before they become outages. So when I like to think of chaos engineering,

23
00:02:26,600 --> 00:02:30,260
to me, it's, you know, you're going to inject failure into a system, you're going to break

24
00:02:30,260 --> 00:02:36,400
things on purpose. Like for example, maybe you think, well, I've set up my database infrastructure

25
00:02:36,400 --> 00:02:45,180
correctly. If I shut down one of my replicas, then a clone should kick off and then a new replica

26
00:02:45,180 --> 00:02:52,240
will be put in its place. Or for example, if I shut down my primary, then it will be replaced by a

27
00:02:52,240 --> 00:02:56,960
replica. And I think that it's going to take exactly this long for it to happen. But the thing is, if

28
00:02:56,960 --> 00:03:02,620
you don't actually inject that failure, inject that shutdown, inject that killing of that machine,

29
00:03:02,840 --> 00:03:07,900
then you don't ever know. And the other thing is it can change by day. So say on a Monday, when you

30
00:03:07,900 --> 00:03:13,480
have like heavy traffic Monday morning, if you're like a heavy traffic business, a lot of people starting

31
00:03:13,480 --> 00:03:19,700
work on Monday, then that could actually make your cloning process slower because you have a lot of

32
00:03:19,700 --> 00:03:26,220
things happening in your system. Whereas it could be much faster on say a Saturday or a Friday even.

33
00:03:26,780 --> 00:03:32,540
But that's why it's really good to be doing these experiments continuously. So that then makes us

34
00:03:32,540 --> 00:03:36,760
think, yeah, like who is the right person to be running these experiments? Because you do want it to

35
00:03:36,760 --> 00:03:42,600
be controlled. You want it to be scientific. You want people to be thinking through what kind of things

36
00:03:42,600 --> 00:03:48,200
they want to learn from doing the experiments. And to me, there are many different teams that actually

37
00:03:48,200 --> 00:03:53,120
should be doing chaos engineering. And I think that actually your entire company should be aware

38
00:03:53,120 --> 00:04:00,620
that it's happening. So when I previously worked at Dropbox, right now I'm working at Gremlin. So

39
00:04:00,620 --> 00:04:05,540
yeah, I've been at Gremlin for a year and a half and I joined as the ninth employee there. But before

40
00:04:05,540 --> 00:04:09,440
that, I was working at Dropbox doing chaos engineering as the SRE manager for storage.

41
00:04:09,720 --> 00:04:14,640
So that's where I did chaos engineering. I was a site reliability engineering manager for all of Dropbox's

42
00:04:14,640 --> 00:04:22,140
storage, which is a lot of storage. It's for 500 million customers, more than that even. And it's all of the block

43
00:04:22,140 --> 00:04:27,660
data and metadata. So there, when we did chaos engineering, we did a lot of different things like

44
00:04:27,660 --> 00:04:36,620
DNS failover. We would do a lot of work on the MySQL machines to shut down replicas, to shut down MySQL

45
00:04:36,620 --> 00:04:42,080
process on the machines and make sure that everything worked correctly in terms of hardware management,

46
00:04:42,080 --> 00:04:49,900
because it was actually a lot of bare metal or on-prem. And to me, that makes sense. Like, you have your team

47
00:04:49,900 --> 00:04:54,660
that knows their system really well. So we're the storage team. We're SREs. We know what the failure could

48
00:04:54,660 --> 00:04:58,760
potentially look like. So we're going to plan out those experiments. We're going to run them and going

49
00:04:58,760 --> 00:05:03,760
to run them continuously, different times of the day, different days, every single week. I used to do

50
00:05:03,760 --> 00:05:08,220
at least three experiments a week and we would continuously repeat them. And then that, you can

51
00:05:08,220 --> 00:05:13,440
learn a lot. And I actually got a 10X reduction in incidents for that specific team by doing that sort

52
00:05:13,440 --> 00:05:17,640
of work, which was awesome. And I talked about that at a few conferences and it was really great.

53
00:05:17,740 --> 00:05:22,460
You know, it went from having high on-call load to really not having much on-call load at all.

54
00:05:22,600 --> 00:05:28,700
And that means you don't have to be getting paged on the weekend or in the evening. You know, you actually get

55
00:05:28,700 --> 00:05:32,400
to have some time to spend with your friends and your family, which is so much better.

56
00:05:33,600 --> 00:05:40,400
And also it reduces burnout, which is great for SREs too. And then that makes you then think, okay,

57
00:05:40,500 --> 00:05:46,560
if one department can get so much value out of chaos engineering, then it makes you think what other teams

58
00:05:46,560 --> 00:05:51,960
could potentially get value out of it. Because I think like a good thing to do actually is to focus on the teams

59
00:05:51,960 --> 00:05:57,760
that have the most problems first. Like that's actually a good area. Like look at the teams that have a high on-call load

60
00:05:57,760 --> 00:06:02,520
where it is a critical system, where there is a large number of outages or incidents.

61
00:06:02,520 --> 00:06:07,420
Like I always say that, like focus on the big wins and the critical systems first,

62
00:06:07,420 --> 00:06:10,100
instead of looking at low hanging fruit.

63
00:06:11,220 --> 00:06:17,020
Because the thing is, you're going to get a lot more value out of it if you focus on your critical systems when you're doing chaos engineering.

64
00:06:18,020 --> 00:06:24,360
And then that generally makes people get a little bit worried because they think, well, you know, it's a critical system.

65
00:06:24,480 --> 00:06:28,220
I don't want to inject failure. But the thing is, you know, failure is always going to happen.

66
00:06:29,220 --> 00:06:32,980
Machines are always going to get rebooted. Containers are always going to randomly die.

67
00:06:33,660 --> 00:06:41,380
All sorts of problems are happening anyway. But if you actually inject that failure in a really controlled way during the middle of the day,

68
00:06:41,460 --> 00:06:45,540
while everyone's there, you can measure it, you can monitor it, you can look and learn from it.

69
00:06:45,540 --> 00:06:48,500
That's so much better than just being too scared about the failure.

70
00:06:49,020 --> 00:06:52,160
Because definitely your, you know, your replica is going to shut down.

71
00:06:53,020 --> 00:06:55,080
And different types of machines are going to shut down.

72
00:06:55,220 --> 00:06:57,700
So that's just something that we all know will happen.

73
00:06:57,780 --> 00:06:59,960
You'll have problems with kernels and all that sort of thing.

74
00:07:00,660 --> 00:07:07,320
So, you know, you mentioned that you were doing this at Dropbox as an SRE and that that's something that an SRE can,

75
00:07:08,440 --> 00:07:16,620
I mean, this might kind of require just maybe illuminating for us and our listeners, you know, an SRE,

76
00:07:17,620 --> 00:07:24,660
what's the relationship between an SRE and an app development team or a service team, right?

77
00:07:24,660 --> 00:07:33,320
Like if I'm running, if I'm providing the MySQL and other database services to, you know, a bunch of enterprise developers,

78
00:07:33,320 --> 00:07:36,860
what's the relationship between an SRE?

79
00:07:36,940 --> 00:07:42,920
And then we have kind of, in our world, we have like kind of the platform engineers and the folks who are providing that underlying platform

80
00:07:42,920 --> 00:07:45,620
and building in a lot of reliability.

81
00:07:45,840 --> 00:07:48,360
Cloud Foundry has a lot of that kind of built in already.

82
00:07:48,820 --> 00:07:54,580
But to me, there seems like there's still these other places for failure that would benefit from this approach.

83
00:07:54,580 --> 00:08:02,340
And the backing services and at the code level itself of the application are two that jump out for me.

84
00:08:03,000 --> 00:08:09,860
And so does this mean that, you know, you should have an SRE dedicated to those services?

85
00:08:10,220 --> 00:08:13,120
Can they float across different services and app teams?

86
00:08:13,200 --> 00:08:14,460
Like what's the relationship there?

87
00:08:14,900 --> 00:08:15,120
Yeah.

88
00:08:15,520 --> 00:08:20,800
I honestly think like every team really should be doing chaos engineering and thinking about failure injection.

89
00:08:21,100 --> 00:08:29,820
We actually have like, you know, a lot of people talk about it from infrastructure, backend systems, databases, critical services.

90
00:08:30,420 --> 00:08:34,300
But then also at Gremlin, our UI engineers do chaos engineering.

91
00:08:34,880 --> 00:08:39,020
So we have built something called ALFI, which is application level failure injection.

92
00:08:39,540 --> 00:08:44,340
And what they can do is we can actually trigger for a failure to be created.

93
00:08:44,340 --> 00:08:50,580
And you can actually then see in staging when this failure occurs, what happens on the UI?

94
00:08:50,580 --> 00:08:53,080
Like what kind of error message is displayed?

95
00:08:53,340 --> 00:08:58,020
Do we have like graceful degradation instead of just the entire page not working?

96
00:08:58,220 --> 00:08:58,960
Something like that.

97
00:08:59,020 --> 00:09:04,820
And one of the nice examples that a lot of people would realize is usually when you're using Netflix,

98
00:09:05,280 --> 00:09:12,320
you have this nice like hero banner when they're trying to, you know, show you some type of new movie or TV show that they want you to watch.

99
00:09:12,380 --> 00:09:15,640
And it's this big hero banner across the top of the screen when you open it up.

100
00:09:15,660 --> 00:09:21,200
And then below there, there are little tiles with different types of TVs and shows and movies to select.

101
00:09:21,660 --> 00:09:28,020
But if there's something wrong with that movie that they're trying to premiere and encourage people to watch,

102
00:09:28,100 --> 00:09:29,780
then they'll just remove the hero banner.

103
00:09:29,980 --> 00:09:31,000
They can just get rid of it.

104
00:09:31,120 --> 00:09:34,940
But it would be so much worse if, you know, it wasn't working.

105
00:09:35,120 --> 00:09:41,760
And then the hero, like people tried to click on it and it didn't load or the image didn't load or there was a big X or, you know,

106
00:09:41,780 --> 00:09:46,720
you can imagine all the different types of things that could go wrong and how it could look bad to the user.

107
00:09:46,900 --> 00:09:51,560
But that's why it's really great when you have your UI engineers thinking about graceful degradation too.

108
00:09:52,380 --> 00:09:55,940
And this is something that, you know, it does take, it's a big cultural shift.

109
00:09:56,340 --> 00:10:06,640
But for me, for example, like Dropbox, it was like, all right, if we trigger this failure as the databases team and we shut down this machine,

110
00:10:06,640 --> 00:10:08,840
we actually would work with other teams.

111
00:10:09,000 --> 00:10:12,680
So we would have a Slack channel and we would have multiple teams in that channel to say,

112
00:10:12,880 --> 00:10:14,240
hey, what errors do you see?

113
00:10:14,320 --> 00:10:16,120
Do you see any problems with your service?

114
00:10:16,420 --> 00:10:21,000
Can you easily reconnect to the database once this new primary comes up?

115
00:10:21,260 --> 00:10:22,560
Did you see any issues at all?

116
00:10:22,900 --> 00:10:27,140
And that was really important because you want to make sure that you squash any problems there.

117
00:10:27,160 --> 00:10:30,540
And you also want to make sure that when new people join a company,

118
00:10:30,540 --> 00:10:34,040
they know how to do on call and they know what errors look like.

119
00:10:34,040 --> 00:10:36,340
They know how to monitor for database issues.

120
00:10:36,460 --> 00:10:37,560
Like it shouldn't be a mystery.

121
00:10:37,720 --> 00:10:42,600
It should actually be that, you know, if you need to connect to your storage layer or your database,

122
00:10:42,820 --> 00:10:44,140
you should know what errors look like.

123
00:10:44,220 --> 00:10:49,320
You should know how to handle errors instead of just saying like, oh, my SQL's gone away.

124
00:10:49,380 --> 00:10:50,140
What does that mean?

125
00:10:50,800 --> 00:10:56,320
So chaos engineering is actually a really great learning opportunity for teams to work across the company.

126
00:10:56,860 --> 00:10:58,420
Really great bonding opportunity.

127
00:10:58,720 --> 00:11:07,520
I've just had so many amazing experiences through using chaos engineering, you know, to help new people on board for on call,

128
00:11:07,920 --> 00:11:11,420
especially if they've never done on call before in that team.

129
00:11:11,500 --> 00:11:16,680
You know, maybe they did a team transfer and this could be someone who is an SRE or someone who's a software engineer.

130
00:11:16,900 --> 00:11:20,440
And you can show them, hey, like this is what the failure commonly looks like.

131
00:11:20,540 --> 00:11:26,360
You can reproduce an old outage and make sure that that outage never happens again and show,

132
00:11:26,600 --> 00:11:29,260
well, we put these things in place and you can see when we trigger this failure.

133
00:11:29,700 --> 00:11:34,680
Last time it did this horrible thing and created this massive outage for maybe days or hours.

134
00:11:34,800 --> 00:11:41,160
But you can see now we actually handle it and the problem won't happen again and we're really confident about it.

135
00:11:41,300 --> 00:11:44,840
So that's a great way to prove that your fixes are actually working.

136
00:11:45,480 --> 00:11:52,860
And then another really interesting place to do chaos engineering is in your CICD pipeline or within your build team.

137
00:11:53,380 --> 00:11:58,980
So you can do something where you can inject chaos engineering experiments every time you do a deploy.

138
00:11:59,460 --> 00:12:04,840
Like, for example, Spinnaker is what's used at Netflix for their CICD pipeline.

139
00:12:04,840 --> 00:12:12,300
And in there you can actually click to say, I want to use chaos monkey for these these deployments.

140
00:12:12,300 --> 00:12:14,340
And then chaos monkey will run for that.

141
00:12:15,140 --> 00:12:15,580
Yeah.

142
00:12:15,720 --> 00:12:18,260
So I think there's a lot of interesting things that we can do.

143
00:12:18,320 --> 00:12:23,100
And we're really just scratching the surface at the moment to see where chaos engineering

144
00:12:23,100 --> 00:12:23,540
can go.

145
00:12:23,660 --> 00:12:29,480
You know, it started off being all about just I'm going to reboot a random machine, which is like a really good start.

146
00:12:29,600 --> 00:12:32,420
I also used to do that when I was working at the National Australia Bank.

147
00:12:32,500 --> 00:12:35,080
So, you know, this isn't just for like tech companies and startups.

148
00:12:35,280 --> 00:12:39,160
I actually first did chaos engineering when I was at the National Australia Bank.

149
00:12:39,160 --> 00:12:43,780
And yeah, we had every single engineer needed to know how to handle failure.

150
00:12:43,960 --> 00:12:49,520
We would do big exercises every quarter, which are commonly called like disaster recovery.

151
00:12:49,660 --> 00:12:52,340
And you need to do it for compliance to keep your banking license.

152
00:12:52,820 --> 00:12:57,760
But that means that, you know, you get hundreds of engineers and other people across the company.

153
00:12:58,540 --> 00:13:12,540
You'll have like people from testing departments, project managers, like TPMs, customer support, customer success, like business owners, all coming to this big location to run the experiments and make sure that failure is handled well.

154
00:13:12,600 --> 00:13:15,840
And then they actually would go around and check all the different experiments off.

155
00:13:16,320 --> 00:13:18,380
Say, yep, you could do a region failover well.

156
00:13:18,540 --> 00:13:23,820
Or if you didn't, then you had to be able to prove that the next time, next quarter, you were able to do it.

157
00:13:23,940 --> 00:13:25,960
So you only had three months to fix it.

158
00:13:26,540 --> 00:13:29,140
So, yeah, I mean, and that was really good.

159
00:13:29,320 --> 00:13:32,340
Like that definitely helped us think in a different way.

160
00:13:32,420 --> 00:13:38,040
Like you're building your services knowing that region failover will happen, which is a different mindset.

161
00:13:38,040 --> 00:13:42,720
And it makes you think when I build something new, I'm going to build it in a way that is more reliable.

162
00:13:42,720 --> 00:13:48,060
But then you also know that, you know, things will go wrong and that's why you need to keep running experiments.

163
00:13:48,720 --> 00:13:48,900
Okay.

164
00:13:49,060 --> 00:13:53,400
So every team kind of needs to be doing some.

165
00:13:53,760 --> 00:14:05,180
Is there a group or does it make sense for any particular team to kind of take a little bit more responsibility for owning and driving the practice?

166
00:14:05,680 --> 00:14:13,280
And, you know, I can imagine like, you know, you mentioned Alfie and kind of injecting that at the UI layer.

167
00:14:13,840 --> 00:14:20,760
But if you've got hundreds of applications, that's a lot of teams to ultimately onboard.

168
00:14:20,760 --> 00:14:23,900
So they're going to be in different stages of having adopted this.

169
00:14:24,440 --> 00:14:24,560
Yep.

170
00:14:24,940 --> 00:14:33,300
What's kind of a best practice that you've seen in terms of starting with one team and expanding?

171
00:14:33,680 --> 00:14:37,580
Like how does this practice kind of spread from one team to another?

172
00:14:37,820 --> 00:14:44,720
Is there kind of a third party in there that owns that maturation process?

173
00:14:45,440 --> 00:14:45,660
Yeah.

174
00:14:45,880 --> 00:14:48,620
So that's a, yeah, it's a really great question.

175
00:14:48,740 --> 00:14:49,900
A lot of people ask this.

176
00:14:49,900 --> 00:14:58,900
And I think from what I've seen in the past, what worked really well is that it is nice to have a team dedicated to owning the outcomes of the chaos engineering experiments.

177
00:15:00,180 --> 00:15:16,020
And at certain companies, you know, that I've worked, it was either the SRE teams or other places, they might be called production engineering or platform engineering, or it could be operations like within any space within DevOps also makes sense.

178
00:15:16,820 --> 00:15:30,020
But the thing there is, what's really important is, you know, if you have a large company, like say when I was at the National Australia Bank, that's like, you know, tens of thousands of people, what you need to do is really be very transparent and open.

179
00:15:30,020 --> 00:15:36,620
And even if you're at a company with hundreds of people or a few thousand people, you know, tell the entire company what chaos engineering is.

180
00:15:36,660 --> 00:15:39,840
And I think the best way to do that, if you can, is through all hands.

181
00:15:40,040 --> 00:15:47,800
So you can just start by having one talk where somebody comes up that says like, all right, I'm going to be the champion of chaos at our company.

182
00:15:48,000 --> 00:15:51,740
And I'm going to tell everyone at the company what chaos engineering is.

183
00:15:51,740 --> 00:16:00,080
And that's really important because you want to make sure that, for example, everyone knows that you're doing something to get good results and to make a positive impact.

184
00:16:00,720 --> 00:16:02,440
And that's a really handy way to do it.

185
00:16:02,520 --> 00:16:05,940
And then you can also have Q&A and you can open it up to everybody.

186
00:16:06,280 --> 00:16:13,220
If you don't do all hands, then it could be something like recording a video and explaining what chaos engineering is and how you're going to do it.

187
00:16:13,560 --> 00:16:15,760
And then sending that out to the entire company.

188
00:16:15,960 --> 00:16:17,860
So people know this is something we're going to do.

189
00:16:17,960 --> 00:16:19,660
This is where it's been done before.

190
00:16:20,080 --> 00:16:22,500
These are the results that those companies have got from it.

191
00:16:22,500 --> 00:16:29,000
You could say like, for me, for a 10x reduction in incidents, reducing on call, reducing downtime.

192
00:16:29,840 --> 00:16:32,100
And there's a lot of great reasons for why to do it.

193
00:16:32,140 --> 00:16:33,880
And then just link to some further reading.

194
00:16:34,340 --> 00:16:40,620
But I think that's a really good place to start is the education of what chaos engineering is and why we want to do it.

195
00:16:41,180 --> 00:16:45,840
And then just get people a little bit more comfortable to the idea of, you know, failure is going to happen.

196
00:16:45,840 --> 00:16:52,520
But if we can control it and inject it proactively to make sure that our systems can handle it, then we'll be in a much better state.

197
00:16:52,900 --> 00:16:54,500
And everyone understands that.

198
00:16:54,620 --> 00:16:57,420
They know that that makes so much sense when you say it to them.

199
00:16:57,800 --> 00:17:00,880
And then when they see it in action, they're like, yeah, this totally makes sense.

200
00:17:01,520 --> 00:17:06,120
And then the next thing that you can do after just doing some education, people will get pretty excited.

201
00:17:06,440 --> 00:17:08,640
And then they'll want to do chaos engineering.

202
00:17:09,100 --> 00:17:13,700
So I think a nice way to start is to have one team get selected to do it first.

203
00:17:13,700 --> 00:17:21,800
And you can run something called a chaos day or alternatively, some people also call it game day.

204
00:17:22,020 --> 00:17:24,940
I like to just call it chaos day because we are injecting chaos.

205
00:17:25,500 --> 00:17:33,360
And what you do there is you just send out an email, say, hey, we're going to record this chaos day session or we're even going to stream it for the entire company to watch.

206
00:17:33,600 --> 00:17:40,980
We're going to inject failure into this system and then we're going to write down some action items after that to say what we need to do to fix it.

207
00:17:41,240 --> 00:17:44,540
But there's a lot of work that you need to do to plan a chaos day.

208
00:17:45,100 --> 00:17:49,520
You know, you need to whiteboard your architecture for the system that you selected.

209
00:17:50,060 --> 00:17:53,420
You need to make sure that you know what your halt or exit conditions are.

210
00:17:53,880 --> 00:17:57,860
And in Gremlin, like, we built a feature that allows you to just click a button to halt all experiments.

211
00:17:57,980 --> 00:17:58,720
That's really important.

212
00:17:59,100 --> 00:18:01,720
You also need to make sure that people understand blast radius.

213
00:18:02,240 --> 00:18:07,820
And that's the idea that you're going to start small and then gradually expand the blast radius for your experiments.

214
00:18:08,380 --> 00:18:10,620
And you also need to have a few prerequisites.

215
00:18:11,260 --> 00:18:20,460
So if you say, for example, pick one team that you want to do those chaos engineering experiments on, you need to also think about, all right, the prerequisites for chaos engineering are monitoring.

216
00:18:20,460 --> 00:18:22,380
You need to have some good monitoring there.

217
00:18:22,880 --> 00:18:25,600
And chaos engineering can help you fix and improve your monitoring.

218
00:18:26,100 --> 00:18:30,040
But that then means that you need to engage the team that owns monitoring at your company.

219
00:18:30,040 --> 00:18:36,140
If you have a dedicated monitoring team, or maybe that is your operations team or your SREs or production engineers.

220
00:18:36,340 --> 00:18:38,960
But you need a rep from there in your chaos day.

221
00:18:39,580 --> 00:18:42,400
And then you also want to have someone that handles alerting.

222
00:18:42,540 --> 00:18:43,600
That could be the same team.

223
00:18:43,680 --> 00:18:45,660
Like, maybe if there's a product that you use for that.

224
00:18:46,240 --> 00:18:51,980
Like, for example, two products could be Datadog and PagerDuty as monitoring and alerting.

225
00:18:51,980 --> 00:18:58,040
And then make sure you have someone in there that can go, all right, actually, yeah, our alert thresholds aren't set correctly.

226
00:18:58,140 --> 00:18:59,260
We need to fix those.

227
00:18:59,640 --> 00:19:01,360
We haven't audited them in a long time.

228
00:19:01,440 --> 00:19:03,780
That might be something that you learn from this experiment.

229
00:19:04,260 --> 00:19:10,320
But then those action items get assigned to a different department and a different team to think more about that holistically.

230
00:19:10,780 --> 00:19:13,220
Well, we learn this from this one chaos day.

231
00:19:13,380 --> 00:19:16,740
We learn that actually, yeah, we found some gaps in our monitoring and alerting.

232
00:19:16,980 --> 00:19:21,320
But let's take a step back and think how we can improve monitoring and alerting across the company.

233
00:19:21,660 --> 00:19:28,160
And then you can enable people and empower them by saying, hey, like, if you want to run your own chaos day, this is how you can do it.

234
00:19:28,560 --> 00:19:34,340
Give you access to a tool, give them education, tell them what kind of experiments they could start with.

235
00:19:34,840 --> 00:19:39,180
So then people feel really empowered to do it and add it into their daily practices.

236
00:19:39,900 --> 00:19:47,000
And just knowing, like, you know, if you measure first, like, this is the most important thing that I've said to everyone is, like, you need to benchmark before you run your experiments.

237
00:19:47,000 --> 00:20:03,820
So you really want to know how many incidents you have, how many outages you have, how long it takes people to work on outages and incidents, you know, how many hours are people spending each week on on-call, are people having to get paid during the weekends and outside of business hours.

238
00:20:04,120 --> 00:20:06,440
All of those things are really important to know first.

239
00:20:06,440 --> 00:20:18,520
And you can even just survey people based on how effective do you think is our handling of incidents within our team and our handling of outages and get them to rate that on a scale.

240
00:20:18,520 --> 00:20:30,580
And then once you've got all those, that data first, then gradually each month, I think for the first three months, definitely rerun those surveys and check again to look at your data and see how you've improved.

241
00:20:30,960 --> 00:20:35,220
And I found that you'll get a massive improvement within three months, definitely.

242
00:20:36,440 --> 00:20:38,200
For me, every single time you always do.

243
00:20:38,740 --> 00:20:41,000
But then, you know, you want to share those results.

244
00:20:41,880 --> 00:20:53,100
So start small in a company, just like the idea of doing chaos engineering, you know, contain the blast radius, start with one team and gradually expand to more teams, but show the results and show how the team benefited.

245
00:20:53,440 --> 00:21:05,480
Help that get that team to then share with other teams, get them to present at all hands, get them to record a video and explain what they learned from doing the chaos engineering experiments and why it was valuable for them.

246
00:21:06,040 --> 00:21:08,620
I think that's just a really, really good thing to do.

247
00:21:08,620 --> 00:21:15,560
And it will get your company just excited about making your services more reliable for everybody that's using them.

248
00:21:17,100 --> 00:21:18,080
Well, it's interesting.

249
00:21:18,320 --> 00:21:21,240
I have two small kids, so every day at my house is chaos day.

250
00:21:22,080 --> 00:21:25,380
But I think Domain can probably relate to that.

251
00:21:25,840 --> 00:21:28,400
So I totally am with you on chaos day.

252
00:21:28,880 --> 00:21:36,680
I do wonder, we talked a little bit before we started recording, just a little bit about Gremlin and kind of the state of the tooling around chaos engineering.

253
00:21:36,800 --> 00:21:38,480
I wonder if you could talk a little bit more about that.

254
00:21:38,820 --> 00:21:43,560
So talk a little bit about the kind of the evolution of the discipline and some of the tools that are built up around it.

255
00:21:43,560 --> 00:21:51,880
So I actually, I recently wrote up an article, which I shared, which is the chaos engineering principles and the history.

256
00:21:52,460 --> 00:21:57,400
So, yeah, that's good because I've looked through all of the history of chaos engineering research for that.

257
00:21:57,900 --> 00:22:02,740
And it was really interesting to just see how things have evolved and where things have come from.

258
00:22:03,400 --> 00:22:13,520
And I would say, like, you know, you see all the time in companies, there is a lot of things happening, a lot of different things going on, a lot of change.

259
00:22:13,960 --> 00:22:16,880
Hopefully, like, your company's doing well and you're getting a lot of customers.

260
00:22:17,000 --> 00:22:17,620
Like, that's ideal.

261
00:22:17,620 --> 00:22:28,760
Like, and I always like to tell people that being an SRE feels like you are trying to build and repair the rocket ship, but from the outside when you're working in a startup.

262
00:22:29,000 --> 00:22:30,220
Like, that's what it feels like.

263
00:22:30,400 --> 00:22:31,860
It doesn't feel like you're inside.

264
00:22:31,940 --> 00:22:33,020
It's all comfortable and great.

265
00:22:33,100 --> 00:22:37,600
It feels like you're in the outside hanging on while you're trying to continuously build it and repair it.

266
00:22:37,800 --> 00:22:41,820
And that is just, like, to me, the best way to describe site reliability engineering.

267
00:22:41,980 --> 00:22:45,080
It's like, you know, there's intense and there's like, whoa.

268
00:22:45,560 --> 00:22:48,080
But, yeah, so that's a good place to start.

269
00:22:48,180 --> 00:22:55,300
But the thing is chaos engineering can help you reduce that feeling because it actually gives you a lot of control around the chaos.

270
00:22:55,960 --> 00:22:56,740
You can look at it.

271
00:22:56,800 --> 00:22:59,560
You can take a step back and go, okay, like, what are our biggest problems?

272
00:22:59,640 --> 00:23:00,540
Like, I like to say that.

273
00:23:00,620 --> 00:23:03,880
What are our top five issues that we have right now that we need to fix?

274
00:23:04,100 --> 00:23:05,780
Let's get the data to prove that.

275
00:23:05,840 --> 00:23:07,560
And then let's just try and cut them down.

276
00:23:07,880 --> 00:23:10,080
And I like to always mention Pareto principle.

277
00:23:10,320 --> 00:23:11,680
That's one of my favorite things ever.

278
00:23:11,680 --> 00:23:18,940
So when I was working at the National Australia Bank, we always used to solve problems using the Pareto principle or 80-20 rule.

279
00:23:19,800 --> 00:23:21,600
And, yeah, the 80-20 rule.

280
00:23:21,720 --> 00:23:25,620
So it's just like, look at your, you know, your biggest problems.

281
00:23:25,900 --> 00:23:27,280
And you always have this long tail.

282
00:23:27,400 --> 00:23:34,260
So if you can say, all right, 80% of our problems are caused by 20% of the issues that we're currently being paged for.

283
00:23:34,400 --> 00:23:40,460
Like, that's usually what it's actually always like when you look up Pareto principle, you'll see, like, wow, that's interesting when you're looking at data.

284
00:23:41,240 --> 00:23:46,680
But then in terms of the history of chaos engineering, I don't think that was ever applied.

285
00:23:47,040 --> 00:23:50,880
Like, that's just the way that I thought about it because of coming from a banking background.

286
00:23:51,200 --> 00:23:53,680
And you're always looking at so much data.

287
00:23:53,920 --> 00:24:02,140
And when you work in a bank, everyone's really encouraged to prove with data first what you want to do and what improvement you're going to make.

288
00:24:02,200 --> 00:24:03,060
You have to benchmark.

289
00:24:03,340 --> 00:24:05,540
Like, you won't be able to do something unless you benchmark it.

290
00:24:05,600 --> 00:24:16,600
You have to do a lot of really rigorous things like change records and say, these are all the different types of things that I've done to make sure that my software is reliable before I put it in production.

291
00:24:16,600 --> 00:24:21,040
And if you can't list those out, then they just won't let you put your code in production.

292
00:24:21,260 --> 00:24:27,880
So it's like, they'll actually call you on the phone and say, hey, like, you know, what you wrote down wasn't good enough for your change record.

293
00:24:28,000 --> 00:24:34,060
Go back and do more work to make sure that it's reliable before we let you put it into internet banking or foreign exchange trading.

294
00:24:34,680 --> 00:24:39,300
I feel like everyone should work at a bank in some way.

295
00:24:39,300 --> 00:24:40,740
Yeah, it's actually really handy.

296
00:24:40,740 --> 00:24:42,020
At some point in their life.

297
00:24:42,800 --> 00:24:53,580
Just, I mean, I, I worked at a bank, but I mean, and I totally, I was in a, I was in the research department, but I totally know what you mean in terms of like this.

298
00:24:53,740 --> 00:24:57,160
The, the paper trail is like very real.

299
00:24:57,640 --> 00:24:59,320
Um, and you learn a lot of rigor.

300
00:24:59,540 --> 00:25:00,480
Everything's linked to your name.

301
00:25:00,580 --> 00:25:00,880
Yeah.

302
00:25:01,000 --> 00:25:03,800
And you know that everything's getting recorded against your name.

303
00:25:03,800 --> 00:25:09,620
I actually once got a fine from the, um, ombudsman because a mortgage wasn't able to be processed.

304
00:25:09,620 --> 00:25:11,500
And then the customer rang and complained.

305
00:25:11,500 --> 00:25:15,660
And then that name got associated with that fine got associated with my name.

306
00:25:15,660 --> 00:25:18,940
So like the, uh, CIO came and said that to me.

307
00:25:18,960 --> 00:25:20,780
I was like, well, okay, this is serious.

308
00:25:21,120 --> 00:25:23,420
And yeah, it's just an interesting thing.

309
00:25:23,480 --> 00:25:26,380
Very different environment, but I think I learned a lot from that.

310
00:25:26,380 --> 00:25:33,800
And then when we look at chaos engineering and where it began, it was more about, okay, like let's just shut down random machines on AWS.

311
00:25:34,220 --> 00:25:34,620
Yeah.

312
00:25:34,620 --> 00:25:36,980
Which like makes a lot of sense when you're doing a migration.

313
00:25:36,980 --> 00:25:42,800
Cause it was from back when Netflix, um, and Bruce Wong is the one who created the term chaos engineering.

314
00:25:43,140 --> 00:25:46,640
He then went to work at Twilio and now he works at Stitch Fix.

315
00:25:46,700 --> 00:25:47,800
He's a good friend of ours.

316
00:25:47,960 --> 00:25:53,220
And yeah, there was this big migration, which was happening, which, um, Adrian Cockroft worked on the migration.

317
00:25:53,220 --> 00:25:54,280
Now he's at AWS.

318
00:25:54,280 --> 00:25:56,580
And he actually spoke at our conference.

319
00:25:56,740 --> 00:26:01,360
He did the opening keynote at ChaosConf about chaos engineering and how it's changed over the years.

320
00:26:01,500 --> 00:26:09,440
But the idea there was they didn't want, um, Netflix engineers to, you know, think that these machines are going to be trustworthy.

321
00:26:09,620 --> 00:26:11,000
They were going to be up all the time.

322
00:26:11,120 --> 00:26:19,100
And like years and years ago, people used to say that machines were good if they had like a high number, high amount of days for uptime, which is like just really bad idea.

323
00:26:19,160 --> 00:26:22,180
You know, it's better actually if they constantly cycle.

324
00:26:22,300 --> 00:26:23,080
So then they're healthy.

325
00:26:23,160 --> 00:26:31,300
Like I always used to think, I like to think of it as a healthy fleet, which means like they've been washed or like looked after when you think about it, like cattle.

326
00:26:31,500 --> 00:26:36,040
Um, you need to do hardware, like upgrades and you need to do security patches.

327
00:26:36,040 --> 00:26:39,160
So having really high uptime actually isn't very good.

328
00:26:39,720 --> 00:26:43,980
Um, so that was just changing that mindset and saying, okay, everyone just get ready.

329
00:26:43,980 --> 00:26:46,260
Like these machines, they're not ours.

330
00:26:46,380 --> 00:26:47,500
We don't control them.

331
00:26:47,560 --> 00:26:48,580
They're owned by AWS.

332
00:26:48,800 --> 00:26:49,840
They're probably going to go down.

333
00:26:49,920 --> 00:26:51,520
They're probably going to bounce every now and again.

334
00:26:51,680 --> 00:26:53,460
We need to make sure that we handle that.

335
00:26:53,940 --> 00:26:57,120
Um, and so it was just killing random machines.

336
00:26:57,580 --> 00:27:09,160
And then after that, um, at the same time, or like even a little bit before Colton and Forney were working at Amazon and they were building an internal tool to do chaos engineering.

337
00:27:09,400 --> 00:27:17,860
And they were looking at actually injecting failure on, um, Amazon because they knew that all these people are going to come and start using Amazon.

338
00:27:18,000 --> 00:27:19,520
This is like many, many years ago now.

339
00:27:19,520 --> 00:27:24,980
And they wanted to make sure that it was reliable and they built this tool and gave it to everybody across Amazon.

340
00:27:24,980 --> 00:27:27,800
So, and people still use that tool internally now.

341
00:27:28,560 --> 00:27:31,580
Um, and yeah, I think that's pretty cool that they did that.

342
00:27:31,580 --> 00:27:40,680
They had a lot of, um, foresight to think this is what people will need and we need to give them like a really nice UI, um, an agent that can run on the machines.

343
00:27:41,160 --> 00:27:44,080
We want everyone to be able to quickly adopt it and use it.

344
00:27:44,460 --> 00:27:51,980
Um, and then after that, then Colton went to work at Netflix and he built Fit, which is a failure injection platform.

345
00:27:52,600 --> 00:27:59,860
And then that was more about like taking the learnings from what he'd done in the past and then applying those to Netflix as well.

346
00:27:59,860 --> 00:28:11,240
And the idea there is, um, the idea is, you know, okay, I want to inject failure, but I want to make it very narrow in scope, kind of like heart surgery, very precise.

347
00:28:11,760 --> 00:28:25,180
Um, and the idea is you could say, I want to inject failure on just my PlayStation 4 at my house, just for my user, my user ID, instead of shutting down, you know, lots of different machines or something like that.

348
00:28:25,180 --> 00:28:27,700
You're just injecting failure in a very precise way.

349
00:28:28,100 --> 00:28:35,780
Um, and then, yeah, at the same time I was working at Dropbox doing chaos engineering on databases, which was a really new field.

350
00:28:35,960 --> 00:28:39,480
No one had done that before because they're all a bit scared to inject failure on a database.

351
00:28:39,700 --> 00:28:45,320
But, you know, I, I mean, I just have this idea that I've been working on databases for a long time.

352
00:28:45,320 --> 00:28:49,820
Um, and when you have, you know, that they're going to fail all the time, that's just what happens.

353
00:28:49,820 --> 00:28:53,760
Like database machines will shut down and you need to have proper replication.

354
00:28:54,060 --> 00:28:55,180
You need to have good backups.

355
00:28:55,280 --> 00:28:56,500
You need to test your backups.

356
00:28:56,620 --> 00:28:58,480
You need to have cloning in place.

357
00:28:59,040 --> 00:29:05,120
Um, that's all really important, but this is a thing that most people were too scared to touch.

358
00:29:05,120 --> 00:29:13,220
And then, yeah, I went and spoke about it at some conferences, what we were doing and, and the results that we'd got and people were like, wow, that's actually really cool.

359
00:29:13,220 --> 00:29:17,820
Um, so a lot more people started to do that as well with MySQL, which I think is great.

360
00:29:18,100 --> 00:29:23,640
And Netflix had done some chaos engineering with Cassandra on AWS, which is also good.

361
00:29:23,780 --> 00:29:35,100
But yeah, there's a lot of things that have happened over the years where it's like, gradually people have felt more confident and tried to see, okay, can I use chaos engineering to fix this issue that I have?

362
00:29:35,240 --> 00:29:37,340
And then they've got great results from it and done it.

363
00:29:37,340 --> 00:29:45,820
Um, and then when I first saw the gremlin tooling, that was after I'd been at Dropbox for two and a half years and it just looked awesome to me.

364
00:29:45,860 --> 00:29:48,240
Like I had no idea that something could be so great.

365
00:29:48,300 --> 00:29:51,120
When I first did try the demo, I was like, wow, this is great.

366
00:29:51,260 --> 00:29:56,080
Cause it had 11 attacks built in and it's got a UI, a CLI and an API.

367
00:29:56,940 --> 00:30:00,420
And then it also has Alfie, which is the application level failure injection.

368
00:30:00,700 --> 00:30:12,300
And some of those attacks are packet loss, packet corruption, latency injection, um, time travel, like changing the clock time, lots of different things.

369
00:30:12,300 --> 00:30:16,280
And there's a lot of experiments there that, you know, I'd never even thought of and I'd never heard of.

370
00:30:16,500 --> 00:30:20,140
No one had ever spoken about them except for Colton and Forney.

371
00:30:20,700 --> 00:30:23,920
Um, so yeah, I just thought, wow, they're really ahead of where everybody is.

372
00:30:24,440 --> 00:30:31,320
And it's been awesome to be able to, uh, use gremlin and learn more about how we can actually identify failure.

373
00:30:31,320 --> 00:30:35,640
And we're identifying failure every single day within our own infrastructure.

374
00:30:35,920 --> 00:30:41,240
We're able to use gremlin to make it more robust and resilient before we say move something into production.

375
00:30:41,240 --> 00:30:43,820
We do a lot of work with gremlin on staging first.

376
00:30:44,340 --> 00:30:45,580
And that's also a common question.

377
00:30:45,580 --> 00:30:50,460
People always say, is it okay to do chaos engineering on staging or in my local development environment?

378
00:30:50,460 --> 00:30:51,520
I'm like, yeah, totally.

379
00:30:51,660 --> 00:31:00,220
Like you can even start by doing it on your own laptop, using Docker for Mac, uh, start doing some chaos engineering there by learning about what it's like.

380
00:31:00,220 --> 00:31:08,640
Cause I think it does have, um, you know, you need to learn what it is like, how it works and gradually ramp up your chaos engineering practice.

381
00:31:09,220 --> 00:31:14,460
Because, you know, I always tell everyone, don't just go in tomorrow and be like, yeah, let's shut down the data center.

382
00:31:14,600 --> 00:31:16,240
Like that's the worst way to do it.

383
00:31:16,300 --> 00:31:21,080
You know, you want to start slow, um, and just contain that blast radius for your own experiments.

384
00:31:21,660 --> 00:31:21,960
Yeah.

385
00:31:23,720 --> 00:31:24,040
Yeah.

386
00:31:24,040 --> 00:31:44,220
So that, I mean, it's kind of interesting, your explanation of the really targeted failure injection versus sort of the, maybe the long-term state, which I think you described was sort of the Netflix model of, you know, not wanting the engineers to treat infrastructure as reliable, which is really just more of a mindset shift.

387
00:31:44,220 --> 00:31:53,220
But, you know, these kind of chaos engineering practices reinforce that mindset because, you know, they, it could affect them.

388
00:31:53,220 --> 00:31:53,400
Right.

389
00:31:53,440 --> 00:31:58,680
So it's on the one hand, you don't want to start with, Hey, we're just going to shut something down and I hope you're ready.

390
00:31:59,460 --> 00:32:02,400
But you do kind of want to get to that point.

391
00:32:02,400 --> 00:32:19,200
It sounds like, I mean, everyone should start small, teams should start small, but is the end goal to be able to have, Hey, this whole company is now operating this way where if we say, we're going to, we're going to attempt a really big failure, um, that could have a really large blast radius.

392
00:32:19,360 --> 00:32:25,180
So like, hopefully everyone's ready for that, but yeah, you definitely want to get there.

393
00:32:25,260 --> 00:32:25,660
Exactly.

394
00:32:25,660 --> 00:32:32,780
Like you said, the thing there is like, you know, to think of, um, all right, I start small with my blast radius and then what does it look like when it's really big?

395
00:32:32,780 --> 00:32:38,060
Like a really big example is your region failover, um, or data center failover.

396
00:32:38,320 --> 00:32:40,400
So that's always really good, right?

397
00:32:40,400 --> 00:32:49,240
If you have, um, an AWS region on the East coast and on the West coast, and then you feel confident enough that you can shut down an entire region.

398
00:32:49,360 --> 00:32:53,740
And that, you know, that's happened like S3 on, um, had a massive outage.

399
00:32:53,860 --> 00:32:58,440
A lot of the internet was down, um, and having problems because S3 wasn't working in a region.

400
00:32:58,840 --> 00:33:00,060
And that was a few years ago.

401
00:33:00,060 --> 00:33:01,920
I was on call for that while I was at Dropbox.

402
00:33:01,960 --> 00:33:06,020
I was the, um, incident manager on call for all of Dropbox at the time.

403
00:33:06,020 --> 00:33:08,020
And yeah, it was a big outage.

404
00:33:08,020 --> 00:33:08,840
It was interesting.

405
00:33:08,960 --> 00:33:10,100
It went for five hours.

406
00:33:10,180 --> 00:33:14,220
It was a really long time as well, but that's the thing, you know, this failure is going to happen.

407
00:33:14,480 --> 00:33:16,120
So you definitely want to be prepared for it.

408
00:33:16,180 --> 00:33:18,160
You want to be able to do region failover.

409
00:33:18,680 --> 00:33:25,260
Um, but the thing is like most AWS services are not built with region failover out of the box.

410
00:33:25,340 --> 00:33:28,940
So you're going to have to make sure that you make that happen yourself.

411
00:33:29,660 --> 00:33:30,700
That's a big thing.

412
00:33:31,060 --> 00:33:31,300
Yeah.

413
00:33:31,300 --> 00:33:44,740
Um, no, we definitely see that a lot with, you know, if what you stitch together versus saying, and with Cloud Foundry, you get a lot of that automated, but it's still something that you want to test.

414
00:33:45,320 --> 00:33:46,420
Um, yes, exactly.

415
00:33:46,420 --> 00:34:10,800
Um, so another question that I have is kind of, you know, thinking about how much harder is it to adopt and introduce these practices with like a legacy monolith versus something that's already say 12 factor kind of designed, if you will, or, uh, you know, as a microservices architecture.

416
00:34:10,800 --> 00:34:13,160
Like, what does that journey look like?

417
00:34:13,480 --> 00:34:18,660
Do folks have to kind of refactor before they can effectively do this?

418
00:34:18,720 --> 00:34:22,200
Or is it, does it just mean that they're going to have to go about it in a different way?

419
00:34:23,480 --> 00:34:23,680
Yeah.

420
00:34:23,840 --> 00:34:27,260
Uh, I love that question too, because a lot of people ask it all the time.

421
00:34:27,260 --> 00:34:33,060
Um, and the thing is like, I've done chaos engineering on monoliths and also on microservices.

422
00:34:33,440 --> 00:34:34,480
So both.

423
00:34:34,880 --> 00:34:40,360
And, um, I think actually chaos engineering is really good when you're doing any type of migration.

424
00:34:40,360 --> 00:34:44,400
And usually every single company is having a migration occurring at any time.

425
00:34:44,500 --> 00:34:45,720
That's like always happening.

426
00:34:45,800 --> 00:34:46,900
There's something that you're doing.

427
00:34:47,320 --> 00:34:49,900
Um, maybe you're migrating from monolith to microservices.

428
00:34:49,900 --> 00:34:56,960
You're migrating to a different type of storage layer or database, or you're upgrading your API or something like that.

429
00:34:56,960 --> 00:35:04,360
Or maybe you're doing a database migration or, you know, deprecating some type of system, moving everything one place to another.

430
00:35:05,020 --> 00:35:08,820
So this is a big thing that customers can use chaos engineering for.

431
00:35:09,380 --> 00:35:15,820
Um, and obviously that's like a really scary thing when you're doing a big migration, because you need to make sure that you think about like data consistency.

432
00:35:16,180 --> 00:35:17,680
You don't lose data.

433
00:35:18,240 --> 00:35:25,920
Um, I know, you know, my space just recently had a thing where they lost like 50 million songs, um, during a data migration.

434
00:35:26,160 --> 00:35:27,460
So yeah, that's the thing.

435
00:35:27,460 --> 00:35:31,420
Like you can do chaos engineering to help you prevent data loss.

436
00:35:31,420 --> 00:35:33,460
Like that's a big reason that I do chaos engineering.

437
00:35:33,980 --> 00:35:39,080
Um, because you want to make sure that you're injecting that failure to see like, if this happens, would we lose data?

438
00:35:39,500 --> 00:35:42,360
Um, but you do it not in production first, right?

439
00:35:42,400 --> 00:35:49,060
You need to, and you really think through it and document it, make it a very, very, um, controlled experiment.

440
00:35:49,360 --> 00:35:51,000
Cause obviously that's quite scary.

441
00:35:51,000 --> 00:35:55,340
If you like lose all of your data for your customers or you have data consistency issues.

442
00:35:55,600 --> 00:36:02,640
Um, but I think that would be really good for people, you know, with cloud foundry when you're thinking, okay, this is how I expect my system to operate.

443
00:36:03,300 --> 00:36:09,500
Um, this is where I want it to be, but then you actually use chaos engineering to test that it is going to work for you.

444
00:36:09,780 --> 00:36:11,780
That's just like the best way to do it.

445
00:36:12,400 --> 00:36:12,620
Yeah.

446
00:36:12,740 --> 00:36:31,400
I mean, I think sort of maybe a little bit of where the question comes from for me is that often I, I hear from folks where they, when they're dealing with a legacy monolith, one of the challenges is that they're so behind on doing anything with that.

447
00:36:31,400 --> 00:36:42,700
Because it's just, it's just really hard for them to do the right thing in any way because they're, it's just so brittle.

448
00:36:42,700 --> 00:36:46,040
And so they're just so they're drowning in updates.

449
00:36:46,040 --> 00:36:46,380
Right.

450
00:36:46,540 --> 00:36:46,900
Yeah.

451
00:36:47,040 --> 00:37:08,840
And so working through those updates, kind of thinking about like, you have to kind of bite off in some ways a lot at the same time, but then you end up in a place where it becomes easier to not only make changes for the functionality, but also to make the kinds of changes where it's like, we're going to test these changes in new ways.

452
00:37:08,840 --> 00:37:15,720
And you have your, your CI pipeline set up so that you're running all kinds of tests, chaos and otherwise.

453
00:37:17,040 --> 00:37:17,440
Yep.

454
00:37:17,700 --> 00:37:18,040
Yeah.

455
00:37:18,180 --> 00:37:26,380
And I think a lot of that is really like a mindset shift to often people feel overwhelmed at the amount of work that you need to do.

456
00:37:26,380 --> 00:37:30,820
But then the number one thing that you have to be really good at is prioritization.

457
00:37:31,140 --> 00:37:36,180
So that's where Pareto principle comes in again and you go, all right, what are the top five things that we need to do?

458
00:37:36,880 --> 00:37:38,960
Let's pick off that top number one thing.

459
00:37:39,180 --> 00:37:41,420
Let's make sure that we have the right engineers in the room.

460
00:37:41,540 --> 00:37:42,440
We have enough people.

461
00:37:42,540 --> 00:37:43,940
We've talked about headcount.

462
00:37:44,320 --> 00:37:45,220
We've talked about budget.

463
00:37:45,360 --> 00:37:46,920
We've talked about how long this is going to take.

464
00:37:46,980 --> 00:37:53,760
You then take that, I think, to your VP of engineering or your CTO and you present that and say, these are the top problems that we want to solve.

465
00:37:53,900 --> 00:37:55,040
This is how we're going to solve them.

466
00:37:55,100 --> 00:37:56,840
This is how many people we need to solve them.

467
00:37:57,340 --> 00:38:04,220
And to me, that's about like stepping up and being really brave and courageous, which I think is something we need to do a lot more in engineering.

468
00:38:04,600 --> 00:38:13,820
Because there's always so many things to do, but if you can prioritize and you can be brave and you can take that to your leadership team and say, look, like, I know I can do this.

469
00:38:13,900 --> 00:38:14,840
I believe in myself.

470
00:38:15,060 --> 00:38:18,100
I'm going to be able to get us a massive reduction in problems.

471
00:38:18,440 --> 00:38:21,080
I'm going to make sure that this outage never happens again.

472
00:38:21,240 --> 00:38:22,680
I'm going to reduce that on call.

473
00:38:22,940 --> 00:38:23,880
This is my goal.

474
00:38:24,180 --> 00:38:28,160
And then you actually get your entire team behind that to work together to make it happen.

475
00:38:28,280 --> 00:38:30,680
Like, you're going to feel awesome at the end of it.

476
00:38:30,680 --> 00:38:35,260
And I think we just have to believe in ourselves and our skills a lot more that we can actually do it.

477
00:38:35,860 --> 00:38:40,660
That's something that I've always been like myself, just personally, as a personality trait.

478
00:38:40,760 --> 00:38:52,500
You know, I used to be a sponsored skateboarder and I love practicing and getting really good at something and just like being able to say, okay, today I'm going to ollie down two stairs, which is like jump down two stairs.

479
00:38:52,540 --> 00:38:54,420
Next week, I'm going to go down five stairs.

480
00:38:54,560 --> 00:39:00,420
And, you know, you practice and you fall over and sometimes it hurts, but you have to get back up and you have to keep trying.

481
00:39:01,180 --> 00:39:03,360
And yeah, to me, that's like a cultural shift.

482
00:39:03,760 --> 00:39:17,120
But a lot of SREs, you know, as a speciality, it's, you'll notice that SREs and production engineers like really care deeply about reliability and don't ever want to lose data and understand that that's like so important.

483
00:39:17,120 --> 00:39:20,440
Because if you lose data, then you don't have a company anymore.

484
00:39:20,440 --> 00:39:29,440
Like if you've lost your customers data and if you don't have a product that's up and available, well, then your customers can't use your product and they're going to lose trust in it.

485
00:39:29,760 --> 00:39:38,780
So, yeah, I think we're always really pushed by that, that there's like deep empathy for customers and just knowing that you have to step up and just make it happen.

486
00:39:42,400 --> 00:39:42,760
Great.

487
00:39:42,880 --> 00:39:43,060
Okay.

488
00:39:43,260 --> 00:39:44,560
Well, we're getting close on time.

489
00:39:44,560 --> 00:39:46,720
So why don't you tell us a little bit more about Gremlin?

490
00:39:46,860 --> 00:39:56,640
I think our listeners love you a little bit more about the company and what you do and how, maybe how they can get in touch and potentially take a look at some of your, at your approach.

491
00:39:57,420 --> 00:39:57,860
Yeah, sure.

492
00:39:57,960 --> 00:39:59,980
So we recently launched Gremlin free.

493
00:39:59,980 --> 00:40:03,320
So you can actually now use Gremlin totally for free, which is cool.

494
00:40:03,420 --> 00:40:09,420
So if you go to gremlin.com, you can use a few of our attacks, which are shutdown and CPU.

495
00:40:09,800 --> 00:40:11,600
And it works across all clouds.

496
00:40:11,840 --> 00:40:13,060
So it's cloud agnostic.

497
00:40:13,060 --> 00:40:19,340
And you can also attack instances or hosts, or you can attack containers or Kubernetes pods.

498
00:40:19,600 --> 00:40:20,480
So that's really cool.

499
00:40:20,740 --> 00:40:21,700
I think that's great.

500
00:40:21,760 --> 00:40:23,440
That only happened just recently now.

501
00:40:24,000 --> 00:40:30,700
And it's been exciting to see everybody start to use chaos and engineering a lot more across many different companies.

502
00:40:30,700 --> 00:40:35,840
And I've seen, been seeing people reach out to me on Twitter, just sharing what they're doing, which is really cool.

503
00:40:35,840 --> 00:40:43,740
If you look up the hashtag free the gremlins, you'll see some cool tweets from people where they've shared like, these are the experiments that I'm running at my company.

504
00:40:43,740 --> 00:40:45,860
And here's what I've learned so far.

505
00:40:45,860 --> 00:40:49,600
So yeah, I definitely encourage you to try out Gremlin because you can.

506
00:40:49,880 --> 00:40:52,560
And yeah, it's totally free now, which is awesome.

507
00:40:53,320 --> 00:41:00,140
And then apart from that, like, you know, the biggest thing that I'm doing right now this week, you know, we're in a startup.

508
00:41:00,260 --> 00:41:01,540
I started, I was the ninth employee.

509
00:41:01,640 --> 00:41:02,720
Now we're over 50 people.

510
00:41:03,180 --> 00:41:08,840
But this week I'm actually running a big game day internally or a chaos day on our own infrastructure.

511
00:41:09,400 --> 00:41:12,840
So yeah, we're doing that this Friday and really looking forward to that.

512
00:41:12,840 --> 00:41:16,900
And we're actually inviting the whole company to come and watch and learn.

513
00:41:17,040 --> 00:41:25,340
So people from every single department are invited to come along and understand how chaos and engineering works and why it's important and what we're learning from it.

514
00:41:25,340 --> 00:41:32,940
And we're going to be focusing on improving our monitoring and alerting to make sure that everything's set correctly and that we feel really good about it.

515
00:41:33,420 --> 00:41:35,500
So yeah, that's a big thing that I'm doing this week.

516
00:41:35,940 --> 00:41:41,000
Is there like, are there snacks, you know, like if you're in the audience, like is there popcorn?

517
00:41:41,600 --> 00:41:43,380
I'm just picturing like, it's like a shot.

518
00:41:43,380 --> 00:41:45,920
Yes, I think it should be like that.

519
00:41:45,920 --> 00:41:46,500
You got a blanket.

520
00:41:46,960 --> 00:41:50,060
You're like, oh my God, oh my God, is that going to fail?

521
00:41:50,160 --> 00:41:53,060
And then it's like, oh, look, was there, oh no, there wasn't an alert.

522
00:41:53,620 --> 00:41:54,280
There wasn't an alert.

523
00:41:54,800 --> 00:41:55,140
I didn't see that.

524
00:41:55,140 --> 00:41:56,540
Like that's what it feels like.

525
00:41:56,600 --> 00:41:57,460
That's what it feels like.

526
00:41:57,500 --> 00:42:05,500
Even when you're doing it on staging or locally, like in your own development environment, I think it's good to have a theme and make it something that is really memorable.

527
00:42:05,500 --> 00:42:07,660
And everyone's excited to come along too.

528
00:42:07,820 --> 00:42:09,820
And they feel like they're going to learn something from it.

529
00:42:10,100 --> 00:42:19,700
So yeah, when I, you know, you know, because we're gremlins, so we can use like the gremlin theme of having all these little gremlins going through and, you know, have some green cupcakes.

530
00:42:19,820 --> 00:42:20,800
We've definitely done that before.

531
00:42:21,280 --> 00:42:26,380
But like, I mean, yeah, you need to think of what sort of incentives you're going to have to get everyone to show up.

532
00:42:26,440 --> 00:42:26,920
That's for sure.

533
00:42:27,320 --> 00:42:27,560
Yeah.

534
00:42:27,740 --> 00:42:35,480
But like, then you don't, you, I don't know if you ever saw the Seinfeld episode where Kramer goes to watch like some kind of surgery.

535
00:42:35,480 --> 00:42:36,980
You mentioned heart surgery before.

536
00:42:37,140 --> 00:42:39,460
So this is, of course, where my brain immediately goes.

537
00:42:39,940 --> 00:42:43,300
It's not like a real surgery, but like a surgery on Seinfeld.

538
00:42:43,840 --> 00:42:52,840
And he's like eating junior mints and he drops a junior mint, like into the patient, into like the open cavity or something.

539
00:42:53,940 --> 00:42:55,560
So you definitely don't want that.

540
00:42:56,020 --> 00:42:58,260
You don't need to have some guards.

541
00:42:58,420 --> 00:43:04,800
The two-way mirrors or something so that there's no, you know, like junior mints getting dropped in.

542
00:43:04,800 --> 00:43:07,220
Yeah, it's a good way, good way to think about it though.

543
00:43:07,220 --> 00:43:16,600
Cause you know, when they first did chaos engineering at AWS, like Jesse Robbins used to do this thing where he'd actually go in the data center and just pull out random cables.

544
00:43:16,600 --> 00:43:18,880
Like that was how they first did it before they built anything.

545
00:43:19,020 --> 00:43:21,200
And he's the person that coined the term game day.

546
00:43:21,620 --> 00:43:30,040
So yeah, that's where it started, where it was like, you know, wild and all sorts of different things were happening and nobody knew what he was doing in the data center.

547
00:43:30,580 --> 00:43:33,040
But yeah, mostly now we do it remotely.

548
00:43:33,460 --> 00:43:34,820
So they're on like a zoom.

549
00:43:35,820 --> 00:43:36,820
That's a lot safer.

550
00:43:37,700 --> 00:43:37,860
Yeah.

551
00:43:38,860 --> 00:43:39,640
Point cables.

552
00:43:39,860 --> 00:43:40,520
I like it.

553
00:43:41,700 --> 00:43:43,800
The four-year-olds could definitely help out with that.

554
00:43:45,120 --> 00:43:49,340
I don't know why TV references are, we've had one so far.

555
00:43:49,340 --> 00:43:54,780
I'm thinking of the office episode when Dwight pulls the fire alarm and everybody thinks it's a real fire.

556
00:43:55,120 --> 00:43:55,820
I'm thinking.

557
00:43:56,980 --> 00:43:58,620
That was chaos.

558
00:43:58,780 --> 00:43:59,800
It resulted in chaos.

559
00:44:00,440 --> 00:44:01,860
Yeah, that's another big thing.

560
00:44:02,020 --> 00:44:09,600
Like, you know, there was, there was that, um, the alert in Hawaii where they said there was like a massive issue there, a natural disaster.

561
00:44:09,780 --> 00:44:12,680
But then it actually wasn't like, that's a form of chaos.

562
00:44:12,680 --> 00:44:19,340
You know, you want to make sure that you're protected against the wrong alarms going off because that can be really bad.

563
00:44:19,540 --> 00:44:19,940
Yeah.

564
00:44:20,360 --> 00:44:21,840
I mean, it's so interesting to me.

565
00:44:21,840 --> 00:44:24,980
I think a lot about chaos engineering through movies as well.

566
00:44:25,440 --> 00:44:29,380
There's been a lot of really interesting examples and I think it's good for us to go.

567
00:44:29,520 --> 00:44:32,500
Okay, yeah, like failure happens and we just need to be ready for it.

568
00:44:32,780 --> 00:44:33,040
Yeah.

569
00:44:33,300 --> 00:44:34,820
It's actually like less dramatic.

570
00:44:35,480 --> 00:44:41,760
Um, but you know, then we can just go back to our sitcom lives instead of the drama lives.

571
00:44:41,760 --> 00:44:48,140
You know, as much as you want to give the Academy Awards to the dramas, you don't actually want to be living that life.

572
00:44:48,520 --> 00:44:48,880
Exactly.

573
00:44:49,320 --> 00:44:50,420
Like, so true.

574
00:44:50,500 --> 00:44:57,760
Like that, you know, I think as someone who's been on call for many, many years, like my favorite type of weekend is one where I don't get paged.

575
00:44:58,180 --> 00:44:59,600
Like that is the dream.

576
00:45:00,100 --> 00:45:06,760
Um, like one where I can actually, you know, go out, have fun with friends, not having to be getting paged all the time.

577
00:45:06,860 --> 00:45:07,560
That's the best.

578
00:45:08,560 --> 00:45:09,820
Get your weekends back.

579
00:45:10,620 --> 00:45:10,960
Exactly.

580
00:45:10,960 --> 00:45:23,120
It was a pleasure talking to you, um, and learning more and kind of just getting, getting a clear picture of sort of where and when and how and who should be doing chaos engineering.

581
00:45:23,440 --> 00:45:28,780
So, uh, thank you for sharing your, your wisdom with us, Tammy, and, uh, look forward to talking to you again.

582
00:45:29,220 --> 00:45:29,900
Thanks so much.

583
00:45:29,960 --> 00:45:31,480
That was really fun to talk with you both.

584
00:45:31,620 --> 00:45:32,360
Thanks for having me.