1
00:00:00,000 --> 00:00:08,420
Well, hello, everybody. It's the 11th of February, 2010. And this is, I forget which episode number, but it's some episode number of the Agile Executive.

2
00:00:09,160 --> 00:00:17,620
And as always, I'm one of your rotating co-hosts, Michael Coté. And I'm joined by one of the other sort of guest co-hosts. Would you like to introduce yourself?

3
00:00:18,460 --> 00:00:22,700
This is Andrew Schaefer. I've been on the Agile Executive podcast a couple times now.

4
00:00:22,700 --> 00:00:29,520
I previously was a co-founder at Reductive Labs, and I've been talking about this Agile infrastructure stuff.

5
00:00:29,920 --> 00:00:38,540
At the moment, I'm sort of between my projects. I have a couple things in the works. I don't want to jinx them, but hopefully there's some interesting things coming down the pipe.

6
00:00:39,300 --> 00:00:47,080
And then we are also joined today by someone that we've talked about on these podcasts and certainly inspired some of my thinking on the topic.

7
00:00:47,160 --> 00:00:53,040
We have John Alspaugh with us today, and he's going to tell us a little bit about how he sees this problem and how he solves it.

8
00:00:54,480 --> 00:01:03,320
Yeah, so my name is John Alspaugh, and I currently am the VP of Technical Operations at Etsy.

9
00:01:03,900 --> 00:01:07,200
I just started there just a couple weeks ago, previous to that.

10
00:01:08,280 --> 00:01:13,520
For about five years, I ran the operations engineering group at Flickr.

11
00:01:15,820 --> 00:01:39,440
Part of my growing up at Flickr, or at least growing the operations team as the site got large, were a lot of different bits of tools and culture that provided for a pretty great platform to move quickly

12
00:01:39,440 --> 00:01:49,180
and to react quickly and to react quickly to all the things that happens to you when you're in an accelerated growth situation within web operations.

13
00:01:49,420 --> 00:02:05,860
So I've been feeling quite passionate in the last couple of years about those topics, the collaboration and the communications between what are historically known as development groups and operations groups,

14
00:02:06,240 --> 00:02:14,200
where those groups meet, where those groups argue to death, and where those groups can actually work together.

15
00:02:14,800 --> 00:02:23,520
Those are the topics that have been on my mind, and how it relates to agile infrastructure, I think, is something worth talking about.

16
00:02:24,040 --> 00:02:33,040
Yeah, and I think I first came across your – or you might have put together this talk earlier in this, but back at, I guess, Velocity last year,

17
00:02:33,540 --> 00:02:37,420
there's – and thankfully there's a pretty nice video recording of it, which we'll put in the show notes.

18
00:02:37,420 --> 00:02:50,740
But you and one of your cohorts at Flickr kind of went through – you sort of dug below the gloss of like, hey, all this high-performance, high-scaling web stuff and cloud stuff is like fantastic,

19
00:02:50,900 --> 00:02:53,280
which is kind of like the level of conversation most people have.

20
00:02:53,400 --> 00:03:01,840
But you actually sort of walk through the processes and the changes that were happening to get the – you know, starting from the top for Flickr itself to deliver that service.

21
00:03:02,380 --> 00:03:09,480
And then more importantly, the kind of – not only the technical changes, but also sort of some of the process and cultural changes that development and operations went through.

22
00:03:10,200 --> 00:03:16,700
And, you know, it's one of the rare talks that you could actually like do something with it after you see it,

23
00:03:16,760 --> 00:03:21,740
other than just think that like there must be some magic idea like hiding around the nooks and crannies somewhere.

24
00:03:22,540 --> 00:03:29,460
So on that note, and this is going back to something you said, John, you said this notion of tools and culture.

25
00:03:29,460 --> 00:03:41,040
And just to get, you know, kind of your take on it, your spin on it, which do you think has a bigger impact or how do you see that feedback loop between better tools and then better culture?

26
00:03:43,860 --> 00:03:54,360
Well, I feel really strongly that the tools will get written if the culture allows the tools to be good.

27
00:03:55,120 --> 00:04:01,720
I think that – I think that – I think if the culture is there, then everything else will follow.

28
00:04:02,040 --> 00:04:10,620
I don't think – I think that there's – there is a natural tendency for engineers on both sides of that proverbial fence

29
00:04:10,620 --> 00:04:19,380
to look to look to technical solutions first and then try to work culture around them.

30
00:04:20,100 --> 00:04:27,500
And I think that I have some experience that would say the contrary should happen.

31
00:04:27,540 --> 00:04:38,420
And, you know, there were – the blurring of the lines on solving technical problems I think is a good thing.

32
00:04:38,840 --> 00:04:49,760
I think that developers – I think that people who write public-facing code can be awesome at writing operational tools.

33
00:04:49,880 --> 00:05:10,380
I think that people who manage infrastructure or have their career or domain expertise in managing infrastructure can bring a lot of really good things to the table in terms of the design of scalable architectures.

34
00:05:11,840 --> 00:05:28,400
And I think that if there's any situation where one – only one of those groups gets to sit at the table when they talk about implementation or process or anything, then you're going to lose out.

35
00:05:28,680 --> 00:05:29,060
Absolutely.

36
00:05:29,580 --> 00:05:36,580
Some of the best tools that we had at Flickr were written by the same folks that wrote public-facing tools.

37
00:05:40,000 --> 00:05:52,720
And by exposing things like fault tolerance and how to handle load feedback and how to write – you know, how to gather metrics from your application.

38
00:05:53,140 --> 00:06:01,560
Exposing those sorts of topics to the people who are writing your application can really, really pay off in the long run.

39
00:06:01,560 --> 00:06:12,540
It means that there's awareness of what your application is going to do, not just when you deploy it, but six months, a year later after you deploy it.

40
00:06:13,020 --> 00:06:15,820
So providing feedback on that.

41
00:06:16,060 --> 00:06:25,960
And I think the vice versa also is – again, I think that you're missing out if you only have developers dictating architecture.

42
00:06:25,960 --> 00:06:34,560
And I think that you're totally missing out if you only have operations people dictating things like monitoring and metrics collection.

43
00:06:35,100 --> 00:06:51,780
Well, you know, to back up the context a little bit, one of the things I was curious to ask you is in sort of traditional IT, whether it's public-facing or traditional operations, public-facing or sort of, I don't know, data centers or enterprise or whatever term you want to use for something that doesn't end in a dot-com.

44
00:06:52,620 --> 00:07:02,240
It seems like – I guess to ask the question one way, why did it end up that you had to start operating this way?

45
00:07:02,440 --> 00:07:12,320
Like what were sort of like the drivers of Flickr or the users of the business or whatnot, to use that term, that sort of made it such that development and ops couldn't stay being two camps?

46
00:07:12,440 --> 00:07:16,440
I mean, was there some sort of like moment of failure where it's like, oh, this is just not going to work anymore?

47
00:07:16,600 --> 00:07:18,560
Kind of like what was that motivation?

48
00:07:18,640 --> 00:07:24,900
And the reason I ask this is because whenever there's some sort of new culture or process change, you know, you get the same response from everyone.

49
00:07:25,000 --> 00:07:29,480
It's like, oh, that sounds great, but my organization is somehow different and it doesn't apply to us.

50
00:07:30,100 --> 00:07:31,920
It's a special snowflake problem.

51
00:07:32,160 --> 00:07:32,660
That's right.

52
00:07:33,020 --> 00:07:33,380
Exactly.

53
00:07:33,820 --> 00:07:41,380
And so I'm curious, like in the instances you've been involved in, like how you kind of got over that hump of like we're not a special snowflake essentially.

54
00:07:42,080 --> 00:07:50,140
Yeah, well, so it's an awesome question and it's something that people ask all the time.

55
00:07:50,140 --> 00:07:53,800
So I had the benefit of growing it.

56
00:07:54,340 --> 00:07:55,660
We didn't have to change it.

57
00:07:56,700 --> 00:08:15,020
The, the, uh, my, my gut instinct tells me that, so I was the third, well, not really the third technical guy, but there were two PHP developers and, uh, there were two PHP developers, um, at Flickr when I started.

58
00:08:15,580 --> 00:08:22,880
And one of them was the guy that wrote all the code, Cal Henderson, and the other guy, Aaron Cope, started like a month before me.

59
00:08:23,420 --> 00:08:41,080
So until a month before I started, Cal was both writing all the public facing PHP, writing all of the metrics, uh, and, uh, and alerts, um, handling everything, racking servers.

60
00:08:41,080 --> 00:08:50,140
So, you know, in, in, in the, in the, in the, at least in the Bay Area, how it works out is, you know, the, the idea of three guys in a garage, they're going to have a web startup.

61
00:08:50,340 --> 00:08:57,840
Well, you can't have to, you don't have to worry about operations if you don't have a product to operate, right?

62
00:08:57,960 --> 00:09:01,180
So you start with a developer or at least one developers.

63
00:09:01,380 --> 00:09:12,020
And at some point you realize, okay, well, managing this stuff or dealing with this other stuff, maybe we should bring somebody in who, uh, can just focus on the operation side of things.

64
00:09:12,100 --> 00:09:16,960
So as a result, Cal wrote a lot of tools that made things easy for him.

65
00:09:17,860 --> 00:09:18,960
He had a developer mindset.

66
00:09:19,700 --> 00:09:26,880
He hadn't, he didn't, he didn't have any preconceived notions of, oh, well, I'm going to use a layer seven load balancer here.

67
00:09:27,080 --> 00:09:29,060
So I better get a quote from NetScaler.

68
00:09:29,360 --> 00:09:30,580
He didn't have any of that.

69
00:09:31,060 --> 00:09:39,200
Um, he didn't, uh, he, he said, you know, for example, perfect example is distributing loads throughout the, throughout the application.

70
00:09:40,280 --> 00:09:46,340
Maybe an ops centric person might say, I'm going to get a load balancer and we'll distribute load that way.

71
00:09:46,340 --> 00:09:49,260
And it's got all these pros and cons for it.

72
00:09:49,640 --> 00:09:59,100
And the application developer historical, or at least in Cal situation, I'm going to get an array, fill it with host names and iterate over it.

73
00:09:59,100 --> 00:10:00,140
However I like.

74
00:10:00,680 --> 00:10:03,000
So this is the same results.

75
00:10:03,300 --> 00:10:09,340
You still have, uh, load being distributed, except ones with an expensive piece of hardware.

76
00:10:09,560 --> 00:10:13,800
And the other one is that the application took on that task of doing it himself.

77
00:10:13,800 --> 00:10:31,420
The part that is subtle is that Cal's thinking in, in, in, uh, in distributing load within the application meant that he had to actually think about how the load was going to be distributed within the application.

78
00:10:31,420 --> 00:10:42,440
It is, in my sense, less about what you would find in a, uh, in a historic, uh, or a typical developer mindset.

79
00:10:42,620 --> 00:10:49,880
So, uh, the, the, what I, what I'm trying to get at is that I didn't, I didn't have to come in to Flickr and change anything.

80
00:10:50,420 --> 00:10:57,180
It was, it was, it was, it started out this way and it just, uh, led by example.

81
00:10:57,440 --> 00:11:03,960
I'd say that both Cal and myself, as time went on, uh, we led by example.

82
00:11:04,200 --> 00:11:22,900
So any of the, any of the operations people on my team, um, respected developers, there was no, uh, well, this is my job or my responsibility and you don't get to touch, you know, developers don't get to touch this or that, or, you know, production is pristine.

83
00:11:22,900 --> 00:11:28,500
So you don't get to touch this or that, um, and it turns out you're the special snowflake then.

84
00:11:30,320 --> 00:11:35,280
So this, well, let me, I'm going to throw a comment out and then I want you guys to both, uh, respond to it.

85
00:11:35,440 --> 00:11:52,760
But I kind of feel like when people talk about agile, whether they're talking about developer practices or as we evolve sort of this, this new, uh, ops angle that the, the best practices really come out of creating the proper habitat and ecosystem.

86
00:11:52,900 --> 00:12:03,420
And then the, they just sort of emerge because I, well, barring ignorance essentially, but it's two, it's 2010 people are doing this.

87
00:12:03,480 --> 00:12:04,760
People are talking about it.

88
00:12:04,760 --> 00:12:09,800
There's no reason that you have to be ignorant of what people are doing and how they're getting their results.

89
00:12:10,200 --> 00:12:19,340
So if you create a culture, going back to the original statement, if you create a culture where that is, it's a culture of learning and it's the proper habitat, then people will just behave this way.

90
00:12:19,860 --> 00:12:28,460
And, and there's a quote, I can't remember who made it, but it's something to the effect of, if you want people to, to follow a process, then you should just start doing it.

91
00:12:28,560 --> 00:12:31,240
If you want people to ignore a process, then you should document it.

92
00:12:32,600 --> 00:12:33,000
Yeah.

93
00:12:33,080 --> 00:12:41,300
No, and, and, and, and I think, you know, the, I, I think the, the patterns of adoption for changing culture in a technical organization that seemed to be six.

94
00:12:41,300 --> 00:12:48,580
One of them that seems to be successful is, is like what, what you're getting to is that it's the easiest way to convince someone.

95
00:12:48,720 --> 00:12:54,540
Well, one of the easier ways to convince someone to do something is to actually kind of reverse it on its head and start talking about the benefits of being that way,

96
00:12:54,580 --> 00:12:59,540
essentially, rather than it just, you know, rather than saying this is a hygienic better way to operate and we should try it out.

97
00:12:59,540 --> 00:13:08,660
And that's, that's the nice thing that, you know, the folks, the folks like, like John and others who are doing this DevOps thing, they have, they have kind of the track record and the stories to tell.

98
00:13:08,740 --> 00:13:14,340
They're not only sort of like, they're both problems that we fixed essentially in, in traditional problems that we fixed.

99
00:13:14,500 --> 00:13:21,520
But what becomes even more interesting are the more, what I always call like the glass half full sort of things, like the new things that you can start doing that are interesting.

100
00:13:21,700 --> 00:13:31,100
And I think, I think those two things together becomes a pretty powerful motivation for, for what seems like an impossible cultural change that, that you get onto and it helps get over that hump.

101
00:13:32,840 --> 00:13:44,160
Yeah. And I think that, um, uh, I would, I would agree and, you know, not to, not to make this any sort of a segue, but I do believe that, um, the, the idea of the infrastructure, um,

102
00:13:44,500 --> 00:13:52,180
is, you know, if you're only a button on a webpage away from pulling up a, uh, for provisioning a server, sure.

103
00:13:52,180 --> 00:14:02,320
There's all of the regular advantages of, of, of, you know, uh, forecasting and economics of scale and, and, and, uh, you know, burst amounts of capacity, blah, blah, blah.

104
00:14:02,320 --> 00:14:14,320
But what it really means is that, uh, you, you've now won, you've, you've pulled away from what is essentially somewhat ineffective.

105
00:14:14,500 --> 00:14:22,080
Um, way of doing things, which is everything needs to be metal and I need to buy, I need to buy the metal and, uh, that sort of thing.

106
00:14:22,760 --> 00:14:29,440
Um, to, again, smearing out some of the, um, some of the concerns, right?

107
00:14:29,460 --> 00:14:34,500
So like we used to say it at Flickford is that everybody owns availability, right?

108
00:14:34,860 --> 00:14:42,420
Operations isn't the only one who, uh, is, is tasked with making the site, uh, fast and available.

109
00:14:42,900 --> 00:14:48,940
And you'd be hard pressed to find a developer who says, yeah, I'd all, no, I just want to write my code.

110
00:14:48,940 --> 00:14:52,940
And I don't really care if the site goes down, like you're not going to find that guy.

111
00:14:53,020 --> 00:14:56,440
Or if you do, he doesn't, you know, he's, he's not a very good developer.

112
00:14:56,880 --> 00:14:56,980
Right.

113
00:14:57,780 --> 00:14:59,420
Um, so he doesn't work there very long.

114
00:15:00,540 --> 00:15:01,240
Right, exactly.

115
00:15:01,340 --> 00:15:12,720
So I think that cloud computing, um, uh, has helped, has helped these, this, this smearing out of not just responsibilities, but domain, uh, expertise.

116
00:15:12,720 --> 00:15:13,680
Domain knowledge.

117
00:15:13,680 --> 00:15:25,400
I think that it's a good thing, absolutely the best thing because it, it, it then entices developers to learn more about how their, how their, their application is, is working with infrastructure.

118
00:15:25,960 --> 00:15:39,540
And it also focuses operations people, um, less on, on how good their tabling is and, and more on, on other things that can be, you know, more important.

119
00:15:39,700 --> 00:15:48,620
And I'm absolutely more than happy in pushing, you know, physicality, uh, to this down the stack, somebody else.

120
00:15:48,720 --> 00:15:51,960
And at Yahoo, that was pretty much the case.

121
00:15:51,960 --> 00:15:57,680
You know, there, there wasn't anybody on my team who ever had to touch a server at Flickr.

122
00:15:57,840 --> 00:16:04,700
We enjoyed having the, the mothership of Yahoo have horizontal platform groups.

123
00:16:04,760 --> 00:16:11,020
There's, you know, we had eight data centers, you know, so there were, and none of them were anywhere near our office.

124
00:16:11,020 --> 00:16:15,980
So it's not like any of my, I had to have anybody on my team that had to go to the colo.

125
00:16:16,560 --> 00:16:30,740
So what that meant was that they could, they could, they could focus on really the, the, the harder problems, the more important problems that come up at least, at least on a fast growing web 2.0, uh, you know, type of site.

126
00:16:31,380 --> 00:16:36,440
So, so to, to, to that point of like, uh, you know, having this sort of process and culture change.

127
00:16:36,440 --> 00:16:50,240
I mean, can you give us kind of like, uh, uh, an abbreviated sort of run through of like, what kind of like the top sort of changes that you see the operations and developers, people going through when they, when they get to the, uh, you know, when they, when they transform into this way of operating.

128
00:16:50,520 --> 00:17:01,600
I, you know, I realize that's, uh, incredibly broad, but, but, you know, I'm just curious to hear like, what, what are the major ways they seem different than, than traditional or, or typical sort of ops and dev people?

129
00:17:03,740 --> 00:17:17,980
Um, well, so the, the obvious example, I think, and, and, and Andrew's, is pointed this out multiple times in various places, but, um, and you call them, I think, I think Andrew, you call them boundary objects.

130
00:17:18,280 --> 00:17:21,620
Um, but one, the most obvious place obviously is deployment.

131
00:17:22,160 --> 00:17:28,580
Um, and this, this, uh, so at, at Flickr, developers push their own code.

132
00:17:29,000 --> 00:17:33,560
Um, it is, if you want to call it continuous deployment, um, which is.

133
00:17:33,740 --> 00:17:39,160
Uh, another way of saying pushes, pushing small changes often.

134
00:17:40,040 --> 00:17:49,680
Um, I'd say that, I'd say that that's a, that's a perfect example of where collaboration and communication, um, can bring those two groups together.

135
00:17:50,060 --> 00:18:01,140
Uh, again, we talked a little bit about it at our talk at Velocity, some of it taken out of context, but, um, you know, the, uh, I, I don't, I don't know how much,

136
00:18:01,140 --> 00:18:15,200
how much detail you want to, you want me to talk about it, but I, you know, I think that, um, deploying, uh, deploying small changes very frequently is it as an operations manager, I want that to happen.

137
00:18:15,340 --> 00:18:18,360
I believe it to be a boon for availability.

138
00:18:18,360 --> 00:18:31,760
It, because it means that, that if there is an issue that the, are my meantime, uh, my meantime to resolution, my meantime to detect all of those things are going to be different because guess what?

139
00:18:31,920 --> 00:18:36,060
That last deploy only had five changes from three different people.

140
00:18:36,540 --> 00:18:37,020
That's it.

141
00:18:37,520 --> 00:18:37,840
You know?

142
00:18:38,320 --> 00:18:43,880
So, so I think that's a good, good, good crystallized example of, of, of where this stuff can happen.

143
00:18:44,920 --> 00:19:02,080
And you go back to the, the classic ideas that the agile sort of founded on and you think about how people do most of their, their web operations, you know, flicker notwithstanding and some of these other people who are moving in this direction, but you essentially have agile, agile, agile, agile, you know, small changes, small changes.

144
00:19:02,260 --> 00:19:06,160
And then your deployment process is essentially a big waterfall, right?

145
00:19:06,300 --> 00:19:09,500
That's, that's like, that's how most people do this.

146
00:19:11,500 --> 00:19:11,900
Yeah.

147
00:19:12,020 --> 00:19:12,180
Yeah.

148
00:19:12,180 --> 00:19:18,880
No, I, I was speaking to like, uh, the local Austin agile group recently about, you know, figuring out what, what you can do with the cloud.

149
00:19:18,880 --> 00:19:24,600
Essentially, you know, these are the kinds of exciting talks I ended up having with, uh, with, with the general audience.

150
00:19:24,600 --> 00:19:31,500
And, and, you know, as, as I, as I was going through what I was going to say to them and talking with them, I was realizing that, you know, it, it sort of is, uh,

151
00:19:32,140 --> 00:19:42,360
and, and I, I think, I think you guys pointed this out, John, in, in the, in the velocity presentation, but you know, the, the, the ops people essentially that they've kind of been stuck in this role of basically their job is to say no.

152
00:19:42,360 --> 00:19:49,520
And, and change management started sort of becomes, you know, no management essentially change management is the art of making sure nothing changes.

153
00:19:49,520 --> 00:19:55,580
And, you know, on the one hand, it's frustrating for everyone because it's sort of like they become a bottleneck or a wall.

154
00:19:55,580 --> 00:20:02,020
But then on the other hand, there's also this, you know, the only time people really notice operations people is when something goes terribly wrong.

155
00:20:02,020 --> 00:20:08,100
Like, you know, there's not like, uh, you know, send flowers cause your email's working perfectly fine sort of things going on with ops people.

156
00:20:08,100 --> 00:20:10,480
No one really thanks them when everything's humming and great.

157
00:20:11,160 --> 00:20:21,660
And so, you know, it's just over the years, you develop this culture that, uh, if you're only going to be punished, then you basically just do everything to avoid punishing and you don't do anything to really bring rewards on.

158
00:20:22,440 --> 00:20:29,540
And, and it is like, you know, when you get down to something as, as, you know, taking those, those like, I don't, you know, vague head in the cloud sort of notions.

159
00:20:29,540 --> 00:20:32,560
And then you apply something like having very small changes, right?

160
00:20:32,780 --> 00:20:33,580
Continuous deployment.

161
00:20:33,580 --> 00:20:40,900
And then like you're saying, you start to understand that like, even if something goes wrong, it's really, it's easier to sort of figure out what's going wrong.

162
00:20:40,900 --> 00:20:46,940
And it should be easier to control something not going wrong in the first place than if you sort of have these giant waterfall sort of processes.

163
00:20:46,940 --> 00:20:59,800
And it's sort of like, uh, as happens all the time in tech world, it's this classic Stockholm syndrome where the person who's trying to work around a problem becomes the embodiment of that problem rather than trying to come up with some way of preventing it from happening in the first place.

164
00:21:00,940 --> 00:21:01,340
Right.

165
00:21:01,700 --> 00:21:02,100
Absolutely.

166
00:21:03,620 --> 00:21:09,260
So John going, you know, it's been eight months since the velocity talk, give or take, and velocity is coming up again.

167
00:21:09,260 --> 00:21:20,140
And I, we, we, we constantly reference velocity in these discussions because I feel like it's sort of the center of gravity for people who are really pushing the practice forward in terms of operations.

168
00:21:20,140 --> 00:21:32,780
And although it's, it's quite web centric, the best practices that are being developed at these companies who are really solving these problems at scale can be applied to, to the enterprise, you know, across the board.

169
00:21:32,780 --> 00:21:54,660
And you sort of, you sort of made mention of the, of the talk, but I've also heard some people, uh, operations people in particular who are, they're complaining a little bit about the talk in the sense that the, the, the development teams they work with somehow the only thing they took away from, from you and Paul on the stage was that now we can deploy whenever we want to.

170
00:21:56,660 --> 00:21:57,060
Yeah.

171
00:21:57,060 --> 00:21:57,140
Yeah.

172
00:21:57,240 --> 00:21:57,780
And yeah.

173
00:21:57,880 --> 00:22:10,700
And they're, I think they're missing the part where you only get to pull, you only get to deploy whenever you want to, if you have a history of non incident deploys.

174
00:22:11,560 --> 00:22:21,900
So, you know, like I said, like we said in the talk is that, you know, you can't, you can't claim that you're agile and nimble, uh, you know, and you deploy 10 times a day if you go down 10 times a day.

175
00:22:22,380 --> 00:22:29,360
And, um, uh, you don't, it's, it's, it shouldn't be used as an excuse to be a cowboy.

176
00:22:29,360 --> 00:22:52,360
The, the whole, the whole point is actually to spread the, the onus or the, the, the burden of thinking about availability, um, across the whole team, instead of having it just sit with the guys who are the least, uh, uh, enabled to, to control.

177
00:22:52,360 --> 00:22:57,660
So, um, you know, I, I, that's, that's, that's, that's was somewhat disconcerting.

178
00:22:57,660 --> 00:23:01,920
I probably blame, I, I blame that we probably didn't spend enough time on that topic.

179
00:23:01,920 --> 00:23:05,900
I, there, there are a couple of folks that are presenting in, at velocity.

180
00:23:06,280 --> 00:23:08,720
I think, I think you should blame Paul cause he's a developer.

181
00:23:10,860 --> 00:23:22,340
Well, you know, towards, you know, uh, even, you know, towards the end of, of my working at, at flicker, actually not even towards the end, you know, you know, that you're in a situation when, uh, you know, you're in a situation.

182
00:23:22,340 --> 00:23:43,340
You know, Paul and myself, uh, the, the sort of twin leaders of the, of, at least of the technical culture there, um, when Paul is arguing for us to be, uh, more conservative, um, and, and I'm, I'm pushing for us to, to, to say yes instead of, instead of Paul.

183
00:23:43,340 --> 00:23:49,740
Paul is the other way, a little bit of role reserve reversal, but I, I think that that's actually, um, healthy if the pendulum goes in the other direction.

184
00:23:50,280 --> 00:24:06,660
Um, yeah, I mean, if you got out of the talk or if you get out of, of, of anything, and it's not just us, you know, Eric Reese, um, uh, um, and, and, and, uh, Tim Fitz talks about very similar things that, that, that happens at, at IMVU.

185
00:24:06,660 --> 00:24:21,380
And, uh, um, but, uh, you know, you, you weren't really listening if you think that, uh, see, well, if, you know, if, if John Osborne, Paul Hammond say that we should deploy 10 times a day, then we should do it.

186
00:24:21,520 --> 00:24:29,900
And, uh, those ops people can, you know, suck it and just let their pages, you know, their pages can catch on fire because those guys are doing it that way.

187
00:24:30,080 --> 00:24:30,280
Yeah.

188
00:24:30,360 --> 00:24:30,600
Yeah.

189
00:24:30,680 --> 00:24:30,880
Yeah.

190
00:24:30,880 --> 00:24:42,960
I mean, I think, I think a lot of the discussion you, you guys have had in that talk and in other places is kind of, uh, you know, the aspiration is to be able to do that, to, to deploy many times a day or, you know, to deploy a lot more.

191
00:24:42,960 --> 00:24:52,300
And, and, and I, I guess the thing people have to realize is like all this other stuff, like the other 95% of like the conversation is all about making that possible essentially.

192
00:24:52,660 --> 00:24:58,580
And you kind of like, you can't, you can't just choose like that five or 1%, you know, you can't just eat the donut.

193
00:24:58,720 --> 00:25:00,920
You got to eat all this other stuff that comes with it.

194
00:25:01,260 --> 00:25:02,240
That's what I call this.

195
00:25:02,300 --> 00:25:03,400
It's the dessert tray.

196
00:25:03,540 --> 00:25:05,800
Agile, you only eat the dessert.

197
00:25:06,280 --> 00:25:11,180
You know, one, one last that I'd like to say about that.

198
00:25:11,280 --> 00:25:21,380
One thing that it's probably not very clear is, um, after we were acquired by Yahoo, the, we were, we had this deployment, uh, methodology.

199
00:25:21,380 --> 00:25:25,300
We were doing the small changes often and we were growing, right?

200
00:25:25,300 --> 00:25:32,640
We went from the 25th most popular, um, Yahoo property to like the fifth, four years later.

201
00:25:33,260 --> 00:25:53,260
Um, and, uh, we, so we had insane growth, but yet this, you know, methodology and through various, you know, reorganizations of Yahoo management or whatever, um, various people from, from further on inside the mothership of Yahoo would say, Hey, I heard this rumor that you guys deploy 10 times a day.

202
00:25:53,260 --> 00:25:54,080
Is that right?

203
00:25:54,240 --> 00:25:58,600
And somebody said something about developers pushing their own code.

204
00:25:58,760 --> 00:26:00,300
Please tell me that's not true.

205
00:26:00,720 --> 00:26:04,160
And, you know, that's the part where the jaws drop on the floor.

206
00:26:04,160 --> 00:26:07,800
And I say, and Hammond and I say, no, actually that is, that's the situation.

207
00:26:08,520 --> 00:26:24,240
But what's missing is that the Yahoo service engineering or Yahoo operations, um, leadership never would have allowed us to continue the deployment methodology if we didn't have the history to back it up.

208
00:26:25,040 --> 00:26:36,020
The most important thing about continuous deployment in my mind is knowing where to turn, where and when to turn up that volume.

209
00:26:36,260 --> 00:26:46,740
If I have a history of incidents that have my, my meantime, my MTTR and my MTTD is really high, well, guess what?

210
00:26:46,980 --> 00:26:49,680
That's, that's an indication that maybe you're going too fast.

211
00:26:50,120 --> 00:26:55,100
Maybe you should slow down a little bit because it, it, you know, the metrics say that you're a cowboy.

212
00:26:55,560 --> 00:27:11,840
So you only get to do that if you have a history to prove that you can, that's, that's, that's, that's, that's what I think is probably missing from, um, from the conversation that, that at least I've put forth in the, in, you know, in, in the past.

213
00:27:12,220 --> 00:27:15,760
So you can go as fast, you can go as fast as you want, but don't rush.

214
00:27:16,760 --> 00:27:19,380
Well, it's just, you know, just, yeah, exactly.

215
00:27:19,540 --> 00:27:20,880
Don't crash the car.

216
00:27:21,940 --> 00:27:31,820
Uh, you know, um, it's again, you know, keep track of, uh, you know, what is your, what is your deployment to incident ratio?

217
00:27:32,540 --> 00:27:49,220
You know, is that, is that ratio really high is what percentage of your, of your deploys, I don't think it doesn't matter how many changes are in each deploy, how many, how many, uh, incidents have you had for every 10 deploys, you know?

218
00:27:49,220 --> 00:27:53,880
So, um, the idea, the goal there is to not have incidents.

219
00:27:53,880 --> 00:27:54,980
That's the whole point.

220
00:27:55,420 --> 00:28:09,140
So, so yeah, the, the, the missing part outside of the donut is, is, you know, there's, I'm, I'm, I'm, uh, breaking down on the food analogy here, but you know, you have to, you have to, you have to, let's go to, let's go to sports.

221
00:28:09,140 --> 00:28:14,380
I think it's, it's like your turnover to, turnover to assist ratio is basically what you're looking at.

222
00:28:15,040 --> 00:28:16,040
Yeah, exactly.

223
00:28:16,040 --> 00:28:25,140
I think, I think, I think to your point, like one of the more interesting things and all, all of this kind of talk, whatever, whatever you want to call it, whether it's agile infrastructure or cloud stuff,

224
00:28:25,140 --> 00:28:39,480
or high performance hoopty or whatnot, is the, the, one of the interesting things that's come, comes true, like, you know, you mentioned MTTR and MTTD and all these things is, is you, I noticed that most people who do this become really metrics obsessed.

225
00:28:39,480 --> 00:28:46,860
And it's sort of like metrics become one of the main things that both developers and operations people begin to be really interested in.

226
00:28:47,000 --> 00:29:02,520
And what, what becomes interesting there is, is they're not so much, they're not only interested in traditional metrics, like, you know, CPU and storage and network bandwidth, but they start, you somehow need to enable them to put their own sort of metrics in there to instrument the systems that they're doing.

227
00:29:02,780 --> 00:29:10,040
So that they can start sort of taking the, you know, they can, they can, they can not be a cowboy by having metrics that tell them when they're getting too close to that, essentially.

228
00:29:10,760 --> 00:29:17,400
And it seems like, it seems like no matter which software you're working on, you're probably going to need some sort of custom metrics.

229
00:29:17,400 --> 00:29:20,020
So you need some system that enables that to be possible.

230
00:29:20,640 --> 00:29:22,360
Oh yeah, of course, absolutely.

231
00:29:22,600 --> 00:29:36,580
And in fact, if you think that, um, you can just take a, a, a, a canned set of metrics from one application in company to another application in company, then you're, you're doing it wrong.

232
00:29:36,580 --> 00:29:47,820
Um, there's, you know, uh, there are, there are application level metrics that are important to some people and, uh, to some applications and others that aren't, you know?

233
00:29:47,820 --> 00:30:01,820
Um, um, Flickr has photos, Etsy has listings, Twitter has tweets, you know, uh, there, there are multiple, multiple metrics for, for multiple applications.

234
00:30:01,820 --> 00:30:14,700
And if you're not really keep, and if you're not keeping track of what your infrastructure is doing alongside those metrics, well, then you're, you're, you're only listening to one guy in the band.

235
00:30:14,700 --> 00:30:16,620
You're not listening to the whole song.

236
00:30:17,320 --> 00:30:25,120
So to, to, to that point, and this is something you, you had in your, in a slide at Velocity, the, the job is to enable the business.

237
00:30:25,340 --> 00:30:27,060
Everyone's job is to enable the business.

238
00:30:27,300 --> 00:30:33,900
And if the business, if the business value isn't delivered, then, then you can't do all the fancy stuff with all the technology.

239
00:30:34,740 --> 00:30:35,140
Right.

240
00:30:35,560 --> 00:30:36,560
Yeah, absolutely.

241
00:30:36,840 --> 00:30:38,940
Cause that's, that's the whole point at the end of the day.

242
00:30:39,360 --> 00:30:39,580
Yeah.

243
00:30:39,580 --> 00:30:46,820
That was when it like down here in Austin, a few weeks ago, we had, uh, like the first ops camp and it was kind of, it was disheartening and also heartening.

244
00:30:46,820 --> 00:30:54,000
It's always our cyclical nature of these kinds of ideas that there was this group that was like the, uh, you know, the, uh, the craft of system of operations.

245
00:30:54,000 --> 00:30:57,460
And, you know, it's a typical sort of like career therapy, which is always fantastic.

246
00:30:57,460 --> 00:30:57,840
Right.

247
00:30:57,900 --> 00:31:04,900
And, uh, and, and, and, and it was funny, like after 10 or 15 minutes, everyone sort of reached this conclusion is like, oh, we're supporting the business.

248
00:31:04,900 --> 00:31:07,120
So we should, you know, we should focus on that.

249
00:31:07,120 --> 00:31:16,900
And, and the disheartening part about that is, is to realize like the last 10 years, there's been all these like big vendors in the industry and processes that are trying to drive everyone towards that.

250
00:31:17,140 --> 00:31:27,740
And that it's still like, uh, it's still like this big realization for people to go through that, that, you know, oh, what we're doing is we have customers who are paying us to do something or, you know, giving us their eyeballs or whatever.

251
00:31:28,280 --> 00:31:39,160
But, uh, you know, in the, in the same way that, you know, hopefully it's, it's a similar pattern to sort of like SOA versus sort of like restful open API things where like the original ideas of SOA were fantastic.

252
00:31:39,160 --> 00:31:43,200
And then it went all like crazy, but then we kind of, you know, fixed it after a while.

253
00:31:43,660 --> 00:31:46,120
So, uh, anyhow, hopefully that'll happen this time.

254
00:31:47,000 --> 00:31:47,400
Yeah.

255
00:31:48,400 --> 00:31:48,760
Yeah.

256
00:31:48,780 --> 00:31:52,380
I mean, that it's, it is also, you know, it's, it is difficult.

257
00:31:52,380 --> 00:31:55,980
It's hard to, um, it's hard to see the, the large picture sometimes.

258
00:31:58,280 --> 00:32:07,440
So on that note, we, we often mentioned velocity and, and as I said earlier, I think it's, uh, sort of the, the center of gravity for, for this sort of thought.

259
00:32:07,640 --> 00:32:11,020
And I know you guys are working on putting together the, the one in June.

260
00:32:11,020 --> 00:32:16,700
So maybe you want to kind of talk a little bit about what, what people could expect to see there or, or why that's exciting.

261
00:32:17,920 --> 00:32:18,360
Yeah.

262
00:32:18,360 --> 00:32:19,460
Uh, yeah.

263
00:32:19,560 --> 00:32:28,260
So I'm, I'm part of the program committee, um, and, uh, and the, the deadline's passed for all of the proposals, but we're going through.

264
00:32:28,280 --> 00:32:29,680
All of the proposals at the moment.

265
00:32:29,760 --> 00:32:36,860
Matter of fact, we have a conference call tomorrow, um, for all the people in the program committee, um, to go over the proposals and that sort of thing.

266
00:32:36,860 --> 00:32:38,780
And, and, and so it's a third year.

267
00:32:38,780 --> 00:32:53,400
And I think that the, I think that the, the agenda and all of the things that's important, um, to, uh, to both the operations and the front end performance communities.

268
00:32:53,400 --> 00:33:03,560
Um, I think there's, um, uh, much, uh, much more broad, uh, and both broad and focused at the same time.

269
00:33:03,560 --> 00:33:04,520
It doesn't make any sense.

270
00:33:04,520 --> 00:33:12,900
Um, but, um, there, there are really good proposals, proposals that I wish people, I wish that there were last year.

271
00:33:12,900 --> 00:33:17,980
Um, but I think that it's, it's, I think that the conference, uh, in general is, is picking up steam.

272
00:33:17,980 --> 00:33:32,480
Um, without, without, you know, without speaking about, uh, uh, any particular proposals, I think a lot of the stuff that we're talking about, um, today, uh, is going to, is, is going to, uh, show up in the conference.

273
00:33:34,540 --> 00:33:39,260
So there's a, there's a question I have just because people are doing this all the time.

274
00:33:39,380 --> 00:33:40,320
They're losing their minds.

275
00:33:40,380 --> 00:33:45,160
I think I have more drama from the cloud computing experts in, in Twitter than any other group.

276
00:33:45,480 --> 00:33:55,880
But when people are talking about cloud computing, there, there's sort of a nebulous idea of, you know, the clouds and like sys admins are going away and we're not going to use physical hardware anymore.

277
00:33:56,540 --> 00:34:10,340
In your experience working at, at, you know, Yahoo, Flickr and Etsy, well, how much do you see these sort of, you know, from Facebook to Google using what are the, you know, the cloud, if you will, technologies?

278
00:34:12,520 --> 00:34:26,380
Oh, well, uh, well, personally, I think that anybody who has had any, uh, experience in, um, in growing, uh, uh, at least a web operations infrastructure, uh,

279
00:34:26,380 --> 00:34:41,540
realizes that especially with applications that are, are, are more social and less static these days, um, anybody who thinks that, uh, that operational concerns are going away is, is fooling themselves.

280
00:34:41,540 --> 00:34:46,440
Um, uh, I, I, I, it's just, it's not a possibility.

281
00:34:46,440 --> 00:34:56,220
If, if, if operations to you means physical people to, you know, people to physically move iron around and cable them up, then,

282
00:34:56,220 --> 00:35:01,360
uh, then, uh, then you probably work on a relatively boring application.

283
00:35:01,960 --> 00:35:20,220
Um, that's, you know, like I said, I've had, there are eight, when there are eight people on the Flickr operation engineering, uh, team and, uh, none of them have to, have to worry about things like BGP and OSPF or swapping drives or going to a polo.

284
00:35:20,220 --> 00:35:23,480
And they don't need those little Velcro wire things or anything, right?

285
00:35:23,580 --> 00:35:24,200
Yeah, exactly.

286
00:35:24,500 --> 00:35:26,580
Uh, and no static guards or anything.

287
00:35:26,740 --> 00:35:32,580
All of those people are very, very busy doing things that are important to the business.

288
00:35:32,720 --> 00:35:35,160
So, no, operations isn't going away whatsoever.

289
00:35:35,560 --> 00:35:46,080
As far as cloud computing is, as, uh, um, sort of as a topic, I think that the answer, whether people talk about it or not, is, should be already clear.

290
00:35:46,140 --> 00:35:59,440
And those people who use the, the, the smug mugs and the animotos and the, um, picnic, the people who use cloud computing, or at least, uh, uh, I don't know, whatever.

291
00:35:59,440 --> 00:36:07,020
Or insert, insert, insert debate about definition here, um, infrastructure as a service or whatever, Amazon web services.

292
00:36:07,340 --> 00:36:14,480
Um, those people who use it successfully use it for exactly what it, it is designed for at the moment.

293
00:36:14,880 --> 00:36:25,260
Um, I don't, I could be wrong about this, but, you know, so Don McCaskill at smug mug is a perfect example.

294
00:36:25,260 --> 00:36:27,600
He's always a really good example of cloud computing.

295
00:36:27,600 --> 00:36:33,660
He uses it for, uh, processing images and storing photos.

296
00:36:34,160 --> 00:36:38,600
Um, as far as I know, I could, I, I certainly, uh, could be wrong about this.

297
00:36:38,740 --> 00:36:43,500
I believe that web servers and databases are still on premise for him.

298
00:36:43,960 --> 00:36:50,520
Um, I don't know if he's moved his database layer, um, to, uh, to Amazon web services just yet.

299
00:36:50,520 --> 00:37:15,360
I know that picnic has not, and that's just because an application specific requirement that you're building dynamic web pages and the, the, the boxes that front that, that serve those web pages needs to have low latency, uh, um, generally, uh, random, uh, access workloads, um, uh, to resources.

300
00:37:15,360 --> 00:37:27,940
Uh, AKA databases that are right next to it and have to have some, uh, some, um, reasonable amount of persistence or durability.

301
00:37:28,560 --> 00:37:50,500
So, um, you know, um, you know, I still think that there's, um, there's, there probably will be a day when every piece of what is, what, what is deemed to be a, uh, of a full, or at least a full lamp architecture or, or, or, uh, uh, at, at large volumes, um, maybe there's, you know, something like that.

302
00:37:50,500 --> 00:38:04,540
But in the, in the meantime, uh, there, there's this, there's this spectrum of two guys in a garage, small site on one side and Google, Facebook, Flickr on, on the other side.

303
00:38:05,060 --> 00:38:21,740
And at some point your application or parts of your application are simply just too big for either the economics or the, uh, uh, the architectural limitations to make it worth moving to the cloud.

304
00:38:22,220 --> 00:38:26,080
Um, if that was the case, Facebook would be on Amazon web services.

305
00:38:26,700 --> 00:38:39,260
Yeah, no, I mean, it's an interesting point you're making about the, the sort of cloud services that are available where, you know, there's, you, you may not be able to treat it as generic as you may think you'd be able to be able to treat these services.

306
00:38:39,260 --> 00:38:46,500
That it's, it's good to make sure that you're still matching, you know, what capacity you need and what functionality you need to sort of the service that you're using.

307
00:38:46,500 --> 00:38:56,060
And it's not sort of like this panacea of, you know, it's replacing my data center essentially, which is, is, is sort of practical advice that it's good to hear ahead of time before you get,

308
00:38:56,060 --> 00:38:57,900
you discover it on your own essentially.

309
00:38:59,040 --> 00:39:03,080
But, uh, anyhow, but, but with that, I mean, I mean, it's, uh, we, we should, uh, wrap it up here.

310
00:39:03,140 --> 00:39:05,900
I mean, I appreciate you, uh, spending so much time talking with us about it.

311
00:39:05,940 --> 00:39:09,960
It's, it's always nice stuff to, to, uh, you know, just discuss casually.

312
00:39:09,960 --> 00:39:11,000
So that's fantastic.

313
00:39:11,500 --> 00:39:18,560
Yeah, this is, I love talking about this stuff and, um, I'm, I'm, I'm happy to, um, to gab on about it.

314
00:39:18,640 --> 00:39:22,740
I mean, you guys are, uh, you guys are great for, for allowing me to ramble.

315
00:39:23,060 --> 00:39:25,800
And, and, you know, I have to say now that I know you're at Etsy,

316
00:39:25,920 --> 00:39:29,520
I'm, I'm very happy about that because, you know, Etsy needs to be kept up so that one

317
00:39:29,520 --> 00:39:31,940
of my favorite blogs, Regretsy, keeps getting good content.

318
00:39:32,600 --> 00:39:36,780
So now that, now that I know that's safe, you know, there's plenty of stuff to read in

319
00:39:36,780 --> 00:39:37,160
the future.

320
00:39:37,240 --> 00:39:39,140
My wife will be happy and as will I be.

321
00:39:39,640 --> 00:39:41,200
You can, you can, you can sleep well at night.

322
00:39:41,460 --> 00:39:41,840
That's right.

323
00:39:41,920 --> 00:39:42,120
Yeah.

324
00:39:42,760 --> 00:39:46,080
Well, absolutely, uh, absolutely entertaining that Regretsy.

325
00:39:46,180 --> 00:39:50,360
And I'll tell you, as soon as I tell people that I work at Etsy now, um, the number of

326
00:39:50,360 --> 00:39:53,420
people who say, oh, my wife will be so happy.

327
00:39:55,380 --> 00:39:58,080
It's like, I can't count how many times people have said that.

328
00:39:58,480 --> 00:39:58,880
Absolutely.

329
00:39:59,080 --> 00:39:59,460
Well, great.

330
00:39:59,540 --> 00:40:02,680
Well, uh, well, thanks again and, uh, we'll see everyone next time.

331
00:40:03,480 --> 00:40:04,020
Thank you.

332
00:40:04,620 --> 00:40:05,220
Thanks, guys.