Introductions, Reductive Labs and Flickr
0:00Well, hello, everybody. It's the 11th of February, 2010. And this is, I forget which episode number, but it's some episode number of the Agile Executive. And as always, I'm one of your rotating co-hosts, Michael Coté. And I'm joined by one of the other sort of guest co-hosts. Would you like to introduce yourself? This is Andrew Schaefer. I've been on the Agile Executive podcast a couple times now. I previously was a co-founder at Reductive Labs, and I've been talking about this Agile infrastructure stuff. At the moment, I'm sort of between my projects. I have a couple things in the works. I don't want to jinx them, but hopefully there's some interesting things coming down the pipe. And then we are also joined today by someone that we've talked about on these podcasts and certainly inspired some of my thinking on the topic. We have John Alspaugh with us today, and he's going to tell us a little bit about how he sees this problem and how he solves it. Yeah, so my name is John Alspaugh, and I currently am the VP of Technical Operations at Etsy. I just started there just a couple weeks ago, previous to that. For about five years, I ran the operations engineering group at Flickr.
Growing an Operations Team Through Accelerated Growth
1:15Part of my growing up at Flickr, or at least growing the operations team as the site got large, were a lot of different bits of tools and culture that provided for a pretty great platform to move quickly and to react quickly and to react quickly to all the things that happens to you when you're in an accelerated growth situation within web operations. So I've been feeling quite passionate in the last couple of years about those topics, the collaboration and the communications between what are historically known as development groups and operations groups, where those groups meet, where those groups argue to death, and where those groups can actually work together. Those are the topics that have been on my mind, and how it relates to agile infrastructure, I think, is something worth talking about. Yeah, and I think I first came across your – or you might have put together this talk earlier in this, but back at, I guess, Velocity last year,
Below the Gloss of High-Scale Web
2:33there's – and thankfully there's a pretty nice video recording of it, which we'll put in the show notes. But you and one of your cohorts at Flickr kind of went through – you sort of dug below the gloss of like, hey, all this high-performance, high-scaling web stuff and cloud stuff is like fantastic, which is kind of like the level of conversation most people have. But you actually sort of walk through the processes and the changes that were happening to get the – you know, starting from the top for Flickr itself to deliver that service. And then more importantly, the kind of – not only the technical changes, but also sort of some of the process and cultural changes that development and operations went through. And, you know, it's one of the rare talks that you could actually like do something with it after you see it, other than just think that like there must be some magic idea like hiding around the nooks and crannies somewhere. So on that note, and this is going back to something you said, John, you said this notion of tools and culture. And just to get, you know, kind of your take on it, your spin on it, which do you think has a bigger impact or how do you see that feedback loop between better tools and then better culture? Well, I feel really strongly that the tools will get written if the culture allows the tools to be good.
Culture First, and Everything Else Follows
3:55I think that – I think that – I think if the culture is there, then everything else will follow. I don't think – I think that there's – there is a natural tendency for engineers on both sides of that proverbial fence to look to look to technical solutions first and then try to work culture around them. And I think that I have some experience that would say the contrary should happen. And, you know, there were – the blurring of the lines on solving technical problems I think is a good thing. I think that developers – I think that people who write public-facing code can be awesome at writing operational tools. I think that people who manage infrastructure or have their career or domain expertise in managing infrastructure can bring a lot of really good things to the table in terms of the design of scalable architectures.
If Only One Group Sits at the Table, You Lose
5:11And I think that if there's any situation where one – only one of those groups gets to sit at the table when they talk about implementation or process or anything, then you're going to lose out. Absolutely. Some of the best tools that we had at Flickr were written by the same folks that wrote public-facing tools. And by exposing things like fault tolerance and how to handle load feedback and how to write – you know, how to gather metrics from your application. Exposing those sorts of topics to the people who are writing your application can really, really pay off in the long run. It means that there's awareness of what your application is going to do, not just when you deploy it, but six months, a year later after you deploy it. So providing feedback on that. And I think the vice versa also is – again, I think that you're missing out if you only have developers dictating architecture. And I think that you're totally missing out if you only have operations people dictating things like monitoring and metrics collection.
Why Traditional IT Ended Up Somewhere Else
6:35Well, you know, to back up the context a little bit, one of the things I was curious to ask you is in sort of traditional IT, whether it's public-facing or traditional operations, public-facing or sort of, I don't know, data centers or enterprise or whatever term you want to use for something that doesn't end in a dot-com. It seems like – I guess to ask the question one way, why did it end up that you had to start operating this way? Like what were sort of like the drivers of Flickr or the users of the business or whatnot, to use that term, that sort of made it such that development and ops couldn't stay being two camps? I mean, was there some sort of like moment of failure where it's like, oh, this is just not going to work anymore? Kind of like what was that motivation? And the reason I ask this is because whenever there's some sort of new culture or process change, you know, you get the same response from everyone. It's like, oh, that sounds great, but my organization is somehow different and it doesn't apply to us. It's a special snowflake problem. That's right. Exactly. And so I'm curious, like in the instances you've been involved in, like how you kind of got over that hump of like we're not a special snowflake essentially. Yeah, well, so it's an awesome question and it's something that people ask all the time.
Two PHP Developers and No Preconceptions
7:50So I had the benefit of growing it. We didn't have to change it. The, the, uh, my, my gut instinct tells me that, so I was the third, well, not really the third technical guy, but there were two PHP developers and, uh, there were two PHP developers, um, at Flickr when I started. And one of them was the guy that wrote all the code, Cal Henderson, and the other guy, Aaron Cope, started like a month before me. So until a month before I started, Cal was both writing all the public facing PHP, writing all of the metrics, uh, and, uh, and alerts, um, handling everything, racking servers. So, you know, in, in, in the, in the, in the, at least in the Bay Area, how it works out is, you know, the, the idea of three guys in a garage, they're going to have a web startup. Well, you can't have to, you don't have to worry about operations if you don't have a product to operate, right? So you start with a developer or at least one developers. And at some point you realize, okay, well, managing this stuff or dealing with this other stuff, maybe we should bring somebody in who, uh, can just focus on the operation side of things.
Tools Written by Someone With a Developer Mindset
9:12So as a result, Cal wrote a lot of tools that made things easy for him. He had a developer mindset. He hadn't, he didn't, he didn't have any preconceived notions of, oh, well, I'm going to use a layer seven load balancer here. So I better get a quote from NetScaler. He didn't have any of that. Um, he didn't, uh, he, he said, you know, for example, perfect example is distributing loads throughout the, throughout the application. Maybe an ops centric person might say, I'm going to get a load balancer and we'll distribute load that way. And it's got all these pros and cons for it. And the application developer historical, or at least in Cal situation, I'm going to get an array, fill it with host names and iterate over it. However I like. So this is the same results. You still have, uh, load being distributed, except ones with an expensive piece of hardware. And the other one is that the application took on that task of doing it himself. The part that is subtle is that Cal's thinking in, in, in, uh, in distributing load within the application meant that he had to actually think about how the load was going to be distributed within the application.
Nothing Had to Be Changed, It Started That Way
10:31It is, in my sense, less about what you would find in a, uh, in a historic, uh, or a typical developer mindset. So, uh, the, the, what I, what I'm trying to get at is that I didn't, I didn't have to come in to Flickr and change anything. It was, it was, it was, it started out this way and it just, uh, led by example. I'd say that both Cal and myself, as time went on, uh, we led by example. So any of the, any of the operations people on my team, um, respected developers, there was no, uh, well, this is my job or my responsibility and you don't get to touch, you know, developers don't get to touch this or that, or, you know, production is pristine. So you don't get to touch this or that, um, and it turns out you're the special snowflake then. So this, well, let me, I'm going to throw a comment out and then I want you guys to both, uh, respond to it. But I kind of feel like when people talk about agile, whether they're talking about developer practices or as we evolve sort of this, this new, uh, ops angle that the, the best practices really come out of creating the proper habitat and ecosystem.
It Is 2010, There Is No Excuse for Ignorance
11:52And then the, they just sort of emerge because I, well, barring ignorance essentially, but it's two, it's 2010 people are doing this. People are talking about it. There's no reason that you have to be ignorant of what people are doing and how they're getting their results. So if you create a culture, going back to the original statement, if you create a culture where that is, it's a culture of learning and it's the proper habitat, then people will just behave this way. And, and there's a quote, I can't remember who made it, but it's something to the effect of, if you want people to, to follow a process, then you should just start doing it. If you want people to ignore a process, then you should document it. Yeah. No, and, and, and, and I think, you know, the, I, I think the, the patterns of adoption for changing culture in a technical organization that seemed to be six. One of them that seems to be successful is, is like what, what you're getting to is that it's the easiest way to convince someone. Well, one of the easier ways to convince someone to do something is to actually kind of reverse it on its head and start talking about the benefits of being that way, essentially, rather than it just, you know, rather than saying this is a hygienic better way to operate and we should try it out. And that's, that's the nice thing that, you know, the folks, the folks like, like John and others who are doing this DevOps thing, they have, they have kind of the track record and the stories to tell.
The Glass-Half-Full Case for Cultural Change
13:08They're not only sort of like, they're both problems that we fixed essentially in, in traditional problems that we fixed. But what becomes even more interesting are the more, what I always call like the glass half full sort of things, like the new things that you can start doing that are interesting. And I think, I think those two things together becomes a pretty powerful motivation for, for what seems like an impossible cultural change that, that you get onto and it helps get over that hump. Yeah. And I think that, um, uh, I would, I would agree and, you know, not to, not to make this any sort of a segue, but I do believe that, um, the, the idea of the infrastructure, um, is, you know, if you're only a button on a webpage away from pulling up a, uh, for provisioning a server, sure. There's all of the regular advantages of, of, of, you know, uh, forecasting and economics of scale and, and, and, uh, you know, burst amounts of capacity, blah, blah, blah. But what it really means is that, uh, you, you've now won, you've, you've pulled away from what is essentially somewhat ineffective. Um, way of doing things, which is everything needs to be metal and I need to buy, I need to buy the metal and, uh, that sort of thing. Um, to, again, smearing out some of the, um, some of the concerns, right?
Everybody Owns Availability
14:29So like we used to say it at Flickford is that everybody owns availability, right? Operations isn't the only one who, uh, is, is tasked with making the site, uh, fast and available. And you'd be hard pressed to find a developer who says, yeah, I'd all, no, I just want to write my code. And I don't really care if the site goes down, like you're not going to find that guy. Or if you do, he doesn't, you know, he's, he's not a very good developer. Right. Um, so he doesn't work there very long. Right, exactly. So I think that cloud computing, um, uh, has helped, has helped these, this, this smearing out of not just responsibilities, but domain, uh, expertise. Domain knowledge. I think that it's a good thing, absolutely the best thing because it, it, it then entices developers to learn more about how their, how their, their application is, is working with infrastructure. And it also focuses operations people, um, less on, on how good their tabling is and, and more on, on other things that can be, you know, more important. And I'm absolutely more than happy in pushing, you know, physicality, uh, to this down the stack, somebody else.
Eight Data Centers and Nobody Touching a Server
15:48And at Yahoo, that was pretty much the case. You know, there, there wasn't anybody on my team who ever had to touch a server at Flickr. We enjoyed having the, the mothership of Yahoo have horizontal platform groups. There's, you know, we had eight data centers, you know, so there were, and none of them were anywhere near our office. So it's not like any of my, I had to have anybody on my team that had to go to the colo. So what that meant was that they could, they could, they could focus on really the, the, the harder problems, the more important problems that come up at least, at least on a fast growing web 2.0, uh, you know, type of site. So, so to, to, to that point of like, uh, you know, having this sort of process and culture change. I mean, can you give us kind of like, uh, uh, an abbreviated sort of run through of like, what kind of like the top sort of changes that you see the operations and developers, people going through when they, when they get to the, uh, you know, when they, when they transform into this way of operating. I, you know, I realize that's, uh, incredibly broad, but, but, you know, I'm just curious to hear like, what, what are the major ways they seem different than, than traditional or, or typical sort of ops and dev people?
Boundary Objects, and Deployment as the Obvious One
17:03Um, well, so the, the obvious example, I think, and, and, and Andrew's, is pointed this out multiple times in various places, but, um, and you call them, I think, I think Andrew, you call them boundary objects. Um, but one, the most obvious place obviously is deployment. Um, and this, this, uh, so at, at Flickr, developers push their own code. Um, it is, if you want to call it continuous deployment, um, which is. Uh, another way of saying pushes, pushing small changes often. Um, I'd say that, I'd say that that's a, that's a perfect example of where collaboration and communication, um, can bring those two groups together. Uh, again, we talked a little bit about it at our talk at Velocity, some of it taken out of context, but, um, you know, the, uh, I, I don't, I don't know how much, how much detail you want to, you want me to talk about it, but I, you know, I think that, um, deploying, uh, deploying small changes very frequently is it as an operations manager, I want that to happen. I believe it to be a boon for availability.
Five Changes From Three People
18:18It, because it means that, that if there is an issue that the, are my meantime, uh, my meantime to resolution, my meantime to detect all of those things are going to be different because guess what? That last deploy only had five changes from three different people. That's it. You know? So, so I think that's a good, good, good crystallized example of, of, of where this stuff can happen. And you go back to the, the classic ideas that the agile sort of founded on and you think about how people do most of their, their web operations, you know, flicker notwithstanding and some of these other people who are moving in this direction, but you essentially have agile, agile, agile, agile, you know, small changes, small changes. And then your deployment process is essentially a big waterfall, right? That's, that's like, that's how most people do this. Yeah. Yeah. No, I, I was speaking to like, uh, the local Austin agile group recently about, you know, figuring out what, what you can do with the cloud. Essentially, you know, these are the kinds of exciting talks I ended up having with, uh, with, with the general audience. And, and, you know, as, as I, as I was going through what I was going to say to them and talking with them, I was realizing that, you know, it, it sort of is, uh, and, and I, I think, I think you guys pointed this out, John, in, in the, in the velocity presentation, but you know, the, the, the ops people essentially that they've kind of been stuck in this role of basically their job is to say no.
Change Management as the Art of Changing Nothing
19:42And, and change management started sort of becomes, you know, no management essentially change management is the art of making sure nothing changes. And, you know, on the one hand, it's frustrating for everyone because it's sort of like they become a bottleneck or a wall. But then on the other hand, there's also this, you know, the only time people really notice operations people is when something goes terribly wrong. Like, you know, there's not like, uh, you know, send flowers cause your email's working perfectly fine sort of things going on with ops people. No one really thanks them when everything's humming and great. And so, you know, it's just over the years, you develop this culture that, uh, if you're only going to be punished, then you basically just do everything to avoid punishing and you don't do anything to really bring rewards on. And, and it is like, you know, when you get down to something as, as, you know, taking those, those like, I don't, you know, vague head in the cloud sort of notions. And then you apply something like having very small changes, right? Continuous deployment. And then like you're saying, you start to understand that like, even if something goes wrong, it's really, it's easier to sort of figure out what's going wrong. And it should be easier to control something not going wrong in the first place than if you sort of have these giant waterfall sort of processes. And it's sort of like, uh, as happens all the time in tech world, it's this classic Stockholm syndrome where the person who's trying to work around a problem becomes the embodiment of that problem rather than trying to come up with some way of preventing it from happening in the first place.
Velocity as the Center of Gravity
21:00Right. Absolutely. So John going, you know, it's been eight months since the velocity talk, give or take, and velocity is coming up again. And I, we, we, we constantly reference velocity in these discussions because I feel like it's sort of the center of gravity for people who are really pushing the practice forward in terms of operations. And although it's, it's quite web centric, the best practices that are being developed at these companies who are really solving these problems at scale can be applied to, to the enterprise, you know, across the board. And you sort of, you sort of made mention of the, of the talk, but I've also heard some people, uh, operations people in particular who are, they're complaining a little bit about the talk in the sense that the, the, the development teams they work with somehow the only thing they took away from, from you and Paul on the stage was that now we can deploy whenever we want to. Yeah. Yeah. And yeah. And they're, I think they're missing the part where you only get to pull, you only get to deploy whenever you want to, if you have a history of non incident deploys. So, you know, like I said, like we said in the talk is that, you know, you can't, you can't claim that you're agile and nimble, uh, you know, and you deploy 10 times a day if you go down 10 times a day.
Not an Excuse to Be a Cowboy
22:22And, um, uh, you don't, it's, it's, it shouldn't be used as an excuse to be a cowboy. The, the whole, the whole point is actually to spread the, the onus or the, the, the burden of thinking about availability, um, across the whole team, instead of having it just sit with the guys who are the least, uh, uh, enabled to, to control. So, um, you know, I, I, that's, that's, that's, that's was somewhat disconcerting. I probably blame, I, I blame that we probably didn't spend enough time on that topic. I, there, there are a couple of folks that are presenting in, at velocity. I think, I think you should blame Paul cause he's a developer. Well, you know, towards, you know, uh, even, you know, towards the end of, of my working at, at flicker, actually not even towards the end, you know, you know, that you're in a situation when, uh, you know, you're in a situation. You know, Paul and myself, uh, the, the sort of twin leaders of the, of, at least of the technical culture there, um, when Paul is arguing for us to be, uh, more conservative, um, and, and I'm, I'm pushing for us to, to, to say yes instead of, instead of Paul.
Role Reversal, and the Pendulum Swinging Back
23:43Paul is the other way, a little bit of role reserve reversal, but I, I think that that's actually, um, healthy if the pendulum goes in the other direction. Um, yeah, I mean, if you got out of the talk or if you get out of, of, of anything, and it's not just us, you know, Eric Reese, um, uh, um, and, and, and, uh, Tim Fitz talks about very similar things that, that, that happens at, at IMVU. And, uh, um, but, uh, you know, you, you weren't really listening if you think that, uh, see, well, if, you know, if, if John Osborne, Paul Hammond say that we should deploy 10 times a day, then we should do it. And, uh, those ops people can, you know, suck it and just let their pages, you know, their pages can catch on fire because those guys are doing it that way. Yeah. Yeah. Yeah. I mean, I think, I think a lot of the discussion you, you guys have had in that talk and in other places is kind of, uh, you know, the aspiration is to be able to do that, to, to deploy many times a day or, you know, to deploy a lot more. And, and, and I, I guess the thing people have to realize is like all this other stuff, like the other 95% of like the conversation is all about making that possible essentially. And you kind of like, you can't, you can't just choose like that five or 1%, you know, you can't just eat the donut.
The Dessert Tray
24:58You got to eat all this other stuff that comes with it. That's what I call this. It's the dessert tray. Agile, you only eat the dessert. You know, one, one last that I'd like to say about that. One thing that it's probably not very clear is, um, after we were acquired by Yahoo, the, we were, we had this deployment, uh, methodology. We were doing the small changes often and we were growing, right? We went from the 25th most popular, um, Yahoo property to like the fifth, four years later. Um, and, uh, we, so we had insane growth, but yet this, you know, methodology and through various, you know, reorganizations of Yahoo management or whatever, um, various people from, from further on inside the mothership of Yahoo would say, Hey, I heard this rumor that you guys deploy 10 times a day. Is that right? And somebody said something about developers pushing their own code. Please tell me that's not true. And, you know, that's the part where the jaws drop on the floor. And I say, and Hammond and I say, no, actually that is, that's the situation. But what's missing is that the Yahoo service engineering or Yahoo operations, um, leadership never would have allowed us to continue the deployment methodology if we didn't have the history to back it up.
Knowing When to Turn the Volume Up
26:25The most important thing about continuous deployment in my mind is knowing where to turn, where and when to turn up that volume. If I have a history of incidents that have my, my meantime, my MTTR and my MTTD is really high, well, guess what? That's, that's an indication that maybe you're going too fast. Maybe you should slow down a little bit because it, it, you know, the metrics say that you're a cowboy. So you only get to do that if you have a history to prove that you can, that's, that's, that's, that's, that's what I think is probably missing from, um, from the conversation that, that at least I've put forth in the, in, you know, in, in the past. So you can go as fast, you can go as fast as you want, but don't rush. Well, it's just, you know, just, yeah, exactly. Don't crash the car. Uh, you know, um, it's again, you know, keep track of, uh, you know, what is your, what is your deployment to incident ratio? You know, is that, is that ratio really high is what percentage of your, of your deploys, I don't think it doesn't matter how many changes are in each deploy, how many, how many, uh, incidents have you had for every 10 deploys, you know?
The Goal Is Not Having Incidents
27:49So, um, the idea, the goal there is to not have incidents. That's the whole point. So, so yeah, the, the, the missing part outside of the donut is, is, you know, there's, I'm, I'm, I'm, uh, breaking down on the food analogy here, but you know, you have to, you have to, you have to, let's go to, let's go to sports. I think it's, it's like your turnover to, turnover to assist ratio is basically what you're looking at. Yeah, exactly. I think, I think, I think to your point, like one of the more interesting things and all, all of this kind of talk, whatever, whatever you want to call it, whether it's agile infrastructure or cloud stuff, or high performance hoopty or whatnot, is the, the, one of the interesting things that's come, comes true, like, you know, you mentioned MTTR and MTTD and all these things is, is you, I noticed that most people who do this become really metrics obsessed. And it's sort of like metrics become one of the main things that both developers and operations people begin to be really interested in. And what, what becomes interesting there is, is they're not so much, they're not only interested in traditional metrics, like, you know, CPU and storage and network bandwidth, but they start, you somehow need to enable them to put their own sort of metrics in there to instrument the systems that they're doing. So that they can start sort of taking the, you know, they can, they can, they can not be a cowboy by having metrics that tell them when they're getting too close to that, essentially.
Every Application Needs Its Own Metrics
29:10And it seems like, it seems like no matter which software you're working on, you're probably going to need some sort of custom metrics. So you need some system that enables that to be possible. Oh yeah, of course, absolutely. And in fact, if you think that, um, you can just take a, a, a, a canned set of metrics from one application in company to another application in company, then you're, you're doing it wrong. Um, there's, you know, uh, there are, there are application level metrics that are important to some people and, uh, to some applications and others that aren't, you know? Um, um, Flickr has photos, Etsy has listings, Twitter has tweets, you know, uh, there, there are multiple, multiple metrics for, for multiple applications. And if you're not really keep, and if you're not keeping track of what your infrastructure is doing alongside those metrics, well, then you're, you're, you're only listening to one guy in the band. You're not listening to the whole song. So to, to, to that point, and this is something you, you had in your, in a slide at Velocity, the, the job is to enable the business.
Everyone's Job Is to Enable the Business
30:25Everyone's job is to enable the business. And if the business, if the business value isn't delivered, then, then you can't do all the fancy stuff with all the technology. Right. Yeah, absolutely. Cause that's, that's the whole point at the end of the day. Yeah. That was when it like down here in Austin, a few weeks ago, we had, uh, like the first ops camp and it was kind of, it was disheartening and also heartening. It's always our cyclical nature of these kinds of ideas that there was this group that was like the, uh, you know, the, uh, the craft of system of operations. And, you know, it's a typical sort of like career therapy, which is always fantastic. Right. And, uh, and, and, and, and it was funny, like after 10 or 15 minutes, everyone sort of reached this conclusion is like, oh, we're supporting the business. So we should, you know, we should focus on that. And, and the disheartening part about that is, is to realize like the last 10 years, there's been all these like big vendors in the industry and processes that are trying to drive everyone towards that. And that it's still like, uh, it's still like this big realization for people to go through that, that, you know, oh, what we're doing is we have customers who are paying us to do something or, you know, giving us their eyeballs or whatever. But, uh, you know, in the, in the same way that, you know, hopefully it's, it's a similar pattern to sort of like SOA versus sort of like restful open API things where like the original ideas of SOA were fantastic. And then it went all like crazy, but then we kind of, you know, fixed it after a while.
The First OpsCamp in Austin
31:43So, uh, anyhow, hopefully that'll happen this time. Yeah. Yeah. I mean, that it's, it is also, you know, it's, it is difficult. It's hard to, um, it's hard to see the, the large picture sometimes. So on that note, we, we often mentioned velocity and, and as I said earlier, I think it's, uh, sort of the, the center of gravity for, for this sort of thought. And I know you guys are working on putting together the, the one in June. So maybe you want to kind of talk a little bit about what, what people could expect to see there or, or why that's exciting. Yeah. Uh, yeah. So I'm, I'm part of the program committee, um, and, uh, and the, the deadline's passed for all of the proposals, but we're going through. All of the proposals at the moment. Matter of fact, we have a conference call tomorrow, um, for all the people in the program committee, um, to go over the proposals and that sort of thing. And, and, and so it's a third year. And I think that the, I think that the, the agenda and all of the things that's important, um, to, uh, to both the operations and the front end performance communities. Um, I think there's, um, uh, much, uh, much more broad, uh, and both broad and focused at the same time.
What Is Coming Up at Velocity
33:03It doesn't make any sense. Um, but, um, there, there are really good proposals, proposals that I wish people, I wish that there were last year. Um, but I think that it's, it's, I think that the conference, uh, in general is, is picking up steam. Um, without, without, you know, without speaking about, uh, uh, any particular proposals, I think a lot of the stuff that we're talking about, um, today, uh, is going to, is, is going to, uh, show up in the conference. So there's a, there's a question I have just because people are doing this all the time. They're losing their minds. I think I have more drama from the cloud computing experts in, in Twitter than any other group. But when people are talking about cloud computing, there, there's sort of a nebulous idea of, you know, the clouds and like sys admins are going away and we're not going to use physical hardware anymore. In your experience working at, at, you know, Yahoo, Flickr and Etsy, well, how much do you see these sort of, you know, from Facebook to Google using what are the, you know, the cloud, if you will, technologies? Oh, well, uh, well, personally, I think that anybody who has had any, uh, experience in, um, in growing, uh, uh, at least a web operations infrastructure, uh,
Operational Concerns Are Not Going Away
34:26realizes that especially with applications that are, are, are more social and less static these days, um, anybody who thinks that, uh, that operational concerns are going away is, is fooling themselves. Um, uh, I, I, I, it's just, it's not a possibility. If, if, if operations to you means physical people to, you know, people to physically move iron around and cable them up, then, uh, then, uh, then you probably work on a relatively boring application. Um, that's, you know, like I said, I've had, there are eight, when there are eight people on the Flickr operation engineering, uh, team and, uh, none of them have to, have to worry about things like BGP and OSPF or swapping drives or going to a polo. And they don't need those little Velcro wire things or anything, right? Yeah, exactly. Uh, and no static guards or anything. All of those people are very, very busy doing things that are important to the business. So, no, operations isn't going away whatsoever. As far as cloud computing is, as, uh, um, sort of as a topic, I think that the answer, whether people talk about it or not, is, should be already clear.
Using Infrastructure as a Service for What It Is
35:46And those people who use the, the, the smug mugs and the animotos and the, um, picnic, the people who use cloud computing, or at least, uh, uh, I don't know, whatever. Or insert, insert, insert debate about definition here, um, infrastructure as a service or whatever, Amazon web services. Um, those people who use it successfully use it for exactly what it, it is designed for at the moment. Um, I don't, I could be wrong about this, but, you know, so Don McCaskill at smug mug is a perfect example. He's always a really good example of cloud computing. He uses it for, uh, processing images and storing photos. Um, as far as I know, I could, I, I certainly, uh, could be wrong about this. I believe that web servers and databases are still on premise for him. Um, I don't know if he's moved his database layer, um, to, uh, to Amazon web services just yet. I know that picnic has not, and that's just because an application specific requirement that you're building dynamic web pages and the, the, the boxes that front that, that serve those web pages needs to have low latency, uh, um, generally, uh, random, uh, access workloads, um, uh, to resources.
Databases, Persistence and Large-Volume LAMP
37:15Uh, AKA databases that are right next to it and have to have some, uh, some, um, reasonable amount of persistence or durability. So, um, you know, um, you know, I still think that there's, um, there's, there probably will be a day when every piece of what is, what, what is deemed to be a, uh, of a full, or at least a full lamp architecture or, or, or, uh, uh, at, at large volumes, um, maybe there's, you know, something like that. But in the, in the meantime, uh, there, there's this, there's this spectrum of two guys in a garage, small site on one side and Google, Facebook, Flickr on, on the other side. And at some point your application or parts of your application are simply just too big for either the economics or the, uh, uh, the architectural limitations to make it worth moving to the cloud. Um, if that was the case, Facebook would be on Amazon web services. Yeah, no, I mean, it's an interesting point you're making about the, the sort of cloud services that are available where, you know, there's, you, you may not be able to treat it as generic as you may think you'd be able to be able to treat these services.
Match the Service to the Capacity You Need
38:39That it's, it's good to make sure that you're still matching, you know, what capacity you need and what functionality you need to sort of the service that you're using. And it's not sort of like this panacea of, you know, it's replacing my data center essentially, which is, is, is sort of practical advice that it's good to hear ahead of time before you get, you discover it on your own essentially. But, uh, anyhow, but, but with that, I mean, I mean, it's, uh, we, we should, uh, wrap it up here. I mean, I appreciate you, uh, spending so much time talking with us about it. It's, it's always nice stuff to, to, uh, you know, just discuss casually. So that's fantastic. Yeah, this is, I love talking about this stuff and, um, I'm, I'm, I'm happy to, um, to gab on about it. I mean, you guys are, uh, you guys are great for, for allowing me to ramble. And, and, you know, I have to say now that I know you're at Etsy, I'm, I'm very happy about that because, you know, Etsy needs to be kept up so that one of my favorite blogs, Regretsy, keeps getting good content. So now that, now that I know that's safe, you know, there's plenty of stuff to read in the future. My wife will be happy and as will I be. You can, you can, you can sleep well at night. That's right. Yeah. Well, absolutely, uh, absolutely entertaining that Regretsy. And I'll tell you, as soon as I tell people that I work at Etsy now, um, the number of people who say, oh, my wife will be so happy. It's like, I can't count how many times people have said that. Absolutely. Well, great. Well, uh, well, thanks again and, uh, we'll see everyone next time. Thank you. Thanks, guys.
Machine transcription, lightly formatted. Chapter headings were written from the transcript; speakers are not identified, so nothing here is attributed.