Third Part of the Mini-Series
0:00Well, hello, everybody. It's the 21st of August, 2009, and this is a third part in a little mini-series we have going here at the Agile Executive Podcast, or within, I should say. And as always, this is Michael Coté, one of the co-hosts available at peopleoverprocess.com, and I'm joined by one of the other co-hosts. This is Israel Gath. I'm partnering with Coté on the Agile Executive, and I am really very, very excited about the whole mini-series, and in particular, the piece on Agile infrastructure that we are going to get into today. And with that, let me hand it over to the third participant. This is Andrew Schaefer. I'm co-founder of Reductive Labs. We make Puppet. We're trying to help people build better IT systems with better tools, and we get to see a lot of different approaches and a lot of different infrastructures, and that's led to some of these insights. And as people may recall from the last two episodes, the three of us were just talking about the idea of what Agile infrastructure, Agile operations is, and sort of the motivations and the benefits and kind of what it looks like, if you will. And last episode, we talked a lot about just the idea of broadly how you can introduce this change and problems in the
Narrowing Down to Practices and Technologies
1:19IT department of introducing changes. And next week, Andrew has a presentation that's going to kind of sum up some of the things that we talked about and other stuff, but also get into the details of sort of practices and technologies you might actually use to do this sort of Agile infrastructure stuff. And so it seemed like a nice opportunity to sort of narrow down and sort of conclude, you know, kind of the niftiness of what we've been talking about with Agile infrastructure. And I think to that end, like the first thing we should start talking about is, I mean, we've kind of danced around this in the last few episodes, but we've more talked in the sort of the happy path positive way about what Agile infrastructure would be, but we haven't spent a lot of time talking about sort of like what the opposite of it are, sort of kind of what the existing problems are that it kind of seeks to address or fix, if you will. And so, I mean, with that, like when you're kind of looking at applying these practices and coming up with these things, Andrew, like what, what are sort of the issues in IT that, that, that you're fixing? So the, the project I work on specifically, Puppet, I think that that's the thing that addresses most effectively is this configuration drift, which is essentially the differences between
Configuration Drift, and Why It Reaches Certainty
2:38machines that are supposedly configured identically. In, in most organizations, if you have two web servers or two database servers or two whatever servers, the chance that they are configured identically is very small. And as you get to 10 web servers or hundreds of web servers, then that chance goes to zero quite quickly. Right. And these, these differences, the, the drift between those configurations cause confusion, they cause mistakes and it leads to a lot of downtime. It costs a lot of time and, and the change man, change management practices that most organizations adopt on almost every level are to impede change. And we touched on this before, but the business, the, the business is enabled by change. So changes, it's, it's got to happen if the business is to move forward, but at the same time, it's putting up all sorts of roadblocks in front of itself to prevent itself from the pain that, that, that the change in the past has brought. And what we're trying to do is move
Wary of the Label "Agile"
3:55things in with using better tools and better processes towards what I consider, you know, agile infrastructure utopia. And agile is a word that I, I, I've been practicing agile techniques for five years. I've been at the Salt Lake City agile round table, but I've also got a lot of love for a lot of other ideas. And I, I don't particularly like labels. Um, agile is a can convenient label on some level, but I don't want to, I don't want people to think like, Oh, I can't do this cause I'm lean. That's right. Cause that's, that's silly. Andrew, may I ask for your opinion about the following perspective, which I believe is completely consistent and complimentary to your view. The way I, uh, perceive agile to be, it's really about a set of principles which are universal in nature. For whatever reasons, those principles have effectively been applied first in the software development. Within software development, we are actually talking often
Scaling Up, Scaling Out, and Scaling Downstream
5:12or Jim Highsmith to be more precise is talking often about scaling up. Namely, agile is applied to a larger team, a team of 500 instead of 50 and scaling out, not a team in one site, but a team in three different sites. I often speak about scaling downstream. And what I mean by that scaling to the other functions in the business, be it marketing, sales, professional services, whatever. The view that I have of the agile infrastructure is that it's the application of the agile principles in the domain of IT operations. And this is another scaling or expansion. If you please do agile for me. And it's so very tempting for the reason that in so many data centers, you know, there is one arm that develops software. There is another arm that runs, maintains, operates a software. Absolutely. And the consistency in the principle has tremendous potential in terms of bringing those two
Half the Story Is Borrowing From Software
6:28together. Absolutely. Let me, let me expand on that point. So I think it's, it's, that's half the story. On some level, it's being able to take some of the lessons that we've learned from the agile principles being applied to the software and applying those two to operations and, and that infrastructure level. But I also think that there's, there's some other things that are happening that make it a little more interesting in the technology. So, so not only can you take these, these lessons and apply them to the operations, but there's, there's things that are changing. And there's information and interesting things coming from the other direction. So on one level, it allows you to, to apply some principles. Yes. But on the other level, on the other level, the things that are happening with, with abstractions from the network layer, the storage layer, and the competition, computational layer, and with the abstractions that, that something like the puppets configuration management give you, and then orchestration, things like control tier Capistrano, all these things are code. So you can start to think about building infrastructure, like it's software. So, so I think it's going in both directions. So I don't want people who are, are sys admins. Are sys admins and ops teams to, to sort of lose their roots? Because there's a lot of knowledge, there's a lot of domain knowledge that I don't want people to, to lose touch up touch with. I know as a developer, I'm having a kind of a develop, developer centric view of the world when it comes to technology.
Fusion of Worldviews, Not Assimilation
7:58And that's not always optimal for managing systems or for security. And these people that have that deep domain expertise are required to balance out my sort of shiny new thing, fly by the sea, my pants, developer attitude, if that makes any sense. Are we really talking about fusion then? I think there's a, there's a fusion, but then there's also a recognition of, of worldview. And I think if you, sometimes when you try to, to assimilate cultures, you lose the, the unique benefit of that particular worldview. So I'm a little hesitant to say that it's a total assimilation. Is that me? Yeah, it's extremely good point. In a way it is about the combination of staying close to my roots while I fuse with other roots. So it's about, I think there's two sides to it. So you don't want to forget who you are, but at the same time, you don't want to limit who you are. So you, you are, you know, things, you are things, you, you, you have all this knowledge and domain expertise, but at the same time there's stuff happening around you and, and it's foolish to pretend like it's not there and not learn from it.
A Culture of Learning in Every Role
9:18I think that's sort of the, the, the principle of learning that, that, that I think if you apply that to every role in, in anything, whether it's agile or not, um, from, from business to development, to testing, to product management, to marketing, as long as you are in that culture of learning, then that's going to be an advantage for yourself and for your organization. So it's sort of like, it's sort of like having a, uh, a lack of continuous improvement essentially, which is, is, uh, it's, it's a core problem that causes all sorts of, all sorts of issues, but like, like, you know, I, I mean, I'm curious drilling down on, on, on, on the, the sort of issues, you know, the, the problems, if you will, more like the, uh, I remember at open source world, we had, we were, I was on a panel discussing, um, just, uh, you know, open source IT management stuff. And, and, and the question came up that basically boiled down to, uh, when is, when, when are, is all this IT management stuff going to work together better? And the, you know, the best answer that we had was sort of like, well, it's all just complex stuff. So what are you going to do? It's always going to be difficult, which is kind of a frustrating answer to give to someone. But, but, you know, to applying that question kind of here, like, like you're saying, and it's very true, like the, the role of change management is, is, is usually to limit change as much as possible.
Why Is IT So Brittle?
10:38And, and I wonder, like, what, what's up with IT being so brittle that it's so prone to screwing up when you change something? Like, how, how do you think it's come about that it's so, it's so complex? Well, let's just talk about that for one sec. That, that panel was with my, uh, Conrad at arm. So I'm going to give a shout out to, uh, Luke Kinnis, who's, uh, out there somewhere listening, perhaps. The, the fact is that's not just unique to IT management, right? There's all sorts of things that you want to talk about things, integrating, working together. How many applications out of the box just kind of work together, right? Yep. And, and as engineers, I think we have a tendency to, to love cleverness and complexity. And that, that, that love affair with cleverness and complexity, while it's, um, entertaining to me personally, it doesn't always, it doesn't always give you, uh, the flexibility that you think you're, you're giving yourself. And, and sometimes you're so clever that no one else can figure out what you did. And if you, if you created some IT management script or, or framework or what have you in the programming language, and then you got a new job and you left, the next guy comes in and he's unraveling what you did.
Forced to Change With Information Missing
11:55And, and, but he's still being forced to, to do things because, well, you know, the business needs to change. Then, then there's some inherent, um, information missing, right? So, so sometimes you fill in the blanks and that, that leads to assumptions and, and mistakes as well. Right, right. Yeah. I, I, I guess we can't all be like as artful and cleverness as like, you know, like I think people, when they're trying to be clever in programming are trying, aiming to be sort of like Lewis Carroll, right? Where you've got a, like, you know, a book and writing full of like, you know, little puzzles and riddles and all sorts of things. But, uh, you can't really come off that well polished and entertaining in the end. I mean, that's, that's a, a complex, clever book, like, you know, the Alice in Wonderland series, but, uh, it's actually difficult to achieve cleverness that's enjoyable and entertaining to consume, if you will, to be, to be mad. So I, I might be dating myself, but there, there's, I think it's Superman two with Richard Pryor and he's, he's trying to, he's trying to figure out what kryptonite's made of. And, and he gets to where he's done an analysis on this, this compound and he knows what everything is except for this, this small fraction of a percent. And, and, and, you know, his techniques can't figure it out. And he, he looks at his, uh, pack of cigarettes and he sees tar. So he just fills it in with tar. And I think that's how, that's how a lot of, uh, problems get solved. When, when we get to the end, we're just, uh, well, this might work.
The Wall of Confusion
13:13That's right. So are, are, are there any other, like, problems to kind of lay the foundation of, of what, kind of what we're fixing here? So there, there's the configuration drift problem, which is, um, a big problem in systems management. There's also, when you're talking about organizational problems, there tends to be a, a wall of confusion between dev and, and operations. And if you listen to me talk all the time, you're probably sick of hearing me say that. Um, but, but it's certainly a problem. And there, and there's really, we talked about this last time, but there's kind of a wall of confusion between every business unit in, in a lot of cases. Um, but the, but the one between dev and ops is, is particularly, um, antagonistic sometimes because of the, the fact that someone threw something over the wall. And I got, you know, a pager call at two in the morning and didn't make me happy. That's right. It's sort of like a lack of controlling your own destiny. Yep. And, and then I think this is also a problem in, in system management. It's a problem in, in development as well, depending on, on the culture. But I think agile does a reasonably good job of balancing this out with, with a concept of, of shared metaphors and teamwork. But in, in a lot of organizations, the, the sys admins have a very strong hero culture. So you, you have incentives on some level to, to let, um, and I think it's deeply rooted and it's almost psychological.
The Hero Cycle
14:34And some people aren't even necessarily aware that they're doing this, but, but when, when there's a fire and someone comes in and puts it out, then they're the hero. Right. And so you, you kind of create this, this hero cycle. Right. And it, and it doesn't necessarily, I mean, I'm a big believer in trying to figure out how you can align incentives with behavior. And if, if, if your incentive is to be a hero, then you need, you need problems so you can be heroic. Yeah. Yeah. Well, you often get, uh, in all walks of life, you're rewarded, you're rewarded for, uh, solving a crisis as it comes up a lot more than for preventing a crisis or just, you know, uh, taking care of, you know, people don't get rewarded very well for just filling in potholes. They get rewarded when they like help, help, like shut down a riot in the city or something like that. Right. And I think it's particularly bad in ops teams because on some level, and it's going to be totally cultural and it has to do with how they interact with the rest of the business units. But you have, if everything's going well, then no one knows you exist. Right. You know, this reminds me of some of the writings of Rob Austin of Harvard University. His basic thesis is that you have to change your measurement or incentive system every so often because one puts in place a measurement system in order to induce a certain behavior.
People Learn to Game Any Measurement System
16:05Within some period of time, people learn how to game it, how to satisfy the incentive systems instead of how to satisfy the thing that it's aimed at optimizing. And the only way that you have to go around this cycle is to be diligent and change your, uh, performance measurement system at some frequency so that people stick with trying to do what you are trying to accomplish rather than, rather than satisfy the performance, the performance system. Right. Definitely. So there's one more problem I'll just go over. And I think there's a problem in dev too, but I, I really, I dislike this notion of non-functional requirements. We've been over this before a little bit, but I think in, in a lot of agile implementations, uh, there are situations where, you know, you go to the product manager or whatever, you know, whoever's sort of making some of these decisions about how things get prioritized. And you explain to them, if we don't, if we don't, if we don't do this thing or upgrade this thing, then it's going to cause all these problems for us. And it's going to spend, you know, we're going to spend a lot of time fighting fires that we wouldn't have to, if we can just fix this thing or if we can just invest in this infrastructure.
Non-Functional Requirements Lost in the Backlog
17:26And that's, since that's a quote unquote non-functional requirement, it, it often gets, uh, deprioritized and then just sort of sits in the backlog and the, in the churn of the, the backwash of the backlog, if you will, for, for, for, you know, for time and all eternity. And, and, and I think that's a, that's a cultural problem that, well, in some cases it's out of, out of the hands of, of the dev and ops team. I think that needs to be recognized at a higher level up the decision-making, uh, chain. And you can solve a lot of problems and save yourself a lot of money if you invest a little bit in some of these non-functional requirements. Andrew, let me ask you the following question. Uh, it's basically the thought that you triggered in my head. What we oftentimes are doing agile is we break the current organization into a team-based structure. So you have the scrum team and this is the structural unit, uh, the structural organizational unit in whatever we do. When I listen to you, I'm inclined to think that, and tell me whether I'm putting words in your mouth, is that you are thinking in terms of the scrum team or something like the scrum team being expended to include IT.
Putting Data Center People on the Team
18:47Namely, here we have the product owner and here we have the developers and the testers and whoever else. And you are sort of suggesting that the, uh, fractal scrum team would be with members of the IT, uh, data center. And furthermore, that those are ongoing participants because of the context in which they operate, uh, by, uh, its own nature requires different solutions from the R&D side. Is that where you are heading? I'm a, I'm a big fan of this concept of, of fractal teams. And what you just said, I think that some of what I talk about is a little, uh, web ops specific. Um, but in general, if you're building a web application, your infrastructure is that application. Your application is that infrastructure. If those machines aren't kept up, if you don't have someone with the, the skills to, you know, the, the system husbandry, system administration, innovation skills to keep those machines up, you're not delivering business value. You're down, you're done. Right? So if you can take advantage of that domain expertise through the cycle of planning, just like you would any other, uh, domain expert on your team,
Problems Melting Away When Dev and Ops Pair
20:10then you're going to be better off in the long run. And I, and I know, and I've seen problems melt away when you get, when you get dev and ops pairing to solve a problem that crosses that domain boundary. Well, you know, the experience that I had, having been in system management, the better part of my life, the experience, somewhat traumatic experience, uh, that I had is that you need so many people on the bridge when something, something breaks. So you get somebody who's good in the database and somebody who's good in the networking and somebody who's good in the application and so on and so forth. And you actually consume your cycles until you triage the problem at the very least, sometimes beyond that. I think, I hear you talking in terms of what I would suggest calling preventive medicine rather than reactive medicine. You know, instead of when the crisis happens, uh, getting all those people on the bridge together, you put them into the fractal team from the very beginning as a preventive medicine kind of approach. Is that a reasonable reflection of your opinion?
Not Preventive Medicine, Just Healthy Living
21:25I'll, I'll take it a step farther and say that it's not preventive medicine so much as it's just good, healthy living. I would not agree more. And I will drink to that both in the data center and in our personal lives. Absolutely. So, so, I mean, that, that in itself is like kind of gets into the, uh, the sort of practices and technologies that, that we were going to discuss in the second half here. And I mean, it seems like one, one of them, you know, is essentially, it's, it's, it's, as always with these things, it's always like it boils down to a very straightforward, simple minded sounding thing. It's just like have the operations people involved in throughout the, this, the development process, just like you do QA and developers, obviously, and documentation folks and everyone else who's sort of contributing to and then becomes responsible for the end result. Absolutely. So, I'll just, I'll just start with, uh, kind of a list of practices that I think are, are sort of essential to, to what I see as agile infrastructure, agile operations. Um, the baseline, and this, this applies to, to development as much as anything, and everyone sort of takes it for granted, but it's not always applied in operations, is, is gotta be version control. So, in, in, in a lot of cases, you'll, you'll, you'll find systems and there'll be, you know, dot VAK files, um, sort of laying around, uh, and what you get with version control right off the back is the ability to roll back.
Version Control as a Shared Resource
22:50You have history, you have context, you have comments, and, and you also share that. That's a shared resource. You don't have to go inspect each individual machine. If you have a shared resource that you can, and we'll talk about some of these techniques, um, a little bit more. And I don't really care what your version control is, as long as it's Git. Just kidding. Um, um, SVN is great too. Uh, I use SVN for years, and, and it's, um, certainly better than CVS, which I also use, but, I, I think Git is, um, it just, it enables a lot more flexibility, uh, and once you've used it for a while, then, then you don't want to use anything else. So, I'm going to advocate Git. So, so what do, uh, like, like, like, what do you, what do you see, what do people put into version control? All your config files, as a starter, all your, all the, even if you're not going to use, uh, things like Puppet, even if you're not going to use these other tools that are emerging, just version your config files. Just version all the little scripts that you have to, to build things. If, if, if you're, if you're a sysadmin with any chops at all, you, you've got a directory, um, with some Perl scripts or whatever your favorite language is that do things that help you get your work done, right?
Versioning What You Would Otherwise Retype
24:07You're not, you're not, you're not, hopefully you're not typing everything from, uh, you know, first principles every single time you sit down. So, just versioning those is a huge step forward. Andrew, let me ask you the following. You know, uh, uh, in the software development, we check in code every so often, and we have got a whole bunch of tools, which are good at maintaining our code in, uh, what I would say, uh, usable, usable shape, in spite of the fact that there may be many branches and probably branches of branches. Basically, basically the metaphor that I think, uh, I really advocate is that with respect to the IP configuration, similar tools are of immense value. Is that so? Uh, I mean, the other thing that I think, I, I touched on this a minute ago, but I, I don't think I've brought it out too much in this podcast. But as you move to using abstractions, the APIs to bring up machines on, on, you know, vSphere, EC2, cobbler, whatever that gives you your, your base images, and then abstractions, um, configuration management layer.
Once It Is in Version Control, It Is Software Development
25:27And you start to use these sort of tools, then that process looks exactly like software development. It's just code. Uh, you, you can take advantage of all these other things, and, and what you just said applies to that as much as anything else. Because it's, it's, it's now software development. And which tools would you, uh, mention, uh, under this analogy? Namely, our software control tools are very well known, uh, I would think, to just about anyone who reads this blog and listens to the podcasts. At the same time, we are addressing here, uh, a topic which is an extension to Agile, namely, Agile in the system management context, or applying Agile to system management. What tools are the equivalent of the source control systems that we know and love in software development? They're not different. You use the same ones. Not only that, but this is another point, that you should have a shared repository. Particularly if you're working in a, a web shop, and it, it breaks down a little bit if you have to manage, you know, ten, ten different applications.
One Repository for Application and Configuration
26:44But if your business is, is a web app, if you're doing some sort of, you know, software as a service, then the, the operations, the config files, the abstraction layers for the config files, and the application, those should be in the same repository. So there's not a different set of tools. It's the same version control tools that everyone knows and loves. I hear you loud and clear. And thankfully, usually those don't cost very much, depending on, usually they're free. They just cost time to get up and running, which is always handy. Yeah, there's certainly, um, tools you can, you can buy, but I think that, like I said before, if you're not using Git right now, you should go at least try to use it. And there's a little bit of a, I think this is something that I run into a little bit. There's a curve of learning, and there's sort of namespace collision between Git and, and subversion, which makes it a little more confusing than it probably would have been if they didn't have commands with the same name that do different things. But once you, once you figure out how it works, it's free tool, it's Linus Torvalds, it really kicks butt. And once, once you understand Git, quote unquote, get it, you're not going to go back. So, so what are the, uh, you know, like, like, like we were saying, you're going to detail, go into tons of detail on all these things in, in your talk next week.
Always Ship Trunk
28:03But, uh, like, what are some of the other practices and technologies aside from version control? So, so, so just really quick, and I'm stealing most of these from, uh, the person I always talk about, um, in these, John Allspaugh, but if, and this is web app, off specific as well, but, um, always ship trunk. If you have a shared repository and you're always shipping trunk, everyone knows where to look when things go wrong. Um, they, they also do some interesting things at Flickr, um, branching instead of in the, in the version control, you, you put your branches in the actual, um, code. So you can turn features off and on. And that allows you to do interesting things like dark launches where you're going to, you're going to, you're going to test the impact of new features on the infrastructure by running the code, but not exposing it to the UI, not exposing it to the user. And that, that gives you some, um, ability to reason about how it's going to go when you, when you roll this thing out to, you know, quote unquote, uh, live. Now, now, is, is that, is that, is that different than like turning it on for only 20% of your users or is it what, because it sounds like you ship everything except the ability for the user to actually use it.
Twenty Percent of Users, Twenty Percent of the Impact
29:20Well, well, you, you, I think you might do it. Um, and this is something I would consider doing and it's going to depend on the feature of course, but if you, if you turn it on for 20% of your users and, and I've, I've been, uh, bit by this before, then that gives you 20% of the impact. Which might not be a great indicator of what it's going to do with all of your users. And, I mean, and, and this is just, this is just ways to get data and educate yourself because you can never, one thing that you can never account for in software is how users are going to do stuff. And, and so that, that can certainly cause you problems, but what you're trying to do is give yourself as much information as you can before you put the, you know, before you turn the lights on. And so, and so with, with the, with the dark, I mean, with doing this dark launch thing, like that sort of gets you, gives you a sense of what it's going to be like to like, what, what, what, what sort of problem does that address? Well, let's, let's say you want to introduce something that makes suggestions to the user based on their preferences or what they've already done. Some sort of context, um, specific suggestion. And, and that might be some, some computation on the backside and, you know, whatever that problem is, you know, some queries to the database, whatever.
Dark Launching Work Nobody Can See
30:36Um, if you, if you, if you want to turn that feature on in a dark way, you're going to actually run all those queries for everyone. You're just not going to give them the ability to see that happen. Um, and then, so, so basic, basically with the dark launch, you, you can do it with things that don't require user input that, that sort of like, there's all this processing that happens in the backend, so to speak at the non UI layer. And, and you're testing out that that will work and then the next, and then you're staging this release and the next thing is like, okay, now let's let the user not only see it, but like give it an input or something. You need to use your example, like fine tune the context and then we'll see if that part breaks. I mean, you could also, you should, you could also take it a step further where if, if you required some user input, then you could generate those behind the scenes, right? If it's going to post to some, some form or, or post to some, um, URL, then you, you could come up with what you, you know, and, and obviously you're, you're not, this goes back to what I just said, that you can't predict everything users are going to do, but you can maybe make some assumptions and see what those assumptions, what the impact of those assumptions are on the infrastructure. Andrew, I would like to clarify one point, uh, as I think it's of critical importance in the context that we are discussing.
Under the GUI, in Production
31:55The, the going under the GUI, so to speak, that you recommend and that you indicate that, uh, Flickr is using is in the production system. We are. Yes. Yeah. I think this is of critical importance because, you know, the old session philosophy is that I have an intermediate, uh, environment, which is pre-production. And part of the production environment, part of it is probably the configuration drift that you mentioned, but the other thing is that there is no way to replicate a full production environment, or at least no economically, possibly way. So the immense power of, uh, or, uh, what you are advocating is that this kind of quote unquote testing of features or turning, uh, on and turning off features is actually done in the production system, not in any one of the systems that precedes. Well, you, you would, you would certainly test everything. So you're going to go through, and this is a problem that I, I've also been bit by in my career, is you have this difference, the configuration drift between, um, the test and production system.
Why It Passed in Test and Failed Live
33:14So you test a feature in test, and because of either a configuration file or the state of the database, there's something about that test succeeding in, in the test environment. And when you put it live, then, then there's some assumption about how things are that you fail to account for, and it ends up being a long, painful night. Um, what, what you can do with, with this, and this is sort of the, the next stage. So once you have everything in version control and you're able to, um, build things with these abstractions is you're going to rebuild from source. So everyone talks about rebuilding from source with their application. And, and in some level that, that doesn't make sense with interpreted languages, but what you want to be able to do is from the version control, which has a shared repository for your operations and for your application, go from bare metal to a fully running service. In one step, a fully automated, build from source step, right? What do I do about stored procedures, uh, within this modus? Well, I mean, stored procedures are on the database and you should be able to rebuild your database from scratch. So it applies not, quote, unquote, only to the application or only to the infrastructure.
Every Component, Including the Database
34:32It fully applies to every component in, uh, the overall setting, namely the database, uh, and any others, right? I, I would say that, but I'd also say, um, there's a caveat and it depends on, um, how you want to think about the world and how you want to build things. But I would probably try to steer people away from stored procedures in a lot of cases. Um, I, I, I wrote plenty of PL sequel before, and I know there's, there's a lot of power and flexibility there, but I think that there is a tendency because of the way those systems work, um, to not have that stuff be in version control, and that involves, like, that's a whole nother subsystem, right? So right now we're talking about, um, systems and ops and, and bringing those people into this process. Database administrators, um, I think that that's another, another little, um, sparkle to add to the fractal that, that most people don't, uh, do a very good job of accounting for in an agile process. So, uh, if that, if that, um, resonates with your experience, uh, I'd love to hear it. Very, very much so, and I don't want to speak to Coté, but I believe that it resonates for, uh, his experience as well.
Stored Procedures, and Why Nobody Likes DBAs
35:48Yeah, no, I, I mean, I think, uh, there's a reason most people don't like DBAs, and that's that they're very, uh, they fight tooth and nail to maintain complete black blocks control over their domain. And, you know, applying a lot of, you know, a stored procedure seems more often than not, at least, uh, I don't know, for lack of a better word, sort of politically and management-wise, like, a good, a good weapon to use to sort of build up your tower, your defenses against, you know, your job being optimized away from you. To give the cynical view of it. You know, and, and, and, Ed, what were you saying, Israel? I think I need to go to the various consulting assignments that I had over the past year or so, and look carefully at each one of them in view of the perspective that you offer, namely, how much of it was to preserve a job? Yeah, exactly. Yeah, that, that's, uh, that's one of the, that's probably one of the problems that, uh, that we might have mentioned in a past episode that we didn't mention at the beginning of this is, is people, uh, just wanting to maintain the status quo of their, their day-to-day work life. Right. Which is always, uh, always kind of difficult. So, you know, what, what are, uh, as, as you were talking about, I was thinking, you know, the, it seemed, taking it to, like, silly extremes, it seems like at some point you kind of get to the, uh, at least the idea of, I don't know what you would call it, but kind of analogous to having continuous builds, to having continuous production deploys, where you're sort of just gratuitously doing a deploy, a deploy to see if it works or not.
Building the Whole System From Bare Metal
37:16But, uh, you know, I don't know if that's necessarily a good idea, but having that ability, I think is more, you know, is the interesting thing to, it gets to the idea. It gets to the idea of how, how you build your system, including all the infrastructure that basically you could build it from bare metal from scratch at any one point. And I actually, as much of that as possible. I think it's a great idea. And, and what you really want to do is, what I consider, you want to reduce the fixed cost of deployment. Right. And you want, you want to reduce it to as close as zero as you can get. Right. And, and in general, that's in, and, you know, there's, there's a spectrum, obviously, but in, in organizations that consider themselves quite agile, there's often this waterfall-like step to put whatever they did onto the servers. So, uh, so, so, so are there some more, uh, some more practices and technologies you can tell us about? So we were talking about reducing... Come on, give him a chance to speak about it in the conference. That's right. Well, you got to give us a few more, just, just a couple maybe. And then, is there going to be a recording of your talk or anything or like how, uh... Well, we got email from InfoQ that they want to, uh, record, uh, myself and Paul Nassrat, who's the, he's the, he's the Ops lead at The Guardian.
Test-Driven Infrastructure
38:37Yeah. And we're, we're co-presenting some of these ideas, uh, next Thursday at Agile 2009. Well, in that case, I mean, since there is going to be a record of it, why don't you just, like, really quickly run through what the other ones are, if only by name, and then they'll have the details at the top. Uh, so, there's some notions of, of a test-driven infrastructure. I, I'm kind of on the fence of what that actually looks like and what that means, but it's certainly an interesting idea to play with. Um, I talk about deploying early and often. Um, you're trying to get rid of, uh, this. So, so, as I mentioned a minute ago, you, you have this sort of waterfall-like artifact. That is the deploy process. And if you can move to a more incremental, you know, kind of deploy early and often, uh, mindset, then when you do have issues, it's very easy to reason about them because you have small chain sets that you've just introduced. And you can, you know, you don't spend, you know, the hours kind of backtracking to figure out what, what you've just changed. Right. Um, there's, there's, there's this notion of, of being able to tag everything with, uh, who, what, when. And that's sort of tying your version control into your deployments and being able to correlate the, those tags with, with the metrics. Ah, that's, that's interesting.
A CMDB That Enables Change Instead of Impeding It
39:52That gets you into, like, CMDB territory in theory and things like that. In, in theory, uh, I think it's, it's probably a little more pragmatic, but, and, and you're doing it to enable change where I feel like a lot of the, the ITIL is sort of in that mindset where you're, you're impeding change. Um, but being able to, to see, you know, who, what, and when in relation to, to the metrics and, and who, what, and when is, is, is, you know, the, the requests to your application and looking at all the metrics from, you know, from the disk IO to the CPUs and seeing that on, on one timeline correlated, uh, gives you the ability to, to go into the code and the change sets that you made in both your application and in your infrastructure. Yeah, yeah.
Synchronize Every Clock, and Use UTC
41:09Um, and who, what, and when. Um, this is, this is a very simple thing, but it's, it's been me in the past, and I'm pretty sure it bites a lot of organizations on a regular basis. Uh, synchronize the time on all of your servers and all of your systems. Um, I, I would probably advocate putting everything on UTC, but you, I've seen, and, you know, I've had personal experience where you have one data center or another data center, or these machines got provision some way. And so there's different time zones, and they're on different, uh, time scales, and, and then, you know, you, you always know that in the back of your mind, but then there's that one time when you, you, you know, you see data from one and the other, and you don't make that, that mathematical, uh, calculation right, and it causes all sorts of, uh, confusion that doesn't need to be there. You have to, uh, you have to, uh, be very good about being a little time zones in your head, and you love all that crap, which is a, you're just a side effect. So there's, what about, what about, what about the ecosystem, which includes the partners, namely, within one corporation? It might be difficult, but it's relatively easy to have a policy, uh, of, uh, you know, UTC, that's, uh, our standard.
Synchronizing Across Data Centers and the Cloud
42:26However, in terms of, uh, one data center interacting with another data center, interacting with, um, uh, websites, uh, you know, in the cloud, et cetera, et cetera, et cetera. So, uh, what would be your good recommendation in terms of this kind of synchronization in the lab instead of synchronization within a corporation? Well, that, that's something you're going to have to basically deal with, and this kind of goes back to the, to the beginning where some of this is hard, and some of this is complicated. And, you know, you can, uh, you can write things that, that translate that. If you, if you're going to measure metrics on other people's services, then you, when you make those measurements, you don't necessarily need to ask them for the time. The servers that you're making requests from have their timestamps. So, I think there's ways to, to frame it from, uh, you know, organizationally centric perspective and get those timestamps using your own systems that, that kind of makes that work. Andrew, you remind me of Donald Knuth's classic three big software in the cloud. That's, that's, that's what, um, I started to figure out recently.
Rebuilding as a Habit Gives You Scaling and Testing
43:42I know the feeling. Um, so, when you get to this point where you can build from source, uh, it gives you a lot of flexibility. You're going to migrate everything through the same, same sort of test dev prod process for all your systems. But it's also got, um, the ability to rebuild or to scale. So, once you're in the, the habit of rebuilding your systems, you, it gives you a lot of advantages for testing too. Because, say, say you're, you're doing unit testing. There's often some, some state that you want to, you want to put the system in so that you can test that assumption. Now, in, in my experience, I've seen many times where people have been bitten because of some assumption about the state in the test environment. Now, if that test environment is something that you rebuild and throw away every day, then when you rebuild it, it is in a known state. And that, that gives you power to reason, uh, about what state it's in, what state you want to put it in. And, and that sort of flexibility has, has benefits that kind of cascade across the process, in my opinion. Oh, right, right. That makes sense. And, and, and because you have all this practice, rebuilding systems, there's also disaster recovery, um, implications.
Continuous Training for When Things Go Wrong
44:58Yeah, yeah. And you can do all, you, you, you just are, you're sort of in, to use another, like, metaphor, you're sort of in continuous training, if you will, to sort of, by doing things continually like this, you're always in training to handle when things go wrong. Just because it's sort of like, those are the things, you're doing those practices more often when it comes to just rebuilding systems or troubleshooting them or whatever. And, and, and once you get it to the point where everything's stable and you can rebuild it and, and no one knows who you are anymore, then you can start doing fire drills just for fun. That's right. I don't know about that. Why not? Well, as long as it's a drill, it's often relaxing because it's not stressful. So you just kind of hang out, as I remember. Everyone put, you put on the red baseball cap and leave the building and then, like, go get a soda afterwards. Well, let's say, theoretically, go ahead. Guys, may I suggest that we watch Dr. Strangelove before we go farther into the fire drills? That's right. It's a cautionary tale for applying these, these ideas too much. We don't want to be living in a cave. Oh, the don't stay machine. That's right. Well, I, I think, I, I mean, I, I think we should, we should wrap it up so we don't make our listeners sit in here and listen to too much of it.
Where the Presentation Will Go Deeper
46:17But, but I, I think, I think we kind of got a good sense for what some of these practices are. And once that recording gets put up somewhere, we'll have to put a link to it. And then that'll be sort of like the more, and along with the presentation, hopefully, and that'll be a more in-depth look into it. So in the presentation, we'll, we'll probably go more in-depth on these technical things. We have a, a bit of discussion about, um, use, taking legacy systems, existing, um, systems and practices and trying to migrate towards, um, you know, using these sort of techniques and technologies. And then we also have, uh, a bit of information about, uh, process and culture, um, in addition to, and there's some crossover, obviously, between, uh, the, the technology and the process. But that's, uh, that's what we're shooting for next week. Yeah, it sounds like it'll be exciting. And both of you guys will be at, at, uh, Agile 2009, right? Absolutely. And I'm hoping that my own presentation does not collide with Andrew. Mine is Thursday, 11 o'clock. When is yours, Andrew? Um, I, it's, it's Thursday. Uh, 2.30, I think. It's in the afternoon. I'm not even positive. It's, it's, it's, it's after 11, though. We are safe, sir. I would be, uh, all anxious to attend your presentation and to learn, uh, from you and from the co-presenter. Well, I've already learned plenty from you, but I'm, uh, looking forward to your presentation as well. Delighted. Thank you. Well, well, on that, uh, that happy note, I guess we might as well wrap it up unless there's anything else you guys wanted to mention. We, we, we can sing kumbaya real quick. I don't remember the words. Oh, fantastic. Well, uh, so, that, that sounds great. Well, uh, thanks everyone for listening and we'll see you next episode. Perfect. Thanks. Oops.
Machine transcription, lightly formatted. Chapter headings were written from the transcript; speakers are not identified, so nothing here is attributed.