1 00:00:00,740 --> 00:00:06,920 Well, hello, everybody. It's the 21st of August, 2009, and this is a third part in a little 2 00:00:06,920 --> 00:00:12,580 mini-series we have going here at the Agile Executive Podcast, or within, I should say. 3 00:00:13,060 --> 00:00:18,060 And as always, this is Michael Coté, one of the co-hosts available at peopleoverprocess.com, 4 00:00:18,100 --> 00:00:20,000 and I'm joined by one of the other co-hosts. 5 00:00:22,080 --> 00:00:29,280 This is Israel Gath. I'm partnering with Coté on the Agile Executive, and I am really 6 00:00:29,280 --> 00:00:35,880 very, very excited about the whole mini-series, and in particular, the piece on Agile infrastructure 7 00:00:35,880 --> 00:00:42,160 that we are going to get into today. And with that, let me hand it over to the third participant. 8 00:00:43,080 --> 00:00:49,200 This is Andrew Schaefer. I'm co-founder of Reductive Labs. We make Puppet. We're trying 9 00:00:49,200 --> 00:00:55,020 to help people build better IT systems with better tools, and we get to see a lot of different 10 00:00:55,020 --> 00:00:58,760 approaches and a lot of different infrastructures, and that's led to some of these insights. 11 00:00:58,760 --> 00:01:03,840 And as people may recall from the last two episodes, the three of us were just talking 12 00:01:03,840 --> 00:01:08,920 about the idea of what Agile infrastructure, Agile operations is, and sort of the motivations 13 00:01:08,920 --> 00:01:13,600 and the benefits and kind of what it looks like, if you will. And last episode, we talked 14 00:01:13,600 --> 00:01:19,360 a lot about just the idea of broadly how you can introduce this change and problems in the 15 00:01:19,360 --> 00:01:26,820 IT department of introducing changes. And next week, Andrew has a presentation that's going 16 00:01:26,820 --> 00:01:31,120 to kind of sum up some of the things that we talked about and other stuff, but also get 17 00:01:31,120 --> 00:01:37,500 into the details of sort of practices and technologies you might actually use to do this sort of Agile 18 00:01:37,500 --> 00:01:42,700 infrastructure stuff. And so it seemed like a nice opportunity to sort of narrow down and 19 00:01:42,700 --> 00:01:48,780 sort of conclude, you know, kind of the niftiness of what we've been talking about with Agile 20 00:01:48,780 --> 00:01:52,740 infrastructure. And I think to that end, like the first thing we should start talking about 21 00:01:52,740 --> 00:01:58,600 is, I mean, we've kind of danced around this in the last few episodes, but we've more talked 22 00:01:58,600 --> 00:02:02,180 in the sort of the happy path positive way about what Agile infrastructure would be, but 23 00:02:02,180 --> 00:02:06,380 we haven't spent a lot of time talking about sort of like what the opposite of it are, sort 24 00:02:06,380 --> 00:02:10,360 of kind of what the existing problems are that it kind of seeks to address or fix, if 25 00:02:10,360 --> 00:02:15,680 you will. And so, I mean, with that, like when you're kind of looking at applying these 26 00:02:15,680 --> 00:02:19,620 practices and coming up with these things, Andrew, like what, what are sort of the issues 27 00:02:19,620 --> 00:02:21,420 in IT that, that, that you're fixing? 28 00:02:23,240 --> 00:02:29,700 So the, the project I work on specifically, Puppet, I think that that's the thing that addresses 29 00:02:29,700 --> 00:02:38,720 most effectively is this configuration drift, which is essentially the differences between 30 00:02:38,720 --> 00:02:45,780 machines that are supposedly configured identically. In, in most organizations, if you have two 31 00:02:45,780 --> 00:02:51,920 web servers or two database servers or two whatever servers, the chance that they are configured 32 00:02:51,920 --> 00:02:59,100 identically is very small. And as you get to 10 web servers or hundreds of web servers, 33 00:02:59,100 --> 00:03:03,840 then that chance goes to zero quite quickly. Right. 34 00:03:04,260 --> 00:03:12,940 And these, these differences, the, the drift between those configurations cause confusion, 35 00:03:13,320 --> 00:03:22,240 they cause mistakes and it leads to a lot of downtime. It costs a lot of time and, and the 36 00:03:22,240 --> 00:03:31,260 change man, change management practices that most organizations adopt on almost every level are 37 00:03:31,260 --> 00:03:37,700 to impede change. And we touched on this before, but the business, the, the business is enabled 38 00:03:37,700 --> 00:03:43,900 by change. So changes, it's, it's got to happen if the business is to move forward, but at the 39 00:03:43,900 --> 00:03:50,200 same time, it's putting up all sorts of roadblocks in front of itself to prevent itself from the pain 40 00:03:50,200 --> 00:03:55,360 that, that, that the change in the past has brought. And what we're trying to do is move 41 00:03:55,360 --> 00:04:02,560 things in with using better tools and better processes towards what I consider, you know, 42 00:04:02,960 --> 00:04:11,440 agile infrastructure utopia. And agile is a word that I, I, I've been practicing agile techniques for 43 00:04:11,440 --> 00:04:18,340 five years. I've been at the Salt Lake City agile round table, but I've also got a lot of love for a lot 44 00:04:18,340 --> 00:04:25,080 of other ideas. And I, I don't particularly like labels. Um, agile is a can convenient label on some 45 00:04:25,080 --> 00:04:30,120 level, but I don't want to, I don't want people to think like, Oh, I can't do this cause I'm lean. 46 00:04:31,120 --> 00:04:31,920 That's right. 47 00:04:32,700 --> 00:04:34,340 Cause that's, that's silly. 48 00:04:35,500 --> 00:04:42,500 Andrew, may I ask for your opinion about the following perspective, which I believe is completely 49 00:04:42,500 --> 00:04:55,040 consistent and complimentary to your view. The way I, uh, perceive agile to be, it's really about a set of 50 00:04:55,040 --> 00:05:03,480 principles which are universal in nature. For whatever reasons, those principles have effectively been 51 00:05:03,480 --> 00:05:12,220 applied first in the software development. Within software development, we are actually talking often 52 00:05:12,220 --> 00:05:19,420 or Jim Highsmith to be more precise is talking often about scaling up. Namely, agile is applied to a 53 00:05:19,420 --> 00:05:29,180 larger team, a team of 500 instead of 50 and scaling out, not a team in one site, but a team in three 54 00:05:29,180 --> 00:05:38,500 different sites. I often speak about scaling downstream. And what I mean by that scaling to the other 55 00:05:38,500 --> 00:05:47,000 functions in the business, be it marketing, sales, professional services, whatever. The view that I have 56 00:05:47,000 --> 00:05:55,760 of the agile infrastructure is that it's the application of the agile principles in the domain of 57 00:05:55,760 --> 00:06:09,080 IT operations. And this is another scaling or expansion. If you please do agile for me. And it's so very tempting for the 58 00:06:09,080 --> 00:06:16,080 reason that in so many data centers, you know, there is one arm that develops software. There is another arm that 59 00:06:16,080 --> 00:06:19,080 runs, maintains, operates a software. 60 00:06:19,080 --> 00:06:28,400 Absolutely. And the consistency in the principle has tremendous potential in terms of bringing those two 61 00:06:28,400 --> 00:06:28,400 together. 62 00:06:28,400 --> 00:06:36,400 Absolutely. Let me, let me expand on that point. So I think it's, it's, that's half the 63 00:06:36,400 --> 00:06:44,400 story. On some level, it's being able to take some of the lessons that we've learned from the agile principles being applied to 64 00:06:44,400 --> 00:07:13,400 the software and applying those two to operations and, and that infrastructure level. But I also think that there's, there's some other things that are happening that make it a little more interesting in the technology. So, so not only can you take these, these lessons and apply them to the operations, but there's, there's things that are changing. And there's information and interesting things coming from the other direction. So on one level, it allows you to, to apply some principles. Yes. But on the other level, 65 00:07:13,400 --> 00:07:42,400 on the other level, the things that are happening with, with abstractions from the network layer, the storage layer, and the competition, computational layer, and with the abstractions that, that something like the puppets configuration management give you, and then orchestration, things like control tier Capistrano, all these things are code. So you can start to think about building infrastructure, like it's software. So, so I think it's going in both directions. So I don't want people who are, are sys admins. 66 00:07:42,400 --> 00:07:58,400 Are sys admins and ops teams to, to sort of lose their roots? Because there's a lot of knowledge, there's a lot of domain knowledge that I don't want people to, to lose touch up touch with. I know as a developer, I'm having a kind of a develop, developer centric view of the world when it comes to technology. 67 00:07:58,400 --> 00:08:16,140 And that's not always optimal for managing systems or for security. And these people that have that deep domain expertise are required to balance out my sort of shiny new thing, fly by the sea, my pants, developer attitude, if that makes any sense. 68 00:08:16,400 --> 00:08:19,400 Are we really talking about fusion then? 69 00:08:19,400 --> 00:08:41,400 I think there's a, there's a fusion, but then there's also a recognition of, of worldview. And I think if you, sometimes when you try to, to assimilate cultures, you lose the, the unique benefit of that particular worldview. So I'm a little hesitant to say that it's a total assimilation. 70 00:08:41,400 --> 00:08:42,400 Is that me? 71 00:08:42,400 --> 00:08:56,400 Yeah, it's extremely good point. In a way it is about the combination of staying close to my roots while I fuse with other roots. 72 00:08:56,400 --> 00:09:18,220 So it's about, I think there's two sides to it. So you don't want to forget who you are, but at the same time, you don't want to limit who you are. So you, you are, you know, things, you are things, you, you, you have all this knowledge and domain expertise, but at the same time there's stuff happening around you and, and it's foolish to pretend like it's not there and not learn from it. 73 00:09:18,220 --> 00:09:40,220 I think that's sort of the, the, the principle of learning that, that, that I think if you apply that to every role in, in anything, whether it's agile or not, um, from, from business to development, to testing, to product management, to marketing, as long as you are in that culture of learning, then that's going to be an advantage for yourself and for your organization. 74 00:09:40,220 --> 00:10:10,200 So it's sort of like, it's sort of like having a, uh, a lack of continuous improvement essentially, which is, is, uh, it's, it's a core problem that causes all sorts of, all sorts of issues, but like, like, you know, I, I mean, I'm curious drilling down on, on, on, on the, the sort of issues, you know, the, the problems, if you will, more like the, uh, I remember at open source world, we had, we were, I was on a panel discussing, um, just, uh, you know, open source IT management stuff. 75 00:10:10,220 --> 00:10:17,860 And, and, and the question came up that basically boiled down to, uh, when is, when, when are, is all this IT management stuff going to work together better? 76 00:10:17,860 --> 00:10:26,380 And the, you know, the best answer that we had was sort of like, well, it's all just complex stuff. So what are you going to do? It's always going to be difficult, which is kind of a frustrating answer to give to someone. 77 00:10:26,380 --> 00:10:38,540 But, but, you know, to applying that question kind of here, like, like you're saying, and it's very true, like the, the role of change management is, is, is usually to limit change as much as possible. 78 00:10:38,540 --> 00:10:45,700 And, and I wonder, like, what, what's up with IT being so brittle that it's so prone to screwing up when you change something? 79 00:10:45,900 --> 00:10:50,380 Like, how, how do you think it's come about that it's so, it's so complex? 80 00:10:51,020 --> 00:10:53,060 Well, let's just talk about that for one sec. 81 00:10:53,180 --> 00:10:55,600 That, that panel was with my, uh, Conrad at arm. 82 00:10:55,700 --> 00:11:01,500 So I'm going to give a shout out to, uh, Luke Kinnis, who's, uh, out there somewhere listening, perhaps. 83 00:11:01,940 --> 00:11:06,060 The, the fact is that's not just unique to IT management, right? 84 00:11:06,060 --> 00:11:10,660 There's all sorts of things that you want to talk about things, integrating, working together. 85 00:11:10,660 --> 00:11:14,500 How many applications out of the box just kind of work together, right? 86 00:11:14,820 --> 00:11:15,040 Yep. 87 00:11:15,180 --> 00:11:21,100 And, and as engineers, I think we have a tendency to, to love cleverness and complexity. 88 00:11:22,380 --> 00:11:36,420 And that, that, that love affair with cleverness and complexity, while it's, um, entertaining to me personally, it doesn't always, it doesn't always give you, uh, the flexibility that you think you're, you're giving yourself. 89 00:11:36,420 --> 00:11:41,260 And, and sometimes you're so clever that no one else can figure out what you did. 90 00:11:41,700 --> 00:11:55,680 And if you, if you created some IT management script or, or framework or what have you in the programming language, and then you got a new job and you left, the next guy comes in and he's unraveling what you did. 91 00:11:55,680 --> 00:12:00,140 And, and, but he's still being forced to, to do things because, well, you know, the business needs to change. 92 00:12:00,280 --> 00:12:03,560 Then, then there's some inherent, um, information missing, right? 93 00:12:03,580 --> 00:12:08,680 So, so sometimes you fill in the blanks and that, that leads to assumptions and, and mistakes as well. 94 00:12:09,180 --> 00:12:09,800 Right, right. 95 00:12:09,880 --> 00:12:10,040 Yeah. 96 00:12:10,100 --> 00:12:18,400 I, I, I guess we can't all be like as artful and cleverness as like, you know, like I think people, when they're trying to be clever in programming are trying, aiming to be sort of like Lewis Carroll, right? 97 00:12:18,420 --> 00:12:24,120 Where you've got a, like, you know, a book and writing full of like, you know, little puzzles and riddles and all sorts of things. 98 00:12:24,120 --> 00:12:27,720 But, uh, you can't really come off that well polished and entertaining in the end. 99 00:12:27,720 --> 00:12:38,700 I mean, that's, that's a, a complex, clever book, like, you know, the Alice in Wonderland series, but, uh, it's actually difficult to achieve cleverness that's enjoyable and entertaining to consume, if you will, to be, to be mad. 100 00:12:40,100 --> 00:12:49,000 So I, I might be dating myself, but there, there's, I think it's Superman two with Richard Pryor and he's, he's trying to, he's trying to figure out what kryptonite's made of. 101 00:12:49,500 --> 00:12:57,780 And, and he gets to where he's done an analysis on this, this compound and he knows what everything is except for this, this small fraction of a percent. 102 00:12:58,080 --> 00:13:00,600 And, and, and, you know, his techniques can't figure it out. 103 00:13:00,700 --> 00:13:03,820 And he, he looks at his, uh, pack of cigarettes and he sees tar. 104 00:13:04,000 --> 00:13:05,380 So he just fills it in with tar. 105 00:13:05,380 --> 00:13:09,280 And I think that's how, that's how a lot of, uh, problems get solved. 106 00:13:09,420 --> 00:13:12,700 When, when we get to the end, we're just, uh, well, this might work. 107 00:13:13,320 --> 00:13:13,600 That's right. 108 00:13:14,100 --> 00:13:19,420 So are, are, are there any other, like, problems to kind of lay the foundation of, of what, kind of what we're fixing here? 109 00:13:19,800 --> 00:13:25,160 So there, there's the configuration drift problem, which is, um, a big problem in systems management. 110 00:13:25,480 --> 00:13:31,880 There's also, when you're talking about organizational problems, there tends to be a, a wall of confusion between dev and, and operations. 111 00:13:31,880 --> 00:13:35,760 And if you listen to me talk all the time, you're probably sick of hearing me say that. 112 00:13:36,160 --> 00:13:37,860 Um, but, but it's certainly a problem. 113 00:13:37,860 --> 00:13:43,760 And there, and there's really, we talked about this last time, but there's kind of a wall of confusion between every business unit in, in a lot of cases. 114 00:13:44,360 --> 00:13:53,740 Um, but the, but the one between dev and ops is, is particularly, um, antagonistic sometimes because of the, the fact that someone threw something over the wall. 115 00:13:53,840 --> 00:13:58,840 And I got, you know, a pager call at two in the morning and didn't make me happy. 116 00:13:59,100 --> 00:13:59,720 That's right. 117 00:14:00,080 --> 00:14:02,600 It's sort of like a lack of controlling your own destiny. 118 00:14:03,340 --> 00:14:03,600 Yep. 119 00:14:04,380 --> 00:14:07,700 And, and then I think this is also a problem in, in system management. 120 00:14:07,700 --> 00:14:11,420 It's a problem in, in development as well, depending on, on the culture. 121 00:14:11,880 --> 00:14:19,000 But I think agile does a reasonably good job of balancing this out with, with a concept of, of shared metaphors and teamwork. 122 00:14:19,000 --> 00:14:24,460 But in, in a lot of organizations, the, the sys admins have a very strong hero culture. 123 00:14:25,280 --> 00:14:34,260 So you, you have incentives on some level to, to let, um, and I think it's deeply rooted and it's almost psychological. 124 00:14:34,260 --> 00:14:43,500 And some people aren't even necessarily aware that they're doing this, but, but when, when there's a fire and someone comes in and puts it out, then they're the hero. 125 00:14:43,940 --> 00:14:44,380 Right. 126 00:14:44,600 --> 00:14:47,080 And so you, you kind of create this, this hero cycle. 127 00:14:48,520 --> 00:14:48,920 Right. 128 00:14:48,920 --> 00:14:55,300 And it, and it doesn't necessarily, I mean, I'm a big believer in trying to figure out how you can align incentives with behavior. 129 00:14:55,500 --> 00:15:00,920 And if, if, if your incentive is to be a hero, then you need, you need problems so you can be heroic. 130 00:15:01,260 --> 00:15:01,660 Yeah. 131 00:15:01,720 --> 00:15:01,860 Yeah. 132 00:15:01,900 --> 00:15:17,300 Well, you often get, uh, in all walks of life, you're rewarded, you're rewarded for, uh, solving a crisis as it comes up a lot more than for preventing a crisis or just, you know, uh, taking care of, you know, people don't get rewarded very well for just filling in potholes. 133 00:15:17,300 --> 00:15:22,220 They get rewarded when they like help, help, like shut down a riot in the city or something like that. 134 00:15:22,660 --> 00:15:22,800 Right. 135 00:15:22,880 --> 00:15:31,340 And I think it's particularly bad in ops teams because on some level, and it's going to be totally cultural and it has to do with how they interact with the rest of the business units. 136 00:15:31,500 --> 00:15:36,840 But you have, if everything's going well, then no one knows you exist. 137 00:15:37,620 --> 00:15:38,060 Right. 138 00:15:38,060 --> 00:15:47,100 You know, this reminds me of some of the writings of Rob Austin of Harvard University. 139 00:15:48,340 --> 00:16:04,360 His basic thesis is that you have to change your measurement or incentive system every so often because one puts in place a measurement system in order to induce a certain behavior. 140 00:16:05,060 --> 00:16:20,980 Within some period of time, people learn how to game it, how to satisfy the incentive systems instead of how to satisfy the thing that it's aimed at optimizing. 141 00:16:21,340 --> 00:16:44,480 And the only way that you have to go around this cycle is to be diligent and change your, uh, performance measurement system at some frequency so that people stick with trying to do what you are trying to accomplish rather than, rather than satisfy the performance, the performance system. 142 00:16:45,380 --> 00:16:45,680 Right. 143 00:16:46,020 --> 00:16:46,660 Definitely. 144 00:16:46,660 --> 00:16:52,080 So there's one more problem I'll just go over. 145 00:16:52,200 --> 00:16:58,000 And I think there's a problem in dev too, but I, I really, I dislike this notion of non-functional requirements. 146 00:16:58,200 --> 00:17:11,660 We've been over this before a little bit, but I think in, in a lot of agile implementations, uh, there are situations where, you know, you go to the product manager or whatever, you know, whoever's sort of making some of these decisions about how things get prioritized. 147 00:17:11,660 --> 00:17:19,080 And you explain to them, if we don't, if we don't, if we don't do this thing or upgrade this thing, then it's going to cause all these problems for us. 148 00:17:19,080 --> 00:17:26,080 And it's going to spend, you know, we're going to spend a lot of time fighting fires that we wouldn't have to, if we can just fix this thing or if we can just invest in this infrastructure. 149 00:17:26,080 --> 00:17:42,920 And that's, since that's a quote unquote non-functional requirement, it, it often gets, uh, deprioritized and then just sort of sits in the backlog and the, in the churn of the, the backwash of the backlog, if you will, for, for, for, you know, for time and all eternity. 150 00:17:42,920 --> 00:17:51,640 And, and, and I think that's a, that's a cultural problem that, well, in some cases it's out of, out of the hands of, of the dev and ops team. 151 00:17:51,780 --> 00:17:57,620 I think that needs to be recognized at a higher level up the decision-making, uh, chain. 152 00:17:57,720 --> 00:18:04,660 And you can solve a lot of problems and save yourself a lot of money if you invest a little bit in some of these non-functional requirements. 153 00:18:05,380 --> 00:18:08,800 Andrew, let me ask you the following question. 154 00:18:09,280 --> 00:18:11,980 Uh, it's basically the thought that you triggered in my head. 155 00:18:12,580 --> 00:18:21,160 What we oftentimes are doing agile is we break the current organization into a team-based structure. 156 00:18:21,160 --> 00:18:28,440 So you have the scrum team and this is the structural unit, uh, the structural organizational unit in whatever we do. 157 00:18:29,080 --> 00:18:35,760 When I listen to you, I'm inclined to think that, and tell me whether I'm putting words in your mouth, 158 00:18:35,760 --> 00:18:46,520 is that you are thinking in terms of the scrum team or something like the scrum team being expended to include IT. 159 00:18:47,060 --> 00:18:54,460 Namely, here we have the product owner and here we have the developers and the testers and whoever else. 160 00:18:55,520 --> 00:19:08,200 And you are sort of suggesting that the, uh, fractal scrum team would be with members of the IT, uh, data center. 161 00:19:08,200 --> 00:19:24,260 And furthermore, that those are ongoing participants because of the context in which they operate, uh, by, uh, its own nature requires different solutions from the R&D side. 162 00:19:25,120 --> 00:19:26,500 Is that where you are heading? 163 00:19:27,040 --> 00:19:30,720 I'm a, I'm a big fan of this concept of, of fractal teams. 164 00:19:30,720 --> 00:19:37,360 And what you just said, I think that some of what I talk about is a little, uh, web ops specific. 165 00:19:38,020 --> 00:19:44,140 Um, but in general, if you're building a web application, your infrastructure is that application. 166 00:19:44,700 --> 00:19:46,260 Your application is that infrastructure. 167 00:19:46,500 --> 00:19:54,960 If those machines aren't kept up, if you don't have someone with the, the skills to, you know, the, the system husbandry, system administration, 168 00:19:54,960 --> 00:19:58,800 innovation skills to keep those machines up, you're not delivering business value. 169 00:19:58,800 --> 00:20:00,320 You're down, you're done. 170 00:20:00,840 --> 00:20:01,000 Right? 171 00:20:01,100 --> 00:20:10,480 So if you can take advantage of that domain expertise through the cycle of planning, just like you would any other, uh, domain expert on your team, 172 00:20:10,480 --> 00:20:12,840 then you're going to be better off in the long run. 173 00:20:12,840 --> 00:20:22,280 And I, and I know, and I've seen problems melt away when you get, when you get dev and ops pairing to solve a problem that crosses that domain boundary. 174 00:20:23,440 --> 00:20:41,100 Well, you know, the experience that I had, having been in system management, the better part of my life, the experience, somewhat traumatic experience, uh, that I had is that you need so many people on the bridge when something, something breaks. 175 00:20:41,280 --> 00:20:49,180 So you get somebody who's good in the database and somebody who's good in the networking and somebody who's good in the application and so on and so forth. 176 00:20:49,920 --> 00:20:56,520 And you actually consume your cycles until you triage the problem at the very least, sometimes beyond that. 177 00:20:57,240 --> 00:21:05,100 I think, I hear you talking in terms of what I would suggest calling preventive medicine rather than reactive medicine. 178 00:21:05,800 --> 00:21:22,020 You know, instead of when the crisis happens, uh, getting all those people on the bridge together, you put them into the fractal team from the very beginning as a preventive medicine kind of approach. 179 00:21:22,380 --> 00:21:25,120 Is that a reasonable reflection of your opinion? 180 00:21:25,840 --> 00:21:31,600 I'll, I'll take it a step farther and say that it's not preventive medicine so much as it's just good, healthy living. 181 00:21:32,680 --> 00:21:34,000 I would not agree more. 182 00:21:34,180 --> 00:21:38,240 And I will drink to that both in the data center and in our personal lives. 183 00:21:38,660 --> 00:21:39,100 Absolutely. 184 00:21:39,960 --> 00:21:47,800 So, so, I mean, that, that in itself is like kind of gets into the, uh, the sort of practices and technologies that, that we were going to discuss in the second half here. 185 00:21:47,800 --> 00:21:57,400 And I mean, it seems like one, one of them, you know, is essentially, it's, it's, it's, as always with these things, it's always like it boils down to a very straightforward, simple minded sounding thing. 186 00:21:57,400 --> 00:22:11,080 It's just like have the operations people involved in throughout the, this, the development process, just like you do QA and developers, obviously, and documentation folks and everyone else who's sort of contributing to and then becomes responsible for the end result. 187 00:22:11,540 --> 00:22:11,980 Absolutely. 188 00:22:13,560 --> 00:22:23,080 So, I'll just, I'll just start with, uh, kind of a list of practices that I think are, are sort of essential to, to what I see as agile infrastructure, agile operations. 189 00:22:23,480 --> 00:22:34,320 Um, the baseline, and this, this applies to, to development as much as anything, and everyone sort of takes it for granted, but it's not always applied in operations, is, is gotta be version control. 190 00:22:35,080 --> 00:22:50,040 So, in, in, in a lot of cases, you'll, you'll, you'll find systems and there'll be, you know, dot VAK files, um, sort of laying around, uh, and what you get with version control right off the back is the ability to roll back. 191 00:22:50,300 --> 00:22:56,440 You have history, you have context, you have comments, and, and you also share that. 192 00:22:56,540 --> 00:22:57,380 That's a shared resource. 193 00:22:57,500 --> 00:22:59,960 You don't have to go inspect each individual machine. 194 00:23:00,080 --> 00:23:05,880 If you have a shared resource that you can, and we'll talk about some of these techniques, um, a little bit more. 195 00:23:05,960 --> 00:23:09,480 And I don't really care what your version control is, as long as it's Git. 196 00:23:10,140 --> 00:23:10,660 Just kidding. 197 00:23:11,160 --> 00:23:14,160 Um, um, SVN is great too. 198 00:23:14,340 --> 00:23:28,640 Uh, I use SVN for years, and, and it's, um, certainly better than CVS, which I also use, but, I, I think Git is, um, it just, it enables a lot more flexibility, uh, and once you've used it for a while, then, then you don't want to use anything else. 199 00:23:28,680 --> 00:23:30,100 So, I'm going to advocate Git. 200 00:23:30,480 --> 00:23:35,300 So, so what do, uh, like, like, like, what do you, what do you see, what do people put into version control? 201 00:23:36,020 --> 00:23:48,180 All your config files, as a starter, all your, all the, even if you're not going to use, uh, things like Puppet, even if you're not going to use these other tools that are emerging, just version your config files. 202 00:23:48,400 --> 00:23:53,520 Just version all the little scripts that you have to, to build things. 203 00:23:53,520 --> 00:24:07,840 If, if, if you're, if you're a sysadmin with any chops at all, you, you've got a directory, um, with some Perl scripts or whatever your favorite language is that do things that help you get your work done, right? 204 00:24:07,880 --> 00:24:15,620 You're not, you're not, you're not, hopefully you're not typing everything from, uh, you know, first principles every single time you sit down. 205 00:24:15,620 --> 00:24:19,360 So, just versioning those is a huge step forward. 206 00:24:20,860 --> 00:24:22,600 Andrew, let me ask you the following. 207 00:24:23,000 --> 00:24:46,520 You know, uh, uh, in the software development, we check in code every so often, and we have got a whole bunch of tools, which are good at maintaining our code in, uh, what I would say, uh, usable, usable shape, in spite of the fact that there may be many branches and probably branches of branches. 208 00:24:48,860 --> 00:25:02,880 Basically, basically the metaphor that I think, uh, I really advocate is that with respect to the IP configuration, similar tools are of immense value. 209 00:25:03,380 --> 00:25:04,060 Is that so? 210 00:25:05,420 --> 00:25:12,760 Uh, I mean, the other thing that I think, I, I touched on this a minute ago, but I, I don't think I've brought it out too much in this podcast. 211 00:25:12,860 --> 00:25:27,480 But as you move to using abstractions, the APIs to bring up machines on, on, you know, vSphere, EC2, cobbler, whatever that gives you your, your base images, and then abstractions, um, configuration management layer. 212 00:25:27,480 --> 00:25:34,260 And you start to use these sort of tools, then that process looks exactly like software development. 213 00:25:34,720 --> 00:25:35,540 It's just code. 214 00:25:36,240 --> 00:25:42,740 Uh, you, you can take advantage of all these other things, and, and what you just said applies to that as much as anything else. 215 00:25:43,880 --> 00:25:46,060 Because it's, it's, it's now software development. 216 00:25:46,880 --> 00:25:52,580 And which tools would you, uh, mention, uh, under this analogy? 217 00:25:52,800 --> 00:26:02,960 Namely, our software control tools are very well known, uh, I would think, to just about anyone who reads this blog and listens to the podcasts. 218 00:26:03,500 --> 00:26:15,920 At the same time, we are addressing here, uh, a topic which is an extension to Agile, namely, Agile in the system management context, or applying Agile to system management. 219 00:26:16,700 --> 00:26:24,660 What tools are the equivalent of the source control systems that we know and love in software development? 220 00:26:25,280 --> 00:26:26,000 They're not different. 221 00:26:26,700 --> 00:26:27,720 You use the same ones. 222 00:26:28,920 --> 00:26:33,240 Not only that, but this is another point, that you should have a shared repository. 223 00:26:33,760 --> 00:26:43,520 Particularly if you're working in a, a web shop, and it, it breaks down a little bit if you have to manage, you know, ten, ten different applications. 224 00:26:44,020 --> 00:26:58,840 But if your business is, is a web app, if you're doing some sort of, you know, software as a service, then the, the operations, the config files, the abstraction layers for the config files, and the application, those should be in the same repository. 225 00:26:58,920 --> 00:27:00,440 So there's not a different set of tools. 226 00:27:00,620 --> 00:27:04,320 It's the same version control tools that everyone knows and loves. 227 00:27:05,760 --> 00:27:07,140 I hear you loud and clear. 228 00:27:08,220 --> 00:27:12,240 And thankfully, usually those don't cost very much, depending on, usually they're free. 229 00:27:12,380 --> 00:27:15,360 They just cost time to get up and running, which is always handy. 230 00:27:15,920 --> 00:27:25,060 Yeah, there's certainly, um, tools you can, you can buy, but I think that, like I said before, if you're not using Git right now, you should go at least try to use it. 231 00:27:25,060 --> 00:27:29,780 And there's a little bit of a, I think this is something that I run into a little bit. 232 00:27:29,840 --> 00:27:42,060 There's a curve of learning, and there's sort of namespace collision between Git and, and subversion, which makes it a little more confusing than it probably would have been if they didn't have commands with the same name that do different things. 233 00:27:42,440 --> 00:27:49,440 But once you, once you figure out how it works, it's free tool, it's Linus Torvalds, it really kicks butt. 234 00:27:49,720 --> 00:27:55,740 And once, once you understand Git, quote unquote, get it, you're not going to go back. 235 00:27:56,500 --> 00:28:03,300 So, so what are the, uh, you know, like, like, like we were saying, you're going to detail, go into tons of detail on all these things in, in your talk next week. 236 00:28:03,540 --> 00:28:08,220 But, uh, like, what are some of the other practices and technologies aside from version control? 237 00:28:08,780 --> 00:28:21,400 So, so, so just really quick, and I'm stealing most of these from, uh, the person I always talk about, um, in these, John Allspaugh, but if, and this is web app, off specific as well, but, um, always ship trunk. 238 00:28:21,400 --> 00:28:27,020 If you have a shared repository and you're always shipping trunk, everyone knows where to look when things go wrong. 239 00:28:27,680 --> 00:28:39,220 Um, they, they also do some interesting things at Flickr, um, branching instead of in the, in the version control, you, you put your branches in the actual, um, code. 240 00:28:39,660 --> 00:28:41,680 So you can turn features off and on. 241 00:28:42,240 --> 00:28:58,420 And that allows you to do interesting things like dark launches where you're going to, you're going to, you're going to test the impact of new features on the infrastructure by running the code, but not exposing it to the UI, not exposing it to the user. 242 00:28:58,640 --> 00:29:08,520 And that, that gives you some, um, ability to reason about how it's going to go when you, when you roll this thing out to, you know, quote unquote, uh, live. 243 00:29:08,840 --> 00:29:19,840 Now, now, is, is that, is that, is that different than like turning it on for only 20% of your users or is it what, because it sounds like you ship everything except the ability for the user to actually use it. 244 00:29:20,340 --> 00:29:22,640 Well, well, you, you, I think you might do it. 245 00:29:22,980 --> 00:29:37,280 Um, and this is something I would consider doing and it's going to depend on the feature of course, but if you, if you turn it on for 20% of your users and, and I've, I've been, uh, bit by this before, then that gives you 20% of the impact. 246 00:29:37,560 --> 00:29:42,280 Which might not be a great indicator of what it's going to do with all of your users. 247 00:29:43,080 --> 00:29:54,700 And, I mean, and, and this is just, this is just ways to get data and educate yourself because you can never, one thing that you can never account for in software is how users are going to do stuff. 248 00:29:55,260 --> 00:30:06,440 And, and so that, that can certainly cause you problems, but what you're trying to do is give yourself as much information as you can before you put the, you know, before you turn the lights on. 249 00:30:06,840 --> 00:30:17,780 And so, and so with, with the, with the dark, I mean, with doing this dark launch thing, like that sort of gets you, gives you a sense of what it's going to be like to like, what, what, what, what sort of problem does that address? 250 00:30:18,260 --> 00:30:25,260 Well, let's, let's say you want to introduce something that makes suggestions to the user based on their preferences or what they've already done. 251 00:30:25,340 --> 00:30:28,200 Some sort of context, um, specific suggestion. 252 00:30:28,200 --> 00:30:35,560 And, and that might be some, some computation on the backside and, you know, whatever that problem is, you know, some queries to the database, whatever. 253 00:30:36,080 --> 00:30:44,060 Um, if you, if you, if you want to turn that feature on in a dark way, you're going to actually run all those queries for everyone. 254 00:30:44,180 --> 00:30:47,500 You're just not going to give them the ability to see that happen. 255 00:30:48,080 --> 00:31:00,560 Um, and then, so, so basic, basically with the dark launch, you, you can do it with things that don't require user input that, that sort of like, there's all this processing that happens in the backend, so to speak at the non UI layer. 256 00:31:00,560 --> 00:31:10,440 And, and you're testing out that that will work and then the next, and then you're staging this release and the next thing is like, okay, now let's let the user not only see it, but like give it an input or something. 257 00:31:10,560 --> 00:31:14,100 You need to use your example, like fine tune the context and then we'll see if that part breaks. 258 00:31:14,300 --> 00:31:25,340 I mean, you could also, you should, you could also take it a step further where if, if you required some user input, then you could generate those behind the scenes, right? 259 00:31:25,340 --> 00:31:45,440 If it's going to post to some, some form or, or post to some, um, URL, then you, you could come up with what you, you know, and, and obviously you're, you're not, this goes back to what I just said, that you can't predict everything users are going to do, but you can maybe make some assumptions and see what those assumptions, what the impact of those assumptions are on the infrastructure. 260 00:31:46,040 --> 00:31:54,260 Andrew, I would like to clarify one point, uh, as I think it's of critical importance in the context that we are discussing. 261 00:31:55,120 --> 00:32:07,680 The, the going under the GUI, so to speak, that you recommend and that you indicate that, uh, Flickr is using is in the production system. 262 00:32:08,180 --> 00:32:08,360 We are. 263 00:32:08,360 --> 00:32:08,640 Yes. 264 00:32:08,860 --> 00:32:09,580 Yeah. 265 00:32:09,580 --> 00:32:18,800 I think this is of critical importance because, you know, the old session philosophy is that I have an intermediate, uh, environment, which is pre-production. 266 00:32:25,120 --> 00:32:37,940 And part of the production environment, part of it is probably the configuration drift that you mentioned, but the other thing is that there is no way to replicate a full production environment, or at least no economically, possibly way. 267 00:32:38,620 --> 00:32:58,220 So the immense power of, uh, or, uh, what you are advocating is that this kind of quote unquote testing of features or turning, uh, on and turning off features is actually done in the production system, not in any one of the systems that precedes. 268 00:32:59,680 --> 00:33:02,940 Well, you, you would, you would certainly test everything. 269 00:33:03,160 --> 00:33:14,000 So you're going to go through, and this is a problem that I, I've also been bit by in my career, is you have this difference, the configuration drift between, um, the test and production system. 270 00:33:14,000 --> 00:33:24,920 So you test a feature in test, and because of either a configuration file or the state of the database, there's something about that test succeeding in, in the test environment. 271 00:33:24,920 --> 00:33:33,640 And when you put it live, then, then there's some assumption about how things are that you fail to account for, and it ends up being a long, painful night. 272 00:33:34,160 --> 00:33:39,540 Um, what, what you can do with, with this, and this is sort of the, the next stage. 273 00:33:39,540 --> 00:33:47,040 So once you have everything in version control and you're able to, um, build things with these abstractions is you're going to rebuild from source. 274 00:33:47,220 --> 00:33:50,180 So everyone talks about rebuilding from source with their application. 275 00:33:50,740 --> 00:34:05,900 And, and in some level that, that doesn't make sense with interpreted languages, but what you want to be able to do is from the version control, which has a shared repository for your operations and for your application, go from bare metal to a fully running service. 276 00:34:06,380 --> 00:34:11,880 In one step, a fully automated, build from source step, right? 277 00:34:11,880 --> 00:34:16,040 What do I do about stored procedures, uh, within this modus? 278 00:34:17,460 --> 00:34:25,100 Well, I mean, stored procedures are on the database and you should be able to rebuild your database from scratch. 279 00:34:25,660 --> 00:34:32,000 So it applies not, quote, unquote, only to the application or only to the infrastructure. 280 00:34:32,420 --> 00:34:42,000 It fully applies to every component in, uh, the overall setting, namely the database, uh, and any others, right? 281 00:34:43,380 --> 00:34:51,240 I, I would say that, but I'd also say, um, there's a caveat and it depends on, um, how you want to think about the world and how you want to build things. 282 00:34:51,240 --> 00:34:55,980 But I would probably try to steer people away from stored procedures in a lot of cases. 283 00:34:56,340 --> 00:35:07,460 Um, I, I, I wrote plenty of PL sequel before, and I know there's, there's a lot of power and flexibility there, but I think that there is a tendency because of the way those systems work, um, 284 00:35:07,960 --> 00:35:14,200 to not have that stuff be in version control, and that involves, like, that's a whole nother subsystem, right? 285 00:35:14,240 --> 00:35:20,760 So right now we're talking about, um, systems and ops and, and bringing those people into this process. 286 00:35:21,180 --> 00:35:33,380 Database administrators, um, I think that that's another, another little, um, sparkle to add to the fractal that, that most people don't, uh, do a very good job of accounting for in an agile process. 287 00:35:33,820 --> 00:35:39,200 So, uh, if that, if that, um, resonates with your experience, uh, I'd love to hear it. 288 00:35:40,320 --> 00:35:46,940 Very, very much so, and I don't want to speak to Coté, but I believe that it resonates for, uh, his experience as well. 289 00:35:48,280 --> 00:35:58,400 Yeah, no, I, I mean, I think, uh, there's a reason most people don't like DBAs, and that's that they're very, uh, they fight tooth and nail to maintain complete black blocks control over their domain. 290 00:35:58,780 --> 00:36:15,660 And, you know, applying a lot of, you know, a stored procedure seems more often than not, at least, uh, I don't know, for lack of a better word, sort of politically and management-wise, like, a good, a good weapon to use to sort of build up your tower, your defenses against, you know, your job being optimized away from you. 291 00:36:16,280 --> 00:36:17,880 To give the cynical view of it. 292 00:36:18,860 --> 00:36:21,960 You know, and, and, and, Ed, what were you saying, Israel? 293 00:36:22,600 --> 00:36:38,520 I think I need to go to the various consulting assignments that I had over the past year or so, and look carefully at each one of them in view of the perspective that you offer, namely, how much of it was to preserve a job? 294 00:36:39,580 --> 00:36:40,040 Yeah, exactly. 295 00:36:40,440 --> 00:36:52,700 Yeah, that, that's, uh, that's one of the, that's probably one of the problems that, uh, that we might have mentioned in a past episode that we didn't mention at the beginning of this is, is people, uh, just wanting to maintain the status quo of their, their day-to-day work life. 296 00:36:53,020 --> 00:36:53,380 Right. 297 00:36:53,380 --> 00:36:55,120 Which is always, uh, always kind of difficult. 298 00:36:55,120 --> 00:37:16,080 So, you know, what, what are, uh, as, as you were talking about, I was thinking, you know, the, it seemed, taking it to, like, silly extremes, it seems like at some point you kind of get to the, uh, at least the idea of, I don't know what you would call it, but kind of analogous to having continuous builds, to having continuous production deploys, where you're sort of just gratuitously doing a deploy, a deploy to see if it works or not. 299 00:37:16,500 --> 00:37:25,100 But, uh, you know, I don't know if that's necessarily a good idea, but having that ability, I think is more, you know, is the interesting thing to, it gets to the idea. 300 00:37:25,100 --> 00:37:32,300 It gets to the idea of how, how you build your system, including all the infrastructure that basically you could build it from bare metal from scratch at any one point. 301 00:37:32,900 --> 00:37:34,480 And I actually, as much of that as possible. 302 00:37:34,960 --> 00:37:36,060 I think it's a great idea. 303 00:37:36,260 --> 00:37:42,320 And, and what you really want to do is, what I consider, you want to reduce the fixed cost of deployment. 304 00:37:42,800 --> 00:37:43,240 Right. 305 00:37:43,300 --> 00:37:46,740 And you want, you want to reduce it to as close as zero as you can get. 306 00:37:47,340 --> 00:37:47,780 Right. 307 00:37:48,260 --> 00:38:03,740 And, and in general, that's in, and, you know, there's, there's a spectrum, obviously, but in, in organizations that consider themselves quite agile, there's often this waterfall-like step to put whatever they did onto the servers. 308 00:38:04,460 --> 00:38:09,220 So, uh, so, so, so are there some more, uh, some more practices and technologies you can tell us about? 309 00:38:09,760 --> 00:38:11,660 So we were talking about reducing... 310 00:38:11,660 --> 00:38:15,000 Come on, give him a chance to speak about it in the conference. 311 00:38:17,200 --> 00:38:17,900 That's right. 312 00:38:20,220 --> 00:38:23,380 Well, you got to give us a few more, just, just a couple maybe. 313 00:38:23,860 --> 00:38:27,900 And then, is there going to be a recording of your talk or anything or like how, uh... 314 00:38:27,900 --> 00:38:36,620 Well, we got email from InfoQ that they want to, uh, record, uh, myself and Paul Nassrat, who's the, he's the, he's the Ops lead at The Guardian. 315 00:38:37,060 --> 00:38:37,300 Yeah. 316 00:38:37,980 --> 00:38:42,920 And we're, we're co-presenting some of these ideas, uh, next Thursday at Agile 2009. 317 00:38:43,280 --> 00:38:52,020 Well, in that case, I mean, since there is going to be a record of it, why don't you just, like, really quickly run through what the other ones are, if only by name, and then they'll have the details at the top. 318 00:38:53,180 --> 00:38:56,700 Uh, so, there's some notions of, of a test-driven infrastructure. 319 00:38:57,040 --> 00:39:02,040 I, I'm kind of on the fence of what that actually looks like and what that means, but it's certainly an interesting idea to play with. 320 00:39:02,540 --> 00:39:04,480 Um, I talk about deploying early and often. 321 00:39:05,000 --> 00:39:07,140 Um, you're trying to get rid of, uh, this. 322 00:39:07,500 --> 00:39:12,620 So, so, as I mentioned a minute ago, you, you have this sort of waterfall-like artifact. 323 00:39:13,040 --> 00:39:14,700 That is the deploy process. 324 00:39:15,260 --> 00:39:28,600 And if you can move to a more incremental, you know, kind of deploy early and often, uh, mindset, then when you do have issues, it's very easy to reason about them because you have small chain sets that you've just introduced. 325 00:39:28,820 --> 00:39:34,140 And you can, you know, you don't spend, you know, the hours kind of backtracking to figure out what, what you've just changed. 326 00:39:34,380 --> 00:39:34,520 Right. 327 00:39:35,520 --> 00:39:41,180 Um, there's, there's, there's this notion of, of being able to tag everything with, uh, who, what, when. 328 00:39:41,180 --> 00:39:50,640 And that's sort of tying your version control into your deployments and being able to correlate the, those tags with, with the metrics. 329 00:39:50,960 --> 00:39:51,940 Ah, that's, that's interesting. 330 00:39:52,060 --> 00:39:55,840 That gets you into, like, CMDB territory in theory and things like that. 331 00:39:56,700 --> 00:40:09,920 In, in theory, uh, I think it's, it's probably a little more pragmatic, but, and, and you're doing it to enable change where I feel like a lot of the, the ITIL is sort of in that mindset where you're, you're impeding change. 332 00:40:09,920 --> 00:40:39,900 Um, but being able to, to see, you know, who, what, and when in relation to, to the metrics and, and who, what, and when is, is, is, you know, the, the requests to your application and looking at all the metrics from, you know, from the disk IO to the CPUs and seeing that on, on one timeline correlated, uh, gives you the ability to, to go into the code and the change sets that you made in both your application and in your infrastructure. 333 00:40:39,900 --> 00:41:09,880 Yeah, yeah. 334 00:41:09,880 --> 00:41:11,300 Um, and who, what, and when. 335 00:41:11,540 --> 00:41:19,540 Um, this is, this is a very simple thing, but it's, it's been me in the past, and I'm pretty sure it bites a lot of organizations on a regular basis. 336 00:41:20,000 --> 00:41:24,320 Uh, synchronize the time on all of your servers and all of your systems. 337 00:41:24,960 --> 00:41:39,060 Um, I, I would probably advocate putting everything on UTC, but you, I've seen, and, you know, I've had personal experience where you have one data center or another data center, or these machines got provision some way. 338 00:41:39,860 --> 00:41:58,340 And so there's different time zones, and they're on different, uh, time scales, and, and then, you know, you, you always know that in the back of your mind, but then there's that one time when you, you, you know, you see data from one and the other, and you don't make that, that mathematical, uh, calculation right, and it causes all sorts of, uh, confusion that doesn't need to be there. 339 00:41:58,340 --> 00:42:05,340 You have to, uh, you have to, uh, be very good about being a little time zones in your head, and you love all that crap, which is a, you're just a side effect. 340 00:42:06,340 --> 00:42:15,460 So there's, what about, what about, what about the ecosystem, which includes the partners, namely, within one corporation? 341 00:42:16,180 --> 00:42:26,180 It might be difficult, but it's relatively easy to have a policy, uh, of, uh, you know, UTC, that's, uh, our standard. 342 00:42:26,180 --> 00:42:39,180 However, in terms of, uh, one data center interacting with another data center, interacting with, um, uh, websites, uh, you know, in the cloud, et cetera, et cetera, et cetera. 343 00:42:39,180 --> 00:42:49,700 So, uh, what would be your good recommendation in terms of this kind of synchronization in the lab instead of synchronization within a corporation? 344 00:42:50,980 --> 00:43:00,280 Well, that, that's something you're going to have to basically deal with, and this kind of goes back to the, to the beginning where some of this is hard, and some of this is complicated. 345 00:43:00,980 --> 00:43:05,440 And, you know, you can, uh, you can write things that, that translate that. 346 00:43:05,440 --> 00:43:14,180 If you, if you're going to measure metrics on other people's services, then you, when you make those measurements, you don't necessarily need to ask them for the time. 347 00:43:14,520 --> 00:43:17,440 The servers that you're making requests from have their timestamps. 348 00:43:17,940 --> 00:43:29,120 So, I think there's ways to, to frame it from, uh, you know, organizationally centric perspective and get those timestamps using your own systems that, that kind of makes that work. 349 00:43:29,700 --> 00:43:35,520 Andrew, you remind me of Donald Knuth's classic three big software in the cloud. 350 00:43:38,200 --> 00:43:41,080 That's, that's, that's what, um, I started to figure out recently. 351 00:43:42,660 --> 00:43:43,820 I know the feeling. 352 00:43:46,400 --> 00:43:52,120 Um, so, when you get to this point where you can build from source, uh, it gives you a lot of flexibility. 353 00:43:52,500 --> 00:43:58,200 You're going to migrate everything through the same, same sort of test dev prod process for all your systems. 354 00:43:58,420 --> 00:44:03,260 But it's also got, um, the ability to rebuild or to scale. 355 00:44:03,440 --> 00:44:09,960 So, once you're in the, the habit of rebuilding your systems, you, it gives you a lot of advantages for testing too. 356 00:44:09,960 --> 00:44:13,040 Because, say, say you're, you're doing unit testing. 357 00:44:13,160 --> 00:44:18,800 There's often some, some state that you want to, you want to put the system in so that you can test that assumption. 358 00:44:19,920 --> 00:44:28,200 Now, in, in my experience, I've seen many times where people have been bitten because of some assumption about the state in the test environment. 359 00:44:28,200 --> 00:44:36,540 Now, if that test environment is something that you rebuild and throw away every day, then when you rebuild it, it is in a known state. 360 00:44:37,220 --> 00:44:42,860 And that, that gives you power to reason, uh, about what state it's in, what state you want to put it in. 361 00:44:43,320 --> 00:44:48,700 And, and that sort of flexibility has, has benefits that kind of cascade across the process, in my opinion. 362 00:44:49,220 --> 00:44:50,640 Oh, right, right. That makes sense. 363 00:44:51,300 --> 00:44:57,200 And, and, and because you have all this practice, rebuilding systems, there's also disaster recovery, um, implications. 364 00:44:58,200 --> 00:44:58,960 Yeah, yeah. 365 00:44:59,220 --> 00:45:12,240 And you can do all, you, you, you just are, you're sort of in, to use another, like, metaphor, you're sort of in continuous training, if you will, to sort of, by doing things continually like this, you're always in training to handle when things go wrong. 366 00:45:12,240 --> 00:45:19,180 Just because it's sort of like, those are the things, you're doing those practices more often when it comes to just rebuilding systems or troubleshooting them or whatever. 367 00:45:19,800 --> 00:45:28,620 And, and, and once you get it to the point where everything's stable and you can rebuild it and, and no one knows who you are anymore, then you can start doing fire drills just for fun. 368 00:45:29,060 --> 00:45:29,720 That's right. 369 00:45:31,140 --> 00:45:32,760 I don't know about that. 370 00:45:35,060 --> 00:45:35,740 Why not? 371 00:45:36,800 --> 00:45:40,380 Well, as long as it's a drill, it's often relaxing because it's not stressful. 372 00:45:40,920 --> 00:45:42,680 So you just kind of hang out, as I remember. 373 00:45:42,820 --> 00:45:47,540 Everyone put, you put on the red baseball cap and leave the building and then, like, go get a soda afterwards. 374 00:45:48,240 --> 00:45:50,900 Well, let's say, theoretically, go ahead. 375 00:45:50,900 --> 00:45:57,740 Guys, may I suggest that we watch Dr. Strangelove before we go farther into the fire drills? 376 00:46:01,000 --> 00:46:01,500 That's right. 377 00:46:01,580 --> 00:46:04,760 It's a cautionary tale for applying these, these ideas too much. 378 00:46:04,880 --> 00:46:06,140 We don't want to be living in a cave. 379 00:46:06,840 --> 00:46:08,220 Oh, the don't stay machine. 380 00:46:09,880 --> 00:46:10,480 That's right. 381 00:46:10,900 --> 00:46:17,180 Well, I, I think, I, I mean, I, I think we should, we should wrap it up so we don't make our listeners sit in here and listen to too much of it. 382 00:46:17,600 --> 00:46:21,900 But, but I, I think, I think we kind of got a good sense for what some of these practices are. 383 00:46:21,960 --> 00:46:24,900 And once that recording gets put up somewhere, we'll have to put a link to it. 384 00:46:25,140 --> 00:46:30,120 And then that'll be sort of like the more, and along with the presentation, hopefully, and that'll be a more in-depth look into it. 385 00:46:30,620 --> 00:46:34,840 So in the presentation, we'll, we'll probably go more in-depth on these technical things. 386 00:46:34,840 --> 00:46:48,000 We have a, a bit of discussion about, um, use, taking legacy systems, existing, um, systems and practices and trying to migrate towards, um, you know, using these sort of techniques and technologies. 387 00:46:48,000 --> 00:46:58,020 And then we also have, uh, a bit of information about, uh, process and culture, um, in addition to, and there's some crossover, obviously, between, uh, the, the technology and the process. 388 00:46:58,180 --> 00:47:00,820 But that's, uh, that's what we're shooting for next week. 389 00:47:01,000 --> 00:47:02,620 Yeah, it sounds like it'll be exciting. 390 00:47:02,660 --> 00:47:06,240 And both of you guys will be at, at, uh, Agile 2009, right? 391 00:47:06,820 --> 00:47:07,220 Absolutely. 392 00:47:07,600 --> 00:47:12,220 And I'm hoping that my own presentation does not collide with Andrew. 393 00:47:12,540 --> 00:47:14,500 Mine is Thursday, 11 o'clock. 394 00:47:14,600 --> 00:47:15,580 When is yours, Andrew? 395 00:47:16,260 --> 00:47:17,900 Um, I, it's, it's Thursday. 396 00:47:18,000 --> 00:47:19,440 Uh, 2.30, I think. 397 00:47:19,500 --> 00:47:20,220 It's in the afternoon. 398 00:47:20,460 --> 00:47:21,360 I'm not even positive. 399 00:47:21,720 --> 00:47:23,180 It's, it's, it's, it's after 11, though. 400 00:47:24,120 --> 00:47:25,260 We are safe, sir. 401 00:47:25,340 --> 00:47:32,980 I would be, uh, all anxious to attend your presentation and to learn, uh, from you and from the co-presenter. 402 00:47:33,440 --> 00:47:38,320 Well, I've already learned plenty from you, but I'm, uh, looking forward to your presentation as well. 403 00:47:39,480 --> 00:47:39,960 Delighted. 404 00:47:40,040 --> 00:47:40,460 Thank you. 405 00:47:41,540 --> 00:47:47,240 Well, well, on that, uh, that happy note, I guess we might as well wrap it up unless there's anything else you guys wanted to mention. 406 00:47:47,800 --> 00:47:50,020 We, we, we can sing kumbaya real quick. 407 00:47:50,020 --> 00:47:51,400 I don't remember the words. 408 00:47:55,400 --> 00:47:56,160 Oh, fantastic. 409 00:47:56,660 --> 00:47:58,480 Well, uh, so, that, that sounds great. 410 00:47:58,560 --> 00:48:01,560 Well, uh, thanks everyone for listening and we'll see you next episode. 411 00:48:02,260 --> 00:48:02,700 Perfect. 412 00:48:02,860 --> 00:48:03,020 Thanks. 413 00:48:03,140 --> 00:48:03,360 Oops.