All right, here we are at SpringOne Platform, the most awesome named conference in the world. Very, very two-name conference there. Well, three, if you don't really account for the lack of spaces. We have Matt Curry here. How's it going? And then Kenny. Hey. As always, fresh in from the Mediterranean. Yes. So how are the scooters there? You told me it's a bad idea to rent a scooter, but give me a review. Well, it's only a bad idea to rent a scooter if you don't know how to drive a scooter, and that doesn't really work out well. So I got chased down by the guy who rented me the scooter after he realized that I wasn't very good at driving scooters. That seems like it maybe isn't going to go well. No. You can visually see that you were not very good at riding the scooter. That's hilarious. It was beyond the level of just, like, comprehension of what was going on. But people made it look so easy, I thought, but I could do it. So what's hard about it? Staying up. What's hard about it? Everything. Yeah. I guess you do have to balance it. And then that's the one where there's, like, shifting on the handle or something? I don't know. Yeah, I didn't get that far. You're just wobbling around. So I noticed you haven't picked up the, like, you know, you didn't bring any affectations back. Like, you haven't said chow or anything to anyone. Have you found yourself doing that? No, no. Okay. Yeah. I don't know much Italian or French. You have this problem that my wife and I always have, where when you travel abroad, and this is kind of regional since we're down in Texas, but we, our instinct is to just start speaking Spanish to people, like, whatever country we were in. Yeah. It's sort of, it must be like our, if it's not English, it's Spanish. And so we have to catch ourselves. How long have you lived in Texas for? Oh, all my life. Well, that explains it. Yeah, yeah, yeah. So it's just ingrained. Yeah. All right. Well, so you gave a talk yesterday. Yes. Which I had a very good excuse for not going to. Yes. I had a better excuse. Yeah, you weren't even here yet, right? Yeah, Kenny and I got booked on the same slot, so it was like, I had to miss his talk,
2:01he had to miss my talk. That is much better. So I think I saw, like, an early, like, overview of your talk or something like that. But, anyways. But I'm sure it evolved. Like, what did you end up talking about? Yeah, so, I mean, what was interesting is I wrote the executive summary a while ago, and then I had to kind of figure out what I was talking about. So I titled the talk, let's see, Taking DevOps to 11 with Unicorns and Rainbows and Sprinkling a Little Bit of Cloud on It. Wasn't CICD in the title or something? It was a very long talk. Or not. No, a very long title. Yeah, yeah, yeah. But what I ended up doing, or how I ended up structuring it, is I basically took everyone back to kind of first principles and the history of, like, the emergence of DevOps as a term and what problems we were trying to solve as a community at the time and how that community of practice kind of came together. Yeah. And then used that as the framing for, like, okay, if this is the foundation, then what are all the things that are important to do in order to kind of get the most out of a DevOps practice or a DevOps transformation? Right, right. So what was the framing you started with? I think, you know, everyone has at least multiple, one or more versions of, like, how DevOps came about. So it's always fun to hear a, I'm sure you have a good, rational, helpful one. Yeah, so I took the Wikipedia definition, which is the, I think, one of the original definitions from 2008. And I kind of got the idea from, I can't remember who, but I saw this rant on Twitter about what DevOps is and isn't, and that was, like, referenced in there. So it's like, hey, this seems like a good spot to anchor the talk. And it was really about that, you know, DevOps is a set of practices in order to be able to more, with more speed, deploy changes to production with quality. So that was kind of the, I think those are the major parts.
4:01I totally botched, like, the official reading of that definition. Yeah, that's pretty much all I ever say nowadays. But it was like, there's practices. Yeah. It's about going faster. It's generally about making changes in production or to production, and we want to do that with quality. Yeah. And then I kind of just, like, broke down each of those four and highlighted a lot of the underpinnings behind each of them. I also spent some time on culture and, like, the principles of culture. Because I was reading the latest State of DevOps report, which is a really good piece of validation, like, because it kind of highlights how my experience is the same or different than other folks' experience with DevOps. And there's a lot of, like, similar themes. And I think, like, one thing that's highlighted in there is how critical the leadership buy-in aspects are. Yeah. And I think it's because, like, the cultural aspects of what comes out of those practices is, in some cases, like, diametrically, you know, it's opposed to what's in the organization. Like, what are some examples of that? Like, high levels of trust. Right. So, like, DevOps is about enabling high levels of trust in teams. Well, if your organization is, like, not that, then it's going to be hard. Yeah, yeah, yeah. So, you know, having autonomous teams is a thing. So, what are the reasons why people don't trust each other? Isn't that a part of identifying what DevOps means for a company? So, I think that the trust isn't as much a personal thing as it is, like, do I trust the teams to go do them themselves? And I kind of hit on this in the talk, which is I talked about the importance of having, like, sensible defaults for things and the importance of taking actions that every developer is doing and, like, moving that into the platform so that you get, like, some kind of centralized enforcement of certain aspects through automation.
6:23You don't want to just, like, roll the dice, and I hit on this a little bit, like, with cloud breaches being one a week and largely related to misconfiguration, you don't really want to just, like, roll the dice on that and be like, we have all these teams, we put smart people in them, so it's probably fine. Like, your CIO is, that's going to keep him or her up at night. Do you think that visibility increases trust in the organization, so if you can actually monitor the behavior of what the developers are doing through the platform, does that help you decide, I guess, what to do to get people to get on the same page and trust each other? I think the trust is about, largely about education and making sure that things are discussed openly and kind of known, because really it's just, like, I have concerns that it's not necessarily lack of trust. Usually, in many cases, it's, like, I have concerns that nobody has addressed, and so I have a lack of trust or, like, some reservations about this thing that we're going to go do. And how do we make sure we know? And it seems like, and another part might be, I mean, tell me if this is wrong, but that is a weird way of putting it, but there is still some mistrust in the way you're doing things, but you've automated making sure that this stuff works, right? Like, there's different ways of automating stuff and enforcing it that maybe previously didn't exist. Like, if you're manually putting together a release, right? Yeah. Of course, you're going to have a lot less trust than if you, like, trust the thing that enforces the way things are done and automated. So, there is still the effect of mistrusting. Right. You just don't have to worry about it as much as you do. I think the way that I kind of look at that is you're taking away responsibilities from the individual teams through layers of abstraction, and because of that, like, you can trust them more for the things that they have responsibility for. Yeah, yeah, yeah. Because they have responsibility for, like, less stuff. Yeah. So, like, do you think that in, I mean, having looked at this and everything else, like, do you think that there's a good answer for, like, scaling up DevOps?
8:31Like, is there any material on that or experiences or, like, you know, every now and then there'll be, like, the Capital One person has, like, 30-minute talk, but I don't know if they really ever cover, like, how you spread that to the entire organization or something like that. So, I think one of the talks I referenced in my own talk is the John Olspot talk and Paul Hammond, which was from Velocity in 09, like, the 10-plus deploys a day. They didn't even call it DevOps at the time. They were, like, you know, collaboration between devs and ops. And something that they hit on, there's a couple, I think, really key things that they hit on there. One is there should be one button to produce a deployable artifact, so there should be a second button to actually deploy said artifact, and, like, that should be the maximum level of effort. And then the second piece that they really hit on is how important it is to have consistency in the infrastructure so that developers aren't spending, like, huge amounts of time trying to figure out why stuff doesn't work. And so, like, I view it as kind of that second part where, you know, especially with the structured platforms, they provide a huge opportunity to kind of do this, like, standardization where it's, like, everything looks very much the same. I can jump from team to team and, like, make sensible assumptions about how things work because they're very consistent in nature. Yeah. My deploys are very consistent in nature, so I think, you know, that's super critical in terms of scaling. Yeah. Yeah, I always have this, like, self-imposed, what would you call it, like, a self-imposed handicapping of this idea of, like, what you really, and I've tried to get over this, and it's self-imposed because, you know, I work for a vendor at Pivotal, and, like, the following highly favors us, right? Right. Which is, like, you should probably just homogenize as much as possible. Like, you want to standardize on the exact same thing and people having the same stack and, like, and, you know, like, use the same build pipeline,
10:36use the same platform, all the same frameworks. Because, to your point, right, like, in a large setting, the more variation you have, like, down in the stack, like, just like you said, the more hassle you're going to have and the more things to deal with. But isn't it more about the communication pathways between your developers and your operators? Like, for instance, if, for some reason, a developer is unable to go forward and they need an operator to respond to a ticket or a request to look into the infrastructure, doesn't that cause problems when they're not able to self-service? I think that, I mean, I think there's two dimensions of that. You know, sometimes the operator has experience that the developer, like, really needs. And so you're not going to get that through a ticket. I think the challenge, and I hit on tickets a little bit during the talk, but I think the challenge with tickets is, you know, many have demonized the ticket. And, like, in some cases, there's still a need to track and audit requests for work. So, like, it's not the ticket that's the issue. It's the engagement model that's the issue, which is putting a person behind a ticket and they have, like, zero additional context behind, like, you asked me to do a thing in a ticket. Yeah. Which I've bubbled down to, like, ten drop-down boxes because I get tired of people asking me, like, the same things in different ways. I guess, you know, like, sort of technologically, the difference between, like, a physical sticky note, a digital sticky note, and some Kanban thing, and then, like, a ticket is not really that much. Right. It just depends on what you do with this thing. Although, to be, I don't know if fair is the right word, but to be descriptive, your ticket systems, if they're, like, five, ten years old, they can get a little crazy. Right. Yeah, they can. Like all the different fields and stuff, but still. And people do crazy things with Jira these days as well. Yeah, no doubt. We have all kinds of crazy workflows and who knows what, but, you know, Kenny, I think, like, one thing that is huge is kind of the alignment of product.
12:36So, like, you start moving teams to be more product-focused and making both your dev and infrastructure folks align to, like, customer outcomes and business impact. And being able to, like, push that forward is something that helps kind of with the engagement model. Yeah. Obviously, it's, like, super ideal to have dev and ops folks sitting side by side. And I think if you're building a platform team or another DevOps-type team, like, the cross-functional aspect of the experience of the different team members is hugely important. Right. Definitely have a tried-and-true infrastructure person, someone that understands, like, networking and servers and storage. You should have somebody that really understands, like, the operating system. You should have somebody that really has a good foundation in, perhaps, like, middleware. And then an application developer, like, somebody was asking me after the talk, he said, well, certainly you don't put software developers on this team if this is, like, coming out of infrastructure. And I was like, of course you do. Like, why wouldn't you do that? Like, if you don't have the diversity of experience, you create artificial blinders to some extent about what's possible when it comes to problem solving. Yeah, so to revisit something, I think the last time we talked on tape, as it were, it's weird, I still say tape, but I don't know what else you would say, on SD, but, like, so to that end, you kind of made, maybe I'm reading into it, but an interesting distinction there. You're like, if we have developers working on, like, a service or a platform or an infrastructure thing, you would have operations people there working with you. Now, conversely, what if you had, like, what's been your experience so far with, like, I've got a user-facing, a human-facing, customer-facing application, right?
14:36And I'm just building out whatever that application may be, and I'm not building, like, a service for other developers to use. Like, do you end up actually having operations staff on just, like, a regular apps team? I mean, this is my ongoing, like, thing of, like, is DevOps actually a real thing, right? Like, is the idea of it a real thing, which I still don't quite know. I mean, if you look at John and Paul's talk, like, they kind of ask the question of, like, how many people who would label themselves as, like, infrastructure folks have, like, made a change to the code base, like, ever? Yeah, and I mean... And the user-facing application, and, like, I think, like, pretty much nobody... And then to interrupt you, right, like, the thing that, like, the way that I boil down the salient thing for SRE thing is, the different, the thing about the major leap for SRE people is you have ops people who code, right? And so, like, you're putting dev in ops, but you don't necessarily, like, if I'm working on a, let's pick something here, you have an application that allows you to register and pay for a conference and optionally book one of the three hotels they have. And so, like, if you were the application team working on that, like, do you really have, like, an operations person with you? Or is it just all developers working on that? I think it comes down to self-service. Like, how much can you self-service to, I guess, deploy your application into production and keep it running? I mean, I think there was definitely a time when you needed to have, like, that full cross-functional team for DevOps to really exist. But, you know, I think what we've seen is for the folks, the unicorns that do DevOps, like, there's always some semblance of structured platform, whether it's for Google or, you know, a lot of the other .coms have some something, you know. It's not necessarily Cloud Foundry, in many cases it's homegrown, but there's some level of abstraction where a developer basically says, like, here's my code.
16:37It's like, go run it. Right. Yeah, and so, I mean, that's the thing I've kind of settled on is, one, like you said, you have operations people who are building services and stuff for others to use, to build on top of, right? And, of course, you need operation expertise there because it's inherently operation stuff going on in there. And then you have, you might have, you also have sort of pure ops people who, in this kind of way of thinking, their responsibility is, like, to automate things totally, right? Like, totally with the sense of a little footnote that says, like, I mean, if you can't do it, try it three more times and then maybe give up. But, like, so they need to automate things instead of with code or, you know, coding it away. And then the third one is, like, what we were just talking about is, and then eventually you have, like, a collection of services that you've built up or customized or however you've acquired it. And because it's all self-service, like, the application people need to know, like, how the internet works and what that implies for how they do their coding. So they do need to have operational knowledge, but you don't necessarily just have the sysadmin sitting on the team with it, right? Like, it's not quite like that at all. Yeah, I think, like, there's value in having, like, some operational expertise on the dev team if possible, but, you know, like you said, like, I think that's something that's much less common. Yeah. Much harder to come by. And arguably with structured platforms, like, it's less important. But, you know, even with structured platforms, like, folks struggle with, like, networking and making sure the right firewall rules are open. Yeah, yeah, yeah. Really understanding, like, how that, how that all works, and triaging and troubleshooting some of those things is very difficult sometimes. Yeah. Yeah, well, I think, I think just as you, you want your operations people to code, the other side of that is you want your, your developers to have operations knowledge and sense, right?
18:37Yeah. Like you were saying. And I don't, but I don't know if it literally means, in practice, the, you know, the portmanteau of dev and ops on the same team. It's more just like having that, I don't know. I don't know what the answer is, but it's, it's, I mean, I guess DevOps is kind of a community of practice. So it's not even really like we're doing DevOps or we're not doing DevOps. Like, DevOps in itself really came out of the creation of DevOps days, is my understanding of it. And so, you know, the, I mean, it's really just a set of practices. It's not really about roles or titles. Right. So I think that's where people get wrapped around the axle a little much. Yeah. So, but, but yeah, I mean, I think at a minimum, the dev teams need access to infrastructure expertise when they run into those issues. And it's really important that that's like somewhat contextualized. So does it mean that that infrastructure person is sitting on the dev team a hundred percent of the time? Maybe not, but there's some way that that person gets an overall understanding of like what's going on. Yeah. Whether that's through like ruthless standardization, they can make assumptions about stuff or through some other mechanism. I think this comes back to self-service again, right? If you look at a developer, even if they have a platform, maybe it's not a great platform, but in order for them to proceed forward, they're going to need something like expertise, right? If they need expertise to be able to operate their application, that's going to be a blocker for their self-service. So I think that if you look at it from that perspective, it makes a lot of sense.