Running the platform at The Home Depot, with Tony McCulley

Episode 131 · 46:16 · SRT · archive.org · all Tanzu Talk

Hello, everyone. This week, we have Tony on, who's always a good guest, fun to talk with. There was some weird internet connection problems. So every now and then, you'll hear some little glitchy robot sounds, and it might even cut out some of the things he was saying. But I think it's all pretty much in there. But I've left those robot things in there so that you can know when it goes, robot. And with that, enjoy the show. So I had carrots. And I had one carrot I sliced lengthwise. And then I had, we got some egg salad. Egg salad? I was going to say a sandwich, but it's just egg salad. And we need to eat that because we're going out of town for a few days soon, tomorrow. But now I put a mystery ingredient in there that I learned from my wife. I want to give each of you one guess. What is something that would go with carrots? Mystery ingredient. How about you go first, Richard? So it's not an ingredient in the egg salad, just something that goes with carrots, period. Well, it could go with carrots on its own, but I did put it into the egg salad. And it's not like pepper or anything like that. Like it's its own thing. Pickles? Pickles. That's a good guess. Egg salad with pickles is delicious. However, that is not what it is. How about, how about special guest first? Why don't you introduce yourself and then take a guess? Tony McCauley from Home Depot. Uh, so not, not a condiment, something. So not pepper, not a spice. Well, it's not a spice. It is a condiment, but it's, it's, uh, you see. Okay. Um, let's go with, um, I have no idea. No idea. You just totally give up. That's fine. Okay. I mean, if they catch up, we're, we're stopping this podcast. Oh, that's close. That's close. The secret ingredient, the secret ingredient was mustard. Oh, I almost said mustard. I think mustard's okay. Yeah. Yeah. Mustard come in all egg salad. I don't know. Well, it doesn't feel very edgy. Okay. Okay. I, you know, this, this, uh, you know, I got, I got the Albert Heine,

the sort of, uh, grocery store that is, I like to think of it, the H-E-B, uh, of the Netherlands and, uh, their egg salad is a little mild. Maybe there's like a Indonesian or Surinamese egg salad I can get. I don't know if there's mustard. Okay. Okay. You got me there. Normally I would put mustard in there, but my whole point to this story is, uh, uh, I learned about a year ago that you could actually put mustard, just regular mustard on carrots. And that just like blew my mind. That seemed just crazy. But, uh, apparently that's totally normal. In some parts. I'm gonna, yeah, I'm gonna take your word for it and try it. Yeah. It's pretty good. Like I had a, I had a, I had a Dijon mustard, a little spicy, a little horseradishy, even regular mustard. It's, it's a strange combination, but, but, uh, but it works out. It's pretty good. See, if you do, if you do the spicy mustard, I, I can do that on almost everything. So yeah. Yeah. See, exactly. That's, that's, I normally, I, I, that's what I was thinking with the Dijon mustard. It's like, I like, I like some spicy food and I have a theory that like, if you have, you know, I can only eat like one carrot before I just like am really bored, you know, like carrots really good. And then like you get to that second carrot and it's just like, oh man, I feel like I should just put a bunch of dirt in my mouth. Not that it tastes like dirt, but it's just like, there's nothing going on. And, uh, I thought if I had some, some spiciness, it would, it would, uh, I guess metaphorically speaking, spice up the carrot. That's right. Maybe it goes on your tuna pizza from a couple episodes back. Oh no, I still am not into that. That's, that's no good. That is no good. I don't, I don't think fish goes on pizza. That's well, I guess anchovies. That's it. But is that really fish? Anchovies. Yeah. Yeah. It's like the McDonald's of the fish. Yeah. McDonald's version of fish. That's true. Well, we'll, uh, we'll say, you know, I think there's a lengthy fish conversation we could have about anchovies where I think I feel like an anchovy is sort of like a chip and in that a chip is often just an excuse to eat a bunch of queso, you know?

And I don't know, maybe anchovy is just an excuse for salt, but that's, that's a category of food we could dive into, uh, on the next episode you're on, perhaps Tony. Well, why don't you, uh, once you introduce yourself, like you've already introduced yourself, but like, what do, uh, what do you work on? What's your deal? What is my deal? My deal is to, uh, so, uh, we have a bunch of developers at Home Depot who would have thought, and, uh, I think my deal is to make sure they are happy and productive. That is kind of my deal. You know, and I don't know if we discussed this in past conversations, but how did, uh, how did you end up doing that? Well, we started out, uh, running a platform team. So we had this old platform we built and then, uh, we signed a deal with you guys a few years ago and started doing CF. Um, and it turned out CF was really cool for empowering all of our developers and they loved it and they could choose the languages they wanted to work in. And, uh, I guess they enjoyed the way that we ran the platform. Uh, I think we did it a little untraditionally, uh, from typical infrastructure side of things. And then, uh, yeah, so they wanted us to do more and more things. So, um, you know, we started running some dev tool stuff, um, and now we're doing trainings and, uh, enablement stuff and now we're changing it controls. And so we're doing all sorts of cool things. Is, is, is kind of that collection of stuff different than what, what you, uh, you would have anticipated it being? Yes. Yeah, absolutely. Uh, yeah. I mean, when we started four years ago, it was, uh, very, um, like, Hey, we have some things that you need to go run and operate and make sure it doesn't fall down. Uh, and, and that was great. And we did that. Um, and then, uh, I think just the community aspect of things, we built up a pretty big Slack community. Um, and now, I mean, they just like, Hey, why don't you run the API gateway for us? Or, Hey, why don't you, uh, do something about these terrible IT controls? And so we just, we just started taking on more and more things. Yeah. You know, I, and that's, that's, uh, as, as you're going over that, I was, I was

realizing, you know, one, one aspect I've noticed it, uh, with, with groups like yourself, right. Who are the, uh, as you add to summarize what you're saying, making the developers happy and enabling them and all that, uh, like that, uh, that doesn't seem like a traditional operations infrastructure thing to do. And, and I think, I think when I've talked with people who operations, people who are running a platform or who are in that role nowadays, uh, they all, I, force is the wrong word, but I'll use the passive voice. They all end up, uh, sort of doing what you described essentially being like an internal provider and having to, uh, having to getting to like talk with people and, uh, you know, have a community around it and work more closely with developers. And many of them have that same sort of, uh, posture, if you will, that you had that, like, we're, we're trying to do stuff that, uh, makes your life easier, but that, that, uh, people seem to evolve to that position. Yeah. I don't think we set out. I mean, it definitely wasn't in the cards for us and we sort of lucked into it. Um, you know, some of it with some, uh, uh, corralling and, uh, suggestion from, you know, some of the folks at Pivotal we were working with, but, um, yeah, I mean, I think, you know, also for us, we were all software developers, like none of us had systems backgrounds or infrastructure backgrounds. And so, um, for us, it was the only way we knew how to run things. How big is this team, Tony, that that's doing this function? Yeah. So the, uh, the platform team themselves, um, I think there are seven people now. Um, uh, we have some part-time help from some infrastructure people here and there, uh, for capacity build outs. And then, and now I have three other teams. So we have, um, a DevTools team, which is about seven or eight people, and they run things like Artifactory, GitHub, um, Slack. Uh, they do some of the Pivotal Tracker API stuff, um, Confluence. Um, and then we have another team, a pipeline engineering team. So they're doing some Spinnaker work and then they're also

looking at, um, some sort of metrics and insight stuff. Um, and then we have another team that is software engineers and they're sort of working on, uh, filling all the gaps. So anything that we have to build custom, uh, to either automate some security stuff or compliance stuff for collaboration and sharing, that's another seven or eight. So there's about 45 to 50 people now, um, that work with me. Wow. That's awesome. So is that pipeline engineering team, are they helping build pipelines for specific teams or just building kind of the infrastructure patterns or models and other teams then build their pipelines for? Yeah, it's a good question. We have a, um, I think we were taking a slightly different approach. So the original approach was to sort of build one pipeline to rule them all. Um, and I think what we landed on was, um, and having done this at a couple of other companies, um, we didn't want to be too prescriptive with the CI portion of that. There's just too many languages and testing utilities and a bunch of other things. So we still let teams build their own CI environments. Uh, we help share best practices, but, um, but what we're working on now with Spinnaker is more of the CD part of it. So, um, actually getting your things into production, that's what that team is working on. Yeah. So speaking of that, so you're a, uh, you know, you at Home Depot, you yourself have been in cloud found your world for a long time, you know, good success, good stories. You've told us over the years now, I know you're doing some more Kubernetes sort of stuff at Home Depot. I remember when we first talked to you, like, I don't know, you know, a couple of years ago, I was like, you know, containers, we don't need any of that. We're doing, we're focused on apps and things like that, but it seems like some of the use cases have expanded. You're doing some Kubernetes work. Can you tell us about how you kind of see those two things living? Yeah, it's actually a really, uh, it's a very spirited conversation going on right now. Um, so we've had some people interested in Kubernetes for a while. Um, one of the things that's happened recently is, um, functions. Um, so we have a lot of teams now with, uh, SRE groups and, um, you know, a lot of these SRE groups, I think want Kubernetes and we have a lot of engineers that want Kubernetes. I've been like, well, what, what are we going

to be using this for that we can't use CF for? Um, and there are some valid use cases that have come up. So we're seeing more and more, um, some vendored software that we buy, um, that, that comes packaged as a container and needs to run somewhere. Um, and we're also looking to migrate our old platform to something a little bit more modern and Kubernetes seems to be a good fit for that. Um, but in terms of, uh, uh, we still have a lot of engineers, I think that expect that when Kubernetes gets rolled out, it's going to be, you know, you get a cluster and you get it. It's like the Oprah Winfrey of containers. You know, it's like everyone gets a cluster and I'm not sure if that's the right pattern that we want to go. So we're still having some conversations, but as a whole, I feel, um, for 80, 85% of our workloads that we would still use CF. And I still think that's the better option. So, so how do you, how are you thinking through the, uh, uh, the, you get a cluster sort of strategy? Like that, uh, like I'm thinking of there, there's a presentation that, uh, that Rabobank gives where, uh, they, they kind of go over there, like, here's, here's the, the infrastructure of the platform that we think we should provide. Like there's four types and here's like, here's, you know, and now that I think of it, he doesn't really go into a detailed analysis, but it's sort of like across, uh, VMs, containers, and like, uh, pivotal, as pivotal cloud foundry and like functions. Here's, we kind of like figured, thought about what developers need. And, and this, this option fits these scenarios for them. Uh, and kind of like you were saying, most of their scenarios fit into, uh, using CF for it, but like, I don't know, how do you put together, like figuring out if every development team should run on like a Kubernetes cluster versus be a function or run in VMs or be on CF? What's the, what's the analysis there last year where we're like, uh, what's our platform of the future? Um, what is that going to be based on? And we had a lot of technical thought leaders that were

like, uh, it has to be Kubernetes. It's the future. Um, and then I think when you stop focusing on the technology and all the buzzwords and you're like, yeah, but in terms of our engineers and how they're going to interact with this platform, what are those things? And when you abstract all that away, it's like the list they came up with. It was like, well, that's cloud foundry. Like, um, and so I think that's where we struggled is for me is that we have a lot of people that want to use Kubernetes for Kubernetes sake, um, versus like, well, what problems are we actually solving? Um, what do you, what do we want the interactions and the patterns and the day-to-day lives of our engineers to be like? Um, and I think where we're struggling is there's a lot of what I call resume building. There's a lot of people that just want to work with whatever this thing is that's eating the social media in the technology world, uh, versus, versus trying to get work done. Um, and so I think that's where we're struggling is, um, there are some valid use cases for Kubernetes and I think that want to hire, you want to the best talent at your company and they come here and they're like, yeah, I can't wait to work with Kubernetes. And you're like, well, uh, we kind of want you to sell hammers and fix like, you know, two day delivery. Um, so, so there's a, that's, that's part of the challenge. Well, you know, along those lines, uh, you know, one of the, it's almost, uh, uh, what's the word, uh, slightly like cliche at this point, but you often hear in the container in the Kubernetes world, right? That like, like Kubernetes is a platform for platforms. And I guess that means it's not a platform itself or something, but like, what, uh, what, what do you take that phrase to mean that it's like, and does that kind of fit, fit into like your, your own view of like where you would fit it into the, the platform you're providing? Yeah. I mean, that's the way I see it is. I, I see Kubernetes sort of as a infrastructure as a service, uh, plus or star. Right. Um, so I, I, if there were services or platforms that we wanted to expose to our engineers, um, you know, we would back those with Kubernetes

when it made sense. Uh, and so the example would be, um, if a team wants a pipeline, right, they wouldn't get a cluster and build their own pipeline. They would get pipeline as a service backed by Kubernetes. Um, I don't think it would be Kubernetes itself or a cluster itself. Like I don't think cluster as a service is the right, uh, the right pattern to expose to teams for the most part. Is there certain types of package, whereas Tony you're seeing, is it databases or you started to do more with things like Kafka and messaging? I mean, what kind of commercial or package software do you find yourself throwing at these? Um, well, there's not a lot. I mean, there's only been a couple, that's the thing. So there's only been a couple that have come up so far. Um, you know, to be honest, my head is, I think, um, I, I still think we're figuring out a lot of, um, I, I personally, I think it's really early and, um, what I'm trying to, uh, is figure out the right balance of, there's a lot of really smart, talented engineering and companies that are going after solving these problems and, and how involved do we want to be in that? Or, um, do we just need to play with it enough to kind of stay, you know, up to speed with the landscape so that once it is figured out, you know, we can be a fast follower. So, um, you know, to be honest, I think, um, you know, I, I've always felt that, um, you know, the developer experience on Cloud Foundry is awesome. And, you know, I think it's sort of ironic that, um, you know, the, the developer experience friendly version of Kubernetes, like might actually be CF. Right. And it's, uh, so I think, you know, I think it's more of us just preparing for the future, um, staying up to speed, but it's tampering expectations and desires of a large technology base that, that just wants to kind of like, you know, play with things and tweet things and, uh, and put things on their resume and sections and, and, uh, and not get people upset and want to leave and do other things. So that's, that's the hard part. Yeah. One of the things I was thinking about when you were describing that, that challenge is, uh, I think it was in Paradox of Choice or a book I read recently that it's tough for humans because

things lose their novelness with us fairly quickly. Like something that was super awesome. I mean, you know, the new car, the, whatever, you know, a month ago is now like, oh, that's cool. So do you feel like as you're further down this journey, some of the things that were just amazing at the beginning, whether it's getting apps to production on pipelines or CF push or some of these things now can maybe take for granted, are there things you think you take for granted because you've just been doing them a long time? What gets boring after a while? Oh, that is a really good question. Um, I, I think there are certainly things that we take for advantage, uh, take it or, uh, we don't appreciate. Um, I mean, I think scaling still is a big one. And a lot of the, uh, in the CF world, at least a lot of the sort of, uh, health checks and corrections of things that still have a lot of apps, you know, with memory leaks or issues here and there. And, and I think, you know, cloud foundries made it sort of easy to sweep some of that stuff under the rug for some teams. Um, not our most mature teams, but, um, but yeah, I think just the, I think the API itself, um, and how you can sort of abstract away a bunch of the things that engineers need to do on a day-to-day basis and to like, you know, eight API calls or something. Um, I think we take that for granted. And as we look to other platforms, even Google, for example, um, and the fact that they don't have that sort of interaction, um, it makes it really difficult to do some of the things that we've done the last couple of years with compliance and security and stuff, uh, because we don't have that, uh, that sort of front facing, uh, API proxy to everything, uh, underneath. What about that backend too? Cause I mean, you, you, I think started, as you mentioned running a platform team. So you also dealt with the good and the bad of trying to roll updates out to a multi-tenant form and things like that. Did that also mess with your expectations moving forward of what it, what does it mean? Now, hopefully if you're running something as a service entirely, you don't even see the muck of updating operating systems and things like that. But assuming you're not entirely in that world for quite a while, has that messed with your

expectations of what you want a platform to do for you? Yeah, it has. And I think actually now that we're starting to assess, um, other things like Kubernetes or, you know, data service offerings and things, we keep finding ourselves, um, comparing everything, I think subconsciously to CF. It's like, oh, there was 3000 people and we're running it with a team of eight and, uh, we do updates and upgrades during the day when zero downtime. Um, and so now as we're starting to compare to other things, we're like, well, it doesn't do this and it doesn't do that. Or, uh, you know, we could do this, but we'd need like, you know, 15 SRE teams. And, and so, yeah, it is jading. It's jading sort of our assessment of everything else, uh, to some degree, at least me internally. Um, I know that we have some folks that are like, oh yeah, we, we should have an SRE team on every team. But, um, I don't feel that way personally, but, um, our platform team has been really good. Um, the fact that they can upgrade, you know, OSs and VMs and stuff without really even having to tell us most of the time is, is pretty incredible. Um, but yeah, I, yeah, the, the, the zero downtime, you know, during the day upgrades, that was a first for us at Home Depot. That was not something that we did before. Uh, you know, you were mentioning, uh, you know, uh, hammers and fixing second day delivery, uh, which, which sounds nice. So, you know, I've been, I've been trying to figure out like, uh, linking those two things together. So you do, you do like, uh, uh, a zero, you know, zero downtime upgrade or, or whatever you might pick as sort of a platform function you provide. And how do you like figure out how that links up to like fixing two day delivery? And then further on, how do you figure out like why two day delivery matters? I mean, intuitively it makes sense, but that's, that's kind of like a corny way of putting it. But to put it from another angle, like I've, I, I keep waiting for like, I, you know, in IT, we have this phrase, the business. And, uh, I hear very few stories about how the business is like, Whoa, we've got this crazy IT function now. So let's just have them like do things

and we're not going to harass them about how much it costs all the time. Like, like we're just more going to focus on like, we want to do this new type of business. And so sure, we don't want to overpay for it, but like more of what we want to do is figure out how to provide this new thing. Right. And, uh, I don't know. So that I'm really avoiding like cliche terms, like IT business alignment or ROI or all that stuff. But like, I'm sort of realizing the, the spreadsheet driven, just like cost accounting way of doing like a business IT alignment, like that never really goes anywhere. Like it's maybe it works for like desktop operating systems, but for everything else, it doesn't seem very useful. So how, how, uh, when it comes to the, you know, selling more hammers and the two day shipping, like, what does that look like when the business people and the, the nerd people get together and try to work together on it? Yeah, I would, I would say, I would love, I would love to say that we've figured that out. Uh, the, so I think recently a big change for us is we just hired, um, so we have our own, uh, product organization now. So we just hired our first chief product officer. Um, and there is a group that's being built out. Um, and I think there's been a couple of big initiatives that have been embraced and funded, um, that, uh, one of those being sort of common services where we're trying to take, you know, 14 or 15, uh, large domains from the retail experience and sort of funding those, um, as, uh, you know, sort of ongoing, uh, things that we should always be caring about and in supply chain, uh, Hey, you know, we, we need these set of features, right. And then how do we know whether or not, um, we're, we're making money or we're saving money by doing them. Um, I can say from our part, what we're trying to do is be much more data driven. So we have all of these, you know, tools, um, uh, we have all these tools, right. So we have tracker and GitHub and cloud foundry and we have pipelines and it's, um, and, uh, another book that I read recently accelerate, or I guess not so

recently, but you know, there's, there's some really good meaty things in there, but, um, it's like, how do you know if it's, if it's true? And so I think one of the big things, one of the big things that we're, that I'm, you know, sort of driving this year is like, Hey, let's be a much more data driven organization. So we have all these metrics and rich data, but how do we know if pushing more frequently has a positive impact on our product teams? Yeah. And so, um, so we're working with our product organization. Um, they have a bunch of things that they care about. That's different philosophy of like, okay, ours where it's what I call like TDD for product. Um, it's, it's, uh, they sort of come up with these things or like, Hey, we want to move some needle by X percent or some, you know, Y value. Um, and we're going to release some things and see if that needle moves. And so I think that's really interesting. And my theory is that when we pair that up with, um, sort of these rich metrics around how frequently teams go to production, um, or how, you know, uh, how many changes they go that don't, that don't cause problems that those have a positive impact on product teams being able to move their needles. So, um, so that's the theory. We'll see. Um, there's a lot of work to do there and we're just starting to, to kick some of that off. But, uh, I think that's an interesting case study for us. Yeah. Yeah. That'd be fun to see how that pans out. Like I think, uh, to, to cite another book. So I've been reading, um, I forget what it's called the flow framework or something. This, the new book from, uh, Mick Kirsten or Kirsten, I always forget how to say his name. Uh, who's, who's like the CEO of task top, but he's done all sorts of other stuff. And, uh, you know, he's trying to figure out essentially like how to, how to apply like lean thinking and value stream maps to the entire process of a software driven business, um, more or less. And, uh, he's got this little like, um, uh, interludes of, of lucky him being at like the BMW plant and seeing how they make cars and everything. Uh, which is fun if you're into that kind of, uh, interstitial stories, but it towards the end of the book, which I'm

not quite finished with. He, he finally makes this interesting point about thinking about value streams where like in manufacturing, uh, doing a value stream is pretty straightforward because I guess you literally have a line, right? So it's a very, like, it's a very straight, uh, series of things. Now I'm sure there's, there's some nuance, but there's even more nuance. And what he does, what he says is he realizes that if you look at all the activities to get software out of the door, and I don't know if it's him, it's like two other people he was working with kind of brought this to him is that it's actually just like a network of things. And so like figuring out how to not only like optimize the way a network flows, but I think back to the point I was meandering about like how to find causation in a network can be very difficult. Right. And so it's, yes. And, and so like, it's, it's making me think that's part of the issue we always have is like intuitively, we know it'd be better to ship software more frequently, but then like, if you do the quarterly, like, you know, how many hammers did we sell? It's it, I don't know how you would do it, but it seems like it might be hard to say like, well, we ship software once a day for this last quarter. And therefore we sold 50% more hammers like, and, and like, and, and maybe wanting to answer that question is like the wrong question in the first place, but I'm pretty sure it's somewhere in there, right? Like somehow that the time and money spent on increasing releases has to link back up to something like that. And I have no idea how that happens in the real world. Yeah. I don't think it does. I think I like that answer. Move on to the next quandary. Yeah, exactly. It's a, yeah, I don't, I don't know if anyone's figured that out yet, but I think ideally, I think theoretically, if you break down these metrics and they, they sort of create a tree of metrics. And so I think as long as the app teams are able to move needles at their level that move needles above them, but it's all theory. Um, and I think, uh, I think the more important thing

is capturing, um, what doesn't work. Right. And I think if you're doing these fast feedback loops, I think what we need to celebrate more often is the things that we didn't invest in that we learned was the wrong thing to do. And I think that, that bringing that to light is as valuable or more valuable. And I don't think we put enough focus on that. So, um, I think there's something there, um, and, and celebrating that sometimes. Yeah. No, you're reminding me. That's another little thing I had squirreled away on this topic is, uh, and, and I liked the way you put it. Like if it's, if it's impossible to show positive causation or not impossible, if it's not, if it's not profitable, so to speak, right. It's, it's not worth your time. Uh, maybe something good to do is, is to avoid things blowing up. Right. Which is, is kind of like a cynical way of looking at it. But I always thought that was one of the better, like, uh, sort of rules of thumb from the, the black Swan guy, Nassim Taleb, which is like, success is great, but what's really going to cause you to not succeed is blowing up. So like, if you can avoid over time, if you can avoid blowing up, you'll be successful, which, which, um, I, you know, which that, I think that's, that is a, again, easy and hard is the wrong word, but that's a much more profitable exercise to go through. Right. And that's why having stability and all these things is good. And not that you shouldn't try the other thing, but it definitely, uh, it has a clear effect. Yeah. Not blowing up. What's the, uh, I mean, so in that, what's the hardest part today about being, getting to production, Tony, what we struggle with is, uh, you know, honestly, just this whole socks thing, man, if you, if you got rid of socks and, and, uh, PCI and compliance, you know, or, um, uh, or, or found some ways to, to, I think, uh, align your company differently so that those things aren't as big of an issue. But, um, yeah,

I think most of it is just us kind of getting in our own way. We have a lot of process and, and, uh, in regulatory things and, you know, that's, there's a finance group and then there's a, there's an audit group and then there's, you know, a technology group and a security group and they're all getting pulled in different directions. Right. And it's, uh, how do you kind of get them all on the same page that, Hey, our goal is to, you know, uh, release software more frequently so that we can figure out what works and doesn't work for our business faster. So, um, getting everyone on the same page about that mission, I think is, is the hardest part. You know, we, we, we kind of scheduled doing this, this talking when we're thinking about, uh, SpringOne platform talks and various stuff that y'all have going on. So y'all have been working on that problem, uh, of automating compliance and, uh, tell us about that. What's, what's that looking like? Yeah, I think, um, we thought it was pretty cool at the time. So a couple of years ago, uh, and it's, we've been iterating on it for a while. Um, what we did, I mean, so I think one of the things that we do differently, um, well, I'll, I'll, I'll just be transparent. So we, so we had an audit internal audit. Um, and it turns out that our, our controls were, you know, just do the right thing, right? Have it, have a change record for everything. And it turns out that, um, when you just do that, there's a percentage of your folks that don't do it. So, um, so it turns out that, uh, you know, we had a handful of teams that were not doing their change records. And so, um, you know, our, our leadership sort of removed everyone from production. So for us at Home Depot, we had people as space developers in production cloud foundry. Um, so we, you know, we didn't do anything to block their production access. And so we had teams, you know, pushing things to production without change records. And, and so we had to remove everyone from space developer, which caused a bunch of problems. We had issues around GitHub access and cloud foundry access. And they were like, Hey, if you're a developer and you can write code and cloud and GitHub, then you can't push your own code and cloud foundry. Um, I was like, well, you know, the whole DevOps thing is to, to write code and deploy code. Right? So what we did is we came up with this thing. Um, we called it locksmith. We put it, um, essentially what we did is we proxied

all of the requests coming into the cloud foundry API and we allowed teams to do whatever they want, as long as it wasn't, you know, something that would change code. And then, so for those API calls, we would go through and make a bunch of service calls to make people feel comfortable that they had done all the right things. So we would look up change records on service. Now, uh, we would see if they've scanned their code and tested and a bunch of stuff. Um, and that worked, um, uh, except that we still had an issue with this sort of segregation of duties concern. Um, have you guys heard of segregation of duties, by the way, is that like a thing? Any, Oh yeah. Yep. All right. So I just want to make sure it wasn't just us, but, um, so to, to get around segregation of duties, what we did is we said, Hey, like, um, it doesn't really matter who pushes the code as long as they can't do it themselves. And so instead of, uh, separating, um, for us internally, the people that write code from the people that push code, uh, we said, Hey, like, uh, anyone, any code that gets deployed has to sort of be approved by two people. And if it is, then, then you can't go to production by yourself. And so we redefined segregation of duties for us internally. So, um, so essentially what this locksmithing does is it, is it looks at the artifact being pushed, make sure that multiple people have signed off on it. Um, so that's how we got around segregation of duties. Um, it's essentially a route service for the cloud foundry API. Um, if that makes sense. Um, but, and we've seen, and so now everyone has production access again in cloud foundry. So everyone's back to being space developers, um, in production, which is, I think, I think kind of rare, but yeah, that's, that's interesting. I mean, I mean, true to the description, you, uh, you, you sort of automated the enforcement of compliance, which, which, which, which seems, uh, seems like it would be better than sort of, uh, well, as you were saying, it's better than having sort of procedure that people need to

follow. I mean, I guess, I guess you just lock down the thing and don't, don't let them do it, but like, it's probably even a richer source of, uh, compliance information, hopefully, and less tedious, which, and I was just thinking it's a, that's a great example of how like, uh, software is a network instead of a factory line, right? Like having to go out to service now to check something and look at all these other things. There's like multiple, like spaghetti bowls that you have to go dig around in and not just, uh, not just sort of like a compliance person sitting there on the line. Yeah. I mean, me personally, I'd rather us not have any of that and just, you know, hold people accountable. And, uh, you know, it, the, the joke I make is if you get audited and there's no one doing anything wrong, do you need it controls? And then, okay, this is what you should do and shouldn't do. Um, I think the problem is like, as a big company, what's the right mix of, uh, like, Hey, we're going to prevent you from doing a couple of things, uh, to show that we're not negligent. But, um, you know, ideally I would just, I would wish that we could just kind of, uh, hold people accountable so that you need less process, but that's hard to do in a big company. Yeah. Yeah. Yeah. There's probably some chart that shows a size of company and amount of process required. You know, years ago when you presented at different conferences, you talk about apps you've got in prod or how many, how much you're pushing. So even amongst all this, is there a stat now that you're proud of? Like even with all the craziness of pushing to prod and using different clouds, what's that thing that is celebrated internally right now? Yeah. So I think there's a few things. So, um, I think in production, you know, we're doing, um, uh, we do 17 billion HTTP. Um, and you know, we're still around 400 to 500 production changes a week, um, in production and of those, the fail rate. So, so now with locksmith, we can see, uh, the code changes that are people making. Is it, is it based on a change or is it based on an incident? Um, and the changes made from incidents is that, uh, less than 1%. So, um, if you look at those, you know, 400 to 500

production changes each, um, 0.6% of those are due to incidents. So, um, yeah, so there's, uh, um, some really cool things going on. And you, uh, you garbled out there a little bit. How much, what did you say at first? How many deploys I think you do? Uh, we do four, yeah, 400, yeah, 400 to 500 a week in production. Yep. And the other number I think was API requests, right? I think you were saying, yeah, there's 17 billion, uh, service requests on each month and production. Those are some big numbers. What, what, like, what are, what are those? Like when it comes to, like, when you're, when you're deploying all those times a week, like what are you actually deploying and what are those calls for? Like, what are, I don't know what, what happens that frequently? Yeah, that's a good question. I don't think we know anymore. Uh, I mean, you know, four years ago I used to know every developer that was on the platform and, you know, would talk to them, you know, for two to three hours a day. And, uh, I mean, today it's kind of taken on a shape of its own. Um, uh, our Slack community, um, you know, the cloud foundry community that we have on Slack internally is, you know, by far probably one of the most, the busiest, except for the one that tells you where free food is. I think that one might be beating us, but, um, the, uh, yeah, the Slack community has been great. I mean, we, our team doesn't really even handle a lot of the sort of newbie questions or, um, you know, some of the, the first level of support questions. Our community does a lot of that for us. So, so finally, um, you know, like you're saying you have a, a chief product officer and, uh, you're, you're, you're helping make the developers lives easier and so forth and so on. All the great things like how, uh, like, like on your, in your group, how do you like product manage what you do? Cause I assume that's part of it, right? In the same way that a, uh, a chief product officer of, of Home Depot comes up with what their customers and I would imagine suppliers

and other people, but you know, you've got a product management going on, uh, in the business side, but do, do y'all do that, uh, for the platform that you run? Yeah. So that's what's new. So that's sort of what I'm heading up. So we just hired a couple of product folks from this product group. Um, and we're looking at our engineers as users. So we're starting to do formal, uh, sort of UX research around that. Um, we're creating journey maps for the different demographics. So another big thing that we're seeing is whether or not the teams are in scope for socks or PCI, that seems to be a big factor into sort of their workflow and journey map. So, so we're doing a lot of formal research around, uh, what the different teams need and what their pain points are and their processes and stuff. And that's interesting to have a, uh, uh, speaking of segregations, like a segregation of compliance complexity, right? Like this, this profile of things needs to deal with all this stuff and this other one, you know, just do whatever, I mean, responsibly, but, but that is, I mean, that's a, that's an interesting way of like dividing things up. So how many, how many, uh, like teams do you have working on apps? Oh, that is a good question. So I think, um, um, we did go through a big hiring spree that you guys may have heard of. I think we ended up hiring 900, uh, engineers last year. So I think, you know, um, for, for aprons as we call them associates, I think we're up to, um, and I think there's about 2,700 of them on the platform. So, um, and we've had a couple of acquisitions and stuff, but, uh, yeah, I mean, it's a, it's, it's a large group. Um, uh, I don't remember what was the question again? Oh, just, just like, you know, just to get a sense of like, so you have all the, the, uh, the astounding numbers of, of kind of the, uh, velocity or movement each week, right. And API calls and deploys. And so like, I mean, how many like applications and therefore how many teams and sort of how many people that

equals, right? Like it's, it's always interesting, um, to think about how much, as you were kind of joking about earlier, how much software development large organizations actually do on their own. It's sort of like, right. I always, I was still, after everything that's been going on in the last five years, like it always seems like the, uh, the sort of hidden pool of software developers out there. Uh, so I'm always curious for a sizing of them. Yeah. So I, I don't know the number of, you know, in terms of CF, uh, things we have 4,000, over 4,000, um, services and production now on CF. Um, and then from teams, um, you know, we, we see some services on there that haven't, but then we still see new things being added all the time. So, um, yeah, I don't know the number of teams, but, uh, we're up to 4,000 apps or services. Yeah. Um, that's, that's a lot from like a couple of years ago, if I remember. Internally, um, was that, um, in terms of, of adoption, we saw nearly double the adoption in the last year. So, um, so that hockey, that, that little curve, um, went up a lot last year. It's starting to smooth out a little bit. So, so like, what, uh, like, like, what do you see being kind of next year? Like what are the, what's the next set of things that you think you'll be working on? Like, do you, do you get the chance to sort of like dream up things that the platform team might want to do? Or are you still like too busy, uh, you know, making sure all the pipes aren't bursting, keeping? Yeah. Um, so our team, I think there's a lot of, uh, there right now, uh, one of the big things that we're looking to help out with is, um, this sort of hybrid cloud thing, right? Which I'm not entirely sure what that means still, but, um, but we do know that we have some things that we want to run, you know, in Google or Microsoft, and we have something that we want to run on prem for lots of various reasons. Uh, but we're not

good at, um, sort of that connectivity either from a networking perspective, um, openly. So I think, uh, or seamlessly, I should say. So I think there's some work to do there. Um, I would say that, um, to, to sum it up, I would say that we've sort of been lucky that we have this big firewall around everything that we do on prem. And so I think that allows us to, um, take certain liberties, you know, or like, Hey, we're behind a firewall, so we don't have to be, you know, really stringent about this or that in terms of service to service stuff. Um, but as you go into the hybrid cloud world, um, and you're sort of going from on prem to the public and then back, uh, there's some, you have to think about those things. So, so API management and sort of, um, and security things are, and, and also being data driven or some of the, the big, probably two top two to three things that we're working on, uh, over the next year. Yeah. And that, that seems like a fruitful area of, uh, seeing if there is a, a, some, some good useful ways of doing like, you know, business to it value mapping stuff in the sense of kind of like you were saying is like, well, if, if everyone around here says we need to do hybrid cloud things, then they must have a reason or they must have a reason is the wrong way of putting it, but like, it will allow us to like explore the reasoning that comes behind, like, we must do this thing and, uh, maybe kind of surface up the type of analysis and sort of, uh, rating you would do for things. And then maybe you can apply that type of, of thinking backwards, but, uh, cause you know, at the end of the day, it might just be sort of like I was talking about earlier. It's just like an intuitive benefit that you just want that, uh, without it being sort of like excruciatingly analyzed with numbers. Right. Well, I think it's interesting is going back to that whole ROI thing. I think for us, we're looking at it slightly differently as well. Um, for me anyway, um, I definitely think there's a business side to that, you know, pushing two times more frequently to production, sells more hammers. But I think what we're interested in learning is,

um, uh, Hey, like, does the tech stack that you choose, how does that impact or the size of your team? We're trying to look at is, you know, what is, uh, you know, if you're in, you know, app engine or CF or Kubernetes, how does that impact your team's ability to deliver? And unless you have a really good business case for choosing one technology over the other, let's be very data driven about the decisions that we make, uh, from team sizing and silos and, uh, and the tech stack choices and stuff. Yeah, no, that makes sense. That, that, that's, that seems like a good, uh, uh, is it a solve or a bomb? It seems like a good way to address like a, uh, data driven question, like, like finances is, uh, get some data. Right. Well, when you have a bunch of people in the room and they're like, Hey, we should go Kubernetes and we should go this. It's, you know, um, instead of just sort of debating I'd rather just like, well, let's just look at the data and let that decide. Um, I think that's, that will be more fruitful. Yeah, yeah, for sure. All right. Well, great. Well, uh, thanks for coming back on and, uh, uh, checking in on us going, going through stuff, or I guess we checked in on you, but whatever. Uh, if, uh, if people wanted to, uh, see what you're up to online or, you know, do you get a Twitter account or any other things you would point them at for a further study? Yeah. Yeah. Um, it's, it's mostly dad stuff, but, uh, you can check me on Twitter at, uh, epic nerd, um, on Twitter. So, and, uh, and LinkedIn, uh, just Anthony McCauley on LinkedIn. That would be great if you were just epic nerd in LinkedIn too. I think that would fit well. Just own the name. Yep. Yeah. Yeah. I think, you know, maybe, was it about five years ago that there was some like, uh, data absurdist ism in, in LinkedIn when they, uh, introduced endorsements and people would get endorsed for like, you know, hot dog eating and things like that. Yeah. I mean, I just endorsed you for mustard on, on carrots like an hour ago. Carrot condiment innovation. I, uh, yeah, those things were hilarious. I never, yeah. Yeah. I'm going to add on the

Coté and Richard that I'm glad to both of yours. That's good. You know, when I, when I enter my, I don't know if it's third or fourth career as a, uh, consumer packaged goods and condiment, uh, marketeer, these are really going to pay off these, these endorsements. That's going to be good. That's right. And then, and then, uh, I might write you, I'm going to write one for you. Okay. I'm going to look forward to it. And are y'all, uh, are y'all still hiring or is 900 enough? Are you still trying to find trash cans for them all? No, no, no. We're definitely hiring. I think, uh, I think we're filling up in Atlanta, but they're still hiring, uh, folks, anyone interested in working in Austin, we still have, uh, some openings there for sure. Oh yes. That's a good, that's a good place to go work. All right. Well, great. Well, uh, thanks again. And, uh, as always, this has been Pivotal Conversations. If you want to, uh, I guess you're already listening to this one, but if you want to go check out some of the other episodes, if you go to soundcloud.com slash Pivotal Conversations, you can see them there. And about every Thursday, if I remember to click the publish button as I did not last week, but eventually did about every Thursday, uh, you can see the full show notes over at pivotal.io slash podcast. And we'll put some links to some other relevant things and, uh, and stuff Tony has done as in the past. And with that, we'll see everyone next time. Bye-bye.

Tanzu Talk · 🍕 Get more pizza.