Step One: Build a Pizza Factory (Ep. 43)

Episode 30 · 52:06 · SRT · archive.org · all Tanzu Talk

So I'm up here with a big hunk of ham that I'm eating, which made me think, so at least for people in the U.S. or people who like to follow the United States sort of traditions, what are your tips for the post-Thanksgiving warm-up exercises to get back into the swing of things, doing work? And let me end with a statement, as I like to do. I find this a challenging part of the year for at least two reasons. Well, I'm not in sales, so it's not because I have to finally go get everyone to buy. If I remember, all great deals are made at December 31st at 11.59 p.m. I think that's what happens then. So I don't have that to worry about. But it seems like you've got this issue of Thanksgiving, and then there's the, as we say nowadays, holidays, the holiday break, and you've got this weird little donut hole of supposed productivity in the middle. So, like, what do you do to motivate yourself? I mean, other than being a responsible, professional human being, I assume. Naturally. Well-paid professionals who should take six weeks off. Yeah, no, I mean, I think it's... The German solution. You and I have the, I think, the Gartner Conference next week. So sometimes it's not a bad deal to have a conference in this season. I think it might keep you a little focused. No, that's clever. That's clever. You're sort of, like, stacking the deck in your favor. Right. Yeah, so I think that's helping me. You know, and that's also a time of year to, a lot of companies have shipped out, and we'll talk about in the news, some cool new tech. So, even if your day-to-day work isn't super inspiring, go, you know, disappear for a day and try something new and get some inspiration that maybe you'll bring into your company next year. So, I try to do those things. I just wrote a giant blog post I put live this morning about, you know, PCF on Azure and Concourse and spent a lot of time on it. It was fun, but it was something that kept me inspired over the Thanksgiving when the kids were too noisy. No, you know, you raise a good, a good, it doesn't really address the question, but it's sort of a tactic.

I had a friend, well, I guess he's still a friend, an old architect friend, this guy Chip Holden, and he used to kind of relish this time of year because so many people would be out of the office, you know, on vacation, and that he would, like, be able to sit there and get a lot of work done. So, he would sort of, not that he would delay his stuff, but, you know, he would plan his stuff around that, which seemed like an interesting tactic. Now, this reminds me, yeah, we are going to be at the Gartner Application Strategies and Solution, which if you do the initialization on that is a slightly tragically named conference, but never mind that, next week in Las Vegas. Now, I was just, hopefully we'll have a few live to tape from recordings there. We are going to have Rita come back to tell us what's going on in the analyst world. I thought that would be apropos or apropos. I don't know how to pronounce Latin. I was not trained in that. But most importantly, you'll be there, right? I don't know if our buddy Andrew Schaefer will be there or not, but I see that it's still at Caesar's Palace, which means we need to schedule going to the, I don't know how to pronounce the French person's name, despite my name, but there's this French guy. And he has a restaurant in there, and there's a little hidden part in the back where they make crepes. I think I said that right. Very challenging episode for me here. But they have, they have a, they have a, a, a, a, a, what do you call the eggs with hollandaise sauce on them? Not Bernays. It's a. No, I know. That's the sauce though, isn't it? The Benedict. They have, they have, they have, they have an eggs Benedict that has lobster in it. And, and I think, I think you and I, and whoever else we can recruit, we need to go order that. Because I, I dream about that all year. And I've tried to recreate it other places. I went where we had the SpringOne platform conference was in the, um, the ARIA, I think. And they had one of these Jean-Philippe French guy, uh, things there, but they did not offer that. It was quite the letdown. The rest of the conference is wonderful and great. Lots of recordings on YouTube. You should check out. But that one item, they, uh, the ARIA could have done better. But this next week, we'll, uh, we'll get one.

And, and listen, if you don't want to eat yours, just order one and I'll eat it. No, I mean, I'm excited about the conference and now eating this food with you. It seems like the tagline should be, let's go make some mistakes. So come to Vegas. Some creamy mistakes. That's right. Uh, well, speaking of the opposite of making mistakes, I think you wrote this up, right? That we did some recent performance tests with, uh, Pivotal Cloud Foundry. Was it Cloud Foundry or Pivotal Cloud Foundry? We did this in concert with the foundation-y folks. So we definitely had some, some help. We, I think we led the effort, but worked with IBM and others to get, uh, one PCF or one CF environment with 250,000 containers sustained in there. So that's a big milestone, not just for the kind of numbers porn that people like to talk about, like, Hey, look at our great stats, but it was also the, the nature of the test was really good for us. I think it was, it wasn't just like, Hey, let's crank as much as we can into this thing. And then shut it off and just prove we did it. It was, Hey, let's actually try to see what it means to sustain load. And what does it mean to deploy more apps when you already have 200,000 in there? And let's run this for a week and let's make sure all these apps are crashing. There are many of these apps are constantly crashing because that's real life and how the system handles it. So it was just a really good sustainable test. We proved this at scale. It ran on the Google compute platform or the Google cloud platform performed great. All of our test material is publicly available. So anyone could recreate this if they have a few thousand VM sitting around. Now, do you, do you have a sense? So those are AIs that we're running, right? I'm always, I'm always looking for a ratio of application to AIs and I've, and I've, I've done lots of research and my conclusion is basically like, it depends, which is always the most helpful and helpless answer. But like, you know, when, when you look at these, like how, in your mind, what does that kind of represent as far as a range of applications? Yeah, we did. So that was the other part of the test I really liked. So we did batches of 25 instances made up of 20 apps, if you will. So you had 20 different types of apps, you had 25 instances.

And so these apps had different profiles, different memory usage, different storage. Some of them had different outbound network rates of when they would invoke external web services, you know, different crashing ratios. So many of these apps were going to crash every 30 seconds, whatever it was. And then we would then 10,000 of those batches in of 25 instances to get to a quarter million. So the idea was to try to not just make this completely contrived, like, hey, here's a bunch of tiny hello world, no JS apps. Hey, look what we just did. Like nobody, that's not real life. Nobody runs that. So we did want to make sure that multiple app instances, different kind of memory consumption, you know, those sorts of things that all of our listeners, our customers deal with, which is, you know, big apps, little apps, a lot of instances, only a couple of instances. You want to make sure that the load balancing works. You want to make sure the DNS updates are fast. You know, that's what has to happen. And none of our customers are running quarter of a million containers yet. We're trying to be well ahead of where our customers are at. But hey, we want to make sure we're staying well ahead of that. So when they start hitting these limits, we're proven at that point. Yeah. So also in Foundation News, they have a new executive director. I think that's the title she's taken, right? Instead of CEO. Grand Poobah. Grand Poobah. So we have Sam Ramsey. He went to go work at Google on their cloud stuff. And then Abby, Abby Kearns, who she, I actually worked with her when I was first here. And now she's the executive director. She's great. She's very enthusiastic about all this stuff. And generally, like all of us, knows what we're talking about. Maybe even beyond generally. She just plain knows what she's talking about. I generally know what I'm talking about. Yeah, we're excited about that. I mean, it'll be good to get, I think, her and Chip, who also took, I think, a CIO or CTO role in the foundation on the podcast. We can harass them a bit. But it's great. It's showing that, look, I mean, it was something James Watters tweeted out that, hey, people who are associated with Cloud Foundry have great career choices. And Sam just took a great leading position. A lot of our customers keep showing this.

A lot of people at Pivotal and others and partners that, hey, when you, you know, bet on a technology that helps change businesses, that seems to do well for you professionally. So Abby's doing great. Sam's doing great. The foundation's super healthy. It's great to see the sort of evolution. And it's always good to have a good bench, right? You never want to have a person leave and things crumble. You always want to know you have an awesome succession plan. So in this case, they did. Yeah, you know, she, we were at, I forget which conference it was, but she was here in Austin for something. And she, she walked, I live in Austin and she walked me down to a coffee shop I didn't even know existed. It was amazing. It was just like, it was basically through a, is copse the right word? Through like a copse of bushes and trees, like, like right on the water, there was this deck coffee shop. And, you know, she's from La-Di-Da, San Francisco land, where they take their coffee really seriously and do all sorts of stuff. And she seemed to approve of this coffee. So that was great. And, you know, also I, I went in, in reading about this transition, like, uh, I didn't realize, well, I didn't realize, I didn't think about the, uh, the Cloud Foundry Foundation being around for like two years. I've had my, uh, my head up in slides so much that I haven't been paying attention to the timeline, just the, uh, just the light box view. And, and it is like, uh, man, it's been around a while. And I think, I think a little, you know, a little to what you're alluding to a bit, like they, uh, I think they've done a good job being like the, the independent, uh, foundation to look over the code and, uh, sort of shepherd all the people working on it. It's, uh, it's worked out pretty well. I think it's nice. Yeah. Those things aren't easy and we'll talk about them more when we get them on the podcast, but, you know, foundations can either be a boat anchor on your progress or they could be help be a good facilitator. So I think in this case, we've, we've proven it to be a pretty good facilitator. Yeah. Yeah. And, and, and for a long time listeners, you might remember that we had, uh, we had chip on a long time ago when, uh, when Apache Con was in Austin and we were overlooking his, uh, his Hyatt view. Good times. Good times. So, uh, so there's also like, like, uh, there's a bunch of Microsoft.

It seems like maybe it's just with selection bias because you, you like Microsoft stuff, but there always seems to be a lot of Microsoft stuff each time we record. Like we have, uh, we have pivotal cloud foundry supported on Azure and, uh, and, and then, and then there's sort of more evidence of, I don't know what you would call it. Like, cause .NET is mainstream, but maybe even going more mainstream that you have Google joining the .NET foundation, which is sort of, uh, you know, it's, uh, we're, we're way beyond cats and dogs. We're through the cats and dogs looking glass to mix metaphors, like, like Microsoft is sort of like normal. Yeah. I mean, it's bizarro world now. I think we've just accepted that that's the case with these sort of things. So, yeah, I mean, you've got visual studio for the Mac, you've got Google and the .NET foundation. You've got, you know, a lot of things going public, as we mentioned, you know, a couple of weeks ago, PCF officially supported on Azure and all their regions, all their fun stuff. So, yeah, you're, you're biasing the audience by saying that somehow I'm a Homer, so I appreciate that on Microsoft news. But yeah, no, they, they, we've, you know, we're doing good stuff with them. You know, we spend a lot of time with that team when it comes to enterprise customers, there's arguably nobody who has more great relationships than Microsoft does. So that overlaps a ton with who pivotal works with. So that's great to see. But then, yeah, sure, industry wise as a whole, they've been doing a lot of interesting things, clearly in open source and trying to, I think, get rid of that reputation as this kind of aggressive closed source company. And, you know, .NET has had a renaissance of sorts and .NET Core is now cross-platform and, you know, just like arguably Spring Boot with Java, kind of resuscitating that from maybe becoming a legacy tech, .NET Core and .NET Foundation and open sourcing, I think has made .NET back to being a true first class citizen, even for people doing startups and not just, you know, the, the enterprises who have a lot of existing apps. I think it's new people looking at .NET as a great choice. Yeah, you know, it reminds me, I was, I was talking with someone over the weekend who works on big, big, I don't know, enterprise IT deals.

And, and they, they were saying, you know, there's this, they, they were in this kind of wait and see thing of like, it seems like there's this wave of just like Microsoft stack stuff out there that just has to sort of not metaphor, well, metaphorically be turned on for people to have access to doing it. And I think not really knowing Microsoft strategy intimately at all, just from what I observe, like it does seem like there's this slow but steady incremental adding of having basically the Microsoft stack supported everywhere, including Azure on its own, which will allow a lot of the, you know, I'm sure I could go to IDC and maybe quantify all the .NET sort of installs. But it seems like there's a lot of technology on that stack that if it were allowed a place to, we'll have to come up with a new term rather than like forklift, but sort of like maybe white glove lift, just sort of moved new stacks that it would pretty easily, to use another metaphor, just sort of like a sleeping giant can kind of roll over and, you know, crush whatever's underneath it. So we'll see. Yeah, they're trying to be everywhere. I mean, they demonstrated SQL Server and Linux. You know, they're showing that we want our software running everywhere if you're Microsoft. That seems to be the thing that they're big on right now. And, you know, I think everyone's willing to oblige. Google loves, I'm sure, Microsoft workloads. Amazon, I think, is one of the biggest payers of software licensing to Microsoft that they have because of all the Windows instances. So, you know what, the more places they run, the more company, the more money the company makes. So then here's one. Hopefully you can explain to me. So we had Oracle last week. This was last week, right? They announced they were acquiring, is it DIN or DINE? DINE. Yeah, I'm not a networking person, as people know. But like, so they basically, it's like a DNS service, right? It's like the Akamai of DNS. Yeah, it's DNS plus and a really good company had that giant denial of service attack that seemed to take the Internet down in October. So that's how most people all of a sudden realized there was a company called DINE that ran a lot of the Internet.

But, but like, what, what do you make of that? I should have a whole podcast one day called What Do You Make of That? But like, but like, that doesn't seem like a core Oracle business. Well, I think if they're, I think if they're saying we're in on cloud, that having DNS is big on cloud, you know, Amazon's got Route 53 that does their DNS service. Azure has some DNS services. So I think to some extent, this is saying, look, we want to have the whole path from when you type in your browser saying, I want to go hit this website to resolution to hosting it on our Oracle cloud and data store in our Oracle database. So it's an interesting buy. You wonder if they got a discount post denial of service stuff. But it's a, it's an interesting buy. I mean, you know, I don't know how successful that is, but it shows. I have a, I have a recording of my other podcast. Well, the other main one, software defined talk later. I'll see, I'll see if I can get those two guys' opinion on it. But it seems like I can see that there, there would be this, this corporate strategy deck that you had. And you might have a slide that's like, on the left side is like the old middleware stack and dollar amounts next to Oracle's take on each of those. And then the new middleware stack, right? And I could see that something like that would fit into the new middleware as far as transitioning over. Like, like you're saying, it's a, as we used to call it, a control point that you have in all this stuff going forward. So sure. Sounds groovy. Yeah. Exciting. So then next week we'll have to see, we'll have to talk about what happened in, at reInvent, the big Amazon confab. I'm sure there'll be all sorts of exciting stuff coming out of there. It's always fun to see the, the sort of customers that they have and what they're doing and what, what new services and things Amazon comes out with. But you got any, you got any predictions, anything that you're looking forward to, or is it just going to be lots of, lots of lobster eggs Benedict late to the night?

Yeah. I mean, there's been some little leaks here and there of stuff. Again, I don't think price war stuff that used to be the hallmark of some of these conferences was, Hey, we're slashing prices. And I went to the first reInvent back in 2012, which was great, but it's also gotten to be a really big conference. So you just wonder, you know, does the tone change? Does it become, you know, a little bit more of a convention versus just a conference and, and that sort of thing? Or can you keep some of the customer focus and technical depth or do you start to generalize things? So we'll see in the conference itself, what the vibe is, but technology wise, you know, you'll probably see additional partnerships, additional new services. Of course. The questions just will be is where do they continue to encroach on partners and other cloud providers as we're still in this race war between, you know, Google and Microsoft and others and making sure that who has the coolest features, who has the best software, who's innovating the most. And so everyone's in this race. And so it's, it's fun to watch, I guess, as an observer, it's got to be stressful if you're one of the other cloud providers, because you're always one upping each other. But I think you're going to try to continue to see them raise the bar because there's been enough chatter over the last year or so. It's, Hey, is Azure catching up? Hey, is Google about to make a play? Is Amazon going to get toppled? And you wonder if this is going to be them putting the pedal to the floor and stretching the distance again. Yeah. It's always fun around, around reinvent to see all the, the hand-wringing articles and like major publications about what does it all mean? Like that, that's always, that's always a good effect too, to kind of see how it's percolating out. Like that'll be my podcast. You'll have, what's the deal with that? I'll have the other one. It'll be from the, the what podcasting network, the, the five what's, but yeah, you know, it's a, see, you, you, you, you, that, that was so entertaining. I lost my segue. It's terrible. Well, you're talking about the kind of, you know, there's going to be all the pontificating about what, you know, what does it mean to competitors? What does it mean to customers? What does it mean to Amazon as they continue to, to evolve as a company?

So you'll see a lot of talking head thought pieces, arguably by people who haven't used their technology ever, but that's, that's a different topic. That's fine. I like to comment on stuff I've never used ever that I'm not an expert on. That's a, that, that always works out. Well, so I, I thought, uh, we don't have a guest this episode, as you can probably tell. I think we'll, we'll have some, uh, next week, like I was saying, if not just ourselves. And then we've got another one scheduled to talk more about containers, uh, after that over the next, like two or three episodes. But I thought we could talk more about like, uh, some process stuff and conveniently things I have written. So I don't have to think that much. It would be very exciting. Uh, but there is a sort of a means of a segueing. There was a, uh, I forget when it was sometime last week. There was, there was an article written up in the, uh, New York times that had, had, I think it had, it was, I don't know if it was only one, but it had one of our, uh, off-sited and, and, uh, more publicly speaking customers, all state. There was, uh, Doug there who spoke at our SpringOne platform conference, but it's an interesting article. Uh, you know, I, I, I remember, uh, when, when, when Richard was suggesting it this morning, I remember like, I remember reading this and thinking it was a little weird. Uh, so I went back and read it, but it really, it's, it's only weird because it discusses several things and, and it fit, it would fit well in our, uh, our, uh, mythical podcasting network. Cause it's really seeking to explain, I think, like if, if you look at how cloud technologies are, are driving a new way of thinking about how you, uh, innovate and do it more specific, not more specifically, but mostly how you do your custom written software. There's a little bit of SAS stuff in there as well, but it's sort of looking at a collection of, of how new technologies and new ways of thinking about using them are changing the way companies operate and, and how they go about doing things. Um, and there's a little bit of like, uh, effect in there, right? Like there's the usual hand wringing over like, uh, uh, uh, work-life balance or mix as someone at, at, uh, at chef calls it, but it's a, it's a nice brief take sort of like a dipstick into the state of like corporate it at the moment, which, which I think is, is, uh, worth looking at.

What was your impression of it? Yeah. I always like geeking out over just seeing, you know, our customers in public forums, being able to talk about their success, right? I mean, I think arguably we all work at places to make some sort of difference. So it's fun to see companies like all state transform in part with help from pivotal, but clearly they've powered this a lot on their home with their own talent and skill and folks. So I know, again, it was a good take of that. The, you know, the old way isn't going to help these companies adjust to a new reality. And it's, it's cool to see companies start to adopt that, not wholesale, right? And, you know, Coté, you spent a lot of time with them. It's like the whole company is doing this at the moment, but they've done a nice job incubating it and they've been starting to spread that. And I think that's a really nice pattern for people who still are super intimidated by trying to become cloudy the way they do. Yeah. I was, I forget if I mentioned this here, but I was up in, uh, I was at the, the, the Chicago pivotal labs office where they have, they have some of the, as all state calls it, composed people, they even have a big sign. And, and I didn't get to see the, the main floor that they have there, just the, the sort of part where they started off in labs. And I also talked with several other customers and, and the, the, uh, the stories of, of how these large organizations are adopting agile are pretty similar. And, and it's, it's only a couple of paragraphs, uh, where Doug is talking in there, but like his sentiment is pretty much what everyone, uh, goes over that we talk with. And, you know, it essentially, and I was, I was talking with someone over the, uh, over the Thanksgiving break about this very topic where, uh, you're in an organization, usually a large one. And you have all this for lack of a kinder word bureaucracy, just all this governance and stuff that you have to go through just to like get software out the door. Like the, the, the, one of the conversations I had recently, there was, I kept kind of needling at it. And the other person was having kind of that delightful fun, like, like you can't buy pushing on a bruise or picking out a scab or whatever. It was sort of like, so what you're telling me is that there's a big crisis going on because someone who's monitoring the process hasn't been documenting the process monitoring correctly.

Like nevermind if the software runs or if it has the features you want, it's just the documentation of the process has been done incorrectly. And this is creating all sorts of freak outs. And I think, you know, that kind of thing is emblematic and, and it's sort of, I was, there's another person I was talking with who was, who was saying, uh, the organization they were talking with didn't want to stop and address all their technical debt. And, you know, other than just rolling your eyes, I think, I think the thing you do in all these situations is say like, as long as you're aware that the product you're working on is the process itself and not the actual product, go ahead. Right. Like, like, like if all you want to ship is that you followed a process and not actually ship software, then you're doing a great job. And I think, uh, you see that sentiment, you know, Doug kind of speaks to that sentiment and, and, um, melting or tearing or, or artfully removing as if you're restoring the Sistine Chapel or something like remove, removing these old layers of things and instead replacing them with a much more agile and nimble thing. And, you know, as we're always espousing, a lot of this is enabled by, uh, the automation that you have in cloud technologies, right? It, it doesn't, it doesn't anymore literally take weeks to order and bring online a server or the networking gear. You can just do it within seconds, if not minutes, I guess you're supposed to reverse that little trope, but, uh, you know, you can do it so fast that all these vestigial, uh, governance and things that you had in place. It's not that all of them need to be eliminated, but it's, it's probably much worth your while to revisit this. And, uh, just as one last thing before I, in my little mini monologue here, I was, I was finishing up my, uh, well, I'm trying to finish up. I'm almost done, uh, with, with my, um, we'll have to come up with a snappy title, but the second edition of my cloud native journey, journey, uh, and, and I was revisiting the part that goes over exactly this point, like dealing with auditors and revisiting it. And I was reminded of a, uh, or I should say my to-do list reminded me, there's a passage of target going through this and they had some big old process.

Uh, and this is excerpted in the, uh, the DevOps handbook, but they had a huge amount of process that was slowing down them being able to do something. And so finally someone was like, Hey, we should not do that. And so sure enough, they, they would, it was basically a change, uh, what do you call a cab, a change advisory board or whatever. It was basically like a big sort of cab thing. And someone finally started investigating and asking why they were doing this stuff. And it turns out like no one really knew. It was just sort of like the, uh, the theater they had constructed over time. So it's always worthwhile going in. And if something's really slow and painful, asking why you have to do all this governance stuff end to end and making sure that you, uh, only do what's necessary. Yeah, I picked up, uh, on your recommendation of your, your favorite DevOps books on the new stack. Was it last week? I picked up the, uh, started in scaling DevOps of the enterprise book and started reading through that. And it echoes a lot of what you were just talking about. There is, you know, even where I'm at it's look doing some value stream. And actually, if you document the deployment, that's often going to bring a lot of things to light that maybe people just don't realize of how atrocious it is. I just read the one anecdote where they, one company wanted to see what it would take to get a hello world to production on a new tech and 200, 250 days later, they gave up. And it was, I think to your point, oftentimes it's just about sometimes just documenting what you've been doing and realizing how woefully inefficient it is and how terrible it is. And so people just don't know it's tribal knowledge or just accepted pain. And so often this stuff starts by just bringing to light what you're doing today and then all of you embarrassingly realizing how that's not helping your business anymore. Yeah. It was fun to write a review of that book because I think, I think I, uh, I was reading that book on a plane and I was just like, holy crap, this book is awesome. It was, I mean, I guess it's a testament to like how, how deep and nerdy I am on this stuff nowadays. But like, man, that book is great. That, that, that, that guy writes really well. And his, uh, I mean, like you're saying his general thing, uh, is basically you should really put continuous integration in place and use that as like your, your main to use this word in the wrong way, your main control point on improving your process.

And as long as you have an extreme amount of discipline around continuous integration, all the other good stuff that, that you want to do, uh, will be possible. It's sort of like this never worked for me, but apparently it works for other people. It's sort of like calorie counting when you're dieting, right? It's like nothing about the calorie camp counting, like, like causes you to lose weight, but because you're going through that act, like it makes you pay attention to what you're doing. It's a nice constraint to add to your overall system to, uh, encourage healthy behavior, just like continuous integrations or pipeline. What does he call them? Sort of build pipelines or something, but yeah. The deployment pipeline. Yeah. I mean, that's, I think we hear that from, from customers over and over to their biggest pain point is getting stuff to production. And so as you start to weave back what that means, it's about, you know, where, where are your steps for, where do you have constraints? Where do you have bottlenecks and make sure you're not optimizing in the wrong places. Like that's awesome that your desktop tech is awesome for your devs. If they still swayed for six months to go through a security audit and push to prod, that was meaningless investment. So I think it's value stream mapping. It's putting this stuff up that doesn't sound super fun, but it actually can be. Yeah. So, so there, there's, there's two other things related to this. I mean, this is the, uh, as I was saying last time, this is the kind of stuff that I spend, uh, most of my time working on around here. So I have, I have this, this column over at the register. And, uh, I think the last one I wrote, I, I, I poll tested this in Twitter, this phrase, and everyone told me it was awful. So I decided to use it anyways, because I, I, I figure, you know, that's the way elections work nowadays, but I, it's basically about large aisle or agile in the large. And, uh, as, as a little bit of context, I, I was talking with, uh, with some of my, my, uh, my culty agile friends, and they were telling me there was this sort of argument rebrewing, uh, about, about, uh, about doing agile in the large. And there's basically, and we come across this frequently and I, and I have to admit, even after working on this little piece, I'm still not that much of an expert on it. I need to go read up on this stuff more and, uh, and, uh, talk to some people who are actually doing it.

But, you know, there's processes like, like safe. And I think there's even one called less. And my favorite one is dad, which is the disciplined agile development. I've read about two thirds of that book. Uh, and, and there's other ones. I think, I think, uh, I'm mixing some, you know, there, there's also, you might call them. Like Uber conceptual frameworks, like bimodal it and, uh, and, and things like that. But, you know, it's, it's like this eternal question, or I should say this undying question that we have, which I guess is just a snarky way of saying eternal, but it's like this undying question we have of like, can you scale, scale agile? And, you know, I think, uh, that's, that's a lot of what we spend our, just spent time talking about, but the, the worry that people tend to have, and it'd be interesting to hear from people who listen to this, like what they've encountered is that if you put too much structure around agile, uh, when you scale it, what you end up is basically replicating the same bureaucracy to use that word again, that you had previously. And, and, and there, there's some good critiques of, of agile in the large from, from various people that I linked to in that piece that essentially it's a little bit more than all, but say, cause they sort of half say that essentially these big things are basically just giving your management structure. There's some work to do when, when, when maybe you should look at removing, like we're talking about removing governance and structure, but you know, I, I, I haven't come across like conclusive, conclusive things that show how to scale agile well. And I don't know if that's because it's dependent on, on each organization that you're operating in, or if it's just too new of a concept. So we haven't tested it out, but it is, uh, it's a, it's an ongoing tricky, wicked problem area to, to stare at. It is, you know, and there's conferences around it. Heck, I'm, I'm a program chair for the agile Alliance tech conference next year. And we're, we're grabbing speakers now to, to try to continue to talk about this sort of topics of scaling DevOps and agile and these other things. Cause everyone wants to know what that looks like.

That arguably one of the problems is that everybody comes back from tech conferences today with these techniques that work for these eight person startups. And then they try to implement that at their giant enterprise and get frustrated. And then you do have to think about these things a little differently when they're working in your own culture, with your own teams, with your own tools. And not accidentally do big a agile, which is big tools and methodologies and processes and whatever, and get frustrated. Sometimes it's just about introducing a little a agile and say, we need to be better at collecting feedback, iterating on that, you know, delivering value, and then start to overlay some process. You know, there's just, I think to your point, there's no one size fits all here. There's no single playbook. And that's frustrating because we just want to say, how should I do this? And, you know, there's things you start with. I think you want to talk a little bit about pair of programming today is, is one way to get going with this, but there simply isn't the do this. And you'll get that as much as pivotal labs and pivotal tries to help you morph to that. We still take that into context, how your company runs today. Yeah, yeah. Yeah. I mean, it's like, I like the visuals of your, you go to a conference. It's like people go to a conference and they come back and they're like, the first thing we need to do is make a pizza factory. Like, if we're going to have all these two pizza teams, we need to put in place a process in which to manufacture pizzas. And it's sort of like, you know, laughably as, as the joke obviously goes like, no, no, no, no, you didn't understand. Right. Like, like your job is, your job now is not to build pizzas. It's for those, those teams to actually do things. And yeah, I mean, I mean, going back to like some of the, uh, the anecdotes I was talking about earlier, like, um, in several of those conversations, especially where there's a lot of governance. I was, I was thinking when I was talking with these various people that the part of the issue, uh, is that the customer doesn't really understand, doesn't have expectations around software. And so somehow, uh, they haven't kind of been told how to manage software or what to expect from it.

And so when someone comes up and tells them they're going to be like CMM level four or three and a half or whatever, or they're going to do some ISO thing, or they're going to be safe or less or, or, you know, find their agile daddy or something like, I mean, it's sort of like when, uh, we, you know, we let our front, our front lawn die and we've got these yard people over repairing it. And I don't know what they're doing. I don't understand how to fix that. Like, I have some vague notion that we should put some sod up there and water it, but like, ultimately I have no idea. So like they could just tell me whatever process and I just, I don't know. And I think a lot of the people that are sort of driving budgeting for software, they are in a similar position. And so that's, that's, you know, part of the thing. I think if you want to do agile in the large is you have to go outside of the IT department and really rejigger the expectations that people have about how it can be operating. Cause until you do that, they're just going to be like, well, where's my Gantt charts? Right. Cause that's sort of how they, how they expect things to operate. Well, I think that's a lot of people forget is, you know, whether it's DevOps, whether it's agile, like these things touch the whole company. If you're not rethinking finance and HR and all these other aspects, it's going to be really frustrating really quickly. So then, yeah, as, as, as you alluded to the other thing I wanted to go over, uh, is, is I think, I think previous to this, I wrote, I wrote up a piece on pair programming, which is, uh, you know, uh, I always like to say this is one of the few pieces where I think they didn't rewrite my headline and strap, which usually happens. And I usually have to be a little, but I think they actually used my, my original ones here, which, which was nice. My only other achievement there, there's one of the columns that I wrote where I made a, uh, an allusion to, to the French's reaction to baguettes from Piggly Wiggly. And I think for 2016, that was my crowning rhetorical achievement to, uh, to have that published somewhere. If you, if you remember old Piggly Wiggly anyways, uh, so yeah, I, I, I wanted to write up a piece on pair programming and, um, I was reminded of this because I, uh, I think it was, it was, it was in November, which is still this month.

But over the last month, I, I went to two, um, two of our cloud native roadshows that we did in middle America, uh, over in Kansas city, which it turns out, did you know, Kansas city is actually in Missouri? Who knew? Yeah, it's very confusing. Yeah. I don't, someone, something happened. I don't know if the Mississippi like moved or something, but like it's, someone should have fixed that over at the Rand McNally back in the 1800s. Uh, so you got, yeah, I was up in Kansas city and then Omaha, uh, where, where, uh, you know, that was nice. Anyways, uh, at both areas, we had Garmin, who's a pivotal customer there doing like a, an executive session, which was scheduled to be like a 40, 60 minute talk, but it kind of extended into a 90 minute, uh, conversation, uh, that we had with the room. And one of the things that, I mean, just to do the pie chart in my head, I think the largest share of discussion time was spent talking about pair programming and basically just like, what? And, and so like, it is like, I think, I think if you, I mean, if you're an external facing role at pivotal, you definitely know that pair programming is one of the weirder things that we espouse, but I think internally you just kind of take it for granted. And so it is like, so I wanted to write up a piece. I'm kind of retconning the history here, but this topic comes up a lot and I wanted to finally gather all the research on it. And, and, and it is like, there's not a whole lot of recent research on pair programming, but when you do look at the research, first of all, some of it's a little funny. Cause it's sort of like whatever students enrolled in my CS class, like I had them do some exercises, which I guess is the type of research, but I don't know if it's like incredibly applicable. But in general, if you look at all the research, like basically pair programming works, right? Like, like it works from a, uh, from, from all the efficiency and quality standpoints from an efficiency of money, right? Like, uh, you, for the money that you spend and the time spent, which is just a form of money in this conversation, it's more efficient to have pairs of people, uh, be working on things.

And then also from a quality standpoint, quality consistently is better than if you have, what's the opposite of a pair, a, a unipair. If you just have a solo person. So it is like, and you also like look in surveys and it's, it's one of these things that like is, it's one of the least wide, uh, least followed things out there consistently by surveys. It varies anywhere from like 10 to 15%, but it seems to be like, like thinking, especially in the context of, of, of what I've been rambling on a lot about here, like of all the things you can do to improve a large organization or any organization, the way they do software pairing is probably the easiest process thing to achieve that will scale the most, right? So if there was a mixture of sort of like difficulty of doing it and ability to scale it, something like pairing, because it's basically at the team level, seems like one of the things you would get other than buying Pivotal Cloud Foundry and using it as a centralized place to host all of your software. So it seems like pair programming is like the thing that's the easiest to apply with the biggest buck. And really, uh, it seems like there's only the main thing, there's two main things to get over. And that is, uh, managers thinking that you're basically having productivity, right? And that one's pretty easy to get over because once you pair people up, there's good arguments to be made about, uh, by having two eyes on stuff, you raise the quality and there's the old chestnut about the earlier you find bugs, the cheaper they are to fix, which maybe that's in that leprechauns of coding is true or false. So I forget, but then, but then there's also like, there's all of these proof points from normal mainstream companies that it actually is more effective. Many, many who, uh, have spoken at our conferences that you can see a reference to in that piece and other things. But then too, I think what I've found in investigating this over the past couple of months is that the biggest barrier are actually the senior developers that you have who think that their value is being threatened, right?

So a senior developer might take it as demeaning that they would pair up with a junior person and, or somehow it's like a threat to their job. And many of the reasons that you speak about pair programming is valuable, like diffusing knowledge, not having bottlenecks of individuals. This is the, the low bus number, if you will. Uh, they are kind of threatening to the value that a senior person has the sort of monopoly over knowledge. Um, so that's, that's, that's an issue to get over, but it's the Garmin people who came up with the most interesting, um, anti old programmer barricade, I guess, which was that in most cases in large organizations, senior developers spend most of their time, like in meetings and answering questions because they are the silo of knowledge that people have to come to and extract from. And they don't really get to spend that much time programming anymore. And if they're still a developer and they haven't, uh, uh, switched over to the management track, they probably would like to develop code. And when you pair them up with other people, they actually get to spend most, if not all of their time coding instead of just answering incessant emails, which seems like a nice motivation to have to, uh, get, get your senior people to, to want to actually, uh, do pair programming. If, and I don't think most people, senior people like this, I think there's just a few holdouts that, uh, that don't like it. No, it's a good point. I mean, that's, uh, you know, I've done it before. I've, I've worked in teams recently that did it at scale. And now I sit in an office where people pair all day as well. And like you said, there, there's hurdles to get past. And that initial sense of my gosh, I have to buy higher twice to developers. Now they get the same thing. Like, no, that's not really the case. And some data helps prove that. But as you say, there, there are some challenges. Look, I mean, I watch where teams will struggle because their, their pair is late for work or they're sick and it throws kind of things off or, Hey, you do have personality clashes and it's not even just junior or senior. It's, it's one more introvert versus extrovert or somebody who can't just sit and watch another person type and they have to jump right in and different coding styles, tabs versus spaces.

Like you could have people punching each other. So, you know, there's going to be things that are challenging and the staff you have may not just instantly go to pairing. And that might be a hard conversation to say, look, maybe they switched to other teams and pairing has to be something we specifically do with people who are game for this. Because if you're fighting it the whole time, you're probably not going to get the benefits from it. So, you know, maybe you ease into it. Maybe it is a few hours a day with the team or you start all your new hires with pairing so that they can learn quickly. Or you use it for testing and debugging and say, look, whenever there's testing, you pair with a tester or you pair with another dev. And that's how you do some of your unit tests or whatever it might be to kind of get you started with that, because it seems like people are very visceral around pair programming. Either you hate it or you at least tolerate it slash love it. Like there's very much in the middle, like meh pair programming. I think for the most part, you love it or hate it because either it's forcing you to be social and you'd rather just work by yourself or you love it because it helps you get more knowledge, helps you solve problems faster. And you like the social aspect, but it's, yeah, it's definitely something where, you know, I've also seen people recommending that, hey, look, pairing doesn't just mean you literally sit at the same desk and you both type together. Pairing can also mean, hey, you solve problems together by going to a whiteboard or walking around or, you know, in Pivotal playing ping pong together for 10 minutes. Like it's also about team building and kind of getting outside of your space, not literally just staring at the same monitor for nine hours a day. Yeah, you know, ping pong. You know, if someone was pairing with me, it would be like, hey, let's go make snarky comments and Twitter together for 10 minutes. I think that's how we would blow off steam and bond. I think he would need to pair next week in Vegas. Yes, over some lobster egg benedict. That has to happen. You know, you're reminding me of another thing that came up in the Omaha discussion we were having about offshoring, not just outsourcing, but offshoring. And there was one of the people there had some like Eastern European developers and they were talking about how you do pair programming with that.

And this is a mixture of like what they actually did in theoretic and theoretical stuff. And there was I think there was basically like two hours of work time that they would have. And the discussion we were having is that and this is my suggestion is that part of the problem I've noticed over the years that companies have with offshoring is that they don't trust them. Right. They're just that and this this has to do with all sorts of like good and valid and it's just like people. Right. Like their whole reason. Well, see, I'm even kind of doing it here. But one of the reasons companies will cite for doing offshoring, what do they what do they call it? They don't call it cheap. There's some some magical phrase they use for like it's sort of like economic imperative butter rubbing like like there's some kind of like low cost or something like that. But essentially, because it's cheaper, right? Like that's that's what motivates. Now, it could very well be the case, especially with open source driven products, that it's exactly the opposite of it. It's just that like, as you know, we've had Marcin who lives in Poland on this podcast several times. It's like, well, that's where the talent is. So we we offshore them, as it were. So there's many reasons. But with large organizations, someone was saying, you know, we have this team and like one, they're kind of grumpy because they don't get treated very well. And then to like it's kind of a net loss because it's expensive. Like we have to check their stuff and review it. And there was an interesting discussion about, well, maybe what you should do is remotely pair with them or send people various, you know, get them on planes to pair for like a couple of weeks. And then you start diffusing knowledge across the two teams. And more importantly, you start trusting that these teams know what they're doing. And then once you get to the point that on whatever shore you're on, you can basically review each other's code and you trust that. Then you remove a lot of this this drag that you have with the the follow the sun approach, which, again, that was a mixture of like stuff people have been doing and theoretical things and stuff like that. But it seems like another good way that doing that the idea of pairing could could address some some large organization problems.

Yeah, great point. So and the trust that trust is. Yeah, trust is such the big thing. And that's usually the first part you have to get past is I'm trusting my pair. I'm trusting the team as a manager. And then I'm trusting as an exec that team manager is being productive with the people they have. So, you know, that's the first hurdle to get by, which is probably means you can't wholesale on a Monday saying, hey, we're going to pair on everything. There probably is a at least a bit of a warm up period and prove that it works great. And, you know, you might even rotate pairs. You might rotate teams to make sure that people see the excitement. But there's a reason why Pivotal goes into every engagement and kind of enforces pairing or same with our dojos is, you know, we want the people to pick up skills. We want them to build trust even with their own team and even with us that we're not just consultants coming in. We're coming in as collaborators and partners on something. So it's a total mind shift and it's weird. And I've even seen teams do mobbing where you have 10 people in a room all doing it together to solve a problem and build something, which seems kind of woefully or horribly inefficient. So you can't do that all the time. But even then, it's about using the collective brain to solve hard problems. I think that's a good advancement versus just relying on the lone hero developer who has to go in and solve all the problems. Yeah, yeah. Yeah. So there's three things I was just thinking of. Before we close out. And that is, I'll see if I can remember. But, you know, we talk a lot about this internally and it'll be interesting to see over the next year. Like, I think there's another pairing thing that in our cloud native community we need to figure out, which is sort of like enterprise architect pairing. And I don't really know what that looks like, but there's a lot of discussion that we have. And we've got a whole, like, group of people who help out, like, from the tops down of doing transformation and things like that. But I've been in a lot of discussions since I've been at Pivotal with enterprise architects. And it's not so much that they're having some sort of existential crisis. It's that they're trying to, well, that happens a lot. But they're trying to figure out, like, what is the role of enterprise architecture?

And what is it that this skill set can do to help everyone out? And, yeah, I don't know if anyone actually knows. But I think it's through, like, through pairing with people who have worked on this stuff in the past and kind of documenting what ends up happening that we can accelerate what that role does. Because I feel like going back to the large aisle stuff, one of the things that the large agile disciplines often bring up and that the, I guess you could call them, I think I jokingly referred them as the agile poets, the sort of agile in the small spaces. They never really deal that convincingly with what you do with, like, cross dependencies and things like that. Like, Ron Jeffries has this great piece where he's, like, it's almost like the, is Lao Tzu the Tao guy? It's almost like, how do you deal with cross dependencies? You don't have them. Huzzah! Right? Like, it's just, and so, like, which is, you know, delightful, but doesn't seem realistic. So that would be an interesting thing to, like, figure out pairing on. And I've forgotten the second thing. But it reminded me, so, like, since you're around it all the time, what are the, like, I guess pun intended, but what are, like, the smells of pairing? Like, what does it look like, like, physically when you're moving around? Like, like, how would you describe it? Yeah, I mean, it looks very noisy. So one thing I have to always, you know, get a room when we do the podcast, because it's a noisy place, right? Everyone's pairing. Everyone's talking. You know, people are working on problems. And what's really interesting is when we do interviews, and I'll watch someone come in, because we do interviews through pairing. And so you're going to see, how does the person coach them through? I actually asked at breakfast this morning at the office any tips people had for some of the things for getting started. And, you know, one of the fellows said, look, I have to sit on my hands sometimes, because my inclination is to just jump in and try to solve everything. So I physically sit on my hands, in some cases, so that I don't just jump to the keyboard and kind of dominate my pair. And so sometimes it looks like just a lot of conversation and typing and problem solving. And we don't have touchscreens, because we all use Macs, but all the monitors seem to be smudged, because everyone's touching their monitors to solve problems.

You know, so it just looks like a very interactive, bustling place. Now, for some devs, that is the complete opposite of how they like to work. They like to be in a dark room and coding with headphones on, and nobody bugs them. So this sort of very lively, noisy, problem-solving environment can be, again, very jarring to people who don't like that work style. But when you're in it, there's also a really good energy, and people don't feel like they get stuck very long. Yeah, I think for myself and many people, pair programming is like a NIMBY issue, which is like, that all sounds great, as long as it's not me. Right. But, yeah, I remember early on, here in the marketing groups, I've messed around with, people would try to, like, they would say, pair on things. And it's just like, maybe it's just because I'm a NIMBY on it, but that just didn't work for me. Like, I can't, like, co-write a blog post with someone. But it's, I don't know if it's just, like, I'm resistant or whatever, or maybe that work of the, because really, like, programming is pretty, kind of similar to, like, writing a blog post or, like, putting together a PowerPoint. But there's something about it that, like, I wasn't having that. That was a bunch of nonsense. Crazy talk. Well, with that, maybe later today, the second thing I was going to bring up will pop into my head. And I'll quickly forget it. When my daughter asked for more milk, and I have to tell her yet again, we do not have milk. We're out of milk. But we'll see what happens. But on that note, this has been Pivotal Conversations. As always, if you want to be in the vanguard of people who listen to this, the thin edge of the wedge, the sort of, like, the slowly coming in tide as the tidal wave comes in, you can go to our not-so-secret backend at soundcloud.com slash pivotalconversations. And there's an RSS feed there. You should just subscribe to it. And then you don't even, after that point, even if you don't want to listen to it, it's fine. Just subscribe, because this is the only thing we track.

But hopefully you actually do listen to it. And if you like it, it would be great if you're in Overcast, as many of our listeners are. If you click the recommend thing, you just kind of look at the show notes and scroll up, and there's a little star. I have no idea what that does, but it sounds great. Even better would be if you went into the iTunes store and added a rating or wrote a review. Maybe you could tell us what it smells like to pair with someone, and that would be a nice review to see. I haven't actually checked this episode to see if it was written in a review. So if they had, thank you. And if they haven't, now you've got something to do. It also would be nice if you just tell people that you like this podcast or tweet it. Essentially, help us market it. Barring all of that, you could just tell us that you listened to it, and that will save us enough. It'll keep us nimble and thin in our little dark cave of podcasting, and let the spirit stay alive. So with that, thanks for listening, and we'll see everyone next time. Bye-bye.

Tanzu Talk · 🍕 Get more pizza.