Dieu Cao on PMCing in Cloud Foundry, isolation segments, & cloud-native QA (Ep. 59)

Episode 42 · 49:21 · SRT · archive.org · all Tanzu Talk

Well, you know, I just got from spending a week doing the old business traveling over there in Paris, which, you know, sounds luxurious. But I wanted to ask you this, Richard, and guest, if you care to weigh in. Now, when I get back from a long trip back here to Austin, the first thing I always usually do, there's two things I miss. Well, there's three things. The first thing I miss is like hearing Spanish spoken frequently, but that's, you know, not really in the same category as other stuff. So it's always nice to come here and then you start hearing Spanish. It's comforting. And I guess if you go to Spain, you don't have that problem. But other than a place where they speak Spanish. And then the next is like, you know, second, I like to go to a restaurant and just get free refills on drinks. That's always refreshing. And then also iced tea, unsweetened iced tea. Not always possible to get over there. And then third, it's good to get like a taco or some enchiladas or something like that. And I haven't done the third yet, but I'm almost acculturated back in. But I ask you one, maybe two. When you come back from a long trip, is there any sort of like comforting thing you do? Or does your mind just start rebelling at all the inanity of what it's like to be at home? I love that when you added number three, I thought you were going to say I get to see my family. So I'm glad that tacos actually feed that. So well done. Richard, tacos are my family. That's taco one, two, and three. Guacamole comes to visit. No, so no, my first thing, the best thing when I get on a trip is to get into my own car. It's just a moment of such relief when I'm not stuck in a taxi or whatever kind of car, especially in another country. That drive home is luxurious. You know, that is a good point because you're making me realize that every time I'm going back on the little airport shuttle in my car, I think, am I going to remember how to drive? Like it always feels like a little odd. That's true. That's true. How about yourself, special guest? Do you have any reacculturation comforts that you go to? I miss dim sum, so I always have to get my dim sum when I get back from a trip overseas.

You go to the cart of surprisingly large bill, as I recall how that usually works out. But, yeah, you have no idea what the plate costs that they're presenting to you, but it looks good. Well, with that, why don't you introduce yourself briefly before we get to the news, and then we'll come back for more detail about your areas of interest and expertise. Sure. I'm Yui Cao. I work for Pivotal. I actually wear two hats. So at Pivotal, I'm a director of product. I'm responsible for basically the elastic runtime tile and the things that go inside that. And I also, with the Cloud Foundry Foundation, I wear the Runtime EMC Lead Hat, and I help out with those projects. And amazingly, there's a lot of overlap between those two hats, so that's what I do. Yeah. It's always fun to meet a new hat rack person who's running around with multiple things. Well, we'll get to details of a lot of what, I don't know, the full VIN diagram of your hats is. Maybe we'll figure out if they're caps or other types of hats. But before that, there's just a few news items. There's not a lot going on this week. I think I was waking up this morning having my coffee. And it looks like Microsoft purchased, I guess you would call it a divestiture from Engine Yard, although their association's not so clear in all the stuff here, even though it is clear. But they bought, how do you say it? Dees? Dees. Dees. There you go. A godlike company. Now, I learned a lot more about them reading up on this news this morning than I've known in a while. My main knowledge of them is that they did like a Paz thing a while ago, and they have excellent socks at conferences. People who have even better socks are the, who's the SSD company that has some fire name? SolidFire. They have the best socks at DevOps days, but the god company, even better socks. But it looks like, in addition to hiring like a, other words I can't pronounce, a Kubernetes person back in July,

and no doubt some other people over at Microsoft. Now they've got like, you know, I haven't used it firsthand, but it seems to have evolved into a Kubernetes orchestration layer, along with the registry and some other things in there. So I think we were noting this on my other podcast, Software Defined Talk, a couple episodes ago, that pretty much everyone, as it were, is on the whole Kubernetes train, except, of course, our buddies over at Amazon. So maybe there'll be the lone holdout to be pretty deeply involved in that. Yeah, I mean, if you're Microsoft, it seems like you're just, you know, I don't know, hoarding assets at this point. They support, they've invested in Cloud Foundry. They've obviously done a lot in the Kubernetes space. They've, you know, messed around with lots of good platforms, Mesos, others, Docker. So I think they're just not picking any winners. They're going to offer their customers virtually everything. And I'm assuming in this case, I don't know if they would rebrand and resell this. Do they just want to use the IP to make Azure better? Is it an acqui-hire? Could be a lot of those things. But, you know, either way, Microsoft's getting pretty credible in this space. And while I don't see a lot of opinions coming out of them about any of this, it seems like they're just willing to offer lots of different things. Yeah, maybe they'll put some little Microsoft logos on their socks or upgrade. That's a given. You know, no one really gives away hoodies that much. I guess they are like $30 or $40. So they're very expensive. Yeah, it's like those water bottles we did at SpringOne. You know, I promptly lost mine like three weeks later in Phoenix. I had to get up at like 3 a.m. for a flight, and I guess I just left it. I think I had a black one, and it must have blended in with the little hotel table that I was there. Those are excellent. So, yeah, you know, I was looking through one of their – it's actually a January 2016 framework. And I know that you, Richard, are a connoisseur of cloud platform diagrams, and they have a pretty good one that's worth checking out. Yeah, I saw that. I saw you tweeting about it as well. That's a good hamburger stack, I believe you – Yeah, yeah, I used to call those burger stacks. But we have a new diagram out too, right? Or you were sending one around.

Is that one public yet? We do, yeah. It's actually – I think it's up on our new Pivotal Cloud Foundry page. So it's not news per se, but if you do go to pivotal.io slash platform, there's an entirely new and refreshed PCF page that, I don't know, shockingly tells you what the product does, which is always good in the product pages. And I always like the sort of Hipstertron color palette, sort of like dark black blue with some, like, little lines from the light bikes and everything. It's well done. Right. You stare at it long enough, it actually does start moving. So that's key. So, yeah, it's a great new visualization of what PCF really is. Definitely encourage you to check that out. You'll see us inject that into a lot of stuff now. Yeah. Who fights for the user indeed. So, also, I think there's two items that I ended up grouping together here. One, there was the usual, but always useful because it updates the numbers, capital spend on top cloud companies. And what was it? In 2016, there was like, we'll just round up. There was basically $32 billion in spending building out. I assume there are various hardware and maybe real estate. I don't know how. I'm no accountant. I don't know how that all gets counted. But it said it was basically up 22% from last year. So, there's just a continuous, I think the word might be onslaught, of just like building up these machines out there. Yeah. I mean, it's pretty amazing. That article was a fun read. Some smart sounding fellow said that, you know, game over for anyone trying to get into this market as general purpose. And even if you are kind of in it, whether you're Oracle or IBM or others, AP obviously got out. VMware got out, as we talked about last week. Rackspace is, I guess, kind of out. So, it's tough to be in public cloud if you're not one of the big three if you can't either find a really specific niche to work in or that you can simply somehow out-compete these large providers by being more efficient or some other innovative way. So, it just, I think, continue to double down on things we talked about last week and things we all see is that these big three are just plowing money into facilities and capital.

And to your point, you know, I think we mentioned last week, like, the Rackspace's CEO's comments on not competing in public cloud where they would have, like, 300 million free a year to spend on other things. And, I mean, clearly, if you're competing with someone who's spending, I mean, if you divide, let's say, I think the Amazon figure was at least 10 billion they spent. Like, your 300 million twig is going to be difficult against that, I don't know, concrete monolith. It's a difficult game to get involved in. Yeah, the end of that article has a quote from the fellow who runs infrastructure at Google saying each of their regions cost between 300 and 600 million. It seems like a big range, but that's a lot of money. And then the other one, one of my old friends from the systems management world was linking to this earlier today. Of all places, there was a Reuters story. I don't know how this got on the editorial agenda for them, but it's awfully specific. But it was going over kind of the predicting doom in large banks, rewriting all their COBOL programming. Now, I always have to remember COBOL ends in an O, not an A, but, you know, I've got to verify that. But there was an interesting use case in there of, I think it was the Commonwealth Bank of Australia, that in 2012, they spent about $700 million rewriting all of their old COBOL stuff. And I thought that was interesting because, you know, most people we work with in Pivotal and in Cloud Native land are these large organizations. And, you know, if you extend out the math or so, depending on if you, I always like to round down for estimates, like not round down, but take a, you know, go down to like 500 million instead of 700 million. But if you multiply that across 20 or so banks, and that's like 10 or $15 billion in spending, which, you know, as I like to say, I think we can all agree on money. But it is an interesting, like, an interesting look at the sheer quantity of, I don't know, enterprise software out there and the huge opportunity for like all of the types of stuff we talk about to convert it over to that way of doing things. And I guess there'll be some COBOL left over. There's always like old stuff left.

Yeah, I mean, I think at the IBM event a couple weeks ago, they were still using the number, which had seemed like it's been updated in a while, but like 90% of the world's data is on mainframes. I mean, it was some insane number. And I've seen that for years. And I don't know if that's just not changing or what's going on, but a lot of data is on mainframes. I was at an event and spoke a couple weeks ago, the integration event, and it reminded me how much EDI still runs communication between most companies as they transfer data. Like, it's easy for us to think about microservices and queuing and Kafka. Let's remember most data goes through things like EDI and gets stored in the mainframe and maybe sometimes reset ourselves about all the new hotness. Yeah, I mean, it reminds me, I visited a large company when I was in Paris last week and they had, I don't know, I think this question is more frequent or probably be more frequent over the next year and coming years is essentially, how do we order our now becoming legacy application portfolio and decide which things to start rewriting first and the quantity of it? Everyone has a lot. Everyone has a lot of trepidation, but clearly that's a huge thing on the agenda over the next five years or so. I mean, I guess, I don't know if it's true or not. I guess you're always in theory needing to rewrite your stuff. But at any given day, it seems like you haven't been actually rewriting your stuff enough and you feel overwhelmed by it. But hints are the shackles of success. Yeah, sure. One of the best descriptions I've seen lately of microservices is, you know, back in service-oriented architecture, Coté, our mid-2000 days, a lot of focus on things like reuse. And the focus on microservices actually is replaceability. Like, how easy is it to replace things? And that's just a different way of thinking. But to your point, if you're always going to be evolving your stuff, how easy are you making it to replace individual components versus entire $100 million systems? I would think flexibility matters there. I like that focus more on carving things up for replaceability, not just reuse. All right. So let's jump into you. I want to chat with you a little more about kind of your role on your multiple hats, as Coté mentioned. I don't know, part of me really wanted to focus on some of the open-source governance piece, because I think sometimes these things are seen as a black box.

Or even though it's open-source, it's seen as, well, you know, Pivotal does all this work, and it's not really an open-source project or whatever. So I just thought it'd be fun to chat with you about kind of what makes up Cloud Foundry. What is that experience like? So I thought I'd start with this kind of question of, if you look at Cloud Foundry, you think of it as a thing. But in reality, it's not just a thing. It's a set of, what, a couple dozen projects that make up the whole thing? And can you tell us a little bit about how Cloud Foundry is arranged and what makes up the platform as projects? Yeah, I'm happy to. Right now, Cloud Foundry is arranged into four different PMCs, Project Management Councils. The main ones are BOSCH, and that contains core BOSCH and all of the BOSCH CLIs. There's the runtime PMC, which contains most of the core bits from UA to Cloud Controller to Garden to the CLI. And then there's the extensions PMC, which is really a holding place where we'd like to nurture projects that have extension points on top of the core runtime. And then additionally, our newest PMC is the Open Service Broker API working group. And they've been kind of collaborating with the Kubernetes Red Hat community there on a service broker model that we can do there. And these are all very DevOps, right? I mean, these are independent teams. They all have different leads, I believe. So, I mean, these are all fairly independent units that come together to collaborate. Is that right? Yeah. They're all very independent teams with different PMs. We certainly have teams that are geared towards integration. So, we have a release integration team in the runtime PMC that, you know, basically helps with the tooling and how all of these teams contribute to, you know, a more or less a blessed set of releases that we believe work together, that we can tell the community.

These sets of releases we believe work together, and we can provide you some tooling to help you figure out, well, if you need to change one of the releases within, how can you test that out? What sets of tests can you run? And how can you put that together? That's one of our main integration teams. We have an infrastructure team that works on some of the shared components like etcd and console. So, while each of the individual component teams function independently and the UAA team can prioritize UAA-related things, there's a lot of cross-cutting things that come up. And we all kind of work together to try to support each other in getting our next big feature out. How do you work that? So, I mean, that particular part, did the infrastructure teams and things like that, were they organic? Did they come up after a while where you realized that, I mean, your group here sits at the guts of what Cloud Foundry is. So, clearly, you can break everybody if you don't do things the right way. So, did that stuff just kind of emerge? Did you set that up in the beginning? How did you make sure that when you're making changes to these fundamental pieces of Cloud Foundry that a component like a metrics component in PCF or another component in Bluemix or whatever doesn't collapse? How do you keep it in sync and how did that come to be? So, we keep it in sync with a lot of testing, a lot of pipelines, a lot of integration. This kind of grew out of, there used to be literally a single runtime team that took care of DEAs, HM9000. We're now replacing all of those components. But there used to be literally one team that was maintaining all of this, a four-pair team that would cut across five or six different code bases and three or four different languages and still cut the release and work with the PWS team to get that deployed. And we found that, you know, we just spent a lot of time trying to keep the lights on and not being able to move forward.

So, we started looking at, well, where are some dividing lines? Where can we, where are some natural boundaries where we can divide up the responsibilities? And I think the trust that we have in each other to keep moving forward and keep shipping, that kind of grows from starting with that small little team. I mean, I think I've seen slides from you or Ansi, who heads up engineering, kind of show like, hey, we had these few teams and it kind of exploded to here and now it's geographically all over the place and things like that. So, as you do think about that, I mean, how do you work on that communication piece? How do you make sure that teams that are distributed geographically kind of working for the most part in the same, hopefully, sprint cycle? I mean, you tell me if these teams typically work in the same rhythm, but is there a communication? Do you have a daily stand-up? How do you make sure that teams do stay in sync? There's a variety of mechanisms that we have. We've got each of the individual teams, they've got a daily stand-up for themselves. Some of the teams that are closer to each other, the Diego Garden Container Networking, have their own weekly sync-up where they talk about some cross-cutting things. I have a weekly meeting with each of the teams to talk with them about the health of the team, if there's any, any blocking issues that they're facing from other teams and what are their next big things that they'd like to work on and get some visibility or help with from the other projects. We have, in the runtime PMC, we have in the runtime PMC, a biweekly meeting where we just talk about, the project leads talk about what they've worked on in the past couple of weeks, what their new tracks of work are, and publishing that very publicly on CFDev, our mailing list for wider collaboration. And then we also have a monthly call that's open to the wider community to ask us whatever questions they want about what we're doing and how we do things.

So when you're sorting through, I guess as we would say around Pivotal, the backlog of requests from all these different people, I mean, I guess you're sort of like there's at least three constituencies. Y'all are sort of managing requests from one yourselves, right? That's always fun. And like you're saying, kind of identifying like, well, we spend a lot of time self-managing, so we should figure out fixing that, kind of standardizing things and not having all these different internal scaffolding we have to maintain. I don't know, just to pick one example. And then there are the various vendors that use Cloud Foundry. And then I guess also to some extent, other commercial interests, you know, system integrators and others, and even end users who are doing things. And then I guess there's sort of like the third bucket, which is part of that as well, sort of the actual end users of Cloud Foundry stuff, but also individuals who are interested in it. And like, I don't know, what's the general thinking y'all go through when you're prioritizing things? There's, it's pretty complicated, because there's a lot of different stakeholders with different opinions and different abilities to influence that. There's certainly table stakes around addressing security issues. And that's important to the community and to foundation members who are producing their own distributions, right? That's just table stakes and pretty easy to prioritize those sorts of things. And then there's stability, you know, is, are we having issues in terms of scaling so-and-so component? AWS has a certain workload, and that certain workload is causing components to fall over, so how can we address that? Or Bluemix is having certain scaling issues because they have a lot of different users on their environment. So those kinds of things are pretty easy to prioritize, the security and stability, keeping the lights on sort of table stakes sort of thing.

And for a lot of teams, that may take up, you know, 50% of the time, you know, 70% of the time addressing security issues or stability issues. And then what time you have left over, each of the teams, they all have public GitHub repositories where anyone's free to ask via a GitHub issue, you know, I'd like this kind of feature, this is causing me pain. And that's a direct line into each of the teams to try to plus one, a particular feature set. And the PMs, you know, take that into consideration about, you know, should we prioritize this? Because we're getting a lot of plus ones about one thing or another thing. The HA Proxy Bosch release, we were getting a bunch of requests about, you know, can we expose more features in HA Proxy? And one nice thing was the Stark and Wayne, they put out a community Bosch release that did a lot of these things. So we reached out to them and asked, you know, hey, can you come in and contribute that to the core Cloud Foundry? And now we're going into the incubation process. So we get a lot of signal just via GitHub issues or PRs that the community is able to communicate with us through or through the CFDev mailing list. Now, our commercial customers, IBM's commercial customers or SAP's commercial customers or Pivotal, there's additional conversations that we have with stakeholders from Pivotal and IBM where Pivotal needs a certain feature set. And what's interesting is that a lot of these are somewhat cross-cutting, like isolation segments is something that a lot of Pivotal customers were very interested in. And IBM was similarly very interested in it. And we thought about them a little bit differently, but we collaborated on what the MVPs should be and what the different phases were.

And we kind of worked on that together. So I think, interestingly, we have a lot of common ground on what we want to do. And the edges where there's maybe conflict, those are kind of few and far between. And we try to mediate those by listening to each other and trying to carve out time for feature sets that people are interested in. And at the end of the day, we are governed by contribution. So we look at, well, IBM's contributing, you know, maybe 10%, 20%, 25% of the engineering effort for a particular team. They should have some sway over that 25% of the team's bandwidth, what they should do next. So I mean, does that, you know, we think about the occasional conflict that does come up? Because as you mentioned, we're all member companies of the Cloud Foundry Foundation, but we all are selling competing offerings and we have different business needs each company needs. So who becomes the arbiter of that? Is it the PM? Does it go up to, you know, someone like Chip at the CFF who kind of is the CTO up there? How do you, you know, if there is a design difference or even a prioritization difference, how do we start to come together and what sometimes finally wins the day? A lot of the times it's the PM. Some of the cross-cutting features, I help to facilitate that conversation and we try to find common ground. At the end of the day, Chip can step in. We very rarely have gone to that point. And I'm thankful to say we've never actually had to take anything to a vote because we've tried very hard to satisfy everyone's concerns just through the normal give and take that occurs. So we might say, hey, you know, we realize you want to prioritize this. Could you help us by sending out over a couple of engineers, you know, to spend time on this?

So very recently, Microsoft is interested in adding support for SQL Server to UAA and CAPI. We didn't quite have bandwidth in the CAPI team's current set of priorities for SQL Server, but the Microsoft team, we had a discussion with them. And now they're contributing a developer to help us get it going and get it going in a repeatable way, not just throwing a PR over the fence, but helping us with our CI and making sure, you know, we can maintain that support into the future. That's exciting. Does the release cadence help? I mean, CF release seems to go out every, I don't know what we're at now, three, four, five, six weeks. I mean, it's fairly frequent that a new CF release gets cut. So does that sort of constant rhythm help that someone doesn't feel like if their priority doesn't get taken care of this release that I'm not waiting a year, like there's a decent chance it comes pretty quickly? Does release cadence kind of help mitigate the risk? I think so. I think that's key. David Sabeti, who's the PM on the release integration team, he's got a reminder on his calendar every Tuesday to, you know, try to push forward a CF release. I think if we can get to a weekly cadence with shipping, you know, this compatible set of bits works together and we believe that nothing is worse than before. It's only better than it does mitigate the concerns people have about can I get this one more thing in or this one more bug fix in and really slowing down our delivery cadence there. So one thing that's always interesting to, I don't know, for me, keep an eye on is as we mentioned, and you, I don't even know if you had a chance to, or you kind of cut yourself off from laundry listing all the various projects that you get involved in, let alone the ones that are in the whole Cloud Foundry world.

But are there components in there that get used frequently like on their own or separately or all of these things sort of like interdependent with each other such that like you need all of them to be something and it would just be like a headless Voltron or something. I mean, every now and then this idea of using one part on its own comes up and in the other, in other parts of the world of whatever you want to call it, our cloud native ecosystem, there's always a lot of focus on the separate components rather than the whole. But, you know, I wonder if, in your experience or opinion, like having using any of these things separately makes any sense or it's just a bonkers notion. Yeah, I think we're trying, we're in favor of reusability where it makes sense. The UAE team, their release is actually used independently in a few places, not having to do with Cloud Foundry at all, because they're solving an important problem around OAuth there, and I think that that's really great. The etcd release is sometimes used independently, and I believe that team is also working with the team working on Kubo. The HA proxy release that we had in CF release was used as the base for the HAProxy Bosch release, I believe, and that is certainly also used independently. MySQL release and the Postgres release, they're used as the base for, you know, your own independent relational databases as well. We're looking at trying to, you know, at the garden and the Groot FS layers, where we can share some capabilities with other container orchestrators.

So we're in conversation with them about where's some common ground where we can share some libraries, if not the entire components, but at least share some libraries, where we're all trying to do a similar thing. And then, you know, on the other end of the, I don't know, the stack of sophistication, as it were, I mean, you've been involved in the Cloud Foundry world for quite some time, as I recall, right? Yes. Well, since 2014. Sure. Long ago. But, you know, I only have sort of like a slight dilettante study of all this, but it seems like, and correct me if I'm wrong, it seems like there's a history of naming in the Cloud Foundry world, where it started off with the usual, just like, really silly names that engineers will come up with, right? Whether they're like recursive names or like named after robots or whatever. And then at some point, there was kind of like a softening or a slight maturation, right? Like you would have something like Garden or Diego. But then nowadays, it seems like there's a policy of like, name the thing the thing that it does, right? Like, don't come up with some funny name for it. Like, be extremely descriptive of what it is. And so like, is there some like policy on naming things? Or is my characterization of how it's evolved accurate or anything? What's been the name with like the first bucket of silly names? And nowadays, and I don't mean silly in a judgmental way. It's fun. But like, it seems like there was a bunch of just like, you know, let's come up with a funny name for something. And nowadays, it's just like very, I don't know how to describe it. Straightforward? I know what you mean. Like droplets versus, you know, isolation segments is the most boring corporate name, but makes total sense given what it is. Exactly. Versus something else. I think it's just an evolution where we found that even within people who are familiar with Cloud Foundry, it was getting kind of hard to keep track of what a particular component does or what it means within the system.

So, you know, I tell Jules Friedman all the time that he is not allowed to name anything anymore. Because he'd always come up with interesting, punny names. But for someone coming in from the outside or wanting to reuse a particular component, it makes it just that much harder to find your component and want to reuse it. And just as the ecosystem or the number of projects grows and grows, we want to really reduce the barrier to entry. And naming kind of gets in the way sometimes. So what are some of those fun punny names that are now on the restricted list? I guess I'm not on the restricted list, but that are representative of fun instead of easy to comprehend ability. Also, Diego used to have a number of interesting component names. There was a Tinker, Taylor, Soldier, and a Circus. There was the theme there, and we ended up renaming all of those components to be a little bit more transparent about what they do. And the container networking team, they used to have a release called Netman. And they've since renamed it to CF Networking, to be a bit more clear there. The bit service, we still internally call the team Flintstones, but they wanted to call their release the Flintstones team, and that was an extrapolation from... I don't even remember how we got to Flintstones, but it made sense at the time.

There was some association with Pebbles, and then it somehow went over to Flintstones. But the service, the team's now called the bit service team. So trying to be more clear and a bit more discoverable and a bit more accessible to new people coming into the Cloud Foundry ecosystem. Yeah, I guess as a sort of maybe more of an almanac than a grimoire, but as far as the grimoire of naming metaphors, like the Flintstones are pretty approachable, but you don't want to just throw like, what's the guy's name? Alacare, a Tinker Tailor spy thing at someone and have him figure out all the intricacies of what those metaphors mean. So speaking of, I mean, since you've done some work and also presenting and things on this topic, so we talked about like some of the isolation stuff last week, but just give us like an overview of what that bundle of capabilities is. And you were kind of alluding to this earlier, like why it became a high priority thing to work on and what motivated people to want it and made you all choose it as a thing to develop. Yeah, isolation segments, I'm pretty excited that we've got the capability out there now. The CLI just became available last week. So the big things there are trying to reduce complexity for operators. So for a number of companies, they end up deploying completely separate cloud foundries for many different reasons. And we're trying to look at, well, that's interesting. Why do you have 16 cloud foundries deployed at your company? You know, that seems like a lot of overhead, a lot of complexity that, you know, and some of the reasons were that, well, we are looking for ways of segmenting the workload

so that it could just be a quality service saying, okay, these apps are higher priority, so maybe we'll stripe these across three AZs and maybe the dev environment will only stripe across two AZs or one AZ because they're lower priority or they get the, you know, cheaper hardware whereas our prod workloads get, you know, SSDs or something like that. So being able to make that a little easier for operators to say these workloads can have a better time on the platform without having to have a wholly separate login space and org management and management of quotas and whatnot that you have to do with every single new cloud foundry that you stand up. So there's sort of like at least two main motivations. One of them is, as you say, you know, you're looking to solve a problem and then you set up 16 foundations and now you have 16 problems, which is exciting. And that problem essentially is there is a, I don't know, a certain amount of management required, operational things and integrating. And then also what comes with that, as you're alluding to, is verifying that you've set all of your security stuff up correctly and you have everything configured. So you've got to go to all 16 of these instances to make sure that it's acting in the way you intended and you desired. And so it would be nice if you just had like one setup for all of that. And then the reasons people are motivated to do that, I mean, as the name would imply is, and I see this all the time, is they still, they have this mentality of dev and production, at least, or prod, as the kids like to say. And, you know, so at least you have that, but you might also have QA and other things. You might have all these different foundations out there. And so I guess the feeling is that like previous to having isolation segments, like you can't really effectively, I'm trying not to use the word isolate,

but make it seem like to the end users and the various processes that are running in there, that they're in their own foundations. So you need controls in there to shield them off from other things. And I mean, do you think what's in there would allow people to fold down 16 foundations now? Or like, when do you think, when do you think that would be, you know, when's there going to be like a CF make it all one foundation with hyphens underneath there? I don't know if we're ever going to be able to get to CF make it all one foundation for certain customers. There are certainly, like some of the things are use cases that people want to solve is disaster recovery. And any two ways you look at that, you just need to. Right. Unless your mean time to recovery is really, really quick. But for the most part, a lot of people try to solve that with two. I think I could see maybe a reduction by half in certain environments, where maybe that 16 foundations goes down to eight, because you still have regional needs. You need your apps to be running close to the data, wherever that may be. So there's still some additional foundation, additional use cases that we need to look at and see how could we solve that and reduce the overhead. But I think right now with the capability we have for some customers, we could probably reduce the number of foundations they have by half. And that's pretty big. So when you're, I mean, I know from following some of what you all had to build for isolation segments, sometimes these things that appear straightforward are obviously complex under the covers. Are there other things that make up, especially your domain, that maybe are a little underrated in terms of how, you know, what they have to do? And where there's, I mean, Cloud Foundry is a really complex distributed system. It's exciting, but you can take a lot for granted,

because we purposely shield people from a lot of the guts. What are some of the things underneath the covers, though, that maybe we should appreciate a little more? I've always thought CF push just, it's beautiful, but it's hiding away so much complexity. It's a massive orchestration of what's going on between, from the time we take your bits and put it together with a build pack and make it runnable code so that all you had to do was push that application source that you really care about. And then how we actually place it into and keep it running, how much we work hard to make sure if your instance goes down, if you go, ooh, we'll try to bring it back, if your route availability and how we can scale out horizontally across a lot of our components, and how much we do under the covers to make that CF push experience as transparent as this. I mean, the health stuff's interesting. I was reading this morning a little bit just on some of the other things public clouds do for things like scaling and health recovery. And they still focus a lot on the VM. Like, if the VM is unhealthy, we'll replace it, but your app could be flapping and they don't really notice or can't do anything. So, again, it can be underrated that CF's looking at the app health and it's also looking at host health because we're doing this dual scheduling thing. But that's a really interesting sophistication that, like, we're actually making sure your app instance is healthy. Even if the host is doing great, if your app isn't healthy, we'll take care of that. Good. Well, I just have one more question for you. You had mentioned, you know, that cool example of Microsoft pitching in some folks and this and that. Do you incorporate many public pull requests or the most contributors from people who have gone through the full-time committer process? I believe CF Foundation has some guidelines for what it takes to be a full-time committer, at least to be a real committer.

So, what does it look like? How could I be a random person who wants to improve the log regator and actually send a pull request? Yeah. You'd have to sign our CLA agreement. But, yeah. What does it mean if I sign that? That more or less signs over the IP for the work that you've done and allows us to take it in as an open source and you agree to the license that we have for Cloud Foundry. But more or less, it's very easy to submit a pull request and improve the code base. If you want to become a full-time committer, that's a six-week process where we help you onboard onto a team and get up and going and learn how to run a Bosch Lite and deploy Cloud Foundry. And we have a number of – a lot of our teams actually have a good mix from Pivotal and IBM, SAP, and H8 – well, SUSE now. I was going to say HP. But we have a good – a really good mix and a growing mix of contributors on all of the core teams now. Well, so I had – before we wrap up, just one, I don't know, mildly related question. Now, if I was speed reading your LinkedIn profile correctly, it looks like you spent about half your career developing and the other half in QA, more or less. My calendar edition is pretty poor. But I'm curious, like, given that and now that you're sort of – to be on DevOps-y, on the other side of the QA-dev divide, like, how do you think – how do you think in the past, like, seven or so years, like, QA has changed? Like, I get – and I ask because I ask this question – I get asked a question a lot by sort of sad-looking QA people, and they're like, but what happens to me with all of this?

And so, like, both in your day-to-day work and then just, like, the way you think about it, like, what's the involvement of QA in all this? What does a QA person do? Well, I think that I find, you know, the QA functionality and having come from that background, like, I think it's important. But I think what we've done with Cloud Foundry is we've tried to absorb that into the development team. So, we actually did have explicit exploratory testers, which is a bit different from QA, but we've converted all of those into devs on the team to, like, spread their knowledge and their thinking about testing and whatnot into all of the development teams and trying to take that in as a core responsibility into the development. I think the biggest change was just, you know, not quite being a black box tester anymore or even what they called white box tester, where it's all on the outside. You're thrown over a bunch of code, thrown code over the fence and told, you know, just can you make sure our code doesn't have bugs in it? And I think really shifting that responsibility back to the teams and ingraining in the teams that QA mentality, that that's their responsibility, that's part of definition of done, because they can help to build the right amount of tools at the right complexity level. Because I used to build so much automation that was outside, not trusting the dev team at all, to write a good unit test or anything like that. And it was just very brittle and a huge time sink. And I think shifting that responsibility to the dev team so that they can build the right tools at the right layer is one of the things that Pivotal believes in, in our pairing processes. And one of the things that we've tried to instill into Cloud Foundry.

The way you describe sort of like traditional QA, it sounds like it flies under the bold flag of a whole lot and not very much fun, right? Like just sort of like writing your own things around stuff that's brittle and always kind of being sort of like, I don't know, cynical in your approach to making sure things are working well, which certainly you have to do if you're like separate groups. I think they call it the checks and balances. But, you know, I wonder, like hearing you describe it, for the serial listeners out there, not the podcast, but people who just listen to this in order. Like in the previous episode, Andrew Schaefer and I were talking about the Google SRE book. And I think there's an interesting framing. I don't know if you think this would apply to how it seems like QA is best done nowadays, but of course a Google SRE person, which is the people who are responsible for keeping Google stuff up and running and helping make sure they have a good platform that works to dramatically shortchange the description. But they're charged with basically spending 50% of their time doing like sysadmin stuff and the other 50% of their time developing software that helps run their stuff. And it seems like, again, I don't know if that balance is great for a QA person, but that sort of simplicity and clarity of how you spend your time seems to be, I don't know, it seems like what we want. I don't know who we is. But what the industry wants from QA people nowadays is for them to start being programmers more. Yeah, I think there's that. And why not, right? Why wouldn't we or why couldn't they be programmers more? Yeah, I mean, that's certainly what comes out is if you can automate a lot of the QA toil, as it were, then what becomes needed and valuable is a, I don't know, a meatware computer figuring out interesting things to do rather than just the automated stuff. And then also writing all the stuff that automates QAing. Well, thanks for being a guest on here. It's always nice to get an overview of various parts of the Cloud Foundry world. And yeah, so if people want to look up things that you're doing, correct my math about the time you've spent doing your QA versus development or otherwise find you.

Like, what are some places you have online for people to find you? You've caught me. I don't really update my LinkedIn profile at all. Well, that's fine. And that's probably because I'm actually pretty happy working on Cloud Foundry. So I'm not trying to keep that too up to date. The places you can find me, I'm on Twitter at UETCow on Twitter. I'm sure that's probably going to be hard to sound at. But otherwise, you can reach me on CFDev or on the Cloud Foundry Slack at UE, at D-I-E-U on the Cloud Foundry Slack. So I'm reachable in a number of places. All over the place there, more or less. Well, great. Well, as always, thanks for listening. This has been Pivotal Conversations. The best way to get all of these episodes, as I was implying earlier, is if you should just subscribe to the feed and they'll automatically download to whatever it is you listen things in. And while you're doing, you know, if you want to find that, you can go into iTunes or Overcast or the podcast app or whatever and just search around for it and you'll be able to find it. And if you're not into that kind of thing and you're one of these people who, for whatever reason, likes to listen to things in web browsers, you know, there's no judgment about that. You go to SoundCloud.com slash Pivotal Conversations and see all of our past episodes forever. And also, if you want to see the full show notes, we post those. It's usually about trailing by a few days or a week. But we post all those notes over at Pivotal.io slash podcast. And we'll put links to most all of the stuff that we mentioned, including the thrilling news items. And it's always nice if you actually write to us or just do a little tweet thing, whether it's in whatever social world you're in, just so that we know that people are out there. That's always encouraging. As fun as it is to talk into a microphone to disembodied voices, it's good to know that there's listeners as well. And with that, we'll see everyone next time.

Bye-bye.

Tanzu Talk · 🍕 Get more pizza.