Hey everyone, it's Coté again. This week we have another interview from VMware Explore 2022. We've got DIMMs. And in this episode, we talk all about how the CNCF, the foundation around Kubernetes, how it operates and runs, how the various special interest groups or SIGs work, and some lessons learned from previous communities and, you know, things that they're up to. So if you wonder how the open source world of Kubernetes runs, take a listen to this interview. And hopefully next week we'll be back with our news format and an interview as well. But before we go over that, why don't you go check out the conference we have coming up December 6th to 8th, SpringOne. If you go to spring1.io, you can check out what we've got going on there, the speakers. There's going to be a lot of great stuff going over software development from Java and .NET, probably a little bit ago, just the kind of stuff programmers would be interested, but also things for the platform engineers and DevOps, kind of cloud people there, all the way up to the enterprise architect and executive level of people talking about how they're getting better at doing software practices and things like that. A lot of the stuff we talk about here on the podcast. Anyways, when you go to spring1.io, when you register, which I'm sure you will, you can use the code COTE200 to get $200 off. That's COTE200, C-O-T-E-200. And with that, enjoy this interview. Like all of us, we've been around for a while, I'm assuming, and seen many iterations of foundations and things like that. And I think digging into the operational stuff is actually interesting to see how a foundation works. And then going up higher, right? Like the first thing I was interested talking about is like, what do you think like the Kubernetes foundation or, you know, not the CNCF, whatever, like has like learned about incrementally improving how foundations work, right? Like, like what's kind of the new, you know, I mean, just to kind of explain the context, right? So I think the furthest back foundation I can remember is like the ASF, right? And then, you know, Eclipse was after that and then OpenStack and there's many other ones. And each, each of them kind of like builds on the one past it. So I'm curious, like when you look at,
2:00at, you know, the, the CNCF and Kubernetes work, like what's, as y'all are on the board, it looks like, what's like, ah, here's a new thing. That's going to be awesome that we're going to do. Oh, that, that's funny that you ask that. Um, so there's a bunch of us who float between the foundations, right? Like we started work in Apache software foundation and then we moved on to, you know, I've, I've done some work in Eclipse foundation as well, but more, uh, in the OpenStack foundation. Um, so, and we talk to each other all the time. Um, you know, so, uh, we keep asking each other, Hey, uh, what is the new thing that you're trying? Um, you know, looks like we need people, in, um, testing or we need, uh, to set up some infrastructure for testing, right? What did you do? How did you do it? And what worked, what didn't work? Yeah. And, and, and to interrupt you, that seems like one of the major, whatever, uh, innovations of the OpenStack foundation is that like this testing and performance thing is kind of a big deal. We literally picked it from them. Uh, and in fact, when I started working at CNCF and Kubernetes community, I was known as the OpenStack guy, but no longer, right? Right, right, right. I'm the Kubernetes guy. Yeah, yeah, yeah. So, you know, we, we tend to embrace the people as they come in as well. Uh, right. So, uh, so, so like, like what's, what's a practice that you think is kind of like a incremental improvement in the foundation world that, that, that y'all are doing? Um, so right now we are all talking about sustainability. How do we make sure that, uh, things are there for the longer run? Right. Um, because as you know, um, you know, um, you know, Kubernetes is, has a few years under its belt and now we have to figure out, like, how do we make sure that, uh, it works for people who are not nine to five on the west coast and it should work for people across the globe and it should work for people who are, um, you know, working part time, um, who are, who are students, um, who want to contribute. Um, you know,
4:04I joke about saying in, in, in, in a year or so, uh, Kubernetes is going to be the, uh, a code base, which has a no spelling errors and grammar errors. So I w I want to challenge this a little bit. Okay. So sustainability, right. That can lead to a lot of mediocrity too, can it? And how do you continue to take risks that are really important and making decisions that are really important because, yep, you know, you could like have a lawn and like, you like put fertilizer on that lawn all the time and like, but, but someday, like you may have to change it up. You may have to dig that lawn up. You may have to put in some bushes. Absolutely. So, uh, the way we do it is, uh, through controlled experiment, experimentation. Uh, and I'll give you one example of how we do that. Uh, right. So, uh, you know, uh, like two years ago, there was a group of, uh, folks, uh, in the community that were interested in like, how does, how do we do batch processing, right? Like how do I schedule if I want to have a ML or a job, which needs like exactly 25 pods, um, you know, spread across the nodes, uh, how do I, and it's a binary thing. You do all 25 or you don't do any of them, right? So, uh, they were interested in that kind of challenge and we kind of added the scheduler extensions to, you know, give them, hey, go forth and experiment and come back to us with, uh, your learnings. Now they have actually, it ended up, uh, uh, adding projects like Volcano and a few other, um, uh, things, right? Yeah. Now they are coming back to us to say, hey, uh, we want to have a work group in Kubernetes and a work group in CNCF level so that we, now we know what to do and how to do it right. What is the minimum set of things that we need in the base layer in Kubernetes itself? And then, okay. Okay. So what, so let's say, let's take this to another level then.
6:11So what if there are like real poor decisions maybe that were made based on a set of precedents that really had an impact on the core? I can give you two. Um, one is, uh, we recently, we are in the process of deprecating what we call pod security policy and the other one is, uh, completely removing Docker shim, uh, that we had within, um, you know, the Kubernetes repository. Um, those were two big mistakes, uh, in terms of the Docker shim, it slowed us a lot. Um, especially because we ended up defining the CRI extensions and we still had this piece of code that was within the Kubernetes community and it was slowing, uh, the work that was being done by signal in terms of adding new features. So we had to get rid of it. The pod security policy was really hard for people to use. So we ended up redesigning, uh, a new set of, uh, resources that, that is easier for people to understand. Like we wanted to make sure that the 80% of the use cases are covered with the simple stuff instead of somebody using a really complicated pod security policy. They have something easier to work with. So, so, and then my last question for, for, for this part of the show. Okay. Um, so what are some of the issues in the core right now that are just not working like pod security policy? They're always going to come up. Yeah. Um, so, so this is one of the reasons why we did the annual, uh, review. Uh, we have, uh, sent, we have, this is the second time we did the annual review. Uh, and we have enumerated a list of problems that are there in the community, whether it is people oriented, whether it is technology oriented and things like that. Um, and you, you, you, when you read that, you will get a sense of like, hey, here is where we need to invest in. And, uh, here is where we need help. Um, so simplest of things,
8:14uh, testing, right? What we have found is, um, there are people who come to different SIGs and want to work in those SIGs and want to do well in those SIGs. But what we are losing in the process is we don't have people who can work across the SIGs, right? So when we run into flakiness in the CI jobs, we don't know who to talk to, right? And the number of people who can understand things that happen, uh, across the SIGs is very few. And that is what we need investment in. And we have to make things better, easier. Uh, you know, in fact, this 125 release was the release where we had the least amount of flakiness. So the CA signals that we were getting out of the SIG release folks was, you know, the green thumbs up all the time. And we were very happy. Uh, we've been struggling with a flakiness in the CI jobs and people just end up doing a retest of all the jobs. And frankly, that costs more money, right? Like if you have a single PR and it runs a set of jobs and if it passes, you know, you know, you have good signal. But if you have to retest 100, uh, you know, 10 times, then you're essentially increasing the cost of your CI for, for, you need some anti flakiness, like some, uh, some, I don't know, anti dandruff shampoo for the organization. Yeah. That's tragic. That's what comes to mind. But so, you know, you covered a little bit and we've talked a little bit about, um, introducing new things into the overall project. And, you know, we, we talked a couple days ago with, uh, Lucas from, uh, loft. And so one of the discussions we had with him is like, you know, between the, the more open source community stuff that you do. And then as you go up into loft, like what you do, like one of the, one of the things I think we're interested in is where is that cut line of like, we're not going to do that then. And, and, or, or to put it in a more positive way, that's a great thing for someone to add to the community on their own, the overall ecosystem, I'll use that word. Right. And so like, and, and the reason I asked that, well, not, I mean, I'm curious, but you know, especially us being a VMware, like, and everyone else is interested in
10:17like improving it for developers. Yeah. And so there must be a line somewhere where, where you're like, well, we're not going to do actual code stuff. So like, but where is that level where you feel like the, the kind of the CNCF and the Kubernetes ecosystem is like, yeah, we don't really do that. Like that's, that's someone else's thing. Yeah. Um, so I'll give you one example where, uh, we have a, a line that we have not crossed yet, which is, um, how do you stitch multiple, um, Kubernetes clusters together? All right. That's one example where, uh, people are innovating in the ecosystem. There are startups, there are different projects. Loft has the V cluster thing, for example, right. Um, so we have, we did two experiments in that area, but we ended up giving up on that idea. For example, we did a cube fed one and cube fed two, um, Kubernetes Federation literally. And, uh, those were experiments that we tried, but we did not bake those things into the Kubernetes code base itself. Right. We kept it outside, uh, but we were innovating, we were trying something out and we didn't like what we saw and we kind of like gave up on it. So, uh, if. Why'd you give up on it? Uh, we, because we didn't like the APIs, we didn't like the way it worked. So, uh, what did you not, I, I, I keep digging a little bit. What did you not like about the APIs? Uh, so, uh, what, uh, like, QFed V1, right. Um, that was a kind of like, uh, I know Solomon is here. Uh, so, uh, it was like Swarm, right. Uh, you know, you have the same API, but underneath you have something else. Uh, so that, that was one experiment that we ended up trying. And, you know, that was not a good choice for us. So we said, okay, fine. Let's remove that. Let's try QFed V2 where we'll re-imagine the APIs. Uh, but that didn't seem enough for us. And no,
12:17the way we end up figuring these, this out is nobody's using it, right. Uh, there's very few people using it and we are not getting enough feedback from the people saying, hey, we are using it. Uh, you know, if you can make this small thing better than, you know, if we don't have that feedback loop coming in from the community, then it means that it's not of much value to, uh, people in the community and we stop doing, uh, you know, things where, uh, hey, one vendor is driving it and so we are adding some code and then, uh, you know, but if, if you see it's a dead end, then we stop, we deprecate, and we get it off it in a, uh, new release. So like another, another topic that, um, you know, I, I talk with analysts and press and, and users, uh, companies that use Kubernetes and like maybe, maybe it's because I'm from VMware, but they always like to throw out like lock-in and things like that. And, you know, like how like, like that is like, if we, we want to make sure that we have a, you know, you, you probably remember Simon Phipps, not that he's gone, but like, yeah, he had this great phrase, the freedom to leave. Like, that's really what, you know, his theory was, that's really what people are looking for, uh, with things. And like, um, I mean, how at a, at a foundation level, like, how are y'all thinking through, like, we want to make sure that we have that freedom to leave, that like, like, like, how do you engineer that into? Cause I mean, you know, obviously it's like, oh, it's a source, right. But like, there, there must be a lot more thinking put into it as far as the entire ecosystem and how, how, uh, it all works. Right. Uh, you know, some of the obvious answers there is like, we make sure it's Apache license. Right. Right. Um, and one example of freedom to live is, um, you know, Grafana labs, they were working on a project called cortex, uh, in, in the community. And, uh, they said, okay, we are not getting the kind of engagement that we are getting from, uh, that we need. Uh, and, uh, we are trying, we are trying to do all the engineering by ourselves. So, you know, out we go. Right. Uh,
14:22so they ended up doing, uh, open source, single vendor open source, uh, called me, my mirror or me mirror. I don't know exactly how to pronounce it. So, you know, now they are off doing their own thing. And that's the same for anything that I've listed out as extensions, right? Like, um, you know, I, uh, listed a whole bunch of, uh, extensions, uh, and be like, take the CNI, for example, or the CSI. How many windows do we have for CSI? How many windows do we have for the CNI? Right? A lot of them. Yeah, exactly. And those kinds of things were baked into Kubernetes right at the beginning. And then we kind of like kick them out saying, hey, go explore, uh, uh, the, the things that you want to do. And you don't have to talk to us. You don't even have to like, you know, uh, get, uh, unless you run into a problem, you don't even have to do anything, uh, with us. And you can innovate on your own. Right? Uh, so you have Andrea from VMware. We have, uh, Cilium. We have, you know, so many other, uh, projects. Uh, we works had a project too, right? Like, so we've met, um, so, uh, that is an example of like how we are un, un blocking the innovation, um, you know, in the community. So I just have a last question for myself that I'm curious about. What gets you so excited about the work that you do? What is it that impassions you to talk, get up there and like give this very thorough and deep explanation of SIGs? Uh, the people, right? Uh, I look forward to talking to, uh, all the friends that I made over the years, right? Uh, and it's not even necessary that they are doing things with me right now. You know, uh, the other day I was talking to Henry Handel from AWS, right? And, uh, the last time I spoke to him was like 15 years ago, but it felt like yesterday that I talked to him, right? So it is the people, uh, that bring you back to the work and it's the P like software happens when people get together, right? Uh, so, uh, and sometimes I also think that the people problems are more difficult than the software problems as well. Uh, because wearing the different community hats, we've seen,
16:25you know, issues, uh, uh, people related to people and those are harder to solve, uh, than, you know, the software stuff. So it is, uh, essentially, you know, the people that end up making things good for me. So you, you probably know the dates better than me, so you can fill it in, but, uh, KubeCon North America is coming up this in October. Yes. Yeah. Yeah. Yeah. Good enough. Yeah. Yeah. Third week, I think. Yeah. Yeah. And I mean, over, over recent years, like that conference has grown incredibly, I mean, it's over 10,000 people now, something like that. And, and so for people who want to go to that, what's your advice for like, you know, you can't talk to 10,000 people or like go to all the sessions, like, right, like how, how should someone like plan out having a good, useful time at that, that, that event at that scale? Um, so, uh, what I would try to figure out is like, what do you want out of it? Right. Um, so, uh, for example, when I go to KubeCon, I go there to meet my friends, right? Like I'm not really interested in the sessions as such, but I go there so that I, you know, we work with, uh, each other for, you know, for a very long time. It's like a couple of times a year when we can actually get together. Right. So that is important to me. Um, but you know, for example, uh, we coach and mentor, uh, students that are coming to, um, KubeCon and we actually have, uh, scholarships for people to come as well. What I tell them is like, you know, plan ahead, um, figure out which sessions you want to go to, uh, figure out who do you want to talk to, make an effort and come up with a topic that you can talk to somebody about so that you can have a conversation and they will remember you rather than, uh, you know, just, uh, doing a handshake and walking away. Like you're not going to remember, uh, the face or the name, but if you talked about something, then you'll say, okay, I remember that guy because, or that girl, because, you know, uh, we talked about, you know, how to do something better in this
18:29part of the code base. And that becomes something that you can follow on once you go back from KubeCon. Totally. So here's another thing that I would ask. What do you want to know? What are you trying to learn? So then what do you ask? What are those questions that you want to ask people? Yeah. Like write those questions down. Yeah. Because then you can find those, then you can talk to those people. Yeah. And when you're talking to people, you might just think of a question. Yes. It's that preparation. There's also pancakes apparently at the time. Oh, yes. Yes. Yes. Yes. Yes. This is the pro advice from the journalist. Yeah. Doing your prep work. Thank you. Knowing what you want to talk about. Thank you very much, Michael. I think, you know, I haven't really like narrow, thought about that as, as an aspect. It's sort of like organically evolved if you go to enough conferences, but that is a good idea of, I like your idea of people remember a topic. Yes. Chances are they remember a topic more than a person. Right. For better or worse. Yeah. But, but that is, that is good to kind of prep out. Like, here are things I want to talk about and find out about and the specific questions that I ask people because otherwise you just kind of ad hoc float around. Right. With, with things. Yeah. So that, it's almost like, and if you have the, and if you have the, you know, the interest, write something down, like write a, write a blog post, even, you know, write a, write a series of tweets saying, these are the questions I have, you know, I'm looking to try to get these questions. So that's a lot of nice prep work you can do beforehand. And it helps you kind of with your profile. Not everyone wants to do that, but it can be a way. Yeah. Do it on the flight over. I mean, you know, you've probably seen. Yeah. Yeah. The flight. The flight. Are you really going to watch another Marvel movie? Like what, what? No. Yes. All right. Well, great. It was, it was, it was really, it was really fun talking with you and getting that, that, that kind of overview of the, uh, the internal workings of, of how things float around there. So thank you, Michael. Yeah. Well, we'll, we'll see you around. Thank you, Alex. You're welcome. Thank you. Yeah. Thanks.
20:29Bye. Bye. Bye. Well, I hope you enjoyed that. If you want to get the show notes for this episode and find out how to subscribe to Tanzu Talk as a whole, you can go to TanzuTalk.com. You should subscribe to it and all of your podcast players, whether you listen to it or not, if you have children or loved ones, or even enemies, and you somehow can subscribe to the podcast on their phone or their web browser or signing up for an account for them. You should do that too, whether they listen or not, you know, that's a bonus as far as I'm concerned, in addition to the downloads that we get. But kidding aside, if you are interested in more content like this and hearing case studies about people talking lots of things about programming and DevOps and platform engineering, you should come to our annual conference, December 6th and 8th, 2022, in case you're listening in the future. It's going to be in San Francisco. And if you go to spring1.io, you can check it out, look at the speakers and the talks that we have. It should all be pretty fun. So if you like it, when you go to spring1.io, you can register. And if you use the code COTE200, you can get, that's right, $200 off your registration with the code COTE200. And with that, that's it this week for Tanzu Talk. And we'll see everyone next time. Bye-bye.