So, Paul, you're big into food. I have a question for you. Now, do you have a position on drinking broth? It depends. I have opinions about this thing called bone broth and how it's really just stuck with a bunch of woo wrapped around it. What is woo? Well, I think of woo like a, I guess like a, I'm going to probably get myself into trouble, but like a chiropractor or a homeopathic doctor or a... Mystical. Mystical. Yeah. That makes sense. Okay. So, you think beef broth has a certain amount of mysticism associated with it? No, bone broth. Bone broth. Oh, okay. Okay. Bone broth is really just stock because stock is just bones that you boil in water to make a delicious stock with. And then you take that stock and you can either turn that into broth, which is kind of like a stock without really much in it, or you have a soup, which is when you actually add other things into it to make a sort of more wholesome, hearty thing. Okay. Well, this is delightfully more than I was expecting. So, let's, let's dive into this just a little bit and then we'll get to the technical stuff. So, broth is a superset of stock and bone broth? Like, broth is the whole collection of things? Yeah. So, bone broth is just like a marketing term that people have added so they can sell it as part of a diet that people will buy their, like, body transformation books over. It's not really a separate thing from regular broth or stock. Okay. Okay. So, it's a... They'll argue that the process is different and they'll put nutrients out of the bones with extra vinegar and a bunch of other nonsense. But the science ain't there. I see. I see. Okay. Okay. So, the mysticism aside, do you have... Is drinking broth, is that okay? Is that a thing or is it just flavor? Like, what is a terrible idea?
2:01What do you think? If it's good enough for Baby Yoda, it's good enough for me. When does the soup become a stew? I think it's a stew when there's more other stuff than there is soup. The particulate water level is higher. Big chunks. I think it's sort of like, I mean, I guess everyone says, people say that the body is mostly made up of water, but we would not call the body water. There is like, it is more on the stew side, philosophically thinking. I guess it's pre-stew. Yeah. I guess I just figured it's when a fork comes into the equation. You're in stew territory. That's probably a good way to look at it. When you need to upgrade from a fork, a spoon to a fork. And I don't think a spork quite counts. Well, this isn't Kentucky fried chicken. So, right. Spork's not welcome. That's good. That's good. Well, good. I think we've covered the formal introductory part of the podcast where we talk about food or the weather. So, that's good. So, Paul, why don't you introduce yourself? My name is Paul. What do you do around here? I'm on the developer advocacy team at, I guess, now VMware, since the acquisition went through recently, previously pivotal. And I sort of focus on Kubernetes and other more operations slash DevOps things as opposed to squiggly lines and periods and equal signs in Visual Studio Code or whatever we're using these days when we're writing code. So, I'm a developer advocate, but really I'm more of a ops advocate. Well, you know, they need advocating too. Yeah. That's what I always tell people is you do the ops stuff and the DevOps. And of course,
4:01the Kubernetes, Kubernetes, the kubes, all of that stuff. The kub is very needy. So, we definitely have some Kubernetes. So, I wanted to have you on to sort of kick off the year, talking about speaking of Kubernetes and operational stuff. And, you know, I realized, well, you know, I know you well, but I knew it would be especially fun because I watched that video you did in Warsaw a while back. By a while back, I mean last month. So long ago. It seems like a long time ago. But yeah, yeah, I hadn't seen you do the overall Kubernetes arc of everything. And it was very good. It was nice. Everything kind of fit together well. And, you know, you also got to do a speak in Warsaw, which I'm sure was fun. And I think I spoke in that same room. That's like in a big university building somewhere. Yeah, yeah, it was. Yeah. Anyways, so we all have only a couple little news items that I wanted to go over and just some pointers. And then we'll dive into, you know, Paul's exploration. As Paul mentioned, I think it was on December 30th, 2019, that VMware finalized its acquisition of Pivotal. So we're all VMware people. I think we all have VMware email addresses as well, which is very exciting. We'll have to trade with each other what they are. It's always fun when you join a large company to see what your email address ends up being. And as you can imagine, as another thing, I've been doing a lot of figuring out what's going on in VMware world. And somehow I stumbled upon, I'll put a link to it in the show notes, which we post at pivotal.io slash podcast. Maybe it'll be another URL sometime in the future. But there's a good video from, I think it's like VMware tech days that goes over what Project Pacific is. And like, I think there's, it's for me, been one of the more mysterious sort of things that VMware has been doing in the Kubernetes space. But this thing finally, this thing, this video finally, I think has a really good technical explanation of what's going on there if you wanted to dive into it more.
6:01So I would refer to you to that. And I think it's from one of the project, product managers. And then also we and Forrester released this thing called a total economic impact or T-E-I, which unfortunately it's too bad they couldn't forget how to make it spell Thai because you can't really, you can't really like pronounce T. How would you pronounce that if you have, I can't think of a, I guess, I guess Terry in French, but that's a T-H-E-I. There's gotta be some way you would play. Yeah. Yeah. Anyhow, that's a report that we have out. You can read. And it goes over. I think, I think if I remember it's based on like five or six or eight or so customers that Forrester interviewed. Something in that area. Cause we did one last year or what they did one last year. And it kind of goes over the, the improvements that you get, the return on investment and productivity and stuff. And it's fun reading through. It's always nice to have aggregated numbers like that. Yeah. It's fairly realistic. I mean, they'd kind of create an aggregate customer out of all the interviews and talk about the general benefits you get. And it's real anecdotes. It's not made up stuff. So it's, it's actually a pretty useful read, as you say. So with that, Paul, Kubernetes, give us an overview. Wow. Okay. An overview of Kubernetes. Yeah. What is it? It's a thing. What runs containers for you? See, that's a good start. This is one of the conclusions. As, as I was, you know, watching your, your overview, watching yet another overview of Kubernetes, I was like, you know, as always, I complexify things too much. There's a slide you had where you're like, oh, it's really easy to run this on your laptop. And then here's a bunch of servers. And now you need to do that. And it's sort of like, it's after all these years, I often forget that simple jump that like the whole point is like, you need to run like thousands of these and you know, that seems like a hassle. Have you changed your opinion? I mean, if we go back to the very beginning, why do people care about containers? I mean, we've been talking about it for years, but has your thinking evolved at all about even in the first place, like why containerize? So, I mean, I think the value of containers,
8:02especially, you know, the, the canonical, you know, Docker image version of containers is the ease of building and the ease of sharing. I think that kind of trumps most of the other things, especially now we have a bunch of options around the lightweight VMs that behave like containers. And I know, I mean, I think 12 months ago, I kind of wrote a snarky blog post about the future of Kubernetes being lightweight VMs. And I still feel that there's a strong place there because there's still a lot of angst and concern about the real secure isolation of containers and what that looks like in a multi-tenant world. And I think it would be a lot of work to make a multi-tenant safe container hypervisor, whereas a VM hypervisor is more or less multi-tenant safe from the outset. And being able to, instead of having a bunch of servers with hypervisors with a container engine, sort of all stacked up, being able to just have the Kubernetes control plane and a bunch of hypervisors, sorry, multiple Kubernetes control planes, either sharing that pool of hypervisors and sharing those resources from a infrastructure point of view, from a cloud provider point of view, even from a, you know, inside your data center, private cloud point of view, having that brings a lot more potential for higher density of, you know, more bin packing of your workloads. So, and I, and I think that bin packing side of things, the better density is probably the killer app there, more so than the security and the multi-tenancy.
10:02So I still think we're going to see it go that way. But I still think we're going to continue to see the, that Docker file, Docker hub type system for building and sharing. But building and sharing for you are the kind of main things. Yeah. So it'll still be effectively a container to you or I, but it may be a lightweight VM when it's actually running somewhere. And that's going to be an implementation detail that we don't even care about. Yeah. That's a good way to start wrapping our hands around the impossibly open-ended question of so Kubernetes, which is as, so, so I, as, as you're saying, like the two valuable things that you get, the reason you do it is, is I guess that building and sharing. I mean, is, is that, and, and what is the sharing part of it? Cause I guess the building part is just like, it's easy to package up your software to run basically. And it's fast. And then what's the sharing part? The sharing is the Docker push, Docker pull that you do to a, either the Docker hub up in the public or a private registry like Harbor, where it becomes very easy to say, here's my app. Anyone can run it. And then anyone can go, Hey, I'm looking for Bob's WordPress image. There it is. I can grab it and run it without having to really think about it. I think that's the difference, right? I mean, Coté from you and I, we're all of us, we're all old people now. I mean, from the years ago of, here's 20 set of instructions to get this thing up and running. I got to start this up, install this, configure that. And Hey, it's different than two weeks ago. Cause something else changed. Cause everything is packed together. Right. Probably at that point, me running out of my machine will be completely identical to you doing it. Cause all the dependencies, all the config is literally wrapped up in one immutable thing. Yeah. Yeah. And if you, if you ever built VM images using a tool like a packer, you know, that packer is probably about the best you get and it's still quite hard,
12:05right? Cause you need to spend a lot of time figuring out which base OS, and then you basically running either a big, long script or you're running chef or puppet or something to build it all up. And it's going to be packaged and it's going to be shipped up to your cloud provider, et cetera. And so that was quite a lot of work for just any, anyone to do for people that building OSes isn't in their wheelhouse. Whereas with Docker and the ability to build on top of other people's Docker images. Now I can be like, well, I just need a Java image and then I know how to put a jar in my, you know, Java image. So I only need to know the copy command in the Docker file versus knowing how to build an entire OS from using the Docker file. As you were mentioning, you know, you got all your bin packing concerns, like sticking as many things in a constrained space as possible, metaphorically speaking. And like how much, how much of like the container of the value from the container Kubernetes world do you think comes from that? Like, is that like a huge deal or is it just like a fun side of it? I think it's a fun side effect and I'm not sure it's even as real as a lot of people think it is. You know, there is definitely more overhead running VMs and there is running containers. But at the end of the day, most of the overhead is in whatever the application you're running is itself. And you're not reducing that. So you might have a couple of percent here or there because of what the VM takes up. But I don't think that it's really that, you know, they used to talk, you know, tens of VMs or thousands of containers. I don't think that's real. That something like that might be real with serverless or, you know, K native where you have like a container on demand based on like web traffic or something that might become more real.
14:06But you're making other sacrifices there. So I don't think the bin packing versus VMs is really that big a difference. And then, of course, Kubernetes and its scheduler have its own way woes. Kubernetes doesn't really try and optimize the bin packing. So it will just distribute VM. It will distribute the containers to say the worker nodes that are there. If you create new worker nodes, it's not intelligent enough to redistribute them to more evenly fill those machines. And so, you know, I don't, I think other systems probably have better bin packing. Like I know VMware does a really good job of moving workloads around to optimize the hardware using Vmotion and stuff like that. Whereas, you know, the default Kubernetes schedulers don't do that. There's no reason you couldn't write your own scheduler to do that. But if you're at the point where you're rewriting your own Kubernetes scheduler, you've probably solved every other problem in computer science, including DNS. So that brings up another, another thing I'm always curious about. So tell me if I get this inaccurate just to like lay out a summary. So, you know, on your laptop, you build a, you put your junk in a container, your stuff. It's not junk. It's wonderful handcrafted things, some software you've written and it's packaged in a container. And now, you know, your buddies can access it and do whatever you can run it as basically just an application that's containerized. I didn't really say very much there, but we'll move on. And then you're like, ah, but then also I want to run like, you know, 500 instances of this and it's like a multi-tiered distributed application. So it's got all these different components running around.
16:06So I have like 50 containerized things that I have to orchestrate running. So now I need a whole other system called Kubernetes. I need an orchestrator. And what Kubernetes will do is it'll let me specify like, here's the, I don't know what words people use for this, but like if I were to draw a diagram of how all these 50 things are related and the dependencies that they have and which one, what each one does, the architecture, here's what it looks like. And here's kind of like, I need something to go off and like place all of these on various computational things on the VMs running and make sure that they're in a state that I like. And, you know, if there's drift from my desired configuration, like blow them away and restart them and then make sure that they can do the networking, talking with each other. And then maybe they also need some storage and, and it's sort of, you know, so Kubernetes does a lot of that stuff. However, is that a good, a good summary from the get go? Like what did I hand wave over? Yeah, I think so. I kind of like to think of Kubernetes as effectively an IS for containers. When you think about any cloud, you're saying, I want CPU, I want memory, I want disk, I want network. And you're requesting the exact same things with Kubernetes, just for containers instead of VMs. Um, you've mentioned describing your, like your architecture. You don't really do that. You describe each individual piece. So you still need to keep your architecture in mind. Uh, it does blur when you get into service meshes where you do kind of start describing your architecture or you let your architecture, um, uh, sort of self-describe itself just by a set of conventions. Yeah. No, no, no, no, no, no, no, that's exactly like the blurry line I'm, I'm interested in. Right. Is like, so, so what, what does Kubernetes not do along those areas? Right. Like, like, because, and, and you brought up one thing that I think it doesn't do, which is basically like,
18:06uh, a service mesh and, and my, you know, a service mesh is basically like, I've got all these sub components and systems in my application, whatever that may be. And they need to work with each other. And the service mesh does, that's the outcome it achieves basically. Uh, you know, being able to look up services and having all of them talk with each other and so forth and so on. And like, that's not part of Kubernetes, right? Or is it, or how does that layer in? It's a tough question. I think. A lot of us aren't really sure where the demarcation should be. Uh, and Kubernetes has been very cautious about how much it takes on. Um, because it does try and follow that, you know, do one thing and do it well. Uh, and you can see that through its architecture, um, through how it doesn't do a lot. And then it uses controllers to do all, a lot of the, uh, value added stuff. Uh, and then things like Istio come along. There are a set of custom controllers. That do even more advanced things. I think as we. Develop opinions that are strong, that work well for everyone, they then start look at like moving towards being part of Kubernetes itself. All right. Certainly anything that's like value adding on top of your application is generally outside of Kubernetes, right? So Kubernetes is below your application. Um, Istio is kind of in between and above your application. Uh, and so I think that's definitely outside the purview of Kubernetes, uh, especially say Istio, it's like actually modifying your application from the inside, right? Cause it's putting these sidecars in and it's doing these weird network tricks to force your application to talk through those sidecars. Uh, so anything that's actually manipulating the behavior of your, uh, application, a should not be done by a platform that potentially someone else is
20:08running. Uh, and also be in my opinion should be done with very thoughtful consideration, uh, because now your app doesn't behave the way you asked it to. And therefore what, what things is it, you know, what strange obscure bugs could be introduced that might surprise us later on. I like that rule of thumb. I like that rule of thumb that if, if you're modifying how your application works, it probably shouldn't be in the Kubernetes layer, like, and, and probably being an operative word there that like, you know, maybe there's something that you would put in that layer, but like, it's likely that it's something else you, you would add on top of it. Cause that's, that's, that's pretty clear about where the, uh, the cut line should be. Yeah. When you do that, I mean, one thing I want to back up on Paul, cause you mentioned, I like the IS for containers. I think that's a good thing. When Coté was pushing on kind of the architecture piece, I mean, it's still true that people say a container should be one process, right? You shouldn't be jamming a bunch of different, like a whole multi-tier architecture into a container, although you physically could, I guess, but that's still the rule of thumb, right? That you should be putting one process. You will have more components in your architecture than a traditional, maybe three VMs representing each tier. Am I right? Yeah, absolutely. And actually that's one of the smart things that Kubernetes did with the pod concept is the pod is your, like your workload, not your container, right? And so the pod can run more than one container. And so the pod basically brings up the actual execution environment for your container. So it brings up the C groups, et cetera. And then your various containers share those things. So they share the network namespace. They share the IP addresses. They share volumes. If you have volumes. And so that allows you if you need to have more than one container for your application. The canonical example of course is PHP applications where you have PHP FPM and
22:12you have NGINX and they're usually sharing a socket over a file system. And so you would run those in the same pod in different containers, sharing a volume or sharing local hosts for communication. But many architectures you'd have multiple pods, right? In a traditional app. Yeah, exactly. So if you were like, if we're WordPress, you might have PHP and NGINX in the one pod, but you certainly wouldn't put my SQL in that same pod because it's not part of that, that piece of the architecture. It's not that like atomic scalable unit. And so that's important though, because as you said, right, you're, you're imposing your architecture still onto this system. It's not doing it for you. So when someone looks at those primitives, they have to do this, right? Right. I can still make horrible mistakes in this layout. No. And so how do I lay this out? And no YAML is the language of, of love and of Kubernetes. So how am I using YAML? Yeah. Pause for Paul to smile. How do you use that to lay out? I mean, lay it out for us, right? Like though I'm an architect or I'm a developer. Now I've got a bunch of containers. I'm ready to deploy. What's my next step? How do I think about those lines? How do I think about what goes in pods? How do I, how do I do this correctly? Yeah. So you think of a pod as your like a canonical unit of scale, right? So if you've got a traditional three tier app, it's simple. You have three pods or, you know, three replica sets or three deployments, right? Then you have to connect them together. So you have services and generally if it's one pod talking another pod, you just have the default service, which gives you a cluster IP. And that basically becomes the IP address that you would address a given pod. So you assign a service to a pod. So it's WordPress, the, the,
24:12the pod, therefore you have WordPress, the service, and that gets that IP address. And that IP address is accessible from anywhere in the cluster. And it also inherits a DNS name from kube DNS. So if it's called WordPress, you now have a DNS name called WordPress that any pod can access on that name and get to your application. If it's a replica, a replica set or a deployment where you have more than one copy, it low balances those for you. And so you've got your WordPress, then you need your database. So you have a WordPress service. You have a database service. Your database is probably called a database or WordPress dash database. And so in your WordPress configuration, you can actually hard code the, the path to your database as WordPress dash database, because you, you know, that's what your service is always going to be called. And then, so that's how you get the WordPress and the database talking. So now you have a working application, but you want someone outside to be able to access it. So the WordPress service, you need to elevate that to be either a node port, which gives you an ephemeral port on your worker nodes that will NAT through to it, or you use a service type of load balancer, which will actually configure a load balancer to balance it down. And that gives you a static IP and a sensible port of 80 and the ability to then add DNS to it and have a general purpose. This is how you access a website on the internets. And then you have a step further where you have a bunch of apps working together, and you might use an ingress controller, which lets you put those layer seven rules about which service should be communicated with based on like the URL path.
26:13And that sort of gives you your kind of one branch of an application. Obviously WordPress is not a microservice. But if you have a bunch of microservices, they would talk to each other via those cluster IPs, via their DNS. And so you're getting that very rudimentary service discovery from DNS. And that's usually enough for a very basic set of microservices where there's not a lot of complication. You know what everything is, and you can just run it. When you get to more complex scenarios, that's when you need to look towards things like service meshes where you don't necessarily need to know the entire architecture for what's going on. You can kind of let the service mesh help you with more advanced routing, more advanced service discovery. And then you get some added things like if you really want TLS communication between each of your microservices, if you want cert-based authentication between your microservices, you can do those things through a service mesh. Obviously, there's a whole heap of considerations about latency, if you're going to have some external process injecting TLS communication on top of your application. But those are all things. If you're at that point, you're probably thinking about those effects anyway. If not, well, you'll get woken up at 3 a.m. every day. I mean, it seems like the two things you really have to grok in this is kind of the placement questions of like, where do I stick things? Because there's replicas, there's tiering, and then network connectivity. Like, I feel like I have to know both of those things to be dangerous here. So in a traditional, I don't know, whether it's a startup or a big company, who's doing these things? Like, maybe I'll be fired before the end of the podcast, but are developers supposed to use Kubernetes? Or realistically, should I stop when I built a container and then some other mysterious person comes in and lays this stuff out right?
28:16Yeah, look, assuming you're not a brand new startup that has never computed before, you probably already have someone or something that's managing your infrastructure, right? So whether you're an enterprise or a startup, someone is clicking around in the Amazon UI or someone is using CloudFormation or someone is using Terraform, whoever is doing that in your organization is probably the one that's going to be doing a lot of the work defining the Kubernetes resources. I think it's not really something a developer should have to know. But in your org, it might be something you've decided that they should know because of the way you've structured yourself, because you're targeting a specific, you know, organization slash software architecture. And you sort of Conway's lured yourself into a particular way of working. So that, like, is another favorite topic of mine to contemplate in the Kubernetes world, which is, like, you know, there's this notion of, like, it's a platform for building platforms. And, like, what does that mean? Yeah, so I have some opinions. So when you have an IT group, an operations group, a DevOps group, you're either building a platform or you are the platform, right? You can be a platform made out of meat that Jira is your API, or you can build a platform made out of Kubernetes or out of Amazon or out of whatever your, you know, vegetable of choice is and wrap some extra tools around that, you know, wrap some monitoring, some logging, some access control, and then you have a platform, right? Which is ideally self-service. So the goal is to get self-service application deployment to your developers. And that could be through a real API.
30:18It could be through something Cloud Foundry-ish. It could be through something doing GitOps. So it could be a combination of Concourse and Spinnaker. But the goal is to get to that self-service to your app devs without them having to do a bunch of things that require large portions of their brain, right? Because you want large portions of their brain focused on writing software that doesn't suck, right? I guess the cut line, the thing is that, like, as we were kind of discussing, like, so you get Kubernetes up and running and you still have to, like, figure out how you're going to deploy and manage and specify everything, right? There's not like, it's not so self-service as one would assume. No. And, like, backing up, even more than that, you need to figure out how to actually build and share your images, right? You don't get a build service with Kubernetes. You don't get an image registry with Kubernetes. So, you know, you could, if you were small and able to do so, you could use GitHub and GitHub Actions and Docker Registry and cover those things. But most of us can't do those things for any number of reasons. So we build things that look like those things and enable those for our developers. And so, you know, I was reminded of this when you're going over, like, you need to add in a service mesh, right? So it would seem like a service mesh would be a platform that you build on top of this platform. Like, is that maybe not complete, but it's an example of, like, you use the, I think a word Kubernetes people use a lot is, like, there's primitives that are defined. But you use these primitives to say, like, all right, hey, primitives, here's how you can organize yourself into a civilized way of doing service management. And it's kind of an opinion of that. And then, I guess there's all sorts of other components that you would have, in a platform that you can use those primitives to build.
32:19Here's the thing. And so, you know, related to that, so, you know, you got to, you got to, you got the cloud native 12 to 16 factors of doing software, right? The sort of cloud native way of developing software. And, like, is cloud native software, like, based on the constraints of containers and Kubernetes things? Or are containers built around that? Like, what's the chicken and the egg issue between those two things? 12 factors are really built around the constraints of a PaaS-like platform. I believe they originally came out of Heroku. And having a similar PaaS, obviously, we like them. The interesting thing with Kubernetes is you can start to break some of those factors, right? Because Kubernetes is a more feature-packed platform than PCF or Heroku. Heroku is an opinionated platform versus Kubernetes being an unopinionated platform. So, basically, when we are building the platform on top of the platform, we're adding our own opinions to that platform. And those opinions don't necessarily mean all 12 of the factors. So, with Kubernetes, you can add persistent volumes. So, backing services become less of a concern. You can do configuration by a number of different ways. So, doing your configuration as environment variables. You can handle logging in different ways. So, doing your logging only to stand it out becomes a potentially optional thing. So, the 12 factors, like, they're always good practice. But they're not as important on Kubernetes as they are on a PaaS-like system. And that gives you the added benefit of, if you really want to, you can run any old garbage pretty successfully on Kubernetes.
34:19Right? So, any legacy app, if it runs on Linux or can be forced to run under Wine or something similar like that, you can run in Kubernetes with a little bit of work. And run them quite successfully. And so, it's also becoming a place where you can look at running your legacy applications as well as your fancy new cloud native applications. So, really, with the unopinionatedness or openness, you're not, you can basically do whatever application model you want, for better or worse, right? Like, there's not, it might be best to do things in a cloud native way because it's more scalable and you can take advantage of all the fun stuff of cloud things. But really, like, you could do whatever, as you're saying, with refuse. It's a kitchen drawer filled to the brim with very sharp knives. And you can grab any of those knives out of the drawer that you want. Just be careful because they're pointing in different ways and you'll cut your hand off. Now, how many knives do you think you actually need in your kitchen drawer? Some people say, like, two, right? What do you think the, what do you think the bare minimum is? Yeah, I mean, I like knives, so I have a lot. But really, two, maybe three. Maybe like a paring knife and a chef knife, right? And I guess if, I guess if you make a lot of, oh, yeah, that's true. If you make a lot of bone broth, maybe a cleaver. I went for shaving with, maybe Paul's pretty, pretty manly. That's right. No, I use the axe to shave. That's true. I'm going to carve up these parsnips and then shave with the same knife. Yes. Yeah. If you look at, like, Chinese cleavers, for example, they have, like, a lot of them just use cleavers, right? And they have, like, the vegetable cleaver and the meat cleaver. And so they've figured out a way to effectively have the same knife be multi-purpose. And it just depends on the thickness and the angle of the blade based on what, so it's
36:25always good. Like, I watch a lot of Sichuan chefs on YouTube, and they're always using these big cleavers that we would think are for chopping bones to, like, do very, very fine knife work on vegetables and stuff. Mint some garlic. Right. Hmm. Yes. How did we get on the knives? What happened? Oh, it was, there was a metaphor that Kubernetes is like a kitchen drawer full of sharp knives. And you can do whatever, you can shave, mince garlic, chop bones in half, whatever style you want to do, you can apply it to there. So what, like, so if we were to look forward, as I like to do towards the end of a discussion, like, what are the big, let's say, let's say, two to three things that the Kubernetes community is trying to, like, sort out now? Like, what, like, is it like multi-cluster management? Or is it sort of like, I don't know, instead of YAML, we should use XML? Or like, what are the big, like, issues that people are dealing with? Yeah, I think multi-cluster management is definitely a thing. And that's from both the, like, the operation side of managing Kubernetes itself, but also for, I have my app, I need to deploy it to multiple regions, to multiple clusters. What does that look like? And so there's been various federation efforts with Kubernetes, with varying levels of success. And by varying, I think I mean, all of them were unsuccessful. I'm not sure where the current state of that is. But there's been a couple of attempts that haven't really gone anywhere. I think a lot of us are now starting to really think about good ways to handle the build and storage of images. So like stepping, stepping back from Kubernetes and saying, it's time to add some somewhat opinionated pieces to Kubernetes around building and sharing those images.
38:29So I think if you talk to any vendor that sells Kubernetes, they'll probably tell you that they have a thing in the pipeline to handle those things. And that's sort of like, it seems like one of the major connections between developers and Kubernetes stuff is like the image registry, essentially, right? I mean, even yesterday, right? We shipped Spring Boot 231 that with one kind of build command will actually generate a build packs produced container. I tweeted about it last night and people liked it because it's super easy. Even a dumb person like me can add one command and immediately I have a Docker image that's packaged it up correctly, right? So you're not guessing on which base image to use. You're not guessing on how to stack it all up. Nope. I'm just a dev. I just build and I happen to get a container in my registry. That's really nice to link the world. Yeah, yeah, that absolutely is. And I think the good thing about that is it also makes it very easy to do such thing in a CI system because obviously in the real world, you're not actually building an image on your laptop and sending it to a registry and then running it for production, right? You have hopefully a CI system that is doing that so you can sort of have some traceability of where the image came from and how it was built and keep some sort of bill of materials and stuff. So then maybe the last question, because I think where we've ended up, you know, along with meat cleavers, things like that, is there's sort of this picture of like, so Kubernetes basically like takes your containers and like, as you're saying, it's an IaaS for that layer, right? It like places them, it places them on, on computers and make sure that you have proper network access between things. And then you also need some storage and boom, there you go. And there's a bunch of other stuff in there. Right. But that's sort of the end goal. Now, then as we were sort of just getting into like, but then you need to like actually deploy software to it. So you need to develop the software and that needs to coordinate and you got to do this. You got to have some registries, all these things like, and you know, what do you, especially based on your experience in the OpenStack world, like, like, what do you think is a good
40:33way to like layer on these upper layers of things? Like, like, like, is it good for the Kubernetes community to like have an opinion about that or to sort of like, let other people handle that? Like how, what, how do you like to do that kind of stuff historically? I think it makes sense to let other people handle that, let it sort of settle itself down and then look at what the, the leading options are. Um, I feel like if you try and force opinions on the people, um, and they don't like those opinions, they're more likely to just throw away your entire body of work than they are to use the 80% that works for them. Right. So baby in a bathwater problem. Exactly. So for better or worse, you are better off poking, focusing at the things where you can have, um, unopinionated implementations of them, um, with easy ways to build the opinionated things on top. Yeah. What's the, you want to have, uh, you want to have the, uh, common wisdom, like it, like if, if everyone shares it, it's not really an opinion. It's just sort of like the, uh, the common wisdom of it. Yeah. And, you know, it's often not the ideal thing. It's often not the best implementation, but you know, you can be chasing the perfect implementation forever and never actually run any software, or you can figure out, you can be pragmatic, figure out what works and then build upon that. Yeah. Yeah. That makes, that makes sense that like in the overall community of users and people who do it and vendors and everything, like it's good to, uh, have a multiplicity of things you're trying to do with at, at, at the layer that isn't settled yet. And kind of, uh, yeah, not, not even, not even the it's, it's nuancedly inappropriately phrased to say, let the best one win, but it's more like let the one that everyone wants to use when, right. Which, you know, is, seems like a much better. Well, great.
42:33Well, uh, if people want to, uh, uh, keep up with you in a non-spooky way, see what you're up to. What, what would you point them to? Uh, so they can find me on Twitter, uh, at Pete Chikovsky. Uh, I have a very hard to pronounce name, uh, and a very hard to spell name so that, uh, we can put it in the, in the show notes, how to spell my name. And then that's right. Uh, unfortunately all the short, easy to spell ways of saying my name were already taken by other people on Twitter. Uh, so I'm stuck with a hard to spell name. Uh, otherwise I can be found all over the place at conferences. Uh, I'll be at Bosdom and config management camp in Belgium in like a week's time. Yeah, that's soon. Uh, yeah, that's soon. And then I believe we're starting to plan out our SpringOne tour events for this year. And we have maybe Seattle coming up in March. That's right. I think March 9th or 10th or something like that. Uh, and I believe I'll actually be giving a, like three and a bit hour Kubernetes workshop where I will be giving everyone access to a cluster and teaching them, uh, piece by piece how to actually deploy and run applications on Kubernetes. Oh, that'll be exciting. That should be fun. Yeah. Yeah. I think I'm also giving a workshop, so I will not be able to attend. Hmm. And, and by, I think I am. So I need to figure that out, put together, make sure I have solid content. Yes. That's, that's always key. Well, all right. Well, as, as always, this has been, uh, well, it's still Pivotal Conversations because we haven't renamed it, but just so the audience knows, I think, I think what I'm thinking of me and me and Dormain, Dormain and I is, uh, where I think we're going to call it Tanzu talk just because I think like we were discussing it and I think, I think that title, especially because talk, it kind of matches the tone that both of us have on this whole podcast, which
44:36is, you know, slightly goofy, but helpful. And it's kind of like, you remember, uh, UHF with weird owl, it was that show talk. So we may not, we may not have a finding a marble and oatmeal or, you know, red snapper or anything like that, but we'll try to have it be as, as, as entertaining and, uh, save the channel. Will you have ads for a spatula city? That's right. Nothing says I love you more than a spatula. I, you know, speaking of the kitchen drawer, I, every time I grab a spatula, I think that's, I think that those exact words, spatula city. Uh, well with that, we'll see everyone next time. Bye-bye. What's the rule with two vowels are together that like one of them silences the other or something, or am I remembering that incorrectly? Isn't there some rules as far as how you pronounce thing when two vowels appear with each other? No. No, that's yeah. Awkward silence there. Yeah. You never know that I spend most of my time writing for a living. The only two vowels rule I know is, uh, I before E and less it is preceded by C. The exception of neighbor and way, right? Isn't that the thing? And total economic impact. Right. Yes. Yeah. Okay. Third option. Anyhow. Mm-hmm.