Containers Ain’t No Sriracha Sauce (Ep. 41)

Episode 28 · 51:49 · SRT · archive.org · all Tanzu Talk

Well, I was going to say it's fall down here, but I realize aside from the other side of the world, it's fall everywhere, if I recall. So it's it's which which over over over there, I think it's upside down fall. If I remember, I forget. But I my my wife and my son are up. It's like the day before elections are up block walking in Ohio. So they send me all these lovely pictures of like these these red and yellow colored leaves. It's not nice down here in Texas. We don't have that. It's either green, brown or gone. There's no there's no variance. So I was wondering, do you guys in your neck of the woods, do you have pretty fall colors? What do you and what do you do for that? Like there's lots of pumpkins. What's the fall decoration that's rolling in? Yeah, here in here in Seattle, we have, as you'd expect, fall weather. It's cold. It's wet. There's red leaves everywhere. But it's actually really nice to have seasons. I moved up from L.A. a year ago where it was either hot or not as hot. So it's nice to have proper seasons now. How about yourself, James? The Bay Area. Oh, no, go ahead. Sorry. That's fine. I feel like the Bay Area is the land of the perpetual almost nice. Depends on what you like, but it's always just a little too cold all year round. And so the trees just don't know what to do with it. Oh, yeah. So I hear that in some parts of the Bay Area, they've actually genetically modified the trees to turn fall colors at like higher temperatures. But that's probably not true. Now, this raises another thing I'm always envious of y'all who live over in the Bay Area. It's like you can always wear a light jacket. Like if that's just the thing that you feel like that day, put on a light jacket. It's fine. So you must always wear a light jacket. I guess it's just a different – you can always versus must. I agree. Equivalent. Just subtly different. Subtly different. I value the thickness of my hoodie based on the sequence. That's right. There you go. Yeah. Yeah. Yeah. Well, you know, it would be nice. I mean, do you concur with that, James? You have sort of like the one thing? We have some trees in our – I live in Palo Alto, so it's slightly different than the – like Berkeley where Aruncia is.

So we definitely get some of the trees, turn colors, and lose their leaves. But it's – I grew up in Nebraska, and that was a much more intense fall and transition. Nebraska, where they have winter, as I recall. Yes. That's quite a thing. Yeah. Huh. That's one thing I've never really experienced. We don't really have winter anywhere I've ever lived. Well, it's probably for the better. Well, why don't – we've got a couple of, as you've heard already, guests with us this week. Why don't you guys introduce yourself? Why don't we go in the order that you answered the leaf question? If anyone remembers. Sure. I guess that would be me first. So my name is Unsi Fakhri. I run R&D for Cloud Foundry at Pivotal. My name is James Baer. I work on the product side of Cloud Foundry. I've been working on the team a little over five years now, so I've been around for a while. Very good. So we brought you characters on today because there's – you know, containers are a hot topic. It's a lot of good information out there. There's a lot of, frankly, bad information out there. So we thought it would be good to bring you on and just talk a little bit about, you know, no FUD, no BS. Let's just talk about containers and PCF and containers in general from a couple of smart folks. So we're going to pepper you with a few things here today. Sounds good. Yeah. So first off, I'll start with you, James. What the hell is a container? Why do devs even care about them? Well, that's a great question. I think there's been a lot of confusion about this because one of the things that has happened, especially ever since Docker really took off on popularity, things get very confusing because people first hear – a lot of people first hear about containers with Docker. And it turns out that with Cloud Foundry, we've been using containers well before there was Docker for several years.

And really, it's about an isolation property in the Linux kernels is what we started with with Cloud Foundry. And what we realized early on is that the first versions of isolation we had in Cloud Foundry were just using very simple isolation mechanisms, and they weren't very robust to something that an enterprise would expect. Certainly not as robust as the level of isolation you get in a complete virtual machine. And so one of the things we've done is increase the levels of isolation by using more advanced features of Linux. And you get these great properties compared to a virtual machine. These containers are really fast to start up, and really quickly you can get benefits out of them, and so they feel faster. And so that's probably the biggest benefit to using these containers is you get this improved isolation from really something that just would be something basic, like using a separate Linux processor or a separate Linux directory. You can actually make it such that those things don't see each other, but you get the better speed than you would if you used a complete separate virtual machine. Yeah. Good. Unsee, do containers get you excited? When this came out, did this light you up, or was this just interesting to you? I think at some level it was mostly an implementation detail around isolation, and then I think the brilliance of what Docker ended up doing was making it fairly easy for folks to interact with these containers and create them, tear them down. But most importantly, specify what goes in them. So when I get asked what is a container, I like to think of it as three separate things. It's the isolation primitives that James is talking about, which if you step back and think about it, it's really impressive what the kernel folks have done. They've taken a single-tenant system and wedged multi-tenancy into it and made it possible to isolate processes across different axes within the one kernel.

So that's just bonkers that it works, but it works, and it's been getting better and better and more advanced. So it's isolation and resource constraining. And then it's sort of a description of what goes in the containers. Now you have your box. You've put up the isolation walls. And when you put in the box, what's that file system? What are the files and the configurations that get mounted in? And the last thing is sort of this metadata of, okay, now that you have your box with stuff in it, what do you want to run? What do you want to do with your box? And how do you communicate what you run to other people? And what Docker really did, I think, was made it easy to create the container, easy to define what goes into it, and easy to sort of specify what you want to run. And that just makes it more, I think, accessible, especially primarily right on the local development environment. The thing that, I have a story. So when I was in grad school, I took a really silly little CS class about, the details don't matter. But anyway, I just remember we spent the first two classes trying to configure our workstations, right? Like this is the bane of developers everywhere. It's like, oh my God, I don't care about the configuration. And I just want to get started from a known good point. And so I think one thing that we've seen is containers help make that easier. And by containers, I specifically mean having like a community of those base images makes that easier. Yeah, you know, that's a weird thing. I forget when I started paying attention. This is back when I was an analyst. And like, initially, you read about like containers and LXE and all this stuff. And from my perspective, being a very way above the value line type of person, as it were, I'm just like, what is all this nonsense, right? Like about isolation and stuff like this is, I hate security. I don't want to read about security. And then somewhere in the last maybe two years, there's been this flip of like, no, no, it's not about security. It's about developer productivity. And it's like, I don't know if there's other sort of historic things where something so annoying as, I mean, I guess all of computers, but like, you know, as weird as security isolation turns into this almost, from my perspective, polar opposite thing.

And it seems like that's the huge jump of reorientation that makes a lot more sense nowadays. And it is, I don't know why it seems weird to me, but it seems weird that this, this essentially security thing, like makes developers more productive, which I don't think most people would ever associate security and developer productivity together. But I guess, I guess what it does is sort of freeze up, to put it in another wacky way, it allows developers to be a lot more sloppy in this, therefore quick. They don't have to worry about their entire stack of things, but it is an odd, you know, and it's also like, I always love it when all the Solaris weirdos come out and they're like, we've had this forever. Right. It's like, it's like when the mainframe people come out and tell you how they did DevOps in the 1910s or whatever. And so it's, it's a good, uh, it's a good rebuke for like, you know, well, you did have that, but it didn't really like wrap around this usage of it or, or get so, so much widespread acceptance. So, I mean, given the, as it, whether it's the, uh, mainframe guys from the 1910s or, uh, or Docker today, I mean, so all, are all these container products using different engines? Like, is there commonality here? I mean, I guess, James, start with you. When you look at these sort of components, yes, there's Rocket, there's Docker, there's all these other components, there's orchestration on top. When you look at these components, you know, where do you, where are you starting to see some standardization? Yeah, so that's a great question. In the, when we first started working with the technology, we, we actually started in Cloud Foundry using, uh, um, some very basic stuff that I referenced earlier was really primitive stuff. Like, we're, we're just make up a Linux, uh, user on the fly and, and using basic Linux user permissions and, uh, file, file permissions in the file system. And we quickly found that that wasn't going to be good enough. So one of the other things that, in addition to the security that you may not, you think like, oh, sure, it's the really, um, paranoid enterprise security person. But there's also something that with multi-tenancy, this noisy neighbor property, that when you use the, the better levels of Linux isolation in a container, now you have a lot more confidence that you can put mixed workloads, um, from different tenants on the same place and they're not going to disrupt each other.

And so we started looking at, uh, other engines and one of them was called LXC. And we, uh, we, we played around with it for a while. We found was, uh, this, it wasn't giving us, it was too much of a black box. We couldn't quite understand what was happening under the hood. And so we evolved to, um, this was before Docker was even available. Um, we evolved and wrote our own, uh, low level tool called Wharton and, you know, time goes on and that was really successful for us because we could really go down deep and, and control all the things we wanted to do to provide our, our enterprise platform, the properties we wanted. And then time went on and, uh, Docker emerges and gets a lot of adoption. Uh, we saw other projects come out from Google. There was one called, uh, let me containerize that for you. And so we started seeing that, okay, there's a lot of other stuff going on out here. And what I'm really pleased to say now is that we've actually gone far enough to where we actually share the same, uh, Linux, uh, container runtime that Docker has. It's called run C a whole bunch of, uh, the industry got together for this thing called the, um, open containers project. And Docker took the core part of their insides of their, of Docker engine and made it, uh, open source. Um, and which it had been the entire time at this time, they said the governance of this thing will be open source. We can all work and, and move on this together. And so now all of cloud foundries, uh, um, container systems, whether it's the hosted versions we have, uh, pivotal web services, or it's our, um, enterprise product that we ship to customers. All of those customers have this, uh, shared run C backplane now that does all of the really basic fundamental stuff, like setting up the container file system, setting the Linux namespaces, setting up the C groups, all that stuff is shared now. So that's, that's something we're really excited about. So like, uh, I think, uh, going back to your question, you know, what, what are the shared components?

I think it's just really important to always remember that at the end of the day, all the heavy lifting is being done by the kernel. The nice thing about something like run C is that it's, it's almost like the thinnest possible layer on top of that. Um, to provide a bit more of a user friendly API. Um, and so having, you know, cloud foundry and Docker and other things begin to shift and, and run on top of run C, um, helps in many ways. And the one that really, I think is the most compelling to me is there's this shared shim layer, the shared code base that is doing a lot of the finicky kernel calls. Um, that's where you're likely to, you know, mess up. That's where you're likely to like, you know, have a, a security vulnerability or an issue if you do that wrong. And, and what's nice is now there's this shared thing that has just a ton of eyeballs on it. And we all get to, to benefit from, um, that shared community ownership of that, but then have our own opinions on top of it. And so garden, for example, garden run C specifically gets to implement the garden API, which is, uh, you know, we, we see it as like the orchestrator friendly API. It does just enough to support an orchestrator and like get out of the way. Um, Docker on the other hand, right. They take run C and then they just start to layer on more and more and more stuff to the point where it's this sort of massive thing that can do, you know, local workstation and multiple, you know, clusters. And it's just, which sounds good on paper, but like really it becomes, it becomes a very leaky abstraction. That's very hard for, um, you know, others to either wrap around or use or really trust as it sort of moves and breaks and evolves, um, frankly, too quickly. So what's nice is when, when run C came out, we immediately jumped on that. We, we resisted like switching cloud foundry to run on top of Docker for a long time because there wasn't that clear governance. There, there was a sense that, you know, what Docker is today and what Docker will be in the future. Like what, what is that? You know, there's not a lot of control, not a lot of stability or, or, um, certainty there.

But when run C came out, we're like, oh, that's, that's perfect. That's exactly the right abstraction that we want to build on top of. And so, so we did, and not only did we like build on top of it, we then like shipped up an update that like helped just move all of our customers over transparently, um, from our old sort of homegrown, uh, solution on top of run C. So let's drill on it just a little more cause I know, Hey, maybe we're to blame for it sometimes, but there gets to be confusion about PCF and Docker. Like, do we support that? Are we somehow, you know, mortal enemies sworn to the death or, you know, are we, what are we trying to do to help customers with this stuff? So I guess the first question, aunty, even for you is sure does cloud foundry support Docker? Yeah. So, so I go, I go back to like, what is a container, right? Um, and, and, and that, that mental model of, uh, isolation. Container content, like the file system that gets mounted in and then the metadata to run, run and launch stuff. So, you know, we, we use the same Docker primitives to build the isolation that, that, you know, run C basically. And we, uh, support 100% fetching Docker images, sort of the Docker tarball as it were, um, and mounting it into the container. Like, we don't mess with it. We just grab it using all of the same layer and caching technology that Docker's built. We, in fact, use the same libraries. This is all open source stuff after all to mount your container in. And then we can fetch and translate as much of the Docker metadata as makes sense to run on a 12 factor platform, um, to, to then enable you to just basically run your Docker images on the platform. So, yeah, we, we, we support Docker. Um, so if I've built some beautiful container and I love immutable infrastructure and then, you know, James for you. So if I take that Docker image and I point PCF at it and say, go pull this image and run it, do we break that thing apart and, and PCF it ties it and kind of break the immutability up? Or is, or is, do we just take that container and run it the same way Docker does?

Yeah, it's, it's very, you're going back to NC's answer, um, to provide a little bit more depth, uh, the, the Cloud Foundry user is going to reference a Docker image to the platform. And what the platform does is hand, hands that off to this, uh, you know, run C based layer. And it's going to talk to the Docker registry V2 API, pulling down the layers, just like it would happen in the Docker engine. But in this case, inside of a PCF platform. So all that stuff is exactly the same. The V2 registry is going to say, here's the list of layers that you need. And then the, the local, um, uh, run C based, uh, container runtime is going to say, well, these are the ones that I, these are ones that I already have. So I guess I won't download those. And then they get assembled and started up into, uh, the container aspects that the Nancy was talking about. So all that stuff is exactly the same. It's not different. We don't do something special, um, for retrieving a Docker image, putting it on disk and starting it into a container. There are some additional things we do around that, that supplement the experience. For example, we have this concept called application security groups. And what that allows us to do is to set up a default policy that, um, Cloud Foundry administers can assign to all the applications that run on the platform to whitelist specific network endpoints that you can get to inside of your container. Um, but you can do some other things like log. This is a very kind of enterprise feature for enterprise security people. So you can log the first, um, outbound packet that's from a TCP connection. So now, like if someone's doing a bunch of port scanning, um, inside your, inside your platform, you can figure out who is doing that because we can log that. So those are some of the things that we, you know, add in and around, uh, the, the Docker support that we have. But fundamentally, it's the same as running a Docker image of Docker engine. It's just that we have some extra things that, that we do. Once you have that, um, that the platform adds as advanced features.

One, one thing I might add to that, like the, the things that we do on top, they're really to contextualize, right? They're to contextualize the application within the context of Cloud Foundry. And so the application security group is a good example of that. Another example is that you get all of the environment variables that you may have defined in your Docker metadata. Remember, that's that third category of what a container is, that metadata, that's important. But you also get, um, the environment variables that Cloud Foundry will typically provide. And those would not have been baked into your container. In fact, those are, those are more dynamic, right? They would be the services that you've bound your application to, um, at runtime. Uh, they would be information about, um, your application within the context of a running Cloud Foundry cluster. And so we do, you know, we do layer that, uh, that value on top, um, and we manage it for you. And so, uh, again, that it's important to have that mental model, the, the, the bits themselves, those are not changed. They come in and, and we just, we just mount them directly. So, so when, when, what, uh, like that process that you guys just went through, what, what do people call that? Cause, and, and I asked that because, uh, the, uh, uh, the vocabulary of container fun, as far as I can tell, has two words, container and orchestration. But I imagine it's a lot more like, uh, Baroque and, and, uh, varied than that, but like, what's, what's that one that, the part of, uh, initializing and setting up? Is there a name for that thing? Uh, I don't, honestly, I don't really don't think so. Um, that's, you know, it's creating the container and by that, I mean the isolation groups and it's mounting the file system. And then it's, it's, it's launching, uh, the processes within the correct context and configuration. Just like a good old fashioned booting up. Just about. Yeah. That's, that's right. Yeah. Yeah. No. Yeah. So, so then, so then how does that, so we kind of went through that process. How does that compare with like what happens when you use a build pack? How, how are those two things, uh, different or similar? Uh, sure. James, do you want to?

Yeah, sure. I can, I can, I can talk to that and you can add some color. I mean, to me, the, the big difference with, with build packs and Nancy has this awesome haiku that the people reference a lot. It's here's my source code running on the cloud for me. I do not care how. And when you're using something like a build pack as a developer, that's primarily doing something like Java or node or Ruby, I'd actually don't have to think about how to construct my image. I can, you know, give my source code to the platform. So we call that CF push where my, my application artifacts get handed to Cloud Foundry and Cloud Foundry will go through this life cycle of inspecting that source code saying, oh, it looks like a Java application. I'm going to give you the latest secured version of the JVM and the latest secured version of Tomcat if you're running that kind of application. And it puts it all together for you. And you don't actually have to take responsibility for the, any of the, the base level operating system components or the middleware runtime that, that's on top of that. If you contrast that with the Docker, um, you're running a Docker image, that responsibility is put onto the developer. And that's, that's something that might be surprising to developers that they really have to start thinking about, well, is the version of bash that's in my container, um, up to date and secured is the version of open SSL. Um, I actually don't normally pick that. That's something that the operating system middleware administrator does, but now I'm responsible since I'm producing this artifact. So that's really the big difference to me is, is the, do you want the extra levels of control and responsibility? Um, it's a trade-off for the, for the additional control and the ability to specify everything. Um, I, now I also have the responsibility to, to maintain it and you get that with the Docker image and we support that completely. We find that a lot of developers really like the simplicity of just giving the platform, their application and having the platform responsibility for the assembling of that. Um, image and they actually don't have to think about that too much because they can just say, here's my, here's my code.

You create the image for me and you run it. And, and that's really the, it's a, it's a dramatic simple, um, simplification of the user experience because, um, otherwise you have to think about all those lower layers. Yep. So, so conceptually, I like to think of it, um, this way, right? So with the, with the, with the build pack approach, um, we end up with three independent, uh, sort of layers, conceptual layers. There's the operating system layer that has, you know, tar and gzip and lib open SSL, stuff like that. Um, there's the source code layer, which we can then compile. And then there's the dependencies that the source code needs to pull in in order to run layer, right? And so the build pack is responsible for, for managing those dependencies and compiling the source. And then the underlying, uh, root file system, that operating source layer, that's something that, you know, we are responsible. Cloud Foundry is responsible for providing, um, that multi-layered approach means that we can actually independently swap those layers out. So if there's a vulnerability in open SSL, we just need to ship a new version of that lower, lower layer, that operating system layer, run a rolling deploy. Automatically, everything is patched with no downtime. No one has to wake up. No one has to do anything. It just works, um, with, uh, with a sort of Docker image approach where you choose to specify the contents of your container as basically a single tarball or a collection of, you know, Docker layers, but that are then ultimately like sort of conjoined to form the image. Um, we don't have that much flexibility or really the ability to manage that as a platform. It's sort of a bit more of a free for all where the developer has opted for flexibility, but at the cost of taking over complete control of the whole enchilada, right? All of these layers. So now in that context, when there is a, when there is a vulnerability, you would have to sort of manage your whole fleet of Docker images and Docker image producing pipelines. It's to make sure that they're up to date and have the latest security patches.

So I know what I, yeah, I mean, I know when I came over here and it started at pivotal and started to really see what the platform was, was really doing. I think that was the one that maybe is more underrated because people do think of containers as helping that upstream. Like, Hey, let me just bundle my app. And then it's the output of a CI process. And look, I want the control of the layers and don't, you know, don't tell me what to do. And so that can be very efficient for that initial push, but that operational component, when I have a thousand microservices out there with 5,000 containers and, you know, that dirty cow vulnerability comes out or an issue comes out with Tomcat or something like that. Can you actually repush that many containers? You know, do you have the source? Do you have it all hooked up to CI? That seems like that's a much more complicated process than a restage operation in PCF. Right. And for DockerCon in particular, you didn't even need to restage. It was literally an operator saying, okay, I have the latest root effects, deploy the platform. And it was done. Let me ask kind of like a slightly abstract question, which is, well, you know, I think even in our ancestry line of products, we used to use this word. People used to use this word fabric all the time to talk about like a layer of something, which I never quite understood back then. A layer of stuff you would run your stuff on to be mega abstract. And it seems like there's yet again this thing going on where there's, on the one side, no one can see, but I'm holding my two fists up. On the one side, there's sort of like managing by atomic unit, whether that's like a machine, a server, a VM, a container, whatever. But just as you guys were just outlining, you've got to go like monkey around with each of those little atomic units as needed. And then on the other side is yet to be named new thing. I guess maybe we say platform, but there's like fabric where the haiku applies.

And it seems like, I mean, I don't know. I mean, first of all, like, I don't know if that idea that I'm kind of depicting is valid, but it seems like people always gravitate between those two things. And they really, developers at first seem to really like the atomic unit, but then sometimes they like go over to the fabric side of things and vice versa. But it seems like, like we've just gone over, you know, fabric is like build pack and atomic unit is like your container thing. And that seems to be like the eternal sort of, I always forget if this is the appropriate definition of the world word, but dichotomy of packaging applications that we, you know, run around in and sort of an infinite loop. So I ended a question with a period there, but like, I mean, is that, is that, uh, is that stuff you guys think through? Like when you're, you're messing around with our platform? Cause it seems like those are the two poles of how you would manage and run your software that I've seen throughout my career. Yeah. So I think that, so, so first off, everyone likes choice, right? And so like with PCF, you can, you can choose like the fabric mode or you can bring your own image. That's fine. Right. Just to be clear. Um, what, what gets interesting though, is if you think more and more deeply about the fabric and these, these are things that we want to eventually get to. We don't quite have them, um, like in line to, to start delivering on, uh, ASAP, but you know, what can a platform do if you've given it your source code and entrusted managing that source code? Like could, one could imagine a world in which you give us the source code and you give us, you know, uh, uh, like a, like a simple test or a smoke test to validate that the source code is working. And then we can take that, right? Compile it for you, get the dependencies for you, validate that those dependencies are, um, uh, secure and up to date, run the test, see that it works, deploy your app, perhaps a canary deploy to see that it works. Um, and then if there is a vulnerability in the dependencies today, we require the, um, uh, application developer to go and restage the application.

And that's because, you know, when you change a dependency, that's a, that's a more tightly coupled piece of the application. And sometimes the application can break. So we, we, we defer to the developer to make sure that things work well. But if you, if you ship your source code with, with a, with a test, um, then you could imagine the platform automatically taking care of that sort of thing for you. Right. And so when you take the fabric approach, you can start to really layer on a lot of value, a lot of automation, a lot of, um, you know, like hands off, don't worry about it. We've got it just by enabling developers to more semantically describe their stuff. Like, here's my code. Here's my test. Take care of it for me. Right. Um, so those, those are some of the places that we can imagine going to, um, in the future. But again, back to going back to what I said for now, yeah, you're right. There's this two world universe, right? It's like, do I, do I, do I speak in terms of atoms or do I speak, do I break the atom and speak in terms of its constituent pieces? And again, cloud foundry does support both. And, and, and, and then, and then as an, as another, uh, uh, adjoining question there, uh, like, like, do you, uh, do you, do you think, do you think that when like developers and architects, like, are they, are they thinking through this idea of like, uh, like, uh, I don't know. I want some amount of AI-ness in how my code is deployed. Like, like, are people interested in that kind of thing? Or are they kind of just like heads down on like the normal mode of like, I got to have my APIs and my implementation or, or are there people who are sort of like coming up with the language to describe a lot of what you were just saying? Cause that, that seems to always be one of the, the main problems with programming is until you, until you actually just like the gang of four people that until you actually name these patterns that people are doing, like they don't get widely used. And so like, like who's, like what people are circling through, like, what does it mean to like tell the fabric, these are my tests? Like, I don't even know what you would call that kind of development. Sure. Sure. Uh, sorry. I just got a random text that distracted me. Um, uh, that, that's how development happens. Right. Exactly. Distraction. Um, I, I, uh, I think that there are examples of these sort of nouns emerge, like, you know, what is a blue green deploy?

What is a canary deploy? These are exactly, that's a good example. Right. This is straight language. I look at Spinnaker as a, as a sort of an abstraction or a tool that like builds on top of these, uh, these ideas. And, and, um, so I think there's, there's, there's a lot of movement there. Yeah, no, I like that. That's, that's cause it, cause it kind of, uh, at least in my own written history of like containers where a security thing turns into developer productivity. It's similar where like another seemingly boring thing, pipeline management, like, like turns into a, it enables a whole different way of thinking about your application and the way you're doing it. So maybe there's, there's, there's, that's, that's an interesting area to go poke around in and see what, uh, vocabulary people are using. Yeah. If we, so I mean, back to the security part of it though, and we talk about fabrics and platforms. I mean, so I guess, James, where do we see where PCFs apps actually doing some good things on container security or in general, where's the industry getting better on container security, whether that's preventing things from breaking out or whether that's, you know, the security within things reaching the container. Are we doing better at that as an industry and how's PCF trying to help? Well, sure. That's true. Thanks for the question. I think that, uh, one of the things that we read already talked about this move for pivotal to adopt for on C and one thing that really helped with that for us, and it improved our, our container security story is that, uh, because run C has so many eyeballs on it and so much of a community behind it, as once he was referencing, um, it already works really well with app armor, which is this, um, basically a security subsystem that can be applied to Debian based systems. And works really well on Canonical Ubuntu, which is what, uh, the operating system that we build on top of inside of Pivotal Cloud Foundry. And so that just increased our ability to, um, secure these containers by using even more Linux primitives. Um, another thing that, that we were able to do in the last couple of months is to run our, uh, Linux containers, um, as unprivileged users by default, which is great.

That means that, um, even if, um, even if you're able to, to get out of the container, there's still, uh, some Linux primitives that are protecting, um, you from being able to see things you're not supposed to be able to see. And so that, that is another example of, of container security. And then, so you, you might ask yourself, well, every once in a while, there's still these zero days, uh, right. That you mentioned dirty cows being one of them. And so with, with Pivotal, we've made it, uh, pretty straightforward for, um, these pipelines to always be taking in these, um, security updates that they're coming directly in from the vendors, whether that's, uh, from Canonical or something like, um, Run-C itself or, you know, wherever these vulnerabilities are being found. And we test them and certify all those layers together and know that they're work. And if that's a high CVE, we'll ship that within 48, eight to 72 hours to all of our customers and make it really straightforward for them to update those. So in the past, what we've seen is, you know, I, I guess dirty cow came out about a month ago, right? Um, I, I can't remember the exact date, but, um, I've been seeing these exploits of people sharing them on, um, the social media platforms of, you know, with, uh, some, someone using dirty cow to get out of a Docker container. And here, here we are a month later, um, from when the vulnerability was announced. And if you're a pivotal customer within two or three days of that vulnerability being announced, you can already be patched because we're, you know, have this really sophisticated, um, update mechanism. And so I think that's, that's one of the ways of just making security very straightforward and simple to keep up with is something that, um, is a really big challenge if you take it all on yourself, because we've seen just a lot of our customers struggle to stay up to date with patches, frankly. That's just that, that mundane, um, exercise of just always keeping up to date and trying to keep track of all your dependencies is really difficult to pull off. Yeah. I mean, Antti, do you see that as well when we think about trying to bake security in versus asking the developer to necessarily harden and secure it, what they're deploying?

Right, exactly. I mean, you should secure your code, of course, and do good practices. But when you are taking on more responsibility with a container, there's a risk there. So where do you see that value of platforms trying to give that to you and forces forcing you to bring it yourself? Well, that, that's exactly right. So what's, what's super cool here is dirty cow is a great example of like, uh, a vulnerability in like that OS layer, right? Which we can patch for you. But if you look at what's happened over the last two or three years, um, as the container space has like really evolved rapidly, had more eyeballs on it, had more scrutiny around security. What we've been able to do is evolve the platform through point releases where we've, we've, um, sort of changed our security stance, changed our opinions around security, made them more robust. So James mentioned app armor, James mentioned, um, you know, user name spaces. That's what the unprivileged user thing is. Like we rolled that out, um, without, um, any developer having to worry about anything without any operator having to do anything except simply apply a rolling deploy. And so what you have is, is, you know, like the, uh, a platform that's aligning with industry standards around security and, and keeping up with them and providing that value transparently so that you don't have to think about it. Right. We're just constantly improving our stance, um, and, and, and shipping that transparently. And one of the ways that we can do that is because we are opinionated, right? We're opinionated about how we structure the workloads that land on the platform. We're opinionated about how we run those workloads. And we know that we can turn those security features on and that it will be safe that we won't break anything as we do it. Um, so, so I, there's, there's a ton of value there and no one has to think about it, right? It's just, it's just, uh, you know, the Cloud Foundry and Pivotal working hard to, to bring that value to the table. It's like, it's like, as, as always, my, uh, my buddy security is the, uh, is the fat boy scout on productivity. It's sort of like the, uh, it's, it's the, the one constraint limiting your whole thing. And so if you can free that up, then you can, uh, you know, I don't know. I just like saying fat boy scout, but hopefully you can, you can speed things up and, uh, not be limited by them. That, that, that would be nice.

Uh, so, you know, I, I, along those lines, so you, you have like, what, uh, when, when, when, when you guys are talking with people and thinking through it, like if I'm like, uh, there's never one, there's always like a pack of them. I don't know what the plural is. There's always like some pack of enterprise architects who are like planning out what to do over the next, uh, six to 15 years, six months to 15 years or something. I'm being a little hyperbolic there, but like, what, like, what, what's the kind of advice you would give them about like what to do? with their portfolio? Like, should they just be like lift and shifting everything over or like staging things out or doing nothing? Like what, what's kind of like your default advice other than just, you know, putting everything on Pivotal Cloud Foundry as soon as possible. But like, how do you, how do you sort of advise them to meter this stuff out and identify what works in their portfolio and, and, uh, what projects to use and things like that? So it's, I think it's, it's this big mixed bag, right? There's, there's a lot of complexity there. There's a lot of different moving pieces. Um, I like, I always like to frame it as a journey, right? Like you're, you're on a journey towards a, a more cloud, cloud native world where you have that agility. You have that ability to put, you know, more and more of your workloads on a platform where you don't have to care about stuff that's below the value line, right? How, how you, how you make progress on that journey. I mean, it's, it's a very, rich and complicated question that really depends on the context and where the, the individual customer is at. Um, some things do make sense to lift and shift. Some things are like, James, what do you think of lifting and shifting then? Right. Yeah. Oh, please. Well, for, for, for me, lifting and shifting, um, you have to ask yourself why you're doing it. So, um, if something's just in keep the lights on mode and it's not being actively developed, then, and it's actually running fine where it is, then, then you have to ask yourself, why, why would I lift and shift? This right now? What value would I get out of doing that? Because when you lift and shift, you have to put forward some effort. And then the, then the question I'd have is the effort that the lift and shift effort, um, applied to this particular spot going to be better than the things I'm actively investing in and maybe accelerating those. And so I think that's, that's one obvious thing to look at is, um, if it's, if it's running where it is and it's okay, unless like one, one benefit you could get from lifting and shifting, for example, is, you know, we were just talking about, you know, keeping up to date with security patches.

Maybe that's a problem that you're, you have. And so lifting and shifting a workload can make sense. But, uh, in general, like when, when people come with this question to us, we often recommend that they take something that they're actively investing in. And then, uh, and then, uh, and then that they're more, more apt to be able to make, you know, their own choices. But first we'd like, we'd like to help them with something that they really care about. We'll do it with them and kind of pair up with it. And then, uh, and then, uh, they're more, more apt to be able to make, you know, their own choices. We'd like, we'd like to help them with something that they really care about. We'll do it with them and kind of pair up with it. And then, uh, and then they're more, more apt to be able to make that, you know, informed choice going forward. Yeah. I mean, I think, I think, I think to your, to, to y'all's point, like a lot of lift and shift stuff is like, uh, it's sort of like, you're thinking about technology, like it's a condiment, like, oh, here's this new container liquid in a bottle and I'm just going to squirt it on my hot dog and it tastes better. But like, I think there's plenty of technologies that are like that, like that you can just condiment your way to awesomeness, but it doesn't really like, it's a lot of trouble to like, just squirt some of the stuff on your hot dog. Whereas really like, you're talking more about like a, like a hamburger or a meatloaf or something. Like, it seems like it's a much more fundamental thing. And therefore you pick, you pick projects that are, uh, sort of ongoing in their creation to start using it and learning about. So you get the full benefits of everything instead of just sprinkling it on top. Well, I want to, I want a hot dog right now. So thank you. I'm getting hungry. I mean, I think I want a burger actually. Yeah. I mean, where was this condiment metaphor going to go? Aeolies, relishes, it was going to go, uh, somewhere exciting. Uh, I mean, so I wanted to ask you that kind of building on that a little bit is, you know, we talk about security, we talk about lift and shift, but where are those difficult problems that we have to solve when working with containers? I'm thinking about things like persistent storage and networking. And yes, I could containerize my Oracle database, but why would I ever do that?

Or, you know, can I really network? Well, where are you still seeing James to start with some of these container challenges that, Hey, the industry is starting to solve, but companies who are using them should at least keep an eye out when they're trying to pick the workloads that they would containerize themselves versus where does PCF help with some of those challenges? Yeah, that you've nailed the ones that, uh, that come top of mind for me are persistence and networking are ones that I think, uh, you know, speaking specifically about Spivill Cloud Foundry, we're still exploring and investing in enhancing our capabilities there. It's a lot easier to run a stateless workload than it is to run a stateful one. There's just more to it. And so with persistence, we're actively working on, um, being able to, to mount shared volumes, something like an NFS store or something like that into, into your container. We've demoed that at CF Summit and, uh, uh, we're hopefully going to be exposing that, uh, into Pivotal Cloud Foundry customers relatively soon. And then in the networking space, uh, for a while, we've been working on this, uh, kind of network address translation. Um, is that's the, that kind of been the working model where, um, everyone, everyone's on this flat network and we're using these application security groups. I spoke of earlier to, to kind of control IP tables rules for, for inbound outbound traffic. Um, in the, if you look at, uh, where people want to go with networking and they want to be able to have, um, you know, more explicit controls on the networking. So we're looking at something, um, based off of a CNI, which is a container networking interface, uh, that, uh, is being used in the Mesos and Kubernetes communities, uh, to, um, hook into some of the other software defined networking stacks, like whether that's something like NSX or, um, Calico or any of these other, um, SDN solutions out there. And what that allows us to do is, is turn what has normally been a mechanism where you have to turn to network administrator via filing a ticket or something like that, actually program the network with API calls.

And so we're, we're, we're kind of moving in that direction, um, both, uh, this persistence and networking to be able to make that more automatable self-service by developers and with some kind of guardrails that have been set up by administrators. Those are the kind of the, the emerging areas that we're still, you know, enhancing. And, uh, you know, that I mentioned before, as we enhance those things in our minor point releases, we just keep raising the waterline on, on the whole capabilities of the overall platform. And you'll be able to just, you know, take advantage of those as we deliver those. Yeah. I mean, honestly, where would I access stateful data in a PCF world? Do I call off to VMs elsewhere that are Bosch created or something? How, as a customer, how do I look at this environment and say, I've got some stateful data. It's not just, you know, elastic web apps. Where am I supposed to stick this stuff when I work with a container-based platform like PCF? So, to today, they're external to the system or, yeah, on a Bosch-managed virtual machine. We're, we're slowly starting to evolve that story. So, with the, um, sort of the first step with the persistence offering is to say, well, today, the only service you can bind is, um, uh, a service that I can connect to, like, over TCP or UDP, right? Over, over, over the network in that way. Um, and so it has to be, like, an external database or it has to be, uh, you know, an external message bus. But the next step is going to be that, well, actually, we can begin to mount, um, data into the container. So, like, with NFS or an NFS-compatible, um, uh, storage system. What that lets you do, it's, it's less about, now I can run my database on Cloud Foundry. There's a lot of just complexity to orchestrating that sort of thing correctly, managing backups, worrying about lifecycle events. And, you know, we're, we're still not at the point where we think that, well, I don't think we'll ever get to the point where we think that developers really should be worrying about that. We'd rather kind of continue our, our data service offering story on top of Bosch, where there's just a lot more control and stability. But, um, what you can begin to do is to say things like, well, I have just a pile of data over here, you know, in my existing, you know, disk array.

How do I just get at it efficiently from my computer, from my, from my container? And I don't, you know, I don't have the option of putting it all in a blob store. I don't have the option of, of, of, of wrapping it in a shim. I just want to get at it and have my containers process it. We'll, we'll be able to start to have answers about that so that you can see data as just another service that you can bind to your container. And it will just appear, um, that's, that's the, that's the next step. And what we're looking to do is just play that out with a few key customers, see how that, how they, uh, how they experience that. See, get some feedback on it and decide whether or not that is a place to, you know, continue to invest or to pivot and look in a, in a different direction to bring this to the platform in a way that actually makes a lot of sense. Given, given our emphasis and our workloads that we bring on the platform. Awesome. Well, I could, I could probably ask you another four hours of questions, but let me wrap up with one that hopefully contextualize. Some of this, so we could talk about containers and infrastructure forever. One concept that we talk about a lot at Pivotal with customers is this value line. The idea that, Hey, things below that, like. Orchestration or cloud orchestration is super necessary, but not sufficient to be a platform. So, I mean, when we think about containers in our world, I mean, James, you're a remarkably well-read in the community about how we all think about this kind of stuff as well. Ansi, as well as how should customers be thinking about containers in their ecosystem? When, when do they say that this is a very important enabling technology, but I don't stop here. So, I mean, how do you help frame that as a technology that's a piece of something more? James, for. Yeah, sure. I mean, I, I like to, to think about it in terms of just the developer productivity. It's, you know, containers aren't all that interesting unless they actually help you accomplish something. So, if you're, if your business is trying to be competitive in a new area and in order to do that, you, you need to evolve with software really quickly to try out new ideas and see if they can come to market fast. That's really what you should be focused on and, you know, picking and choosing the technologies that are going to help you do that.

And, you know, containers are, are part of that story likely in some, in some way, shape or form, but exclusively just on containers as your primary noun that you're looking at. It's, it's, it's really hard to see how it ties back to, you know, enabling your, your business to do more and more with software. And so, I, I, I put the focus on that time to value metric and whatever, you know, technologies you can, and we have strong opinions at Pivotal about, you know, and, and a pretty good track record of actually being able to demonstrate being point to customers that we've done this with. And so that's, that's why we don't, you know, promote the container in and of itself as the end all be all because, you know, it's really about this, you know, time to value for your business and, you know, that's, that's where we want to focus. Great. Antti, your thoughts on that? How you think about that? I, I, I agree. I think that that last phrase, that time to value, you know, containers are a tool. There's lots of tools out there. What, what's going to help your developers get, you know, all of the cruft out of the way and focus on rapidly iterating what they deliver to your customers? Because it's only when you have that rapid iteration that you can course correct quickly. And it's only if you can course correct quickly in response to feedback that you can build experiences that your customers are going to love, that they're going to value. And that's how you win in the software economy, right? So, are containers an important part of that? Yes, they are, but they don't, they don't kind of come in like a, a magical, you know, condiment that makes all hot dogs incredibly delicious and compelling, right? They're, they're, they're part of a bigger recipe and you have to, you have to get the whole, the whole package deal right. Yeah. It's not just sriracha sauce. No. Which, which, as we all know, makes everything more delicious. Well, uh, yeah, I, I think, I, I think, I think that's, uh, that's great. That's actually a good overview. And, and, uh, I, I appreciate the, uh, uh, trying to come up with names for the nameless things there in the middle.

But, uh, yeah, if, if people want to follow up with you guys, uh, like where, where should they, uh, where should they find you? You're, are you in the Twitter there? Yeah, I'm Jambay on Twitter, J-A-M-B-A-Y. Yeah. And I'm Unsi Jo, O-N-S-I-J-O-E. And, oh, that reminds me. I had a question for you specifically. So, every time I see an email from you, I see this flat field of sand and then a head. Like, what's the story with that picture? Uh, I, I, I decided to bury, yeah, no, I, we were on the beach one day and I dug a hole about as deep as I am tall and then sat in it and then took that picture. It was delightful. It was very fun. So, what's it like to be buried in sand up to your neck? Like, is it distressing or relaxing? Like, can you get out of there? It's way too, it's way too tempting to have, like, a really lame container joke in here. So, I'm going to avoid it. It's, it's, uh, it's, it's actually really calming. Yeah. It's just like, there's this uniform pressure and it's warm and it's nice. I, I enjoyed it. Yeah, yeah. I enjoyed it a lot. It does, it does. I've gotten a lot of mileage out of that profile pic. Yeah. I'm breaking into a sweat thinking about it right now. So, we need to wrap this up. Well, well, on that note, as always, uh, well, thanks to the two guests and thanks for listening. This has been Pivotal Conversations. You can find us in our not so secret backend at soundcloud.com slash Pivotal Conversations. Or if you go to, uh, I believe it's blog.pivotal.io slash Pivotal Conversations. It's almost a tradition that I get that URL totally wrong, but you can just search around for Pivotal Conversations and you'll find the extensive show notes and everything that we have. It's always great if you enjoyed this episode or if you didn't enjoy it, but still want to leave a favorable review. If you go into iTunes and leave a review, or if you're listening in overcast, you can open it up and click on the little recommend thing. Uh, and, and with that, you know, it's always nice also to just to tell people, uh, that, that you like it and tell us it keeps us, uh, you know, being dancing pickles and everything. And we'll see everyone next time.

Bye-bye.

Tanzu Talk · 🍕 Get more pizza.