Container networking & storage with Usha Ramachandran (Ep. 61)

Episode 48 · 41:14 · SRT · archive.org · all Tanzu Talk

Well, Richard, I don't have anything funny to say today, so we should just jump right into it. Do you want to introduce yourself, special guest? Sure. Well, thank you, firstly, for having me. My name is Usha Ramachandran. I am a product manager. I work at Pivotal, and I lead the container networking efforts for the open source Cloud Foundry platform as well as from the Pivotal side. In addition to container networking, I also PM the volume services features for Pivotal Cloud Foundry. Awesome. So storage and networking. Yeah. All the pluggable things. Yeah. So, I mean, I guess we'll have to get to it, but it seems like you ended up with the two most difficult things in all of cloud computing. So congratulations. Well, thank you. They're also two of the most interesting things. Yeah. Difficulty is also the source of interestingness. Well, before we get to that, before we get to the delight of complexity, as per usual, we have a few news items to go over. Last week, I think in our space that people care about, probably one of the bigger exciting things. I was out of town. It's quite tragic. Otherwise, I could have gone to, well, I don't know if I would go. I could have gone and hung out on the sidewalks, visited with my friend John Willis and other people at DockerCon. But it looks like there was basically like two announcements, mostly around toolkit things. Either building your own Linux distro, not distro, but your own Linux sort of, what would you call it? I can't say image because that's confusing, but your own Linux thingy that you would use in your various Docker stacks. And then if I understand it correctly, and that was called Linux Kit, there's this thing that they called Mobi because they got whales. And it's basically, if you were to look at all of the sort of like platform that you build out of all the Docker stuff, you know, the container itself, the orchestration, the command line tools,

I think the Mobi thing is basically a toolkit for building that sort of, for sort of like building distros, to use my terminology that you would have with that, which I think, you know, several, several, a couple episodes ago, we talked about kind of the layout of all the various Cloud Foundry components and how you can kind of, we didn't really talk about how you can combine them, but any large platform has this idea of components that you can assemble together. And I'm trying really hard not to make like a early 90s cartoon reference. Yeah. Yeah. I mean, some of the news last week was coming out from, you know, all of a sudden github.com slash Docker slash Docker all of a sudden redirected to all the Mobi stuff. And so there was confusion about, did we rebrand it? I mean, wasn't that a really valuable name? Why did that disappear? So it was just some confusion about what is this new thing and what's rolling into that? What is this, what's its relation to upstream Docker? And I think it's all kind of being clarified, maybe it wasn't handled perfectly, but you know, either way containerization moves forward. Yeah. It seemed like after the, uh, after the, the storm of, of confusion subsided, it was fine. Not, not that big of a deal, but yeah. And then meanwhile, we had, we had our own release of something spring cloud data floor, data flow, spring cloud data flow. Maybe one day we'll get into floor management. So we understand how much potential there is when you, when you're stealing, but, uh, like I, I actually haven't looked over the, the release notes for that. I mean, we talked about this many episodes ago, kind of, kind of abstractly, but what's, what's bundled up in that release? Yeah. It's the first release candidate of a version one, two. And for those who aren't familiar with data flow, it's a way of assembling data processing pipelines as a series of microservices instead of big monolithic sort of ETL or even ESB processes. So what was neat in one too, the reason I wanted to highlight it was we're just doing more and more around simplifying the data pipeline. So compose tasks is the name of this feature, but you're actually building more of these stateful, uh, transition logical steps. Like do this and Hey, if the result of this is this, then jump to this step.

So a little more workflow based versus just linear sort of sequencing that we're familiar with in data flow. And last week for fun, I spun up my own data flow server with spring cloud data flow and tried it out and was really impressed with where it's going. So between that, plus the real time metrics and monitoring, this is getting to be a pretty legit tool. I know our customers are really looking at this for a lot of use cases. So it's just neat to see a really great constant evolution of data flow. People should check that out. Yeah. Well, you know, as I, I forget how many episodes ago it was, but when we talked about this, uh, some time ago, I mean, I think, I think the, uh, things, things like spring cloud data flow as they evolve are interesting because there's a huge amount of enterprise applications that as boring as they are, are basically wrapping a workflow and processing both manual and automated around moving a piece or pieces of data along a, um, well, to use the word again, a workflow. So, and you can think about it in insurance companies or banks. I mean, the one that most people are familiar with would be like a mortgage application, uh, where there's a lot of information that you want to combine together beyond what you fill out as, as a user and have the bank or the lender eventually make some decision about, uh, if they're going to give you a loan or not. And, you know, that's a pretty simplistic thing, but anytime that you're just, uh, spent sending a lot of information, heavy decision-making that involves multiple stakeholders making a decision, ultimately it's, uh, in addition to many other use cases, it's, it's, it's something that having a data processing is helpful for. And then also kind of in our realm of things like, uh, a lot of that, you can't really accomplish with like off the shelf things. Like there's no, uh, there's no Zen desk for like mortgage application stuff. I mean, I, there probably are actually, but like highly customizing it and making it fit into the way that you do this stuff is where, uh, if you are someone lending money, it's part of your actual core business and probably competitive advantage. Yeah. I mean, what's exciting to me about this stuff. I also spent some time screwing around with spring cloud stream last week and getting it working in all sorts of ways, but just this idea of making it simpler to build event-driven apps and not depending on centralized integration experts to, to connect all my stuff.

Instead, if you're a good developer who knows Java in this case, you can build an easily, easily build an event-driven app that hooks up to Kafka or rabbit or whatever and process data that does not take some sort of specialist skillset anymore. I think that's kind of exciting that integration doesn't have to be the bottleneck when you're doing these sort of continuous delivery of systems. And then speaking of filling out mortgage apps, maybe one day everyone will have the pleasure of just speaking to their computer, like in that movie to fill out a mortgage app and, and they can, they can answer their voice, their, uh, their price band and everything. But I mean, that's just a terrible segue. There's also a, uh, a Google speech API that became generally available. Yeah. I mean, two things interested me about this first, it's, it's more easily accessible machine learning stuff. So for anybody who says, look, I want to have some pre-trained models. I don't have access myself to some crazy voice data set or things like that to train some model to recognize voices. I can use what Google has. So it's a great reason to, uh, just start using machine learning, but then secondarily from a pivotal perspective, this is one of those things that helps you choose which cloud do you want to use? Like this would be a reason to maybe run PCF on Azure. If you're on GCP, if you're doing tons of this sort of machine learning stuff and you'd love that, or you might use Azure if you're doing more enterprise integration or Amazon for certain application services that these sort of first party services can help you decide where's the right place to run a particular workload. I think that's great because the infrastructure layers is starting to commoditize. Before we, uh, we, we stop optimizing our special guests, uh, inbox cleaning up time. As I was mentioning earlier, uh, I want, I wanted to mention just a few sort of meta things, if you will. First of all, we've got a bunch of cloud native roadshows. I think the, the Chicago one, I forget if the Minneapolis one has happened already, but this week on, uh, on the 28th of April, you got the Atlanta one. And then we've got DC Charlotte, all sorts of ones, all sort of like major cities you might think about. And if you go to pivotal.io slash roadshow, uh, you can look that up. And these are, these are free, you know, events that you can go to.

And, uh, I think they even have free lunch and maybe some breakfast. You can get my favorite, my favorite type of sausage or those flat patties that you get at like these, uh, these roadshow events. I don't know, I don't know what they do with them, but somehow I like them. They're, they're sort of like, I don't really like pop tarts, but you know, people enjoy the, the weirdness of pop tarts and hot pockets. And I think the sausages generally like that breakfasts. Yeah. I mean, you would generally a sausage patty versus link fellow. Oh, see, I like, I like a good sausage link when I may be getting a hotel breakfast, especially like in Europe. Cause they're very different. I don't, I don't know what's up with them, but I think I remember reading, I should go look this up on the old Snopes. That in America you have, you can only have like 20 or 30% fat in a sausage, whereas everywhere else they don't have that rule. That sounds like a total made up thing, but it's delightful. But you said it with authority. So I'm buying in. That's right. That's right. Uh, but anyways, we got, you know, it's not only in the U S that we have these, there's all sorts of roadshows. And I was just looking through the slides that we're doing this morning and they're impressive. Like demoing all the concepts of while you would want to do cloud native and going over what, uh, what DevOps is and how to think about the Google SRE style. Uh, things in your own shop and stuff like that. Yeah, this is not a sales roadshow. This is Google and pivotal together showing off cloud native practices and having good discussion. So I know we've had some really great turnout so far. So absolutely folks should, should jump on that. We're going all around the world for this. So it should be fun. And then just, just a couple more things. So we also have spring days in Chicago, New York, and Atlanta, the Atlanta one. And I was just at DevOps days, Atlanta. So I remember this is in July and that's, that's as the name would imply. I think it's a, it's a two day conference that goes over, um, a lot of, uh, spring concepts from the spring framework to boot, to cloud, maybe even spring cloud data flow. So, but, uh, definitely figuring one of those cities that should be a fun event. And then finally, uh, I was just noticing on our eBooks page, which is that pivotal.io slash eBooks. We have a preview edition of Cornelia's new book, but I haven't even looked at, but it's basically, I think an overview of, um, cloud native thinking and, and architecture and design.

So with that, let's bring back in our, uh, our special guest. Yeah, absolutely. We're really happy to, to have you want to talk a little bit about container networking and volume services and some of these things you've worked on. And then also just, I'm fascinated into how, you know, our teams work with these broader sets of other companies that are part of the cloud foundry foundation and how we work together across organizations and figure out these releases. And you are sitting here and something that's really central to how a lot of other teams are going to work, especially things like networking. So how do we, how do we kind of keep this a well-oiled machine? So I just thought it'd be fun to continue to talk to some engineering folks and, and learn about how we run open source and how we run product at pivotal. So you, uh, you touched on it, but anything, so it's mainly volume services and the network container networking, or those are separate teams, correct? You're not running, that's not one mashed together team. They are separate teams. And, um, for the volume services, there is an open source team with a PM. And so I, I really just focus on the pivotal side of it, like getting into, um, the ERT tile and getting it out to pivotal customers. Um, so that's not as much of like a hands-on, uh, you know, day-to-day, uh, PM management kind of job. Um, for the container networking side though, I, I am the product lead on the, uh, OSS side as well. So I, uh, you know, do a lot of the prioritization work with the foundation, work with, um, our engineering team in getting all, getting all these features delivered. And then I also work on the pivotal specific side of it, which is getting it in, you know, as a beta feature into PCF 1.10. And then fingers crossed, hopefully, um, maybe going GA pretty soon. Um, so that's, you know, it's, it's a little bit of a more complete, uh, ownership of, of the project there. Yeah, that makes sense. So for some of those who aren't network admins and, you know, just kind of take for granted how networking happens within a Cloud Foundry environment, I guess, start me off by walking through kind of what's a traditional flow today.

Hey, if I have application A and application B and that, you know, one's a front end, one's a backend service and they talk to each other, how does that flow through Cloud Foundry today? If I wasn't doing any special container networking? Um, yeah, so the flow for Cloud Foundry today is really, really simple. Um, and that's how I think most container networking solutions start off with, which is that your containers, they don't have, um, their own individual IP addresses outside of the host that they actually reside on. So if I have a front end and a backend app, um, their identity basically is, is your host IP and some random port. So the idea is if your front end wants to talk to your backend today, um, it'll send traffic out of, uh, the host and it'll go out as if it had coming from the host IP with some random source ports that's known as a SNAT or a source NAT. And NAT means network address translation. Um, it'll go all the way out to your load balancer, come back through to the Cloud Foundry entry point, which is the go router, and then be routed to an instance of the backend app. So the overall traffic path is, is really not direct. It, it, it, it hairpins through the router and it uses the go router as a method of service discovery to find, um, where your backend app is, is located. And that makes sense. Of course, I'm sorry for interrupting. I was just thinking, I'm sure that makes sense, especially when you have containers that come and go, when you have apps that scale back and forth, it inherently, it's nice to not know, to have to worry about where is all my stuff. Um, but I guess there's cases where people do want to talk and not go through that hairpin step. Um, yeah, exactly. So it, it, it, it's definitely a very simple, um, and easy way to think about this. But when you start looking at microservices and a whole lot of maybe private microservices and also latency sensitive microservices, maybe they don't want to go hairpin through the router.

Um, the other reason why you might want to talk directly between applications is so that you, you could apply application level policies that are more at the app level, um, and then apply them directly when you're communicating between applications. And in order to do that, perhaps you also need, um, some form of application identity so that, you know, when your traffic is reaching your destination, that this, this indeed came from the application that it, you know, um, was supposed to come from. And so you can, you can then configure more granular policies and more, um, application specific policies. So what's an example of a policy? Like, what does that look like to an in person? And what, what am I setting up in a policy? Um, so mostly when I talk about policy or it's network policy, and it's basically, um, if you're a firewall administrator, then, you know, this, this, this is something that you're probably really familiar with. But, um, for Cloud Foundry, we really want to talk about things in terms of applications. And so when you look at a policy for an application, it's really saying that, hey, my front-end app is allowed to talk to my back-end app, and it's allowed to do this using a specific protocol. So, for example, TCP or UDP, and then a specific port where you're allowed to connect to your back-end app. And so this defines, you know, what kind of network traffic you're basically going to allow into your back-end app. When I was first using this, I thought that was great, just because you didn't have to, I mean, your policies didn't have to know your containers, they just had to be app-aware. So it was this more app-aware policy, which seems much more scalable, maintainable than saying, well, these container IPs can talk to these container IPs, but it's going to change all the time. So do we handle all that complexity of saying, front-end and back-end need to talk to each other, apps are scaling, apps are being redeployed, we're keeping track of the IP mappings? Yeah, exactly.

And I think that's one of the unique things about this policy model, which is that as an application developer or even an operator, you don't have to worry about IPs and, you know, how they come and go. And that's one of the things that's unique about containers is that they're so ephemeral that your application can scale up and scale down and really keeping track of IPs is a big pain when you think of a system like it. So you're right, it's having an application-level policy really frees up the developer or the operator from really thinking about IP addresses and mapping them to applications and rules. When you're thinking about kind of setting up these policies, did we build up our own SDN, software-defined networking stack? Did we use anything that was off the shelf? Were we reimagining what this stuff looks like? I know we went through a couple of revisions as we built this out. So what was that journey? How did we decide what technology to create ourselves versus what sort of standards, if any, to use that were already in industry? You're right. It has been quite a journey. And I joined this journey, I think, partway through. I've been at the level about a year now. So there were some decisions that were made before I actually joined the team. And the primary thing that I think that is the basis for most of our feature support is that we wanted to make the container networking stack pretty pluggable. And we wanted to have app-level policies that stayed true to kind of the Cloud Foundry focus on the application developer. And so that's kind of where, you know, that's how most of the feature set that we planned for, that's where it basically came from. And so in terms of the different components and the different existing technologies available to us, we really wanted to make the stack based on a standards-based specification.

So that, you know, even if we didn't have the most sophisticated SDN solution that was batteries included, as we call it, a consumer could actually, you know, take a different SDN stack and plug it into the Cloud Foundry stack. So we based the networking stack on the container networking interface specification, or CNI. And so this would enable a different CNI plug-in to basically come in and integrate with us and provide more richer SDN capabilities. For the batteries included solution, we went with something pretty simple. Our focus was really to get the app-to-app direct connectivity and the direct policy working. And so we just used, in our beta release, we've used Flannel, which is a CNI plug-in that's available. And it's on, you know, GitHub and it's pretty widely used. And we used Flannel along with a VXLAN-based encapsulation for the batteries included in our first release. But there are other CNI plug-ins that could potentially provide richer SDN capabilities down the road. So if I can play a little bit of the straight man doofus here, those would be a long roundup to, like, some pretty basic questions. But, like, I remember several years ago, maybe this is five years ago, like, SDN was really a big deal. And I was like, oh, I should understand networking. And the end result is I still don't understand networking, which is delightful for me. But the only way that I, like, summarized it is that – and tell me where I'm wrong here as I get to ultimately the sort of, like, what's going on here or why this stuff matters is as designed. And I'm sure I'm supposed to know if this is TCP or IP or whatever, but I'll just say networking. The way that networking, as we know it works, is it's extremely static.

Like, essentially, the baseline case is, like, you've got a network admin and they log into some blinking cursor thing. And maybe they use a tool for it. But they basically say, this computer has this IP address and these people can talk to it. And, like, that's sort of, like, what you start with is someone having to manually configure stuff. And then the problem is if you have, I don't know, anything, virtualization and cloud things and ephemeral containers, they're always coming and going. And you can't keep up manually with – you're like Lucy at the Chocolate Factory to, like, keep up with, like, doing all this. So, you need some way of automating all of this networking stuff, which I always feel like, to some extent, SDN was trying to do. But I never really got a sense for, like, well, to be very naive about it, why that was a problem in the first place. Like, it seems like it should have worked that way all the time. And then I couldn't really ever understand how all the SDN stuff sorted out. It got pretty confusing about commercializing versus, like, what's the IEEE up to? And I don't know. So, I lost track. And then I moved on with the rest of my life. I don't know. I mean, is that sort of, like, the core of the problem, that automating networking at the speed of ephemeral containers is difficult? So, I think part of the – you know, you actually summarized it really well around SDN and kind of the challenges of it. I think part of the problem is that the way networking has evolved is that it started off from a pretty static kind of space where you had VLANs and switches and routers and not a whole lot of programmability and, you know, mix and match kind of things. And the way it's evolved and, you know, and the way it's evolved and, you know, now we have SDN. And even in the past five years, I think SDN itself has matured and really kind of focused itself on the problems that it's trying to solve. So, I do think that networking as you start talking about containers or, you know, even before this VMs, it has become a little more challenging where your traditional networking rules of, hey, we'll just put everything on one VLAN and have it connect to, like, a top-of-rack switch and then, you know, up to another switch is maybe not as flexible and not as extensible.

And I think we definitely see more of that as you bring in containers that come and go where your IP is no longer enough to actually identify a specific server or a specific application, right? I think you need to have maybe a little more intelligent way of configuring policy for things to talk to each other or even, you know, giving identity in terms of IP addresses to it. So, I think that, you know, networking in itself is pretty confusing. Yeah. And there's a lot there. Yeah. I mean, it seems like for the past, I don't know how many years, many years, basically at the application stack and even sort of like whatever is right below the application stack, we've just been like pushing networking far further than sort of like the core technology of it can handle. So, you know, folks like yourself and teams of people who are working on it are trying to figure out how to, like, layer on all of the extra infrastructure stuff we would need for networking to keep up with the way we want to architect and run our applications. Right. And I think it's also changing a lot, like the problem statement is perhaps a little bit different from way back when, when you had, you know, more static entities that where you could use, you know, just an IP address and like a five-tuple firewall rule to basically protect, protect, you know, protect, you know, protect communication between different entities on the network. Right. I think, yeah. And the other challenge that, you know, this comes up with, you know, ever so often around, we're running out of IP address space, so we should move to IPv6. And so there's, there's, there's, there's a lot could change, especially with the new needs that containers or, you know, containers put on the system.

And just in terms of scale and in terms of how much they move around and, and also just kind of moving the network stack up one level. I haven't, you're reminding me, I haven't heard people like freaking out about running out of IP addresses in a long time. Well, I think they're already out now. I think we're out of IPv4. That was a popular, like, you know, Y2K-ish meltdown moment that people were, were going through for a while. So that's maybe, maybe I just don't run in the same circles anymore, but I remember everyone used to get excited about like Microsoft and people buying up big blocks and how we're all going to be like, you know, eating dog food on the internet. Cause there'd been an apocalypse of, of IP address. Yeah, it hasn't happened yet. That's good. So, so then just to summarize, like, it seems like in the, the, the, the container to container and overall networking work that that's going on here, there's an, and add, add and correct me if I'm wrong, but there's kind of three things being accomplished. One is by controlling all of, all of the routing and making sure that, that traffic goes where it should and gets cut off by from where it doesn't. You're essentially adding some security levers or capabilities that you have. So you can feel more secure that bad traffic isn't going to go somewhere. And then maybe a little more subtly that the wrong traffic isn't going to go somewhere. So if you want to, if you've got some, I don't know what the kids call it nowadays, but if you've got some problematic node that's spewing out nonsense, you'd like to kind of shut it off and, and not have it, not have it at least DDoS yourself and shut things down. And then I think, I think the second thing is, is just the basics. A lot of what you're talking about is in a distributed application, you would like to be talked, you would like to talk to other nodes or services or processes, and you've got to figure out how to reach them. So you need something beyond just a directory of finding out what their address is and then resolving that to the right place. And then the third one, I don't know, in my simplistic way is sort of like, I don't know if it's strictly for performance, but sometimes, if not often, it would be better instead of, as you say, hairpinning, which is kind of going to, to a third source to know where to go.

And, you know, it would be, instead of hairpinning, you just go directly to something that's closer to talk to a peer, if you will. It'd be like, if I had to go all the way down to downtown Austin to ask my neighbor if they would pay for repairing the back fence, I could just go over to my neighbor's house and ask them instead. But like, are those three things kind of a good summary of like the benefits that you get when, with all the networking stuff that we're working on? Yeah, that is, that is a good summary. I think that there's one other subtle aspect of this, which is that, which, which is that you can now have other SDN solutions come in and integrate as well and provide a richer set of policies. Right. And this is what, this is what Richard was asking about a little bit is the, the actual, and this is your batteries included or not included part is the actual way all of this is accomplished is essentially pluggable. So you can other, other SDN things under it, which might bring you different features or implement them well. So the way that, the way that, that y'all have coded it up is you can swap that stuff in and out, or if you just want to use the batteries that always have some funny, weird name, you know, in the real world to play out the metaphor, not weird here, I guess. Anyways, you can always use the batteries that are included if you don't want to customize your power source. Yeah. I don't want to trivialize it and say that you, it's, you know, it's just an easy, like swap in and out. There is some integration work involved, but it at least opens up that door to, to have some, some sort of a pluggability. Yeah. That was always back when, you know, in, in any sort of application development, that's always the, the delightful snarkiness is of like, yes, just every week we're swapping in and out our implementations super easy. But, but, but, but at least you have that ability to, to spend the effort to, to move to other things. Yeah. One thing you subtly mentioned there, Coté and Usha was the, you know, as we think about discovery though, because, because these things are coming and going all the time, you know, this app, I'm clearly not going to hard code a bunch of IP addresses that say, this is how I go talk to this other app.

Like that would be insane that things are changing all the time. So this seems like a really great value prop for something like service registries where they're, you can cache it locally and it's constantly refreshed, but I need some way to look up the container addresses, right? Uh, exactly. And so the, one of the key use cases and the one that's actually running, um, on pivotal web services right now is, um, the use cases of using spring cloud services with container networking, um, where you can use the client side load balancing. Uh, capabilities of, of ribbon, um, to basically register your containers IP instead of a route. And so, um, you know, that, that's something that we've, we've actually tested and proved out and, and we see as, as a real use case. Um, so using spring cloud services, direct registration with container networking, so that's, so that's basically your, your, um, service discovery mechanism, so to speak with this new feature. Yeah, it's wild stuff. So I mean, the networking stuff, I think checks a few boxes for folks who maybe want to bring more complex network topologies to the platform and didn't want to just have traditional web apps that go through a router all the time. The other piece then is volume services, which has been, as long as I've been in this community, one of the knocks on things like cloud founder, even any container solution is where does my state go? And we talk about stateless apps, at least as the container itself, not storing state so I can replace it or whatever, but I still have to store data because any app without data is kind of lame. So it seems like most modern apps look at things like object storage or, you know, a Redis cache or Gemfire or using relational databases, whatever. But with your work with volume service, the goal was to say, look, if you've got apps that depend on NFV or NFS sort of V3 volumes, and I want to actually mount a volume to my container and write stuff that can survive container, you know, destruction and replacement, you're good to go. So, did you see this, I mean, did we see this as a big sort of mission critical thing for people trying to move older apps to the platform?

Is that where this came from? Pretty much. So I think that even for volume services, there are a number of different use cases that people have in mind. For the first release, it's currently a beta feature in PCF 1.10. We wanted to tackle the lift and shift use case, which is that I have applications that either, you know, they're third party applications, so I don't control them, and I can't rewrite them, or they're applications that are so old that I really don't want to rewrite them. But I want to bring them to cloud foundry, and these applications have an inherent dependence on having a file system. And that's either because they're using file systems as a mean of communication between applications, so like a source app would, you know, write a file, and then a desktop will basically read that same file and work on that data. Or because they just, you know, they were built with the assumption that they're going to have a file system for perhaps storing data that's required for their application to function. So that was the primary motivation, at least for the initial set of features. And so most of these kinds of users would have an existing, and we went with NFS v3 based on a lot of the investigation and discovery that was done before this. And so the idea is that you have an existing NFS v3 share, and you can bring up applications that are able to mount directories that are in that shared volume. So that's basically the feature there. So we're in beta right now. Are you looking for active feedback? How can listeners, even I guess with container networking as well, how can they, or would you like them to try things out and provide feedback? How would they actually do that?

What sort of feedback are you looking for? Yes, it would be great if we could get feedback. So for container, both for container networking and for volume services, we are looking for feedback around, you know, whether these features solve the actual use cases, whether there are gaps that, you know, that kind of are like, okay, you know, this is what's preventing me from actually deploying this feature in production. So there are, you know, different ways of trying these things out. So for container networking, for example, it's been running in Pivotal web services for some time. And so that's one way to try that out. It's also available in PCF 110, like both of these features. So a great way to actually provide feedback is to go through the Pivotal account teams to give feedback. I, you know, other than that, we're on the open source side, we're also really, really active on Slack. So that's another way to also get feedback. So I wanted to, you know, as we start to wrap up, I just want some of your opinions on, you know, as you do have this scope where you're looking at this work in open source with Pivotal directly, how do you kind of manage that work that I believe, I mean, correct me if I'm wrong, at least let's pick one, the container networking on the open source side. That's a bunch of different companies participating. Is that right? Yes. So our team has four people from Pivotal, and then we have one IBM on our team. And so it rotates on and off. We usually have representation from both IBM and Pivotal, usually on the container networking team. And then how do you manage the priorities and then backlog? Are you the final word? Do you balance that with other people? How do you kind of prioritize the right work? So I, we basically work pretty closely with the Garden and the Diego teams on Cloud Foundry, and then with Yui, who's the PMC lead. And we try and prioritize things in a way that's kind of aligned with where Cloud Foundry wants to go as a whole.

In terms of the container networking piece, you know, we usually follow the usual Cloud Foundry method of like, you know, putting out a proposal and then soliciting feedback and then implementing some of the features. So in terms of prioritization, it's been, I think it's the feedback from the community as well as from the Pivotal side is that we did need a more pluggable networking stack. So, you know, you know, that's, you know, that direction was set. So it wasn't really, it wasn't really tough to prioritize that. Sure. And in terms of prioritizing new features, it's the same thing, like kind of figuring out what are the top problems that both the open source users as well as digital customers are facing. And generally, you see some common trends because there are common problems that, you know, everyone's facing networking. Sure. Hopefully, we pick the right thing to solve. Yeah, I bet. So as you do try to run this and make sure that, you know, with all these different components, whether it's Diego or networking or all these different layers, Logregator, how do you make sure that your piece doesn't accidentally turn the whole concourse pipeline red? Like, how do you make sure this stuff integrates correctly? How quickly do you get feedback if you did happen, you know, someone on the team checked in something that broke something upstream? How do you maintain kind of the flow without breaking everybody? So that's, you know, it's, it's still something that we're not perfect, but there are times where we have broken someone's pipeline. But usually, like, it's, we get a pretty good idea of whether we're going to break something just based on our own continuous integration pipelines.

So we have a pipeline that basically runs and checks, like, upgrades and checks our own, like, acceptance tests. It also runs something known as CAPS, which is the Cloud Foundry acceptance test, and makes sure that it's all green before we actually cut a release. And then the other aspect of it is also just documenting changes. So since we are, you know, a fairly young project where there are oftentimes manifest changes and, and, and other breaking changes that perhaps downstream consumers would also have to make. So just kind of being more upfront with what those changes are and warning downstream consumers of these changes. Those are some ways that, you know, we, we try to make sure that we're good citizens. I think, I think that's a good place to end how you ensure that you're a good citizen and don't make everything else go red. Thanks for being a guest on here. It's, it's always good for me to, as, as is evident, educate myself a little bit. I appreciate it. And so if people want to, well, there's, there's two things. We usually just ask one standard question here, but if people want to dive more into either volume services or container networking, like what would you recommend they look at? Both volume services and container networking have a whole bunch of documentation on Cloud Foundry.org for the CF documentation. And then there's also a lot of pivotal documentation as well. In addition, we have a few blog posts that we posted around, you know, what are some of the new features that we introduced as well as what are some future things that we want to take on and, and some suggestions for how we might solve those. And as I mentioned, we're, we're a pretty friendly team on, on open source Slack. So, you know, that's, that's another way, a pretty easy way to reach us and give us feedback. And then also people are interested in keeping up with yourself.

Like, do you have, you have a Twitter account or anything like that? So my Twitter account is Usha Rusha. So I am, I, you know, that's one way to reach me also on LinkedIn and LinkedIn slash Rusha. Well, well, thanks again. And as always, thanks for listening. This has been Pivotal Conversations. If you want to get the most recent episodes, the best thing to do is to either look in the aforementioned Overcast or podcasting app or whatever a stitcher is. And you can search around there and find us and subscribe to it. And that'll just get every episode downloaded right after we post it. Unfortunately, we don't have the technology to have it downloaded before we post it. We'll work on that at some point. And if you want to go, you know, the place that we hosted is actually at SoundCloud.com slash Pivotal Conversations. You can find the RSS feed there and past episodes and things like that. And finally, we always post the full show notes at Pivotal.io slash podcast. And it's always handy if you share us in your various social media areas, in your Twitters, your Facebooks, your Linkedins. If you think the episode's good, it's nice to, if you recommend it to people so more folks can hear about it. And if you want to reach out to us either on Twitter, I'm Coté, just C-O-T-E. What's yours, Richard? Yeah, I'm at rsarother. So follow us and harass us. Or if you're trying to be one of these people who, like, doesn't engage in social media and does all your deep work and you barely use email, you can email us at podcast at pivotal.io. And with that, we'll see everyone next time. Bye-bye.

Tanzu Talk · 🍕 Get more pizza.