1 00:00:00,000 --> 00:00:05,000 Well, Richard, I don't have anything funny to say today, so we should just jump right into it. 2 00:00:05,260 --> 00:00:07,740 Do you want to introduce yourself, special guest? 3 00:00:08,100 --> 00:00:12,100 Sure. Well, thank you, firstly, for having me. 4 00:00:12,700 --> 00:00:16,420 My name is Usha Ramachandran. I am a product manager. 5 00:00:16,420 --> 00:00:27,620 I work at Pivotal, and I lead the container networking efforts for the open source Cloud Foundry platform as well as from the Pivotal side. 6 00:00:28,120 --> 00:00:35,460 In addition to container networking, I also PM the volume services features for Pivotal Cloud Foundry. 7 00:00:36,320 --> 00:00:38,860 Awesome. So storage and networking. 8 00:00:39,560 --> 00:00:42,780 Yeah. All the pluggable things. 9 00:00:42,780 --> 00:00:50,320 Yeah. So, I mean, I guess we'll have to get to it, but it seems like you ended up with the two most difficult things in all of cloud computing. 10 00:00:50,760 --> 00:00:51,680 So congratulations. 11 00:00:52,400 --> 00:00:55,700 Well, thank you. They're also two of the most interesting things. 12 00:00:56,100 --> 00:00:59,800 Yeah. Difficulty is also the source of interestingness. 13 00:01:00,460 --> 00:01:08,540 Well, before we get to that, before we get to the delight of complexity, as per usual, we have a few news items to go over. 14 00:01:08,900 --> 00:01:14,860 Last week, I think in our space that people care about, probably one of the bigger exciting things. 15 00:01:14,940 --> 00:01:17,180 I was out of town. It's quite tragic. 16 00:01:17,440 --> 00:01:20,180 Otherwise, I could have gone to, well, I don't know if I would go. 17 00:01:20,380 --> 00:01:25,440 I could have gone and hung out on the sidewalks, visited with my friend John Willis and other people at DockerCon. 18 00:01:26,020 --> 00:01:30,940 But it looks like there was basically like two announcements, mostly around toolkit things. 19 00:01:31,300 --> 00:01:36,400 Either building your own Linux distro, not distro, but your own Linux sort of, what would you call it? 20 00:01:36,640 --> 00:01:43,020 I can't say image because that's confusing, but your own Linux thingy that you would use in your various Docker stacks. 21 00:01:43,640 --> 00:01:49,540 And then if I understand it correctly, and that was called Linux Kit, there's this thing that they called Mobi because they got whales. 22 00:01:50,320 --> 00:01:58,500 And it's basically, if you were to look at all of the sort of like platform that you build out of all the Docker stuff, 23 00:01:58,500 --> 00:02:02,840 you know, the container itself, the orchestration, the command line tools, 24 00:02:03,300 --> 00:02:07,000 I think the Mobi thing is basically a toolkit for building that sort of, 25 00:02:07,000 --> 00:02:11,900 for sort of like building distros, to use my terminology that you would have with that, 26 00:02:11,960 --> 00:02:15,040 which I think, you know, several, several, a couple episodes ago, 27 00:02:15,040 --> 00:02:20,240 we talked about kind of the layout of all the various Cloud Foundry components and how you can kind of, 28 00:02:20,360 --> 00:02:22,080 we didn't really talk about how you can combine them, 29 00:02:22,200 --> 00:02:27,220 but any large platform has this idea of components that you can assemble together. 30 00:02:27,660 --> 00:02:31,080 And I'm trying really hard not to make like a early 90s cartoon reference. 31 00:02:32,340 --> 00:02:35,620 Yeah. Yeah. I mean, some of the news last week was coming out from, you know, 32 00:02:35,620 --> 00:02:40,640 all of a sudden github.com slash Docker slash Docker all of a sudden redirected to all the Mobi stuff. 33 00:02:40,740 --> 00:02:44,780 And so there was confusion about, did we rebrand it? I mean, wasn't that a really valuable name? 34 00:02:44,880 --> 00:02:50,760 Why did that disappear? So it was just some confusion about what is this new thing and what's rolling into that? 35 00:02:50,840 --> 00:02:52,760 What is this, what's its relation to upstream Docker? 36 00:02:53,100 --> 00:02:56,000 And I think it's all kind of being clarified, maybe it wasn't handled perfectly, 37 00:02:56,000 --> 00:02:58,860 but you know, either way containerization moves forward. 38 00:02:59,920 --> 00:03:06,620 Yeah. It seemed like after the, uh, after the, the storm of, of confusion subsided, it was fine. 39 00:03:06,820 --> 00:03:09,060 Not, not that big of a deal, but yeah. 40 00:03:09,060 --> 00:03:13,420 And then meanwhile, we had, we had our own release of something spring cloud data floor, 41 00:03:13,520 --> 00:03:16,160 data flow, spring cloud data flow. 42 00:03:17,200 --> 00:03:19,400 Maybe one day we'll get into floor management. 43 00:03:19,400 --> 00:03:27,460 So we understand how much potential there is when you, when you're stealing, but, uh, like I, I actually haven't looked over the, the release notes for that. 44 00:03:27,480 --> 00:03:33,380 I mean, we talked about this many episodes ago, kind of, kind of abstractly, but what's, what's bundled up in that release? 45 00:03:33,580 --> 00:03:37,040 Yeah. It's the first release candidate of a version one, two. 46 00:03:37,040 --> 00:03:47,360 And for those who aren't familiar with data flow, it's a way of assembling data processing pipelines as a series of microservices instead of big monolithic sort of ETL or even ESB processes. 47 00:03:47,360 --> 00:03:53,540 So what was neat in one too, the reason I wanted to highlight it was we're just doing more and more around simplifying the data pipeline. 48 00:03:53,700 --> 00:04:00,380 So compose tasks is the name of this feature, but you're actually building more of these stateful, uh, transition logical steps. 49 00:04:00,380 --> 00:04:03,640 Like do this and Hey, if the result of this is this, then jump to this step. 50 00:04:03,640 --> 00:04:09,460 So a little more workflow based versus just linear sort of sequencing that we're familiar with in data flow. 51 00:04:09,600 --> 00:04:16,560 And last week for fun, I spun up my own data flow server with spring cloud data flow and tried it out and was really impressed with where it's going. 52 00:04:16,720 --> 00:04:22,240 So between that, plus the real time metrics and monitoring, this is getting to be a pretty legit tool. 53 00:04:22,340 --> 00:04:25,620 I know our customers are really looking at this for a lot of use cases. 54 00:04:25,620 --> 00:04:29,340 So it's just neat to see a really great constant evolution of data flow. 55 00:04:29,400 --> 00:04:30,420 People should check that out. 56 00:04:30,420 --> 00:04:38,560 Yeah. Well, you know, as I, I forget how many episodes ago it was, but when we talked about this, uh, some time ago, I mean, I think, I think the, uh, things, 57 00:04:38,660 --> 00:04:56,220 things like spring cloud data flow as they evolve are interesting because there's a huge amount of enterprise applications that as boring as they are, are basically wrapping a workflow and processing both manual and automated around moving a piece or pieces of data along a, um, well, to use the word again, a workflow. 58 00:04:56,220 --> 00:04:58,940 So, and you can think about it in insurance companies or banks. 59 00:04:58,940 --> 00:05:13,960 I mean, the one that most people are familiar with would be like a mortgage application, uh, where there's a lot of information that you want to combine together beyond what you fill out as, as a user and have the bank or the lender eventually make some decision about, uh, if they're going to give you a loan or not. 60 00:05:13,960 --> 00:05:32,920 And, you know, that's a pretty simplistic thing, but anytime that you're just, uh, spent sending a lot of information, heavy decision-making that involves multiple stakeholders making a decision, ultimately it's, uh, in addition to many other use cases, it's, it's, it's something that having a data processing is helpful for. 61 00:05:32,920 --> 00:05:40,640 And then also kind of in our realm of things like, uh, a lot of that, you can't really accomplish with like off the shelf things. 62 00:05:40,640 --> 00:05:44,440 Like there's no, uh, there's no Zen desk for like mortgage application stuff. 63 00:05:44,440 --> 00:05:56,600 I mean, I, there probably are actually, but like highly customizing it and making it fit into the way that you do this stuff is where, uh, if you are someone lending money, it's part of your actual core business and probably competitive advantage. 64 00:05:56,980 --> 00:05:57,380 Yeah. 65 00:05:57,520 --> 00:05:59,020 I mean, what's exciting to me about this stuff. 66 00:05:59,020 --> 00:06:11,560 I also spent some time screwing around with spring cloud stream last week and getting it working in all sorts of ways, but just this idea of making it simpler to build event-driven apps and not depending on centralized integration experts to, to connect all my stuff. 67 00:06:11,780 --> 00:06:24,400 Instead, if you're a good developer who knows Java in this case, you can build an easily, easily build an event-driven app that hooks up to Kafka or rabbit or whatever and process data that does not take some sort of specialist skillset anymore. 68 00:06:24,400 --> 00:06:30,840 I think that's kind of exciting that integration doesn't have to be the bottleneck when you're doing these sort of continuous delivery of systems. 69 00:06:31,340 --> 00:06:44,340 And then speaking of filling out mortgage apps, maybe one day everyone will have the pleasure of just speaking to their computer, like in that movie to fill out a mortgage app and, and they can, they can answer their voice, their, uh, their price band and everything. 70 00:06:44,340 --> 00:06:46,320 But I mean, that's just a terrible segue. 71 00:06:46,540 --> 00:06:50,400 There's also a, uh, a Google speech API that became generally available. 72 00:06:50,620 --> 00:06:50,760 Yeah. 73 00:06:50,820 --> 00:06:56,180 I mean, two things interested me about this first, it's, it's more easily accessible machine learning stuff. 74 00:06:56,200 --> 00:06:59,380 So for anybody who says, look, I want to have some pre-trained models. 75 00:06:59,500 --> 00:07:05,440 I don't have access myself to some crazy voice data set or things like that to train some model to recognize voices. 76 00:07:05,440 --> 00:07:06,640 I can use what Google has. 77 00:07:06,640 --> 00:07:16,120 So it's a great reason to, uh, just start using machine learning, but then secondarily from a pivotal perspective, this is one of those things that helps you choose which cloud do you want to use? 78 00:07:16,220 --> 00:07:18,760 Like this would be a reason to maybe run PCF on Azure. 79 00:07:18,760 --> 00:07:34,620 If you're on GCP, if you're doing tons of this sort of machine learning stuff and you'd love that, or you might use Azure if you're doing more enterprise integration or Amazon for certain application services that these sort of first party services can help you decide where's the right place to run a particular workload. 80 00:07:34,620 --> 00:07:38,200 I think that's great because the infrastructure layers is starting to commoditize. 81 00:07:38,480 --> 00:07:43,440 Before we, uh, we, we stop optimizing our special guests, uh, inbox cleaning up time. 82 00:07:43,440 --> 00:07:48,500 As I was mentioning earlier, uh, I want, I wanted to mention just a few sort of meta things, if you will. 83 00:07:48,580 --> 00:07:51,460 First of all, we've got a bunch of cloud native roadshows. 84 00:07:51,660 --> 00:08:00,160 I think the, the Chicago one, I forget if the Minneapolis one has happened already, but this week on, uh, on the 28th of April, you got the Atlanta one. 85 00:08:00,160 --> 00:08:05,120 And then we've got DC Charlotte, all sorts of ones, all sort of like major cities you might think about. 86 00:08:05,120 --> 00:08:08,900 And if you go to pivotal.io slash roadshow, uh, you can look that up. 87 00:08:08,900 --> 00:08:12,580 And these are, these are free, you know, events that you can go to. 88 00:08:12,640 --> 00:08:15,460 And, uh, I think they even have free lunch and maybe some breakfast. 89 00:08:15,460 --> 00:08:21,360 You can get my favorite, my favorite type of sausage or those flat patties that you get at like these, uh, these roadshow events. 90 00:08:21,360 --> 00:08:25,220 I don't know, I don't know what they do with them, but somehow I like them. 91 00:08:25,220 --> 00:08:30,600 They're, they're sort of like, I don't really like pop tarts, but you know, people enjoy the, the weirdness of pop tarts and hot pockets. 92 00:08:30,600 --> 00:08:34,400 And I think the sausages generally like that breakfasts. 93 00:08:34,440 --> 00:08:34,620 Yeah. 94 00:08:34,640 --> 00:08:37,040 I mean, you would generally a sausage patty versus link fellow. 95 00:08:38,160 --> 00:08:43,460 Oh, see, I like, I like a good sausage link when I may be getting a hotel breakfast, especially like in Europe. 96 00:08:43,460 --> 00:08:44,760 Cause they're very different. 97 00:08:45,040 --> 00:08:49,360 I don't, I don't know what's up with them, but I think I remember reading, I should go look this up on the old Snopes. 98 00:08:49,360 --> 00:08:56,440 That in America you have, you can only have like 20 or 30% fat in a sausage, whereas everywhere else they don't have that rule. 99 00:08:56,860 --> 00:09:00,260 That sounds like a total made up thing, but it's delightful. 100 00:09:00,440 --> 00:09:01,380 But you said it with authority. 101 00:09:01,380 --> 00:09:02,360 So I'm buying in. 102 00:09:02,580 --> 00:09:03,000 That's right. 103 00:09:03,100 --> 00:09:03,520 That's right. 104 00:09:03,820 --> 00:09:08,340 Uh, but anyways, we got, you know, it's not only in the U S that we have these, there's all sorts of roadshows. 105 00:09:08,340 --> 00:09:11,320 And I was just looking through the slides that we're doing this morning and they're impressive. 106 00:09:11,720 --> 00:09:19,100 Like demoing all the concepts of while you would want to do cloud native and going over what, uh, what DevOps is and how to think about the Google SRE style. 107 00:09:19,100 --> 00:09:22,260 Uh, things in your own shop and stuff like that. 108 00:09:22,360 --> 00:09:24,800 Yeah, this is not a sales roadshow. 109 00:09:24,880 --> 00:09:29,600 This is Google and pivotal together showing off cloud native practices and having good discussion. 110 00:09:29,600 --> 00:09:31,660 So I know we've had some really great turnout so far. 111 00:09:31,660 --> 00:09:34,160 So absolutely folks should, should jump on that. 112 00:09:34,200 --> 00:09:35,540 We're going all around the world for this. 113 00:09:35,740 --> 00:09:36,300 So it should be fun. 114 00:09:36,720 --> 00:09:38,320 And then just, just a couple more things. 115 00:09:38,320 --> 00:09:43,240 So we also have spring days in Chicago, New York, and Atlanta, the Atlanta one. 116 00:09:43,240 --> 00:09:44,660 And I was just at DevOps days, Atlanta. 117 00:09:44,780 --> 00:09:48,360 So I remember this is in July and that's, that's as the name would imply. 118 00:09:48,360 --> 00:09:57,120 I think it's a, it's a two day conference that goes over, um, a lot of, uh, spring concepts from the spring framework to boot, to cloud, maybe even spring cloud data flow. 119 00:09:57,120 --> 00:10:00,620 So, but, uh, definitely figuring one of those cities that should be a fun event. 120 00:10:00,620 --> 00:10:06,280 And then finally, uh, I was just noticing on our eBooks page, which is that pivotal.io slash eBooks. 121 00:10:06,280 --> 00:10:17,300 We have a preview edition of Cornelia's new book, but I haven't even looked at, but it's basically, I think an overview of, um, cloud native thinking and, and architecture and design. 122 00:10:17,300 --> 00:10:20,980 So with that, let's bring back in our, uh, our special guest. 123 00:10:21,200 --> 00:10:21,840 Yeah, absolutely. 124 00:10:21,940 --> 00:10:28,260 We're really happy to, to have you want to talk a little bit about container networking and volume services and some of these things you've worked on. 125 00:10:28,260 --> 00:10:40,600 And then also just, I'm fascinated into how, you know, our teams work with these broader sets of other companies that are part of the cloud foundry foundation and how we work together across organizations and figure out these releases. 126 00:10:40,600 --> 00:10:47,660 And you are sitting here and something that's really central to how a lot of other teams are going to work, especially things like networking. 127 00:10:47,660 --> 00:10:50,620 So how do we, how do we kind of keep this a well-oiled machine? 128 00:10:50,620 --> 00:10:57,340 So I just thought it'd be fun to continue to talk to some engineering folks and, and learn about how we run open source and how we run product at pivotal. 129 00:10:57,340 --> 00:11:06,620 So you, uh, you touched on it, but anything, so it's mainly volume services and the network container networking, or those are separate teams, correct? 130 00:11:06,700 --> 00:11:08,880 You're not running, that's not one mashed together team. 131 00:11:09,040 --> 00:11:10,260 They are separate teams. 132 00:11:10,260 --> 00:11:15,500 And, um, for the volume services, there is an open source team with a PM. 133 00:11:15,500 --> 00:11:24,940 And so I, I really just focus on the pivotal side of it, like getting into, um, the ERT tile and getting it out to pivotal customers. 134 00:11:25,220 --> 00:11:33,500 Um, so that's not as much of like a hands-on, uh, you know, day-to-day, uh, PM management kind of job. 135 00:11:33,500 --> 00:11:42,040 Um, for the container networking side though, I, I am the product lead on the, uh, OSS side as well. 136 00:11:42,040 --> 00:11:52,560 So I, uh, you know, do a lot of the prioritization work with the foundation, work with, um, our engineering team in getting all, getting all these features delivered. 137 00:11:52,560 --> 00:12:01,700 And then I also work on the pivotal specific side of it, which is getting it in, you know, as a beta feature into PCF 1.10. 138 00:12:01,700 --> 00:12:06,800 And then fingers crossed, hopefully, um, maybe going GA pretty soon. 139 00:12:07,340 --> 00:12:13,420 Um, so that's, you know, it's, it's a little bit of a more complete, uh, ownership of, of the project there. 140 00:12:13,560 --> 00:12:14,420 Yeah, that makes sense. 141 00:12:14,620 --> 00:12:25,960 So for some of those who aren't network admins and, you know, just kind of take for granted how networking happens within a Cloud Foundry environment, I guess, start me off by walking through kind of what's a traditional flow today. 142 00:12:25,960 --> 00:12:35,200 Hey, if I have application A and application B and that, you know, one's a front end, one's a backend service and they talk to each other, how does that flow through Cloud Foundry today? 143 00:12:35,200 --> 00:12:37,400 If I wasn't doing any special container networking? 144 00:12:37,740 --> 00:12:41,940 Um, yeah, so the flow for Cloud Foundry today is really, really simple. 145 00:12:41,940 --> 00:12:55,880 Um, and that's how I think most container networking solutions start off with, which is that your containers, they don't have, um, their own individual IP addresses outside of the host that they actually reside on. 146 00:12:55,960 --> 00:13:06,000 So if I have a front end and a backend app, um, their identity basically is, is your host IP and some random port. 147 00:13:06,000 --> 00:13:23,500 So the idea is if your front end wants to talk to your backend today, um, it'll send traffic out of, uh, the host and it'll go out as if it had coming from the host IP with some random source ports that's known as a SNAT or a source NAT. 148 00:13:23,820 --> 00:13:26,180 And NAT means network address translation. 149 00:13:26,180 --> 00:13:39,780 Um, it'll go all the way out to your load balancer, come back through to the Cloud Foundry entry point, which is the go router, and then be routed to an instance of the backend app. 150 00:13:39,780 --> 00:13:44,580 So the overall traffic path is, is really not direct. 151 00:13:44,580 --> 00:13:56,020 It, it, it, it hairpins through the router and it uses the go router as a method of service discovery to find, um, where your backend app is, is located. 152 00:13:56,520 --> 00:13:57,720 And that makes sense. 153 00:13:57,720 --> 00:13:59,200 Of course, I'm sorry for interrupting. 154 00:13:59,340 --> 00:14:09,880 I was just thinking, I'm sure that makes sense, especially when you have containers that come and go, when you have apps that scale back and forth, it inherently, it's nice to not know, to have to worry about where is all my stuff. 155 00:14:09,880 --> 00:14:13,960 Um, but I guess there's cases where people do want to talk and not go through that hairpin step. 156 00:14:14,300 --> 00:14:15,060 Um, yeah, exactly. 157 00:14:15,240 --> 00:14:21,120 So it, it, it, it's definitely a very simple, um, and easy way to think about this. 158 00:14:21,240 --> 00:14:31,980 But when you start looking at microservices and a whole lot of maybe private microservices and also latency sensitive microservices, maybe they don't want to go hairpin through the router. 159 00:14:32,440 --> 00:14:49,440 Um, the other reason why you might want to talk directly between applications is so that you, you could apply application level policies that are more at the app level, um, and then apply them directly when you're communicating between applications. 160 00:14:49,600 --> 00:15:06,640 And in order to do that, perhaps you also need, um, some form of application identity so that, you know, when your traffic is reaching your destination, that this, this indeed came from the application that it, you know, um, was supposed to come from. 161 00:15:06,740 --> 00:15:14,600 And so you can, you can then configure more granular policies and more, um, application specific policies. 162 00:15:14,800 --> 00:15:16,500 So what's an example of a policy? 163 00:15:16,620 --> 00:15:18,960 Like, what does that look like to an in person? 164 00:15:18,960 --> 00:15:20,580 And what, what am I setting up in a policy? 165 00:15:20,820 --> 00:15:31,780 Um, so mostly when I talk about policy or it's network policy, and it's basically, um, if you're a firewall administrator, then, you know, this, this, this is something that you're probably really familiar with. 166 00:15:32,320 --> 00:15:37,200 But, um, for Cloud Foundry, we really want to talk about things in terms of applications. 167 00:15:37,380 --> 00:15:51,100 And so when you look at a policy for an application, it's really saying that, hey, my front-end app is allowed to talk to my back-end app, and it's allowed to do this using a specific protocol. 168 00:15:51,320 --> 00:15:58,740 So, for example, TCP or UDP, and then a specific port where you're allowed to connect to your back-end app. 169 00:15:58,820 --> 00:16:05,240 And so this defines, you know, what kind of network traffic you're basically going to allow into your back-end app. 170 00:16:05,240 --> 00:16:12,160 When I was first using this, I thought that was great, just because you didn't have to, I mean, your policies didn't have to know your containers, they just had to be app-aware. 171 00:16:12,400 --> 00:16:21,380 So it was this more app-aware policy, which seems much more scalable, maintainable than saying, well, these container IPs can talk to these container IPs, but it's going to change all the time. 172 00:16:21,380 --> 00:16:30,720 So do we handle all that complexity of saying, front-end and back-end need to talk to each other, apps are scaling, apps are being redeployed, we're keeping track of the IP mappings? 173 00:16:31,960 --> 00:16:32,920 Yeah, exactly. 174 00:16:33,420 --> 00:16:48,060 And I think that's one of the unique things about this policy model, which is that as an application developer or even an operator, you don't have to worry about IPs and, you know, how they come and go. 175 00:16:48,060 --> 00:17:05,340 And that's one of the things that's unique about containers is that they're so ephemeral that your application can scale up and scale down and really keeping track of IPs is a big pain when you think of a system like it. 176 00:17:05,460 --> 00:17:17,300 So you're right, it's having an application-level policy really frees up the developer or the operator from really thinking about IP addresses and mapping them to applications and rules. 177 00:17:17,300 --> 00:17:25,480 When you're thinking about kind of setting up these policies, did we build up our own SDN, software-defined networking stack? 178 00:17:25,560 --> 00:17:27,240 Did we use anything that was off the shelf? 179 00:17:27,340 --> 00:17:29,460 Were we reimagining what this stuff looks like? 180 00:17:29,660 --> 00:17:32,440 I know we went through a couple of revisions as we built this out. 181 00:17:32,540 --> 00:17:33,600 So what was that journey? 182 00:17:33,700 --> 00:17:39,340 How did we decide what technology to create ourselves versus what sort of standards, if any, to use that were already in industry? 183 00:17:39,720 --> 00:17:40,260 You're right. 184 00:17:40,300 --> 00:17:42,260 It has been quite a journey. 185 00:17:42,860 --> 00:17:46,480 And I joined this journey, I think, partway through. 186 00:17:46,980 --> 00:17:49,040 I've been at the level about a year now. 187 00:17:49,200 --> 00:17:54,180 So there were some decisions that were made before I actually joined the team. 188 00:17:55,560 --> 00:18:08,980 And the primary thing that I think that is the basis for most of our feature support is that we wanted to make the container networking stack pretty pluggable. 189 00:18:09,740 --> 00:18:21,300 And we wanted to have app-level policies that stayed true to kind of the Cloud Foundry focus on the application developer. 190 00:18:21,980 --> 00:18:31,540 And so that's kind of where, you know, that's how most of the feature set that we planned for, that's where it basically came from. 191 00:18:32,000 --> 00:18:45,460 And so in terms of the different components and the different existing technologies available to us, we really wanted to make the stack based on a standards-based specification. 192 00:18:46,020 --> 00:18:54,280 So that, you know, even if we didn't have the most sophisticated SDN solution that was batteries included, as we call it, 193 00:18:55,620 --> 00:19:02,680 a consumer could actually, you know, take a different SDN stack and plug it into the Cloud Foundry stack. 194 00:19:03,340 --> 00:19:10,340 So we based the networking stack on the container networking interface specification, or CNI. 195 00:19:11,940 --> 00:19:22,260 And so this would enable a different CNI plug-in to basically come in and integrate with us and provide more richer SDN capabilities. 196 00:19:22,900 --> 00:19:27,340 For the batteries included solution, we went with something pretty simple. 197 00:19:27,800 --> 00:19:34,860 Our focus was really to get the app-to-app direct connectivity and the direct policy working. 198 00:19:35,620 --> 00:19:43,840 And so we just used, in our beta release, we've used Flannel, which is a CNI plug-in that's available. 199 00:19:44,760 --> 00:19:48,960 And it's on, you know, GitHub and it's pretty widely used. 200 00:19:49,460 --> 00:19:59,060 And we used Flannel along with a VXLAN-based encapsulation for the batteries included in our first release. 201 00:19:59,520 --> 00:20:06,360 But there are other CNI plug-ins that could potentially provide richer SDN capabilities down the road. 202 00:20:06,360 --> 00:20:14,320 So if I can play a little bit of the straight man doofus here, those would be a long roundup to, like, some pretty basic questions. 203 00:20:14,520 --> 00:20:19,760 But, like, I remember several years ago, maybe this is five years ago, like, SDN was really a big deal. 204 00:20:19,840 --> 00:20:21,480 And I was like, oh, I should understand networking. 205 00:20:22,000 --> 00:20:26,220 And the end result is I still don't understand networking, which is delightful for me. 206 00:20:26,620 --> 00:20:39,160 But the only way that I, like, summarized it is that – and tell me where I'm wrong here as I get to ultimately the sort of, like, what's going on here or why this stuff matters is as designed. 207 00:20:39,680 --> 00:20:44,320 And I'm sure I'm supposed to know if this is TCP or IP or whatever, but I'll just say networking. 208 00:20:44,640 --> 00:20:48,400 The way that networking, as we know it works, is it's extremely static. 209 00:20:48,980 --> 00:20:55,060 Like, essentially, the baseline case is, like, you've got a network admin and they log into some blinking cursor thing. 210 00:20:55,620 --> 00:20:57,200 And maybe they use a tool for it. 211 00:20:57,200 --> 00:21:03,020 But they basically say, this computer has this IP address and these people can talk to it. 212 00:21:03,440 --> 00:21:07,540 And, like, that's sort of, like, what you start with is someone having to manually configure stuff. 213 00:21:08,120 --> 00:21:17,880 And then the problem is if you have, I don't know, anything, virtualization and cloud things and ephemeral containers, they're always coming and going. 214 00:21:17,880 --> 00:21:23,960 And you can't keep up manually with – you're like Lucy at the Chocolate Factory to, like, keep up with, like, doing all this. 215 00:21:24,000 --> 00:21:31,520 So, you need some way of automating all of this networking stuff, which I always feel like, to some extent, SDN was trying to do. 216 00:21:31,660 --> 00:21:37,760 But I never really got a sense for, like, well, to be very naive about it, why that was a problem in the first place. 217 00:21:37,980 --> 00:21:41,400 Like, it seems like it should have worked that way all the time. 218 00:21:41,500 --> 00:21:46,220 And then I couldn't really ever understand how all the SDN stuff sorted out. 219 00:21:46,220 --> 00:21:50,080 It got pretty confusing about commercializing versus, like, what's the IEEE up to? 220 00:21:50,220 --> 00:21:51,000 And I don't know. 221 00:21:51,060 --> 00:21:51,800 So, I lost track. 222 00:21:51,980 --> 00:21:53,660 And then I moved on with the rest of my life. 223 00:21:53,780 --> 00:21:54,040 I don't know. 224 00:21:54,080 --> 00:22:00,240 I mean, is that sort of, like, the core of the problem, that automating networking at the speed of ephemeral containers is difficult? 225 00:22:00,240 --> 00:22:08,920 So, I think part of the – you know, you actually summarized it really well around SDN and kind of the challenges of it. 226 00:22:09,520 --> 00:22:28,460 I think part of the problem is that the way networking has evolved is that it started off from a pretty static kind of space where you had VLANs and switches and routers and not a whole lot of programmability and, you know, mix and match kind of things. 227 00:22:28,660 --> 00:22:34,120 And the way it's evolved and, you know, and the way it's evolved and, you know, now we have SDN. 228 00:22:34,120 --> 00:22:43,920 And even in the past five years, I think SDN itself has matured and really kind of focused itself on the problems that it's trying to solve. 229 00:22:44,000 --> 00:23:09,760 So, I do think that networking as you start talking about containers or, you know, even before this VMs, it has become a little more challenging where your traditional networking rules of, hey, we'll just put everything on one VLAN and have it connect to, like, a top-of-rack switch and then, you know, up to another switch is maybe not as flexible and not as extensible. 230 00:23:10,760 --> 00:23:27,540 And I think we definitely see more of that as you bring in containers that come and go where your IP is no longer enough to actually identify a specific server or a specific application, right? 231 00:23:27,540 --> 00:23:40,460 I think you need to have maybe a little more intelligent way of configuring policy for things to talk to each other or even, you know, giving identity in terms of IP addresses to it. 232 00:23:40,960 --> 00:23:47,000 So, I think that, you know, networking in itself is pretty confusing. 233 00:23:47,640 --> 00:23:47,920 Yeah. 234 00:23:48,280 --> 00:23:49,320 And there's a lot there. 235 00:23:49,880 --> 00:23:50,140 Yeah. 236 00:23:50,520 --> 00:24:02,260 I mean, it seems like for the past, I don't know how many years, many years, basically at the application stack and even sort of like whatever is right below the application stack, 237 00:24:02,260 --> 00:24:08,880 we've just been like pushing networking far further than sort of like the core technology of it can handle. 238 00:24:09,620 --> 00:24:23,940 So, you know, folks like yourself and teams of people who are working on it are trying to figure out how to, like, layer on all of the extra infrastructure stuff we would need for networking to keep up with the way we want to architect and run our applications. 239 00:24:24,900 --> 00:24:25,340 Right. 240 00:24:25,340 --> 00:24:47,300 And I think it's also changing a lot, like the problem statement is perhaps a little bit different from way back when, when you had, you know, more static entities that where you could use, you know, just an IP address and like a five-tuple firewall rule to basically protect, 241 00:24:47,300 --> 00:24:53,880 protect, you know, protect, you know, protect communication between different entities on the network. 242 00:24:54,660 --> 00:24:55,220 Right. 243 00:24:55,220 --> 00:24:56,000 I think, yeah. 244 00:24:56,160 --> 00:25:06,360 And the other challenge that, you know, this comes up with, you know, ever so often around, we're running out of IP address space, so we should move to IPv6. 245 00:25:06,360 --> 00:25:15,580 And so there's, there's, there's, there's a lot could change, especially with the new needs that containers or, you know, containers put on the system. 246 00:25:15,580 --> 00:25:26,000 And just in terms of scale and in terms of how much they move around and, and also just kind of moving the network stack up one level. 247 00:25:26,000 --> 00:25:31,460 I haven't, you're reminding me, I haven't heard people like freaking out about running out of IP addresses in a long time. 248 00:25:31,460 --> 00:25:32,860 Well, I think they're already out now. 249 00:25:32,980 --> 00:25:34,280 I think we're out of IPv4. 250 00:25:34,640 --> 00:25:40,400 That was a popular, like, you know, Y2K-ish meltdown moment that people were, were going through for a while. 251 00:25:40,920 --> 00:25:51,140 So that's maybe, maybe I just don't run in the same circles anymore, but I remember everyone used to get excited about like Microsoft and people buying up big blocks and how we're all going to be like, you know, eating dog food on the internet. 252 00:25:51,140 --> 00:25:53,560 Cause there'd been an apocalypse of, of IP address. 253 00:25:53,700 --> 00:25:55,020 Yeah, it hasn't happened yet. 254 00:25:55,580 --> 00:25:56,200 That's good. 255 00:25:56,260 --> 00:26:10,220 So, so then just to summarize, like, it seems like in the, the, the, the container to container and overall networking work that that's going on here, there's an, and add, add and correct me if I'm wrong, but there's kind of three things being accomplished. 256 00:26:10,220 --> 00:26:19,640 One is by controlling all of, all of the routing and making sure that, that traffic goes where it should and gets cut off by from where it doesn't. 257 00:26:19,980 --> 00:26:24,280 You're essentially adding some security levers or capabilities that you have. 258 00:26:24,280 --> 00:26:27,960 So you can feel more secure that bad traffic isn't going to go somewhere. 259 00:26:28,160 --> 00:26:32,840 And then maybe a little more subtly that the wrong traffic isn't going to go somewhere. 260 00:26:33,040 --> 00:26:47,120 So if you want to, if you've got some, I don't know what the kids call it nowadays, but if you've got some problematic node that's spewing out nonsense, you'd like to kind of shut it off and, and not have it, not have it at least DDoS yourself and shut things down. 261 00:26:47,120 --> 00:26:50,200 And then I think, I think the second thing is, is just the basics. 262 00:26:50,200 --> 00:27:00,200 A lot of what you're talking about is in a distributed application, you would like to be talked, you would like to talk to other nodes or services or processes, and you've got to figure out how to reach them. 263 00:27:00,200 --> 00:27:09,100 So you need something beyond just a directory of finding out what their address is and then resolving that to the right place. 264 00:27:09,580 --> 00:27:25,340 And then the third one, I don't know, in my simplistic way is sort of like, I don't know if it's strictly for performance, but sometimes, if not often, it would be better instead of, as you say, hairpinning, which is kind of going to, to a third source to know where to go. 265 00:27:25,660 --> 00:27:31,840 And, you know, it would be, instead of hairpinning, you just go directly to something that's closer to talk to a peer, if you will. 266 00:27:31,880 --> 00:27:41,040 It'd be like, if I had to go all the way down to downtown Austin to ask my neighbor if they would pay for repairing the back fence, I could just go over to my neighbor's house and ask them instead. 267 00:27:41,680 --> 00:27:49,700 But like, are those three things kind of a good summary of like the benefits that you get when, with all the networking stuff that we're working on? 268 00:27:50,100 --> 00:27:52,360 Yeah, that is, that is a good summary. 269 00:27:52,360 --> 00:28:06,400 I think that there's one other subtle aspect of this, which is that, which, which is that you can now have other SDN solutions come in and integrate as well and provide a richer set of policies. 270 00:28:07,240 --> 00:28:07,400 Right. 271 00:28:07,540 --> 00:28:18,080 And this is what, this is what Richard was asking about a little bit is the, the actual, and this is your batteries included or not included part is the actual way all of this is accomplished is essentially pluggable. 272 00:28:18,080 --> 00:28:23,540 So you can other, other SDN things under it, which might bring you different features or implement them well. 273 00:28:23,680 --> 00:28:37,280 So the way that, the way that, that y'all have coded it up is you can swap that stuff in and out, or if you just want to use the batteries that always have some funny, weird name, you know, in the real world to play out the metaphor, not weird here, I guess. 274 00:28:37,960 --> 00:28:42,980 Anyways, you can always use the batteries that are included if you don't want to customize your power source. 275 00:28:43,280 --> 00:28:43,720 Yeah. 276 00:28:44,080 --> 00:28:49,220 I don't want to trivialize it and say that you, it's, you know, it's just an easy, like swap in and out. 277 00:28:49,440 --> 00:28:59,660 There is some integration work involved, but it at least opens up that door to, to have some, some sort of a pluggability. 278 00:29:00,020 --> 00:29:00,420 Yeah. 279 00:29:00,460 --> 00:29:11,780 That was always back when, you know, in, in any sort of application development, that's always the, the delightful snarkiness is of like, yes, just every week we're swapping in and out our implementations super easy. 280 00:29:12,500 --> 00:29:18,060 But, but, but, but at least you have that ability to, to spend the effort to, to move to other things. 281 00:29:18,520 --> 00:29:18,880 Yeah. 282 00:29:18,900 --> 00:29:32,360 One thing you subtly mentioned there, Coté and Usha was the, you know, as we think about discovery though, because, because these things are coming and going all the time, you know, this app, I'm clearly not going to hard code a bunch of IP addresses that say, this is how I go talk to this other app. 283 00:29:32,360 --> 00:29:34,620 Like that would be insane that things are changing all the time. 284 00:29:34,720 --> 00:29:43,940 So this seems like a really great value prop for something like service registries where they're, you can cache it locally and it's constantly refreshed, but I need some way to look up the container addresses, right? 285 00:29:45,280 --> 00:29:46,080 Uh, exactly. 286 00:29:46,260 --> 00:30:02,340 And so the, one of the key use cases and the one that's actually running, um, on pivotal web services right now is, um, the use cases of using spring cloud services with container networking, um, where you can use the client side load balancing. 287 00:30:02,360 --> 00:30:09,980 Uh, capabilities of, of ribbon, um, to basically register your containers IP instead of a route. 288 00:30:09,980 --> 00:30:18,240 And so, um, you know, that, that's something that we've, we've actually tested and proved out and, and we see as, as a real use case. 289 00:30:18,360 --> 00:30:31,700 Um, so using spring cloud services, direct registration with container networking, so that's, so that's basically your, your, um, service discovery mechanism, so to speak with this new feature. 290 00:30:32,360 --> 00:30:33,260 Yeah, it's wild stuff. 291 00:30:33,340 --> 00:30:43,000 So I mean, the networking stuff, I think checks a few boxes for folks who maybe want to bring more complex network topologies to the platform and didn't want to just have traditional web apps that go through a router all the time. 292 00:30:43,060 --> 00:30:52,520 The other piece then is volume services, which has been, as long as I've been in this community, one of the knocks on things like cloud founder, even any container solution is where does my state go? 293 00:30:52,640 --> 00:31:02,140 And we talk about stateless apps, at least as the container itself, not storing state so I can replace it or whatever, but I still have to store data because any app without data is kind of lame. 294 00:31:02,480 --> 00:31:11,280 So it seems like most modern apps look at things like object storage or, you know, a Redis cache or Gemfire or using relational databases, whatever. 295 00:31:11,620 --> 00:31:27,380 But with your work with volume service, the goal was to say, look, if you've got apps that depend on NFV or NFS sort of V3 volumes, and I want to actually mount a volume to my container and write stuff that can survive container, you know, destruction and replacement, you're good to go. 296 00:31:27,680 --> 00:31:33,620 So, did you see this, I mean, did we see this as a big sort of mission critical thing for people trying to move older apps to the platform? 297 00:31:33,700 --> 00:31:34,560 Is that where this came from? 298 00:31:34,560 --> 00:31:35,560 Pretty much. 299 00:31:35,740 --> 00:31:42,720 So I think that even for volume services, there are a number of different use cases that people have in mind. 300 00:31:43,400 --> 00:31:50,580 For the first release, it's currently a beta feature in PCF 1.10. 301 00:31:51,080 --> 00:32:07,880 We wanted to tackle the lift and shift use case, which is that I have applications that either, you know, they're third party applications, so I don't control them, and I can't rewrite them, or they're applications that are so old that I really don't want to rewrite them. 302 00:32:07,880 --> 00:32:18,200 But I want to bring them to cloud foundry, and these applications have an inherent dependence on having a file system. 303 00:32:18,720 --> 00:32:31,040 And that's either because they're using file systems as a mean of communication between applications, so like a source app would, you know, write a file, and then a desktop will basically read that same file and work on that data. 304 00:32:32,540 --> 00:32:44,500 Or because they just, you know, they were built with the assumption that they're going to have a file system for perhaps storing data that's required for their application to function. 305 00:32:44,500 --> 00:32:49,120 So that was the primary motivation, at least for the initial set of features. 306 00:32:50,020 --> 00:33:04,820 And so most of these kinds of users would have an existing, and we went with NFS v3 based on a lot of the investigation and discovery that was done before this. 307 00:33:05,500 --> 00:33:18,740 And so the idea is that you have an existing NFS v3 share, and you can bring up applications that are able to mount directories that are in that shared volume. 308 00:33:19,400 --> 00:33:22,660 So that's basically the feature there. 309 00:33:22,800 --> 00:33:23,740 So we're in beta right now. 310 00:33:23,780 --> 00:33:25,200 Are you looking for active feedback? 311 00:33:25,300 --> 00:33:31,640 How can listeners, even I guess with container networking as well, how can they, or would you like them to try things out and provide feedback? 312 00:33:31,860 --> 00:33:33,120 How would they actually do that? 313 00:33:33,460 --> 00:33:34,980 What sort of feedback are you looking for? 314 00:33:36,260 --> 00:33:39,220 Yes, it would be great if we could get feedback. 315 00:33:39,760 --> 00:33:58,280 So for container, both for container networking and for volume services, we are looking for feedback around, you know, whether these features solve the actual use cases, whether there are gaps that, you know, that kind of are like, okay, you know, this is what's preventing me from actually deploying this feature in production. 316 00:33:59,280 --> 00:34:03,960 So there are, you know, different ways of trying these things out. 317 00:34:03,960 --> 00:34:08,860 So for container networking, for example, it's been running in Pivotal web services for some time. 318 00:34:10,000 --> 00:34:12,160 And so that's one way to try that out. 319 00:34:12,220 --> 00:34:16,780 It's also available in PCF 110, like both of these features. 320 00:34:17,520 --> 00:34:24,420 So a great way to actually provide feedback is to go through the Pivotal account teams to give feedback. 321 00:34:25,140 --> 00:34:30,220 I, you know, other than that, we're on the open source side, we're also really, really active on Slack. 322 00:34:30,340 --> 00:34:32,860 So that's another way to also get feedback. 323 00:34:33,120 --> 00:34:41,900 So I wanted to, you know, as we start to wrap up, I just want some of your opinions on, you know, as you do have this scope where you're looking at this work in open source with Pivotal directly, 324 00:34:42,080 --> 00:34:49,700 how do you kind of manage that work that I believe, I mean, correct me if I'm wrong, at least let's pick one, the container networking on the open source side. 325 00:34:49,840 --> 00:34:51,680 That's a bunch of different companies participating. 326 00:34:51,780 --> 00:34:52,120 Is that right? 327 00:34:53,300 --> 00:34:53,700 Yes. 328 00:34:53,820 --> 00:35:00,960 So our team has four people from Pivotal, and then we have one IBM on our team. 329 00:35:01,280 --> 00:35:02,820 And so it rotates on and off. 330 00:35:02,880 --> 00:35:08,600 We usually have representation from both IBM and Pivotal, usually on the container networking team. 331 00:35:09,400 --> 00:35:11,760 And then how do you manage the priorities and then backlog? 332 00:35:11,960 --> 00:35:12,960 Are you the final word? 333 00:35:13,100 --> 00:35:15,060 Do you balance that with other people? 334 00:35:15,200 --> 00:35:17,320 How do you kind of prioritize the right work? 335 00:35:18,860 --> 00:35:26,700 So I, we basically work pretty closely with the Garden and the Diego teams on Cloud Foundry, and then with Yui, who's the PMC lead. 336 00:35:26,700 --> 00:35:35,260 And we try and prioritize things in a way that's kind of aligned with where Cloud Foundry wants to go as a whole. 337 00:35:35,920 --> 00:35:53,980 In terms of the container networking piece, you know, we usually follow the usual Cloud Foundry method of like, you know, putting out a proposal and then soliciting feedback and then implementing some of the features. 338 00:35:54,980 --> 00:36:11,640 So in terms of prioritization, it's been, I think it's the feedback from the community as well as from the Pivotal side is that we did need a more pluggable networking stack. 339 00:36:12,080 --> 00:36:18,060 So, you know, you know, that's, you know, that direction was set. 340 00:36:18,140 --> 00:36:22,040 So it wasn't really, it wasn't really tough to prioritize that. 341 00:36:22,480 --> 00:36:22,700 Sure. 342 00:36:23,420 --> 00:36:38,060 And in terms of prioritizing new features, it's the same thing, like kind of figuring out what are the top problems that both the open source users as well as digital customers are facing. 343 00:36:38,060 --> 00:36:46,380 And generally, you see some common trends because there are common problems that, you know, everyone's facing networking. 344 00:36:46,840 --> 00:36:47,200 Sure. 345 00:36:49,240 --> 00:36:50,980 Hopefully, we pick the right thing to solve. 346 00:36:51,580 --> 00:36:52,080 Yeah, I bet. 347 00:36:52,280 --> 00:37:00,080 So as you do try to run this and make sure that, you know, with all these different components, whether it's Diego or networking or all these different layers, Logregator, 348 00:37:00,220 --> 00:37:04,500 how do you make sure that your piece doesn't accidentally turn the whole concourse pipeline red? 349 00:37:04,700 --> 00:37:06,880 Like, how do you make sure this stuff integrates correctly? 350 00:37:07,060 --> 00:37:12,840 How quickly do you get feedback if you did happen, you know, someone on the team checked in something that broke something upstream? 351 00:37:12,960 --> 00:37:16,380 How do you maintain kind of the flow without breaking everybody? 352 00:37:18,060 --> 00:37:27,060 So that's, you know, it's, it's still something that we're not perfect, but there are times where we have broken someone's pipeline. 353 00:37:27,860 --> 00:37:37,400 But usually, like, it's, we get a pretty good idea of whether we're going to break something just based on our own continuous integration pipelines. 354 00:37:38,080 --> 00:37:45,780 So we have a pipeline that basically runs and checks, like, upgrades and checks our own, like, acceptance tests. 355 00:37:46,020 --> 00:37:55,260 It also runs something known as CAPS, which is the Cloud Foundry acceptance test, and makes sure that it's all green before we actually cut a release. 356 00:37:56,660 --> 00:38:02,240 And then the other aspect of it is also just documenting changes. 357 00:38:02,240 --> 00:38:16,040 So since we are, you know, a fairly young project where there are oftentimes manifest changes and, and, and other breaking changes that perhaps downstream consumers would also have to make. 358 00:38:16,140 --> 00:38:25,240 So just kind of being more upfront with what those changes are and warning downstream consumers of these changes. 359 00:38:25,760 --> 00:38:29,740 Those are some ways that, you know, we, we try to make sure that we're good citizens. 360 00:38:29,940 --> 00:38:36,280 I think, I think that's a good place to end how you ensure that you're a good citizen and don't make everything else go red. 361 00:38:36,280 --> 00:38:37,620 Thanks for being a guest on here. 362 00:38:38,280 --> 00:38:42,620 It's, it's always good for me to, as, as is evident, educate myself a little bit. 363 00:38:43,360 --> 00:38:44,260 I appreciate it. 364 00:38:44,320 --> 00:38:46,820 And so if people want to, well, there's, there's two things. 365 00:38:46,860 --> 00:38:55,380 We usually just ask one standard question here, but if people want to dive more into either volume services or container networking, like what would you recommend they look at? 366 00:38:57,580 --> 00:39:07,580 Both volume services and container networking have a whole bunch of documentation on Cloud Foundry.org for the CF documentation. 367 00:39:07,580 --> 00:39:11,180 And then there's also a lot of pivotal documentation as well. 368 00:39:11,180 --> 00:39:27,060 In addition, we have a few blog posts that we posted around, you know, what are some of the new features that we introduced as well as what are some future things that we want to take on and, and some suggestions for how we might solve those. 369 00:39:27,380 --> 00:39:31,720 And as I mentioned, we're, we're a pretty friendly team on, on open source Slack. 370 00:39:31,840 --> 00:39:37,740 So, you know, that's, that's another way, a pretty easy way to reach us and give us feedback. 371 00:39:37,740 --> 00:39:40,800 And then also people are interested in keeping up with yourself. 372 00:39:41,020 --> 00:39:43,080 Like, do you have, you have a Twitter account or anything like that? 373 00:39:43,420 --> 00:39:45,860 So my Twitter account is Usha Rusha. 374 00:39:47,660 --> 00:39:55,860 So I am, I, you know, that's one way to reach me also on LinkedIn and LinkedIn slash Rusha. 375 00:39:56,160 --> 00:39:57,060 Well, well, thanks again. 376 00:39:57,120 --> 00:39:58,780 And as always, thanks for listening. 377 00:39:58,920 --> 00:40:00,440 This has been Pivotal Conversations. 378 00:40:00,640 --> 00:40:09,300 If you want to get the most recent episodes, the best thing to do is to either look in the aforementioned Overcast or podcasting app or whatever a stitcher is. 379 00:40:09,380 --> 00:40:12,160 And you can search around there and find us and subscribe to it. 380 00:40:12,540 --> 00:40:16,260 And that'll just get every episode downloaded right after we post it. 381 00:40:16,640 --> 00:40:19,980 Unfortunately, we don't have the technology to have it downloaded before we post it. 382 00:40:20,180 --> 00:40:21,640 We'll work on that at some point. 383 00:40:22,100 --> 00:40:27,300 And if you want to go, you know, the place that we hosted is actually at SoundCloud.com slash Pivotal Conversations. 384 00:40:27,420 --> 00:40:31,020 You can find the RSS feed there and past episodes and things like that. 385 00:40:31,440 --> 00:40:35,960 And finally, we always post the full show notes at Pivotal.io slash podcast. 386 00:40:36,420 --> 00:40:44,200 And it's always handy if you share us in your various social media areas, in your Twitters, your Facebooks, your Linkedins. 387 00:40:44,580 --> 00:40:49,800 If you think the episode's good, it's nice to, if you recommend it to people so more folks can hear about it. 388 00:40:50,080 --> 00:40:55,180 And if you want to reach out to us either on Twitter, I'm Coté, just C-O-T-E. 389 00:40:55,480 --> 00:40:56,400 What's yours, Richard? 390 00:40:56,840 --> 00:40:58,000 Yeah, I'm at rsarother. 391 00:40:58,300 --> 00:40:59,960 So follow us and harass us. 392 00:41:00,300 --> 00:41:10,900 Or if you're trying to be one of these people who, like, doesn't engage in social media and does all your deep work and you barely use email, you can email us at podcast at pivotal.io. 393 00:41:11,160 --> 00:41:13,320 And with that, we'll see everyone next time. 394 00:41:13,560 --> 00:41:14,100 Bye-bye.