Networking in Kubernetes, and, The SpringOne Missing Track

Episode 242 · Tue, 18 Oct 2022 12:02:25 +0200 · 33:48 · SRT · archive.org · all Tanzu Talk

well this week we don't really have uh news to go over uh instead i'm gonna uh you'll you'll hear an interview from uh duffy at at covalent and uh he's gonna go over uh some ebpf stuff from explore us 2022 wait what year has this been 2022 it is 22 i'm really excited by the ebpf stuff uh i i started to read about it recently and it's really oh we'll have to we'll have to have an episode where you kind of explain what the deal with that is and maybe you can do a youtube short where you explain it all like ebpf in 60 seconds that that would be fun i would love to know enough to be able to record that video i don't yet but uh but yeah it looks fascinating so i thought what we would do uh right before uh i play that interview which will be coming up uh and it's an interview again from explore 2022 that alex williams of the new stack and i did uh with with duffy after his talk i kind of vaguely remember it so it'll be fun to listen to it again but you know we have we have the the best conference for developers and uh platform engineer types and operators and also managers who are getting their organization better at software coming up december 6th to 8th in san francisco SpringOne and uh i thought uh ben and i would share one talk we're we're looking forward to there uh this this episode now if you're listening to that and you want to register you can get 200 off with the code Coté 200 and you know ben you you pointed out uh yesterday when we were talking about this there's actually a track missing from SpringOne what what is that track i think this was a good insight there is i think every conference always has this missing track whenever you see the agenda there's a track missing the hallway track oh it's the track right it's the place where you can go to solve your problems whatever problem you have there's a hallway track for you and and i think i think that was a uh you know obviously that's being a little clever but that's that's fine that's good

because it is i think especially you know after three or so years it's it's uh easy to forget that like a huge part of the value is uh not just seeing people and hanging out with them but getting the chance to actually ask questions and kind of like i don't know most people who are implementing this kind of stuff in the old uh the old application development and and cloud native and container stack and all this like there's a whole bunch of idiosyncratic weird stuff uh that they have to put up with and really having someone to discuss it with is uh is quite handy like one of my colleagues was reminding me not reminding me he was telling me back in uh at our last in-person conference or maybe the second to last i think 2019 was the last one right ben if i remember it might have even been in austin of all places you mean in person yeah yeah yeah yeah yeah yeah yeah austin definitely yeah and and there were there was uh you know there was there was i think it was a panel discussion or a keynote or a talk and like someone from cerner had been talking about how they had been using uh um you know uh spring and and uh uh i think at the time pivotal cloud foundry stuff and then someone from wells fargo was up there and then afterwards you know uh people were able to talk to those individuals about how they were securing things and uh setting them up and if you imagine that all over the conference especially when it comes to the the development topics since there'll be so much spring stuff there you really get the chance to just uh go talk with people and have the conversation you're having to uh solve things out and to that end the talk i wanted to highlight uh i was looking through the talks again this morning is we have one from uh joseph at the u.s army and it's basically an overview of how they've been um he doesn't actually say it in the abstract but for us industry insiders i'll use i'll use the the summary word how they're doing dev sec ops stuff how they're you know securing the software the code that they have the process of doing it and uh hopefully all the way up to like the the runtime environment they're using i think i forget you know i think the u.s army uses both

the tanzu application service and uh kubernetes based stuff uh like a lot of people but it'll that'll be a good talk one to see how they're securing things because i figure you know uh if if the army's not doing some security thing it probably doesn't need to be done so whatever it is they're doing is probably the maximal amount of security you would ever care about uh and and so that'll be nice but then afterwards uh you know i don't want to put some some work on them but you know you could ask questions uh during the talk or maybe try to uh blockade speakers from leaving the room until the the kind av people are like oh we've got the next speaker coming up could y'all move it out into the hallway but it'll probably it'll be you know setting up your securing all this stuff is um there's all sorts of great standards and ways of doing it but as uh as oran who was a many many a couple of months ago uh talked about when he was on here like a lot of securing this stuff is the non-code stuff around policy and putting together your uh your CISO oriented stuff so that would be another the kind of conversation you could have with them and other people who are there uh so you can see the listing if you go to uh spring1.io and look for uh us army or joseph and i'll put a link to it in the show notes at tanzutalk.com but how about you ben what's what's one talk you think you're you're looking forward to that maybe crosses over into this missing track absolutely so uh oh apologies if you can hear that that's the sound of a platform being built exactly i'm having a platform built so apologies if you can hear that but uh yeah i i've picked out um one of my favorite people to listen to brian chang who is uh taking basically um a whole bunch of people and doing a panel you talked about panels just a second ago and sometimes i find a panel strangely comforting somehow there's lots of people to see there's folks to listen to and they've all got different perspectives on a particular problem so um brian's running running uh the tanzu vanguard panel and it's entitled priceless insights

so uh reading reading a little bit from the description um it's basically the super user community for vmware tanzu coming together to talk about various different problems from different perspectives and the two perspectives that i'm pretty sure you're going to you're going to hear about from this particular talk one is the kind of deep dive into something technical what that could be could it's something that will be decided probably on the day but it'll be something deeply technical i'm sure and then at the same time you'll also have this kind of um management aspects of that you know how do we optimize our organization to realize this new thing whether it's devsec ops or whether it's you know platforms or whatever it might be so i'm really looking forward to that one because there's also an audience q a so it's built into the talk you know it's not sort of a couple of minutes at the end it's kind of we're definitely going to dig deep into the experience of all these folks on the stage and we've got some really really good folks on the stage as well i'll read the list we've got greg mayer who works for vmware i know i know greg and he does some great stuff for example with accelerators on the tanzu application platform and we've got jim cole from great american insurance we've got jim fox also from great american insurance and we've got kelly uh shaffler uh sorry shaffer also uh sorry from uh one magnify she's a senior it director so we've got a good mix of technologists and management type folks in there so i'm pretty sure you'll get um a varied and quite interesting talk as a result so i'm looking forward to that one priceless insights by the vanguards well i have to say you you picked as as we used to say before i was born a real humdinger there because i uh i also had that one on my list of of talks i was looking forward to and i think uh i

think there's at least one person on that panel if not two who were we uh we had we had as as they say in the business a webinar or you could just call it an online talk we i i moderated a panel with uh several of the vanguard about uh like the use of metrics in in their organization you know doing software and it was it was great to talk with them and it was a good example of like of like the things you can do in person at a conference is uh you know when we were doing it online but you know when we were discussing it ahead of time and talking with each other there was we came up with uh you know new ideas and new things to talk about with relation to metrics and there was even a few sort of like oh i never thought about it that way and questions the panelists had on each other and how they were tracking things and fun fact uh that's when i was uh in the u.s last and i actually had that conversation in the middle of texas on i-35 uh in a starbucks parking lot because i needed to use the internet which was uh quite thankfully i had a nice car that i could sit in and do it and i went inside and i had some of those sous vide egg things uh the the mediterranean one i believe and did the internet that you were using also have a big key that was changed to like a lavatory seat exactly just like just like in that movie the jerk it was a a tire rim that it was uh it was secured to yep that's right but yeah that would be a good panel and to ben's point right like uh there there should be at least a third of it i would hope if not more well you can ask whatever you want uh whether it's off topic or not and have that conversation going well anyhow uh to check out those two talks and other ones obviously i'll put a link in the show notes to those exact panels at tonsutalk.com but you should go check out the rest of them if you go to spring1.io and you click on i think it's schedule and then you click on breakout sessions you can scroll through those and uh you know there's other stuff there as as ben has mentioned several times there's workshops all sorts of them uh mostly focused on programming but there's some operational ones and i just realized there's

also a whole .net day a whole .net conference uh going on as well for those in the .net world um and if you register uh you can use the code cote200 to get $200 off uh which you know just think of how many hot dogs that would buy uh if if you're into that whether they're real meat beef pork whatever it may be that's a lot of hot dogs you could you could purchase which you know us at vmware are going to be lacking that amount of hot dogs because you'll have give you'll be getting a discount but we would like you to enjoy the hot dogs instead of us now this talk of hot dogs i can't help but think of everything everywhere all at once the new movie oh yeah where there's a complete parallel universe that's where everyone's fingers are hot dogs exactly you gotta see it it's crazy that's right it's it's sort of like the uh the cute and endearing rick and morty instead of being uh you know very nihilistic and and offensive uh indeed well on that note what what you'll hear next is uh from explore 2022 uh so you know we had a we had a stage there and we had five people from i don't know i would say basically the like the like cloud container kubernetes ish software development startup world come out and uh we have one from uh uh we duffy from isovalent did a talk about ebpf about kind of um doing networking stuff as i'm always joking you know layer six and a half uh you know uh networking kind of at the service mesh type of layer uh and kind of how ebpf does that and obviously a little bit about isovalent if as i recall and then afterwards he came over and talked with alex williams uh and i and uh it was a fun interview of which i remember precious little so when i'm editing it in i'll have to uh see how great it was by listening to it uh and we'll be back next week uh with a little bit of news and and other stuff enjoy the interview so duffy cooley yes

you know one of those one of the things you're talking about is the security issues that we see in networks yep is that an identity issue at heart and is that kind of what you're getting at it can be yeah i mean basically you know what we're this is actually one of those things that we learned from kind of from that nice era exercise or from open beast which are network virtualization in general right was that um you know in many of the in many of the things that we have invented before we've kind of re-implemented the network just as it was designed before without really bringing any other capability to it right and so when we were thinking about how you know the shape of cilium and like what we were trying to develop here we realized that we needed to actually support things like identity it as a base primitive in connectivity between things right we need that just to be in there from the beginning so with cilium whenever you actually associate an endpoint whether that's an application in kubernetes or some other application running analytics server somewhere we give it a unique identity and then you can write network policy that is like that that basically allows you to enforce that um network access by identity rather than by ip or by any of that but before i go any further i'm alex williams of the new stack duffy cooley of isovalent and my co-host host of the vmware tanzu podcast michael Coté plus he does a lot of other things but michael we were talking about security but we were also talking about why do we need this new kind of approach that's right yeah you know so i don't really know much about networking so so maybe you can help me out but it seems like and you were touching on this it seems like what i see from my perspective of ignorance in in you know the last 10 years of networking is like

there's this tension between like static and dynamic and like those of us who live way up the stack are always like wanting to change things around all the time to be ephemeral as you were saying and it seems like as you go lower down the network stack they don't like that right and so like we sort of like i don't even know i mean i mean tell me what your perspective on this is but it seems like there's not really even a meeting in the middle it's like us on the top of the stack are sort of like all right well how about we put some new layers on there right and like it seems i don't know i mean that that tension seems weird is the wrong word but it's kind of like an oddity of like usual yeah yeah it's sort of like at some point and and i only mean this in like an exploratory best way but like shouldn't the network people figure that out or or like what where does like where do the cultures like differ and like why so many layers i mean there's there's quite a there's quite a lot of there's quite a lot quite a lot to chew on there so i'm going to begin from the beginning i would say that like you know in my career i have actually i've worn many of those hats right i've worked as a network architect for dsl networks across the country and i've worked all the way up into you know basically reinventing networking and network virtualization and now i'm kind of working on a networking solution that really implements itself with the application layer i would say that for the most part what i've seen uh over time is that a lot of network architectures are now kind of moving toward a place where they are considered plumbing and you have to think about that plumbing as highly available can handle all the water or whatever else is passing through right you know that that is the biggest part of your challenge right and where you're hooking that plumbing up are servers at this point right and that's kind of like that has become the way that we think about that underlying network for the most part yeah um and now but that that doesn't really but that introduces a new set of problems right so now that we've got this plumbing

thing worked out we're really good that we've come up with clause networks we've come up with all kinds of great ways to handle moving all of these bits back and forth how can we you know revisit or improve the security model of those applications that are consumers of that network yeah now for the most part i would say that this is where that division comes in this is what what i mean by that things have changed a bit right a lot of network architects are just focused on the plumbing problem anymore and the application and the developers and the platform security folks and those folks are now suddenly on the hook for solving that security problem right they've they've kind of like that by they i mean the networking people have pushed responsibility upward and in in some actually kind of awesome sleight of hand right like it because usually like you know again i have an application development background right and you're always it's not exactly how you put it but you're always trying to be ignorant of what's below you yeah right and you want and this is almost like a reversal of that flow of like hey how about you're very knowledgeable of like the layer right right below you exactly and now but what's interesting now with like different orchestration systems like kubernetes and and and others that are out there what we've seen is that like we're you know the platform folks and people who are writing software for the platform folks are are thinking about how can we make it so that people who are in the application business can just tell us what should be okay what should be allowed or not allowed right because at the end of the day you know as an application developer which services your application should be talking to right right and me as a networking guy i i don't care right like i just want to know i just want to know like what you should be doing so that i can put some reasonable card laurels around that so right right you you want you want the people as a network person up the stack to specify what they need i mean when i hear myself saying that it sounds ridiculously obvious but as opposed to giving you proactively being in charge of of all the intricacy of the networking stuff that happens as as you're saying is something as complicated as an app running in kubernetes right because there's a lot

going on there one of the other shifts in paradigm in this particular case that has always really blew my mind is that you know from a traditional networking background you spend a lot of your time defining you spend a lot of your time like looking for primitives that you can use as identity so that you can write firewall policies that restrict access between things in multiple areas right yeah say i have databases over here and they have financial records and i have my applications over here and they have like access to that financial record information and i want to make sure that only those ones that are supposed to have access to that financial information have it sure so i need some form of identity again right i need to have i have some place to start and so traditionally what i've had to do was i'd say these applications are only going to have these ip addresses and then i can say these ip addresses have access to the database and woe be to anyone who figures out how to get an ip address into that subnet because they'll be able to get around it and do all that stuff what's changed though is that with a lot of these orchestration systems now what we're writing network policy is on is label sets so we label every endpoint you see this in right lumio you see this in kubernetes you see this in everything yeah right you just write you you label everything and then you say things with this label can talk to things with this label which as a networking guy was a huge paradigm shift right right it made this whole thing so much simpler than it ever had right because not only do you escape uh numbers as your primary currency but like you're saying you can start having groups of things that are labeled so i think it it gets back to identity yeah and again just the search for it and like how do we how do we make that identity either as atomic as it needs to be or at least in a way that we can you know enforce that that security there but can we back up a little bit because yeah why we why do we need all this new stuff oh yeah that was the question i was that was a question that michael was posing like like i was reading a story you have on the psyllium site about the history of networking essentially yeah and a lot

of it gets down to the hardware versus the software story in many ways but but why what what what was what was happening of 10 15 years ago that put us on this path you know i i've i got to work at core os for about a year before it was acquired by red hat and one of my absolute favorite talks that um branded phillips who was our cto yeah i think he really raised the point that where things change in a really approachable way that really blew my mind he said every day you know another 20 000 people are logging onto the internet for the first time yeah and worldwide at the time i think it was there were 20 000 individuals working in it to understand this problem ah we are significantly outmanned yeah like like we cannot possibly with the technology and the practices and the and the procedures and everything else that we had 15 years ago solve the problem of today let alone the problem of why why was that at capacity because there are more people consuming more services consuming the internet putting their photos up getting gmail getting all of these applications and what resulted from that on the back end as a result with more people and more people and more people so what resulted is basically a change in the way that we think about infrastructure now we you know at the time we were thinking you know what if you if you rack 20 servers and you have an application stack on each of those 20 servers that was enough but it was an overload issue yeah and so you couldn't add more servers yep because you couldn't have enough cable enough buildings yeah enough machines and this is where you end up with like containerization taking over right because we figured out a way to actually put hundreds of these applications onto a single server and we figured out a way to actually manage the life cycle of them so that you could change them very quickly without having to go through all of the time it would take to change that 20 application stack on 20 servers probably but now the the networking as i understand it for instance the ebpf is as much about the application the

service maybe the load balancer uh the uh the security layer the firewall i mean you were talking about facebook for instance right you know how they how do they use it they uh they they leverage ebpf to actually handle what they call um i think it's called cantran or they it's their load balancing layer in front of all of the properties at facebook so if you have any service that is a part of a facebook page there will be a load balancer usually more than one between the the your phone and say that particular service and all of those load balancers are implemented in ebpf so is ebpf enough i think ebpf gives us quite a lot of capability to dig into this problem in a very material way right like and again it changes the way that our entry point is what isn't enough for sure is if we're only looking at packets as they enter the network or leave the network because like you know like i was saying if you look at like if you're looking at evaluating networking equipment today or firewalls or any of that stuff today right you you measure the capability of that piece of equipment in packets per second right this kind of speaks to that scale problem that we're talking about packets per second that's that's the way you measure it if it's a firewall or network switch or any of these things that's what you're looking at with uh ebpf we can actually make decisions about whether to allow or deny traffic drop packet i mean drop drop this traffic before it ever even moves up the stack and becomes a packet we can see that traffic and make this routing decisions about it before it even enters the network uh before it ever hits the wire or any of that stuff so really with ebpf it gives us the capability of actually moving so much closer to the source of this of this huge scale problems that we can really have a very significant effect in changing the way we think about the solution

so you know in in the uh uh the kubernetes world there's this constant sort of consternation about like well developers should do this or they should not do this like you know get involved in things and i and like what's your thinking and experience in the excuse me in the networking space like is who does this right like like is it because obviously someone needs to come in everything you know what you're just describing there's decisions to make there and like even day-by-day interaction like is there is there now like container network admin like like what what kind of roles are filling that need because i assume that like if i really know how to like convert a screen into pdfs like i'm an application developer i'm probably not doing that unless i'm desperate but like like what is that role and and do people do it like you know i think it's i think it's uh i think it's developing like for for quite a while i would say it was the platform it was the platform team in general that was the network right now it's kind of moving into more like a sec a sec devops kind of role and and that sort of thing where you have the people who are responsible for enforcing kind of the security posture of the company working alongside development teams to help them understand what changes need what changes are needed and how to implement those things right and in some companies this is actually like a production engineering world right but it actually does move more toward like if you are going to be a consumer of this platform uh we still need to be able to protect and enforce a reasonable security posture for the company yeah so we need to basically ensure that you get those resources so that you can be a good consumer of that platform that you can accomplish your goals and still not like blow up the security posture of yeah right and so that's where i i feel like culturally it's changing right like traditionally it's it's been like platform uh it's been the platform team whoever gave you that account to that to that um to that system but now i feel like it's kind of moving more toward people solving this

in a production a production management or production systems role where you have embedded people in different teams responsible for the outcome or the the usage of that platform right and they're responsible to ensure that they're communicating up like what right right and so i guess i mean i don't know i'm just making this up in my head but there's kind of like three three functions when it comes to like like networking as you're discussing in the kubernetes world and other things there's there's the function of just like uh your application being able to operate right like you know because basically i don't know with exceptions that are good or bad like if you're running an application in kubernetes it's a distributed application so you need a network just to like live and you don't want to think about it at all right right right where am i going to get this ip address no right and i i thought about the deployment that should be enough totally totally and then you had yeah exactly then you have the second concern that's sort of like more external facing like the performance of things like especially if it's higher scale where you need to add in more load balancers and interact with cdns and everything yeah but then the other thing that also seems like a new and interesting maybe exciting uh like thing is like you can do security better like you can do security i don't even i don't know it's another thing i don't know about but it seems like you're given more handholds you're given more opportunity to like do stuff uh which is nice to layer on top of everything else there yeah i mean something you'll hear a lot of times you know in this industry is a thing called micro segmentation right micro segmentation is the idea that we can either through the construct of identity or you know understanding how to actually get really atomic with network policies and those sorts of things be able to describe how to segment just some small port of an application from some other small part of an application right right rather than gross green controls are you allowed in the data center or are you not that's like a that's like an external

firewall problem right we're really familiar with how to solve that problem yeah yeah but can application talk to application b application a talk to application b how do you solve that problem right your west east not yours but the west east metaphor yeah yeah you know yeah and that's where that's where things like leveraging upf to kind of implement this stuff they put us as close to that endpoint as it as we can get right we can dynamically change the network policy or the or or the um security posture of that application very quickly without having to without without having to go back and configure all these firewalls and all those other stuff that's the big payoff for this micro segmentation right right right is that if you can do it and you come come up with a reasonable way to implement it and you allow your developer to say for example to look at all the traffic that went by or was attempting to go by the last couple hours and just pick the things that are supposed to be allowed and write network policy this is a feature of the psyllium enterprise ui right if you if you enable that sort of self-service so that people can actually just do the easy thing click the thing and make it and make it so that this is now their network policy and then that goes through an approval process if needed um then you have a much like you have a much more secure posture because you're securing all the way to the application yeah rather than just to the very large data center right lateral movement as i think most folks know in security and security minded lateral movement is your biggest security risk once people get in that's a big problem but if they can get in and own everything else around them that's a whole different problem yeah yeah yeah yeah for sure yeah and and so so like i mean as a as a last thing um you know well let me preface this is by saying like to force yourself to answer the following question you know because the answer is always like you know it depends it's flow right like life everything is flow right like um like

when it comes to this type of networking like what's your model for thinking about when we're done with it not done like we're not going to do it anymore but that like it's now mainstream right like you know dns is done right like there's we're not we're not like scrapping to have it be the way that we do anything so like like like when you think about it like how would we know if we're sort of like done scrapping for this this is an interesting one because i think that i think it comes down you know it's like i've actually so i've worked in kubernetes for quite a few years and that's the same question i think the answer is the same it's going to come down to like when it's boring like i was saying before you're not when you are interacting with a kubernetes api or your ci system or what have you when you're thinking about how you're deploying your applications you're not thinking about the network and that's fine right i don't want you to think about that right i want to enable you to think about the security of that network i want you to be i want to enable you to say my application over here is not talking to this other application and i want to know what's happening on the wire between them i want some observability capability that lets me see that right uh-huh i want to answer that question that these are all things i would consider check marks to tell me the things about right yeah right and and we've done that with psyllium right psyllium is an OS's project i feel like we've actually we're we're at the place now where psyllium is a stable solid option for pretty much any of these environments and i feel like it's in a better position to call itself done than a lot of the others that i've seen in space right right so where do you go from here then well we're going from here is i think you know within isovalent we we say sometimes that if there is a way to leverage ebpf in the stack that would give us some competitive advantage of this market and the answer is yes so if we look at tetragon for example one of the other security things we haven't talked about tetragon gives us the ability to actually what's happened to know what's happening again at that what is that then tetragon what is that it is the it is the ability to instrument what's happening at the application layer right so before i said you can see like traffic moving back and

forth yeah with tetragon i can show you which process made the connect call i can tell you what its linux capabilities which are like the permission system in front of the linux kernel i can tell you what capabilities it has i can let you write application policy that allows organized access to specific things within your underlying file system right i can actually move you know to different places in the evpf stack and provide primitives that allow you to improve the security for all of that as well and that's what's next for us it's like that piece of it the service mesh piece of it i think you know that's that's where we're headed all right that was fun it's like every like i said i'm always pretty ignorant so every time i talk with a networking person i feel a little bit better yeah i got it that's right i got it for now a little bit more well tomorrow alex we have a whole lot of interviews yeah five or six of them so if anyone's watching in the live stream turn in tomorrow and thanks so much for uh coming up here it was it was fun to hear about it was uh you got a nice way of explaining stuff that's not uh thanks toffee too technical i appreciate it appreciate it i'll see you next time bye bye

Tanzu Talk · 🍕 Get more pizza.