Platform Automation and Kubernetes with T-Mobile and Altoros

Episode 153 · 49:05 · SRT · archive.org · all Tanzu Talk

domain t-mobile i think we've heard from t-mobile before they uh i've done a lot of good stuff uh with pivotal so yeah yeah i think um no i don't think we've had t-mobile on the podcast but we right only had t-mobile speak at cf summit and i've heard them speak at SpringOne platform um and it's a great story uh so it's it's always exciting to hear and i know that the team's really been kind of pushing the boundaries and the limits of a lot of stuff and so i think a lot of folks stand to gain by just hearing what the t-mobile team is up to and we've also got uh ryan on from altoros and i think that's also another thing that's really valuable for folks is to understand how you know other folks in the community consultants like altoros are are how they're helping because i think some people might assume that sure those are great for getting started getting off the ground but you know here's a team like t-mobile that's three years into the journey four years into the journey um and you know there's still a lot of help that uh folks at altoros could do so i can't wait to hear more about this i'm i'm eager to learn yeah so why don't we start uh talking a little bit uh to to t-mobile matt and uh muhammad are online joining us uh and i think you know we're going to drill into some of the things you've done around platform automation and some of the benefits there uh especially working with pks maybe we could start uh either matt or muhammad and just a quick intro uh introduce yourselves and then maybe just tell us a little bit about like set the foundation uh kind of your role um and and and how you guys uh kind of got started with pks so my name is mama salman been with t-mobile for last uh five to six years uh i joined uh pks

pks team uh last year and we started using kubernetes on pks so i think there was a huge ask of automation automation around that time and uh how we are going to manage um uh our pipelines and our uh upgrades and our third-party software upgrades uh uh because we knew we are going to grow really fast so uh we started working on concourse with altoros that's that's when we brought in uh brian from altoros and started pairing up with him to make sure uh we don't run into any kind of issues going forward uh so we we started with uh one foundation we built it manually and after that we started i started um uh working on automation pipelines uh like platform automation and stuff like that so uh i think we uh we started with uh we are at uh 10 to 12 upgrades now on pks level and we did it all by uh platform automation and uh there are some other kubernetes uh kubernetes level pipelines that we worked on i think we can talk about it too later on that actually helped us build a cluster within one hour 24 node cluster uh a prox a prox one hour uh with all third party uh software installs on top of pks and um customer ready cluster i'll let mad at some some some of his thoughts to it yeah my name is matt burphy uh i've been with the pks team for about a year and a half uh here at t-mobile um like a lot of things that someone said uh one of the nice things that uh pivotal's brought forward for us is the platform automation uh in the past uh we've had to kind of write or um create our own pipelines and um really start from ground zero and pivotals brought forward the pivotal automation uh support which has really helped us a lot in um um just building our foundation and then once a foundation is built we move on with the pks

cluster installed um i believe we're at uh 3.0 right now right we just went to platform automation version we are at t-01 and um we just tested that with ryan um and updated one of our foundation with that other than that uh yeah as it's working for us uh really smooth we did some we did uh entire foundations i believe there are 10 foundations how many foundations i think we're up to 12. 12. good foundation within one night using uh platform automation patching uh applying a security patch on all the all the foundation so it's working pretty pretty good for us um uh and uh ryan has helped us a lot in that in that project and and um uh we continue uh work with him um on all the pipelines and and uh and making things better yeah it brings a lot of uh a lot of a lot of good things for us like uh uh even when you get down to ssl certificate management and bbr backups all of that's being provided with the platform automation which has really helped us out tremendously so and then you know maybe just to help folks listening who may not be as familiar with um pcf or or pks specifically you know you mentioned up to 12 foundations for for pks which is pivotals kubernetes distribution what does the foundation represent uh that's um it's a lot like uh in pcf cloud foundry um uh we a foundation we call it's like an ops manager installation um so once that is once that's done you can build several kubernetes clusters within that that ops man installation so um that's we we just call that a a foundation here but uh okay so that that's sort of the the bosch layer underneath exactly okay exactly exactly um and then on top of that you're running you actually

then build the the kubernetes clusters themselves and you mentioned that your platform automation is not just building out the foundation but it's it's building the clusters on top of it um is do i have that right that that is correct so we have uh seven hardware regions if i recall correctly uh and then we have 12 uh pks foundations on underneath running on top of that and then and then we have uh i don't remember the number exactly on kubernetes cluster because we are every day building clusters every day uh but we are over we are over approximately 40 clusters running uh running on the all those 12 pks foundations so uh that's how that's how the infrastructure is laid out here at t mobile and it's all on on-prem on-prem solution uh del hardware running in our data centers okay so you've got you've got about 40 clusters running across the various foundations and you mentioned you're building clusters every day so you're not building them and then letting them run for a while uh like how often are they getting rebuilt and and what what's the reason behind rebuilding these clusters well but maybe every day was a stretch but uh we're um we we do have sandbox environments where we'll tear them down and build them up and and they're for our use um as a matter of fact we have one cluster that uh is rebuilt once a week via a pipeline uh cicd stuff so um to you once we roll out a production cluster or a um a non-production environment for our developers uh that's pretty stable and we we don't just go rebuilding those so okay but however we do use the pipelines to patch those and upgrade and various things but when once they're built and delivered to the customer they're they're pretty stable uh we have a sandbox cluster that actually rebuilds every every week just to make sure we are up to date everywhere software wise and everything that uh that we want to apply

we apply on that cluster first before going to the customer cluster uh and that's our that's our patching standards and applying any patches to our current kubernetes cluster that's that's the path we follow so that's how the rebuilds of the cluster uh works here but once the customer are on the cluster customer facing cluster doesn't get rebuilt or any it's just we apply patches on that and that's uh that's actually pipeline and um and um a hundred percent uptime on on those most of the time um uh there's no customer impact and um when we apply patches and and uh or up or update the foundations um on those on those clusters most of the time however we we still are in uh we are still ramping up we are building clusters and and uh new environments um um quite often so we're we're not gonna we're not stable here at 12 installs of ops fan and bosch you know that that's going to be increasing quite a bit so okay great i think you had a question yeah well i just wanted to bring ryan into the conversation uh and you know we definitely want to get into some of the before and after we'd love to hear uh you know what what the what the process was like before you implemented a lot of this platform automation uh and and i i suspect part of the reason uh you're working with ryan and altoros was to was to help with that automation uh but ryan just bringing you into the conversation could you just tell us a little bit about um altoros for those who don't uh know the company uh and and your i guess your perspective you know from a 30 000 foot level on on platform automation yeah of course jeff um so altoros has been going as a consultant company for the last 18 years it was founded in 2001 um we've got our headquarters out in the san francisco bay area um but we also operate at a global scale scale we have offices in the us and norway denmark the uk and bs start in

ireland at the moment and i work remotely with t-mobile um we have about uh 350 staff now um of which we have about um just over 30 uh cloud foundry consultants and we have a whole um team dedicated to uh pcf and cloud foundry um so i've been working basically as a consultant with several different companies that are implementing pivotal cloud foundry and pivotal kubernetes uh services um and i just get a large uh view of uh different different ways that it can be deployed and some of the challenges that are in enterprises today and just with that experience i can take that from many different companies and i've used it to apply to t-mobile um in terms of platform automation um it's really been an iterative process um we originally started with pcf and there was a a suite of uh templates or pipeline examples that uh were called pcf pipelines uh that we used to use we started implementing that automation just because when you're doing things manually you can make a lot of human error and mistakes and when t-mobile is growing so rapidly um it's easy for technical debt to spiral out of control so we just want to keep things fast and uh deploy things as quickly as possible so um there's been massive improvements and uh pcf pipelines has been built upon to kind of create this uh product platform automation um and it's seen several improvements throughout the years and it's become stable and it was now like generally available as of may there so we're running on 3.0 like uh selman mentioned and uh yeah it's it's been a great experience so far so it's it's really helped reduce the amount of time to keep uh code and the platform up to date especially when there's such a fast release cycle from pivotal with security patches on the operating system with stem cells um coming out every other week um it's really important that we keep the the kubernetes environment secure up to date and that we can rebuild it often if there's any

issues or errors yeah it sort of reminds me of that adage of uh you know if it if it hurts do it more often uh so you know things like uh building out foundations or patching which as you mentioned is happening all the time with the the volume of and velocity i should say of uh stem cells and patches that pivotal releases so um rather than make that something that is a painful one-off exercise that just has to be repeated automated uh so and it's interesting to hear that this has been kind of a journey i remember when pcf pipelines uh and back in like 2017 or so was starting to get used by a lot of customers so it's great to hear how it's evolved yeah we've definitely seen the potential and um it's it's it's kind of one of those things where cloud security is quite different these days and it's definitely a big question for large enterprises such as t-mobile it's like how do we ensure that the platform uh and these foundations and these environments are secure so we try to implement the the three hours of security which is rotate uh repave and repair um and that that is that is um uh one of the main strategies with such a large distributed platform such as pcf and kubernetes um the the quicker that you can rebuild something uh the the less likely it is that security breaches will actually cause a significant impact and uh i think matt said yesterday a great thing was like uh we want to fail forward and fail fast um matt and uh and muhammad i love the way that you guys were talking about your customers but it might be worth clarifying here your customers that you're talking about you're talking about t-mobile's internal developers correct correct uh but you've got a very sort of product orientation towards how you think of those developers they're not just these other

people who work at my company who keep asking me for stuff right like they're your customers so you're you're thinking a lot about i can just tell from the language you're using you're clearly thinking a lot about the experience that those developers have what can you tell tell me about that experience now that um t-mobile's been kind of ramping up with providing kubernetes environments to those developers to those customers um and and rolling out pks what's the the developer experience that that you strive for and and what have you kind of learned along the way uh so we our customers are mostly internal um uh t-mobile uh developers and um uh since we are a new platform one year old so we are making sure everybody's aware uh what we are doing and um so we are hosting this uh pks 101 kind of pks slash tke 101 um uh classes uh at t-mobile making sure customers are aware of it how how things works at in t-mobile world we just had a um uh a tk 101 class in in bellevue in our headquarters where uh 40 developers showed up and then kind of uh bent over environment um and and showed them how um and help them deploy a simple app on our uh kubernetes cluster which are running on pks so uh the experience has been really good since uh last one year and uh we have seen a lot of improvements 100 percent uh some of that went 100 100 on pks environment i think four of our apps are running taking 100 production traffic at this moment and we are uh we continue continue to make a lot of poc work with our uh app teams and making sure that they they have whatever they need to be successful on this platform yeah it's been a learning experience for both the developers and us um with pks um we're able to um patch you know in the middle of the day provided that the developers have used 12 factor rules and um have followed certain standards uh for microservices or container

type orchestration platforms so um uh it's definitely important to us that even when you know our non-prod environment um to us everything's prod because if if our non-prod environment goes down that's where all the development is happening um so um it's important to both us and the developers and it's it's it's been a learning experience and we're we're starting to get good at it so it's pretty exciting some uh some of the patches you applied in the past um on the pks layer uh with the 100 percent um production traffic running on them and uh a few times we didn't hear anything from the customer because the patching went so smooth uh we got we got into some issues uh later on but we are uh can we continue to make improvements on our environments and making infrastructure layer better uh to uh best fit for our customers so you mentioned something that's that's kind of interesting and um i love the idea of the classes where you're you're kind of engaging with those developers and teaching them what they need to know so they could be successful in the platform and that sounds a lot like what i remember um hearing from the t-mobile team that they were doing a couple years ago with pcf back then just using what's now called the the pivotal application service um and matt you mentioned that uh folks are or should be able to have you know no downtime for their apps running in pks provided that they're following 12 factors so what i wanted to ask even before you mentioned that but now i'm really curious is really what's sort of different that you've seen especially in terms of how you need to enable those developers to be successful what's different between um pks and the the application service that that you've been offering to those developers for some time i wouldn't say there's much difference um it's kind of the same the same concept um some some things that we're we're you know we there was a lot of lessons

learned with pcf we were on a long journey there and i think we've we've uh done a good job of learning those lessons and applying them here in pks land um take for instance in our prod environment we have two prod environments and we encourage our our customers to um spread them out across both environments so if there is something critical uh where they can't have pod disruption at all um uh hopefully they have um they can switch their a10 services or their level loading services to say the prod 2 environment and then we can do whatever we need in the prod 1 environment so things like that there's some lessons learned we've used yeah so it doesn't sound like there's much of a kind of shortcut or anything it's not like by using pks it's somehow easier like i think there's sometimes a perception that trying to get my application to be ready to run in the application service is oh it's hard it's too much work right because you have to make it you know minimum of four factors but it sounds like really best practices even in pks you should be doing the same thing yeah yeah definitely um pks seems uh pretty popular here because uh the pcf world um there are some stricter rules for microservices um okay so it is perceived it is perceived oh sorry to cut you off go ahead go ahead just just to add to what matt's saying there is uh the pivotal application service pcf it can be a quite opinionated platform um for web applications and you have to follow 12 factor applications i believe that kubernetes gives you more control um at a lower level at the container level there is like a large data workloads and anything that needs persistent stateful storage um is a good fit for pks so it's really when the application and customers like developers which are our customers come to us we try to go for an assessment with them to kind of see where

their application is the best fit a lot of the time for legacy systems um it's easier to just do a lift and shift instead of uh rebuilding the application from scratch um there's no real shortcut uh if you want your application to run in the cloud these days you should really be following the 12 factors um for creating applications um but sometimes that's not possible and you need to get something um migrated quicker um so pks uh and kubernetes people believe it's easier but there's so much of things that you have to uh integrate into your um microservice container application system such as like logging and monitoring and there's a whole different range of things that go into it so there's no real shortcuts you should still be applying best practices and a lot of them from uh pivotal uh pivotal's application service we try to apply those and uh teach customers about that even if they're going to pks and i wonder you you ryan have seen uh you know as a as a consultant you work across a number of different companies what is your experience been uh not just t-mobile but but broader in terms of uh your customers thinking about when to use pas versus pks a more opinionated platform like the application service versus kubernetes you mentioned you have kind of a criteria you you look to uh when deciding which which is the best fit for different applications how has that played out in the real world uh in the different customers you've worked with yeah um so that's a really good question jeff um so basically um we we do a different a load of different uh capabilities one of them is uh very popular uh is the the pivotal dojos the platform dojos um and some application transformation can go into that as well um but but when when a customer is having we try to identify pain points um it could be something from not being able to deploy or maybe there's some downtime or there's latency or errors um on the vm level or maybe they're just looking to cut costs and move from vms

to containers um but i do see a lot of similar patterns across a lot of customers um so we've kind of created some um projects and uh deliverables that are fairly common to to a lot of people uh and a lot of large enterprises uh but i do see them the same questions being asked time and time again um so there's no real easy solution um but we try to take those lessons learned from from our companies and customers uh and put it into like best practices and recommendations uh and we often make a lot of blog posts about it um but yeah it's really it's really about teaching the developers um how to do cloud native development and what those best practices are uh and if that falls into place then it's really really easy and simple to use and they see much much faster deployments um much more reliability with the production workloads and we actually have the developers owning the applications so that if there is an issue um it's fixed much much faster so if we come back to um the platform automation kind of journey that the pks team at t-mobile has been on you mentioned uh muhammad that uh you that built the first foundation manually and then started to introduce platform automation working with ryan how did that sort of uh how did it come to be that you started to work with ryan and alturos on that platform automation project for pks so uh ryan has worked um with us previously on a pcf site so we knew that ryan can provide us um a lot of help there and um so that's why we kind of um brought in ryan and and um to use the expertise uh over there and then um kind of build the testing environment and started using um platform automation over there making sure the upgrades will go smooth and um with a minimum customer impact and um and um kind of uh kind of started uh testing the whole whole platform

automation uh layer over there making sure ops manager upgrades are a smooth pks style um um uh creates a smooth with uh with minimum customer impact uh so uh kind of kind of making the start making improvements over there making sure uh uh uh making sure our customers are um uh at at minimum impact when we do these upgrades and i can kind of start to expand uh from there and and the upgrades in np environment and then uh then prod and uh uh right now we are uh 100 percent automated on 99 percent automated on um on uh pks uh pks layer okay yeah we we have a really we have a really good uh relationship with ryan and alturas um especially with uh in the pas world and um using concourse and the nice thing about uh platform automation is it's the same tool used in pas as well as pks so it was a real good fit to have alturas and ryan help us out to get the pks stuff off the ground in the cicd world yeah just to add to what matt's saying there um just to put it into a bigger high-level perspective um we have really small operations team i've mostly been working with matt and salman um and you're talking about like a team of two or three operators or devops uh engineers um running all these foundations and thousands of containers and thousands of pods for for hundreds of developers and like hundreds of applications um and really that's that's why the automation is essential um there's no way that you can manage something with such a small operations team and i see this in so many customers is like the ops team can be really really small and if you have the correct automation in place not just with uh platform automation um but uh expanding out into the infrastructure automation with terraform as well um it's it's really the platform automation is very modular and it gives you

the building blocks to kind of piece together what bits you want and then expand upon that so we've expanded the platform automation to include building out the the kubernetes clusters with the the pks cli and we've expanded that into building out the low balancers um uh that connect up to pks and uh configuring the ingress and configuring splunk collectors and just configuring the monitoring and we've hooked all that into the platform automation that pivotal has provided with but uh it's really um building blocks that we can use and then expand upon so the the idea is to automate everything so that we can have this small team of two or three people running the entire platform and it basically runs itself okay so what i'm hearing is you know these aren't just raw kubernetes clusters that are getting spun up for those internal customers they're they're pretty well outfitted with a lot of um you know important infrastructure around it like the load balancing and uh splunk logging aggregation and um so you're really building out a more uh whole platform for those developers that is just built on kubernetes uh just yeah and that you know rings familiar with folks like kelsey hightower out there tweeting it's like hey kubernetes is a platform for building platforms it's not really the developer experience itself do i have that right yeah absolutely there's so much more to it um and we've just kind of expanded it and expanded it and everything that we need so it's at a click of a button we run the pipeline and it builds out a full cluster that's ready for the developers to go on instantly um and if we rebuild it we have everything there um again from the login to the dashboards to the cluster role bindings to the persisted storage at us attached uh and then we also run like sonoboy

which is like a conformance tool um to make sure that uh the best practices and standards are met with the kubernetes cluster and that it's configured correctly um so we try to package everything and we do make it a platform so that um it's kind of similar to the pivotal application service that everything that that's provided we also provide in kubernetes um all in one yeah as soon as we run the pipeline on on a kubernetes cluster we make sure that the cluster is customer ready for them to deploy their workload on it and making sure um uh that we don't have any manual work after that uh that's the whole point of uh kubernetes cluster pipeline was delivering a customer ready cluster with all third-party components on on top of pks layer and what by by working with ryan um so thank you by the way for kind of explaining the background and and the the good experience and relationship that you've had with ryan and alturos that made it kind of a natural fit for building out the kubernetes space platform um but muhammad and matt i'm curious what did having ryan uh helping with this platform automation build out what did that free you guys up to go do uh well it it did definitely help us uh because we're also involved uh uh salmon and i in kubernetes support as well and um certainly before ryan came we were really buried in the concourse stuff trying to home cook this ourselves um and just you know kind of going through public get repos and looking for things and just really trying to get a grasp on concourse and had no time for um actual kubernetes support so it's really free to have to be able to do um things like that we're now now we're able to take on different tasks and you know more r d stuff or gangway or um any new stuff coming up in the kubernetes down the line so it uh it truly uh uh once the hard work is over it it does kind of run itself

until you know an upgrade or something like that or um it's just made rolling out a new environment so much easier uh and i think you know we've been me and matt and someone we've been parried like pretty much every day for the past six months which is how long this engagement's been running and we picked up even small things hopefully they've learned um anything from restful apis to bash scripting just doing the whole pair programming really is good for knowledge transfer and skills transfer so hopefully that they if if they need to add even more automation in the future they can build it themselves uh would you agree with that matt definitely definitely it wasn't like you came in here and did all the work for us it's uh uh definitely a lot of knowledge transfer taught us how to fish i guess for the old analogy that's how it's used yeah that never gets old no it's so many different scenarios but but it's a good it's a good point and i wonder maybe if you could expand a little bit on and this is i guess kind of tangential to the main topic but you know what what is the relationship expand a little bit on the relationship uh working with uh a service provider like alturos how do you how do you build that trust you know what how does it actually work on the ground how how do you develop that that working relationship when you're coming from you know somebody coming from the outside how do you make the most of that engagement well again for us uh we we had already had ryan for uh pas support but i i guess what it started with is uh just um you know uh someone and i told our management hey look you know we're kind of swimming here we need some help um we need someone like ryan and alturos to come in and kind of show us the ropes and bringing ryan in and uh having a meeting and kind of set in the bar and this is what we expect uh we we had weekly meetings on uh how we're doing take things off the list that are done and adding things and um so the trust was built uh quite early in the process so it's uh i don't know do you have anything more to put i mean we were doing a lot of

stuff uh a lot of stuff manually uh uh before ryan came in and we were installing um a lot of components on kubernetes cluster uh uh kind of manually and and uh we had some pipelines separated out pipelines that would do work but we kind of needed something that uh ties everything together and then kind of needed somebody who knows um who knows how these pipelines and architecture work on the from the from the on the ground level so that's when that's when uh ryan uh comes into play and kind of uh coming back to the pcf player ryan already worked with pcf team uh i think last last year a couple of years ago so we kind of had that trust already so as soon as we uh we started talking about this ryan was the first person who who came to our mind and and kind of um started um talking to alturos how to bring him in again and uh work with us on pks uh pks side so uh yeah i mean we man we achieved a lot and uh on platform layer um third-party installation uh building kubernetes cluster customer ready um that's that's how it went yeah and uh just just to add to what salmon and matt said there what i see from customers is uh there is a lot of stuff done manually like large enterprises are under so much time pressure to get production workloads in on the really strict deadlines and what ends up happening happening is uh all of this stuff is done manually to kind of do proof of concept and kind of prove the value and value is really important but once you've got that initial value you then have to go back and say can we automate this um because uh from now on i like if you're if we're doing anything new like as the projects went on the scope's gotten bigger and bigger and uh new things have been added so rather than building something else we we initially try to build it out automatically first time um rather than building it out and revisiting it um later and because it never really gets done it gets pushed back and pushed back and

higher priorities come so what i try to get matt and salmon to do is anytime we've got a new thing to build out it has to be done automatically out of the box and we try to automate it first thing um and that that it takes longer to automate something but the benefits long-term pay off um because you're not having the repeat tedious processes over and over again um which if you're a linux system admin uh you probably that's the worst thing you can be doing in the job right so you got to put in that investment early uh and it pays off over time yeah yeah and and it also helps us extend extend things across across the environment and you know um i mean uh we couldn't imagine building 4d clusters um uh different set of uh different set of eyes and hands and and um every cluster would look a little bit different in that scenario but when when automation comes into play you know uh you know everything is going to look exactly the same the way you want it to be yeah even just the little things like naming standards can make a huge difference in trying to find something when you've got tons of data centers and tons of resource pools on premise and just trying to find like troubleshoot and bug if everything's named consistently the same way it just makes it much much easier to operate troubleshooting so just one one thing that uh matt you had mentioned uh the like the weekly meetings just in terms of how you're kind of i guess running the engagement with el toros you mentioned these weekly meetings where sort of things come off of that backlog and then and new things come on um which sounds a lot like you know know iteration planning meetings and this is not a a consulting engagement where things were all defined up front as like okay here's a statement of work and this is what's going to happen in the next six months it's like you know every week what's going to get done actually changes a little bit and that that also ties then um to me to at least the way i heard it to

what you were saying ryan in terms of every new thing that shows up like the scope keeps changing and growing but every new thing that shows up we make sure to automate it the first time um and i just i mean if am i thinking about that right in terms of part of the way that you're working with el toros is with things like weekly iteration meetings where you know what we thought we needed that and we actually decided we don't need that this other thing is more important and that's what we're going to build and by the way we're going to build it automated the first time definitely definitely we um it was a definitely a living breathing document almost like a real loose sprint type meeting it's like okay what did we get done this these two weeks um yeah we're done with that and oh by the way um well you know pks 1.3 came out and uh we haven't really built something around an upgrade or um let's make sure we have this down so that comes on the list and uh we just all kind of check our things and ryan gives us a yep that looks good or hey let's do this or that um it was a very uh fluent engagement with him it wasn't like you said uh we're going to do these 10 things and it's going to cost this much and it's got to be done in six months um it was it was quite flexible which was awesome it's something that we really needed so yeah i can i mean just you know that obviously uh aligns really closely with the the pivotal way of working on building software right so not necessarily setting up kind of a hard uh it's you know waterfall style statement of work or something but um having a a general kind of guiding light right um north star if you will but then having that fluidity that agility if you if you will if you must so it sounds like that would be kind of a secret success that could be passed on to others in terms of how they think of working with consultants

um whether it's alturos or someone else is kind of looking for the ability to engage in that iterative way because it aligns to the iterative journey that you've been on with pks platform automation and has allowed you to kind of map to what those internal customers your developers really need and build out a platform that that meets those needs and and so and not to put too fine a point on it but it would be great i think we could reiterate the value that platform automation delivers to your customers i.e the developers um you know you talked about obviously there's some security benefits um but i think you know what about things like you know getting new capabilities new features and pcf out to them quickly i mean if you could if you had to you know tick off the top two or three benefits to developers for really automating platform operation what would they be i i i i think for one big thing is uh it makes us very um uh efficient when when a developer team comes and says you know we have a new product coming out a new phone's coming out a new model uh we're going to need you know three clusters and we need them tomorrow that's something that's actually realistic and we could spin up and deliver to them um with with with our solid configuration proven and um with their namespaces and give them give them the keys and say go ahead and go and and to to add to the uh uh to that at the pks layer we almost did like all almost nine upgrades coming from 1.1.5 to 135 now and we have used platform automation and that and those upgrades include new features that we would need in in uh pks style and that eventually the customer would uh need it or those feature uh would help us out in in some Kubernetes cluster configuration so i can't imagine doing those upgrades manually from 1.1.5 to one three one three five that we are currently right now on on uh on almost uh uh 24 uh pks foundations right and and ultimately again i'm a

i'm a very concrete person so let's let's take that one step further benefits for the developers so how does that benefit um t-mobile's actual customers who are you know uh have phone plans and you know how does how does this all translate to value for them in the business well well for one instance i'd say about a month and a half ago two months ago a cve came out that was that was quite important that could expose a lot of data and customer data for that matter on some of our sensitive environments and we were able to upgrade all of our environments very quick and uh without much interruption to anyone and that cve was behind us where in the old days we would have had thousands of boxes to patch would have had to put a group together to make sure everything was done and all of that uh this was literally running five to six pipelines and we were done so right that's one nice thing when when a security bulletin comes out or something needs to be patched immediately um this allows us to do that quite well yeah that's a great example obviously security is you know top of mind for a lot of people a lot of consumers uh and so this you're able to keep them safer more quickly uh and that's that's obviously valuable to the customer and that's hopefully gonna you know that provides a somewhat of a differentiation for t-mobile uh to to you know retain customers and win new customers so uh that's a great example fantastic and i think we should also link in the show notes here um your colleagues uh james and brendan presented at kubecon maybe six months ago on you know another sort of dimension to the journey towards building the the on-premise kubernetes platform um and they touched on kind of another not so much on the platform automation side but on some other dimensions to that so i i think that would be a good compliment for folks to check out but that was six months ago so i know there's been probably some updates and changes and it does sound like the platform automation story has come from a long way uh so that is very cool to hear thank

you so much yeah absolutely and uh yeah as we wrap up i mean what's what's next on your agenda uh in terms of you know maybe next six months to a year what's what do you hope to accomplish let's say we we have you back on the show uh this time next year what what do you think we'll be talking about small more more more and more automation minimum uh minimum manual work uh for us for us for our customers and making sure uh uh making sure we are uh we are up to date and um on our on our customer providing all the features um in automated fashion uh that would be uh our goal and uh that's our um company goal as well i'm making sure um we are 99 automated across the board on on pksdk land yeah but we'll be much larger we'll be definitely much larger there'll be uh there's more more infrastructure coming in more foundations being built more clusters being built more containers everything and uh just just to add to that as well um we've been working on a pks reporter tool um that runs and collects information from all of the different pks foundations all the different uh kubernetes clusters that are running there the different plans the different sizes what kind of workloads are on it and from that reporter we're trying to output that data in a meaningful way and generate an automatic architecture diagram which basically gives us a high level overview of all of our environments what versions they're at what kind of state they're in um similar to monitoring but more of the architecture diagram um we've been working with uh fabric js to kind of draw that out um patch together from some json and then we can export that as a png image or uh send it to as a report to stakeholders and uh basically share it out to developers to kind of show hey we're at the latest and greatest features um you've got all the benefits of kubernetes with your scalability and uh you're you can basically run

anything um on on these clusters because they're underutilized and so on and so forth so that that that's something that's in the works at the moment uh and hopefully it'll be of uh benefit once it's complete that sounds cool can i can i sneak in maybe one more question which is all this automation that you've built around kubernetes you're building around pks you're not building around open source kubernetes um why what's the difference that from your perspective why is it why have you chosen to do this around pks and not just open source kubernetes i i think that um going with pks allows us to um have you pivotal do all the research when new things are coming out right you have a whole team that um is building your your technology as new technology comes out um you're able to soak that in and maybe a month or two down the line once pivotal gets their pks package with the new kubernetes versions out and the new new bells and whistles um is delivered to us rather than have us have a whole team that is dedicated to research and development or research and you know of all the new things that are coming out with kubernetes we kind of put that on you and then it's delivered to us and and also give us gives us flexibility to focus more on automation and our customers customers definitely yeah okay that makes sense you know you want to be if you have a team of doing r d instead of having them do the same r d that it's probably happening in other places have them focus on what what matters at t-mobile okay and and our our uh our mts uh mts uh mts i james webb and and um um our principal on team just here see uh these guys work really closely with pivotal to bring in new features and we have a um

Tuesday call every day every week with pivotal to kind of go over what we need and uh what are features coming in in new versions so uh that's kind of nice and i think as well is that kubernetes is is a very complex system like it's rapidly uh being developed like there's new features every other day and there's like a massive community behind it uh putting new features in and we really need somebody to check those features are production ready um i believe and maybe you can edit this out or whatever but t-mobile um tried to do open source kubernetes before um and it was just too much to manage it's too complex and that that's why at altaurus we recommend pks is because uh managed kubernetes is is much easier to run especially when you don't have a full team of hundreds of engineers to kind of actively develop and work on it and troubleshoot it um it's just too complicated um so for for enterprises that are not purely technology focused um and want to stay on the cutting edge technology uh a managed kubernetes solution is always the best approach in my opinion okay thank you well said thank you so much for for coming on the show it was really great to hear about um you know the the the journey that t-mobile has been on around building out the platform automation for the the pks environment um and how how far along you guys have come in a relatively short amount of time with really building out a very developer centric platform uh at t-mobile so congratulations can't wait to hear more absolutely thank you thank you

Tanzu Talk · 🍕 Get more pizza.